Both Mayweather and Pacquiao were skewered by the media and fans for taking so long to finally get into the ring with each other. They hyped that fight up with each other and talked so much shit for years, and the "superfight" (LOL) almost didn't happen, and when it did it Manny was out of his prime and people didn't really care anymore. Of course spectators can be involved in stuff like this, even if they don't have money on the line. It is a very common and natural thing for people to "blow up" sports figures if they don't do their jobs or fail to back up their talk.
If you are going to claim that someone is "free" and is not a top player and disparage each other as competitors, you better be willing to actually back it up. Words are wind. So you are absolutely right, they don't have to go through with this money match. That is their prerogative. But to say that its "ridiculous" to blow them up for not playing after everything they've said to each other, is silly to me. That's a very natural human reaction when somebody talks the talk but doesn't walk the walk.