It can all be summed up...
...in one word:
Style.
To answer your key question first, you are obliged to take no actions. Middle of the floor with no one using them, dumbbells are fair game.
But there are a few variables that you didn't provide. What kind of gym was it? How big? How many other people were there at the time? Is it the kind of gym where behavior like his would normally be tolerated? If you had whomped his ass, is it the kind of gym where that would have been tolerated? All of these are pieces in the puzzle that are needed to complete the big picture.
Now, for what it's worth, here's how I would have handled the situation (and, in fact, how I have handled similar situations successfully).
1. Never blow your cool, but never offer more of an apology than a quick, "Sorry, man."
2. If the other guy acts like a jerk, never take your eyes away from his eyes. Looking directly at a person who is making an ass of himself can be very intimidating in itself. It shows that you have the self-confidence not to avert his looks.
3. Regardless of the truth, make him think that you're able to kick the shit out of him - whether you are able to or not doesn't matter, what he thinks you are able to do matters. But never blow your cool.
4. Make him think that you're more psychotic than he is. But never blow your cool. Do it with style.
5. Do not hesitate to threaten him with arrest or expulsion from the gym. But once again, never blow your cool, and do it with style. Am I being repetitious? Yep. Because most of you who are reading this are immature hotheads that can't begin to fathom the consequences of a real fight. Except bad_man, who somehow managed to keep his cool.
6. I hate to quote Patrick Swayze in Road House, but I've never heard it put better. When you're dealing with an asshole, "Be nice. Always be nice." Do what you need to defuse the situation but, even if you have to kick ass (and in 99% of these case you should not have to do so), be nice.
If it gets to the point of potential physical violence, and only if it gets to that point, you have two choices. One is to walk or run, and the other is to face it down. If you decide on the latter, never blow your cool, and play it with style.
Remember that guys who want to fight or who are willing to fight are, by nature, fighters. And they're probably good at it. So here is how to defuse them. Say, "Look, bro. You're bigger than me [if that's the case], and you're probably faster and stronger. You probably know how to fight pretty well, and I don't. But that means that you're gonna be pulling punches, and I'm not. Because I'm more psycho than you. And I don't know how to pull a punch. You you have to think carefully: Is it worth going to the hospital? Because if you want to follow this through, you're gonna have at least one broken bone. No bullshit, bro. You need to think carefully about whether it's worth it."
While you're saying this, you are looking directly into his eyes the entire time. You are saying it quietly and with confidence (even though you might be shitting while you say it), and while you are overtly calm, you are communicating that your adrenalin is rising even as you speak. And when the guy backs down (and he will, if you've done it right), make sure you communicate that there are no hard feelings and that you're really a nice guy. Let him save face, because it's the right thing to do, especially if you are really a nice guy. And during the entire time, if there are others around, take all steps you can not to embarrass teh other guy, or you will have backed him into a corner (which you should never do) and he may become more psycho than you.
Here's a fact: Most of the guys on this board who say, "I'd kick his ass" are under 21, and they'll hopefully grow out of it. (If they don't, the most eloquent thing they will ever say in life is something like, "Would you like fries with that burger?") Those guys who threaten violence and who are over 21 are on their way to being such piss-ant, immature failures in life (not job, status, or finance, but life itself) who wre so insignificant that you don't have to be bothered with them. If you are better than they are, you don't have to be bothered with them. And if they force their way into your life, get them out in any way you can, first nonviolently, and then making sure you cover your ass legally (because they may actually be shrewder than you in the end run).
As for the gym, if you don't like the assholes, change gyms. Seriously, you can whine about them here, and you may even achieve a little catharsis (look it up). But they'll still be at the gym when you go back. So either have their butts kicked out of the gym, or change gyms. It's that simple.