I'd have to say that it depends on the type of racks that you're dealing with, and the servers. I actually carry a few tools (SOG Power Pliers, Gerber MultiPliers and a Latshaw Pocketwrench II). In my computer bag, I also keep for laptops and internal parts, a Husky HD-74501 W and an HD-74502 W. (small screwdriver and torx set, single handle with a total of 8 tips each, but very small; the torx goes down to T4, and they fit into spaces a 1/4" drive screwdriver won't). At my desk, I keep a monkey wrench, a socket set, a Stanley Wonder Bar II, and a screwdriver/nutdriver set.
The Pocketwrench is great for just holdings nuts while you tighten from the other side (and tightening rails from Sun where you have no clearance to get a wrench in there). The reason for the SOG and the Gerber is that they have different size philips bits -- the Gerber I use for rack screws, the SOG for removing cards and such. I also find that two plier-like devices also comes in handy when dealing with bolts, as the pocketwrench doesn't always give me the leverage I need for breaking nuts free. The Wonder Bar is the only thing people look at me funny for -- but it's a god send when you have to shift a server up slightly, so you can fit in something below it. (the pocket wrench works okay, but the Wonder Bar has much better leverage).
Oh -- and I've accidentally flown with the pocket wrench, and security turned my bag inside out looking to figure out what it was. As there weren't any sharp edges on it, they let me get on the plane with it.