Specifically, I am asking about screwdrivers with removable tips (something like this), but I think the answer would apply to any magnetic-tipped screwdriver.
Will the magnetic field damage anything in the server?
I Googled this before asking, and the general consensus was "it won't hurt anything". However, the answers I found were from general PC guy forums, not the server community. Around high-dollar equipment, you certainly tend to be more careful!