Physical Access to a machine == opportunity to root the machine.
Do not allow anyone into the server room who you do not want to give access to the equipment on the machine. Or, have physical access (along with KVM or other local/console means) to the controls of the machine restricted if you are going to allow others physical access to the machine room.
The best practice in my mind is to either prevent access entirely to non-admins, to provide a security escort while someone's in the server room who is not authorized for global access (i.e. vendors), or to keep key-locked hardware in place and restrict keys to subsets of authorized users/admins. The last part is the best-practice for most colocation spaces where you as a customer will rent space.
Also: If you have the opportunity, make sure you have an "airlock" system that requires two forms of access, which prevents "tailgating". In our case, these are punch and card-scan locks. The entry into the foyer requires that you punch a code into a lock. Once you're in the foyer, you need to scan an ID card to enter the actual server room.
Beyond just "It's really a good idea", there's certain industry specific SAP's, laws, or regulations that might be involved. In an educational or government institution, I have specific laws I need to be sure are enforced in regards to access to student information. Similar requirements exist for companies who are publicly traded; they must comply with SOX. The medical industry, or any industry that handles associated identity information along with medical history, must follow HIPPA. Any company that stores credit card transactions must comply with their merchant agreements, which are usually VERY explicit about what the machines are allowed to store and who has access to the machines. Your industry's mileage may indeed vary.