Examine each attack vector.
Physical: This one is pretty straight forward. Is the server in a rack? is it locked? who has the key? How secure is the key? Is the server is a room? Is the room locked? Usb ports are accessible or enabled? etc.
If it is a switch are you using port security? etc.
Network: Less simple, but the most common. Test your ports by scanning all open ports on the server with nmap or wireshark or something. Determine how restricted you want those network services to be depending on how you want them to work and how vulnerable they make you.
For example; a http service, restrict who has access but subnet? by host? by user? Check common vulnerabilities; is indexing turned on, etc.
Human/Social: This is something that isn't easily fixed though IT, normally HR. But here are some things you can do. Password policy; The old school of thought is to change passwords regularly, but this seems to force end users to choose weak passwords. Generating passwords for users tends to make them write them down. You will need to find a policy that best suits you, but you may want to go with educating end users on how to make a strong password, but make the password change once a year or never.
Also educate end users on how to manage passwords, and keep them secret.