If your server has a serial port you can connect through it easily enough. Let us assume the simplest case; 1 server and a cart pc or laptop. Add an entry to your /etc/inittab file similar to:
s0:12345:respawn:/sbin/agetty 19200 ttyS0 vt100
That will tell the server to listen on ttyS0, the first serial port, for terminal like behavior. Plug a cable from that port on the server to your laptop/cart. Using a program such as minicom or the right options on screen connect to your local serial port. Assuming you properly configured the speeds you should drop straight into a login prompt on your server.
For the more complicated case, many different servers, you can centralize the same basic approach. Instead of a mobile cart or laptop, setup a specialized server that will be directly connected to all of your various devices. This can be done with a boatload of serial ports, or something like a Digi console server. From this console server you can then gain out-of-band access to any of your connected systems. I would check out applications like Conserver when working in bulk.
I would recommend, at least, configuring the serial port on these systems even if you don't plan on using an enterprise wide management system. You can't always trust that a box will have video, and at least with the serial port enabled you can work on a system that may not completely boot.
EDIT: After sleeping on it I realized this method will not work for virtual machines. Best case, using a serial connection you could drop into a shell on the VM host, but not any of the guests. We leave this problem as an exercise for the reader.