Think of Rack-Mount servers as your computer towers in a different shape/form. (Same guts inside (of varying power, capability, etc.) but different shape. Modern Rack-mounted machines are thinner, to take-up less space.) Essentially, they are able to be screwed into a rack, along with other rack-mount servers.
OS's are installable onto these computers. It's your choice, which OS you'd like to use. Usually, the vendor you've bought your rack-mount from will have a list of "officially" supported and compatible (with the hardware) OS's which can run on the server you've bought.
Your Rack-mounted server will have a display port (usually VGA) and ports for input devices like the keyboard & mouse (same as a normal Tower computer). In most racks which hold more than one server, a device called a KVM or KVM switch is used to connect all the computers to one screen/mouse/keyboard, so you can switch between the machine that you are working on, and only have one screen, mouse, & keyboard for them all. (Handy!)
You will certainly need an OS for a fileserver.