I'm running windows server 2008 64, but all open-source stuff (mysql, ruby, python, R) is either not that well-tested or has problems. And of course, if you have to compile packages, you are on your own.
I guess it's obvious that FOSS will run better on linux; maybe OSX too. But there are wide differences in terms of distros. Which one has the best reputation for x64? This is for a desktop use (doing scientific simulation, needs lots of ram).
I don't particularly like macs, but could consider it if they have demonstrably better x64 support.