This can go in two ways as I read this; do you want thin clients connecting to a server, or do you want to physically connect this hardware to a powerful server?
Technically you can do either one.
There is a proprietary product that lets you chain via long cables multiple displays, keyboards, etc. from one physical box from the vendor's card in the back of the server.
The alternative is to run X with X-terminals or Windows Terminal Service (depending on platform) to use thin clients over the network to your server.
given that you want to use 3D graphics, I'd strongly advise against it. Hardware is cheap compared to the headaches you're asking for in supporting this kind of setup, and you'll have more weird glitches to deal with in sound redirection and multimedia and graphics performance. Thin clients and such are best for office work...web browsing, email, word processing, etc...even then we had people kill a terminal server by looping an animated weather map that consumed so much memory everyone else's sessions became molasses and they thought it locked up (leading to corrupted profiles as they rebooted the terminal, thinking that rebooted the machine. Um...no.)
Purchase separate hardware. You're wasting your time if you think terminals are going to give performance with graphics stuff as a primary requirement.
EDIT: just saw your update
You may want to figure out if you can instead set up your compilations to be farmed out; i.e., upload to a repository on a powerful server and do the compilation there. I don't know how big your compilation is, or if it's an issue with testing, but I don't know if you'd want to test on the same machine you're developing on in practice anyway (I think it's a best practice to not do so but that doesn't mean developers care about best practices anyway). I know big places like MS have farms that compile things nightly and that developers upload their changes to a central place to have the compiles done, then if someone breaks the build they get beaten mercilessly. It centralizes the code base and allows for backups and source code versioning, as well as offloading some of your concerns about memory and speed for compiles.
Testing the actual software product...hrm...still not sure I would like that if I were developing (or managing developers) just because if something goes wrong it could kill what work they've done or hamper productivity because some bug in code causes a major problem. Others may disagree but I'd feel better separating the code and development from the testbeds.