You can certainly remove gcc. In general, if you find some package you don't want, you can just remove it with
sudo aptitude remove packagename
then if it turns out the package has some dependency that you want, just answer 'n' when it asks whether to proceed.
That being said, IMHO removing the compiler gives a false sense of security. If a skilled attacker gets shell access, you've already lost most of the battle anyway. Secondly, not having a compiler available maybe made sense when servers were running on architectures not generally available in PC's, were running some proprietary Unix variant etc. Nowadays an x86 PC with Ubuntu isn't exactly hard or expensive for an attacker to get his hands on, so if the compiler isn't available he can just copy over whatever binaries he wants.