I honestly don't think it's worth the trouble. Yes, it provides an added level of security, but at significant cost. Security isn't an absolute; all you have to do is be better than other targets that are at a similar risk of being targetted (or that have a similar benefit to the attacker if they're compromised). Given the number of servers that can still be popped open using some simple password brute-forcing, trivial web application vulnerabilities, and massively out-of-date patch levels, if you're actually do all that stuff, then you're so invulnerable relative to millions of other systems that your risk of intrusion is already so low as to be not worth any additional faffing around.
This is, of course, assuming that your risk analysis centers around untargeted attacks. If you're likely to be a target of a particularly skilled or motivated attacker (either because you're a valuable target, like a bank, or people particularly want to take you down, like Sony) then you might want to look at SELinux, or some other niche security products.