Will installing guest additions post a security risk to a host OS?
Probably. Definitely in vmware-tools. The copy/paste buffer and keyboard/mouse shared memory areas allow access to the host. If the virtual is rooted it's trivial to determine the type of virtualization used and hack/install your own client tools. AFAIK, only vmware has had example code to hack over these API channels and that's been patched for quite some time.
That being said the changes of you getting hacked this way are probably close to zero. I wouldn't concern myself until we start to see published hacks using these methods. Or rather, there's plenty of other things to spend time on securing first. Shared file access, not using a DMZ, improper firewalling, weak passwords, bad file permissions are all more important.
Always remember, social networking hacks, mis-configured systems and disgruntled employees and plain stupidity (selling data loaded harddrives on ebay, password taped to the monitor, etc.) are the top four entry points for most reported large system hacks.
Here's one, there have been others - VMWARE - A VMCI privilege escalation on Windows-based hosts or Windows- based guests.
OpenBSD has the right attitude. Don't install it unless it's absolutely needed. You can install the network driver from the client tools on it's own. "Secure" virtuals should not have shared file access, consoles, etc..
The guest is sandboxed. Guest additions are installed on the guest OS. I don't see how it's any more of a security risk than having that machine on your network as a physical system...I suppose you could come up with some scenarios of a more direct threat if you have host-to-guest drive sharing enabled.
Otherwise...why would you think it's a security threat?
There are some vulnerabilities in any software, and virtualization software is no different. However, the vulnerabilities are currently hard to do and little known or understood - thus not much of a threat. There is the threat that a user may try to "break" the virtualized sandbox; however, as I mentioned, this is currently not likely to occur.
You biggest risk is from the host OS or the guest OS being compromised, not the virtualization software itself.