Webmin is where most security questions would come into play, as it is where logins and such happen. Webmin has a very good security history, and its security record is public: http://www.webmin.com/security.html
It's been several years since the last serious root-level or direct data exposure exploit was discovered, though there have been a few XSS vulnerabilities in the past couple of years. No software of Webmin's complexity will be completely bug-free, including security bugs, but we do take security issues very seriously, and they get fixed quickly. I think Webmin core is about on par with OpenSSH in number and severity of vulnerabilities discovered in the past five years, and I think we all agree that OpenSSH has a really good security record.
PCI compliance is entirely possible in a Virtualmin system, as nearly everything related to PCI is provided by the OS (so if your OS is CentOS, then you'd take the same steps you'd take with a non-Virtualmin CentOS system; which isn't all that much). We have hundreds of users who have gone through the PCI compliance process.
Note that the PCI scanner is kinda dumb, and will flag the CentOS (or Debian or Ubuntu) standard Apache package as being old and insecure (and since our Apache build is just a rebuild of the OS-provided package with suexec docroot set to /home, ours also gets flagged)...but the OS vendor applies security patches, which correct security issues. So, you have to add an exception for that particular package. This is well-understood by the PCI folks, and you won't have any trouble from them over this; it'd be more dangerous to build Apache from source, get PCI compliance, and then forget that you'd installed from source. We've seen security issues from this kind of thing a lot over the years, so we definitely recommend you stick with standard packages whenever possible, so that normal updates via yum or apt-get will work (and Virtualmin has an updates notification module on the System Information page to let you know when you have updates, if you aren't running them automatically).
In short, I believe Virtualmin security is at least as good as the competition, though I'm certain no one has a perfect security record, since the target that the most popular products provide is huge. Webmin, cPanel, and Plesk are all prime targets for black hats because they have root privileges, and run on millions of machines (I know Webmin does, anyway, I'm not sure of the numbers for cPanel or Plesk).
And, since jeffatrackaid has gone to the trouble to bring up our competitors forums, I'll mention that Webmin/Virtualmin also has a very active community at http://www.virtualmin.com (and if you like the old school mailing list support process, http://www.webmin.com has the hookup).
Disclaimer: I'm a developer on Webmin and Virtualmin.
In what context do you need hosting automation? Is this for internal use, do you plan on selling services based on this solution?
For shared hosting automation, I prefer Plesk. I find it works very reliably and does not interfere with the underlying OS. Some people prefer cPanel as it allows much more point and click customization, such as choice of PHP version, choice of Apache, etc. Both systems are reliable if you use them as intended, but I like the ability to manage the OS via the default package managers (rpm in my case as I work on Red Hat and the like).
I've dealt with virtualmin before but found it rather cumbersome and not a polished product. This was a couple of years ago so it may have improved.
I think it really boils down to usage. If this is to be used for business purposes, managing important business site, selling hosting, I prefer: Plesk cPanel in that order, and then if you must: DirectAdmin Webmin/Virtualmin
There are a few other hosting automation panels out there but their market share is rather small -- this means a much smaller userbase if you hit an issue.
Plesk and cPanel both have active forum communities to get help and plenty of companies like ours where you can get paid support.