On an existing iPlanet webserver, we now have to install PHP5 and MySQL 5 on Red Hat Enterprise Linux 4. Are there any recommended guides or best practices for safely installing these 2 tools and hardening them from hackers on this public facing server?
First on my list would be to pick a platform which has full support from the distributor - not just legacy support (RHEL4 is now five and half years old and about to go into 'production 3' support mode).
Next on my list would be NOT INSTALLING SOFTWARE WHICH VOIDS the support warranty.
Then I'd have a long hard look at hardening the Operating System - but since you only asked about specific applications I'll skip over that.
Regarding hardening the 2 services....the very basic introduction to Linux hardening from Sans is 15 pages - so I'd strongly recommend you look elsewhere for answers - a quick post on SF will not provide you with a fraction of the questions you need to ask. There are some more guides here.
Do have a look at the suhosin website - even if you stick with the standard distributions of PHP there's a lot of information about why suhosin exists from which you can make informed decisions.
As for mysql - it certainly should not be public facing. In addition to firewalling, you should be disabling all network access if possible - or moving it to a separate server not routable from the internet.
Similarly for mod_security
First of all, note that doing this will basically make the server unsupportable by Red Hat, so you're on your own if trouble comes up.
Now that that's out of the way, you'll find MySQL 5 and PHP 5 packages for EL4 in the CentOS Plus repository. Despite the name, these packages will work on both CentOS and RHEL.