I need to reconfigure a prod env to provide hot failover for a db driven web app written in Perl that is currently running on an Apache HTTPD server, on a Linux OS.
The app is not designed to run in a cluster. Thus, if 2 instances of this app are connected to the same db and exposed to users via a NLB it would result in disaster (a logically corrupted db).
I would like to configure different db credentials for two app instances put under an Apache mod_proxy NLB installed on a 3rd machine. Normally, instance 2 would be shutdown and instance 1 would handle all the load. When something goes wrong with instance 1, I would need a script to send a shutdown command to instance 1, connect to the db and disable instance 1's db credential, restart the db server and finally start instance 2. If anything goes wrong in this chain of events I would be happy for the app to remain unavailable to users.
The problem is that I don't know of an easy way to configure this hot failover scenario. I would be happy to glue a shell script to mod_proxy but don't know if that's possible.
Can it be done without configuring an additional monitoring system that would detect the problem and trigger the required actions?