I'm a student volunteer helping to maintain the web server for a small website, along with a few other students. It's a pretty standard LAMP server running on Ubuntu 12.04 and mostly hosts PHP based sites.
There have been multiple times recently when we've wanted to make a minor config change, one that should take a few minutes at most - for example, changing permissions on some website files to improve security, updating the SSL certificate, making slight modifications to config files, etc. - and accidently mess up somewhere which brings down the entire site. This usually results in a frantic 15 minutes or so afterward trying to get the site back up again.
Fortunately, a few minutes of downtime isn't the end of the world as it's a small, non-commercial site. However, I think it's only a matter of time before we mess up something that cannot easily be fixed, and that could result in some pretty serious issues. Of course we do have backups as well as a secondary server but restoring those backups can still take a significant amount of time.
So here's the question I'm wondering about: Is there a good way to test configuration changes before they go live on the server? I thought about setting up a secondary server configured exactly the same way as the primary server so that commands could be executed/tested on the secondary server first, but I think it would become very annoying to make sure the two servers are always in sync. Is there a better way to do this, short of virtualizing everything and working with snapshots?