As I'm going to change the network configuration of a remote server, I was thinking of some security mechanisms to protect me from accidentally losing control on the server.
The level-0 protection I'm using is a scheduled system reboot:
# at now+x minutes > reboot > ctrl+D
where x is the delay before reboot.
While this works relatevly well for very simple tasks like playing with iptables this method has at least two drawbacks:
- It's not very reactive, ie a connectivity problem should be detected automatically if for example an automatic remote ssh command fails does not work anymore for x seconds.
- It can obviously not work if one need to modify some configuration files and then reboot to test the changes.
Are you guys using some tool for the second point ? I would love to have something able to revert the system configuration in a previously known stable state if I can't join the server X minutes after reboot.
The server is a remote Linux server, with either a Debian-like or a RHEL-like distribution.
I only have acces to this specific server, behind a firewall. All ports are filtered, except port 22 (ssh). So no KVM switch, no iDRAC, etc.
I can have local support on this machine in case of a critical failure but this requires far too much time: it takes three hours to get there by car. And I perfer spending this time on serverfault or developing my own tools to avoid going there.
my actual plan: develop some ugly tool based on mercurial or git and calling a "hg revert; reboot" in a cron. I just wondered is some well tested tools already existed.