After months of neglect, e-mail flames and management battles our current sysadmin was fired and handed over "the server credentials" to me. Such credentials consist in a root password and nothing else: no procedures, no documentation, no tips, nothing.
My question is: assuming he left boobytraps behind, how do I gracefully take over the servers with as little downtime as possible?
Here are the details:
- one production server located in a server farm in the basement; ubuntu server 9.x probably, with grsec patches (rumours I heard last time I asked the admin)
- one internal server that contains all internal documentation, file repository, wikis, etc. Again, ubuntu server, few years old.
Assume both servers are patched and up-to-date, so I'd rather not try to hack my way in unless there's a good reason (i.e. that can be explained to upper management).
The production server has a few websites hosted (standard apache-php-mysql), a LDAP server, a ZIMBRA e-mail suite/server, and as far as I can tell a few vmware workstations running. No idea what's happening in there. Probably one is the LDAP master, but that's a wild guess.
The internal server has an internal wiki/cms, a LDAP slave that replicates the credentials from the production server, a few more vmware workstations, and backups running.
I could just go to the server farm's admin, point at the server, tell them '
sudo shut down that server please', log in in single user mode and have my way with it. Same for the internal server. Still, that would mean downtime, upper management upset, the old sysadmin firing back at me saying 'see? you can't do my job' and other nuisances, and most importantly I'd have to lose potentially a few weeks of unpaid time.
On the other end of the spectrum I could just log in as root and inch trough the server to try to make an understanding of what's happening. With all risks of triggering surprises left behind.
I am looking for a solution in the middle: try to keep everything running as it is, while understanding what's happening and how, and most importantly avoiding triggering any booby traps left behind.
What are your suggestions?
So far I thought about 'practicing' with the internal server, disconnecting the network, rebooting with a live cd, dumping the root file system into a USB drive, and load it on a disconnected, isolated virtual machine to understand the former sysadmin way of thinking (a-la 'know your enemy'). Could pull the same feat with the production server, but a full dump would make somebody notice. Perhaps I can just log in as root, check crontab, check the .profile for any commands that's launched, dump the lastlog, and whatever comes to mind.
And that's why I'm here. Any hint, no matter how small, would be greatly appreciated.
Time is also an issue: there could be triggers happening in a few hours, or a few weeks. Feels like one of those bad Hollywood movies, doesn't it?