I've got kind of a situation here.

We need to get 2 servers under our control. The things we know so far:

  • both are running linux, at least one is running Red Hat
  • they are used as file-server, mysql server, and probably there's a sip server
  • they are running apache and phpMyAdmin 3.4.1
  • It is said that they are set up to mirror their data every 30 minutes
  • the previous administrator said, that they are impossible to break in, and still have a working system.
  • we don't have root passwords or any accounts on these systems

The previous administrator has been fired and has to be considered evil. He could have remote access.

One of the servers is no longer working correctly, and the old administrator said, if one of the servers is switched off, the whole system would break.

We don't know, how good he really is.

Our current plan: - switch off the servers anyway and make a full image. - put an additional firewall around the network - start one server up again - configure the firewall so that only sip traffic is let through

So we hope to get a working system, where the old admin is unable to remotely connect.

Then try to get into the other server to get to the configuration, and really gain control.

Do you any additional ideas? Do you see weak points?

Any thought welcome.


--- Update ---

Neither putting a firewall around the system, nor makeing an image of it had been accepted.

They tried to force us to just try to crack the password without any further precautions, so in the end we walked away.

The accepted answer is what we would have done.

  • 4
    Hire a pro :) I love situations like this. But shouldn't this have been a consideration before the admin was fired? Usually hold severance in exchange for the admin's coöperation. – ewwhite Jul 31 '15 at 13:25
  • 2
    Evil previous admin -> rebuild systems from scratch to ensure they are not compromised. – Thorbjørn Ravn Andersen Jul 31 '15 at 13:40
  • Consider migrating them into a virtual machine you can control better. – Thorbjørn Ravn Andersen Jul 31 '15 at 13:40
  • @ThorbjørnRavnAndersen Building up from scratch was my first reaction. But at the time being the powers that be don't follow me there. – ingo Jul 31 '15 at 13:45
  • @ewwhite Sadly now is too late for before. – ingo Jul 31 '15 at 13:49

This is how I would do:

  • First I would consider the system compromised. This means I would not trust the data, configs, etc unless is carefully checked.
  • I would prepare a live CD and an empty HDD (can be an uncompromised network storage)
  • I would reset the system (not system reboot, but reset button) and I would make an image of the disk. This image should be made read-only, or you should create a backup of it. The image can be used for both forensics and data recovery.
  • I would use a clean trusted VM where I will mount the partitions and I will scan with antivirus all volumes used for file sharing.
  • Even if you have backups you should not trust the data.
  • Install the services on new clean servers, then move the data as you review it

Since you're going to shut down the servers, boot one of them with a live CD/USB key, mount the root filesystem and chroot into it.

Then change the root password and check if there are other users presents and change their password too.

Then you can reboot it, connect to the network and procede with the same procedure with the other server.

  • 1
    <Smacks head> Thanks, I should have thought of that myself. – ingo Jul 31 '15 at 13:50
  • chroot works, as well as simply editing the shadow file to insert new passwords. – Hyppy Jul 31 '15 at 14:19
  • 1
    If sshd is running make sure you remove the keys or he'll be able to get in even with a changed password. – william Jul 31 '15 at 15:15
  • That... really won't be enough. There are any number of other backdoors into that system. However, that having been said, you don't need to start completely from scratch; if you boot from LiveCD / LiveUSB you can transfer important files from the hard drive to another drive (if there are multiple USB ports) and begin piecing things together. A hybrid between starting from scratch, and simply using the old, potentially malicious stuff. – Parthian Shot Aug 1 '15 at 6:17

No need to boot up with a live cd as suggested by @Mr Shunz, since the system is working. Just restart the server, in the boot manager select to edit the boot parameters and add an 1 to the end. The system will boot in runlevel 1 with no network and the root user already logged in. Eventually disable system management programs from starting up automatically at startup, so that things like puppet or chef won't eventually enforce particular users or passwords on the system. Change the root password, restart and then do your investigation on what's on the system using the root user.

  • ...If they were using Puppet or Chef, I'm thinking this wouldn't be an issue to begin with. He could just copy the database image and he'd be good to go. – Parthian Shot Aug 1 '15 at 6:18
  • @Parthian Shot, why no issue? They don't even know what is running on the server. And there might be a local cron job, init script or even login script enforcing some settings, for example applying a local puppet manifest. – stoned Aug 2 '15 at 7:27
  • I suppose so. I guess what I meant to say was "if they were using Puppet or Chef correctly, than the config files and whatnot would be in source control". But you're right the admin could've been using them without anyone knowing. – Parthian Shot Aug 3 '15 at 15:18

This answer will depend on using Grub and rebooting the server, but since they're Redhat servers, Grub is practically guaranteed, and it is a sure way to regain access without booting to another medium. Here goes:

  1. Initiate a reboot. When the bootloader screen pops up, spam the Escape key to stop automatically booting to the fefault kernel.

  2. Using the arrow keys, highlight the kernel you want to boot, and press "e" to edit the line.

  3. Append the following to the kernel line: "init=/bin/bash"

  4. Hit the "enter" key to accept the changes.

  5. Make sure the line you edited is highlighted and hit the "b" key to boot to the edited kernel line.

  6. This will boot into a shell as root before the rest of the OS loads; the filesystem will be mounted as read-only.

  7. You can now use "passwd root" to change root's password. If you find you need to write to a file while in this state, you can execute "mount -o remount,rw /" to remount the filesystem as read/write.

  8. If you have remounted the filesystem as read/write, use "mount -o remount,ro /" to remount as read only. Halt the system using shutdown -H and then physically reboot the server.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.