11

When a server gets rooted (e.g. a situation like this), one of the first things that you may decide to do is containment. Some security specialists advise not to enter remediation immediately and to keep the server online until forensics are completed. Those advises are usually for APT. It's different if you have occasional Script kiddie breaches, so you may decide to remediate (fix things) early. One of the steps in remediation is containment of the server. Quoting from Robert Moir's Answer - "disconnect the victim from its muggers".

A server can be contained by pulling the network cable or the power cable.

Which method is better?

Taking into consideration the need for:

  1. Protecting victims from further damage
  2. Executing successful forensics
  3. (Possibly) Protecting valuable data on the server

Edit: 5 assumptions

Assuming:

  1. You detected early: 24 hours.
  2. You want to recover early: 3 days of 1 systems admin on the job (forensics and recovery).
  3. The server is not a Virtual Machine or a Container able to take a snapshot capturing the contents of the servers memory.
  4. You decide not to attempt prosecuting.
  5. You suspect that the attacker may be using some form of software (possibly sophisticated) and this software is still running on the server.
6

10 Answers 10

6

If you're facing an APT, then your best option is to set up a honeypot and thoroughly investigate all traffic that flows into and out of it, in addition to monitoring the server.

The measure of going through memory is so expensive in terms of time and effort that it's usually not worthwhile unless you've tried every other method, and if you determine that it's worthwhile, it's generally best setting up a honeypot that allows you to easily dump the memory and system state to another machine on the fly so you can do analysis with less threat of being detected while the machine is up and running.

I had one situation where the attacker kept everything in memory to the degree that, except for logs, the machine looked exactly like its image once powered off and back on. They would then hack back in and start using it again because the vulnerability was still there - they didn't need to leave any backdoors for themselves. A memory evaluation could have helped here, but watching the traffic was enough in this case to identify the vulnerability quickly.

Therefore:

The only reason to avoid pulling the power and doing offline disk evaluation is if you're going to go through the pain of doing a thorough memory analysis of the threat while it's in place and operating. If you've gotten to the point where this is necessary, then there is no reason to pull either plug.

If you're not doing a memory analysis, then pulling the power plug is your best bet - pulling the ethernet (or using a shutdown command) is only going to give the attacker's software advance notice - which does matter occasionally.

So:

Pull them both, unless you're doing a memory analysis, in which case, don't pull either.

17

RAM forensics (e.g. /dev/shm) can be helpful.

But I prefer unplugging the power cable (but try to log-in and rsync /proc right before).

The reasons for going for the power cable are:

  1. When you do forensics in a hacked system, you are "stepping all over the crime scene"
  2. The root kit keeps running - not so hard for the malicious to execute something (e.g. system wipe-out) on Network Link Down event.

Kyle Rankin gave a nice Intro to Forensics talk - there he recommends pulling the power cable.

1
  • +1 for "stepping all over crime scene". Unless you know exactly what you are doing, or don't really care to collect exact forensic evidence, then you may want to call in the experts rather than pollute the evidence as you learn how to do forensics...
    – dunxd
    Jan 3, 2011 at 23:57
11

Disconnect the network. The attacker can't pull any additional information if there's no network connection. It's very hard (read: impossible) to do any forensics without power.

3
  • 3
    You can (and probably should) do forensics from off-line (A) via live CD, (B) by moving the hard drive to a different system, or (C) after taking images of the affected hard drives (via live CD). Jan 3, 2011 at 20:43
  • 3
    Often useful evidence is in memory. This needs to be allowed for before any power loss is advised.
    – Sirex
    Jan 3, 2011 at 20:50
  • Also think to disconnect the LOM interface by safety :)
    – Kedare
    Jan 3, 2011 at 21:37
7

In this day and age, it could a Virtual Machine so either method is easy and can even be done remotely. (Virtual machines give rise to the possibility of using snapshots, too, of course)

I'd suggest disconnecting from the network as an automatic first step, because this gives you time to ponder what the next step should be, whether that next step is yanking out the power cord or something else. If you know your corporate "security response procedures" won't allow you to spend time digging through the machine in detail then preserving the contents of RAM might not be that important.

I'd suggest that getting "separating the victim from it's muggers" done in any way is more important than the how, so both approaches are valid. I'd have no problems with pulling the powercord myself, lets put it that way.

3
  • +1 for the VM. If you unplug the network, the rootkit keeps running - not so hard for the malicious to execute something (e.g. system wipe-out) on Network Link Down event Jan 3, 2011 at 21:08
  • 6
    Snapshots of the running system state is such a great idea I'm pissed at myself for not thinking of it. Jan 3, 2011 at 21:44
  • @Alexsandr - I've been trying to think of the actual example and I'm drawing a blank, but I seem to recall a rootkit that stays entirely in memory so would be lost when the plug is pulled. I think it's a case of either way there's a risk of losing evidence.
    – Rob Moir
    Jan 4, 2011 at 11:32
6

This isn't an either/or situation. You generally want to do both -- you want to perform certain forensics (dump of running processes, listening sockets, files in /tmp, etc.) on a system that's been removed from the network, and then perform your remaining diagnostics from a safe environment (i.e. a live CD). There are situations where neither approach is the right one, though, and you need to think about and understand what those might be in your organization.

2
  • If you unplug the network, the rootkit keeps running - not so hard for the malicious code to execute something (e.g. system wipe-out) on a Network Link Down event. Jan 4, 2011 at 8:34
  • That's definitely true, and it's a chance you have to evaluate and decide if you want to take depending on the sensitivity of the situation. Some rootkits are only resident in memory, and if you kill power to the system, you eliminate any chance of figuring out how it got there. You might try blocking traffic out of the switch port while keeping link state, but any rootkit is software and can do anything any other software can do -- there's nothing keeping them from probing the default gateway and setting off the same bomb if it's unreachable. Jan 4, 2011 at 14:18
4

Before you do anything, figure out if you need to preserve evidence for an eventual prosecution. Evidence handling is a very tricky subject and not for the faintly-trained. Once you've answered that, then the trained computer forensics person can take it from there.

3
  • 1
    First, I think one should decide Prosecute or Not?. Kyle Rankin in his talk (goo.gl/g21Ok). Starts by discussing this. To prosecute one would need to provide evidence of substantial monitory loss. Jan 3, 2011 at 21:12
  • 1
    @Aleksander Generally you need to know very, very early in the process whether or not you can re-use the compromised hardware and data or if you'll have to warehouse it and build a new system from completely new parts. This is very likely before you've even proven financial or business reputation losses. Preserving the evidence chain needs to happen the moment someone realized a potentially actionable event has happened.
    – sysadmin1138
    Jan 3, 2011 at 21:21
  • Often best to begin with keeping options open re prosecute or not, as you may not know whether monetary loss exists, so in the early stages doing everything by the book, keeping chain of custody is important.
    – Rory Alsop
    Jan 4, 2011 at 12:14
3

No need to power off the server. You can just disable the connectivity from the border gateway / router. One firewall rule will be enough to discard any further sent/received packet.

1
  • +1 That's one reason why a having gateway is a good idea. But what if you don't have one? If you unplug the direct network cable - the rootkit can receive the Network Link Down event. And isn't logging-in to the rooted server and doing something there on day 0 a bad idea? Jan 3, 2011 at 20:48
2

The answer depends largely on what you mean by "rooted". Pulling the network lead is usually a good idea, particually if you believe this was an active human attacker looking for sensitive information.

Pulling the power is much harder to answer. If you feel there's malicious code running which may cover its tracks, do so. However if you feel there may be evidence of a break-in still in the memory, leave it running.

Overall, it depends if you're dealing with malicious code, disgrunteled staff, or someone with a specific target in mind.

If erring on the side of caution, pull the power. You'll lose a small amount of potential evidence, but may prevent the massive removal of other evidence by automated code.

-- Then again, if you feel the machine has been compromised and you know it has no sensitive data on it, you're very confident of your network security elsewhere and understand the consequences of doing so, perhaps get a forensic boffin in asap and see if you can trace or understand the attack and go from there. That's not normally a good idea, though

1

My answer was before the edit, with the 5 assumptions.
My assumption was that if your server got rooted, you're dealing with a sophisticated, targeted attacker, and that you have no way of knowing when and where the attack came from. (Since the machine was rooted, obviously you cannot trust anything it tells you. However, you might have some off-box information, e.g. IDS...)
I assumed also that you have interest in dealing with your attacker, and not just waving him off like a bothersome fly. If the assumed attacker is a script kiddie, this would be different. I also assumed that, since the attacker is targeted and sophisticated, it's likely that they have custom software running on your machine, that can respond to your actions on it...


Neither.
Check out https://security.stackexchange.com/q/181/33, either way its a bad idea.
It's like giving a couple of bandaids to the black knight... too little, too late, it just ain't gonna help much.


For all those who feel that this answer is too far out there, or just not "security-conscious" enough - you need to realize that it's already too late.
It's not your server anymore, even if you completely disconnect it. Even if you shut down, run all your AVs and whatever - you don't own it anymore, they do.
That's kinda the definition of "rooted".

Now, assuming that they own it, and can hide their ownership from you, you need to consider - what will disconnecting it acheive?
The only thing it will acheive - since it will continue to be rooted regardless - is to let your adversary know you're on to them. Which simply puts their guards up, and gets them to start cleaning up after themselves (if they haven't already), which will just kill any hope of forensics.
You're still not protecting that server, or any of its data. How long has it been rooted? How would you know?

What you're better off doing:

  • Leave the server up and running...
  • Isolate it from the rest of your internal network, other servers, etc - but best to plug in some kind of simulation, so it seems connected.
  • Quietly start realtime / network forensics, run traces, etc.
  • Try and figure out the extent and length of the damage (obviously its different if it happend 2 hours ago, or 2 months ago).
  • Continue from there... Only after mitigating the actual damage, go ahead and clean it up.
  • Of course don't forget to investigate the root causes, and enforce whatever controls to ensure it doesnt happen again...
4
  • 2
    I think this can work in a situation where you can rapidly move it to a simulated/isolated environment, but so few organisations can do this that in reality they want to go for the 'minimise impact' side of things, so unplugging power or network is often the instant removal of that threat. (still +1'ed you as I wish it was like this :-)
    – Rory Alsop
    Jan 4, 2011 at 12:16
  • @Rory problem is you can't minimize impact, the server is already pwned. You ain't getting that back, and any sensitive data is potentially stolen too, since you dont know how long they've had access. That said, the simulated environment is just icing, it's the isolation thats important - and I'd believe most organizations have a solid firewall in place, flip a few access rules and you're gold. That's another reason accessible servers should be in a DMZ - makes that part easier....
    – AviD
    Jan 4, 2011 at 12:45
  • Also, I want to make something clear - the above applies when the OS is rooted. If there is some form of application level vulnerability (e.g. SQL Injection) being exploited in your website, ABSOLUTELY shut it down and pull all the wires you can get your hands on! But OS root is different - they control the entire infrastructure of your machine. You no longer have any control.
    – AviD
    Jan 4, 2011 at 12:54
  • 2
    no, what I meant was minimise the impact beyond that rooted machine. I agree it is totally lost, but unless you can get it isolated quickly, you have no way of knowing what the controller might do to the rest of your systems
    – Rory Alsop
    Jan 4, 2011 at 14:58
0

Having reviewed your assumptions do both. Use a CD/DVD based media to dump the current state. Primarily you will want to be able detemine how you were compromised. You may also want to attempt to recover user data not on your backup images. Yo

Then rebuild the system from the latest backup media that is not contaminated. Verify this with checksums if possible. Alternatively, reinstall the software from installation media, and reconfigure. Reconfiguration should be automatic if you have been using Puppet or cfEngine to configure your server.

Reload user data and scan for root kit components. Verify you have no setuid or setgid programs in the data directories.

Determine and close off the method of access used to infect the system. Stage reactivation of the server allowing time to verify applications are running as expected. Carefully monitor for new infection attempts.

This all assumes the infection is at root level. Web infections can be done by altering the code run by the web server. Allowing the web server write access to its code may make this more easily done. You may still want to treat this case as if the root account has been compromised.

If root on one system has been compromised on one system, you may have more systems compromised. Carefully consider which systems can be accessed without a password from the infected system.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .