11

There were other devs logged in as root that I wanted to kick out.

So I issued

pkill -KILL -u root

Then I realized that I probably killed all kinds of root processes.

Have I damaged my system? I can't reach it through SSH now.

UPDATE: Web server is still running. But I can't connect by SSH now. I have no idea what I've done.

11
  • 1
    You are almost certainly going to need to reboot it. If you can, send it an ACPI shutdown signal instead of just pulling the plug.
    – Skyhawk
    Jan 3, 2013 at 7:00
  • 1
    Iain, IME that's not entirely true. They have to start as root, to bind to the privileged ports, but then spin up a whole bunch of processes owned by whatever non-priv user is configured, for privilege separation. In the case of apache, though, there's usually still a single root-owned process sitting around at the head of it all. But I agree with you that all the processes doing the actual web serving are not root-owned.
    – MadHatter
    Jan 3, 2013 at 7:22
  • 7
    Making mistakes is the best way to learn. Please don't downvote simply because someone made a bad mistake. This question is controversial but should not be closed. Instead I challenge you to read more deeply into the thought behind the question and explain: What happens when I kill all root processes? Why is the host still alive? Why can SSHD not serve requests but the webserver can? How is it possible that the webserver is alive at all? We don't seeem to have a good canonical answer to these sort of questions. The answers could provide valuable explanations about how Unix works. Jan 3, 2013 at 19:05
  • 3
    Thank you Stefan, for using your head and your heart and not compulsively clicking the downvote button like so many others do. If no one ever posts questions about their mistakes on stackexchange sites, for fear of being downvoted, then NO ONE will learn from them. I am helping people with my question, and should not be punished for it. Jan 4, 2013 at 3:42
  • 1
    I up-voted the question because it was good for a laugh, no offense. It's like watching a video of somebody falling into a hole while texting.
    – UncaAlby
    Jan 2, 2018 at 22:50

3 Answers 3

16

A quick answer is that you killed sshd (and lord knows what else) and will be unable to log back into the system using SSH. Unless you have some other method of gaining access to the system (such a remote console, IPMI, etc), you will need to reboot the system which will restore the SSH service and other services.

Hopefully you have physical access to the box, in which case you probably just need to hit the power button. Realize that you killed many processes and be prepared for some corruption. Linux is designed to recover from a system crash, and you essentially triggered a 'manual' crash. Most things should recover fine after a reboot. You may have all sorts of interesting error messages in the logfiles.


Long answer:

This is a great thought experiment and a good job interview question. "What happens if you did X..." This is a fun thing to try on your own private virtual machine, but should never be done on a real box. Everyone makes mistakes. Remember and learn from your mistake. Making mistakes is the best way to learn. Making mistakes on production is a painful lesson that will happen occasionally in your career.

pkill -KILL -u root

This command will send a 'SIGKILL' (e.g. kill -9. KILL is an alias for SIGKILL) to all processes owned by root. It is a very bad thing to do on a system. kill -9 should be avoided except as a last resort.

Your command aggressively killed all process owned by root, the processes were killed immediately and were not given a chance to clean up. To get a sense of what you killed, log into a healthy Linux box and list the processes owned by root, using a command like one of these. You typically do not need to be root to run these commands:

$ pgrep -u root -l
$ ps aux | grep root

You may have killed Init (PID #1) which spawns new processes. Your system may be unable to create new processes. So, it may continue to function for now but is sick and needs to be repaired as soon as possible. As time goes on, the system will get more and more sick. The longer you wait, the worse it will get.

UPDATE: Web server is still running. But I can't connect by SSH now. I have no idea what I've done.

I am guessing that you are using Apache. It appears that the child processes of the webserver are still running because they are not owned by user 'root'. However, the parent webserver process is normally owned by root and you killed it. As a result, new child processes will not spawn. This will be fine for a time, because you probably have enough child processes to serve requests, and typically those child processes will persist until they are killed or they crash. Again, the quickest fix is to reboot the machine.

3
  • I don't have physical access but I will figure something out. I was pleasantly surprised to see that Apache is still working. So the machine is doing fine without root. P.S. if you downvoted the question, I've improved the title. Jan 3, 2013 at 7:06
  • 1
    @ButtleButkus I did not downvote the question. I started answering the question, and then got quite interested in why things work the way they do. Jan 3, 2013 at 19:17
  • 1
    Stefan, thanks for your answer. It is the one answer I upvoted from the start, since it made sense. I got the system rebooted within about 10 minutes and everything seems to be working wonderfully since then. Jan 4, 2013 at 3:39
5

You will most likely have to restart your system as you have killed pretty much every critical service on it. How you do that depends on what tools you have or what transport you have to get to the data centre.

4
  • It doesn't seem like I killed all the critical processes, actually. Otherwise, why would the webserver have still been serving up perfect webpages? Jan 3, 2013 at 7:24
  • @ButtleButkus: Your web server will not be running as root.
    – user9517
    Jan 3, 2013 at 7:28
  • @lain If even a single "critical" service was killed, then I think that would make the entire server go down. It seems like the command actually didn't kill a single critical service. It did kill a convenient service, though: sshd. Jan 3, 2013 at 7:32
  • 3
    Just because a critical service is killed, doesn't mean the machine will go down immediately. I would e.g. consider my fan control daemon critical – when cpu usage goes up, I want the fan to spin more. I could kill the service and have low cpu usage for days, then suddenly 1000 people access my site and my cpu catches fire. And there are many more less noticable ways of doing damage …
    – unhammer
    Jan 14, 2013 at 7:48
4

The system is running because the kernel is running. You can't access sshd because you have killed the daemon. Probably init has been terminated too, meaning you can't create new processes. So, new apache connections might not establish (configuration parameters applied ;)).

You can't send a signal to the kernel threads, that is why the system is running but the root owned services have been terminated and for a normal resurrection, you ought to reboot it.

1
  • Thanks for the interesting details. I gave you an upvote, but I'm giving Stefan the checkmark. Jan 4, 2013 at 3:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .