Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Today I have shutdown'ed a production machine by mistake because I thought I was on my local machine. I know, novice mistake :-(

As a solution for it not to happen again, I was thinking to remove the execution permission for the shutdown command as that machine should be always on.

Is that a good idea? Can you see any unwanted side effect by doing that?

Cheers, Dan

share|improve this question
1  
Most sysadmins do something like this at some point (which window was it again? whoops...) –  Bart Silverstrim Aug 9 '10 at 12:11
3  
Be glad it was only a shutdown. Other people learned that lesson with the dd tool (AKA: disk destroyer) ;) –  pehrs Aug 9 '10 at 12:39
1  
Most of the sysadmin duties & tools can be fatal and deadly. Great power comes with great responsibility. You may chmod 000 your shutdown command, but next time you might mess up with rm, dd, fsck, mv or some other potentially dangerous tool. Everyone of us WILL make mistakes sooner or later. The best thing you can do is to prepare for the worst and make sure you have backups etc :-) –  Janne Pikkarainen Aug 9 '10 at 13:05

10 Answers 10

Completely another aproach how to be warned that you work on productional machine is to mark the terminal. For example the user@machine:~# text could be red at production machines, green at development, etc. Here is nice tutorial how to do this: Color Bash Prompt

share|improve this answer
    
+1, I color code all my machines: White text on a colored background, Orange = Infrastructure; Blue = Production; Purple = Test/Dev. Normal workstation keep the standard black background. –  Chris S Aug 9 '10 at 14:56
    
Another how-to for almost every shell: understudy.net/custom.html –  Chris S Aug 9 '10 at 15:08
    
+1 for a brilliant tip, I'd have never thought of that myself. –  Kenny Rasschaert Aug 29 '10 at 7:24

The best advice I can give you is don't login as root unless you need root access, and make sure you have a different root/sudo password on each machine.

Making shutdown inaccessible is one option but it's not a good one. Either alias shutdown to shutdown -a and touch /etc/shutdown.allow or chmod a-x /sbin/shutdown

Also, where does it end? Are you going to also disallow halt, reboot and init?

share|improve this answer
4  
It never ends. There's always something more, again and again, more and more, it just never stops <bangs head on the wall repeatedly...> –  Bart Silverstrim Aug 9 '10 at 12:16
    
Don't forget "kill". After all, "kill -9 1" is a pretty effective way (or, well, used to be, back in the days) of shutting a unix box down. –  Vatine Aug 9 '10 at 12:44
    
Some shutdowns don't have -a. Plus relying on an alias like that is the same as relying on alias rm='rm -i' - someday it won't be there when you really need it. Besides, shutdown -a only has limited usefulness anyway. –  Dennis Williamson Aug 9 '10 at 14:56

Some points to consider:

  1. What were you doing as root on a production system during production time? Configure your systems so you don't have to be root on them for day-to-day work. You should never be root on a production system without a very good reason.
  2. Learn the important lesson. When you are root you have to check twice before you press enter. SUDO is no protection if you just write your password to continue, without thinking. Prompt colouring, as mentioned by mkudlacek is a very useful tool to help ensure you are not on the wrong system.
  3. Don't mess with the builtin tools. It is likely to break updates and will drive new hires insane. If you want to change something use your own alias file.
share|improve this answer
2  
I don't necessarily agree with this post. While working as a sysadmin it is very common for my production machine accesses to require root. You lock down users, not admins. Note, no down-vote, because it is a legitimate point of view even if it differs from mine. –  PP. Aug 9 '10 at 12:54
1  
In that case you don't have very hard service level requirements on the machines. Software install, patching, emergency bypassing ACL and changes to the network config are the only things you need root for. None of these activities should be done often on a machine in production... And everything else should be possible to do from a superuser account. If you use root instead of superusers on your production systems you typically have a problem... –  pehrs Aug 9 '10 at 16:03

I don't think messing with the permissions on shutdown are the way to handle the situation. Basically you just learned a lesson. Chin up.

I've done the same kinds of things -- getting long chains of ssh sessions, then messing with the routes on one of the machines I was ssh'd through, cutting myself off. I've botched a rsync request, leading to a systematic destruction of a system on the other side of the world. I've run rm -rf / path on a production server. (That time I got to learn how the restores worked.)

So, much older and hopefully a little bit wiser, I now have strict rules I impose on myself.

  • All root prompts end with a #, no matter what other information is in them.
  • Any time I am at a # prompt, I literally sit on my hands before pressing the enter key.
  • If there's any doubt about what I'm about to do or where I really am or how I got there, I cancel out of it and rebuild it again from a known start condition.
  • When I make a mistake (and I still make them, although they are getting both less frequent and more obscure as time goes on), immediately figure out what I've done, who I've impacted, and go confess my sins to them. Then drop everything else and undo the damange as quickly and as best you can.

The nature of my job requires me to spend a lot of time at a lot of varied root prompts, but thanks to my errors in the past I maintain a far better situational awareness than I did when I started.

share|improve this answer

If you're using Ubuntu then try Molly-guard. As discussed in this similar question.

share|improve this answer

Depends, really. You could try just wrapping the command but it means that if you do updates or upgrades that affect that executable, you might forget about it and have it screw up an update. Playing with system shutdown commands can be a PITA, especially if you have new hires or a replacement that ends up not knowing you were playing with system binaries.

Personally I'd look at wrapping the command in a script that identifies the system by name and makes you confirm that's what you really want to do before running the actual binary, or that you have to type in a certain sequence of letters to confirm shutdown before it runs the binary. That should give some pause.

share|improve this answer

You might consider setting the command prompt to include the machine name, at least on servers. It just might help to prevent other commands being run on the wrong machine in the future. It's more effective if the machine names are coloured to make them stand out and you could even colour code to identify server roles.

share|improve this answer

One way is not to use su and/or login directly to root. Better to log into directly into the account AND have different password for root on local machine and ssh key.

Of course it is in addition to watching for red '#' prompt.

share|improve this answer

Yes, removing the execute bit on the shutdown command is the simplest and safest way of preventing accidental shutdowns, especially if you have a desktop environment like KDE on the machine and you want to prevent accidental shutdowns when logging out.

As for new people being confused, I think the first thing they'll do would be ls -l /sbin/shutdown to find out why it doesn't work (especially if they have the good habit of tab-completing executable names). Obviously though you should tell them about any changes you've made.

For extra safety you could add a line to /etc/rc.local to remove the execute bit from the shutdown command so you don't forget to reset it after a reboot.

share|improve this answer

you can just remove /sbin/ from root's path. that way you need to type the full path to execute it, and it usually fixes accidents.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.