138

A few of us at my company have root access on production servers. We are looking for a good way to make it exceedingly clear when we have ssh'd in.

A few ideas we have had are:

  • Bright red prompt
  • Answer a riddle before getting a shell
  • Type a random word before getting a shell

What are some techniques you guys use to differentiate production systems?

15
  • 11
    Is there any way to teach people to avoid ssh'ing to production systems? If you have everything checked into Puppet, for example, you'd only need to log into production systems for troubleshooting.
    – Alex Holst
    Oct 19, 2011 at 20:25
  • 37
    You allow root ssh access on your production systems????!!!!!
    – symcbean
    Oct 19, 2011 at 22:51
  • 7
    @symcbean: that is not what Sionide said. He could have meant after you SSH into your user account.
    – Zan Lynx
    Oct 19, 2011 at 23:37
  • 3
    Is this question becoming a candidate for community wiki?
    – MadHatter
    Oct 20, 2011 at 15:14
  • 15
    You can make it exceedingly clear it's a production system by disabling SSH to it. It can't be any more clear than that. Oct 20, 2011 at 20:48

24 Answers 24

139

The red prompt is a good idea, which I also use.

Another trick is to put a large ASCII-art warning in the /etc/motd file.
Having something like this greet you when you log in should get your attention:

 _______ _    _ _____  _____   _____  _____            
|__   __| |  | |_   _|/ ____| |_   _|/ ____|     /\    
   | |  | |__| | | | | (___     | | | (___      /  \   
   | |  |  __  | | |  \___ \    | |  \___ \    / /\ \  
   | |  | |  | |_| |_ ____) |  _| |_ ____) |  / ____ \ 
   |_|  |_|  |_|_____|_____/  |_____|_____/  /_/    \_\


 _____  _____   ____  _____  _    _  _____ _______ _____ ____  _   _ 
|  __ \|  __ \ / __ \|  __ \| |  | |/ ____|__   __|_   _/ __ \| \ | |
| |__) | |__) | |  | | |  | | |  | | |       | |    | || |  | |  \| |
|  ___/|  _  /| |  | | |  | | |  | | |       | |    | || |  | | . ` |
| |    | | \ \| |__| | |__| | |__| | |____   | |   _| || |__| | |\  |
|_|    |_|  \_\\____/|_____/ \____/ \_____|  |_|  |_____\____/|_| \_|


 __  __          _____ _    _ _____ _   _ ______ 
|  \/  |   /\   / ____| |  | |_   _| \ | |  ____|
| \  / |  /  \ | |    | |__| | | | |  \| | |__   
| |\/| | / /\ \| |    |  __  | | | | . ` |  __|  
| |  | |/ ____ \ |____| |  | |_| |_| |\  | |____ 
|_|  |_/_/    \_\_____|_|  |_|_____|_| \_|______|

You could generate such a warning on this website or you could use the figlet command.

figlet

Like Nicholas Smith suggested in the comments, you could spice things up with some dragons or other animals using the cowsay command.

dragon cowsay

Instead of using the /etc/motd file, you could also call cowsay or figlet in the .profile file.

8
  • 3
    Ooh I like that!
    – Sionide21
    Oct 19, 2011 at 18:46
  • 13
    If you typed in some command you won`t see that any longer.
    – Nils
    Oct 19, 2011 at 19:02
  • 3
    True, that's where the red prompt comes in handy though. Oct 19, 2011 at 19:03
  • 41
    On Windows servers, I use a bright pink desktop colour and bright yellow title bars/colour scheme so that it's really, really obvious. This is our version of a coloured command prompt Oct 19, 2011 at 21:26
  • 8
    I'd like to add ASCII dragons into the mix. Oct 20, 2011 at 15:15
82

Not quite the same thing, but this web site recommends having your developers wear a pink sombrero when making changes to production systems. You could probably have a similar rule for sshing into them.

developer wearing a pink sombrero

3
  • 1
    Haha! +1 because I LOL'd. But seriously, people can look for their missing glasses while wearing them, I doubt that a hat will serve as a "constant reminder". Also, it does nothing to help you remember which window is which. Oct 20, 2011 at 9:38
  • 8
    Yeah, you'd have to change your process to be: 1) Declare a desire to edit production. 2) Get out the sombrero and shut down all other windows. 3) Do production stuff while everybody is watching. 4) Log out of session and remove sombrero. Oct 20, 2011 at 14:36
  • 1
    "2) Get out the sombrero and shut down all other windows" That won't really work in my company, since that's step 2 of all our SOPs. Step 1 is of course "turn on the lights", because, with the windows closed, it can get pretty dark in this office. Jul 17, 2014 at 21:27
52

The biggest I've used is a discrete naming-scheme where prod-systems are named obviously different than test/dev instances. This makes the "Username@Hostname: " style prompt visibly different. And by obvious I mean more than just different words, different formats too:

example: PRD-WEB001 vs DEVEL-BOB-WEB001

This has several things going for it:

  • The extra hypenated block makes it a set-of-three instead of a set-of-two.
  • The first of the set is a different length.
  • The overall length of names is markedly different, which makes the command-line spacing different relative to each other and other text in the window.

And best of all, it doesn't require special terminal-configs for production just to avoid Oops errors.

In my experience, you want something that is a constant reminder of where you are. Login-methods like riddles are good for about 10 seconds, until you forget which window is which. All it takes is to do an ls in the wrong directory to scroll the ominous login-banner out of view, bury the terminal window under a browser window while googling something, alt-tab back to the wrong window and mayhem ensues. Best to have some constant visual cue like a significantly different command-prompt.

2
  • 1
    That works really well for basic systems, but as soon as you change a dev-* to prod-* the devs moan and groan as half their links stop working (no matter how many times you've done it before), and sometimes the system itself limps along half-working until you fix all the numerous references to the old name in config files. Especially with proprietary software installed, and DNS aliases only help so much. Even worse, it's easy to forget the prompt completely while switching windows. Oct 22, 2011 at 7:17
  • 1
    Devs need to create settings that work across dev/test/staging/production (that's what I do, but it's another story). For a working 'red prompt' for bash (harder than it seems), see my answer at serverfault.com/a/479718/79266
    – RichVel
    Feb 17, 2013 at 17:35
44

One thing you need to keep in mind is that this needs to be a persistent reminder, not just an indicator at login time. Very often, someone will have several shells running at the same time in different tabs and move between them. Some will be dev, some production. So when you are running a command, you need to have an indicator at that point. So having a special prompt is the best method, in my experience, with a modified title/tab bar being a nice complement to this for finding the right window/tab easily.

So I'd recommend having a colored prompt (red being the obvious choice) and all caps for the hostname, with similar behavior for the user (privileged vs. non-privileged) as your prompt. Some examples:

colored prompts example

Usually something like

set prompt =  "%{\033[1;44m%}`whoami`@`hostname -s`#%{\033[0m%} "` 

in your shell startup file. This one is for the blue. Replace the 44 with 41 fir red, and 42 for green. Other colors and wild patterns available too.

1
  • For a working 'red prompt' for bash (harder than it seems), see my answer at serverfault.com/a/479718/79266 - also shows git branch, truncated current directory, etc.
    – RichVel
    Feb 17, 2013 at 17:35
15

These are my suggestions:

1) Make sure most commands (rm, chown, chmod, /etc/init.d/* ) on the Production environment require sudo access

2) Use PS1/PS2 to indicate that the user is in a Prod server

bash-3.2$  export PS1="[\u@\h \W]\$ "

This will show the command prompt as

[sridhar@prodappserver901 conf]$

3) If Using Putty/SSH clients, You can always set up unique Background color/profile to make the Production servers stand-out.

1
  • #1 is an interesting idea but impractical because scripts running as daemon users might need to use rm, chmod, etc.
    – Zan Lynx
    Oct 19, 2011 at 23:38
13

Just consider that your second and third ideas help during the initial connection but are of no value when you have multiple terminals open and move from one to another. sysadmin1138's idea of using naming is good when it can be applied but there are plenty of cases where it cannot be.

The only thing I've found to be really worthwhile is a coloured prompt. I like green for dev/testing, red for production and blue for machines in the DMZ. That way, even if I have two machines with the same name (in different networks), such as when preparing a replacement machine, I can still easily tell which one I'm on.

11

The red/special command prompt is good. Another thing might be a quicker auto-logout on those machines using the TMOUT variable. If you have opened many windows the production ones will go away faster.

This should lead to a different behaviour:

  1. Develop
  2. Test
  3. Make your changes to a staging server
  4. Only then make a quick dash to production and deploy there (exactly like you did it on the staging server)
9

Working on a production machine with a plain root account is never ever a good idea.

Have an account with full sudo permissions. Does not allow saving sudo session. Prohibit sudo su. Use separate password for it ( not one you have for your dev machine ). Probably tweak sudo to notify about production identity of the shell before executing the command ( via alias ).

It will make accidental mistakes quite a hard ones. And red prompt never hurts.

4
  • and by "sudo su" you mean "sudo -i" or at the very least "sudo su -", right?
    – Sparr
    Oct 20, 2011 at 11:34
  • Whoa! it's Sparr! Small world :)
    – Sionide21
    Oct 20, 2011 at 14:07
  • 1
    It's impossible to stop people from having a root shell. (sudo bash). You can get a root shell from any 'blacklist' setup. (If your careful, you can stop it with a whitelist setup, but it's typically not guaranteed. )
    – user606723
    Oct 20, 2011 at 15:16
  • 1
    It is not supposed to prevent intended access to root shell - only some common ways people are so used to so that they can do it subconsciously, without paying any attention. Oct 20, 2011 at 16:09
8

I went with the red prompt idea, and found it fairly tedious to find working code for .bashrc.

So here's my version, ready to be included in .bashrc - https://github.com/RichVel/nicer-bash-prompt. It's completely driven by hostname, so as long as you have a suitable pattern to production hostnames (say xyprod01, xyprod02, etc) it will work well, and you can use the same .bashrc in all environments.

It looks like this:

screenshot of red prompt

This creates a nicer bash prompt including red prompt on production hosts - also shows you the current git branch, and last 2 directories in $PWD. It takes care to avoid messing up the prompt display when doing Ctrl/R (reverse searching) in bash.

Also includes an optional feature to sync your bash history across all terminal windows, along the lines of this answer. This is nice but not everyone wants it, so it's disabled by default.

0
5

Though I don't know what your IT setup is like, one solution that might be effective would be to have a special room that you have to go to to SSH into production servers as root. If you have a datacenter this could be the server room itself, but having a separate physical location from which 'normal' work is not done would quite effectively serve as a constant reminder that you're accessing production machines.

5
  • 2
    While that would certainly be effective, for most situations it would also be extremely counter-productive. Oct 20, 2011 at 3:47
  • I agree with John, but this is certainly something to think about.
    – user606723
    Oct 20, 2011 at 15:10
  • @JohnGardeniers: counterproductive? You mean it would hurt productivity?
    – LarsH
    Oct 20, 2011 at 16:59
  • @LarsH, having to unnecessarily go into different rooms to do different parts of your job most certainly does hurt productivity. Your entire network should be manageable from a single location. Oct 20, 2011 at 20:31
  • @JohnGardeniers: understood. When you initially said "counterproductive" I thought you meant its effect was counter to the stated purpose, i.e. to serve as a constant reminder that you're accessing production machines.
    – LarsH
    Oct 20, 2011 at 21:21
4

Just a tweak on above suggestions. I use CDE as my unix desktop and all the production systems I access via a menu in .dt/dtwmrc. On all the dev and UAT systems I keep my normal colour scheme but on prod systems I set the terminal to have a red background. I don't like the look but that's kind of the point.

eta - missed Karol's suggesting basically the same thing

4

When I log into a production machine, I get a paragraph warning me that it's a production machine, as well as a short list of guidelines. There's a number I can call for UNIX support if I don't think I can perform my task alone safely, a reminder that blowing up a production machine can cost me my job, and a reminder that everything I do is logged.

edit: I work in the transportation industry.

3
  • Ouch. While I can understand a need for this, I'm glad I don't work in that kind of environment.
    – LarsH
    Oct 20, 2011 at 16:58
  • 1
    I'm glad I do- I work in transportation, and these systems don't just cost money when they go down, they cost lives.
    – Basil
    Oct 20, 2011 at 17:57
  • I'm glad you're that careful too!
    – LarsH
    Oct 20, 2011 at 21:15
4

Simple answer? Change the color of your shell to red in the shell config. It will be dead obvious and simple to set up. Not only that, but unlike server headers it won't go away after you type in a few commands.

4

My group uses visionapp Remote Desktop (2017 edit: looks like the product got renamed but I think it's the same) to both RDP into Windows machines and SSH into Linux ones. Our connections are grouped into folders by tier and assigned a tab color.

So whenever we open a production connection -- bam! -- we get some bright red in our face:

enter image description here

It's definitely a worthwhile investment if you're a big Windows shop and rely on RDP a lot. I bet there are other great tools if all you need is SSH.

3

In PuTTY you can change the title of the window to something other than the default for a specific saved session. This always remains on the window no matter what you do inside the window. This also shows up in the task bar.

Expand Window, click Behaviour. Enter something in Window Title like:

  * * * * * * * * * * * * PRODUCTION  * * * * * * * * * * * * PRODUCTION  * * * * * * * * * * * *
3

Here's what I did on my Mac. For every server, I add an entry for it in my ~/.ssh/config file, e.g.

Host app13
    HostName server.example.com
    User tom
    PermitLocalCommand yes
    LocalCommand osascript %d/bin/change_terminal_colours.scpt 12 35 35

This Applescript is triggered once the SSH session is established. It sets the terminal background colour to the RGB values provided (or back to default if no colour values are provided). The potentially tricky part is to intercept the end of the SSH session to set the colours back to the defaults. For that, I created the following shell script as ~/bin/ssh as to override the default ssh command. This essentially intercepts and wraps any calls to the SSH command. I tried using aliasing and functions, but this solution worked the best:

#!/bin/bash
/usr/bin/ssh $@
osascript ~/bin/change_terminal_colours.scpt

Here's the source for the change_terminal_colours.scpt script. Put this in your ~/bin directory also:

on run argv
    tell application "Terminal"
        # NOTE: Color values range from 0 to 65535.
        if (count of argv) > 0 then
            set backgroundColor to {(item 1 of argv) * 256, (item 2 of argv) * 256, (item 3 of argv) * 256}
        else
            set backgroundColor to background color of default settings
        end if

        try
            set background color of (selected tab of front window) to backgroundColor
        end try
    end tell
end run

I wrote this solution a week ago and have been using it ever since. I hope others find it of value. I find it works better than any of the solutions I found by Googling.

2

Apart from a unique prompt (which seems to be the most reliable solution), if you're logging in from the same workstation, you can use different profiles for your SSH sessions.

For instance, I have red backgrounds for production systems, green for development, blue for infrastructure (routers etc.) and white for local workstation.

If you use GNOME, there's an easy way to fire up a SSH connection with your desired profile:

gnome-terminal --window-with-profile=production -e 'ssh [email protected]'

The main disadvantage - it's client-side, so a special prompt is still your best bet if you're accessing servers from different locations.

2

Another way to make it clear that you're on a production system is to set the TMOUT (autologout) feature on the production system.

3
  • 1
    Just how does that make it clear? It's also potentially very dangerous, as commands or processes may be running and will suddenly be aborted by having the session logged off. Oct 20, 2011 at 1:24
  • 3
    I just tested TMOUT and it looks like if a long-running command is running it does not terminate the session. It only terminates it when you're sitting at the command prompt. Nils also suggested this and it makes sense to me.
    – sjbotha
    Oct 20, 2011 at 15:16
  • 1
    That is the really nice thing about TMOUT. We used autolog or even killerd before we discovered TMOUT. autolog and killerd are NOT that nice. killerd is also buggy and tends to block one complete CPU...
    – Nils
    Oct 20, 2011 at 20:29
2

Due to the vast regulatory requirements while working in a particular industry, here everyone's activities are key-logged for any future disputes. Aecause of that, access is also restricted and you have to jump through a few "menus" that allow you to access a machine as one of a handful trusted users. By setting up this system, person must conciously pick PRODUCTION or QA environment and then pick the host they want to access from the list. This also has a timeout period so you don't run into situation of logging onto a prod host and forgetting which environment you're in the next day.

2

i use the prompt changes like others here. Its quick and nasty but works fine for my purposes. It'll give you red as a normal user on a production system, and red upper case as root on a prod system.

It could be written in less lines but i do it like this so i can tweak the other 2 cases (non root-non prod) if i want to.

We work on the assumption that a production server does not use DHCP, but you can use any other method for working out if its a prod system. Whatever works for you.

productionSrv=1
grep -qi bootproto=dhcp /etc/sysconfig/network-scripts/ifcfg-*
if [ "$?" -eq 0 ]; then
    productionSrv=0
fi
hostName=`hostname`
userName=`whoami`
if [ $userName == "root" ]
then
    if [ "$productionSrv" == 1 ]
    then
        hostName=`echo $hostName | tr [:lower:] [:upper:]`
        PS1='[\e[0;31m]\u[\e[0m]@[\e[0;31m]$hostName[\e[0m][$?][\e[0;31m][\W][\e[0m][\e[0;31m]\$[\e[0m]: '
    else
        PS1='[\e[0;31m]\u[\e[0m]@[\e[0;35m]$hostName[\e[0m][$?][\e[0;31m][\W][\e[0m][\e[0;31m]\$[\e[0m]: '
    fi
    PATH=$PATH:/sbin/
else
    if [ "$productionSrv" == 1 ]
    then
        PS1='[\e[0;32m]\u[\e[0m]@[\e[0;31m]$hostName[\e[0m][$?][\e[0;31m][\W][\e[0m][\e[0;32m]\$[\e[0m]: '
    else
        PS1='[\e[0;32m]\u[\e[0m]@[\e[0;35m]$hostName[\e[0m][$?][\e[0;31m][\W][\e[0m][\e[0;32m]\$[\e[0m]: '
    fi
fi

4
  • While the script is interesting, the assumption of static IP's only on production systems is not common. In most cases, all servers will have static IP's, both development and production. Oct 20, 2011 at 21:13
  • 1
    horses for courses really. our dev systems use dhcp reservations. You can use any system that works for you, such as compare the hostname, subnet, vlan, whatever.
    – Sirex
    Oct 21, 2011 at 6:55
  • @Sirex, The opposite is also possible. For instance, Amazon's EC2 VPS's use static DHCP leases on production servers and public IP's are forwarded. The instance itself doesn't directly know what public IP it's running behind, at least not at the network interface level. Oct 22, 2011 at 11:01
  • 1
    yup. You could always just have a role rpm package that creates a /etc/PRODUCTION file and test for that. Whatever floats your boat.
    – Sirex
    Oct 25, 2011 at 11:39
1

Have a different (perhaps longer) root password on the production machines.

You could make a special sudo (or sudo wrapper), such that a special message is output for the production machines.

2
  • .. A different root password on prod machines? why didn't I think of that!?
    – user606723
    Oct 20, 2011 at 15:08
  • Also, sudo already has plenty of configuration options. I would be surprised if you can't already do this. A custom sudo is terrible idea because it wouldn't be tested will enough to get the suid bit.
    – user606723
    Oct 20, 2011 at 15:08
1

Here, we have default putty configurations to make the foreground color of the entire screen as bright pink.

  • Prod: bright pink.
  • Regression: pale green
  • Model: pale blue
  • Dev: normal

It works pretty well, but I do like some of the other ideas here.

  • Pro:unless something doesn't use the foreground color.., EVERY prod screen is bright pink.
  • Con: Obviously if you ssh through something other than the default putty config this does nothing though.
1

in the .bashrc/.bash_profile

echo " THIS IS THE PRODUCTION SYSTEM. BE RESPONSIBLE.. " .. 
0

Here's mine

I used ascii-image-converter for the image (forward the result to a file and cat it at the top of your .bashrc). Also using this for full effect:

export PS1="\[\e[1;32m\]\u\[\e[3;31m\]@\[\e[3;31m\]🔥🔥🔥 PROD 🔥🔥🔥\[\e[0;37m\]: \[\e[1;34m\]\w \[\e[0;31m\]$ \[\e[0m\]"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .