Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Long story short: Big old corporation, lots of UNIX/Linux servers.

I inherited responsibility for a bunch of scripts that that left a few years ago. One of them was a script that would be run every $X amount of months to globally update the root password across all of our servers.

The script is a mess of Shell Script and Expect, and it works on the SSH trust that is set up between all of our servers and a central command-and-control server.

The problem is, the script is a giant mess. The Expect commands are trying to account for every possible version of "passwd" that exists on any UNIX/Linux box out there - and they vary quite a bit.

As we're expanding and upgrading a lot of our infrastructure, the script is getting really unmanageable.

My question is: Is there a better way to do this? Assuming there's an already established SSH trust, what's the best way to change the root password on 3000+ servers at the same time?

share|improve this question
2  
Using sudo and eliminating root passwords altogether is not an option, is it? –  Dmitri Chubarov May 22 '13 at 14:28
5  
Hire an intern? :) –  EEAA May 22 '13 at 14:35
1  
@syneticon-dj the other accounts may be created afresh and stored on some centrally maintained database e.g. LDAP or NIS whatever is better supported. It is not the same thing as ripping away local root and replacing it with the one coming from ldap. –  Dmitri Chubarov May 22 '13 at 14:43
1  
We already have several other security measures in place: Sysadmins log into boxes via LDAP-managed accounts. We do use sudo to get elevated privileges. However, the root account still exists and still has a set password. In the event of total failure of other authentication methods, we still need to be able to log in directly as root via a the console port or other management connection. –  Ricapar May 22 '13 at 15:23
1  
If you've got >3k machines, you need management tools ... yesterday! –  warren May 28 '13 at 15:52

11 Answers 11

up vote 15 down vote accepted

Use Puppet.

Puppet is very flexible, easy to maintain and uses SSL. Might sound a bit overkill and you'll have to put some extra effort to build Puppet system up.

But. Most probably this is not the last mass-update you'll be doing to these machines. Puppet will and does save you a lot of time when actual whatever mass-update procedure begins and scripts are very readable/reusable.

At least this worked for me few years back and still I'm able to re-use some of those Puppet recipes (aka scripts). I have also used it in a bit smaller environments just make sure that every machine is actually having a known state.

I have proven it many times (at many companies) that all customised deployment scripts become pain in the butt after a while or when that next guy steps in. And as long as you carry a cellphone, old scripts will and do haunt you.

If you think that this actually sounds good, here's a great Puppet tutorial with virtual environment included to get you started.

share|improve this answer
4  
Agred, but do not use a user{"root":} to set the password. Instead, use an exec{"chpasswd -e ..."}, which is much safer. –  Dennis Kaarsemaker May 22 '13 at 19:33
6  
Please correct your answer, puppet uses SSL, not SSH. Agree with the rest, though ansible seems like a lightweight alternative much better suited for this use case. Just pip install ansible in a GNU/Linux box, build a list of hosts and ansible -m user -a "password=$crpyt". No agents needed. Manages GNU/Linux, AIX and Solaris out of the box. –  dawud May 22 '13 at 19:51
    
True it uses SSL, not SSH. I use to be maintainer in a system where Puppet was used via SSH by executing a client - long story short. Answer corrected. Thanks dawud pointing that out! –  tomi May 23 '13 at 14:53
    
I provided my own answer below, but I'm accepting yours, as it's probably the most "correct" and manageable long-term solution to the problem. However, it does require a significant investment in terms of infrastructure, planning, licensing, and time in order to roll out Puppet in an environment our size. Some day we'll get there... –  Ricapar May 23 '13 at 19:39
    
@DennisKaarsemaker why is that safer? –  Tom Marthenal May 27 '13 at 20:47

If you can write Perl, the module Net::OpenSSH::Parallel allows to write scripts that perform actions in parallel on remote hosts via SSH quite easyly.

It contains a sample script for changing passwords that you can use as a base. As it seems that you have an heterogeneous environment you would like to group the hosts by type and use a different dialogue handling sub for every one.

share|improve this answer

I've used the Perl module Authen::PAM on Solaris with great success. Here's a sample script:

#!/usr/bin/perl

use Authen::PAM;

my $username = 'root';
my $password = '1234567';

die qq{Error: Unknown user\n} unless getpwnam($username);

die qq{Error: You must run this as root.\n} unless ($> == 0);

my $pamh;

sub my_conv_func
{
    my @res;
    while ( @_ )
    {
        my $code = shift;
        my $msg = shift;
        my $ans = "";

        if ($code == PAM_PROMPT_ECHO_OFF() )
        {
            if (($msg =~ /^New Password:/i) or ($msg =~ /^Re-enter new Password:/i))
            {
                $ans = $password;
            }
            else
            {
                die qq{Unknown message: $msg\n};
            }
        }
        else
        {
            print qq{$msg\n};
        }

        push @res, (PAM_SUCCESS(), $ans);
    }
    push @res, PAM_SUCCESS();

    return @res;
}

ref($pamh = new Authen::PAM("passwd", $username, \&my_conv_func)) || die "Error code $pamh during PAM init!";

my $res = $pamh->pam_chauthtok;

print $pamh->pam_strerror($res),"\n" unless $res == PAM_SUCCESS();

exit 0;
share|improve this answer

I don't know about "best", and whether it's possible for all of the non-Linux *nix machines in your mix, but have you looked at puppet or cfengine for this kind of activity?

There are also commercial (highly expensive) tools for identity management out there, two I've seen/used in the past are Oracle Identity Manager and the Novel equivalent.

share|improve this answer

After continuing to research this, I've learned a few things...

First and foremost, this is a really annoying task to automate, especially across many different environments. The most correct answer to this question is probably @tomi's: use Puppet.

Eventually I hope to get Puppet to manage out infrastructure, but deploying to the entire enterprise's UNIX servers for a root password change right now isn't that feasible of an option.

After reading many manpages and lots of Google-fu, I managed to come up with a script that loops through a list of target servers, opens an SSH connection, and runs one of the following:

# Solaris
# Generate new pass via crypt(newpass,salt) and insert it into /etc/shadow

# AIX
$ echo "root:newpass" | chpasswd -f NOCHECK

# Linux
$ echo "newpass" | passwd root --stdin

# IBM VIO (Virtual I/O)
$ echo "echo \"padmin:newpass\" | chpasswd -f NOCHECK" | oem_setup_env

# IBM HMCs (Hardware Management Consoles)
$ chhmcusr -u hscroot -t passwd -v "newpass"

It does quite a bit more than running just those commands, but those above are what work the magic.

I couldn't find any simple way to non-iteratively change the password on Solaris - so we resorted to modifying /etc/shadow on the fly.

share|improve this answer
2  
Please do not put cleartext passwords on commandline, as it gets logged in shell history. Or set the shell history to /dev/null before you do it. –  Marcin May 23 '13 at 19:51
    
yes , also do not put password in script, check my script to add encrypted passwd –  Rahul Patil May 29 '13 at 5:56

Recently I have done this Using Bash Script..

#!/usr/bin/env bash

# Change Password of Remote Server Using SSH

#--------------------------------------------
# Define User which you want to
# Change Password in remote server
#--------------------------------------------
Uname="root"
# Create Password in Encrpyted Form Using below command,
# and store in this script
# perl -e'print crypt("YourPassword", "salt")' ; echo -e
# then copy and past in following varible,
# password should be single qouted*

Password='safv8d8ESMmWk'

Update_Pass() {
  ssh $ruser@$Server_ip  -p $port "echo ${Uname}:${Password} | chpasswd -e"
}

Show_Help(){
cat <<_EOF
Usage $0        [OPTION]..
Mandatory arguments to long options are mandatory for short options too.
  -i, --ip     <IP_ADDR_OF_SREVER> IP Address Of Remote Server
  -u, --user   <Username>          Username Of Remote Server    <Default User is root>
  -p, --port   <port>              Port Of Remote Server        <Default is 22>

Note:- For Security Reason Do Not Provide Password to the script, because
       it will get save in history, so do not provide it,
       script will prompt for password

Report $0 bugs to loginrahul90@gmail.com
_EOF
}



Main() {

        case $1 in
           -i|--ip) Server_ip=$2;
                    ruser="$4"; [[ -z $ruser ]] && ruser=root
                    port="$6";  [[ -z $port  ]]  && port=22
                    Update_Pass ;;
                *)  Show_Help ;;
        esac
}

Main $*
share|improve this answer

This is my solution so far. still need to see if it works on multiple systems

#!/usr/bin/env bash

ChangePassword()
{
    echo "changing password for server $ServerIp"
    ssh root@$ServerIp "echo root:${NewPassword} | chpasswd" < /dev/null
}

CreatePassword()
{
    while true;
    do
        echo "Please enter the new password :"
        read -s NewPassword <&0
        echo "Confirm the password :"
        read -s ConfirmPassword <&0 
        # /proc/${PPID}/fd/0

        if [ "$NewPassword" == "$ConfirmPassword" ]
        then
            break
        else
            echo "Passwords do not match"
            echo "Try again..."
        fi
    done
    ChangePassword
    echo "end of createpassword"
}

SetUpPasswordlessSSH()
{   
    echo "enter the old password from server $ServerIp when asked"
    ssh root@$ServerIp mkdir -p .ssh
    cat .ssh/id_rsa.pub | ssh root@$ServerIp 'cat >> .ssh/authorized_keys'

    echo "Passwordless SSH is now available"
    echo "Now you can change the password"
    CreatePassword
}

NoSSH()
{
    echo "Passwordless SSH for this server with ip $ServerIp is not yet set up."
    read -p "Do you want to set it up now? " -n 1 -r <&0
    echo "" 
    if [[ ! $REPLY =~ ^[Yy]$ ]]
    then
        break
    else
        SetUpPasswordlessSSH
    fi
}

AcceptHostKey()
{
    read -p "Do you trust the server? " -n 1 -r <&1 
    echo ""
    if [[ ! $REPLY =~ ^[Yy]$ ]]
    then
        break
    else
        SetUpPasswordlessSSH
    fi
}

Main()
{
    while read -r ServerIp <&9
    do
        echo  "Server $ServerIp ..."
        status=$(ssh -o BatchMode=yes -o ConnectTimeout=5 $ServerIp echo ok 2>&1)
        if [[ $status == ok ]]
        then
            echo "creating password"
            CreatePassword
            echo "password changed"
        elif [[ $status == "Permission denied"* ]]
        then
            NoSSH
        elif [[ $status == "Host key verification failed"* ]]
        then
            echo "Error: $status"
            AcceptHostKey
        else
            echo "ERROR OCCURED FOR SERVER WITH IP: $ServerIp"
            echo "Error: $status"
        fi
    done 9< servers.txt
    history -cw
    clear
}

Main $*
share|improve this answer

You could use pdsh to execute your command on multiple hosts at the same time.

share|improve this answer
    
And what command are you going to run? passwd is different on the different versions mentioned. pw not always available.... –  Chris S May 22 '13 at 15:22
    
We have the part of running commands on all of the boxes down pretty well.. The problem is the actual changing of the password. As far as I know, passwd is always user-interactive. –  Ricapar May 22 '13 at 15:27
    
passwd on RHEL Linux at least does have a --stdin parameter –  AngerClown May 22 '13 at 16:16

In addition to puppet: SaltStack Another approach - automate execution using SSH libs either sequentially or in parallel using Fabric http://docs.fabfile.org/en/1.6/, Capistrano or similar that do not require much time/effort to deploy.

share|improve this answer

Two options:

Use puppet

Use run deck. Run deck is a server which allows you to execute commands on hundreds of machines simultaneously. You can group machines into groups, in order to execute commands on just a subset of machines.

http://rundeck.org

share|improve this answer

I think the format of the /etc/shadow file is pretty standard across linux distros. Can you just write a simple awk script to update the password.

cat /etc/shadow| awk -v pass='NEWPASSHASH' -v now=`date '+%s'` 'BEGIN{ OFS=FS=":"} /^root/ {$2=pass; $3=now} {print}' > /tmp/shadow && mv /tmp/shadow /etc/shadow

I would make sure to test it so you don't lock your self out ;)

share|improve this answer
    
I believe editing /etc/shadow is the way to go. However, won't the redirect wipe out /etc/shadow before the edit? This looks like a case for ed or ex. –  mpez0 May 22 '13 at 17:34
3  
The format of shadow may be, but the hashes aren't guaranteed to be interpreted by the same algorithm everywhere. Plus your little script just removed all entries from shadow altogether. :} –  tink May 22 '13 at 17:38
    
lol oops, should be fixed. I hope you didn't sudo run that tink ;) –  HaxElit May 22 '13 at 21:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.