Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

On one of the servers that I manage i've just noticed that there are hundreds of "ssh-agent" processes running. I'm at a loss as to where to even start here.

Is there something that I've forgotten to setup that is keeping the ssh-agent processes running after a session has disconnected?

# ps ax | grep ssh-agent
  306 ?        Ss     0:00 ssh-agent
  312 ?        Ss     0:00 ssh-agent
 ...
 5076 ?        Ss     0:00 ssh-agent
 5087 ?        Ss     0:00 ssh-agent
 5098 ?        Ss     0:00 ssh-agent
 5119 ?        Ss     0:00 ssh-agent
 ...

This continues past PID 37000.

The server is running Ubuntu 12.10 LTS.

Here is the sshd_config

# cat /etc/ssh/sshd_config
# Package generated configuration file
# See the sshd_config(5) manpage for details

# What ports, IPs and protocols we listen for
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 768

# Logging
SyslogFacility AUTH
LogLevel INFO

# Authentication:
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes

RSAAuthentication yes
PubkeyAuthentication yes
#AuthorizedKeysFile %h/.ssh/authorized_keys

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
PasswordAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

#MaxStartups 10:30:60
#Banner /etc/issue.net

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

After clearing the ssh-agent processes they have slowly started to come back. As requested here is the output from ps fauxw

$ ps fauxw
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0      0     0 ?        S    02:23   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [kworker/0:0]
root         5  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [kworker/u:0]
root         6  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [migration/0]
root         7  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [watchdog/0]
root         8  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [migration/1]
root         9  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [kworker/1:0]
root        10  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [ksoftirqd/1]
root        11  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [watchdog/1]
root        12  0.0  0.0      0     0 ?        S<   02:23   0:00  \_ [cpuset]
root        13  0.0  0.0      0     0 ?        S<   02:23   0:00  \_ [khelper]
root        14  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [kdevtmpfs]
root        15  0.0  0.0      0     0 ?        S<   02:23   0:00  \_ [netns]
root        16  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [kworker/u:1]
root        17  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [xenwatch]
root        18  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [xenbus]
root        19  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [sync_supers]
root        20  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [bdi-default]
root        21  0.0  0.0      0     0 ?        S<   02:23   0:00  \_ [kintegrityd]
root        22  0.0  0.0      0     0 ?        S<   02:23   0:00  \_ [kblockd]
root        23  0.0  0.0      0     0 ?        S<   02:23   0:00  \_ [ata_sff]
root        24  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [khubd]
root        25  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [kworker/0:1]
root        26  0.0  0.0      0     0 ?        S<   02:23   0:00  \_ [md]
root        28  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [khungtaskd]
root        29  0.0  0.0      0     0 ?        S    02:23   0:01  \_ [kswapd0]
root        30  0.0  0.0      0     0 ?        SN   02:23   0:00  \_ [ksmd]
root        31  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [fsnotify_mark]
root        32  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [ecryptfs-kthrea]
root        33  0.0  0.0      0     0 ?        S<   02:23   0:00  \_ [crypto]
root        41  0.0  0.0      0     0 ?        S<   02:23   0:00  \_ [kthrotld]
root        42  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [khvcd]
root        43  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [kworker/1:1]
root        62  0.0  0.0      0     0 ?        S<   02:23   0:00  \_ [devfreq_wq]
root       166  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [jbd2/xvda1-8]
root       167  0.0  0.0      0     0 ?        S<   02:23   0:00  \_ [ext4-dio-unwrit]
root       266  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [kjournald]
root       831  0.0  0.0      0     0 ?        S    02:23   0:00  \_ [flush-202:1]
root         1  0.0  0.0  24324  1336 ?        Ss   02:23   0:00 /sbin/init
root       278  0.0  0.0  17232   340 ?        S    02:23   0:00 upstart-udev-bridge --daemon
root       280  0.0  0.0  21468   796 ?        Ss   02:23   0:00 /sbin/udevd --daemon
root       341  0.0  0.0  21464   680 ?        S    02:23   0:00  \_ /sbin/udevd --daemon
root       347  0.0  0.0  21464   684 ?        S    02:23   0:00  \_ /sbin/udevd --daemon
root       401  0.0  0.0  15188   188 ?        S    02:23   0:00 upstart-socket-bridge --daemon
root       452  0.0  0.0   7264   508 ?        Ss   02:23   0:00 dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.lea
root       615  0.0  0.0  50032  1096 ?        Ss   02:23   0:00 /usr/sbin/sshd -D
root      1679  0.0  0.0  73436  1328 ?        Ss   02:32   0:00  \_ sshd: app-user [priv]
app-user  1778  0.0  0.0  73436  1112 ?        S    02:32   0:00  |   \_ sshd: app-user
root      2644  0.0  0.2  73436  3600 ?        Ss   05:44   0:00  \_ sshd: ubuntu [priv]
ubuntu    2789  0.0  0.0  73436  1540 ?        S    05:44   0:00      \_ sshd: ubuntu@pts/0
ubuntu    2790  0.2  0.4  25972  8500 pts/0    Ss   05:44   0:00          \_ -bash
ubuntu    2935  0.0  0.0  16992  1216 pts/0    R+   05:48   0:00              \_ ps fauxw
syslog     632  0.0  0.0 335656  1412 ?        Sl   02:23   0:00 rsyslogd -c5
102        642  0.0  0.0  23816   536 ?        Ss   02:23   0:00 dbus-daemon --system --fork --activation=upstart
root       697  0.0  0.0  14504   440 tty4     Ss+  02:23   0:00 /sbin/getty -8 38400 tty4
root       703  0.0  0.0  14504   448 tty5     Ss+  02:23   0:00 /sbin/getty -8 38400 tty5
root       712  0.0  0.0  14504   444 tty2     Ss+  02:23   0:00 /sbin/getty -8 38400 tty2
root       713  0.0  0.0  14504   444 tty3     Ss+  02:23   0:00 /sbin/getty -8 38400 tty3
root       720  0.0  0.0  14504   448 tty6     Ss+  02:23   0:00 /sbin/getty -8 38400 tty6
root       725  0.0  0.0   4328   432 ?        Ss   02:23   0:00 acpid -c /etc/acpi/events -s /var/run/acpid.socket
root       726  0.0  0.0  19112   572 ?        Ss   02:23   0:00 cron
daemon     728  0.0  0.0  16908   260 ?        Ss   02:23   0:00 atd
mysql      752  0.2  8.6 1340724 150636 ?      Ssl  02:23   0:31 /usr/sbin/mysqld
whoopsie   809  0.0  0.0 187720  1616 ?        Ssl  02:23   0:00 whoopsie
root       838  0.0  0.8 486052 14604 ?        Ss   02:23   0:00 /usr/sbin/apache2 -k start
www-data  1003  0.0  2.5 492704 44156 ?        S    02:23   0:06  \_ /usr/sbin/apache2 -k start
www-data  1007  0.0  2.7 497052 47088 ?        S    02:23   0:05  \_ /usr/sbin/apache2 -k start
www-data  1471  0.0  2.3 492272 41496 ?        S    02:23   0:06  \_ /usr/sbin/apache2 -k start
www-data  2171  0.0  2.0 492048 36052 ?        S    03:27   0:03  \_ /usr/sbin/apache2 -k start
www-data  2253  0.0  2.2 493532 38700 ?        S    03:51   0:03  \_ /usr/sbin/apache2 -k start
www-data  2379  0.0  1.9 490472 34048 ?        S    04:27   0:03  \_ /usr/sbin/apache2 -k start
www-data  2578  0.0  1.9 489936 32948 ?        S    05:27   0:01  \_ /usr/sbin/apache2 -k start
www-data  2580  0.0  1.8 489936 32200 ?        S    05:27   0:01  \_ /usr/sbin/apache2 -k start
www-data  2611  0.0  1.8 490164 32444 ?        S    05:34   0:00  \_ /usr/sbin/apache2 -k start
www-data  2612  0.1  1.8 489920 32152 ?        S    05:34   0:01  \_ /usr/sbin/apache2 -k start
root      1109  1.5  9.1 1762540 157820 ?      SNl  02:23   3:16 /usr/local/crashplan/jre/bin/java -Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=
root      1127  0.0  0.0  14504   444 tty1     Ss+  02:23   0:00 /sbin/getty -8 38400 tty1
app-user  1662  0.0  0.0  12568   324 ?        Ss   02:30   0:00 ssh-agent
app-user  2074  0.0  0.0  12568   316 ?        Ss   03:00   0:00 ssh-agent
app-user  2197  0.0  0.0  12568   316 ?        Ss   03:30   0:00 ssh-agent
app-user  2282  0.0  0.0  12568   320 ?        Ss   04:00   0:00 ssh-agent
app-user  2399  0.0  0.0  12568   320 ?        Ss   04:30   0:00 ssh-agent
app-user  2494  0.0  0.0  12568   320 ?        Ss   05:00   0:00 ssh-agent
app-user  2593  0.0  0.0  12568   320 ?        Ss   05:30   0:00 ssh-agen
share|improve this question
    
I would be worried about the fact that all of them have been started at exactly the same time. This might mean that either somebody tried a fork bomb, or there is a script which is launching all of them. Have you ever checked what their parent PID is? –  mauro.stettler Sep 2 '13 at 9:24
1  
@mauro.stettler: nothing allows us to say they have been started at the same time. 0:00is the processor time consumed. But the PIDs used does look like a loop creating them... –  Kwaio Sep 2 '13 at 9:45
    
@Kwaio oh, right, that's the processor time:) sorry for the mixup –  mauro.stettler Sep 2 '13 at 9:47
1  
Seems like a fork going wild. Can you give me the result of ps auxf | grep ssh-agent ? Also have a look at tychoish.com/rhizome/9-awesome-ssh-tricks {the "reagent" part}. It may help –  Nikolaidis Fotis Sep 2 '13 at 9:47
1  
ps fauxw | less will allow you to verify if they aren't what we call a fork bomb, meaning process forking themselves endlessly until the running pid limit is reached, effectively breaking your system. If not, you will at least be able to soo what is spawning them and why they aren't killed when it is done. –  Kwaio Sep 2 '13 at 9:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.