3

I have a number of systems that rely on application-level mirroring to a secondary server. The secondary server pulls data by means of a series of remote SSH commands executed on the primary. The application is a bit of a black box, and I may not be able to make modifications to the scripts that are used.

My issue is that the logging in /var/log/secure is absolutely flooded with requests from the service user, admin. These commands occur many times per second and have a corresponding impact on logs. They rely on passphrase-less key exchange. The OS involved is EL5 and EL6. Example below.

  • Is there any way to reduce the amount of logging from these actions. (By user? By source?)
  • Is there a cleaner way for the developers to perform these ssh executions without spawning so many sessions? Seems inefficient. Can I reuse the existing connections?

Example log output:

Jul 24 19:08:54 Cantaloupe sshd[46367]: pam_unix(sshd:session): session closed for user admin
Jul 24 19:08:54 Cantaloupe sshd[46446]: Accepted publickey for admin from 172.30.27.32 port 33526 ssh2
Jul 24 19:08:54 Cantaloupe sshd[46446]: pam_unix(sshd:session): session opened for user admin by (uid=0)
Jul 24 19:08:54 Cantaloupe sshd[46446]: pam_unix(sshd:session): session closed for user admin
Jul 24 19:08:54 Cantaloupe sshd[46475]: Accepted publickey for admin from 172.30.27.32 port 33527 ssh2
Jul 24 19:08:54 Cantaloupe sshd[46475]: pam_unix(sshd:session): session opened for user admin by (uid=0)
Jul 24 19:08:54 Cantaloupe sshd[46475]: pam_unix(sshd:session): session closed for user admin
Jul 24 19:08:54 Cantaloupe sshd[46504]: Accepted publickey for admin from 172.30.27.32 port 33528 ssh2
Jul 24 19:08:54 Cantaloupe sshd[46504]: pam_unix(sshd:session): session opened for user admin by (uid=0)
Jul 24 19:08:54 Cantaloupe sshd[46504]: pam_unix(sshd:session): session closed for user admin
Jul 24 19:08:54 Cantaloupe sshd[46583]: Accepted publickey for admin from 172.30.27.32 port 33529 ssh2
Jul 24 19:08:54 Cantaloupe sshd[46583]: pam_unix(sshd:session): session opened for user admin by (uid=0)
Jul 24 19:08:54 Cantaloupe sshd[46583]: pam_unix(sshd:session): session closed for user admin
Jul 24 19:08:54 Cantaloupe sshd[46612]: Accepted publickey for admin from 172.30.27.32 port 33530 ssh2
Jul 24 19:08:54 Cantaloupe sshd[46612]: pam_unix(sshd:session): session opened for user admin by (uid=0)
Jul 24 19:08:54 Cantaloupe sshd[46612]: pam_unix(sshd:session): session closed for user admin
Jul 24 19:08:55 Cantaloupe sshd[46641]: Accepted publickey for admin from 172.30.27.32 port 33531 ssh2
Jul 24 19:08:55 Cantaloupe sshd[46641]: pam_unix(sshd:session): session opened for user admin by (uid=0)
Jul 24 19:08:55 Cantaloupe sshd[46641]: pam_unix(sshd:session): session closed for user admin
Jul 24 19:08:55 Cantaloupe sshd[46720]: Accepted publickey for admin from 172.30.27.32 port 33532 ssh2
Jul 24 19:08:55 Cantaloupe sshd[46720]: pam_unix(sshd:session): session opened for user admin by (uid=0)
Jul 24 19:08:55 Cantaloupe sshd[46720]: pam_unix(sshd:session): session closed for user admin
Jul 24 19:08:55 Cantaloupe sshd[46749]: Accepted publickey for admin from 172.30.27.32 port 33533 ssh2
Jul 24 19:08:55 Cantaloupe sshd[46749]: pam_unix(sshd:session): session opened for user admin by (uid=0)
Jul 24 19:08:55 Cantaloupe sshd[46749]: pam_unix(sshd:session): session closed for user admin
Jul 24 19:08:55 Cantaloupe sshd[46778]: Accepted publickey for admin from 172.30.27.32 port 33534 ssh2
Jul 24 19:08:55 Cantaloupe sshd[46778]: pam_unix(sshd:session): session opened for user admin by (uid=0)
Jul 24 19:08:55 Cantaloupe sshd[46778]: pam_unix(sshd:session): session closed for user admin
Jul 24 19:08:55 Cantaloupe sshd[46857]: Accepted publickey for admin from 172.30.27.32 port 33535 ssh2
  • What syslog daemon is in use? Have you tried configuring that to send ssh to the bitbucket? – Zoredache Jul 25 '14 at 5:26
  • 1
    It's rsyslog in this case, but I'm thinking that connection reuse is the way to go... Some nagging issues with RHEL, though. – ewwhite Jul 25 '14 at 5:27
3

Specifically answering the question regarding whether it's possible to reduce the overhead of spawning more SSH connections: Yes. You can use the ControlMaster feature present since OpenSSH 5.5. This blog post will have more details: http://puppetlabs.com/blog/speed-up-ssh-by-reusing-connections

I'm not sure if that would affect how much logging happens. But, depending on how the application is written, it may be possible to use this feature without actually modifying the application, just reconfiguring OpenSSH. Assuming it even uses OpenSSH as the client, or a different client with this support. There's just not enough info here to tell for sure.

| improve this answer | |
  • Great answer, I didn't know that feature... However, currently, redhat el5 and el6 are not using openssh 5.5. Also, this only changes the way SSH socket is established, and I think the logging will not be changed. I don't really see a solution (except log filtering) else than re-writing the scripts to connect once and then loop into command handling (using perl / expect is quite easy to do). – tonioc Jul 25 '14 at 11:31
  • I figured out that the ControlMaster feature was back ported into EL5 and EL6, but the super critical ControlPersist feature was not. As a result, I have to keep a master connection open. The logging does subside, so this takes care of log noise reduction and streamlines the connections. – ewwhite Jul 25 '14 at 12:33
1

Since you're using rsyslog you have the option to filter messages quite easily. A basic example to discard messages is this one:

#/etc/rsyslog.cof 

# this is original log file including all authpriv messages regarding admin
authpriv.* /var/log/secure.admin

# These rules filter the remote key based logins for admin
:msg, contains, "Accepted publickey for admin from 172.30.27.32" ~
:msg, contains, "session opened for user admin" ~
:msg, contains, "session closed for user admin" ~
authpriv.*     /var/log/secure

The tilde ~ is the instruction to discard messages containing the previous string.

A slightly more advanced and better syntax (untested) is :

if $syslogfacility-text == 'authpriv' and 
 ($msg contains 'for user admin' or $msg contains 'publickey for admin from 172.30.27.32') 
then /var/log/secure.admin

if $syslogfacility-text == 'authpriv' and not
 ($msg contains 'for user admin' or $msg contains 'publickey for admin from 172.30.27.32') 
then /var/log/secure

Where each if statement goes on a single line.

| improve this answer | |
0

An update on this...

I've solved this on my RHEL/CentOS EL6 systems by enabling the SSH ControlMaster feature:

On the connecting host's /home/username/.ssh/config file:

Host *
ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h-%p
#ControlPersist 600

The ~/.ssh/sockets/ directory needed to be created manually.

The ControlPersist directive is a persistency feature that has not been backported to the OpenSSH available in EL6. I wish it were, because without it, the multiplexing connections terminate once the initial connection dies.

I've circumvented this with a quick-and-dirty Monit script on the connecting host.

check process ssh-control
        matching "MNn"
        start program = "/usr/bin/ssh -MNn destination" as uid username
        stop program = "/usr/bin/pkill -u username ssh"

Where is the system I'm connecting to and is the service account of the application replication system (admin in this case).

The ssh -M is used for master-mode and SSH connection sharing.

-M Places the ssh client into “master” mode for connection sharing. Multiple -M options places ssh into “master” mode with confirmation required before slave connections are accepted.

The result of all of this is clean logs... and that the flurry of ssh commands initiated by the replication server all reuse the open socket created by the master connection.

Once this feature makes it into the EL6 OpenSSH package, I'll be able to get rid of the Monit script and just use the ControlPersist parameter.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.