2

I run a few Rightscale CentOS AMI based instances on Amazon EC2. Two months back I found that our SSHD security is compromised( I had added host.allow and host.deny for ssh). So I created new instances and done an IP based ssh that allows only our IPs through AWS Firewall(ec2-authorize) and chnaged the ssh 22 default port to some other port but two days back I found I was not able to login to the server and when I tried on 22 port the ssh got connected and I found that sshd_conf was changed and when I tried to edit sshd_config I found root had no write permission on the file. So I tried a chmod and it said access denied for 'root' user. This is very strange. I checked secure log and history and found nothing informative. I have PHP, Ruby On Rails, Java, Wordpress apps running on these server.

This time I did a chkrootkit scan and found nothing. I renamed the /etc/ssh folder and reinstalled openssh through yum. I had faced this on 3 instances on CentOS(5.2, 5.4) I have instances on Debian as well those working fine. Is this a CentOS/Rightscale issue. Guys, what security measures I should take to prevent this.


I doubt this can be an internal hacker too since we allow only our IPs to ssh.

So I am planning to user a keylogger which can log all keystrokes from each user and terminals. Could you suggest me some Linux keylogget best suits for this purpose?


I am sure it's and ssh attack, but no idea how they did. CentOS 5 still uses openssh 4.3 so I compiled openssh 5.5 and found that they have changed the attributes for /usr/bin/ssh and /usr/sbin/sshd. I had to use lsattr filename the chattr -ia filename to remove the attributes and successfully compiled openssh. Now I am trying to setup chroot and setup Osiris for precaution.

Please share your thoughts. Also I need a keylogger logs all tty keystrokes which one you suggest?

Thanks for your support.

3
  • could it be that recent wordpress vulnerability?
    – Knox
    May 13, 2010 at 14:21
  • @Knox I was googling about that vulnerability.... But could not find much information. Could you please tell me more on that?
    – user37143
    May 17, 2010 at 11:45
  • I thought i had heard of a recent zero-day vulernability. However, in researching this further, I think although a number of wordpress servers were taken over, they were either outdated versions of wordpress or had their passwords compromised in some way. Here's a summary of the attacks: securecomputing.net.au/News/…
    – Knox
    May 17, 2010 at 12:29

4 Answers 4

5

Basically, you're screwed.

If you were actually exploited, they could have added themselves a backdoor into the SSHD binary, or any other file on the system...Since the source is available, this is easy to do, and it's extremely difficult to spot if you didn't plan ahead.

If you didn't have Tripwire or similar installed before the problem happened, you're basically going to have to check every single file on the system against a known-clean install.

Easier to just move your app to a clean server.

The best prevention is a file integrity scanner. Basically, it checks every file on the system, and makes a hash of it. Then it periodically re-scans, and notifies you of any changes. Makes clean up a breeze.

I agree that you probably didn't get exploited through SSH...I've almost never seen that happen, and when it does happen, it's usually because of an unpatched vulnerability in the daemon. Brute forcing a SSH password is usually a lost cause.

Edit: I've been reading up on Osiris this afternoon, and it looks pretty cool. It periodically re-scans and sends you an email of the changed files. The enterprise version of Tripwire offers real-time checking, but of course you have to pay for it. There is an open source version of Tripwire as well, which has picked up the reins from the much much older open source tripwire, and is pretty solid as well.

2
  • Hi Thanks. I have Aide installed now and runs on a weekly cron is there any integrity checker that can notify immediately when some system file changes?
    – user37143
    May 13, 2010 at 15:37
  • @user37143 - see tripwire/osiris as suggested in my post.
    – sh-beta
    May 13, 2010 at 16:48
2

Before anything else: make doubly certain with anyone else who has root/sudo access that they did not reconfigure this. You don't want to go chasing an attack that didn't happen.

Note: an attack scenario sounds unlikely. Most people smart enough to compromise your machine aren't stupid enough to change such obvious things as the port SSH is listening on. Why bother?

Once you've decided that your machine was compromised, consider anything you've typed or stored on that server public information. Change passwords, SSH keys, etc.

Next, you need to determine the most likely point of entry. Unless you've got some ridiculous passwords, I doubt very much that SSH was the original compromise. Astronomically more probable is that someone gained access via one of your webapps.

  • Isolate each of your apps to its own brand new EC2 instance. They're cheap. Security compromises can destroy your business. Doing this will help you to isolate any vulnerabilities to a single app, which has the handy side effect of pointing the finger when/if an attack does occur.

  • Check your versions for known vulnerabilities. Or take the shotgun approach and upgrade everything to its newest version.

  • If any of your apps are running as root, fix that now.

  • If any of your apps are running outside a chroot, fix that now.

  • Create a central syslog server with no external-facing servers. Set every one of your apps to log as verbosely as possible via syslog to this server. When/if an attack occurs, you'll need the forensics to determine where it came from and how they got in.

  • If you're allowing passworded SSH access to these servers, don't. Change to ssh keys.

  • Consider installing Tripwire, Osiris, or another integrity-monitor to determine when something bad happens.

A sophisticated attacker will try to cover his tracks. If you don't have the monitors and safeguards in place to record what's going on in, chasing him down is going to be an exercise in futility. Keep your system as safe as possible and ensure that any odd activity will not go unnoticed.

EDIT

Since you mentioned some permission-denied errors for root, verify that root is the only user assigned to UID 0 (getent passwd 0). If not, you're definitely compromised.

7
  • Never used Osiris, and I've been looking for a Tripwire replacement. Do you like it? May 13, 2010 at 14:52
  • You mean some thing like Splunk for the log monitoring? I started using Aide but it's on a weekly cron. How can I get notified suddenly as something changes? If any of your apps are running outside a chroot, fix that now. Sorry I did not get what you mean this. I run all apps on /home/user/public_html with the /home/user got 755. How they make the sshd_config not even editable by root?
    – user37143
    May 13, 2010 at 15:40
  • @satanicpuppy - i've only touched Osiris in passing, but I've heard very good things.
    – sh-beta
    May 13, 2010 at 16:38
  • @user37143 - serverfault.com/search?q=chroot But really, if you don't know what a chroot is you're in way over your head trying to address security compromises. Consider hiring a consultant or outsourcing your hosting.
    – sh-beta
    May 13, 2010 at 16:44
  • upgrading after finding a security breach is kind of a moot point. chances of the distributions package management utility to completely wipe out the compromise is next to nil. if you want, assess where the attack came from, and how you were compromised. but after that, wipe everything and start over.
    – cpbills
    May 13, 2010 at 16:49
1

If any binares have been changed you can run "rpm -aV". This will md5 match the rpm installed agianst the binary on disk (and also check permissions and such like). Its not exactly tripwire - and wont detect if any user uploaded content has changed obviously but if somethings been changed in /usr or /bin etc it should show up.

It will flag up a bunch of false positives for stuff in /etc which has been deliberately changed, so you'll need to review its output after you've ran it.

0

Are you sure this isn't Rightscales "rightscripts" (chef) just paving over your changes?

Its extremely unlikely your sshd got comprimised unless you're using six character lowercase english word passwords.

2
  • I got a strong passwords generated by python scripts
    – user37143
    May 14, 2010 at 4:30
  • Hi. I tam trying to check whether the Rightscripts and disable it. But I am not able to remove the daemon rightscle
    – user37143
    May 14, 2010 at 9:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .