Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Apache's error_log shows lines like the following:

--- snip ---
which: no ruby in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no locate in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no suidperl in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no get in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no fetch in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no links in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no lynx in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no lwp-mirror in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no lwp-download in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no kav in (/sbin:/usr/sbin:/bin:/usr/bin)
--- end ---

The architecture is:

Internet -> Load Balancer -> Varnish -> Apache

There are several web servers behind the load balancer and I have checked at least one of them with rkhunter (link) and couldn't find anything suspicious.

Versions:

  • CentOS 5.7
  • Varnish 2.1.5
  • Apache 2.2.3
  • PHP 5.2.17

Does this mean that someone has executed the command which through Apache? How can that happen?

Thank you so much.

share|improve this question
add comment

1 Answer

up vote 2 down vote accepted

It is fairly clear you have been hacked. The commands mentioned would be of interest to a hacker who has found a vulnerability letting them execute commands as the apache user (probably through a web application vulnerability). For example, links and lynx would let the attacker download additional programs, as would lwp-*. These commands would never be executed by a legitimate web app so it must be an attacker.

See this thread for an attack with a similar signature - this is actually a privilege escalation.

The first thing is to take the system offline - a tradeoff decision but since you have no idea how completely the system is owned, it's the safe thing to do.

You should attempt to find out when you were hacked - e.g. restore backups from 4 weeks ago onto a separate system, and compare with a backup done last night. This should be a whole machine backup as the attacker might have got into root.

Once you know when you were hacked, you can restore from a backup that was made before the attack (but still keep the system offline so it can't be exploited again). The vulnerability is still there, so the attacker could get in again if the system was online - so you also need to quickly find and close the vulnerability - the complete machine backup from last night is important so you can find how they got in.

See My server's been hacked EMERGENCY for more complete advice on how to recover from a hack.

Good luck.

share|improve this answer
    
Thank you so much. –  Unai Rodriguez Mar 20 '12 at 8:06
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.