Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

My wordpress EC2 server was probably hacked. When running iftop command I can see that my server sends and recieves data from a hostname name "i157panamamails.com".

This hostname never goes away and eating lots of bandwidth. I want to prevent the server from communicating with this hostname and deny it completely. I tried putting a deny from in the .htaccess and restart the Apache2 server (using Ubuntu 10.0), but it didn't help, I an still see that hostname connection active. Maybe the connection is not from Apache2, but from what I know this is the only server installed.

How can I prevent the server to deny this hostname?

More info:

  • in etc/hosts I have only 127.0.0.1

  • in etc/resolve.conf I have:

    nameserver [some_ip_address] <--- this IP is for EC2, so not problem here

    domain ec2.internal

    search ec2.internal

  • The IP in the resolve.conf doesn't resolve to host name

  • Using lsof command on port 22/21/80 Can't find the host name in the connections. Port 21 has one connection for root user listening, but port 21 is not open in EC2 security groups and port 443 (HTTPS) has no connections

  • It seems that the connection is not on port 80 of apache because lsof shows only my connection at a specific time frame and I see the connection for that hostname still active in the network print. I assume that another software that acts as a server might initialize that connection on an already open port 80 that is not related to Apache

  • Running NetHogs on eth0 shows me that a program sshd: ubuntu@pts1 under user ubuntu which is continuously receiving and sending data (not Apache2). Looks suspicious

  • I also notice a lot of connection (~50) from a PROGRAM 37:80-[ip_addresses] on port 80 and other ports. Those PROGRAM(s) do not send or receive data (most of them)

  • Blocking the hostname in the host.deny didn't help

Update: I've find out the IP and blocked it in IPtables (both OUTPUT and INPUT). The IP server doesn't send data to that IP, but it dos receives a 240 bytes every second or so.

share|improve this question

3 Answers 3

up vote 2 down vote accepted

Okay, first of all you can run tcpdump on your server to analyze traffic.

tcpdump -s 65535 -w ~/traffic_capture.pcap 

Then you can read from this file with command:

tcpdump -vv -r ~/traffic_capture.pcap 

or download it your computer and check it with wireshark.

Also you can use iptables to log all of your outgoing traffic(here is rule):

iptables -A OUTPUT -j LOG 

By default log of this traffic will be in /var/log/kern.log.

Also you can run iftop with flag -n which will prevent iftop from doing hostname lookups.

Quote from manpages of iftop:

   By  default, iftop will look up the hostnames associated with addresses
   it finds in packets. This can cause substantial traffic of itself,  and
   may  result in a confusing display. You may wish to suppress display of
   DNS traffic by using filter code such as not port domain, or switch  it
   off  entirely, by using the -n option or by pressing r when the program
   is running.

If you are sure that you got some kind of shell or malware, then you can try use maldet or ai-bolit to check for malware in apache's directories.

share|improve this answer
    
You thirds option allowed me to see the IP address! .. checking if I can block it. –  Idan Shechter Aug 31 at 8:32
    
The ip resolves to Netherlands, the same country that ddos my xmlrp.php which I blocked in htaccess earlier. We are in the right direction –  Idan Shechter Aug 31 at 8:35
    
Great that could help you. You can block this ip address on iptables completely. –  Navern Aug 31 at 8:38
    
@Navem How can I enable the ip again if I want it to work again, just in case? –  Idan Shechter Aug 31 at 8:39
1  
You can check which line blockig rule is with iptables -L -v -n --line-numbers and then run iptables -D <CHAIN> <LINE-NUMBER> to delete rule from specific chain on specific line. –  Navern Aug 31 at 8:44

You should try iptables for a quick fix.

iptables -A OUTPUT -d 202.54.1.22 -j DROP

@see http://www.cyberciti.biz/tips/linux-iptables-6-how-to-block-outgoing-access-to-selectedspecific-ip-address.html

On EC2, if you're running in a VPC, you can edit your ACL on-the-fly and block this IP.

However, if you were indeed hacked, this is just the first step. Take a look at the process that's doing those connections, lsof will help, figure out why it's running and who installed it and remove it from your machine if needed.

The second step would be to harden your security: - upgrade wordpress to the latest version; there are some popular exploits for this platform - change all IAM and master account passwords - rotate all aws access keys and ssh keys - make sure you cannot ssh using passwords - if possible, allow ssh only from specific IPs, via Security Groups

share|improve this answer
    
Hi, the problem that I can't reverse the hostname to an IP. This is urgent, please guide.. –  Idan Shechter Aug 31 at 6:40
1  
Hmm, yes I saw that. I cannot ping it either. Take a look in /etc/hosts and see if you have some rules only for this server. –  VladFr Aug 31 at 6:43
1  
Also, /etc/resolv.conf maybe someone changed your DNS settings. You should see IPs of DNS servers in there. –  VladFr Aug 31 at 6:43
    
I update the question with info. In resolv.conf I see a nameserver with IP address (See the question body). Any idea if the IP should be there? –  Idan Shechter Aug 31 at 6:49
    
The IP should be there, it should point to a aws nameserver, usually from the class 10.x.x.x or 172.x.x.x –  VladFr Aug 31 at 7:00

It might be outbound connection rather than inbound. Check below activity:

  1. Top command. You might want to stop other services so that it is more obvious which process is using CPU.
  2. Ping i157panamamails.com and see whether it can resolve to an IP on your server. If it does, block it on iptables.
  3. Block all outbound connections on your AWS security group if you don't need outbound connections.

Apart from those reactive measures, I would also recommend you to install host based intrusion detection software such as OSSEC to detect unauthorized access and changes to your system.

share|improve this answer
    
using top, I see that apache2 and mysqld are the main ones that are consistently have active CPU. I assume that if someone hacked the server, the server might not have a process footpring (or am I wrong)? –  Idan Shechter Aug 31 at 8:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.