0

we are hit by a strange attack, the attacker even emailed to challenge us.

The output of "netstat -n" is just a few "ESTABLISHED" and "TIME_WAIT" typically, yet the server's website cannot be reached from outside. And when typing "ssh 127.0.0.1" on the Console window, it takes almost a minute to get the prompt for password. CPU and memory usage are very low.

The output of "tcpdump -l" averages merely a few lines per second. Most of them are TCP connection with "S" or "R" flags from port 80 of this web server to some remote ips. Strangely, these IPs do not show in the output of "netstat -n"! And these IPs are similar to other ips the attacker used before.

And the symptom (with delayed ssh to 127.0.0.1) will even last after I completely unplug the server from network. Only after around 10 minutes will the symptom suddenly disappear. And this behavior can be repeated.

The server is behind a hardware firewall, that only allows TCP connection to this web server. Is there something that I didn't know, ie, is there some way to let the server remember some TCP requests, which doesn't show by netstat, yet will tie the server resources, so it can hardly handle any subsequent TCP requests?

Improve this question
2
  • What operating system is the server?
    – RedGrittyBrick
    Nov 20, 2010 at 19:58
  • FreeBSD with latest patches.
    – Joe
    Nov 20, 2010 at 20:14

5 Answers 5

Reset to default
1

Network Computing has an article on TCP SYN Flooding Attack and how to handle them.

Improve this answer
1
  • Thanks, that described the initial wave of the attack, which was revealed by the thousands of tcp connections with all kinds of flags when issuing "netstat -n". However, now "netstat -n" only shows a few lines of tcp connections, yet the connection is still slow. This is also mentioned in your link: "So the attacking system only needs to send as few as five packets per minute for each port targeted on the attacked system." But still, will these packets be visible by netstat? Right now nothing suspicious shows there.
    – Joe
    Nov 20, 2010 at 20:29
0

I did not have answer what is causing this - but I have questions to clarify situation. You did no tell if the server load is high ? If the answer is yes - what cause this load - check with top command. Can you try to run tcpdump -ntuap ? Seems only option -n is not enough. Does it run any other service like DNS, FTP ? Is it possible your ssh delay to be caused by DNS resolving problem or high load? How did you connect to the server with ssh if it is disconnected? Does the server has access to the DNS servers if it is disconnected (no resolving = slow ssh connection)? Can you check apache's option "keepalive" on and set it to off lower the interval for it.

Improve this answer
1
  • The server load is very light. It only runs apache. I sit in front of the server to type "ssh 127.0.0.1". tcpdump indeed showed it was trying to talk to DNS server from time to time, although the output of "tcpdump -l" was just a few lines per second. The ISP's DNS server is fast. I can check apache's option, but at that time there was only a few connections to the web server. I will try "tcpdump -ntuap" when the attack resumes.
    – Joe
    Nov 20, 2010 at 20:19
0

Speaking without going into specifics, the combination of symptoms you've described lend themselves most readily to a diagnosis of "rootkit", as it would be difficult (albeit hardly impossible) to generate them any other way.

Improve this answer
3
  • I can not rule out that possibility, although at that time both servers in that subnet got this same problem, while the other server has even less exposure to the internet. They both have latest patches, and essentially only runs apache.
    – Joe
    Nov 20, 2010 at 20:16
  • If more than one server is affected I would check out DNS.. ssh can have long delays when it tries to resolve incoming connections. Setting "UseDNS" to false in sshd_config may help.
    – mfarver
    Nov 20, 2010 at 22:10
  • You are absolutely right, after changing UseDNS to no in sshd, ssh 127.0.0.1 is instant now. Still, the website is not available. This is very worrisome, as netstat shows just a few connections.
    – Joe
    Nov 21, 2010 at 4:06
0

It turned out the attack used a protocol that's not used by our site, so is blocked by our edge router. Therefore although it saturated our outside pipe, little legitimate traffic luckily fell through into our inside servers.

Improve this answer
0

Check packet rate on interfaces with:

netstat -bdh 1

Check what system is doing with: 

systat -v 1

Check some of bsd-limits with: (LIMIT/USED)

vmstat -z

PS. Are you using apache? Check it's stats

PPS. Try putting nginx before it.

PPPS. Are you using accf_http kernel module?

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .