Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This question already has an answer here:

I'm getting a lot of hits on my server. This server normally gets little to no traffic yet there's is constant hits every time I bring the server back up. I get the following error first ip_conntrack: table full, dropping packet then sooner or later my httpd runs out of memory and my server becomes unresponsive. Any ideas on how to fix it?

latest head of my access_log. I changed http to hxxp

122.193.164.5 - - [27/Mar/2011:23:48:35 -0700] "GET hxxp://pubs.acs.org/templates/jsp/_style2/_achs/css/atypon-main.css HTTP/1.0" 200 174299 "hxxp://pubs.acs.org/doi/abs/10.1021/ac100095u" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)"

218.29.188.217 - - [27/Mar/2011:23:48:38 -0700] "GET hxxp://rotator.adjuggler.com/servlet/ajrotator/913831/0/vh?ajecscp=1301294917498&z=pdn&dim=753179&kw=&click=http://ad.yieldads.com/clk?2,13%3B5900475f5cba1a74%3B12efb38a54b,0%3B%3B%3B1304299909,cl1GAPp3GABp04QAAAAAAEfOIQAAAAAAAgAAAAIAAAAAAP8AAAABGF1nJgAAAAAAJ6sXAAAAAAD1YSwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAn.A8AAAAAAAIAAwAAAAAAS6U4-y4BAAAAAAAAADY2ZjM3ZGE0LTU5MDctMTFlMC04MzUwLTAwMzA0OGQ3MjBhOABmlSoAAAA=,,http%3A%2F%2Fwww.healthcarefinancenews.com%2F, HTTP/1.0" 200 1181 "http://ad.yieldmanager.com/iframe3?cl1GAPp3GABp04QAAAAAAEfOIQAAAAAAAgAAAAIAAAAAAP8AAAABGF1nJgAAAAAAJ6sXAAAAAAD1YSwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAn.A8AAAAAAAIAAwAAAAAAwMqhRbbzxT.AyqFFtvPFP1yPwvUoXM8.XI.C9Shczz9mZmZmZmbWP2ZmZmZmZtY.AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAbr8TXwhPZCb-NEWYczMEV.VtRMDgbQFgGd6CwAAAAAA==,,http%3A%2F%2Fwww.healthcarefinancenews.com%2F,Z%3D300x250%26s%3D1603578%26_salt%3D954499605%26B%3D12%26m%3D2%26u%3Dhttp%253A%252F%252Fwww.healthcarefinancenews.com%252F%26r%3D1,66f37da4-5907-11e0-8350-003048d720a8" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040206 Firefox/0.8"

117.41.182.55 - - [27/Mar/2011:23:48:38 -0700] "GET hxxp://www5.tellgames.com/media/games/images/tellgames/120x90/02470dca7676598b9381e4c5dc2eef05.jpg HTTP/1.0" 200 4883 "http://us.tellgames.com/index.php?category=17&sortby=play&referer=ad2games" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 95)"

117.41.186.191 - - [27/Mar/2011:23:48:37 -0700] "GET hxxp://s0.2mdn.net/1361550/K2147_NBRD_FYEA_728.jpg HTTP/1.0" 200 41371 "hxxp://ad.doubleclick.net/adi/N3340.161249.ADNETIK.COM/B5252096.3;sz=728x90;click=http://ad.z5x.net/clk?2,13%3B6b9391cec2a21533%3B12efb389ce8,0%3B%3B%3B2955295377,s5mFAKglGQBtfoAAAAAAAJJyIQAAAAAAAgAAAAYAAAAAAP8AAAABGB5.JwAAAAAAd0IfAAAAAABy8CsAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABdhBAAAAAAAAIAAwAAAAAA6Jw4-y4BAAAAAAAAADY1YTAxMzY4LTU5MDctMTFlMC1iMTJmLTAwMzA0OGQ3NTRlMABwpioAAAA=,,http%3A%2F%2Fwww.providesearch.com%2F,;pc=[TPAS_ID];ord=[timestamp]" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040614 Firefox/0.9"

173.252.208.155 - - [27/Mar/2011:23:48:38 -0700] "GET hxxp://ads.smowtion.com/st?ad_size=160x600&section=1739112 HTTP/1.0" 200 1336 "hxxp://www.consumerhealthdigest.info/category/health-information" "Mozilla/5.0 (X11; U; SunOS sun4u; en-US; rv:1.4) Gecko/20030701"

61.139.105.162 - - [27/Mar/2011:23:48:38 -0700] "GET hxxp://therugged.com/wp-content/uploads/2011/01/Steph61-80x53.jpg HTTP/1.0" 200 2980 "hxxp://www.therugged.com/category/lifestyle#player" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1"

share|improve this question

marked as duplicate by Tom O'Connor Aug 25 '13 at 21:36

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

    
The links in these logs are not on my server –  user76112 Mar 28 '11 at 7:46
add comment

4 Answers

up vote 1 down vote accepted

Are these domains you are hosting? I suspect not.

I've seen a big increase lately in scanning for open http proxies on my machines - it looks like you may running an open http proxy (which is just as bad as running an open mail relay - worse even, since most people now implement carious mitigations like RBL and SPF).

Disable proxying / add authentication / restrict to your LAN addresses.

OTOH if you really are webmaster for all these domains then have a look at mod_evasive and mod_security.

share|improve this answer
    
No these domains are not mine. –  user76112 Mar 28 '11 at 8:16
    
I tried disabling all Proxy mods for apache. That didn't stop it. –  user76112 Mar 28 '11 at 8:34
    
Thanks. I just followed this site wiki.apache.org/httpd/ProxyAbuse and I was able to deny the proxy with 403 Forbidden. –  user76112 Mar 28 '11 at 9:15
add comment

From the time-stamps, it does not seem like a very high hit-rate but from the IPs, it seems to be originating from all over. Most web-servers should be able to handle a few hits a second. However, you can try a few things to mitigate your problem.

  1. If some of these connections are blocking the connections by holding onto an open connection, you could reduce the keep-alive time-out for each connection.
  2. Check that your httpd is not consuming too much memory by reducing the maximum number of listening processes and threads.
  3. Park your web-server behind a reverse proxy like varnish/pound and filter the destination connections at the edge, dropping invalid connections immediately.
  4. Beef up your server to be able to handle the larger number of connections. Regularly test things by using siege or apache bench to ensure that you can handle a reasonable load.
share|improve this answer
    
1) I don't have keep alive on. 4) This web server should get a few hits a day. I'm having a hard time looking to reduce the number of processes and threads, any tips on what I should look up? –  user76112 Mar 28 '11 at 7:44
    
#1 - you need to specifically turn it off, otherwise, it may be on a default value. #4 - it depends on your webserver, you will need to consult the documentation. –  sybreon Mar 28 '11 at 8:19
add comment

Here's a little IPtables script that will limit the number of connections from individual IPs in a specified period. It's stopped this kind of thing for us in the past. It uses the recent module (which most modern OSs will load as an IPtables module automatically).

It does look as though those HTTP hits might not be causing the Web Server issue but this will just limit all traffic from an IP so won't be concerned with just HTTP hits if you remove the --dport option. As with anything IPtables related: use with caution (try it locally first if it's a remote Server).

iptables -F; iptables -X; iptables -nL

iptables -I INPUT -p tcp --dport 80 -m state --state NEW -m recent --set

iptables -I INPUT -p tcp --dport 80 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl -j DROP

iptables -I INPUT -p tcp --dport 80 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl -j LOG --log-level info --log-prefix IP-DROP:

share|improve this answer
    
Oh I forgot, this will help remove your conntrack errors and should be pretty harmless to increase "echo 131072 > /proc/sys/net/ipv4/ip_conntrack_max". YMMV. HTH, JR –  Jonathan Ross Mar 28 '11 at 9:02
add comment

Use a reverse proxy, I would suggest Nginx because its light weight.

Setup a server section for your domain and one as a default for catch all requests. Any requests that arrive on the catch all domain return a 404 on them.

Also turn off KeepAlive by setting it to 0 and you probably want to disable logging for the default section till the attack goes down, else it will fill up our disk.

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.