0

Recently I noticed that the CPU load of my server is increasing rapidly. In any day, the CPU load does not reach 2.5. I have the following server:

Intel® Xeon® E3-1270 v2 Single Processor - Quad Core Dedicated Server
CPU Speed: 4 x 3.5 Ghz w/ 8MB Smart Cache
Motherboard: SuperMicro X9SCM-F
Total Cores: 4 Cores + 8 Threads
RAM: 32 GB DDR3 1333 ECC
Hard Drive: 120GB
Smart Cache: 8MB

When the server reaches 4.5, I immediately logon to my server using ssh and issue this command netstat -na |grep :80 |wc -l to see how many connection I have. To my surprise it reaches more than 950 connections.

When I look into the IP address by issuing this command netstat -na |grep :80, I saw that there this ip 210.4.99.44 is consuming more than 600 connections.

When I look into the access.log I found the following for this IP address:

210.4.99.44 - - [06/Jul/2013:10:59:23 +0800] "GET /java/4552/sites/all/modules/ctools/css/ctools.css HTTP/1.0" 200 59506 "http://www.mysite.com/java/4552/site$
210.4.99.44 - - [06/Jul/2013:10:59:23 +0800] "GET /forums/programming/c/sites/all/themes/arras/custom.css HTTP/1.0" 404 24875 "http://www.mysite.com/forums/pr$
210.4.99.44 - - [06/Jul/2013:10:59:23 +0800] "GET /comment/reply/3673/21593/misc/drupal.js HTTP/1.0" 200 32289 "http://www.mysite.com/comment/reply/3673/21593$
210.4.99.44 - - [06/Jul/2013:10:59:23 +0800] "GET /forums/programming/c/sites/all/themes/arras/arras.css HTTP/1.0" 404 25012 "http://www.mysite.com/forums/pro$
210.4.99.44 - - [06/Jul/2013:10:59:23 +0800] "GET /forums/programming/c/sites/all/themes/arras/arras-blue.css HTTP/1.0" 404 24921 "http://www.mysite.com/forum$
210.4.99.44 - - [06/Jul/2013:10:59:23 +0800] "GET /comment/reply/3673/21593/modules/system/system.messages.css HTTP/1.0" 200 32507 "http://www.mysite.com/comm$
210.4.99.44 - - [06/Jul/2013:10:59:23 +0800] "GET /comment/reply/3673/21593/modules/system/system.theme.css HTTP/1.0" 200 32691 "http://www.mysite.com/comment$
210.4.99.44 - - [06/Jul/2013:10:59:23 +0800] "GET /comment/reply/3673/21593/modules/user/user.css HTTP/1.0" 200 32378 "http://www.mysite.com/comment/reply/367$
210.4.99.44 - - [06/Jul/2013:10:59:23 +0800] "GET /comment/reply/3673/21593/misc/jquery.once.js HTTP/1.0" 200 32586 "http://www.mysite.com/comment/reply/3673/$
210.4.99.44 - - [06/Jul/2013:10:59:24 +0800] "GET /comment/reply/3673/21593/misc/textarea.js HTTP/1.0" 200 32543 "http://www.mysite.com/comment/reply/3673/215$
210.4.99.44 - - [06/Jul/2013:10:59:24 +0800] "GET /forums/programming/c/sites/all/modules/views/css/views.css HTTP/1.0" 404 24931 "http://www.mysite.com/forum$
210.4.99.44 - - [06/Jul/2013:10:59:24 +0800] "GET /comment/reply/3673/21593/misc/jquery.js HTTP/1.0" 200 32424 "http://www.mysite.com/comment/reply/3673/21593$
210.4.99.44 - - [06/Jul/2013:10:59:24 +0800] "GET /comment/reply/3673/21593/modules/comment/comment.css HTTP/1.0" 200 32556 "http://www.mysite.com/comment/rep$
210.4.99.44 - - [06/Jul/2013:10:59:24 +0800] "GET /forums/programming/c/sites/all/modules/video_filter/video_filter.css HTTP/1.0" 404 24868 "http://www.mysite$
210.4.99.44 - - [06/Jul/2013:10:59:24 +0800] "GET /comment/reply/3673/21593/modules/system/system.base.css HTTP/1.0" 200 32663 "http://www.mysite.com/comment/$
210.4.99.44 - - [06/Jul/2013:10:59:29 +0800] "GET /forums/programming/visual-basic-60/sites/all/themes/arras/ie6.css HTTP/1.0" 404 24973 "http://www.mysite.co$
210.4.99.44 - - [06/Jul/2013:10:59:29 +0800] "GET /forums/programming/visual-basic-60/sites/all/themes/arras/arras-blue.css HTTP/1.0" 404 24048 "http://www.mysite$
210.4.99.44 - - [06/Jul/2013:10:59:29 +0800] "GET /forums/programming/c/c/sites/all/modules/ctools/css/ctools.css HTTP/1.0" 404 24945 "http://www.mysite.com/f$
210.4.99.44 - - [06/Jul/2013:10:59:29 +0800] "GET /forums/programming/visual-basic-60/sites/all/modules/video_filter/video_filter.css HTTP/1.0" 404 25067 "http://www.$
210.4.99.44 - - [06/Jul/2013:10:59:33 +0800] "GET /sites/default/files/download/donk90/file1.zip HTTP/1.0" 200 133016 "http://www.mysite.com/sites/default/fi$
210.4.99.44 - - [06/Jul/2013:10:59:34 +0800] "GET /forums/programming/c/c/misc/jquery.js HTTP/1.0" 404 24918 "http://www.mysite.com/forums/programming/c/c/mis$
210.4.99.44 - - [06/Jul/2013:10:59:34 +0800] "GET /forums/programming/visual-basic-60/sites/all/themes/arras/suckerfish.css HTTP/1.0" 404 25088 "http://www.mysite$
210.4.99.44 - - [06/Jul/2013:10:59:34 +0800] "GET /forums/programming/visual-basic-60/sites/all/themes/arras/custom.css HTTP/1.0" 404 24853 "http://www.mysite$
210.4.99.44 - - [06/Jul/2013:10:59:34 +0800] "GET /forums/programming/c/c/misc/drupal.js HTTP/1.0" 404 25001 "http://www.mysite.com/forums/programming/c/c/mis$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/sites/all/themes/arras/html-elements.css HTTP/1.0" 404 24943 "http://www.mysite$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/modules/user/user.css HTTP/1.0" 404 25182 "http://www.mysite.com/forums/$210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/sites/all/themes/arras/arras.css HTTP/1.0" 404 24962 "http://www.mysite.$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/sites/all/themes/arras/geshifilter-languages.css HTTP/1.0" 404 24957 "http://www$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/public:/geshi/geshifilter-languages.css HTTP/1.0" 404 24855 "http://www.mysite$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/sites/all/modules/geshifilter/geshifilter.css HTTP/1.0" 404 25081 "http://www.so$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/modules/system/system.base.css HTTP/1.0" 404 24883 "http://www.mysite.co$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/modules/system/system.menus.css HTTP/1.0" 404 25044 "http://www.mysite.c$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/sites/all/modules/views/css/views.css HTTP/1.0" 404 24914 "http://www.mysite$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/modules/field/theme/field.css HTTP/1.0" 404 24926 "http://www.mysite.com$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/modules/system/system.messages.css HTTP/1.0" 404 24976 "http://www.mysiteste$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/sites/all/modules/ctools/css/ctools.css HTTP/1.0" 404 24960 "http://www.mysite$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/modules/system/system.theme.css HTTP/1.0" 404 25088 "http://www.mysite.c$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/modules/book/book.css HTTP/1.0" 404 24941 "http://www.mysite.com/forums/$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/modules/forum/forum.css HTTP/1.0" 404 24989 "http://www.mysite.com/forum$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/misc/jquery.once.js HTTP/1.0" 404 24950 "http://www.mysite.com/forums/pr$
210.4.99.44 - - [06/Jul/2013:10:59:35 +0800] "GET /forums/programming/visual-basic-60/modules/node/node.css HTTP/1.0" 404 24970 "http://www.mysite.com/forums/$
210.4.99.44 - - [06/Jul/2013:10:59:36 +0800] "GET /forums/programming/visual-basic-60/modules/poll/poll.css HTTP/1.0" 404 25049 "http://www.mysite.com/forums/$
210.4.99.44 - - [06/Jul/2013:10:59:36 +0800] "GET /forums/programming/visual-basic-60/misc/drupal.js HTTP/1.0" 404 24885 "http://www.mysite.com/forums/program$
210.4.99.44 - - [06/Jul/2013:10:59:36 +0800] "GET /users/jvidals/sites/all/themes/arras/ie6.css HTTP/1.0" 404 24955 "http://www.mysite.com/users/jvidals/sites$
210.4.99.44 - - [06/Jul/2013:10:59:36 +0800] "GET /forums/programming/visual-basic-60/modules/comment/comment.css HTTP/1.0" 404 24889 "http://www.mysite.com/f$
210.4.99.44 - - [06/Jul/2013:10:59:36 +0800] "GET /forums/programming/visual-basic-60/misc/jquery.js HTTP/1.0" 404 24928 "http://www.mysite.com/forums/program$
210.4.99.44 - - [06/Jul/2013:10:59:36 +0800] "GET /users/jvidals/sites/all/themes/arras/suckerfish.js HTTP/1.0" 404 24915 "http://www.mysite.com/users/jvidals$
210.4.99.44 - - [06/Jul/2013:10:59:36 +0800] "GET /users/jvidals/sites/all/themes/arras/suckerfish.css HTTP/1.0" 404 24946 "http://www.mysite.com/users/jvidal$
210.4.99.44 - - [06/Jul/2013:10:59:38 +0800] "GET /sites/default/files/download/VincentProgrammer/phonebook.zip HTTP/1.0" 200 46500 "http://www.mysite.com/sit$
210.4.99.44 - - [06/Jul/2013:10:59:47 +0800] "GET /java/4552/misc/jquery.js HTTP/1.0" 200 45624 "http://www.mysite.com/java/4552/misc" "WE 9.50"
210.4.99.44 - - [06/Jul/2013:10:59:53 +0800] "GET /tutorials/php/php-tutorial.html HTTP/1.0" 200 45632 "http://www.mysite.com/comment/reply/3673/sites/all/the$
210.4.99.44 - - [06/Jul/2013:10:59:54 +0800] "GET /tutorials/php/php-tutorial.html HTTP/1.0" 200 45640 "http://www.mysite.com/comment/reply/3673/sites/all/mod$
210.4.99.44 - - [06/Jul/2013:10:59:58 +0800] "GET /tutorials/php/php-tutorial.html HTTP/1.0" 200 45640 "http://www.mysite.com/comment/reply/3673/modules/field$
210.4.99.44 - - [06/Jul/2013:11:00:26 +0800] "GET /sites/default/files/download/user/voting_system.zip HTTP/1.0" 200 1227639 "http://www.mysite$

Note that this is just a few sample pages that this IP address is accessing.

I hurriedly block the IP address using iptables and the CPU load drops quickly to 0.8...

Is this some form of a DDOS attack?

I thought that a DDOS attack is coming from different IP Addresses, but what happen to me is there is only one IP address. And also I thought that a DDOS attack just send a packet using ping command. But the log in my server seems that the IP address is opening a web page on my site (just like a normal visitor), those it does not ping my server.

The IP address does not stop sending request for almost two hours until I block it.

So what kind of attack is this? And what tools they are using? I am thinking that this is an "ab" benchmarking tool. But I haven't tried it yet so I can't make sure they are using that.

BTW, after blocking this IP address, the connection drops to 280 something. So I believe that this IP is really attacking my server.

Any help please.

7
  • 3
    Well, its certainly not DDos if its only one ip, rather DOS Jul 6, 2013 at 16:55
  • So what kind of attack is this? Why it seems that it can take my server down? If I did not noticed it, may be it consumes all the CPU load on my server.
    – jaYPabs
    Jul 6, 2013 at 16:59
  • It looks like to just be a web spider. Has this occurred from the same IP often? If it's loading your server like that you may want to put something like Varnish in front of your server to serve those static files.
    – Nathan C
    Jul 6, 2013 at 17:02
  • @NathanC I just noticed it today, one moment while I checked the yesterday's log.
    – jaYPabs
    Jul 6, 2013 at 17:04
  • @NathanC I can't find this IP add on yesterday's log and on last two days log also.
    – jaYPabs
    Jul 6, 2013 at 17:07

2 Answers 2

4

I wouldn't worry about this. It looks to be a crawler that decided to visit and they frequently hit pages rather quickly - and from the looks of it there's tons of stylesheets on your site. I highly recommend using some sort of caching (Varnish, nginx front-end, a CDN if needed) to lessen the burden of Apache.

4
  • Out of curiosity: how can you tell from looking at that log that this is a crawler (rather than a DOS attack?) Jul 6, 2013 at 19:48
  • Because scrapers, web downloaders and web indexers (search engines) typically are just sequentially reading through your website. DOS would be several thousand continuous requests for a handful of pages Jul 6, 2013 at 21:00
  • I'm actually using nginx right now except varnish and CDN. May be i'll try varnish first. Is there a way to limit the # of connection from the same ip address using iptables?
    – jaYPabs
    Jul 6, 2013 at 22:52
  • Yes, there is.
    – Nathan C
    Jul 6, 2013 at 23:18
4

First, it's not a DDOS attack without the "Distributed" (Multiple IPs). Since this has caused problems, you can properly call it a DOS attack.

But from the looks of it, it wasn't intended to be a denial of service attack. It looks more like a crawler. (We can tell because the URLs aren't random, they appear to be in a 'follow all links on a page' pattern.) The crawler was going entirely too fast, but it's an easy thing to do. For example, try "wget in crawl mode".

Adding a caching layer will NOT help against crawlers (unless your site is very small and fits entirely in the cache). A better fix is to run mod_evasive -- that will at least force the crawler to slow down. Evading real (D)DOS attacks is complicated, but this was NOT a "real" attack.

But let's take a closer look at those logs:

"GET /.../file1.zip HTTP/1.0" 200
"GET /../jquery.js HTTP/1.0" 404
"GET /../suckerfish.css HTTP/1.0" 404
"GET /../custom.css HTTP/1.0" 404
"GET /../drupal.js HTTP/1.0" 404

It looks like your pages contain a lot of invalid links. That multiplied his attack (since he followed every one of them). Also, do you really need a zip file link? That's probably expensive to generate. Consider trying to block robots from your site (or from your zip files) via robots.txt. It might also help to post a policy on crawling on your site (and/or provide a dump of your data directly!)

Pro tip: If your pages are generated by a script (i.e. mod_php or mod_perl), then you're using Apache as an app server. Every time you need to serve a static request, you're wasting a few hundred MB of RAM. Move your images/javascript/css to a different web server with a smaller footprint. One way is to setup a 2nd Apache server configured without modules, or nginx in reverse proxy mode. But an even simpler way is to use a CDN server like S3 or CloudFront. Then your app server won't be involved on "static" requests at all, and the page will go faster since it can download static elements in parallel with the dynamic elements.

4
  • Using PHP does not necessarly mean using Apache.
    – Bogdacutu
    Jul 6, 2013 at 21:59
  • Actually I compress and minify all js and css, but wondering why this bot can access separate js and css file. Is the bot also downloading all my files? If this is the case? I am actually serving downloadable files on my free programming website. I'm already using nginx. Is using varnish is what you mean by reverse proxy? Adding additional server would cost me another money, so I can't do it right now.
    – jaYPabs
    Jul 6, 2013 at 22:59
  • BTW, you are right about the invalid links. But I don't know why this is happening. It seems that instead of /sites/all/modules/views/css/views.css, the bot is now accessing /forums/programming/c/sites/all/modules/views/css/views.css which is not right. Normally visitors or even bot should not actually access this css file as it is used only on admin side. And I am minifying css and js file as well.
    – jaYPabs
    Jul 6, 2013 at 23:26
  • Sorry, I did assume you were using PHP in Apache. If you're using Nginx, you're fine. You need to use "limit_req_zone" in nginx instead of mod_evasive on Apache. Jul 19, 2013 at 19:52

Not the answer you're looking for? Browse other questions tagged .