Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Okay, an Ubuntu server I manage fell victim to a DDoS attack today. Usually this is unpleasant but not that big a deal. A few high server load moments & then it passes. Today was clearly different. For the record, I have years of server attack scars. But today felt different. Read on.

The solution was solved by re-enabling a ModSecurity rule that prevents curl access via a call to a pure IP address. Meaning if a hostname is mygreatsite.com and the IP address is 123.456.78.90 and I have a virtual host for mygreatsite.com & that is enabled. And ModSecurity is active with the rule that prevents curl access via a call to a pure IP address, then a visitor would get the proper content when they visit mygreatsite.com, but are blocked by ModSecurity with a 403: Forbidden if they attempt to access the site via 123.456.78.90. Great problem solved!

But while the attack was happening, three things were clear:

  1. Apache Processes Through the Roof: The Apache process load as reflected in Munin was through the roof & consistently 3x higher than normal.
  2. No Corresponding Web Traffic Logged: There was no corresponding traffic logged via Apache & reflected in AWStats. This was the case for a few hours. This is across all logs on all virtual hosts on the server including the default virtual host that would be connected to a bare IP address.
  3. Server Load Was Moderately High, But Not Typical Attack Level: While overall server load was high, it was only in the single digits—4, 5, 6, 7, etc…—and not a typical DDoS where loads can go double and even triple digit like 20, 30 & even 120 (!!!).

To me item number 3 was the most unusual part of this attack. I have not experienced a DDoS where server load was basically equivalent of a heavy traffic day & not an insane server load connected to the attack.

Additionally, running elinks to fetch Apache server-status from the command line showed the connection pool filling up like crazy within minutes of a restart:

elinks http://localhost/server-status?refresh=1

And past any of that, I just noticed an Ubuntu security patch today specifically for curl issues. And this specific issue piqued my interest:

libcurl incorrectly validates wildcard SSL certificates containing literal IP addresses.

So here is the question: What the heck happened? I know re-enabling a ModSecurity rule stopped the attack, but could it have in any way been connected to the libcurl issue? And if so, how can I detect or prove that? And past that if the issue was with libcurl—which I assume is coming from weaponized clients using the flaw—then does that mean that someone can simple keep a hacked older version of curl around to DDoS again? And writing that, any hacker can just download source code & patch it to hack so why the flood today?

My guess is the patch to curl made more people aware of the flaw & the “enthusiasm” of script kiddies around the globe brought a mad flood of scanning around within hours of the exploit being found? But again, this confuses me because it’s on the client curl install, correct? I can’t imagine curl on the server side would be victim to this.

share|improve this question
1  
possible duplicate of I am under DDoS. What can I do? –  Jacob Apr 16 at 3:13
2  
It's not really a DDoS if you can stop it on your end. –  Jacob Apr 16 at 3:13
    
Are you sure that the attack traffic isn't showing in the Apache logs? The "main" vhost might have a different access log than the default vhost which is getting these requests? –  Shane Madden Apr 16 at 3:15
    
I appreciate the duplicate flag, but not to invoke Passover too much, but to me & my experience this attack was different from every other attack I have experienced. The timing of the curl patch & my re-enabling of the related ModSecurity rule connected to curl seems oddly too perfect. –  JakeGould Apr 16 at 3:15
1  
It looks to me like it was a SYN attack. It is only a fraction of a connect to your webserver, meaning that it will deplete the amount of available connections. It being only a SYN request and perhaps some more data, as opposed to a complete HTTP request, it would not show up in the apache logs. –  Jarmund Apr 16 at 7:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.