Okay, an Ubuntu server I manage fell victim to a DDoS attack today. Usually this is unpleasant but not that big a deal. A few high server load moments & then it passes. Today was clearly different. For the record, I have years of server attack scars. But today felt different. Read on.
The solution was solved by re-enabling a
ModSecurity rule that prevents
curl access via a call to a pure IP address. Meaning if a hostname is
mygreatsite.com and the IP address is
123.456.78.90 and I have a virtual host for
mygreatsite.com & that is enabled. And
ModSecurity is active with the rule that prevents
curl access via a call to a pure IP address, then a visitor would get the proper content when they visit
mygreatsite.com, but are blocked by ModSecurity with a
403: Forbidden if they attempt to access the site via
123.456.78.90. Great problem solved!
But while the attack was happening, three things were clear:
- Apache Processes Through the Roof: The Apache process load as reflected in
Muninwas through the roof & consistently 3x higher than normal.
- No Corresponding Web Traffic Logged: There was no corresponding traffic logged via Apache & reflected in AWStats. This was the case for a few hours. This is across all logs on all virtual hosts on the server including the default virtual host that would be connected to a bare IP address.
- Server Load Was Moderately High, But Not Typical Attack Level: While overall server load was high, it was only in the single digits—4, 5, 6, 7, etc…—and not a typical DDoS where loads can go double and even triple digit like 20, 30 & even 120 (!!!).
To me item number 3 was the most unusual part of this attack. I have not experienced a DDoS where server load was basically equivalent of a heavy traffic day & not an insane server load connected to the attack.
elinks to fetch Apache
server-status from the command line showed the connection pool filling up like crazy within minutes of a restart:
libcurl incorrectly validates wildcard SSL certificates containing literal IP addresses.
So here is the question: What the heck happened? I know re-enabling a
ModSecurity rule stopped the attack, but could it have in any way been connected to the
libcurl issue? And if so, how can I detect or prove that? And past that if the issue was with
libcurl—which I assume is coming from weaponized clients using the flaw—then does that mean that someone can simple keep a hacked older version of
curl around to DDoS again? And writing that, any hacker can just download source code & patch it to hack so why the flood today?
My guess is the patch to
curl made more people aware of the flaw & the “enthusiasm” of script kiddies around the globe brought a mad flood of scanning around within hours of the exploit being found? But again, this confuses me because it’s on the client
curl install, correct? I can’t imagine
curl on the server side would be victim to this.