Could this be a DOS attack?

Question

I'm a bit out of me league here (we're a reasonably small firm, I'm a software dev stuck with doing sysadmin when needed), but I thought I'd ask the smart people at ServerFault about my problem before we called in our 3rd party IT support firm.

We're experiencing a massive traffic spike at the moment, similar to a spike we experienced in October, which went away by itself. If you'll see our ISP's internet usage monitor:

You'll notice that in the last 2.5 days we've been maxing out our ADSL2 (~20mbps) connection. Ironically, it was Australia day (a public holiday) for one of those days.

We own a Fortinet Fortigate Internet appliance which does our logging and internet connectivity. Here is its snapshot of our usage: This one taken yesterday:

This one today:

You'll see that the connection was being absolutely maxed out until we arrived in the office yesterday morning, then it was pretty much maxed out (a lot higher than usual, as you can probably gather from the Internode monthly history image) until we left and then it started 100% usage again. Finally, at around 11ish Internode finally capped us (odd, given we'd been over our limit by heaps for the past 2 days).

We have a subscription to FAMS, Fortinet's online logging and reporting service. We also have our Fortigate export our logs to a syslog server. I've looked at FAMS and this is what the top service usage by destination log looks like:

As you can see, there's only around 8 or 9 logged there, which is about normal for us, at least it's no where near the 167gb that we've been logged on Internode as using.

This puzzles me - clearly the Fortigate appliance has some sort of log of the traffic, as its utilization snapshot has it there, but in the detailed logs (syslogs didn't show much, but I don't know how to parse them in an efficient way, I've just been watching them stream in) there is nothing.

My question is, any ideas what sort of traffic this could be? I'm thinking perhaps the Fortigate doesn't bother logging certain types of traffic (ICMP ?) and we're being DOS'ed through that type of traffic. I should mention we do have publicly accessible URLs that are password secured, but our uploads are not included in our quota so I don't think that is it.

Any tips on where I should look? Or should I just call in the big guns (or perhaps just wait till it goes away like last time...)

EDIT: Here is another report from FAMS, this one goes by web requests I believe, unfortunately I can't get a report across all ports for this:

Answer 1

This link pointed me to the answer: http://forums.adobe.com/thread/391741

The problem was adobe updated and our fortigate router didn't like each other, causing an infinite loop. I would have thought that sort of thing should show up in the firewall logs, but I had a look at the 'requests' version rather than megabytes and one computer was trying to head to adobe for updates.

Looking at the thread, this was the problem:

• Computer was trying to auto update adobe
• Adobe starts the download of the update file
• The Fortigate has http virus scanning, it caches the file and gets ready for virus scanning
• Adobe thinks that the delay means that the download hasn't worked, so it scraps it and asks for it again
• This goes on and on, the file never gets to the client PC, meaning that it is never actually logged in the full fortinet logs.

At least its something along those lines, for now I've turned off the fortigates virus scanning of HTTP requests. But I will look into just blocking adobe from everything, or modifying the scanner settings.

Thanks everyone for all your help - I appreciate it!

Answer 2

Question:

On your webserver - are you seeing a ton of ip addresses in the logs over and over again... AND are they all pulling the same file or query ...

Generally a dos will have some stream to it that you can follow if you get deep into the logs

for a temporary (or perm solution - )check into a firewall off your network) This may help if it is a ddos

www.CloudFlare.com - we use this for a very highly political website about terrorism. The site has not seen a DDOS now for well over 4 months - and we used to battle it literally every single week.

Good news - its free -) and while it is meant as a firewall - it generally will also act as a free CDN service.

Answer 3

The traffic history shows the heavy traffic as inbound on the WAN interface and the graph shows the bulk of the traffic as being HTTP. Have you checked to see what the destination ip addresses are? Are they web servers, streaming media servers, etc? Could this be someone in your office downloading files from the internet, streaming music or videos, etc? I would be very surprised if a DOS attack were able to ramp up that much traffic. Are you able to see both the source and destination ip addresses in the graph? That would give you a better ides of what's going on.