I'm a bit out of me league here (we're a reasonably small firm, I'm a software dev stuck with doing sysadmin when needed), but I thought I'd ask the smart people at ServerFault about my problem before we called in our 3rd party IT support firm.
We're experiencing a massive traffic spike at the moment, similar to a spike we experienced in October, which went away by itself. If you'll see our ISP's internet usage monitor:
You'll notice that in the last 2.5 days we've been maxing out our ADSL2 (~20mbps) connection. Ironically, it was Australia day (a public holiday) for one of those days.
We own a Fortinet Fortigate Internet appliance which does our logging and internet connectivity. Here is its snapshot of our usage: This one taken yesterday:
This one today:
You'll see that the connection was being absolutely maxed out until we arrived in the office yesterday morning, then it was pretty much maxed out (a lot higher than usual, as you can probably gather from the Internode monthly history image) until we left and then it started 100% usage again. Finally, at around 11ish Internode finally capped us (odd, given we'd been over our limit by heaps for the past 2 days).
We have a subscription to FAMS, Fortinet's online logging and reporting service. We also have our Fortigate export our logs to a syslog server. I've looked at FAMS and this is what the top service usage by destination log looks like:
As you can see, there's only around 8 or 9 logged there, which is about normal for us, at least it's no where near the 167gb that we've been logged on Internode as using.
This puzzles me - clearly the Fortigate appliance has some sort of log of the traffic, as its utilization snapshot has it there, but in the detailed logs (syslogs didn't show much, but I don't know how to parse them in an efficient way, I've just been watching them stream in) there is nothing.
My question is, any ideas what sort of traffic this could be? I'm thinking perhaps the Fortigate doesn't bother logging certain types of traffic (ICMP ?) and we're being DOS'ed through that type of traffic. I should mention we do have publicly accessible URLs that are password secured, but our uploads are not included in our quota so I don't think that is it.
Any tips on where I should look? Or should I just call in the big guns (or perhaps just wait till it goes away like last time...)
EDIT: Here is another report from FAMS, this one goes by web requests I believe, unfortunately I can't get a report across all ports for this: