0

I work for a company on my university's campus, helping with general IT problems and some web development. But lately there has been a problem that has me and my boss completely stumped. We, plus one contractor, make up the entire IT department, so I'm reaching out to you for help.

All around the office, we have wall jacks. These collect in a closet down the hall and all plug into a switch. This switch, along with our individual server jacks, plugs into another switch, and that switch plugs into our firewall hardware. Then the firewall is connected out to our campus network. Our campus internet is, well, very fast. I don't know exactly the terms, tiers, etc., but we have thousands of students and downloads can run as fast as 10 MB/s at night; uploads are sometimes even faster. I think we're practically ISP level. In short, I have a lot of faith that it is not the campus side of things that is causing a problem, combined with other evidence I'll mention in a moment.

So our symptoms: web browsing is fast. Web pages, images, etc. load instantly. No problems there. But then when I go to download something, the download starts fast but very quickly (a matter of seconds) drops to nearly 0. Often it will actually drop to 0 and time out. This happens with even very small files, 1 MB or less.

It smells to me like a QoS sort of thing. I'm not entirely sure, and I wanted to get your opinions first. My boss is hesitant to touch our firewall, much less let me touch it, and it was set up and is managed by a consultant remotely.

These problems don't seem tied to a time of the day. I've tried downloads after 5:00 and still the same thing happens.

From my desk, I can turn on my wireless adapter and pick up the campus wireless access point. If I unplug ethernet and connect to it, downloads are fast. This adds to my suspicion that it's limited to our company network.

Also, a number of weeks ago the consultant upgraded our firewall firmware. Suddenly everything was very fast. I tested with downloads from Sun and speedtest.net and things were blazing fast, as they should be with our campus internet! It was wonderful, and I figured the slow speeds were an old firmware bug. In a matter of days, things steadily declined until they were back to the old symptoms.

Oh, and we have antivirus installed on every computer, and we keep it up to date. Though I suppose the possibility is still there that someone could have spyware which is bogging down our internet, in which case what is the easiest/best way to find this out? (maybe this should go in a separate question)

Thank you for your patience in reading all of this. Do you have any ideas as to what I can try? Is this something that you've experienced before? What sort of tools or methods can I use to try and diagnose the problem?

P.S. everything here is Windows. Windows Server 2003 and 2008 on our servers, and Windows XP on employees' machines.

Update: We are submitting a ticket to the university to just take a look and see if they see anything unusual and/or can suggestion methods for us to try and pinpoint our problem. Hopefully they'll be helpful! I'll update this to let you know what goes on.

Update again: We found a hub (yes, a HUB) right between our campus connection and our firewall. It had only those two ethernet cables plugged into it, nothing else. After removing the hub, our speeds have jumped up to several mbps and no more dropped downloads. However in talking with the campus, we got them to run a gigabit line to our firewall in place of the 100mbps line, and we also upgraded all of our switches to gigabit. As of friday, we are at about 65 mbps up and down (according to speedtest.net at 8am)!! Go NC State!!

Improve this question

4 Answers 4

Reset to default
1

What does the Campus IT department say? Some areas on the local campus here are bandwidth throttled based on what the traffic is. Torrents and streaming audio/video get throttled heavily. Internal traffic is fast unless it is student A streaming to Student B. Traffic is shaped based on content or the area you are in. Engineering prof's lab= lots of bandwidth. Dorm room Not so much. Perhaps your area is receiving less bandwidth from the campus network.

Improve this answer
2
  • I suppose it is worth asking if there's anything they can do to help diagnose the problem, but I believe the campus doesn't do any shaping because of our gigantic bandwidth. People host servers in their dorm rooms because the dorm internet is the same as everywhere else. In addition, the symptoms going away after the firewall reboot suggest something more like ManiacZX talked about in his answer.
    – Ricket
    Jan 22, 2010 at 18:37
  • What is the make/model of firewall on your segment of the network?
    – Dave M
    Jan 22, 2010 at 19:03
0

Since the firmware update seemed to help temporarily and then went away over time it may not have been the firmware update itself but the fact that the firewall was rebooted in the process.

I would test a download, power cycle the firewall, test again.

If power cycling temporarily remedies your issue, then you have your smoking gun against the firewall.

One explanation I could think of would be a college network is going to be riddled with junk, all those students with poorly secured systems riddled with spyware sending all kinds of traffic around. The firewall may just be getting overloaded from all of that and the power cycle clears it up temporarily.

It could also just in fact be a faulty unit.

Since firmware was just updated, it either isn't software or it is still an open issue and so probably won't be seeing an instant remedy that way.

What kind of firewall is this? If it is a simple SOHO router, just try tossing another in place and see how it acts, can very simply swap the other back.

Improve this answer
5
  • I'm not so sure though. Sure, the reboot seems to pin the firewall, but let's say for example that the bottleneck is a giant bittorrent stream with hundreds of connected peers. After a firewall reboot, all the connections are closed, things are fast, but then over time the torrent software broadcasts itself to trackers again, peers reconnect and everything is gradually slowed down again. The firewall would have nothing to do with something that. It's the kind of thing that spyware might do, right?
    – Ricket
    Jan 22, 2010 at 18:40
  • True, that goes back to my suggestion of swapping firewalls as a test. Also, if it is more than a simple SOHO router, you should be able to see at least the total bandwidth used and possibly per host how much bandwidth they are using to show if it is purely your traffice being eaten up.
    – ManiacZX
    Jan 22, 2010 at 20:41
  • Also, if you are that concerned about spyware, run a scan on everyone's computers with something like MalwareBytes. Keeping active AV on computers is fine, but it still lets stuff slip through that dedicated ones like MalwareBytes seem to destroy with no effort.
    – ManiacZX
    Jan 22, 2010 at 20:43
  • Another option if you really want to be definitive, shut down all the workstations in your network except one you know should be clean and do your testing.
    – ManiacZX
    Jan 22, 2010 at 20:45
  • On Friday before leaving work I wrote a set of scripts that would distribute and run HijackThis on every computer in the office and save logfiles into a network folder. Unfortunately, all of our computers are laptops, so at 5pm on Friday they are all either shut down or taken home... I plan to run the script sometime tomorrow (Monday) and then analyze each logfile and see what comes up. Do you know of a better anti-virus/spyware program that can run silently and report into a logfile?
    – Ricket
    Jan 24, 2010 at 20:27
0

I bet there's a firewall that captures downloads to virus scan them before they are delivered to you. I've seen this situation where downloads will seem to stall, and then after a few minutes the entire thing will come down all at once, but what was happening was, the download was going to the AV box, virus scanned, and then (if clean) sent to the requester all at once.

Improve this answer
1
  • No, I don't think this is it. For the most part, downloads will come in at an extremely slow rate - around 10kB/s or less. But often, partway through, it slows down until it just reaches 0 and times out. I can double check but I'm pretty sure our firewall appliance does no antivirus scanning.
    – Ricket
    Jan 23, 2010 at 18:13
0

We found a hub (yes, a HUB) right between our campus connection and our firewall. It had only those two ethernet cables plugged into it (the campus connection, and the firewall), nothing else - so it was completely pointless and throttling the connection to half-duplex. After removing the hub, our speeds have jumped up to several mbps and no more dropped downloads.

However, in talking with the campus, we got them to run a gigabit line to our firewall in place of the 100mbps line, and we also upgraded all of our switches to gigabit. As of this past Friday 4/16, we are at about 65 mbps up and down (according to speedtest.net at 8am)!! Go NC State!!

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .