0

This case is a bit weird to understand, so I decided to ask here and see if someone can give me some guesses about what can be the root cause of the issue I'm about to describe.

Our company has 2 main switches, which are:

  • HPE ProCurve 1800-24G Switch (J9028B)
  • D-Link DGS-1016D

These two switches are connected to one another using one Cat6 UTP cable. They're almost at full capacity, but they work fine. There are also a few other unmanaged switches in our network (about 2 or 3) which distribute/repeat the communication to/from the 2 main switches mentioned above.

Up until 2 months ago we had a DSL connection with a crappy DSL modem from the ISP, this DSL modem was connected to one of the main switches (D-Link DGS-1016D), everything worked fine except the internet speed which was crappy by nature. About 2 months ago we got a fiber optic internet connection and got a fiber gateway from the ISP. The fiber gateway is 1Gbps and supposedly is top-notch (according to the ISP, of course). The DSL modem was removed from the network and the fiber gateway took its place, same IP and same spot in the switch.

Here comes the strange part(s)... The first issue I noticed was that when I logged into this fiber gateway, through its web interface, the web interface would become unresponsive, packet loss would start, and the internet would drop after about 2 or 3 minutes - the gateway soft rebooted itself - every time I logged in. Weird thing is that when I log into the same device without it being connected to our network's switch (using a single isolated connection laptop-gateway) then everything works fine. The home page loads fine, no packet loss, no drop in connectivity nor reboots.

After tinkering around a bit I found that, at least to some extent, the issue is caused by the gateway's home page. I'm not sure what it does, it shows a little graph of the network topology and gives you a count of devices connected to the LAN (which is off by at least 60%). The reason I came to this conclusion is because I set-up a JS script in the browser, using TamperMonkey, which redirects the browser to a different page of the gateway immediately after log in. This way the hanging was remedied permanently and I thought "well, f#ck it, everything is fine", until about a month ago when things became weirder.

Almost every day now, multiple times a day, eventually the gateway will have a packet loss ratio of 60+%. This is my big issue at the moment, I don't know to which degree the gateway is the issue, but there is some additional information that may be useful: the packet loss happens simultaneously on the LAN and WAN side, and on the WAN side the ping requests are handled "directly" by the gateway, there isn't any networking device of our own - the fiber optics cable connects directly to the gateway, which in turn connects to one of the switches.

The first few times I power-cycled the gateway, then power-cycled the switches, and then one day I noticed the strangest thing yet: when the gateway starts dropping packets, the temporary solution is as easy as unplugging for a couple seconds the cat6 UTP cable that bridges the 2 main switches and then plugging it back in. After this, the gateway will behave normally for a couple more hours, 3 tops, and then goes back into the same problematic behavior. It's as if the gateway becomes overloaded, yet a 10 year old DSL modem didn't with the same network configuration.

At this moment I have a few theories that I'll have to test:

  • Problem is (one of) our switches. The reason why I haven't tested this yet is because I haven't a switch at hand, would have to buy one. It still doesn't account for the fact that there is no issue with the old DSL modem;
  • Problem is the gateway itself; would need to buy a decent router/gateway to test this too;
  • I think I'm forgetting another theory I had...

I tried getting the ISP to swap for another gateway (different model), but they refuse given that the issue only arises when connected to our network. The gateways are not professional grade, but how bad can they be compared to the old home DSL modem?!

Well, this is the issue that I'm facing. Any thoughts? Suggestions?

Thanks in advance.

  • The boxes handed out by ISP's are sometimes very bad, especially at handling a lot of traffic or a lot of devices on the network, and contrary to what you might think, they are not getting better, so that box would be my first suspect. Have you asked your ISP if they have other customers with a similar amount of traffic/a similar number of devices on their network? To explore other possibilities: Can you configure (one of) the switches to limit the traffic between them? Can you sniff the traffic on the link between the switches, when the problem occurs? – Henrik Dec 28 '18 at 16:20
  • The ISP's tech support seem surprised when I bring the subject up, as if it doesn't happen. But it is likely that most corporate customers, at least with networks big enough, implement their own professional grade solutions which then work fine. Unfortunately only one of the switches is managed, the only thing I could try is change the port speed for the gateway and see how it behaves. As for sniffing the traffic, I guess I could mirror one of the ports in the managed switch, but I'm unsure what I would be looking for. – ner0 Dec 28 '18 at 18:12
  • 1
    I've had similar issues in my home network when I've been running too many connections while the ISP:s device has been handling NAT. So, if the ISP:s device is handling NAT for you, check if it's possible to get a professional-grade router that can do that, and set the ISPs gateway device in bridging mode? – Jenny D Dec 30 '18 at 9:38
  • Yeah, that might be the actual issue. It's one of the options I'm considering, the main one in fact, so the possibility of getting a good router is very much on the table. – ner0 Dec 30 '18 at 13:56
0

Without more insight into your network - with managed switches - this is some guesswork. You've already located the ISP gateway as a source of problem. It soft rebooting itself is some proof of that. Chances are high it might just be the only problem.

Check with the ISP if

  • there's a firmware upgrade, increasing stability
  • the memory of the router is sufficient for your network - monitor it for some time if possible
  • you can get the router exchanged, possibly even for the same type - you might have a bad device
  • there are options to get yourself a decent router and run it with the WAN connection

ISP support might know about problems and play dumb or you've got a bad device/firmware version/configuration. The point is that the ISP gateway is the responsibility of the ISP and you need it to work.

  • Yes, the gateway is the high probable culprit here. From the ISP's perspective, their responsibility is limited to getting the internet to reach the gateway, period. I've tried explaining to them that the gateway might not be capable of handling the relatively high volume of traffic (it's also our secondary DNS server, because, well... 0.0.0.0). They do not care. They also have higher-level support contracts which cost a fortune for the same or less speeds and include the option to lease professional-grade routers. Buying the router ourselves seems to be the solution at this point. – ner0 Dec 30 '18 at 13:54
  • I bridged the ISP router with a Draytek 2820 that we already owned. When the packet loss issue occurs, the Draytek is much more resilient, meaning that it still loses packets but not enough to prevent from remotely connecting to it. Rebooting the router doesn't solve anything, only disconnecting the bridge between both switches does. I am still at loss how disconnecting/reconnecting a bridge between 2 switches in a matter of a couple seconds is enough to solve the issue for a few hours. – ner0 Jan 7 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.