4

I have a HP ProCurve V1810-48G switch with a few servers running Citrix XenServer connected to it (everything in one rack). The switch is almost in its default configuration (no VLAN, no port mirroring/monitoring, no other routers connected other than the gateway to the internet).

During copying of a few hundred GByte of data from server27 to a NFS-mounted directory on server18 I noticed network related error messages for other servers in the same rack, as if they were no longer able to send/receive traffic to each other or their users, such as error messages from external web monitoring services that a particular website is no longer reachable.

After cancelling the copy command everything was normal again.

Note that all of the mentioned servers are connected to the same switch and are located in the same IP network. I always thought that a connection between two servers on one switch will not affect any other server connected to the switch.

I have then hooked up the switch to a zabbix monitoring server. Here is the screenshot: switch network traffic diagram You can see here that the outgoing traffic from server27 (bottom right) to server18 (second row left) seems to affect every single server in the rack. I have also suspended the copy process once, and you can see the drop in network traffic for everybody else.

Also you can see gaps in the diagrams where the zabbix server (server21) was unable to connect to the switch.

Checking the network traffic on the server side (instead of the switch side) showed that there is only the normal traffic, not the huge volume shown on the diagrams above.

Some commenters have pointed out that traffic between two ports in a switch should not affect any other port. This diagram however suggests that there is a subtile problem somewhere. The traffic of just 20 MByte/s affects the connectivity to all other systems.

6
  • Can you expand on what the physical network layout is like? The servers in the other data center are directly connected to this same switch? Sep 2, 2012 at 21:23
  • The switch and the servers are all in the same rack.
    – nn4l
    Sep 2, 2012 at 21:30
  • Posting the "network related error messages" would be helpful.
    – joeqwerty
    Sep 2, 2012 at 21:52
  • My first thought was that the backplane doesn't have enough switching bandwidth to provide access to all ports at full duplex, but that switch has a 104 Gbps backplane, which enough for all 48Gbps ports plus the 4 SFP's Sep 2, 2012 at 22:48
  • It certainly looks like the traffic from server27 is being broadcast. Have you inadvertently configured a strange netmask/broadcast address on one of your servers nics? Can you run a tcpdump on one or two of the servers that are not the intended targets of the traffic and see the type of packets they are receiving?
    – Matt
    Jan 29, 2013 at 18:23

3 Answers 3

1

It's not too far from "have you tried turning it on and off", but have you updated the firmware? If you look at the release notes, there are a few ARP-related fixes.

https://h10145.www1.hp.com/downloads/SoftwareReleases.aspx?ProductNumber=J9660A

As far as gathering more information so people can help troubleshoot, do you have logs from the switch itself when this is happening?

Can you share what configuration changes, aside from management information, have been made from the default state?

Are either of the hosts in question running Xen server? Do you see the problem between any other hosts (now that you've got historical graphing, you should be able to check if this happens elsewhere)?

0

we need to clarify the situation to help give you the best answer.

Can you confirm that this is all the same rack or that the switch definitely has default configuration, such as port configuration, vlan, gateway.

i.e. DC1
rack 1:
HP Switch
server_A
Server_B

DC2
rack 2:
HP Switch
server_C
Server_D

Otherwise,sounds like from the explanation you have given that server_C and Server_D are at different data center but are still in the same network segment In order to communicate. If this is the case then you are using the max local resources to transfer the data and that would cause your issues, if the servers_C and _D use DC2 rack2 HP switch as the backbone.

Questions for you: Do you work at the data center or sys admin to the servers? Can you provide the the network segment? server_A and B on network x.x.x.x etc How is _C and _D connected to _A and _B?

Please be more specific on location of server_C and Server_D. I.E. they are located in buidling x or rack x

Thanks.

1
  • all servers are in the same rack. I have updated the article with more information.
    – nn4l
    Jan 23, 2013 at 16:24
0

Under normal situations, no, Traffic from Server A to B will not affect C to D. Also, a unidirectional flow from A to B is only utilizing the full bandwidth in one direction, Gigabit Ethernet is bi-directional, so you're really only using half of those ports' bandwidth.

The only possible problem from the Switch's "end" of the deal would be a configuration issue. It could be port mirroring/monitoring (where is sends all the traffic from one or more other ports to a single "monitor" port). Or there could be some usage policy (I don't think the 1800 series support bandwidth policies or similar though). You could also have vLANs setup, with a bottleneck router somewhere (just being plugged into the same switch doesn't necessarily make them all in the same broadcast domain, even if they are you could have multiple subnets in one broadcast domain, forcing the use of a router again).

2
  • The switch is almost at its factory settings. Please see additional information that I added just after your comment.
    – nn4l
    Jan 23, 2013 at 16:06
  • Wow, something is really screwed up there... Check the MAC addresses of all the servers (make sure they're all different), the ARP table of the Server 18 and 27, are you able to get more than ~20MBps out of it? Maybe check on the switch too make sure it has picked up all the MACs.
    – Chris S
    Jan 23, 2013 at 16:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .