Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a HP ProCurve V1810-48G switch with a few servers running Citrix XenServer connected to it (everything in one rack). The switch is almost in its default configuration (no VLAN, no port mirroring/monitoring, no other routers connected other than the gateway to the internet).

During copying of a few hundred GByte of data from server27 to a NFS-mounted directory on server18 I noticed network related error messages for other servers in the same rack, as if they were no longer able to send/receive traffic to each other or their users, such as error messages from external web monitoring services that a particular website is no longer reachable.

After cancelling the copy command everything was normal again.

Note that all of the mentioned servers are connected to the same switch and are located in the same IP network. I always thought that a connection between two servers on one switch will not affect any other server connected to the switch.

I have then hooked up the switch to a zabbix monitoring server. Here is the screenshot: switch network traffic diagram You can see here that the outgoing traffic from server27 (bottom right) to server18 (second row left) seems to affect every single server in the rack. I have also suspended the copy process once, and you can see the drop in network traffic for everybody else.

Also you can see gaps in the diagrams where the zabbix server (server21) was unable to connect to the switch.

Checking the network traffic on the server side (instead of the switch side) showed that there is only the normal traffic, not the huge volume shown on the diagrams above.

Some commenters have pointed out that traffic between two ports in a switch should not affect any other port. This diagram however suggests that there is a subtile problem somewhere. The traffic of just 20 MByte/s affects the connectivity to all other systems.

share|improve this question
    
Can you expand on what the physical network layout is like? The servers in the other data center are directly connected to this same switch? –  Shane Madden Sep 2 '12 at 21:23
    
The switch and the servers are all in the same rack. –  nn4l Sep 2 '12 at 21:30
    
Posting the "network related error messages" would be helpful. –  joeqwerty Sep 2 '12 at 21:52
    
My first thought was that the backplane doesn't have enough switching bandwidth to provide access to all ports at full duplex, but that switch has a 104 Gbps backplane, which enough for all 48Gbps ports plus the 4 SFP's –  Mark Henderson Sep 2 '12 at 22:48
    
It certainly looks like the traffic from server27 is being broadcast. Have you inadvertently configured a strange netmask/broadcast address on one of your servers nics? Can you run a tcpdump on one or two of the servers that are not the intended targets of the traffic and see the type of packets they are receiving? –  mtm Jan 29 '13 at 18:23

3 Answers 3

It's not too far from "have you tried turning it on and off", but have you updated the firmware? If you look at the release notes, there are a few ARP-related fixes.

https://h10145.www1.hp.com/downloads/SoftwareReleases.aspx?ProductNumber=J9660A

As far as gathering more information so people can help troubleshoot, do you have logs from the switch itself when this is happening?

Can you share what configuration changes, aside from management information, have been made from the default state?

Are either of the hosts in question running Xen server? Do you see the problem between any other hosts (now that you've got historical graphing, you should be able to check if this happens elsewhere)?

share|improve this answer

we need to clarify the situation to help give you the best answer.

Can you confirm that this is all the same rack or that the switch definitely has default configuration, such as port configuration, vlan, gateway.

i.e. DC1
rack 1:
HP Switch
server_A
Server_B

DC2
rack 2:
HP Switch
server_C
Server_D

Otherwise,sounds like from the explanation you have given that server_C and Server_D are at different data center but are still in the same network segment In order to communicate. If this is the case then you are using the max local resources to transfer the data and that would cause your issues, if the servers_C and _D use DC2 rack2 HP switch as the backbone.

Questions for you: Do you work at the data center or sys admin to the servers? Can you provide the the network segment? server_A and B on network x.x.x.x etc How is _C and _D connected to _A and _B?

Please be more specific on location of server_C and Server_D. I.E. they are located in buidling x or rack x

Thanks.

share|improve this answer
    
all servers are in the same rack. I have updated the article with more information. –  nn4l Jan 23 '13 at 16:24

Under normal situations, no, Traffic from Server A to B will not affect C to D. Also, a unidirectional flow from A to B is only utilizing the full bandwidth in one direction, Gigabit Ethernet is bi-directional, so you're really only using half of those ports' bandwidth.

The only possible problem from the Switch's "end" of the deal would be a configuration issue. It could be port mirroring/monitoring (where is sends all the traffic from one or more other ports to a single "monitor" port). Or there could be some usage policy (I don't think the 1800 series support bandwidth policies or similar though). You could also have vLANs setup, with a bottleneck router somewhere (just being plugged into the same switch doesn't necessarily make them all in the same broadcast domain, even if they are you could have multiple subnets in one broadcast domain, forcing the use of a router again).

share|improve this answer
    
The switch is almost at its factory settings. Please see additional information that I added just after your comment. –  nn4l Jan 23 '13 at 16:06
    
Wow, something is really screwed up there... Check the MAC addresses of all the servers (make sure they're all different), the ARP table of the Server 18 and 27, are you able to get more than ~20MBps out of it? Maybe check on the switch too make sure it has picked up all the MACs. –  Chris S Jan 23 '13 at 16:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.