Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

VM migration from one ESXi host to another is causing a strange graph signature that is very reproducible:

Strange network behavior

I have no idea why this is happening. The ESXi server is a Dell PowerEdge 2850 using an Intel PRO/1000 MT for its vKernel port. The switch in question is a Dell PowerConnect 6248. The issue occurs on all of our 2850s. I have not made any changes to the more esoteric ESXi configuration options. The same file transfer between two 2950s running ESXi 5 on the same switch will result in a nice consistent graph that averages around 70MB/s.

Any insight or suggestions are greatly appreciated, even if the insight is "PE2850s are end-of-life and ill-suited for virtualization." Not averse to buying network hardware. Thanks!

share|improve this question
1  
Have you looked at the IO on the volumes to see if there's a bottleneck there? That graph looks kinda like there's a buffer getting filled/emptied/refilled during the transfer. Also; 2850s are old! :) –  Nathan V Nov 27 '12 at 10:17

2 Answers 2

up vote 1 down vote accepted

I agree with Nathan that this looks like a buffer filling on the receiver. You could confirm this with a packet capture, look for the receiver returning "TCP Zero Window". This is the receiver saying "stop, you're sending me too much!"

Another possible cause could be TCP Segmentation offload (TSO) on the sender. These old e1000 cards only had 64k buffers to process offloading. If you're doing a big TCP transfer and overwhelming this buffer, you'll see "TX Unit Hang" messages logged on the transmitting system. When the buffer overruns, the transmit unit in the NIC resets, which results in momentary stopping of traffic before transfer resumes. Disable TSO to work around this, solve it by getting better NICs.

share|improve this answer
    
You're probably right, but I'll never find out as this was a few jobs ago now. –  Joel E Salas May 18 '13 at 18:40
    
Oh well, hopefully it'll help someone else out via search results. Thanks for the upvote and answer. –  suprjami May 18 '13 at 23:09

I never resolved this question but I was able to reproduce this graph signature on a few other hosts running ESXi 4.X. The solution was invariably to upgrade to ESXi 5.0+.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.