Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

[edit Nov.16th 2009: Thank you for the responses. I'm no longer on this project so the problem is out of my hands. Be sure I will return with other problems :-) ]

I have a strange occurence of regular databursts over my WAN links from 2 sites to a site with vCenter.

What happens is that a vCenter server is managing local ESX 3.5 servers and 2 from 2 sites across a WAN link. Each server sends an approximately 3MB worth of TLS data (less than 10% of the time it varies to higher or lower) every 15 minutes (with a margin of 2 minutes). So far, I've not been able to single out a process that causes it. I looked through all applications on each site. So far, it seems to originate from one server on each site.

Although it may be coincidence and therefore not relevant, I found that one server, with very few exceptions, does a burst at 00:00 on the hour. The other 3 during the hour are a bit off the 15 minute mark but back at the top of the hour, you can sync your watch on it. The other server follows 5 minutes after that with no such precision. But, as said, it never differs more than 2 minutes.

Servers are ESX 3.5, vCenter is 2.5.

share|improve this question

3 Answers 3

up vote 2 down vote accepted

HI, I'm almost certain that this is performance counter data, VC requests a block of performance metrics from each ESX server periodically. Try changing the class of captured metrics in VC to a lower or higher value (i.e. more or less data captured) to see if the data sizes correspondingly go up or down, if they do then you simply need to tune the collection sizes and times.

share|improve this answer

I'd expect that to be management traffic; vCenter Server need to confirm hosts are alive and send them tasks; the hosts need to confirm their licenses are valid and obtain tasks from vCenter. Each node also needs to communicate with the others in it's cluster to confirm which nodes are alive.

You might be able to track down what the traffic is by looking at the port numbers, and tracing that back to an owning process (Process Explorer and Wireshark on the vCenter server would probably help here).

share|improve this answer
    
I have used Process Explorer and Wireshark. A network specialist used Orion NetFlow traffic analyzer which reports traffic going from 443 to 'random high port'. So far we indeed suspect the type of traffic you mention. It seems a bit high however and on occasion, specifically last week, it used all bandwidth, making the site unreachable for 10 to 30 seconds. One site has an ESX server with 3 VMs and the other the same with 2 VMs. All VMs are W2K3 R2. I'm about to leave right now, so looking up the Wireshark dumps will have to wait until tomorrow. In any casy thanks for the pointers. –  Erwin Blonk Jun 30 '09 at 14:33

The general purpose management traffic (heartbeats, performance counters for the hosts etc) should be very low - a few kbits/sec per host at most. A lot depends on how you've structured your hosts within vCenter but even so this does not strike me as normal. Have you got any third party tools installed? Is there any other reason why the hosts or any one of the Guests within the hosts would be talking back to vCenter. Anything like the VMware Infrastructure Management Assistant appliance, or anyone using the PErl or Powershell VMware CLI tools within those guests?

The fact that the bulk of the traffic is over port 443 indicates to me that something odd is going on - VC (and the various CLI tools) use port 443 to handle the initial SSL initiation but then switches to port 902 for the real traffic.

share|improve this answer
    
(apologies for letting the question rest for so long - as it is, it might take a few weeks before I can really get back to it, despite it being a problem) From the reports I get no traffic over 902 at all. I look at it again with that in mind. –  Erwin Blonk Aug 5 '09 at 13:14

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.