Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Server does not respond to ping anymore. In vsphere client, when reconnected after the outage, there is a hole in network-stats - everything else looks normal. (no high load)
After 1 to 3 minutes it's on again....

I connected to ESX-SHELL and checked the hostd.log on the outage-time: ( 16:00 it was down )

    /var/log # cat hostd.log | grep 03T15:58
2013-10-03T15:58:53.361Z [33E87B90 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0x5c33c50] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:ha-property-collector]
2013-10-03T15:58:53.361Z [33E87B90 verbose 'Vmomi'] Arg version:
2013-10-03T15:58:53.361Z [33E87B90 info 'Vmomi'] Throw vmodl.fault.RequestCanceled
2013-10-03T15:58:53.361Z [33E87B90 info 'Vmomi'] Result:
2013-10-03T15:58:53.362Z [34101B90 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(error:Transport endpoint is not connected) with error N7Vmacore15SystemExceptionE(Connection reset by peer)

    /var/log # cat hostd.log | grep 03T15:59
2013-10-03T15:59:05.437Z [FFE16D20 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2013-10-03T15:59:08.051Z [FFE16D20 info 'Vmomi' opID=5E00A82D-00001173] Activation [N5Vmomi10ActivationE:0x33a6b980] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:ha-property-collector]
2013-10-03T15:59:08.051Z [FFE16D20 verbose 'Vmomi' opID=5E00A82D-00001173] Arg version:
2013-10-03T15:59:08.051Z [FFE16D20 info 'Vmomi' opID=5E00A82D-00001173] Throw vmodl.fault.RequestCanceled
2013-10-03T15:59:08.051Z [FFE16D20 info 'Vmomi' opID=5E00A82D-00001173] Result:
2013-10-03T15:59:08.052Z [33540B90 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(error:Transport endpoint is not connected) with error N7Vmacore15SystemExceptionE(Connection reset by peer)
2013-10-03T15:59:23.869Z [34080B90 info 'Vmomi' opID=A660F300-000009D8] Activation [N5Vmomi10ActivationE:0x33aa77f8] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:ha-property-collector]
2013-10-03T15:59:23.870Z [34080B90 verbose 'Vmomi' opID=A660F300-000009D8] Arg version:
2013-10-03T15:59:23.870Z [34080B90 info 'Vmomi' opID=A660F300-000009D8] Throw vmodl.fault.RequestCanceled
2013-10-03T15:59:23.870Z [34080B90 info 'Vmomi' opID=A660F300-000009D8] Result:
2013-10-03T15:59:23.870Z [FFE16D20 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(error:Transport endpoint is not connected) with error N7Vmacore15SystemExceptionE(Connection reset by peer)


/var/log # cat hostd.log | grep 03T16:00
2013-10-03T16:00:01.371Z [33540B90 verbose 'ResourcePool ha-root-pool'] Root pool capacity changed from 10948MHz/29172MB to 10948MHz/29171MB
2013-10-03T16:00:01.555Z [33581B90 verbose 'SoapAdapter'] Responded to service state request
2013-10-03T16:00:36.595Z [33B80B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root

...these errors repeat through the log, but it does not come to an outage there ....

then later when it just got on again it does this:

    2013-10-03T16:03:34.084Z [335C2B90 verbose 'ha-license-manager' opID=A660F300-000009FE] Load: Loading existing file: /etc/vmware/license.cfg
2013-10-03T16:03:34.099Z [335C2B90 verbose 'Default' opID=A660F300-000009FE] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)
2013-10-03T16:03:34.105Z [335C2B90 error 'Default' opID=A660F300-000009FE] Unable to parse MaxRam value:
2013-10-03T16:03:34.105Z [335C2B90 error 'Default' opID=A660F300-000009FE] Unable to parse MaxRamPerCpu value:
2013-10-03T16:03:34.106Z [335C2B90 error 'Default' opID=A660F300-000009FE] Unable to parse MinRamPerCpu value:
2013-10-03T16:03:34.173Z [33581B90 verbose 'ha-license-manager' opID=A660F300-000009FF] Load: Loading existing file: /etc/vmware/license.cfg
2013-10-03T16:03:34.186Z [33581B90 verbose 'Default' opID=A660F300-000009FF] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)
2013-10-03T16:03:34.191Z [33581B90 error 'Default' opID=A660F300-000009FF] Unable to parse MaxRam value:
2013-10-03T16:03:34.191Z [33581B90 error 'Default' opID=A660F300-000009FF] Unable to parse MaxRamPerCpu value:
2013-10-03T16:03:34.191Z [33581B90 error 'Default' opID=A660F300-000009FF] Unable to parse MinRamPerCpu value:
2013-10-03T16:03:38.895Z [34101B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2013-10-03T16:03:53.865Z [33581B90 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0x32fcfab8] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:ha-property-collector]
2013-10-03T16:03:53.865Z [33581B90 verbose 'Vmomi'] Arg version:
2013-10-03T16:03:53.865Z [33581B90 info 'Vmomi'] Throw vmodl.fault.RequestCanceled
2013-10-03T16:03:53.865Z [33581B90 info 'Vmomi'] Result:
2013-10-03T16:03:53.866Z [330C2B90 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(error:Transport endpoint is not connected) with error N7Vmacore15SystemExceptionE(Connection reset by peer)

..you see, checking license / default multiple times in a second... does this look like some kind of restart ?

Update: what I've seen in vmkwarning.log

  /var/log # cat vmkwarning.log  | grep 03T15
2013-09-03T15:58:31.990Z cpu1:107391)WARNING: LinScsi: SCSILinuxQueueCommand:1193:queuecommand failed with status = 0x1056 Unknown status vmhba0:0:0:0 (driver name: ahci) - Message repeated 12 times
/

this alos happens frequently through the log, but can it be the cause ?

also in syslog.log there is another error repeating randomly: ( last before outage )

2013-10-03T15:33:16Z smartd: [warn] t10.ATA_____ST33000650NS________________________________________Z297GT49: above TEMPERATURE threshold (34 > 0)
2013-10-03T15:33:17Z smartd: [warn] t10.ATA_____ST33000650NS________________________________________Z297C3CE: above TEMPERATURE threshold (34 > 0)

right before the outage is also this in vmkernel.log:

2013-09-03T15:58:31.990Z cpu1:107391)WARNING: LinScsi: SCSILinuxQueueCommand:1193:queuecommand failed with status = 0x1056 Unknown status vmhba0:0:0:0 (driver name: ahci) - Message repeated 12 times
2013-09-03T15:58:31.990Z cpu1:107391)ScsiDeviceIO: 2303: Cmd(0x4124007bcfc0) 0x28, CmdSN 0x800e0067 from world 107389 to dev "t10.ATA_____ST33000650NS________________________________________Z297C3CE" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0$

thank you for any hints on this ....

its: VMware ESXi 5.1.0,799733

I noticed that there is an update available - but to get it installed I would need some downtime which I'm trying to avoid if possible...

share|improve this question
add comment

2 Answers

Turned out that a switch was drdos'ed - was digging in the wrong direction.

The errors I reported also happened before this. I will update to latest version. Maybe the errors will be reduced then.

Maybe a moderator could remove the whole thing, to not confuse others.

share|improve this answer
    
This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. –  slm Nov 21 '13 at 13:33
    
This answer is from the question asker, @slm –  Dan Nov 21 '13 at 15:30
    
@Dan - realized that, stil doesn't answer the Q. Should've been either a flag raised to a mod or a comment directed to a mod on the Q. –  slm Nov 21 '13 at 15:49
add comment

There are a lot of potential issues here... But we're missing the details of the hardware configuration (server make/model, NIC configuration, storage, etc).

  • Is this a recent problem?
  • Was the system previously stable?

General advice: Since this is an old revision of VMware ESXi, you should update the system to the current patch level. If this is a standalone system and not attached to a vSphere cluster (with vCenter), you will experience some level of downtime in order to apply updates.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.