I have a VMware ESXi server which has been up and running for nearly 200 days. The past few days, when I try to connect to it using the VMware vSphere Client, I am unable to connect. After entering my username and password, I see the little spinning wheel and the status line say "Connecting..." and then "Loading Inventory...", and then I receive an error message:

Error Message Screenshot

The server 'my.host.name' could not interpret the client's request. (The remote server returned an error: (503) Server Unavailable

Call "ServiceInstance.RetrieveContent" for object "ServiceInstance" on Server "my.host.name" failed.

I am able to SSH in to the VMware ESXi server. All virtual machines seem to be up and running fine, so I want to know beforehand if I need to bring them down for maintenance!!! If methods proposed in your answer would interfere with running VMs please clearly state that so I know to prepare for downtime. Thanks!

How can I resolve this error condition on VMware ESXi?

(I'd post the version number but I don't know how to get it without the vSphere console!)

EDIT: Roughly a month after I asked this question, the server inexplicably rebooted itself. I don't know if it panicked or what happened... but after it rebooted, this problem was gone. So I can't test / confirm any of the answer unless the problem reappears (which I hope it does not!)

You need to restart the vmware management services. Luckily, this is easy (since you have SSH access) and non-impactful to the virtual machines.

In a nutshell, SSH to the esx frame as root and then run one of the two following commands (depending on if it is esx/i):

For ESX:

service mgmt-vmware restart

For ESXi:

/sbin/services.sh restart

See this article for full details

  • Thanks. This probably would have solved the problem! Sadly the server crashed a couple weeks ago and the problem is gone now. – Josh Nov 3 '11 at 12:21

I Solved the issue by uninstalling widecap ServiceInstance.RetrieveContent Error

This VMware KB article seems to match your described symptom. Verify that your DNS is up and running from the ESXi server's point of view.

Option 2: Can you verify, that your vCenter Server is up, and the service is started?

  • Well, it sure seems to match except that it says "it is not possible to connect to the ESX host using vCenter Server [...] for 30-40 minutes" and it's been working fine in my case for ~200 days... Also, DNS is working... – Josh Sep 25 '11 at 16:23
  • You wrote that you could ssh into the server. Did you verify that DNS is working on the server? In this communities.vmware.com/message/1316434 thread a guy had identical problem as you describe. And it was DNS problem :). – Paweł Brodacki Sep 25 '11 at 16:49
  • When I ssh in, I see /etc/resolv.conf does contain a nameserver line, and I can ping both google.com as well as the hostname of my server, so I'm reasonably sure DNS is workig. I don't have a vCenter server, only a single ESXi server. – Josh Sep 26 '11 at 12:36
  • If you can ping them by name, then DNS does seem to work. In that case, there may be nobody listening... Take a look at kb.vmware.com/selfservice/microsites/… and see (netstat -in) if anything is listening on the ports, the client connects to. You may need to restart management service. – Paweł Brodacki Sep 26 '11 at 12:58

We experienced a similar issue as this and it ended up being failed SAN LUNs that were directly attached via Fiber Channel HBAs. apparently one of the two filers had a failover event but didn't fail over cleanly therefore the ESXi host could not declare those paths dead and had an influx of LUN level block issues with HBA busy, bus busy, abort commands littered in the vmkernel.log.

VMware support was able to walk us through the issues after the SAN file head clusters were restored to an active/active state (NetApp). The "cat /var/log/vmkernel.log | grep sense | less" hex errors showed numerous LUN level issues (D:0x2), bus busy (H:0x2), HBA busy (D:0x8), abort commands (H:0x5) from timeouts which points to a SAN filer not being properly failed over and still reporting itself as available

After the SAN filer restoral for the paths/LUNs we issued the command " /sbin/services.sh restart" which completed & we were able to vClient into the Host again, the web & join it back to the existing cluster to clear the "orphane" "unnamed" VMs that were residual leftovers.

On my vCenter 6.5 appliance, the vpxd service core dumps and procudes this error.

Only workaround/solution so far: block access to ESX host until all vCenter services are started.

Now, a shell script/systemd unit on vCenter creates iptables firewall/packet filter rules on boot. Once the vCenter services are started and load average drops below 0.5, the script deletes the iptables rules. Only now vCenter is able to "see" the ESX hosts and is happy for a while. If the issue reappears, I restart vCenter.

The shell script:

#!/bin/bash
# /usr/local/bin/block-esx-access-on-boot.sh

export ESX_HOSTS="ESX1-IP,ESX2-DNS,ESX3-IP"
export LOAD_THRESHOLD="0.5"

sleep 5

LOAD="$(cut -d' ' -f1 /proc/loadavg)"

echo "Waiting for 1min loadavg ${LOAD} > ${LOAD_THRESHOLD} ..."
while [ "$(echo "${LOAD} > ${LOAD_THRESHOLD}" | bc)" == "0" ] ; do
  echo "Waiting for 1min loadavg ${LOAD} > ${LOAD_THRESHOLD} ..."
  sleep 3
  LOAD="$(cut -d' ' -f1 /proc/loadavg)"
done

echo "Blocking outgoing transfers to ${ESX_HOSTS}"
iptables -A OUTPUT -d ${ESX_HOSTS} -j DROP
iptables -L OUTPUT

while [ "$(echo "${LOAD} < ${LOAD_THRESHOLD}" | bc)" == "0" ] ; do
  echo "Waiting for 1min loadavg ${LOAD} < ${LOAD_THRESHOLD} ..."
  sleep 60
  LOAD="$(cut -d' ' -f1 /proc/loadavg)"
done

echo "Allowing outgoing transfers to ${ESX_HOSTS}"
iptables -D OUTPUT -d ${ESX_HOSTS} -j DROP
iptables -L OUTPUT

The systemd unit:

# /etc/systemd/system/block-esx-access-on-boot.service

[Unit]
Description=Block ESX Access on Boot
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/block-esx-access-on-boot.sh

[Install]
WantedBy=multi-user.target

https://gist.github.com/quatauta/a1ac390633006996fbc547da9bd01ef9

I received this error right after my successful upgrade of vcenter 5.0 to 5.1. I noticed several alerts (under EVENTS (tasks & events)) within vcenter from service accounts that I had setup in the past for various items (kaspersky vsheild and orion syslog svc accounts) that were showing access denied. I added these accounts to the local administrators group on vcenter and my problems went away.

Before I discovered this however to provide myself a workaround I would simply restart vmware server service and then I could log in no problem and access vm consoles. After 5 minutes or so the consoles would go black and I could no longer access them. If I logged out and tried to go back into vcenter I would get this error:

Call "ServiceInstance.RetrieveContent" for object "ServiceInstance" on Server "my.host.name" failed.

So the rule is if you can get into vcenter go check the EVENT logs and see if there are any access denied alerts. If there is add these accounts to the local admin group on vCenter.

We ran into the same issue. VMWare Support states that vCenter is out of sync with Single Sign On (SSO). Simple reboots of the SSO server while vCenter server is powered off should resolve the issue:

Here’s the sequence:

  1. power down the vCenter server.

  2. then reboot the SSO box and wait for all the VMWare services to come back up on this box

  3. power up the vCenter server

  4. restarted vcenter server services in proper sequence (directory, kdc, certificate service, idm, sts, inv service and then vc service

I got this after changing the vCenter's IP address

  1. I used #6 https://rlevchenko.com/2016/03/24/vcenter-503-service-unavailable/ to enable the shell.

  2. ssh into the vcenter server

  3. Command> shell

  4. vi /etc/hosts changed the IP there

  5. service-control --stop --all

  6. service-control --start --all

rebooting our vCenter server helped solved this issue for us.

we could not vMotion or build off templates without getting the 503 error. I've also seen this in the past where rebooting the vCenter doesn't help and we needed to reboot a host. which means the VMs on that host went down too.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.