Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

VMWare VpxClient.exe has an issue with Data Execution Prevention (DEP), which results in quite a lot of the functionality of the client being unavailable.

Symptoms:

  • Unable to access the console
  • Error message "Unable to get the window handle for the 'AxQuickMksAxCtl' control. Windowless ActiveX controls are not supported".
  • Error message "Object not set to an instance of an object".

Turning off DEP for this application using the system control panel application does not work, it fails with the error "This program must run with data execution protection(DEP) enabled. You cannot turn off DEP for this program."

VMWare recommend turning off DEP globally using BCDEdit.exe. Clearly this is undesirable.

I have found the solution, and I am posting it here to help others.

share|improve this question

1 Answer 1

up vote 1 down vote accepted

The reason for the issue is that DEP is incompatible with some of the components. (Guessing: This is probably an ATL 7.1 activex control as there is a known issue with this version of ATL).

The reason you are unable to disable DEP for the VpxClient.exe is because the flag NXCOMPAT is set in the executable, which forces DEP on, and prevents you from opting-out the exe.

This became the default with recent versions of Visual Studio, so probably wasn't a conscious decision on the part of VMWare.

The solution is to use EDITBIN.exe to modify the VpxClient.exe binary removing that flag.

Essentially this, in an administrative CMD session:

cd /d "C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher"
copy VpxClient.exe VpxClient.Original.exe
"C:\Program Files\Microsoft Visual Studio 10.0\vc\bin\vcvars32.bat"
EDITBIN.EXE /NXCOMPAT:NO VpxClient.exe

(Obviously you will have to change this for e.g. a 64-bit system, or if your tools are in a different location).

It will now work, but give a warning as the digital signature is no longer valid.

Now all we need is VMWare to do this before they sign the binary.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.