I have a Windows application which creates up to 35 processes and it's working OK when it's running from cmd. But when it is executed as a service on the same machine it is able to create only 20 processes and all other are killed because of some kind of resource exhaustion problem.
The problem is persistent on one Windows 2003 server but not reproducible on other servers.
Can it be because the system has run out of desktop heap? http://support.microsoft.com/kb/184802
How can I check it?
Yes, it's possible that's what happening. You can monitor the Desktop Heap with the (wait for it....) Desktop Heap Monitor! You can get it here, or it's the third hit on Google for "desktop heap". Other hits on that search phrase give a lot of detail on diagnosing and fixing the issue, so do some reading, load that tool, and you're off to the races! Give it a few shots and then post back here letting us know if that helps you.
Is it an application you created? If so you could probably have it log information about available memory and resources to a file to help narrow down what is happening. Is anything showing up in the logs for Windows? What account is the service running under? If you try running the service as your user that you're testing under does the same thing happen? Different environment variables, or a different security context?
What is the process doing? It could be something with an odd security setting. There may be a tool with sysinternals that can help you debug the issue, like using procmon. That may show where the file or registry setting is getting a denied or fail error, depending on what the process is doing. Even if procmon couldn't help there may be other tools in the suite that will. Google for sysinternals (free from Microsoft) and see what their stuff can tell you.
