I am having a similar issue, and ran across this KB article yesterday - http://support.microsoft.com/kb/304101/en-us.
The article discusses fixing an issue when trying to backup large volumes; but when you read the details, it seems like it could be related (Insufficient system resources exist to complete the reqeusted service).
The article describes a way to check your system is affected by the issue, and some registry changes that may correct the behavior
I have not implemented this yet. The issue occurs only on a production system (the dev is unaffected). We are still assessing.
Might be helpful to you...