SCCM 2012 SP1 - DownloadContentFiles() failed with hr=0x80041013

Question

We noticed our Automatic Deployment Rules for Software Updates failed to automatically download and apply this month's patches from Microsoft although they are correctly listed in the Catalog.

The Automatic Deployment Rules list their Last Error Code as 0X87D20417 and the Last Error Description as "Auto Deployment Rule Download failed". Re-running the rules manually reproduces this error. Deleting and recreating the Automatic Deployment Rules also reproduces the same error.

Looking at the SMS_RULE_ENGINE log shows the following errors:

Error   Milestone   004 6/19/2013 3:42:21 PM    SCCM.ad.example.com SMS_RULE_ENGINE 8706     Content download failed.   Message: Failed to download one or more content files.   Source: SMS Rule Engine.   
Error   Milestone   004 6/19/2013 3:42:07 PM    SCCM.ad.example.com SMS_RULE_ENGINE 8706     Content download failed.   Message: Failed to download one or more content files.   Source: SMS Rule Engine.   
Error   Milestone   004 6/19/2013 2:45:44 PM    SCCM.ad.example.com SMS_RULE_ENGINE 8706     Content download failed.   Message: Failed to download one or more content files.   Source: SMS Rule Engine.   
Error   Milestone   004 6/19/2013 2:43:29 PM    SCCM.ad.example.com SMS_RULE_ENGINE 8706     Content download failed.   Message: Failed to download one or more content files.   Source: SMS Rule Engine.   

If I look through the ruleengine.log (which is presumably the log file that the higher level SMS_RULE_ENGINE log within SCCM gets generated from) and coordinate the Package ID for the relevant Deployment Packages that the Automatic Deployment Rules are supposed to place these updates in I find the following:

Contents 16821586 is already present in the package "0040000F". Skipping download.  SMS_RULE_ENGINE 6/19/2013 3:41:58 PM    9068 (0x236C)
Downloading contents (count = 10) for UpdateID 16829711 SMS_RULE_ENGINE 6/19/2013 3:41:58 PM    9068 (0x236C)
List of update content(s) which match the content rule criteria = {16821659,16821660,16821661,16821662,16821663,16821664,16821665,16821666,16821667,16821668}   SMS_RULE_ENGINE 6/19/2013 3:41:58 PM    9068 (0x236C)
Downloading content with ID 16821659 in the package SMS_RULE_ENGINE 6/19/2013 3:41:58 PM    9068 (0x236C)
Failed to download the update from internet. Error = 4115   SMS_RULE_ENGINE 6/19/2013 3:41:58 PM    9068 (0x236C)
Failed to download ContentID 16821659 for UpdateID 16829711. Error code = 4115  SMS_RULE_ENGINE 6/19/2013 3:41:58 PM    9068 (0x236C)

At this point, I have three different errors that I believe are all generated by the same event. Of course, they may not be which is why they are all included here. I did coordinate the times in the log files and am reasonably confident they are all related to the issue with the Automatic Deployment Rules.

• 0X87D20417 - From the SCCM Console's Automatic Deployment Rules
• 8706 - From the SCCM's Console's Monitoring SMS_RULE_ENGINE log
• Error code = 4115 - From the SCCM site server logs from [SCCMInstallationPath]\Logs\ruleengine.log

It looks like we're not able to download those updates. Apparently the place to troubleshoot that kind of thing is the PatchDownloader.log. And 'lo there is yet another error recorded there:

Trying to connect to the \\SCCM.ad.example.com\root\sms\site_REV namespace on the SCCM.ad.example.com machine.  Software Updates Patch Downloader   6/19/2013 3:42:21 PM    9068 (0x236C)
Connected to \\SCCM.ad.example.com\root\sms\site_REV    Software Updates Patch Downloader   6/19/2013 3:42:21 PM    9068 (0x236C)
GetContentFileInfoForDownload() failed for ContentID 16821994. hRes = 0x80041013 .  Software Updates Patch Downloader   6/19/2013 3:42:21 PM    9068 (0x236C)
ERROR: DownloadContentFiles() failed with hr=0x80041013 Software Updates Patch Downloader   6/19/2013 3:42:21 PM    9068 (0x236C)

I can coordinate the Content IDs in the PatchDownloader.log back to the Error: 4115 entries recorded in the ruleengine.log so, as mentioned previously, I'm pretty sure were looking at the same event generating all of these different errors but if someone knows better please correct me.

If I use CMTrace's Error Lookup tool it tells me the following about hr=0x80041013.

Provider load failure

Source: Windows Management (WMI)
-----

And sure enough if I look at the WMI namespace the Software Updates Patch Downloader is connecting to, it doesn't look quite right:

\SCCM.ad.example.com\root\sms\site_REV

Our Site Code is actually 004 and funnily enough the first three letters of our organization start with REV. Mighty coincidental if you ask me. Furthermore, this is not the first SCCM install that existed here and it turns out the previous SCCM 2007 had the existing Boundaries, Collections and Packages migrated over to our new install. Smoking gun? Not quite. It used a different site code as well. Perhaps the REV site code was used for a temporary test installation of SCCM 2012? Perhaps not. Institutional knowledge doesn't have any record of the REV it and the migration we performed before I was hired.

However - our old PatchDownloader.log from the SCCM 2007 instance shows the Software Updates Patch Downloader connecting to the site_$SITECODE WMI namespace. Unfortunately, I don't have logs from our current 2012 install from May where I could confirm the correct WMI namespace is being referenced.

Trying to connect to the root\SMS namespace on the SCCM07.ad.example.com machine.   Software Updates Patch Downloader   8/3/2011 3:18:37 PM 25128 (0x6228)
Connected to \\SCCM07.ad.example.com\root\SMS   Software Updates Patch Downloader   8/3/2011 3:18:37 PM 25128 (0x6228)
Trying to connect to the \\SCCM07.ad.example.com\root\sms\site_DOR namespace on the  machine.   Software Updates Patch Downloader   8/3/2011 3:18:37 PM 25128 (0x6228)
Connected to \\SCCM07.ad.example.com\root\sms\site_DOR  Software Updates Patch Downloader   8/3/2011 3:18:37 PM 25128 (0x6228)
Download destination = \\SCCM07.ad.example.com\WSUSContent\be128fa4-0c6b-418a-893d-3450e38c658d.1\windows-kb890830-v3.21.exe .  Software Updates Patch Downloader   8/3/2011 3:18:37 PM 25128 (0x6228)
Contentsource = http://download.windowsupdate.com/msdownload/update/software/uprl/2011/07/windows-kb890830-v3.21_2aba440b72071ff17cad1ca2a39f0e40aa85c76e.exe . Software Updates Patch Downloader   8/3/2011 3:18:37 PM 25128 (0x6228)
Downloading content for ContentID = 31068,  FileName = windows-kb890830-v3.21.exe.  Software Updates Patch Downloader   8/3/2011 3:18:37 PM 25128 (0x6228)

OK. It really does look like an issue with WMI namespaces. Somewhere in the depths of SCCM, something is telling the Software Updates Patch Downloader to connect to \\SCCM.ad.example.com\root\sms\site_REV instead of \\SCCM.ad.example.com\root\sms\site_004.

On a WAG, I checked likely tables in SQL database for references to REV to no avail..

SELECT * FROM SysResList WHERE SiteCode = 'REV';
SELECT * FROM SiteControl WHERE SiteCode = 'REV';
SELECT * FROM SiteControlNotification WHERE SiteCode = 'REV';
SELECT * FROM Sites WHERE SiteCode = 'REV';
SELECT * FROM Sites_DATA WHERE SiteCode = 'REV';
SELECT * FROM SiteWork WHERE SiteCode = 'REV';
SELECT * FROM PkgServers WHERE sitecode = 'REV';
SELECT * FROM PkgStatus WHERE sitecode = 'REV';

To further complicate things I'm seeing multiple explanations of the 0x80041013 error.

WMI Troubleshooting Tips says that it is a failure to load a WMI provider:

WBEM_E_PROVIDER_LOAD_FAILURE - 0x80041013

The Provider Event Troubleshooting Classes are a great resource, but may be a little overwhelming. The MSFT_WmiProvider_LoadOperationFailureEvent class is one that I've found useful quite often. Most Provider Load Failures I've encountered have been the result of bad component registration (either in the registry or WMI), or permissions related.

Whereas WMI Error Constants from MSDN says that it is a permissions issue:

WBEM_E_ACCESS_DENIED 2147749891 (0x80041003) Current user does not have permission to perform the action.

The only other information I could find on the 0x80041013 error was a fellow who posted on TechNet who seems to have had the same issue as me, even down to the issue where he had a previous installation of SCCM that whose WMI namespace was being mistakenly referenced (e.g., site_REV instead of site_004). He ended up nuking the entire WMI namespace as well as the parts of the SMS_ProviderLocation. I'm not sure I want to do that.

At this point, it's been a long day, we need to get these servers patched and my head hurts. Any advice?

Answer 1

Perhaps the REV site code was used for a temporary test installation of SCCM 2012? Perhaps not. Institutional knowledge doesn't have any record of the REV it and the migration we performed before I was hired.

This hunch was correct. I got a hold of my predecessor and apparently the first and unsuccessful attempt to migrate from SCCM 2007 to SCCM 2010 used the REV site code. How it managed to sit dormant in WMI all this time and why it got "activated" is a complete mystery to me.

I very carefully re-read the solution in this TechNet post which advised deleting the old namespaces and decided to try that. I kind of hesitate to mark this as an answer even though it did resolve this issue, it indicates that I implicitly approve of it, especially since I couldn't get anyone "official" at Microsoft to confirm whether or not this was a safe approach or what the consequences were for doing this. That being said, make sure you have complete backups of your SCCM server or at least a much more intimate knowledge of WMI before proceeding. You could very easily break everything doing this, especially if like me, you're not familiar with WMI and how deeply SCCM leverages it.

I used wbemtest to connect to the root\sms namespace on our SCCM server. From there I used the [Enum_Instances...] button and search for __NAMESPACE as the superclass. I deleted the entry for the REV site code. I then did the same Enum_Instances for the SMS_ProviderLocation as the superclass and deleted the old site code from that namespace. Re-running the Automatic Deployment Rules and reviewing the PatchDownloader.log showed the successful downloading of each Windows Update.

I would greatly appreciate more information about how these namespaces are used by SCCM and exactly how this fixed the issue if anyone has more detailed information.