I am attempting to deploy the Splunk UniversalForwarder as an SCCM application using an MSI Deployment Type to a small group of testing servers and am encountering an uncharacteristically confusing issue with the MSI installer.
The Deployment Type itself is pretty straight forward. I uploaded the MSI file (splunkforwarder-6.2.1-245427-x64-release.msi) to the SCCM Site Server and let the Detection Method be auto-filled from the MSI. The only thing I did was change the install string to include the following arguments:
msiexec.exe /lv splunkinstall.log /i "splunkforwarder-6.2.1-245427-x64-release.msi" AGREETOLICENSE=YES DEPLOYMENT_SERVER="splunkd.security.contoso.com" /quiet /norestart /qn
Due to the inclusion of the
/lv switch I can go to the C:\Windows\CCMcache folder and view the installation log:
GetPreviousSettings: Error: DetermineContextForAllProducts failed witht: 0x65b. GetPreviousSettings: Error 0x80004005: Failed to GetInstalledSplunkSettings. GetPreviousSettings: Info: Leave GetPreviousSettings: 0x80004005. CustomAction GetPreviousSettings returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox) Action ended 14:20:23: GetPreviousSettings. Return value 3.
A little Google-fu and I find this posting over at Splunk's Community Support but I'm not terribly confident in correctness of the answers; however the failing installer step is
GetInstalledSplunkSettings so maybe I should go digging through the registry to see if stale Product or Installer keys are erroneously causing the installer to try to uninstall nonexistent instances of UniversalForwarder.
With a combination of ProcMon and manually searching through the registry I managed to locate the following keys that seem to be related.
HKCR:\Installer\Products\B0271F4D65C5D084FA81634DC56AD4A HKCR:\Installer\Features\B0271F4D65C5D084FA81634DC56AD4AE HKCR:\Installer\UpgradeCodes\13631B46466632F4FA2E89CF8E9602DB HKLM:\SOFTWARE\Classes\Installer\Features\B0271F4D65C5D084FA81634DC56AD4AE HKLM:\SOFTWARE\Classes\Installer\UpgradeCode\13631B46466632F4FA2E89CF8E9602DB
ProcMon Events during installation:
10:17:31.8751924 AM MsiExec.exe 7436 RegQueryKey HKLM SUCCESS Query: HandleTags, HandleTags: 0x0 10:17:31.8752103 AM MsiExec.exe 7436 RegOpenKey HKLM\Software\Microsoft\Windows\CurrentVersion\Installer\Managed\S-1-5-18\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE NAME NOT FOUND Desired Access: Read 10:17:31.8752409 AM MsiExec.exe 7436 RegQueryKey HKU SUCCESS Query: HandleTags, HandleTags: 0x0 10:17:31.8752584 AM MsiExec.exe 7436 RegOpenKey HKU\S-1-5-18\Software\Microsoft\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE REPARSE Desired Access: Read 10:17:31.8752831 AM MsiExec.exe 7436 RegOpenKey HKU\.DEFAULT\Software\Microsoft\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE NAME NOT FOUND Desired Access: Read 10:17:31.8753062 AM MsiExec.exe 7436 RegQueryKey HKLM SUCCESS Query: HandleTags, HandleTags: 0x0 10:17:31.8753230 AM MsiExec.exe 7436 RegOpenKey HKCR\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE SUCCESS Desired Access: Read 10:17:31.8753486 AM MsiExec.exe 7436 RegQueryValue HKCR\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE\ProductName SUCCESS Type: REG_SZ, Length: 38, Data: UniversalForwarder 10:17:31.8753716 AM MsiExec.exe 7436 RegCloseKey HKCR\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE SUCCESS 10:34:31.5741168 AM MsiExec.exe 7560 RegEnumKey HKCR\Installer\Products SUCCESS Index: 11, Name: B0271F4D65C5D084FA81634DC56AD4AE 10:34:31.5744153 AM MsiExec.exe 7560 RegOpenKey HKCR\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE SUCCESS Desired Access: Read 10:34:31.5744407 AM MsiExec.exe 7560 RegQueryValue HKCR\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE\ProductName SUCCESS Type: REG_SZ, Length: 38, Data: UniversalForwarder 10:34:31.5744637 AM MsiExec.exe 7560 RegCloseKey HKCR\Installer\Products\B0271F4D65C5D084FA81634DC56AD4AE SUCCESS
Now things get interesting! I remove the keys expecting success, run the Client Application Deployment and Evaluation Cycle and I get the same disappointing results. I doubled check the NTFS permissions for those keys and SYSTEM has Full Control which seems to disprove this answer indicating that the
0x80004005. Hmmmmm. What if I run the installation manually?
Same installer, same installation arguments (copy and pasted right out of AppEnforce.log), run from a Run As Administrator cmd.exe prompt and it fails with the same error at the same step. If I go back and remove the registry keys again, and re-run the installer from cmd.exe it works! LOL WUT?
- If the registry keys are present the installer fails at the
DetermineContextForAllProductsinstallation step both as run by the SCCM client and manually by me.
- If I remove the registry keys the SCCM client tries to run the installer and fails at the
- If I remove the registry keys and I manually run the same installer with the same arguments from a Run As Administrator cmd.exe prompt it SUCCEEDS!
- I get the same results on two very different servers.
Any ideas? I'm happy to provide a whole copy of my ProcMon events or any other information if that would be helpful.