Stay a while and listen. This is going to be a long one. We've been having a problem the last couple of months with Updates on very specific Microsoft Server 2003 R2 Machines. This problem has confounded me and our resident server administrator to no end as we can not make heads or tails of what exactly is causing the problem.
Here are the details:
Our Network: This network is a medium/small sized organization with approximately (500) computers and 20+ servers. The primary DC and and critical services are all on Server 2012 (r1) while some of the older application/services run on 2008 R2. Additionally, there are a smattering of Server 2003 R2 machines running mostly legacy applications and services which will either be replaced or become obsolete next year.
The issue: The issue is that among these few 2003 R2 servers, three of them are exhibiting strange behavior. We were first alerted to this behavior when they suddenly (in the same Update cycle) failed to properly install updates and we were alerted through our monitoring software.
Windows Update: On the problem machines, we are able to connect to the Windows Update URL just fine, but any attempt to actually check for updates will result in a
Error number: 0x80244016
in the Windows Update page. This error code corresponds to an invalid URL. As we are going through WSUS the URL should be and (we have checked on all the servers) to the FQDN and port of our WSUS server \NOC3.sub.domain.com:8888 . We found that the traffic was getting blocked at this point through a winHTTP proxy setting. So, when this problem occurs, we can use
and clear the proxy settings. Run windows updates and it will successfully connect.
This is where it gets interesting: Now, this would be all good and fine, but it gets better. So the next windows update comes around last month, and lo-and-behold, the exact same issue with the exact same servers occurs again. Now, we are a little suspect about why the same proxy settings have reapplied themselves. Naturally, we believed it may have been an old policy that was not properly disabled or possibly just incorrectly applied. So we both scoured all applicable GPO's for the servers in GPM on the Domain controller looking for any applied policies that may have anything to do with this strange, now reoccurring, policy(?) that is causing our servers to somehow revert to this (wrong) proxy server setting. There were no policies applied that have any mention of proxy servers. Then, I ran a RSOP on the machine.
It gets stranger: So after running the RSOP, everything seemed to have loaded fine. Until I attempted to open the Local Machine --> Administrative Templates folder. First of all, it took an inordinately long time to open the folder. Then when it finally did run, error messages associated with AER_####.adm would appear. Not only errors, but you would have policy settings in multiple languages displayed in the same folder. Now, I do not read Arabic, Chinese, Italian, or Japanese very well, but I am going to assume that these are duplicates of the policy setting in different languages. I found that by clearing the .adm files from the %root%\system\inf folder would, at the very least, allow for the RSOP to load the Administrative templates without error. *Note: later research showed that the AER####.adm files are associated with error reporting; which is strange when you consider what happens next:
First attempt at resolution: So, on a hunch, I moved the extraneous .adm files to another folder; then I ran a
on the affected server. *note: this server still had the 'stuck' proxy server setting in place. After a restart, the proxy settings seemed to have stayed with the no proxy setting which we want.
But alas, the joy was temporary: During this recent update cycle, we were alerted that the servers were once again not getting updates. The same problem, the same errant proxy server setting. *Note: the proxy server setting is pointed to an internal server which we once used as a web proxy so we are not thinking "security risk" atm. I also checked the RSOP once again, and the extraneous .adm files were back in place. So at this point, I am convinced that the problem lies on whatever is modifying our policy such that these new AER####.adm files are being generated (a policy or some external device/program which is making changes to our policy). I have found no references to anything like this anywhere on the web, and I do not understand the internal mechanics of Group Policy as they are applied to server 2003 R2 os to delve any deeper at this point. So I ask you, help!
Conclusion: So here is what I know for sure:
- The problem is only on 2003 R2 Servers
- The problem does not affect ALL 2003 R2 Servers
- There are no policies that are applied (or even active) that have any reference to this current proxy setting which seems to reapply itself at irregular intervals
- Updates will work fine after manually disabling proxy setting on the local servers
- Once again, it will not stay disabled
Any advice or insight into this issue would be great. Thanks for your time!