Whilst Joel's answer leads me to it, I did not solve the problem with the above. I find that changing the application pool to the specific user did not help. There are a few more related areas to check.
The FastCGI.Impersonation value in php.ini must be set to 1. This can be verified in the task manager and see what account the PHP process is currently running under.
And then the there is an IIS Authentication setting in IIS Manager under the website object. By default, Anonymous Login is enabled and when you look at the setting it uses a default user account (in my case it is IISUSER). It allows the option to change the Anonymous user to the Application Pool, in which case, it will work if Application Pool is set to the correct user.
Alternatively, forget completely about the Application Pool and directly set the user account as the default anonymous login account.
I am leaving my answer here and see if other can feedback on potential pit falls of this method of changing default anonymous user.