1.) The first hit on the server and the first invocation of the PHP script in question is actually causing a number of configurations to be parsed, interpreters (possibly) loaded, scripts loaded, modules in the web server coming online, etc. In subsequent queries it's almost invariably faster.
2.) Even under ideal conditions there are a number of moving parts in play during your 0.5 - 2 sec - between your machine going through the setup of several TCP sessions (each requiring a few msec of handshaking), your browser identifying itself and working out capabilities with the server, your browser finally submitting its request and then the server having to parse that information and pass it up and down the stack to then finally format it into HTML to send back to you.
There's quite a bit of sophistication in the implementation of modern scripting / web platforms. It may seem a bit heavy when compared to pinging a server or putting up a static "Hello world!" html page, but the generalized capabilities are pretty impressive.
The other point, of course, is that you're describing a completely stock system. There are almost always things that can be done to tune it to better respond to your particular workload.