Why this is happening
The behaviour is probably down to browser caching, but the confusion is due to how the responses are displayed. This is a big assumption on which this whole answer hinges, so forgive me if this is incorrect.
I find Chrome browser (hit f12, 'Network' tab) is better at illustrating this than Firefox.
Each time I click on a link to the page, all resources are fetched from the server, responding with 200s.
What is likely happening is that when you 'follow a link' (which should be the same at directly entering the url) you are seeing 200 responses which are a mix of real requests and browser cached responses. This is normal behaviour. Chrome illustrates 'from cache' responses by explicitly stating 'from cache' against each resource in the network tab. I believe FF illustrates this as greyed responses in the timeline.
In both browsers when items are retrieved from cache they still display the status response. This is from the last server response, but this is still a cached response.
However, when I refresh the page (specifically, F5 in Firefox), all resources return a 304 and - of course - the page loads much faster as a result.
When you hit F5, you are forcing a request to be sent, not to disregard the cache entirely, but check again at the server if it has changed. Your request headers contain the the
If-Modified-Since because it is still available from cache. A 304 is returned, and your browser uses the cached version confident that the server version has not changed.
Following a link = Use cached data.
F5 = Send requests again.
Ctrl+F5 = Send requests again but also disregard the local cache.
The main page returns a 200 in both cases.
Browser caching is likely disabled for the main page (perhaps all .html content by default) by way of the response headers returned for it. Because of this, it cannot send a
If-Modified-Since in the refreshed request, because it doesn't exist locally in cache, and so there is no date to compare content with. Because there is no
If-Modified-Since sent in the request, the server must respond with another 200 with the full page content.
In the refresh case,
If-Modified-Since headers are sent with the requests to the resources.
Because the resource items are available from cache and so there is a date to send - the date when the items were added to the browser's cache.
However, in the 'clicking a link' case, they are not. What's the
reason for that, and can I control it?
Because the browser doesn't send If-Modified-Since headers when retrieving from local cache. The 200 "response" is a cached one. You don't need to control that as such.
None of this explains why it loads faster when you hit F5. Are you positive this is correct?
On a separate note, be aware of browser 'heuristic caching'. This is the behaviour the browser adopts when caching isn't explicity defined by the reponse headers, and is essentially a 'best guess' behaviour. Naturally it differs for each browser.