A bit of background
I have ~150 scripts that directly or indirectly pull data from an IBM Cognos instance that I have no control over. I do not have access to the Cognos API, so the eventual source of all data for these scripts is simulating a web browser signing in as a normal user to run and download the reports. This is slow (5 seconds to 90 seconds depending on the report). Previously this was done by a Perl script that would get duplicated every time someone wanted to download a different report. This meant we had very duplicated code, and often were still downloading reports every day after they were no longer being used. I wrote a microservice to download an arbitrary report from Cognos to try to replace all the Perl scripts. You make a HTTP GET request to a url indicating the report you want, the microservice simulates a browser downloading the report, and you get back a CSV or JSON file (depending on which you requested). This has been great for reducing complexity, since each script can request data from a report in one or two lines right above the place where it is needed. When the report is no longer needed everything to clean up is obvious. We also don't have to synchronize schedules any more (ex: Perl script to download data at 7:30, Python to process it at 7:35).
The specific problem
Many of the reports are computationally expensive, and I recently received a mass email from the Cognos server administrator that was basically "Quit running the same reports over and over and try to run reports in off-peek hours". For us off-peak hours are late at night. Many of the scripts do things that should happen when people are in the office (ex: we have one that sends out phone calls to specific people), so in many cases I can't adjust the times the scripts run. In all cases, data which is up to 24 hours old is fine.
My thought was that since each report is just a HTTP GET request, I could just put a cache in front of the micro service. The problem though is that while we run quite a few reports every day, most of them are only run once a day. With most HTTP caches I am aware of, this would result in every request being a cache miss.
This is what I would like to happen: The first time a request comes in for a URL, it is a cache miss. The URL is downloaded form the microservice and passed to the client. Then the server knows it can expect to see a request for that URL again, so during off-peek hours it refreshes the cache. The next day when the script makes a request for the same URL it is a cache hit, and the response is returned immediately. If a URL is not hit in some period of time, the cache removes it from the list of URLs to be refreshed every night.
The general problem
Basically I'm looking for a HTTP cache that will learn what URLs are going to be hit and download them before they are requested. Does such a thing exist already or am I going to have to write that?