2

A bit of background

I have ~150 scripts that directly or indirectly pull data from an IBM Cognos instance that I have no control over. I do not have access to the Cognos API, so the eventual source of all data for these scripts is simulating a web browser signing in as a normal user to run and download the reports. This is slow (5 seconds to 90 seconds depending on the report). Previously this was done by a Perl script that would get duplicated every time someone wanted to download a different report. This meant we had very duplicated code, and often were still downloading reports every day after they were no longer being used. I wrote a microservice to download an arbitrary report from Cognos to try to replace all the Perl scripts. You make a HTTP GET request to a url indicating the report you want, the microservice simulates a browser downloading the report, and you get back a CSV or JSON file (depending on which you requested). This has been great for reducing complexity, since each script can request data from a report in one or two lines right above the place where it is needed. When the report is no longer needed everything to clean up is obvious. We also don't have to synchronize schedules any more (ex: Perl script to download data at 7:30, Python to process it at 7:35).

The specific problem

Many of the reports are computationally expensive, and I recently received a mass email from the Cognos server administrator that was basically "Quit running the same reports over and over and try to run reports in off-peek hours". For us off-peak hours are late at night. Many of the scripts do things that should happen when people are in the office (ex: we have one that sends out phone calls to specific people), so in many cases I can't adjust the times the scripts run. In all cases, data which is up to 24 hours old is fine.

My thought was that since each report is just a HTTP GET request, I could just put a cache in front of the micro service. The problem though is that while we run quite a few reports every day, most of them are only run once a day. With most HTTP caches I am aware of, this would result in every request being a cache miss.

This is what I would like to happen: The first time a request comes in for a URL, it is a cache miss. The URL is downloaded form the microservice and passed to the client. Then the server knows it can expect to see a request for that URL again, so during off-peek hours it refreshes the cache. The next day when the script makes a request for the same URL it is a cache hit, and the response is returned immediately. If a URL is not hit in some period of time, the cache removes it from the list of URLs to be refreshed every night.

The general problem

Basically I'm looking for a HTTP cache that will learn what URLs are going to be hit and download them before they are requested. Does such a thing exist already or am I going to have to write that?

3
  • 1
    i aint complete understand your points but either nginx or squid infront of your server could help you as reverseproxy and cache often requested items
    – djdomi
    Dec 14, 2019 at 22:26
  • @djdomi You're correct that I could solve part of the problem (often requested items) with nginx or squid. I'm mostly asking about the other part (requests that come it at bad times).
    – 9072997
    Dec 14, 2019 at 22:32
  • try first nginx and give us Feedback.
    – djdomi
    Dec 14, 2019 at 22:37

1 Answer 1

0

Many of the scripts do things that should happen when people are in the office (ex: we have one that sends out phone calls to specific people), so in many cases I can't adjust the times the scripts run. In all cases, data which is up to 24 hours old is fine.

Decouple data gathering from the action.

  • Get the data at some time of night, say 0315. (Odd times not on the hour may be less busy.)
  • Process the data, generating reports or whatever.
  • Schedule any deferred actions, say at 0800. Absent any way you already do this, on a UNIX or Linux box consider at jobs.

My thought was that since each report is just a HTTP GET request, I could just put a cache in front of the micro service. The problem though is that while we run quite a few reports every day, most of them are only run once a day. With most HTTP caches I am aware of, this would result in every request being a cache miss

If you are set on (ab)using HTTP as cache, look more carefully at the options of caching software. For example, Squid can force non-compliant override of expire times. Then your script can GET the data with a miss overnight, and GET it again with a hit when you actually want it.

The tricky bit here is setting a cache time that has hits and misses where you want. Personally, decoupling the data gathering and action schedules seems like a cleaner solution to me.

2
  • My issue with decoupling data gathering and actions is that I have to manually maintain a list of what reports are needed by what scripts. I'm not tossing that idea out, I'm just saying it's not ideal. As to the http cache: I could have my microservice return Expires headers that are always at night, so I wouldn't be doing anything too custom as far as overriding expire times on the cache. The question remains though of how to warm the cache. I appreciate the suggestions but it's looking increasingly like if I want this I'm going to have to write it.
    – 9072997
    Dec 16, 2019 at 1:07
  • Of course somebody has to maintain lists of data, reports, and actions. That is necessary for any flexible reporting system, and your requirements are getting sophisticated. Warm a cache by doing the same GET, just don't do anything with the data yet. Dec 16, 2019 at 12:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .