19

This is a pretty long question, so bear with me.

I wanted to stress my Akamai Server logged in from an AWS instance. So, I started running ab benchmark. However, they seemed ridiculously fast to download ~3 MB video files. Naturally I wanted to see what's going on. This is what I did to get the file

curl -v -o /dev/null

The above completed in ~5 seconds.

Next, I ran the same command again. This time, it completed in ~200ms! Naturally, my intuition says the file is being cached somewhere.

My questions:

  1. Does curl cache files? If so, is there a way to ignore it?
  2. If curl doesn't, does the ubuntu abstracts a cache beneath curl? If so, is there a way to ignore it?
  3. Given the requirements, do you think there could be a benchmarking tool apart from ab that could serve the purpose?

Thank you, Akshay

15

The curl client isn't caching files, but the remote server network might well be. Try adding an arbitrary query string variable to the URL to see if you can reproduce it.

  • Thank you for your answer. I couldn't add arbitrary query string as the Akamai server that I use doesn't accept any query params! (forcing error as it relies on salted token digest of timestamp and URL). However I was able to generate multiple tokens for the same path (essentially multiple URLs) and you are absolutely right. curl wasn't caching any file - the remote server was. Go CDN! :) – Akshaya Shanbhogue Jun 11 '15 at 19:04
6

Belatedly, try:

curl -v -H "Cache-Control: no-cache"

That will tell the web server to not cache. Doesn't stop layers below caching unless it's coded to obey the headers.

0

I've used this curl command with a cache buster parameter.

curl http://example.com/static/changing_file?_=$(date +%s)

date +%s prints the seconds since the epoch, if you call the url more than once a second use date +%s.%N to add in nanoseconds.

0

You can use add a random query string using the $RANDOM environment variable:

curl --location --silent "https://git.io/lsf-e2e?$RANDOM"

This worked for me on github raw files.

-2

Maybe your dns is caching the resolution of the name and this is the reason of the diference in time of response.

It's only a theory.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.