Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We are a development team in Europe (Berlin to be precise) that heavily depends on the availability of raw.github.com. Our CI server way to often (sometime in 50% of all cases) reports failed builds due to HTTP 503 or 404 responses to e.g. GET https://raw.github.com/component/delegate/master/index.js

curl -I reveals that we hit a farm of Varnish servers on this side of the ocean, either in Amsterdam or in London. The response headers look like this for example:

HTTP/1.1 503 Connection timed out
Date: Mon, 29 Apr 2013 13:55:00 GMT
Server: Varnish
Retry-After: 0
Content-Type: text/html; charset=utf-8
Content-Length: 422
Accept-Ranges: bytes
Via: 1.1 varnish
Age: 1
X-Served-By: cache-l22-LON
X-Cache: MISS
X-Cache-Hits: 0
Connection: close

or, when it succeeds:

HTTP/1.1 200 OK
Date: Mon, 29 Apr 2013 14:03:27 GMT
Server: GitHub.com
Content-Type: text/plain; charset=utf-8
Status: 200 OK
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 100
X-Frame-Options: deny
Access-Control-Allow-Origin: https://render.github.com
X-Content-Type-Options: nosniff
Content-Disposition: inline
Content-Transfer-Encoding: binary
X-Runtime: 12
ETag: "e35e34ca022fff54b6a573fea7b7f42d"
Content-Length: 847
Accept-Ranges: bytes
Via: 1.1 varnish
Age: 0
X-Served-By: cache-a15-AMS
X-Cache: MISS
X-Cache-Hits: 0
Vary: Accept-Encoding
Cache-Control: private

(This does not mean it always succeeds for AMS and never for LON, it actually seems to be random)

The build tools that do the request don't seem to do retries. My question is: how should we approach this problem? One way would be to put our own caching proxy between our CI server and the Varnish cluster and tell it to do retries. The Retry-After: 0 header seems to imply this solution.)

Is there a better way to deal with this?

share|improve this question
1  
Yes, don't depend on raw.github.com. Try cloning the repo for example, this is likely to be significantly less flakey –  Thom Seddon Dec 16 '13 at 10:51
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.