3

I want to setup a cache server for Rubygems, as I'm currently based in Vietnam and the international internet connections are pretty slow. I have been trying to get this done via Varnish, but after hours of googling and trying various things I'm still stuck and can't get it to work properly.

My goal

This a sample request group when I'm installing a gem:

GET http://api.rubygems.org/latest_specs.4.8.gz
302 Moved Temporarily
GET http://s3.amazonaws.com/production.s3.rubygems.org/latest_specs.4.8.gz
200 OK

I want to setup a reverse proxy cache server (e.g. rubygems.mydomain.com) where I can perform the following request and the cache server would follow any redirects internally.

rubygems.mydomain.com/latest_specs.4.8.gz

The redirect location will link to various domains (some rubygems subdomains, Amazon S3, rubygems mirror).

Current State

After fiddling around with nginx, I found this blog post which is pretty close what I wanna achieve. However, I have too little knowledge about how Varnish works to get it working properly.

That's my current config file

import std;

backend rubygems {
    .host = "rubygems.org";
    .port = "80";
}

sub vcl_recv {
    std.syslog(180, "RECV: " + req.http.host + req.url);
    if (!req.url  ~ "^http") {
      std.syslog(180, "FETCH");
      set req.backend = rubygems;
      return (lookup);
    }
}

sub vcl_fetch {
    if (beresp.status == 302) {
        set beresp.http.X-Magic-Redirect = "1";
        return(deliver);
    }
}

sub vcl_hit {
    if (obj.http.X-Magic-Redirect == "1") {
        set req.url = obj.http.Location;
        return (restart);
    }
}

sub vcl_deliver {
    if (resp.http.X-Magic-Redirect == "1") {
        unset resp.http.X-Magic-Redirect;
        return(restart);
    }
    return(deliver);
}

I can perform a request, but it responds with an error:

curl -is http://localhost:8080/latest_specs.4.8.gz
HTTP/1.1 302 Found
Server: Varnish
Content-Type: text/html; charset=utf-8
Retry-After: 5
Content-Length: 376
Accept-Ranges: bytes
Date: Sat, 01 Feb 2014 02:33:47 GMT
X-Varnish: 933109322
Age: 1
Via: 1.1 varnish
Connection: close


<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <head>
    <title>302 Found</title>
  </head>
  <body>
    <h1>Error 302 Found</h1>
    <p>Found</p>
    <h3>Guru Meditation:</h3>
    <p>XID: 933109322</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>

and that's the corresponding syslog output for the request:

Jan 31 18:33:46 precise64 varnishd[2387]: RECV: localhost:8080/latest_specs.4.8.gz
Jan 31 18:33:46 precise64 varnishd[2387]: FETCH
Jan 31 18:33:47 precise64 varnishd[2387]: RECV: localhost:8080/latest_specs.4.8.gz
Jan 31 18:33:47 precise64 varnishd[2387]: FETCH
Jan 31 18:33:47 precise64 varnishd[2387]: RECV: localhost:8080http://production.s3.rubygems.org/latest_specs.4.8.gz

So, the request to Rubygems is working fine, but following the redirect does not work as expected. I would be glad if somebody could point me in the right direction.

1
  • I think you're missing an adequate backend once the request have been restarted. Anyway, you should also change fill req.http.host and req.url appropiately when following the redirect
    – NITEMAN
    Feb 11, 2014 at 8:37

4 Answers 4

2

At this point when you will receive a 302 status from your rubygems backend, you have to make a request again at the new location specify by the HTTP header Location in the response.

You should end up with something like that:

vcl_fetch {
   if (beresp.status == 302) {  /* The content is on another location */

      /* First change the host of the request*/
      set req.http.host = regsub(regsub(beresp.http.Location, "^http://", ""), "^([^/]+)/.*$", "\1");

      /* Then change the url of the request */
      set req.url = regsub(beresp.http.Location, "^http://[^/]+/(.*)$", "/\1");
      return (restart);
   }
}
2

When I contacted the author of angry-caching-proxy, she responded:

Unfortunately your concern is correct. I have left the company two years ago for which I wrote it for - meaning my interest in maintaining it is fairly low. And it seems that nobody else has picked it up there...

0

If you get that working, please share.
Especially whether it works with the bundler dependency protocol.

There is some good information here Alternatively, you may want to take a look at geminabox

0
0

Another approach is https://www.npmjs.org/package/angry-caching-proxy which provides a proxy that will cache the gems on the way through.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .