3

I have a mercurial web-frontend (hgwebdir.cgi) installed on a server, and an installation of nginx was installed in front of it as a reverse proxy to the web-frontend as my friend suggested. However, whenever a large changeset is pushed (via a script), it would fail. I found an issue ticket @google-code that describe similar problem, and there is a solution that says (#39)

So the server side answer is: don't send the 401 back early. Be as slow/dumb as 'hg serve' and make the hg client send the bundle twice.

How do I do that? My current nginx config

location /repo/testdomain.com {
    rewrite ^(.*) http://bpj.kkr.gov.my$1/hgwebdir.cgi;
}
location /repo/testdomain.com/ {
    rewrite ^(.*) http://bpj.kkr.gov.my$1hgwebdir.cgi;
}
location /repo/testdomain.com/hgwebdir.cgi {
    proxy_pass http://localhost:81/repo/testdomain.com/hgwebdir.cgi;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_buffering on;
    client_max_body_size 4096M;
    proxy_read_timeout 30000;
    proxy_send_timeout 30000;
}

From the access log we keep seeing 408 entries

incoming.ip.address - - [18/Nov/2009:08:29:31 +0800] "POST /repo/testdomain.com/hgwebdir.cgi/example_repository?cmd=unbundle&heads=73121b2b6159afc47cc3a028060902883d5b1e74 HTTP/1.1" 408 0 "-" "mercurial/proto-1.0"
incoming.ip.address - - [18/Nov/2009:08:37:14 +0800] "POST /repo/testdomain.com/hgwebdir.cgi/example_repository?cmd=unbundle&heads=73121b2b6159afc47cc3a028060902883d5b1e74 HTTP/1.1" 408 0 "-" "mercurial/proto-1.0"

Is there anything else I can do on the server because solving it on the server side is preferable :/

Further Findings

Bitbucket seems to have this solved ( Check liquidhg bitbucket project and the Diagnosis wiki page ) on the server side, can't find the config anywhere though :/

  1. What happens next varies depending on your server. Some servers refuse the BODY, simplying closing the pipe from the client and causing Mercurial to fail. Some, like Apache (at least the way I configure it, and that could be part of the problem) and nginx (they way BitBucket.org configures it), accept the BODY, though it may take a few retries. Bottom line: if Mercurial doesn't fail the push, it sends the changeset data at least once to a server that has already told it it lacks credentials (more on this at Blame).
  2. Assuming Mercurial is still running, it resends the "unbundle" request and data, this time with authentication.
  3. Finally, Apache accepts the data successfully. Nginx, OTOH, at least under BitBucket's configuration, seems to reassemble the previous body (the one that lacked authentication) and somehow keep Mercurial from re-sending the whole body.

4 Answers 4

0

I spent ages trying to get hgwebdir working with nginx and it is a bit of a hassle, because hgwebdir won't send the request to the browser for authentication. In the end I solved it like this:

server {
        listen 80;
        listen 10240;
        server_name code.zofrex.com;
        access_log /home/zofrex/websites/code.zofrex.com/logs/access.log;
        error_log /home/zofrex/websites/code.zofrex.com/logs/errors.log;

        location / {
                limit_except GET {
                        proxy_pass http://localhost:81;
                }
                fastcgi_pass 127.0.0.1:10001;
                include fastcgi_params;
        }

        location ~ ^/(HGBot|ZeroBotAHD|zoebot|FZeroZBot|RDPrototype|RSPS3000|zerobot|rmi)  {
                proxy_pass http://localhost:81;
        }

        include defaults;
}

server {
        listen 81;

        access_log /home/zofrex/websites/code.zofrex.com/logs/access_secure.log;
        error_log /home/zofrex/websites/code.zofrex.com/logs/errors_secure.log;

        location / {
                auth_basic           "Restricted";
                auth_basic_user_file /home/zofrex/passwords;
                fastcgi_pass 127.0.0.1:10001;
                include fastcgi_params;
        }

        include defaults;
}

This is the only way you can do it, really, as limit_except can't be combined with fastcgi_pass - in fact no conditionals can, but they be combined with proxy_pass. No idea why.

So how does this work? If a GET request comes in, it's handed off to hgwebdir straight away. If a POST (i.e. a push) comes in, the request upstreams to the proxy on port 81 which always prompts for basic auth, thus ensuring it happens.

Why the location with the huge regex? Those are protected projects that I want to require auth for even read access.

Why the listening on port 10240? I can't remember why that's there.

Oh, and I'm using wsgi with spawn-fcgi, which is why it's fastcgi_pass. This exact config probably won't be right for you but hopefully the same trick will solve your pushing woes.

1

The problem is that httplib is half-duplex and servers that aggressively close sockets after responding (nginx among them) cause httplib to write to a broken pipe. There's a band-aid workaround for this in 1.4, but it doesn't really make things faster, just less likely to completely fall apart. I don't know how to configure nginx to do what you want, but I know that on sufficiently large pushes BitBucket has this same problem.

What version of Apache and nginx are you using? It might be a good way for me to have a more "complete" test setup.

What are you using for authentication? Something in the Apache binary? It could be Apache is slamming the door too quickly after failing the auth.

Also, FWIW, liquidhg is the wonderful work of Brad Olson, and is merely hosted on BitBucket. Just wanted to give credit where it's due.

0

I had a similar problem recently and refused to try the listen 81 workaround. I came up with this. I think we have similar issues, check it out here for my question/answer: Nginx wont send POST to FastCGI backend, but GET works fine?

0

Why not simply do:

server {
    listen 80;
    listen 10240;
    server_name code.zofrex.com;
    access_log /home/zofrex/websites/code.zofrex.com/logs/access.log;
    error_log /home/zofrex/websites/code.zofrex.com/logs/errors.log;

    location / {
        fastcgi_pass 127.0.0.1:10001;
        include fastcgi_params;

        limit_except GET HEAD {
            auth_basic "Restricted";
            auth_basic_user_file /home/zofrex/passwords;
        }
    }

    location ~ ^/(HGBot|ZeroBotAHD|zoebot|FZeroZBot|RDPrototype|RSPS3000|zerobot|rmi)  {
        auth_basic "Restricted";
        auth_basic_user_file /home/zofrex/passwords;
    }

    include defaults;
}

It works since nginx 0.8.48, in older versions there was a bug where fastcgi_pass was not inherited inside the limit_except block.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .