2

tl;dr : Certbot stopped generating certificates, shows a "Redirect loop detected" error, but there is no actual redirect loop.

The following issue appeared around 2023-11-30. I've focused on a single website of a single server, but the same occurs on every server and every website that I tried within our datacenter. (On my personal OVH VPS the issue doesn't occur.)

So, I have this server that hosts a few thousands Wordpress websites, each with its own domain name. I regularly add new sites. To generate a new certificate, I use the following command line :

/snap/bin/certbot certonly --register-unsafely-without-email --config-dir certbot/conf --work-dir certbot/work --logs-dir certbot/logs --webroot --webroot-path /home/wordpress/ -d testcert.numerian.fr,www.testcert.numerian.fr

This worked fine for years, but since last week, it shows me this error :

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: testcert.numerian.fr
  Type:   connection
  Detail: 46.18.231.82: Fetching /.well-known/acme-challenge/s2OgquUDSF4hzNPUse66QmM-KOTPApHXrLcIJqFuQTA: Redirect loop detected

  Domain: www.testcert.numerian.fr
  Type:   connection
  Detail: 46.18.231.82: Fetching /.well-known/acme-challenge/14WgGIBhYGCcUx6BYro1cI4y6UfZ9Z_20uYgFtJQ6-M: Redirect loop detected

Two points :

  1. The config of the nginx server hasn't changed in months, so there is no reason for a redirect loop to have appeared. Furthermore, if I re-create the file manually in this directory, I can access it fine with curl, no redirect at all :

    curl -i http://www.testcert.numerian.fr/.well-known/acme-challenge/14WgGIBhYGCcUx6BYro1cI4y6UfZ9Z_20uYgFtJQ6-M

  2. If I look into the certbot logs, I see something odd : It seems that letsencrypt tries to access the urls

Here is the relevant part of the log :

{
  "identifier": {
    "type": "dns",
    "value": "www.testcert.numerian.fr"
  },
  "status": "invalid",
  "expires": "2023-12-12T08:04:51Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "46.18.231.82: Fetching /.well-known/acme-challenge/14WgGIBhYGCcUx6BYro1cI4y6UfZ9Z_20uYgFtJQ6-M: Redirect loop detected",
        "status": 400
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/290578127326/TNJCjw",
      "token": "14WgGIBhYGCcUx6BYro1cI4y6UfZ9Z_20uYgFtJQ6-M",
      "validationRecord": [
        {
          "url": "http://www.testcert.numerian.fr/.well-known/acme-challenge/14WgGIBhYGCcUx6BYro1cI4y6UfZ9Z_20uYgFtJQ6-M",
          "hostname": "www.testcert.numerian.fr",
          "port": "80",
          "addressesResolved": [
            "46.18.231.82"
          ],
          "addressUsed": "46.18.231.82"
        },
        {
          "url": "http://www.testcert.numerian.fr/phdQX/.well-known/acme-challenge/14WgGIBhYGCcUx6BYro1cI4y6UfZ9Z_20uYgFtJQ6-M",
          "hostname": "www.testcert.numerian.fr",
          "port": "80",
          "addressesResolved": [
            "46.18.231.82"
          ],
          "addressUsed": "46.18.231.82"
        }
      ],
      "validated": "2023-12-05T08:04:51Z"
    }
  ]
}

This has to be new, as this prefixed url couldn't have worked before. However, even after configuring nginx so this prefixed-url would point to the correct file, the certificate generation still fails. This works now, but would have been a 404 error last week :

curl -i http://www.testcert.numerian.fr/phdQX/.well-known/acme-challenge/14WgGIBhYGCcUx6BYro1cI4y6UfZ9Z_20uYgFtJQ6-M

Here is the current relevant nginx config (redirects HTTP to HTTPS, except for acme-challenge, and removes the prefix before well-known/acme-challenge if present). (The rewrite was only added yesterday.)

server {
    server_name _;
    listen 80 default_server;

    root /home/wordpress;

    location / {
            return 301 https://$host:443$request_uri;
    }
    location ~ well-known/acme-challenge {
        rewrite ^(/\w+)(/\.well-known/acme-challenge/.*)$ $2 last;
    }
}

More nginx config here : https://gist.github.com/bveuillez/49727a30cc4534ce35b9c0d79ea1d779

I've tried everything I could think of to fix this issue, mostly reinstalling certbot from scratch, trying various certbot options and messing with the nginx config, but I didn't get anywhere. I would be very grateful for any hint about the source of the issue.


EDIT : further failed attempts

I tried adding the options --debug-challenges -v

/snap/bin/certbot certonly --register-unsafely-without-email --config-dir certbot/conf --work-dir certbot/work --logs-dir certbot/logs --webroot --webroot-path /home/wordpress/ -d testcert.numerian.fr,www.testcert.numerian.fr --dry-run --debug-challenges -v

which showed me the expected values for two URLs, and curl-ing those URLs gave the right value, without any redirect. Still, the challenge failed with "Redirect loop detected".

letsdebug.net also tells me that it sees 302 redirects, of which I find no traces in my nginx logs.

I tried replacing --webroot by --nginx

/snap/bin/certbot certonly --register-unsafely-without-email --config-dir certbot/conf --work-dir certbot/work --logs-dir certbot/logs --nginx -d testcert.numerian.fr,www.testcert.numerian.fr --dry-run --debug-challenges -v

I could see the changed nginx config, I could curl the two URLs and get the right value, and still I get the error "Redirect loop detected".

I tried configuring this domain to have a webroot separate from the whole wordpress thing, in case it came from that somehow :

server {
    server_name testcert.numerian.fr;
    listen 80;

    root /var/www/html;
}

then the same certbot command as earlier, with --debug-challenges, and same as before I got the right values when curl-ing the URL, but then got the error "Redirect loop detected"

EDIT :

I ended up creating a new VM with just nginx and certbot to compare. Not only did certbot work fine on it, it also started working correctly again on all the other servers. I still can't figure out what caused then fixed all this, but at least the issue is gone.

1 Answer 1

2

You do have a redirect loop:

$ curl -I http://testcert.numerian.fr/.well-known/acme-challenge/s2OgquUDSF4hzNPUse66QmM-KOTPApHXrLcIJqFuQTA
HTTP/1.1 302 Found
Connection: close
Pragma: no-cache
cache-control: no-cache
Location: /ZQOLV/.well-known/acme-challenge/s2OgquUDSF4hzNPUse66QmM-KOTPApHXrLcIJqFuQTA

$ curl -I http://testcert.numerian.fr/ZQOLV/.well-known/acme-challenge/s2OgquUDSF4hzNPUse66QmM-KOTPApHXrLcIJqFuQTA
HTTP/1.1 302 Found
Connection: close
Pragma: no-cache
cache-control: no-cache
Location: /RkUQX/ZQOLV/.well-known/acme-challenge/s2OgquUDSF4hzNPUse66QmM-KOTPApHXrLcIJqFuQTA

$ curl -I http://testcert.numerian.fr//RkUQX/ZQOLV/.well-known/acme-challenge/s2OgquUDSF4hzNPUse66QmM-KOTPApHXrLcIJqFuQTA  
HTTP/1.1 302 Found
Connection: close
Pragma: no-cache
cache-control: no-cache
Location: /PmcKW//RkUQX/ZQOLV/.well-known/acme-challenge/s2OgquUDSF4hzNPUse66QmM-KOTPApHXrLcIJqFuQTA

Your web server is doing these redirects, adding five random digits, for some reason. From the config you've posted I can't see what is doing it. You can dump the entire config with nginx -T

3
  • Odd, I didn't recreate s2OgquUDSF4hzNPUse66QmM, on my end the first curl gives a 404 error, not a 302 redirect. Could you try the first curl in my question please ? with 14WgGIBhYGCcUx6BYro1cI4y6UfZ9Z_20uYgFtJQ6-M
    – user153991
    Dec 5, 2023 at 9:21
  • That contains test and does not redirect...
    – vidarlo
    Dec 5, 2023 at 9:28
  • Alright, thank you. Still I need to understand where those 302 you see come from, I'm going to work on this. It has to be related to the issue. I added the result of nginx -T to the question.
    – user153991
    Dec 5, 2023 at 9:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .