I am diagnosing a weird problem with some of my Ubuntu 14 LTS / Apache 2.4 machines with Certbot renewals. Yes, I know Ubuntu 14 LTS is EOL soon; these are some of the straggler machines on the rebuild list.
Essentially, Certbot has been working well for a while on these machines, with renewals handled via a cron job. Sometime between October and now, renewals have been failing with a 403 during the HTTP-01 challenge:
Detail: Invalid response from http://example.com/.well-known/acme-challenge/asdkjhfaklsjdfhalksdjfhkljasdfh: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p" To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
I have diagnosed the problem down to the .well-known directory being denied during the challenges because of Apache config (a
Require all denied on /).
As a stop-gap, I am renewing the certs by hand by a temporary
Require all allowed on /, but want to fix this properly.
So, where does Certbot actually serve up challenges from? I tried a
<Directory /var/lib/letsencrypt/http_challenges> Options Indexes FollowSymLinks AllowOverride None Require all granted </Directory>
but that didn't seem to work. Because of the ratelimiting, guess-and-checks are slow.