0

I have several servers running Debian Stretch and I keep running into an issue where the process will hit a segmentation fault and stop. It will not return to service until I manually run service apache2 restart. I'm trying to figure out what is causing it so I can keep the server up and running but I have been unable to.

The server is running two Wordpress instances (one a public site, the other a private staging site for content purposes). Both are protected by Let's Encrypt via Certbot (I include this because of the [ssl:warn] in the error log below). We have not observed any memory or disk space issues when this has happened. Swap on these servers is hardly ever used.

Here is the output for service apache2 status after a segfault:

# service apache2 status
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Sun 2018-07-08 15:50:24 MST; 29min ago
  Process: 11833 ExecStop=/usr/sbin/apachectl stop (code=exited, status=0/SUCCESS)
  Process: 11828 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
 Main PID: 883 (code=exited, status=0/SUCCESS)

Jul 08 15:50:24 hostname systemd[1]: Starting The Apache HTTP Server...
Jul 08 15:50:24 hostname apachectl[11828]: httpd (pid 11770) already running
Jul 08 15:50:24 hostname systemd[1]: Started The Apache HTTP Server.

Here is the output in /var/log/apache2/error.log:

[Sat Jul 07 17:04:51.693795 2018] [core:notice] [pid 29385] AH00052: child pid 18866 exit signal Segmentation fault (11)
[Sat Jul 07 17:04:51.693918 2018] [mpm_prefork:notice] [pid 29385] AH00169: caught SIGTERM, shutting down
[Sat Jul 07 17:04:52.484310 2018] [ssl:warn] [pid 19421] AH01906: bb7f602e547898d78a02b844d49c34bc.4210997990497fe5b452e5c6c4250620.acme.invalid:443:0 server certificate is a C
A certificate (BasicConstraints: CA == TRUE !?)
/page/8/
[Sat Jul 07 17:04:51.693795 2018] [core:notice] [pid 29385] AH00052: child pid 18866 exit signal Segmentation fault (11)
[Sat Jul 07 17:04:51.693918 2018] [mpm_prefork:notice] [pid 29385] AH00169: caught SIGTERM, shutting down
[Sat Jul 07 17:04:52.484310 2018] [ssl:warn] [pid 19421] AH01906: bb7f602e547898d78a02b844d49c34bc.4210997990497fe5b452e5c6c4250620.acme.invalid:443:0 server certificate is a C
A certificate (BasicConstraints: CA == TRUE !?)
[Sat Jul 07 17:04:52.495766 2018] [ssl:warn] [pid 19422] AH01906: bb7f602e547898d78a02b844d49c34bc.4210997990497fe5b452e5c6c4250620.acme.invalid:443:0 server certificate is a C
A certificate (BasicConstraints: CA == TRUE !?)
[Sat Jul 07 17:04:52.498208 2018] [mpm_prefork:notice] [pid 19422] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2l configured -- resuming normal operations
[Sat Jul 07 17:04:52.498230 2018] [core:notice] [pid 19422] AH00094: Command line: '/usr/sbin/apache2'
[Sat Jul 07 17:04:58.754662 2018] [mpm_prefork:notice] [pid 19422] AH00171: Graceful restart requested, doing restart
[Sat Jul 07 17:04:58.766272 2018] [mpm_prefork:notice] [pid 19422] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2l configured -- resuming normal operations
[Sat Jul 07 17:04:58.766290 2018] [core:notice] [pid 19422] AH00094: Command line: '/usr/sbin/apache2'
[Sat Jul 07 17:05:00.039384 2018] [mpm_prefork:notice] [pid 19422] AH00171: Graceful restart requested, doing restart
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
[Sat Jul 07 17:05:00.050665 2018] [ssl:warn] [pid 19422] AH01906: 2af61f923209309052c60f342e6a0578.4287ae6d0b1c48707d1262e562b6250a.acme.invalid:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Sat Jul 07 17:05:00.051519 2018] [mpm_prefork:notice] [pid 19422] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2l configured -- resuming normal operations
[Sat Jul 07 17:05:00.051528 2018] [core:notice] [pid 19422] AH00094: Command line: '/usr/sbin/apache2'
[Sat Jul 07 17:05:06.063638 2018] [core:error] [pid 19422] AH00546: no record of generation 0 of exiting child 19423
[Sat Jul 07 17:05:06.420374 2018] [mpm_prefork:notice] [pid 19422] AH00171: Graceful restart requested, doing restart
[Sat Jul 07 17:05:06.431243 2018] [mpm_prefork:notice] [pid 19422] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2l configured -- resuming normal operations
[Sat Jul 07 17:05:06.431264 2018] [core:notice] [pid 19422] AH00094: Command line: '/usr/sbin/apache2'
[Sat Jul 07 17:05:07.965690 2018] [mpm_prefork:notice] [pid 19422] AH00171: Graceful restart requested, doing restart
[Sat Jul 07 17:05:07.976624 2018] [mpm_prefork:notice] [pid 19422] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2l configured -- resuming normal operations
[Sat Jul 07 17:05:07.976636 2018] [core:notice] [pid 19422] AH00094: Command line: '/usr/sbin/apache2'
[Sat Jul 07 17:05:07.977526 2018] [core:error] [pid 19422] AH00546: no record of generation 0 of exiting child 19550
[Sat Jul 07 17:05:08.211152 2018] [core:notice] [pid 19422] AH00052: child pid 19531 exit signal Segmentation fault (11)
[Sat Jul 07 17:05:08.211291 2018] [mpm_prefork:notice] [pid 19422] AH00169: caught SIGTERM, shutting down

We have the following software and hardware for the logs above (I can provide anything else that may be helpful):

  • apache2 2.4.25-3+deb9u4
  • Debian Stretch 9.4
  • PHP 7.0.27-0+deb9u1 with FPM
  • mariadb 10.1.26-0+deb9u1
  • 4x cores of Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
  • 8G RAM
  • 512MB swap
  • 95G SSD
2
  • Where is the segmentation fault? Jul 9, 2018 at 0:41
  • @MichaelHampton My apologies. I grabbed a log from a not segfault shutdown. I've updated the post with a log that starts with a segfault and shows its process of trying to start back up. After reviewing the logs more closely, it definitely indicated it was a Let's Encrypt issue and a quick test showed that Certbot was failing to renew due to a disabled vhost. I believe I have fixed this issue now. I will answer my own question to close this.
    – Josh
    Jul 9, 2018 at 1:15

1 Answer 1

1

The issue was caused by Certbot trying to renew certificates. If I ran certbot renew, I ran into these errors (I've sanitized the log a bit to remove domains and IPs):

Encountered vhost ambiguity when trying to find a vhost for domain2.com but was unable to ask for user guidance in non-interactive mode. Certbot may need vhosts to be explicitly labelled with ServerName or ServerAlias directives.
Falling back to default vhost *:443...
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (domain2.com) from /etc/letsencrypt/renewal/domain2.com.conf produced an unexpected error: Failed authorization procedure. domain2.com (tls-sni-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect validation certificate for tls-sni-01 challenge. Requested 942e8fc859beda1b41152fddc9579a1e.feafe6d59b7b25a33c08bca3c4be00e4.acme.invalid from 0.0.0.0:443. Received 2 certificate(s), first certificate had names "www.domain.com". Skipping.

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/www.domain.com.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/domain2.com/fullchain.pem (failure)

Afterward, running service apache2 status resulted in

● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Sun 2018-07-08 18:13:00 MST; 6s ago
  Process: 22401 ExecStop=/usr/sbin/apachectl stop (code=exited, status=0/SUCCESS)
  Process: 22396 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
 Main PID: 13700 (code=exited, status=0/SUCCESS)

Jul 08 18:13:00 hostname systemd[1]: Starting The Apache HTTP Server...
Jul 08 18:13:00 hostname apachectl[22396]: httpd (pid 22323) already running
Jul 08 18:13:00 hostname systemd[1]: Started The Apache HTTP Server.

I enabled the vhost in question, restarted Apache, and re-ran certbot renew and everything worked just fine. It was crashing frequently because certbot tries to renew twice daily by default.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .