1

Running on an AWS ec2 instance, a bit of context, I'm a frontend guy trying to solve an issue with a production web server and have run out of troubleshooting/linux cli knowledge and am out of my comfort zone.

I made a change to the httpd.conf file to add a new domain and followed instructions that said to use sudo service httpd restart so the changes I made would take effect. That caused an error and now the httpd service has stopped meaning none of our sites are being served.

I've tried the following things to start the httpd service again:

  • apachectl configtest returns Syntax OK
  • sudo service httpd start returns the following error:
Redirecting to /bin/systemctl start httpd.service
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

-> running systemctl status httpd.service after that gives this info:

httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2021-01-18 16:00:20 UTC; 1min 20s ago
     Docs: man:httpd.service(8)
  Process: 5943 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 5943 (code=exited, status=1/FAILURE)
   Status: "Reading configuration..."

Jan 18 16:00:20 ip-172-31-38-82.eu-west-1.compute.internal systemd[1]: Starting The Apache HTTP Server...
Jan 18 16:00:20 ip-172-31-38-82.eu-west-1.compute.internal httpd[5943]: [Mon Jan 18 16:00:20.317159 2021] [so:warn] [pid 5943] AH01574: module headers_module is already loaded, skipping
Jan 18 16:00:20 ip-172-31-38-82.eu-west-1.compute.internal systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Jan 18 16:00:20 ip-172-31-38-82.eu-west-1.compute.internal systemd[1]: Failed to start The Apache HTTP Server.
Jan 18 16:00:20 ip-172-31-38-82.eu-west-1.compute.internal systemd[1]: Unit httpd.service entered failed state.
Jan 18 16:00:20 ip-172-31-38-82.eu-west-1.compute.internal systemd[1]: httpd.service failed.
[ec2-user@ip-172-31-38-82 ~]$ 
  • running sudo apachectl -k start gives the following warning but no error:
[so:warn] [pid 5967] AH01574: module headers_module is already loaded, skipping
  • Another topic I saw said that something else might already be running on port 80 and to use sudo netstat -lpAinet to see what else is running, which returned:
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN      3246/sshd           
tcp        0      0 localhost:smtp          0.0.0.0:*               LISTEN      3208/master         
tcp        0      0 0.0.0.0:sunrpc          0.0.0.0:*               LISTEN      2750/rpcbind        
udp        0      0 0.0.0.0:bootpc          0.0.0.0:*                           2971/dhclient       
udp        0      0 0.0.0.0:sunrpc          0.0.0.0:*                           2750/rpcbind        
udp        0      0 localhost:323           0.0.0.0:*                           2772/chronyd        
udp        0      0 0.0.0.0:799             0.0.0.0:*                           2750/rpcbind

And nothing here says port 80 or has the program name apache.

  • There was another command I tried to get the httpd service started which returned an error that said the service couldn't be bound to port 80 (I can't find what the actual command was now). The post mentioned the httpd service needs to be started with root permissions to assign port numbers lower than 1024. I tried checking my permissions using sudo -l and the following was returned:
User ec2-user may run the following commands on ip-172-31-38-82:
    (ALL) ALL
    (ALL) NOPASSWD: ALL

which seems like I have the correct permissions to run it.

And this is where I reached the limit of where I felt comfortable troubleshooting by myself.

7
  • What systemd isn’t capable of determining is that in addition to the logs and errors recorded in journald Apache also maintains its own (error) log. That is usually where Apache will store more useful and verbose (startup ) errors. Usually those will be in a /var/log sub directory
    – Bob
    Jan 18, 2021 at 16:32
  • 1
    @HermanB Thanks for the tip, I had a look at the error log for httpd in there and the most recent one (as well as all the ones before that from my previous attempts) say: [suexec:notice] [pid 5995] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) AH00016: Configuration Failed
    – ebeck
    Jan 18, 2021 at 16:38
  • 1
    Although Suexec is quite particular about many things (see for example : httpd.apache.org/docs/2.4/suexec.html ) that message seems to be a red herring . The AH00016: Configuration Failed message is probably the real issue. You may see more in the other Apache logs (is there maybe a ssl-error-log ?)
    – Bob
    Jan 18, 2021 at 16:55
  • Yes there is an ssl_error_log where the most recent message is [ssl:emerg] [pid 5995] AH02565: Certificate and private key ip-172-31-38-82.eu-west-1.compute.internal:443:0 $. There is also ssl_access_log and ssl_request_log but they are both empty. I know there is only an SSL certificate on one of the domains and it has expired, would that be causing this issue?
    – ebeck
    Jan 18, 2021 at 17:01
  • That error message seems truncated and missing some key detail. - Also an expired certificate is in itself not reason why Apache would fail to start, there is more going on there, but an ssl issue seems to be the root cause
    – Bob
    Jan 18, 2021 at 17:07

1 Answer 1

0

As suggested by Andrew Schulman in the comments on my question, the problem was not with the config but with the the SSL certificate used by one of the domains. The certificate.crt and private.key files did not match which was admittedly my fault for trying to swap them out previously.

After searching the error message AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) AH00016: Configuration Failed which I found thanks to HermanB, I realised it must have been my attempted certificate update that caused the issue and sure enough updating both parts of the certificate fixed it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .