I launched a new instance of an Amazon Linux 2 server (from a daily backup snapshot). Everything looks normal and Apache starts but it will not respond to requests. I have ssh access and I can see that HTTP requests are coming in from the internet. Server has been running fine for months.
[ec2-user@ip-172-31-XXXXXX ~]$ httpd -v
Server version: Apache/2.4.46 ()
Server built: Aug 24 2020 18:54:20
[ec2-user@ip-172-31-90-216 ~]$ wget localhost
--2021-04-08 04:29:35-- http://localhost/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 504 Gateway Timeout
2021-04-08 04:30:36 ERROR 504: Gateway Timeout.
Normally I should receive a 404 response, not a timeout! I can't get my server back online. All the configs look correct. Nothing in the error log that I see.
What am I missing? I can see incoming requests timing out.
I have been trying to get this back online for three hours...
I restarted and rebooted, even reinstalled apache with no luck. Service is running.
[ec2-user@ip-172-31-xxxx ~]$ sudo systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/httpd.service.d
└─php-fpm.conf
Active: active (running) since Thu 2021-04-08 05:04:26 UTC; 57s ago
Docs: man:httpd.service(8)
Main PID: 2996 (httpd)
Status: "Total requests: 0; Idle/Busy workers 83/16;Requests/sec: 0; Bytes served/sec: 0 B/sec"
CGroup: /system.slice/httpd.service
├─2996 /usr/sbin/httpd -DFOREGROUND
├─2997 /usr/sbin/httpd -DFOREGROUND
├─2999 /usr/sbin/httpd -DFOREGROUND
├─3004 /usr/sbin/httpd -DFOREGROUND
├─3006 /usr/sbin/httpd -DFOREGROUND
├─3011 /usr/sbin/httpd -DFOREGROUND
└─3057 /usr/sbin/httpd -DFOREGROUND
Apr 08 05:04:26 ip-172-31-xxxx.ec2.internal systemd[1]: Starting The Apache HTTP Server...
Apr 08 05:04:26 ip-172-31-xxxx.ec2.internal systemd[1]: Started The Apache HTTP Server.
Error log
$ sudo tail /var/log/httpd/error_log
[Thu Apr 08 04:28:57.966581 2021] [suexec:notice] [pid 2309] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Apr 08 04:28:58.049821 2021] [lbmethod_heartbeat:notice] [pid 2309] AH02282: No slotmem from mod_heartmonitor
[Thu Apr 08 04:28:58.049925 2021] [http2:warn] [pid 2309] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Thu Apr 08 04:28:58.056290 2021] [mpm_prefork:notice] [pid 2309] AH00163: Apache/2.4.46 () OpenSSL/1.0.2k-fips configured -- resuming normal operations
[Thu Apr 08 04:28:58.056327 2021] [core:notice] [pid 2309] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Apr 08 04:30:36.009806 2021] [proxy_fcgi:error] [pid 2343] (70007)The timeout specified has expired: [client 127.0.0.1:36884] AH01075: Error dispatching request to : (polling)
[Thu Apr 08 04:31:01.250830 2021] [proxy_fcgi:error] [pid 2341] (70007)The timeout specified has expired: [client 52.7.78.221:11600] AH01075: Error dispatching request to : (polling)
[Thu Apr 08 04:31:01.913877 2021] [proxy_fcgi:error] [pid 2342] (70007)The timeout specified has expired: [client 3.239.43.177:45496] AH01075: Error dispatching request to : (polling)
[Thu Apr 08 04:35:18.853149 2021] [proxy_fcgi:error] [pid 3025] (70007)The timeout specified has expired: [client 127.0.0.1:36934] AH01075: Error dispatching request to : (polling)
[Thu Apr 08 04:35:23.560609 2021] [proxy_fcgi:error] [pid 3031] (70007)The timeout specified has expired: [client 158.51.21.11:59070] AH01075: Error dispatching request to : (polling), referer: https://example.com
Localhost ping test
[ec2-user@ip-172-31-xxxxx ~]$ ping localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=255 time=0.017 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=255 time=0.024 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=255 time=0.026 ms
Netstat Results
[ec2-user@ip-172-31-xxxx ~]$ sudo netstat -plan | grep :80
tcp6 0 0 :::80 :::* LISTEN 2996/httpd
tcp6 0 0 172.31.***.***:80 ***.***.167.232:37174 ESTABLISHED 3057/httpd