0

Our Server hosting PHP web application is facing frequent downtime

Server Information [Nginx, FreeBSD] Web application [PHP 5.6, MYsql 5.7]

I have gone through the Nginx logs and below are my findings

error.log has some below logs

2023/05/31 19:48:16 [error] 1456#100101: *7408 open() "/usr/local/www/html/uploads/files/22816683587.pdf" failed (2: No such file or directory), client: 5.255.231.177, server: localhost, request: "GET /uploads/files/22816683587.pdf HTTP/1.1", host: "somesite.com"

there any several request like this to access random pdf files from server from different Ip's

debug.log has some overflow logs

May 31 18:43:40 ip-XX-XX-XX-XX kernel: sonewconn: pcb 0xfffff800e1910dc8: Listen queue overflow: 193 already in queue awaiting acceptance (265 occurrences) 
May 31 18:44:40 ip-XX-XX-XX-XX kernel: sonewconn: pcb 0xfffff800e1910dc8: Listen queue overflow: 193 already in queue awaiting acceptance (330 occurrences) 
May 31 18:45:40 ip-XX-XX-XX-XX kernel: sonewconn: pcb 0xfffff800e1910dc8: Listen queue overflow: 193 already in queue awaiting acceptance (314 occurrences)

When I checked below netstat result at the time of downtime

root@myIP:~ # netstat -Lan
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address
tcp6  0/0/128        *.443
tcp4  0/0/128        *.80
tcp4  0/0/128        *.443
tcp4  0/0/10         127.0.0.1.25
tcp4  0/0/128        *.22
tcp6  0/0/128        *.22
tcp46 0/0/80         *.3306
tcp4  193/0/128      127.0.0.1.9000
unix  0/0/80         /tmp/mysql.sock
unix  0/0/4          /var/run/devd.pipe
unix  0/0/4          /var/run/devd.seqpacket.pipe

From above I found "tcp4 193/0/128 127.0.0.1.9000" this used by PHP FPM service

I check Php-FPM logs too, there is no such slow process logged that is taking more than 5 second, there are some but those records logged in when the server is in getting slow or down

Some Nginx and FastCGi Paramters are set in Nginx.conf

client_header_timeout 3000;
client_body_timeout 3000;
fastcgi_read_timeout 3000;
client_max_body_size 32m;
fastcgi_buffers 8 128k;
fastcgi_buffer_size 128k;
server_name_in_redirect off;
server_names_hash_bucket_size 64;
server_names_hash_max_size 8192;

We enabled MySQL's slow query log, but nothing logged into it the result is empty.

We contacted AWS as the server is using an AWS instance, they told us that there is high CPU usage sometimes randomly and there is no issue with Database, and it looks healthy

We asked them if we are under a DDOs attack, but they refused it saying that the network stat doesn't show it.

We are struggling to understand and find the exact reason or process behind this server downtime.

Please help, Thanks in advance

Update

kernel: sonewconn: pcb 0xfffff80007326c40: Listen queue overflow: 1537 already in queue awaiting acceptance (167 occurrences)

I increased the limit of listen queue but now that is also getting overloaded

10
  • why did you mention there is no such slow process logged that is taking more than 5 second? there's no other mention at all regarding any slow process - you state that the server is down, not slow - as for the random requests - every public server gets that Commented Jun 2, 2023 at 5:11
  • I just clear out that aspect so that while going through my query one can understand that no process is slow
    – Ekky
    Commented Jun 2, 2023 at 5:18
  • so, what you're saying is that there's no indication in the logs you've looked at as to why the site is down - only a vague suggestion from AWS that there is occasional high CPU usage, but you haven't been able to correlate the high CPU usage with down time ... is that the gist of it? Commented Jun 2, 2023 at 6:08
  • Yes, I monitored HTOP outcomes while sites were running and CPu usage was normal even then server getting down , I suspect "sonewconn: pcb 0xfffff800e1910dc8: Listen queue overflow:" is the issue behind server down time, this is used by "tcp4 193/0/128 127.0.0.1.9000" (PHP FPM)
    – Ekky
    Commented Jun 2, 2023 at 6:14
  • is it possible due to misconfigured Fastcgi, PHP-FPM, or nginx configurations?
    – Ekky
    Commented Jun 2, 2023 at 6:17

1 Answer 1

2

It looks to me like you're on the right track. If you take a closer look at the output of netstat -Lan:

root@myIP:~ # netstat -Lan
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address
tcp4  193/0/128      127.0.0.1.9000

It is telling you that the length of the queue (qlen) is 193 while the max. length is only 128 (maxqlen). Since the queue length exceeds the maximum length of 128, you're dropping connections. This is also confirmed by the log messages like

May 31 18:45:40 ip-XX-XX-XX-XX kernel: sonewconn: pcb 0xfffff800e1910dc8: Listen queue overflow: 193 already in queue awaiting acceptance (314 occurrences)

where the number 193 is also mentioned.

You can confirm that your maxqlen is set to 128:

sysctl kern.ipc.somaxconn

You may want to set it to a higher value like so:

sysctl -w kern.ipc.somaxconn=1024

In order to make this change persistent, so that it survives a reboot, you would to add a line to your /etc/sysctl.conf:

kern.ipc.somaxconn=1024

This larger queue would probably make your Listen queue overflow messages go away and avoid the connection drops / downtime.

However, it should be investigated why there are so many requests and whether this is expected load or if there are some automated / scripted attacks happening. In the later case, it would be a good mitigation to configure fail2ban to blacklist IP adresses that exhaust a threshould of a given number of downloaded PDF files per minute or something like that.

12
  • 1
    Thank you for your help ! I checked output of sysctl kern.ipc.somaxconn which comes 128, So as you suggested I added line kern.ipc.somaxconn=1024 "/etc/sysctl.conf" and rebooted server. But when I again checked output of sysctl kern.ipc.somaxconn it comes same 128 and file "/etc/sysctl.conf" still has new line kern.ipc.somaxconn=1024 kern.ipc.somaxconn: 128 is it overriding from someone else or instead of reboot I need to do something else
    – Ekky
    Commented Jun 2, 2023 at 13:22
  • @Ekky There is a service that should apply the settings from /etc/sysctl.conf. You can check if it is running on your system with /etc/rc.d/sysctl status. You can even try to execute /etc/rc.d/sysctl start and see if the setting is applied or if you get any errors. Please make sure the line with kern.ipc.somaxconn=1024 does not have quotation marks or spaces in it. If everything else fails, please try to add this line to /boot/loader.conf instead of /etc/sysctl.conf. Commented Jun 2, 2023 at 13:53
  • Yes, I confirm that "sysctl is running" additionally from below reference I added same line in "/boot/loader.conf ", serverfault.com/questions/171797/…
    – Ekky
    Commented Jun 2, 2023 at 13:57
  • Still output is below, sysctl kern.ipc.somaxconn kern.ipc.somaxconn: 128 , change is not persisting after reboot
    – Ekky
    Commented Jun 2, 2023 at 14:01
  • @Ekky that's strange. The only reason I could think of is that some startup script overwrites this value. Obviously it is hard to tell where this happens, maybe you can try grep -R "kern.ipc.somaxconn" /etc/ to find occurrences. Commented Jun 2, 2023 at 14:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .