1

Nginx 1.20.1 server working as a reverse proxy consuming a lot o memory and making the service stop requiring a system reboot. ModSecurity library is also configured on Nginx.

You can see in the next graph that the number of established connections keep constant around 200 while the second graph shows memory continually increasing until it reaches 100%.

Resource consumption by Nginx

When I filter the output of the top command COMMAND=nginx I see:

top - 16:31:18 up 1 day,  1:54,  2 users,  load average: 0.26, 0.57, 0.66
Tasks: 179 total,   2 running, 177 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.0 us,  0.2 sy,  0.0 ni, 93.4 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  12010.4 total,    184.6 free,   3818.6 used,   8007.3 buff/cache
MiB Swap:    975.0 total,    968.7 free,      6.3 used.   7858.8 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
26553 nginx     20   0 2624044   2.4g   9004 S   7.0  20.3   1:34.85 nginx
26554 nginx     20   0 2625580   2.4g   8932 S   0.7  20.4   0:18.12 nginx
26555 nginx     20   0 2627508   2.4g   8872 S   0.7  20.4   0:04.52 nginx
10284 root      20   0 2624568   2.4g   7772 S   0.0  20.3   0:29.08 nginx
26556 nginx     20   0 2624572   2.4g   8812 S   0.0  20.3   0:00.93 nginx

Here's the output of pmap -d 26553 command:

26553:   nginx: worker process
Address           Kbytes Mode  Offset           Device    Mapping
0000557a808e7000     164 r---- 0000000000000000 008:00001 nginx
0000557a80910000     824 r-x-- 0000000000029000 008:00001 nginx
0000557a809de000     204 r---- 00000000000f7000 008:00001 nginx
0000557a80a12000       8 r---- 000000000012a000 008:00001 nginx
0000557a80a14000     132 rw--- 000000000012c000 008:00001 nginx
0000557a80a35000     128 rw--- 0000000000000000 000:00000   [ anon ]
0000557a81b7c000  264740 rw--- 0000000000000000 000:00000   [ anon ]
0000557a91e05000 1965680 rw--- 0000000000000000 000:00000   [ anon ]
0000557b09da1000    1536 rw--- 0000000000000000 000:00000   [ anon ]
00007fefb919a000  292672 rwx-- 0000000000000000 000:00000   [ anon ]

I feel it is not normal for Nginx to consume that much memory when working as a reverse proxy. The server has 4 vCPUs and 12GB of memory (it was 6GB until last week when it started starving the server and making it unresponsive). But I'm not sure how to proceed from here.

7
  • 2
    show your config with nginx -T.
    – AlexD
    Commented Dec 27, 2021 at 13:23
  • The main suspect is probably mod_security. If you disable it, do you still experience the same memory consumption growth?
    – AlexD
    Commented Dec 27, 2021 at 14:58
  • See this mod_security issue
    – AlexD
    Commented Dec 27, 2021 at 15:00
  • @AlexD I haven't disabled mod_security yet as it's the main reason we have that server running. I'm gonna plan to do that test. And thanks for sharing the memory leak link, I'm reading it now and trying to understand if a new version of mod_security has been released since I installed it.
    – Ronaldo
    Commented Dec 27, 2021 at 15:40
  • 1
    @AlexD it seems my problem was related to Michal256's post from the link you provided. I was loading modsecurity for every server. I've changed it to the http level and it appears to have fixed it. I'm gonna observe for the rest of the week and if it confirms to be the solution, I'll tell you so that you can write an answer to be accepted. Thanks!
    – Ronaldo
    Commented Dec 27, 2021 at 20:03

1 Answer 1

0

It seems my problem was related to Michal256's post from the ModSecurity memory leak link provided by @AlexD. I was loading modsecurity for every server context. I've changed it to the http level and it has fixed it.

The advice about modsecurity proved to be right. The server hasn't had any memory problems since this change. The only change I had to make was to remove several modsecurity on; and modsecurity_rules_file from server context and add it only once at the http context.

Thanks, @AlexD!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .