Server crash of GoogleCE, Fluentd (ruby) memory usage high might be related?

Question

I'm running a small server (f1-micro) on Google GCE. This server is acting as a reverse proxy, so only a small NGINX has been installed on it, which just proxies for a Google Cloud Function. Usage is very low, hardly getting traffic.

However, about once 1 month (might be even more often), the server crashes without apparent reason. Looking in the Console, CPU usage is through the roof, SSH access doesn't work, and the only solution is to kill it and restart the server. Often I also see high disk I/O.

Logging on to the server, I noticed quite high memory usage by the fluentd daemon, more specifically the ruby process, output of top below:

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                        
   1123 root      20   0   40.7g   1.3g   1.3g S   0.7 233.1   0:40.79 ruby       

    867 ?        Sl     0:00  \_ containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/aec49f162b96e0a58522a7ccdee023ad01f7abefbf996352bf4fe9e4fc976465 -address /run/containerd/containerd.sock -container
    897 ?        Ssl    0:05  |   \_ /opt/google-fluentd/embedded/bin/ruby /usr/sbin/google-fluentd
   1123 ?        Sl     0:40  |       \_ /opt/google-fluentd/embedded/bin/ruby -Eascii-8bit:ascii-8bit /usr/sbin/google-fluentd --under-supervisor

This is after running the server for about 12 hours. The memory usage is just a wild guess, but might explain disk I/O (swap usage). Any clues to why this is happening?

Kind regards,

Jeroen

Answer 1

Have a look at the documentation:

When installing the Logging agent, a minimum of 250 MiB memory is required, but 1 GiB is recommended.

and because f1-micro instance has only 614Mb of RAM it could be the cause of the problem. I've checked on the test instance and it has very limited amount of free memory just after start.

Google Fluentd is preconfigured by default, but you can try to customize it by following this documentation, for example you can change size of buffers. Also, have a look at this recommendations. In my opinion, in such case it'll be better try to use at least [g1-small][5] instance with more amount of memory or disable google-fluentd service if you don't need it.