Syslog-ng and Logrotate creating sparse files

Question

We've been using syslog-ng to log our postgres logs. On top of it we run logrotate every 5 minutes which also gzips the files. Recently, we've noted that these postgres logs contain a lot of null characters at the beginning of the file. We later realized that the space occupied by the null characters are equal to the previous size of the file.

Researching (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=733856) on why sparsing has been taking place we realized that the problem lies with syslog-ng not being able to write the logs from the beginning. We tried writing a postrotate script which would kill -HUP syslog-ng. We tried that, but in vain. Many have sorted out this problem by setting the o_append bit. Does anyone know how to set the o_append bit for syslog-ng so that it does not write from the last maintained write head and starts from the beginning of the file?

Answer 1

Assuming you're talking about /var/log/postgresql/postgresql--main.log: syslog is not your point of control. As wurtel points out, these logs do not come via syslog, but are the stdout/stderr of the postgres process

This is why the standard logrotate for postgres has to be a copytruncate, i.e. copy the file then truncate it to zero bytes. The redirected output will then start writing from the beginning of the file again (at least that's what happen on a standard out-of-the-box system).

Oddly though, if your logrotate isn't copytruncate, then I would expect that after rotate the output would go to the .1 file. Can you share the logrotate config you're using please? I suspect there's a clue in there somewhere.