Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running a Debian server and a couple of days ago my rsyslog started to behave very weird, the daemon is running but it doesn't seem to do anything. Many people use the system but I'm the only one with (legal) root access.

I'm using the default rsyslogd configuration (if you think is relevant I'll attach it, but it's the one that comes with the package).

After I rotated all the log files, they have remained empty:

# ls -l /var/log/*.log
-rw-r--r-- 1 root root 0 Jun 27 00:25 /var/log/alternatives.log
-rw-r----- 1 root adm  0 Jun 26 13:03 /var/log/auth.log
-rw-r----- 1 root adm  0 Jun 26 13:03 /var/log/daemon.log
-rw-r--r-- 1 root root 0 Jun 27 00:25 /var/log/dpkg.log
-rw-r----- 1 root adm  0 Jun 26 13:03 /var/log/kern.log
-rw-r----- 1 root adm  0 Jun 26 13:03 /var/log/lpr.log
-rw-r----- 1 root adm  0 Jun 26 13:03 /var/log/mail.log
-rw-r----- 1 root adm  0 Jun 26 13:03 /var/log/user.log

Any try to force a log writing does not have any effect:

# logger hey
# ls -l /var/log/messages 
-rw-r----- 1 root adm 0 Jun 26 13:03 /var/log/messages

Lsof shows that rsyslogd does not have any log files opened:

# lsof -p 1855
COMMAND   PID USER   FD   TYPE     DEVICE SIZE/OFF       NODE NAME
rsyslogd 1855 root  cwd    DIR      202,0     4096          2 /
rsyslogd 1855 root  rtd    DIR      202,0     4096          2 /
rsyslogd 1855 root  txt    REG      202,0   342076      21649 /usr/sbin/rsyslogd
rsyslogd 1855 root  mem    REG      202,0    38556      32153 /lib/i386-linux-gnu/i686/cmov/libnss_nis-2.13.so
rsyslogd 1855 root  mem    REG      202,0    79728      32165 /lib/i386-linux-gnu/i686/cmov/libnsl-2.13.so
rsyslogd 1855 root  mem    REG      202,0    26456      32163 /lib/i386-linux-gnu/i686/cmov/libnss_compat-2.13.so
rsyslogd 1855 root  mem    REG      202,0   297500    1061058 /usr/lib/rsyslog/imuxsock.so
rsyslogd 1855 root  mem    REG      202,0    42628      32170 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
rsyslogd 1855 root  mem    REG      202,0    22784    1061106 /usr/lib/rsyslog/imklog.so
rsyslogd 1855 root  mem    REG      202,0  1401000      32169 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
rsyslogd 1855 root  mem    REG      202,0    30684      32175 /lib/i386-linux-gnu/i686/cmov/librt-2.13.so
rsyslogd 1855 root  mem    REG      202,0     9844      32157 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
rsyslogd 1855 root  mem    REG      202,0   117009      32154 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
rsyslogd 1855 root  mem    REG      202,0    79980      17746 /usr/lib/libz.so.1.2.3.4
rsyslogd 1855 root  mem    REG      202,0    18836    1061094 /usr/lib/rsyslog/lmnet.so
rsyslogd 1855 root  mem    REG      202,0   117960      31845 /lib/i386-linux-gnu/ld-2.13.so
rsyslogd 1855 root    0u  unix 0xebe8e800      0t0        640 /dev/log
rsyslogd 1855 root    3u  FIFO        0,5      0t0       2474 /dev/xconsole
rsyslogd 1855 root    4u  unix 0xebe8e400      0t0        645 /var/spool/postfix/dev/log
rsyslogd 1855 root    5r   REG        0,3        0 4026532176 /proc/kmsg

I was so frustrated that even reinstall the rsyslog package, but it still refuses to log anything:

# apt-get remove --purge rsyslog
# apt-get install rsyslog

I thought someone had hacked the system, so run rkhunter, chkrootkit, unhide in an attempt to find hide processes / ports and nmap in a remote host to compare with the ports shown by netstat. And I know this doesn't mean anything, but all looks ok. The system also have an iptables firewall that is very restrictive with incoming / outgoing connections.

This is driving me crazy, any idea what is going on here?

[EDIT - disk space info]

# df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                 24G   22G  629M  98% /
/dev/root              24G   22G  629M  98% /
devtmpfs               10M  112K  9.9M   2% /dev
tmpfs                  76M   48K   76M   1% /run
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                 151M   40K  151M   1% /tmp
tmpfs                 151M     0  151M   0% /run/shm

[EDIT - strace info]

Strace looks ok for me

[pid 28824] access("/var/log/auth.log", F_OK) = 0
[pid 28824] access("/var/log/syslog", F_OK) = 0
[pid 28824] access("/var/log/daemon.log", F_OK) = 0
[pid 28824] access("/var/log/kern.log", F_OK) = 0
[pid 28824] access("/var/log/lpr.log", F_OK) = 0
[pid 28824] access("/var/log/mail.log", F_OK) = 0
[pid 28824] access("/var/log/user.log", F_OK) = 0
[pid 28824] access("/var/log/mail.info", F_OK) = 0
[pid 28824] access("/var/log/mail.warn", F_OK) = 0
[pid 28824] access("/var/log/mail.err", F_OK) = 0
[pid 28824] access("/var/log/news/news.crit", F_OK) = 0
[pid 28824] access("/var/log/news/news.err", F_OK) = 0
[pid 28824] access("/var/log/news/news.notice", F_OK) = 0
[pid 28824] access("/var/log/debug", F_OK) = 0
[pid 28824] access("/var/log/messages", F_OK) = 0

The complete strace log can be downloaded from here

share|improve this question
2  
Is the log disk full? –  Jenny D Jun 27 '13 at 10:03
    
Oh sorry, I forgot to add that info, I'll update the question. But there's still enough space to write logs. –  Victor Henriquez Jun 27 '13 at 10:09
    
try and strace -p <pid> or start rsyslog manually under strace and check if it is complaining about something –  jijix Jun 27 '13 at 10:45
    
Good advice, unfortunately I haven't been able to find anything relevant. I've updated the question with the strace log, just in case you can find something I'm missing. I'm really frustrated. –  Victor Henriquez Jun 27 '13 at 11:16
    
run manually in debug mode (-d if iirc), so it won't fork, or use strace's follow forks option. My bad, sorry. –  jijix Jun 27 '13 at 11:41
show 4 more comments

1 Answer

up vote 1 down vote accepted

Most probably it's a file ownership problem. rsyslog starts running as root but then drops privileges and runs as user syslog (configuration directive $PrivDropToUser).

syslog files (auth.log, daemon.log, etc.) initially are owned by syslog:adm but if you change ownership to root (as it seems from your file list) then no matter if you HUP (i.e., reload) rsyslog or restart it, that it will be denied to open those files due to the lack of privileges.

If the change of ownership happened after log rotation, then check the create option of your logrotate configuration. Either configure it like create 0644 syslog adm in /etc/logrotate.d/rsyslog or even better, define it globally at /etc/logrotate.conf omitting the mode, owner and group, simply like this create (which is the default configuration by the way), in which case the same values of the file will be used. Consult man logrotate for the full details.

Some versions of rsyslog include a directive $omfileForceChown as a workaround for the external change of file ownership, but it's not recommended. The recommended way is to configure properly the ownership and permissions. Further information about this issue may be found following that link.

share|improve this answer
    
This. For more discussion and detail, see the rsyslogd bug on launchpad: bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/940030 –  JustinC Aug 27 '13 at 21:34
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.