1

Nearly every time syslog-ng writes to the current log file, it touches the log file from 6 days prior. It is not writing any data into the old log files, just updating their last modified timestamp. Screenshot of log files

I’m not 100% sure why it stopped updating the last modified timestamp of the files from the 26th and 27th just before noon the last two days, but I think that may just be coincidence. I restarted the syslog-ng service and the VMware host’s syslog service around that time the last two days and that may have stopped the touching of old files. That being said, I restarted the services again this morning, and it is still happening.

We are currently capturing logs from 24 VMware hosts. Each host’s log files are stored in their own directory with a new file each day. This only happens to the log files from one host. All of the host names are very similar (group1-esx01…08, group2-esx01…08, group3-esx01…08), so it doesn’t seem like a filtering issue explains why it only happens to the logs located in one folder.

We use the default syslog-ng.conf file and put our configurations in a single file located in the conf.d directory. That file looks like this:

#Global network listener
source s_network {
    network(
        ip("1.1.1.1")
        port(1514)
        max-connections(100)
        transport("tls")
        tls(
            key_file("/etc/syslog-ng/cert/PrivateKey.pem")
            cert_file("/etc/syslog-ng/cert/PublicKey.pem")
            peer_verify(optional-untrusted)
        )
    );
};
#VMware
#Dir for each host, file for each day.
destination d_vmware {
    file(
        "/var/log/vmware/$HOST/$YEAR$MONTH$DAY.log"
        perm(0644)
        create_dirs(yes)
    );
};

#Only get data from machines who's hostname starts with the prefixes of the various blade groups.
filter f_vmware {
    host("group1-esx*" type(glob)) or
    host("group2-esx*" type(glob)) or
    host("group3-esx*" type(glob));
};

#From global listener to VMware dest
log {
    source(s_network);
    filter(f_vmware);
    destination(d_vmware);
};

I figure finding someone who has experienced this before is quite the long shot. However, I'm hoping that someone at least has an idea of where to start.

Thanks for your time.

Edit: Quick update, at 12:12pm the 20170728.log file stopped having its last modified timestamp updated and the 20170729.log file starting having its last modified timestamp updated.

Since then, I set 20170729.log to be immutable and I get 'permission denied' when attempting to touch the file as root, as expected. However, it is still having its last modified time updated! I'm not seeing syslog-ng report any errors either. Strange stuff.

Edit2: I found the solution! By default, syslog-ng uses the timestamp in the messages it receives as the timestamp of the log entry, which makes sense. So, there was a good chance the issue was with the host sending the log messages to the syslog-ng server. I think it was sending an empty message with the wrong date in the timestamp on a regular basis. Anyway, instead of diving deep into everything that can generate log messages in ESXi, I just migrated the VMs off the affected host and rebooted it. That seems to have solved the issue for now, and that is good enough for me since we will be replacing this set of servers in the near future.

0

You must log in to answer this question.

Browse other questions tagged .