4

A couple of hours ago we got a system crash on Ubuntu 12.04. We checked all the log files and there is nothing suspicious to blame to.

Last stuff that was logged was some dovecot activity. There are no kernel panic messages. Nothing.

It is a new server (new hardware) we are testing before production. And because it is new hard, I'm suspicious the problem may be due to some faulty hardware.

We already run memtester with no problem detected. I'll be happy to hear from other hardware testing tools (the machine has SSD).

Anyway, the thing I wanted to ask you is a different one. The strange thing is on every open file at the moment of the crash we found the next sequence of symbols was written into them: "@^@^@^@^@^@^@...".

For example, on the syslog log file we got:

Apr 16 15:53:56 odyssey dovecot: pop3-login: Aborted login (auth failed, 1 attempts): user=<info>, method=PLAIN, rip=46.29.255.73, lip=5.9.58.177
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^ [these continues for about 1000 chars...] ^@^@^@^@Apr 16 15:55:12 odyssey kernel: imklog 5.8.6, log source = /proc/kmsg started.

We got all these symbols in all open files. These include: syslog, mail.log, kern.log, ... But also on some logs that are output by php scripts run in CRONs from user accounts (not root).

So, any idea why all open files got these characters written during the crash? This is pretty bad since the crash corrupted many files (we don't even know which other ones may be affected). We are suspicious that all open files (in write mode maybe) at the moment of the crash got all these symbols inserted. Why is that?

BTW [in case it helps], the system automatically rebooted after the crash but Apache did not start. There were not traces in /var/apache2/*log why apache did not start. After running a "service apache2 start" it started with no problems. Also, we rebooted the machine manually and Apache also started on reboot. But it did not start after the crash and no errors were reported.

Thanks guys!

  • Smells like hardware issues. Bad RAM, motherboard, etc. – Michael Hampton Apr 16 '14 at 16:45
  • Probably. Any tool to test the motherboard? In any case, the strange thing is why we got all these ^@^@^@^@^@^... symbols. Did anyone get something like this before? Is it a symptom of any possible problem? – landabaso Apr 16 '14 at 16:49
  • 1
    I have seen those symptoms, caused by two different root causes. One was on a 2.6 kernel using ext2 file system, and the pattern of NUL bytes suggested ordering of write operations were the reason. The other is on NFS where multiple clients writing to the same file can trigger race conditions leading to corruptions. Which file system are those files on? Which mount options are you using? Is the machine an NFS client and/or server? – kasperd May 27 '14 at 19:17
  • 1
    It's a "/dev/md2 on / type ext4 (rw)" on a "3.11.0-15-generic #25~precise1-Ubuntu SMP". In fact, it is a server machine with SSD disks (no NFS). – landabaso May 29 '14 at 6:27
  • 1
    @Blates It's been 5 years since I was last using NFS. And I have no more information about the specifics of the NFS race conditions leading to this. All I can tell you is that I was experiencing the corruption myself. My ~/.bash_history file was the most frequent victim. – kasperd Mar 7 '16 at 18:00
0

Those ^@ are almost certainly binary zeroes. That is, xxd corruptfile | tail -3 will probably emit something like:

#######0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
#######0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
#######0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

The write was signaled to the kernel, but the contents never got flushed to disk. Therefore, the file was extended, in anticipation of a write, so the file is inadvertently sparse.

This is especially likely if you're not using a journaling filesystem, as a journal should cause the write to rolled back if it wasn't completed properly (as it wasn't, due to the crash).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.