Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Trying to get at the root of why a server went down for 20 minutes, I looked at the syslog for that time period and saw this:

Jan  3 07:50:01 tools CRON[17085]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Jan  3 07:55:01 tools CRON[17773]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Jan  3 07:55:01 tools CRON[17774]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Jan  3 08:19:44 tools kernel: imklog 4.2.0, log source = /proc/kmsg started.
Jan  3 08:19:44 tools rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="470" x-info="http://www.rsyslog.com"] (re)start

EC2 shows this CPU utilization during the time of the crash (in the middle, before 13:00 UTC).

EC2 monitoring details CPU usage

So there is nothing even visible during that time period. None of our other EC2 instances crashed and I can find no evidence that it was an application error. In fact, this happened on our tools server (apache, mongodb, and redis). Monit was also running, but there are no suspicious logs at the time of the crash.

What could have caused this crash and what do the ^@'s mean in the syslog?

share|improve this question
    
Looks like binary output. –  Tim Jan 3 '12 at 16:42
4  
This ^@ is just a funky representation of binary 0. It is used in some viewers/editors. No idea why string of zeroes appeared in your syslog, not an usual thing. –  kubanczyk Jan 3 '12 at 17:28
6  
^@^ They're bats. Your server is haunted. –  Tom O'Connor Jan 3 '12 at 18:01
    
I'm gonna blame the Chaos Monkey. –  Shane Madden Jan 4 '12 at 2:17

1 Answer 1

up vote 1 down vote accepted

The answer came on the AWS forum: https://forums.aws.amazon.com/message.jspa?messageID=308434

Reed,

There was a problem with the underlying hardware that caused the underlying system to crash. The artifact of the syslog lines, could be just a representation that the system was in mid write when it crashed, or the lines could have come from the the cron process that was running on the previous line.

Nathan

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.