General tips for interpreting error logs [closed]

Question

Reading log files can be pretty frustrating as, by nature, their content says as much about the developer who penned them as the problem itself.

Do you have any general purpose tips for interpreting error logs (eg: "google is your friend" or "some error codes occur more than others" or "remember that warnings and errors are very different")?

Answer 1

Let developers troubleshoot production issues once in a while. This will do wonders for your logging. :)

Answer 2

About a specific common situation when you have all of these at the same time: (1) a problem in a distributed environment (2) a huge pile of debug info scattered over co-operating servers and different logfiles (3) no documentation for interpreting the logs (4) nothing on google (5) no clue (6) ping-pong players instead of vendor's support.

• First of all, make sure that the time is synchronized in the entire environment (ntp). If it is not, forget about trying to find out inter-host relationships from their log files.
• Do not pick up a random "error" from a random log to blame. Read the log chronologically, remembering that "error" line may be as well result of normal software operation and always been there.
• Compare logs from proper operation to the logs from problem situation. At what point they cease to match? (vimdiff might be useful)
• If during test cases you have the functionality to insert your own custom log messages, use it. (like logger in syslog)
• On analysis, if you catch yourself switching between many huge logs back and forth, trying to catch the flow of action - try to merge the logs. (Use sed to place time on first column. Use cat+sort to merge multiple files. And of course grep -viE for filtering unnecessary lines.)

Answer 3

My habit with server logs is: review them regularly, and investigate/resolve the issues I find. I do this proactively - not waiting until the users are howling about a system outage. The main reason this is effective, really boils down to a couple of old sayings:

A stitch in time saves nine. Obviously if you're solving issues while they're small, you're ahead of the curve, and users/management will have less reasons to yell at you; that's a good thing.

Practice makes perfect. I think this is the greater advantage to the sysadmin. By getting in there regularly and proactively reading the logs, you're gaining experience and familiarity. You're learning what those cryptic log messages mean - and which are trivial, and which are a big deal. The process of investigating messages you don't immediately understand (which will be a lot of them at first!) teaches you a lot about the internals of the OS and the apps running on it.

Usually when I get a new system to manage, it will have quite a few errors in the log, many which recur fairly regularly. The prior admin often shrugs them off with something to the effect of "not real sure what that's about, but the users never complained, so I didn't consider it broken enough to fix!"

My goal with such systems is to revisit the logs weekly until I have solved or understood every new error that comes up; then relax my log reviews to monthly. Clean logs are easier to read!

Answer 4

A good program supports logging levels. And usually logs are worthless without timestamps.

Most linux distributions come with a logwatch tool; learn to use it and configure it's ignore settings. The trick is to set the pain threshold appropriately, so that nothing critical is ignored, but not so spammy that administators write mail rules to file and ignore logwatch mail.

Answer 5

I don't believe any general purpose tips can be made to interpret error logs, except that you must research each error on a case-by-case basis, e.g. with Google or by reading source, to understand it.

For handling something like syslog, especially when aggregating many machines, a general purpose suggestion can be made. Keep a list of patterns to ignore, and a list of patterns to alert on immediately. Generate a daily report that excludes the "ignore" messages. (Or even watch the log file in real-time excluding the ignorable messages). Use this report to add to the ignore list and to the alert list. For patterns identified as real errors, deliver an alert to admins in real-time. Ideally your ignore list should be thorough enough that you can read through the messages that fall through, and your alert list should be simple enough that you can investigate every one that you are alerted on. Be able to handle floods of alerts from a broken system that you can not fix right away. It is worthwhile to keep two additional levels of patterns - those worth reviewing but that are not likely to be a problem, and those worth alerting on but not disrupting someone.

Failing to do this in a Unix environment is probable the single most significant (costly and damaging) commonly-made oversight.

Answer 6

Consult the documentation about the log files that the developers handed over along with the application.

What? There's no documentation? Time for an AttitudeAdjustmentTool

More seriously, documenting log files and how to interpret them needs to be make one of the developers' tasks. Their job isn't done when the code is, it's done when the operations people can run the application and keep it running, and that means documentation, handover meetings, designing for manageability etc.

Answer 7

Don't make assumptions about log files.

Field formats need to be checked. For example: are dates dd/mm/yy or mm/dd/yy?; are numeric fields decimal, hex, octal or something else? Are timestamps consistent (others have mentioned the importance of syncing time between devices: check it has been sync'd or work out what the source of a timestamp would be and correct it)?

Are all devices/processes logging at the same log level and to where you would expect them to?

Is logging consistent between different revisions of the same software? (checking that log outputs are consistent with previous versions and with the documentation should be on the list for testing new software revisions but can be overlooked)