13

We have a CentOS OS that became unresponsive this morning to external network traffic. It is a virtual machine. I was able to reboot the VM. After logging back in, I found the following in the /var/log/messages file, repeating over and over, up to the point of the reboot:

Jan 21 06:53:01 PBX kernel: audit: backlog limit exceeded
Jan 21 06:53:01 PBX kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Jan 21 06:54:01 PBX kernel: printk: 8 messages suppressed.
Jan 21 06:54:01 PBX kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Jan 21 06:54:01 PBX kernel: audit: audit_lost=1130 audit_rate_limit=0 audit_backlog_limit=320

I read on another forum that the following command could identify the source of the backlog traffic:

[root@PBX log]# aureport --start today --event --summary -i

Event Summary Report
======================
total  type
======================
486  USER_ACCT
486  CRED_ACQ
486  USER_START
485  LOGIN
477  CRED_DISP
477  USER_END
6  USER_LOGIN
3  USER_AUTH
2  CONFIG_CHANGE
2  CRED_REFR
1  DAEMON_START

Can anyone advise me as to what next steps I should take in preventing this issue from happening again? I'm not particularly familiar with the purpose of the backlog or what the output of the event summary report means.

Improve this question
2
  • Can you rule out a storage problem? Logs are not written if the storage is inaccessible, but the kernel remains running - at least for a while.
    – the-wabbit
    Jan 23, 2012 at 1:22
  • Storage is local and hasn't showed any signs of trouble. I think it's more likely that useful info is not being logged.
    – YWCA Hello
    Jan 26, 2012 at 5:24

2 Answers 2

Reset to default
5

You can increase the backlog by modifying -b 320 in /etc/audit/audit.rules to something larger and see if it has any effect, but these amounts you show us still very few audit results, so I doubt the audit error has anything much to do with the system freezing in itself. Its probably just a sympthom of something else happening.

Check /var/log/audit/audit.log to see what events have been logged to see if they can be of any use to your debugging.

Improve this answer
2
  • audit.log essentially went quiet about 2 hours before we noticed the problem (this happened in the early morning). Then, messages started up again with the reboot. I'm hoping this isn't another linux freeze scenario where no real answers are found from the logs :/
    – YWCA Hello
    Jan 21, 2012 at 19:00
  • Note that on RHEL7-based system, you need to change the file /etc/audit/rules.d/audit.rules because the /etc/audit/audit.rules gets rewritten at restart of auditd.
    – Bruno Mairlot
    Jun 10, 2015 at 8:07
4

There is multiple solution:

  1. To lengthen the backlog, add or edit /etc/audit/audit.rules by adding or editing "-b 320" to "-b 8192".
  2. change the priority by editing priority_boost from 3 to 4 or 5 in /etc/audit/auditd.conf.

To find out about what problem cause this issue, run aureport --start today or aureport --start today --event --summary -i

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .