Recently the disk usage on one of my machines has been climbing steadily. Upon investigating, I found that the
directory was full of core fump files from cPanel. about 106GB worth! all the files seemed to be due to 'tailwatchd'
> [/usr/local/cpanel/install]# file core.9049 core.9049: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'tailwatchd'
so i looked at the log files for tailwatchd and found they were due tomemory issues...
> [/usr/local/cpanel/install]# tail -n 10 /usr/local/cpanel/logs/tailwatchd_log Out of memory! Out of memory! Out of memory! Callback called exit, <GEN15> line 35842. END failed--call queue aborted at /usr/local/cpanel/bin/tailwatchd line 9, <GEN15> line 35842.
thus it seems like a memory issue... it's strange because this only started happening out of the blue a day ago. i did install Munin about two weeks ago, but thats the only major change on the box i'm aware of. could that be a culprit? munin shows the constant increase in commited memory from day one of being installed... but the disk usage only started growing a day ago.
keep in mind i'm a programmer that has been forced to wear the system admin hat the last week or so. let me know if more details would help.