Monit won't run

Question

I have two identical EC2 instances (the second is a replica of the first), running Gentoo. The first instance has monit running which monitors a single process and some system resources and functions great.

In the second instance, monit runs but quits right away. The configuration is similar on both instances so are the versions of monit.

monit.log shows:

[GMT Oct  3 08:36:41] info     : monit daemon with PID 5 awakened

Final lines on strace monit show:

write(2, "monit daemon with PID 5 awakened"..., 33monit daemon with PID 5 awakened ) = 33
time(NULL)                              = 1349252827
open("/etc/localtime", O_RDONLY)        = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb773a000
read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0"..., 4096) = 118
_llseek(4, -6, [112], SEEK_CUR)         = 0
read(4, "\nGMT0\n", 4096)               = 6
close(4)                                = 0
munmap(0xb773a000, 4096)                = 0
write(3, "[GMT Oct  3 08:27:07] info     :"..., 33) = 33
write(3, "monit daemon with PID 5 awakened"..., 33) = 33
waitpid(-1, NULL, WNOHANG)              = -1 ECHILD (No child processes)
close(3)                                = 0
exit_group(0)                           = ?

No core dumps (ulimit -c shows unlimited)

monit -v shows:

monit: Debug: Adding host allow 'localhost'
monit: Debug: Skipping redundant host 'localhost'
monit: Debug: Skipping redundant host 'localhost'
monit: Debug: Adding credentials for user 'xxxx'.
Runtime constants:
 Control file       = /etc/monitrc
 Log file           = /var/log/monit/monit.log
 Pid file           = /var/run/monit.pid
 Id file            = /var/run/monit.pid
 Debug              = True
 Log                = True
 Use syslog         = False
 Is Daemon          = True
 Use process engine = True
 Poll time          = 30 seconds with start delay 0 seconds
 Expect buffer      = 256 bytes
 Event queue        = base directory /var/monit with 100 slots
 Mail server(s)     = xx.xxx.xx.xxx with timeout 30 seconds
 Mail from          = (not defined)
 Mail subject       = (not defined)
 Mail message       = (not defined)
 Start monit httpd  = True
 httpd bind address = Any/All
 httpd portnumber   = 2812
 httpd signature    = True
 Use ssl encryption = False
 httpd auth. style  = Basic Authentication and Host/Net allow list
 Alert mail to      = [email protected]
   Alert on         = All events

The service list contains the following entries:

System Name           = xxxx
 Monitoring mode      = active
 CPU wait limit       = if greater than 20.0% 1 times within 1 cycle(s) then alert else if succeeded 1 times within 1 cycle(s) then alert
 CPU system limit     = if greater than 30.0% 1 times within 1 cycle(s) then alert else if succeeded 1 times within 1 cycle(s) then alert
 CPU user limit       = if greater than 70.0% 1 times within 1 cycle(s) then alert else if succeeded 1 times within 1 cycle(s) then alert
 Swap usage limit     = if greater than 25.0% 1 times within 1 cycle(s) then alert else if succeeded 1 times within 1 cycle(s) then alert
 Memory usage limit   = if greater than 75.0% 1 times within 1 cycle(s) then alert else if succeeded 1 times within 1 cycle(s) then alert
 Load avg. (5min)     = if greater than 2.0 1 times within 1 cycle(s) then alert else if succeeded 1 times within 1 cycle(s) then alert
 Load avg. (1min)     = if greater than 4.0 1 times within 1 cycle(s) then alert else if succeeded 1 times within 1 cycle(s) then alert

Process Name          = xxxx
 Group                = server
 Pid file             = /var/run/xxxx.pid
 Monitoring mode      = active
 Start program        = '/etc/init.d/xxxx restart' timeout 20 second(s)
 Stop program         = '/etc/init.d/xxxx stop' timeout 30 second(s)
 Existence            = if does not exist 1 times within 1 cycle(s) then restart else if succeeded 1 times within 1 cycle(s) then alert
 Pid                  = if changed 1 times within 1 cycle(s) then alert
 Ppid                 = if changed 1 times within 1 cycle(s) then alert
 Timeout              = If restarted 3 times within 5 cycle(s) then unmonitor
 Alert mail to        = [email protected]
   Alert on           = All events
 Alert mail to        = [email protected]
   Alert on           = All events

-------------------------------------------------------------------------------
monit daemon with PID 5 awakened

Ran emerge --sync before emerge -va monit which installed monit v5.3.2. When that didn't work i've downloaded v5.5 from their website and compiled from source which did not work either.

Answer 1

Found out what the problem was:

The file /var/run/monit.pid contained some strange information, it looked like an md5 hash for some reason.

Once i've removed this file and restarted monit, everything worked fine. What alerted my suspicion was the line monit daemon with PID 5 awakened which was very strange because a newly run process should have a much higher PID so i went checking in the .pid file.

Answer 2

I have loaded Monit on an EBS-backed EC2 instance (using AWS's standard Linux build).

Monit was unreliable for me when starting / restarting due to a configuration error on my part. In the configuration file /etc/monit.conf I had the following line, which was incorrect:

set idfile /var/run/monit.pid

That should have been

set idfile /var/run/.monit.id

Someone out there might find this helpful in the future.