I have an open alert in Datadog for a monitor that measures CPU idle on our servers. It reports that one of our servers has hit 0% CPU idle. But when I run top on the server in question, it shows load averages of close to 0, 100% CPU idle (or close to), and very few processes using up CPU. Every time I run top, it does show 100% stolen CPU for a second but then that drops to 0% and idle goes back up to 100%.
Looking at the Datadog dashboard for that server, the graph just dives off a cliff at the time of the alert. CPU idle drops to 0, but so does CPU system, CPU user, CPU stolen, etc.--it's as if the agent has stopped reporting CPU entirely, except there's no "no data" alerts or anything like that that I can see.
I've tried restarting the datadog-agent on the server, to no avail. The agent logs show me nothing. NST time seems to be pretty close to sync--about 0.02 seconds ahead. I feel like I can probably ignore this alert for this server, but at best that means we have an alert that does nothing. Is there something obvious I'm missing? (I'm incredibly totally new to this, unfortunately.)