We have several 2012 Mac minis in our server room that reliably disconnect from 1GbE network (but otherwise appear to still be working) when the adjacent servers are under heavy CPU load.
We suspect RFI / EMI but can't be 100% sure as we don't yet have a good way to measure it and isolate the source. Other servers (over 50, all kinds including a bunch of xserves) in the same room are fine. Taken out of the server room, Mac Minis are fine.
The weird parts:
- the disconnects started happening after we added three new servers (render nodes) to an existing cluster of three, pretty much pinpointing them as the source of the issue
- no issues when those servers are idle
- issue seems to start when all six are under 50% or higher CPU load
- all other devices in the server room - lots of them, all kinds - have no issues; only the Mac minis are affected
What would you recommend to troubleshoot and address the issue? (Outside of powering down those new render nodes - which we do need and which don't seem to affect anything else.)
What we've tried:
- put in a known good 2012 Mac mini and observe the same issue (we don't have Mac minis others than 2012)
- move Mac minis around the server room; location within the room doesn't seem to matter
- collect CPU utilization data from render nodes and other devices; response times and timeouts from Mac minis and other network devices using Solarwinds NPM; observe strong correlation between CPU load spikes on render nodes and network issues on Mac minis
P.S. What we haven't done yet:
- to check for a possibility that it's just one server misbehaving, disconnect servers one at a time while putting remaining ones under 100% CPU load
- research RFI / EMI measurement and isolation equipment and services
- attempt to ground the Mac minis (their power cables are 2-prong, aren't grounded) or otherwise shield them from RFI / EMI
We do need the Mac minis as we're running out of Xserves, and certain apps only run on Macs.
P.P.S. Apologies if this is off-topic and/or in the wrong forum.
Thanks for any ideas!