Reaching out for any other ideas after banging my head against this problem for a week.
We have approximately 100 identical systems using the AAEON PICO-ITX BT01 motherboard with J1900 Celeron processor.
Every system runs on Debian Jessie with kernel 3.16.0-0-686. Each system is imaged using the same clonezilla image in the same manner.
We're experiencing an intermittent failure mode that manifests in one of three ways (although I believe all 3 to be the same root cause)
1) At the very end of the BIOS splash, it freezes and does not recover. While it shows error code 99, this code is always displayed in the instant before the kernel boots so my feeling is that this BIOS code is not diagnostic (it's just the last thing on the screen). Disabling the bios splash shows nothing useful, just the bios version and the error code.
2) In the very initial stages of the kernel boot it reports that CPU cores cannot be woken up. The system then hangs and does not recover.
3) Immediately after the BIOS splash, screen output ceases and the system hangs and does not recover.
This does NOT happen with every board, although they are all from the same production run and use the same hardware (having said that, we've swapped SSDs and seen the same issue, so I don't believe it's the SSD module).
After seeing this crop up in the field, I instituted a testing procedure where a system would be imaged and then a crontask set up to reboot 60 seconds after boot. We'd burn in systems this way and since they did not recover upon failure, after 24h we'd see which systems were still rebooting and which did not pass the test.
I am asking here to see if anyone has any other ideas, essentially. I've been in constant contact with the board manufacturer and they have two affected systems that they're testing, no results yet. I can run any test necessary on the systems I have here, both the boards that have passed and those that have failed.
There is one more major thing. A hard reboot (a power cut) ALWAYS allows the system to boot. A system identified as faulty will fail to boot in MOST CASES during the first soft reboot after power is reapplied to the mainboard. I have only ever seen this failure mode during a soft reboot.
It's been quite the mystery and aside from this I love the hardware and would love to keep buying it.
Thanks guys and gals.