4

Reaching out for any other ideas after banging my head against this problem for a week.

We have approximately 100 identical systems using the AAEON PICO-ITX BT01 motherboard with J1900 Celeron processor.

Every system runs on Debian Jessie with kernel 3.16.0-0-686. Each system is imaged using the same clonezilla image in the same manner.

We're experiencing an intermittent failure mode that manifests in one of three ways (although I believe all 3 to be the same root cause)

1) At the very end of the BIOS splash, it freezes and does not recover. While it shows error code 99, this code is always displayed in the instant before the kernel boots so my feeling is that this BIOS code is not diagnostic (it's just the last thing on the screen). Disabling the bios splash shows nothing useful, just the bios version and the error code.

http://imgur.com/ifse045

2) In the very initial stages of the kernel boot it reports that CPU cores cannot be woken up. The system then hangs and does not recover.

http://imgur.com/hZdzq74

3) Immediately after the BIOS splash, screen output ceases and the system hangs and does not recover.

This does NOT happen with every board, although they are all from the same production run and use the same hardware (having said that, we've swapped SSDs and seen the same issue, so I don't believe it's the SSD module).

After seeing this crop up in the field, I instituted a testing procedure where a system would be imaged and then a crontask set up to reboot 60 seconds after boot. We'd burn in systems this way and since they did not recover upon failure, after 24h we'd see which systems were still rebooting and which did not pass the test.

I am asking here to see if anyone has any other ideas, essentially. I've been in constant contact with the board manufacturer and they have two affected systems that they're testing, no results yet. I can run any test necessary on the systems I have here, both the boards that have passed and those that have failed.

There is one more major thing. A hard reboot (a power cut) ALWAYS allows the system to boot. A system identified as faulty will fail to boot in MOST CASES during the first soft reboot after power is reapplied to the mainboard. I have only ever seen this failure mode during a soft reboot.

It's been quite the mystery and aside from this I love the hardware and would love to keep buying it.

Thanks guys and gals.

  • 1
    I had a somewhat similar issue in the past. The problem turned out being a bug in the motherboard firmware. The issue was that the systems would hang at the end of POST after a reboot. They would only boot properly if powered down and back on. Having a specific RAID controller in the system is what triggered the bug in my case. You should be working with the manufacturer or vendor to solve this since more sales to you are dependent on it working. – yoonix May 13 '16 at 22:49
  • Definitely in close contact with the manufacturer, they have their techs in California working on it apparently. Basically my goal was to get my ducks in a row as much as possible to avoid the answer being "it's an OS problem" or something similar. It SEEMS to be happening before the OS even begins to load. It's encouraging to see someone else with the same issue, because it's almost exactly as you describe. – Dave May 14 '16 at 4:00
0

The cause of this fault was found working with the manufacturer, it turned out to be a faulty BIOS revision. We both tested rolling back the BIOS and the problem disappeared.

That should have been an early diagnosis step, but live and learn!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.