9

I am having some problems with a Dell 1950 server. I am installing RHEL 4.6 along with Oracle and some other software on here.

I am randomly getting an error message saying "kernel: journal commit I/O error" on my ssh session and on the monitor I have hooked up to the server I see an error scrolling by that says "EXT3-fs error (device sda5) in start_transaction: Journal has aborted."

It has happened several times but never at the same point during the install. Actually, this last time the system was up and running and I was just trying to import a database into oracle.

This has happened on several hard drives, so I'm pretty sure that is not the problem. This makes me think the raid controller is going bad.

What do you guys think?

** UPDATE **

Pretty sure it was a bad hard drive. I threw another drive in the server and it's been running for about 48 hours with out problems.

5 Answers 5

9

I've seen those errors before, but not during the install process.

It means that the drive got enough errors that the OS took it to read-only mode. If you could find the full logs, there'd probably be some I/O errors that retried and worked before the full-on failure errors you saw. Something with actual blocks mentioned.

It's a storage system error. It's definitely the RAID card, the drives in the RAID array, the cables from the card to the drives, the backplane the drives connect to, the slot the raid card is plugged into, the power supply for the hard drives, or something else in between the CPU and the actual storage blocks.

2

Three possibilities come to mind:

  1. There's memory problems (they often cause "random" crashes). If you have ECC ram in there, then obviously it's less likely.

  2. There some problem with the Bus. I've had the same problem with a broken APIC controller on a Tyan dual Opteron motherboard a few years back. There were other log entries that hinted at it, but the bulk of the symptoms were random corruption on disk drives with automatic read-only remounts. In my case I knew it wasn't disk related because it was an external FC RAID box and it was fine.

  3. The RAID controller is bunk.

This is in the order I'd consider the problems.

2
  • Probably not memory problems; those would be more likely to cause segfaults and more random errors, not be restricted to just the storage.
    – freiheit
    Jun 10, 2009 at 20:23
  • True. But in an install or early boot situation, the bulk the memory usage is the buffer-cache so the problems tend to appear there first. Once the machine's been running some load for a while user process dominate memory I/O and so the segfault's prevalence. That being said, a PE1950 should have Xeon processors and ECC ram so the RAM should be able to detect it and report it to Linux. Jun 10, 2009 at 20:34
2

It could be the RAID controller going bad like you said (try a spare if you have one.) It could be the driver for the controller (check for alternative drivers if available, even if performance is worse, it's good to have a reference point.) It could be the kernel (less likely though in RHEL, it's quite well tested.) It could be bad RAM messing up the block cache.

A hardware problem is the most likely cause, though, based on the seemingly random error behavior.

2

Check that the disk isn't full - in particular the root partition. Use df to see the files system disk usage:

df -h

Look for partitions near or equal to 100% utilization

-5

try:

shutdown -rF now

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .