Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am having some problems with a Dell 1950 server. I am installing RHEL 4.6 along with Oracle and some other software on here.

I am randomly getting an error message saying "kernel: journal commit I/O error" on my ssh session and on the monitor I have hooked up to the server I see an error scrolling by that says "EXT3-fs error (device sda5) in start_transaction: Journal has aborted."

It has happened several times but never at the same point during the install. Actually, this last time the system was up and running and I was just trying to import a database into oracle.

This has happened on several hard drives, so I'm pretty sure that is not the problem. This makes me think the raid controller is going bad.

What do you guys think?

** UPDATE **

Pretty sure it was a bad hard drive. I threw another drive in the server and it's been running for about 48 hours with out problems.

share|improve this question

4 Answers 4

up vote 8 down vote accepted

I've seen those errors before, but not during the install process.

It means that the drive got enough errors that the OS took it to read-only mode. If you could find the full logs, there'd probably be some I/O errors that retried and worked before the full-on failure errors you saw. Something with actual blocks mentioned.

It's a storage system error. It's definitely the RAID card, the drives in the RAID array, the cables from the card to the drives, the backplane the drives connect to, the slot the raid card is plugged into, the power supply for the hard drives, or something else in between the CPU and the actual storage blocks.

share|improve this answer

Three possibilities come to mind:

  1. There's memory problems (they often cause "random" crashes). If you have ECC ram in there, then obviously it's less likely.

  2. There some problem with the Bus. I've had the same problem with a broken APIC controller on a Tyan dual Opteron motherboard a few years back. There were other log entries that hinted at it, but the bulk of the symptoms were random corruption on disk drives with automatic read-only remounts. In my case I knew it wasn't disk related because it was an external FC RAID box and it was fine.

  3. The RAID controller is bunk.

This is in the order I'd consider the problems.

share|improve this answer
    
Probably not memory problems; those would be more likely to cause segfaults and more random errors, not be restricted to just the storage. –  freiheit Jun 10 '09 at 20:23
    
True. But in an install or early boot situation, the bulk the memory usage is the buffer-cache so the problems tend to appear there first. Once the machine's been running some load for a while user process dominate memory I/O and so the segfault's prevalence. That being said, a PE1950 should have Xeon processors and ECC ram so the RAM should be able to detect it and report it to Linux. –  Alexandre Carmel-Veilleux Jun 10 '09 at 20:34

It could be the RAID controller going bad like you said (try a spare if you have one.) It could be the driver for the controller (check for alternative drivers if available, even if performance is worse, it's good to have a reference point.) It could be the kernel (less likely though in RHEL, it's quite well tested.) It could be bad RAM messing up the block cache.

A hardware problem is the most likely cause, though, based on the seemingly random error behavior.

share|improve this answer

try:

shutdown -rF now

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.