5

I'm having an issue where after the server has been up for a period of time (~week/few days) the server will start reading corrupt data. For instance when I run a sha1sum of a file after a fresh boot it remains the same. However after a while I will start to get segfaults and from then on whenever I read this file I get a different sha1sum.

I've checked S.M.A.R.T with long tests and I've run an extended memtest86+(12 passes)

My lspci is as follows:

00:00.0 Host bridge: Advanced Micro Devices [AMD] RS780 Host Bridge
00:01.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge (int gfx)
00:06.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge (PCIE port 2)
00:07.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge (PCIE port 3)
00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
00:12.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller
00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
00:13.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller
00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 3c)
00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller
00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge
00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
00:18.0 Host bridge: Advanced Micro Devices [AMD] K10 [Opteron, Athlon64, Sempron] HyperTransport Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K10 [Opteron, Athlon64, Sempron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K10 [Opteron, Athlon64, Sempron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K10 [Opteron, Athlon64, Sempron] Miscellaneous Control
00:18.4 Host bridge: Advanced Micro Devices [AMD] K10 [Opteron, Athlon64, Sempron] Link Control
01:05.0 VGA compatible controller: ATI Technologies Inc Radeon HD 3300 Graphics
01:05.1 Audio device: ATI Technologies Inc RS780 Azalia controller
02:00.0 Ethernet controller: Atheros Communications Atheros AR8121/AR8113/AR8114 PCI-E Ethernet Controller (rev b0)
03:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. Device 3403

I could really use some help on this, do you have any idea what could cause this? It's really frustrating me as it seems to trigger entirely randomly and will not go away until I reboot. I'm also use KVM for virtualization as well as MD for software RAID on this server and the processor is a Phenom II X4 965. I don't believe it's the software raid however as this affects files also hosted on non-raid partitions so I don't know.

Update 21 Jun 10 Ok, just had the motherboard replaced. Still have the same error. No CPU errors I can find; disks all report fine with smart test. Does anyone have any idea whatsoever what this could be ? I am pulling my hair out over here.

Update 22 Jun 10 So I've checked the logs and tried another filesyste, still the same thing. This is all on the host VM too btw.

3
  • Have you checked your system logs when this happen? dmesg, syslog, messages etc.
    – pauska
    May 31, 2010 at 12:27
  • I have, all I get are a bunch of segfaults from various programs, python in particular seems prone to dying over this. The server is running but in the "screwed up" state right now so if anyone has any ideas for commands to run to test that would be great.
    – Jonathan
    May 31, 2010 at 12:45
  • Just adding this here: it appears this is about Ubuntu 10.04. Also, FWIW, we have a small cluster of poweredges running in software RAID-1 with some uptime. Perhaps that helps to reinforce your point that it's not a software RAID problem.
    – Shtééf
    May 31, 2010 at 13:49

3 Answers 3

1

My instincts tells me this is a hardware problem, possibly thermal related (as it shows up after som run time). Most likely you have a problem with the southbridge or related hardware.

Consider running some extensive transaction tests on the southbridge, or simply replace the motherboard.

That your OS stays stable but you have random IO errors typically rules out the CPU/Memory, as errors in those tend to make the OS crash and burn along with the other software. But the majority of the Kernel is read from disk at the boot and never swapped out, so a Linux system can be surprisingly stable even if it can't read propperly from the disk.

2
  • Hi Perhs, I like the sound of that, what sort of software could I use to run transaction tests on the southbridge? I'm going to see about RMA'ing the motherboard; thanks for your help.
    – Jonathan
    May 31, 2010 at 13:43
  • ltp.sourceforge.net/tooltable.php is a starting place to find suitable tests for a subsystem in a linux computer
    – pehrs
    May 31, 2010 at 14:06
0

Does corruption happen on the host itself or in guest machines? There's a known bug in qemu-kvm that leads to data corruption in large virtual disks (see https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/574665 for example)

0

I agree with @pehrs that it is worth looking into the thermal aspect of this since the problem creeps up over time. What kind of server do you have? Most rackmounts these days come with a good number of sensors that can be used to monitor hardware health. Check out lm-sensors. If it is a Dell server, the Dell OMSA package may be helpful. I'm sure other big players have their own proprietary packages as well.

I might as well throw out a few other ideas too-- these don't really match the scenario you described of the problem only appearing after a while but they can't hurt.

As far as error logs are concerned, are you getting any error messages in the logs from the disk or RAID subsystem? Or in dmesg? The Linux Software-RAID HOTWO has some information on the kinds of errors you would be looking for. Something like a bad cable might not show up in the drive's SMART self-tests but you would definitely see some error messages logged.

What is the RAID configuration? Anything in /proc/mdstat? If (for example) the server had a 3 drive RAID 5 and one of the drives was bad that could cause problems.

Also, check the firmware revision for your motherboard/SCSI card/etc and see if it is up to date or if there are any bugs related to disk I/O that have been fixed.

You must log in to answer this question.