1

The problem: I have a lot of Disk IO errors on my server, these are causing multiple server failures.

  • VMs are rebooting because of IO errors "task xyz/sdaX blocked for more than 120 seconds"
  • Backup not working, because VSS needs to much time.
  • Writing to HDD Disks not possible or transfer is extrem slow with massive retry events
  • Disks are disappearing and stay disappeared until I power cycle the server

Windows: "The IO operation at logical block address X for Disk (2|5|7|8) was retried"

Linux: "Buffer I/O error on dev sdX1, logical block Y, lost async page write"

My Server:

Mainboard: Supermicro XDRi
CPU: 2x E5-2630v3
RAM: 8x32GB DDR4 (8x Samsung M386A4G40DM0)
Disks:
4x WD Red 3TB
2x WD Red 6TB
2x SM863 2TB
1x Intel SSDSC2BX200G4 200GB
1x Samsung 940 Evo - 256GB
OS: Hyper-V 2012 R2
Controller: Onboard Intel C612 | HighPoint Rocket 2720SGL | HighPoint Rocket 640L
Raid: I'm not using any hardware raid - I use MS Storage Spaces, but the described problem occurs even without any software raid.

What I tried:

  • Changing all Sata / SAS cables (2x!)
  • Changing the sata controller (2x!)
  • change the hdd bay slot
  • Tested every single disk at my workstation - no smart / write / read error
  • Reinstalled the host system
  • Installed older / newer driver
  • Updated bios / firmware
  • Reset Bios Settings / Disabled power saving options
  • CPU / RAM Test

I can reproduce the IO errors if I write data to the disks (only hdds - no issues with my ssds) - windows or linux - it does not matter.

Do you have an idea, what that could be?

Improve this question
4
  • Are you using RAID? How many VMs/how much I/O do you do simultaneously? If you are hammering the drives you simply might be over-stressing the drives. There are ways to fix it: 1. Use RAID, this will speed up your I/O 2. 1 VM per drive or as many as you can use before you get errors. 3. Use SAS drives, they are faster.
    – Andriy Lysak
    Apr 29, 2016 at 21:10
  • No hardware raid - all VMs are on my SSD, so the VM I/O should not be the issue, and my Linux test was without any VM - just the single hdds
    – HEGE
    Apr 29, 2016 at 21:28
  • do you have a spare power supply to check whether replacing it solves anything?
    – Ale
    Apr 29, 2016 at 21:40
  • no,I will try it with a second external psu for the disks
    – HEGE
    Apr 29, 2016 at 22:18

1 Answer 1

Reset to default
1

It seems that the power plug cables were not ok, I changed the power plug cables from the psu to the backplane, now everything is working - I was able to test 1,5Gb/s without a single disk I/O error.

Still can't imagine how this could happen.

Improve this answer
1
  • 1
    The cables might have had a bad contact (oxidized contacts, bad quality connector, etc.), and a minimal vibration induced for example by the ventilation system might be enough to cause micro-interruptions of the power supply, which are hard to detect and can cause any kind of weird behaviour. An alternative explanation to this could also be that there was insertion resistance, and on current spikes by the disks the power supply voltage they received was falling, with the same possible unexpected behaviours.
    – Ale
    May 11, 2016 at 21:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .