Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I was resizing my hard drive last night and was not successful.

My system and drives are local on an ESXi 5.1 vm.

I'm running Debian 6 x64 and have a 2TB mount that I was resizing. It was about 1.8T that I was resizing to the full 2TB.

I ran e2fsck on /dev/sdc1 and it had no issues. I than ran resize2fs and let it run for a few hours. After checking ps, it showed the state in sleep (D+) with no activity. I couldn't kill the process so I rebooted and now I cannot do anything with the drive.

Now I'm missing the /dev/sdc1 drive and I cannot do anything that will read that drive; fdisk, cfdisk, e2fsck, resize2fs all freeze attempting to do anything related to this drive.

What can I do to recover this drive? I unfortunately do not have a backup of this drive. I just want to be able to mount the drive to recover my data or at least read the data on the drive somehow.

dmesg output

[  722.816205] sd 2:0:2:0: timing out command, waited 180s
[  722.816329] sd 2:0:2:0: [sdc] Unhandled error code
[  722.816336] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[  722.816343] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[  722.816352] end_request: I/O error, dev sdc, sector 0
[  722.816443] Buffer I/O error on device sdc, logical block 0
[  722.816534] Buffer I/O error on device sdc, logical block 1
[  722.816619] Buffer I/O error on device sdc, logical block 2
[  722.816703] Buffer I/O error on device sdc, logical block 3
[  903.215677] sd 2:0:2:0: timing out command, waited 180s
[  903.215802] sd 2:0:2:0: [sdc] Unhandled error code
[  903.215805] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[  903.215808] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[  903.215817] end_request: I/O error, dev sdc, sector 0
[  903.215943] Buffer I/O error on device sdc, logical block 0
[  903.216037] Buffer I/O error on device sdc, logical block 1
[  903.216124] Buffer I/O error on device sdc, logical block 2
[  903.216211] Buffer I/O error on device sdc, logical block 3
[  903.216337] ldm_validate_partition_table(): Disk read failed.
[ 1083.615097] sd 2:0:2:0: timing out command, waited 180s
[ 1083.615216] sd 2:0:2:0: [sdc] Unhandled error code
[ 1083.615218] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1083.615222] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1083.615231] end_request: I/O error, dev sdc, sector 0
[ 1083.615316] Buffer I/O error on device sdc, logical block 0
[ 1083.615411] Buffer I/O error on device sdc, logical block 1
[ 1083.615496] Buffer I/O error on device sdc, logical block 2
[ 1083.615580] Buffer I/O error on device sdc, logical block 3
[ 1264.014510] sd 2:0:2:0: timing out command, waited 180s
[ 1264.014667] sd 2:0:2:0: [sdc] Unhandled error code
[ 1264.014669] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1264.014672] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1264.014680] end_request: I/O error, dev sdc, sector 0
[ 1264.014766] Buffer I/O error on device sdc, logical block 0
[ 1264.014855] Buffer I/O error on device sdc, logical block 1
[ 1264.014939] Buffer I/O error on device sdc, logical block 2
[ 1264.015022] Buffer I/O error on device sdc, logical block 3
[ 1444.413968] sd 2:0:2:0: timing out command, waited 180s
[ 1444.414098] sd 2:0:2:0: [sdc] Unhandled error code
[ 1444.414105] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1444.414113] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1444.414124] end_request: I/O error, dev sdc, sector 0
[ 1444.414246] Buffer I/O error on device sdc, logical block 0
[ 1444.414363] Buffer I/O error on device sdc, logical block 1
[ 1444.414450] Buffer I/O error on device sdc, logical block 2
[ 1444.414536] Buffer I/O error on device sdc, logical block 3
[ 1624.813488] sd 2:0:2:0: timing out command, waited 180s
[ 1624.813596] sd 2:0:2:0: [sdc] Unhandled error code
[ 1624.813603] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1624.813611] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1624.813620] end_request: I/O error, dev sdc, sector 0
[ 1624.813699] Buffer I/O error on device sdc, logical block 0
[ 1624.813786] Buffer I/O error on device sdc, logical block 1
[ 1624.813870] Buffer I/O error on device sdc, logical block 2
[ 1624.813953] Buffer I/O error on device sdc, logical block 3
[ 1624.814071] Dev sdc: unable to read RDB block 0
[ 1805.212836] sd 2:0:2:0: timing out command, waited 180s
[ 1805.212951] sd 2:0:2:0: [sdc] Unhandled error code
[ 1805.212953] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1805.212956] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1805.212963] end_request: I/O error, dev sdc, sector 0
[ 1805.213052] Buffer I/O error on device sdc, logical block 0
[ 1805.213143] Buffer I/O error on device sdc, logical block 1
[ 1805.213230] Buffer I/O error on device sdc, logical block 2
[ 1805.213317] Buffer I/O error on device sdc, logical block 3
[ 1985.612363] sd 2:0:2:0: timing out command, waited 180s
[ 1985.612556] sd 2:0:2:0: [sdc] Unhandled error code
[ 1985.612558] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1985.612561] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1985.612568] end_request: I/O error, dev sdc, sector 0
[ 1985.612691] Buffer I/O error on device sdc, logical block 0
[ 1985.612785] Buffer I/O error on device sdc, logical block 1
[ 1985.612878] Buffer I/O error on device sdc, logical block 2
[ 1985.612966] Buffer I/O error on device sdc, logical block 3
[ 2166.011692] sd 2:0:2:0: timing out command, waited 180s
[ 2166.011806] sd 2:0:2:0: [sdc] Unhandled error code
[ 2166.011808] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2166.011811] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 18 00 00 08 00
[ 2166.011818] end_request: I/O error, dev sdc, sector 24
[ 2166.011899] Buffer I/O error on device sdc, logical block 12
[ 2166.011998] Buffer I/O error on device sdc, logical block 13
[ 2166.012083] Buffer I/O error on device sdc, logical block 14
[ 2166.012168] Buffer I/O error on device sdc, logical block 15
[ 2346.411194] sd 2:0:2:0: timing out command, waited 180s
[ 2346.411377] sd 2:0:2:0: [sdc] Unhandled error code
[ 2346.411384] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2346.411392] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 18 00 00 08 00
[ 2346.411403] end_request: I/O error, dev sdc, sector 24
[ 2346.411505] Buffer I/O error on device sdc, logical block 12
[ 2346.411628] Buffer I/O error on device sdc, logical block 13
[ 2346.411730] Buffer I/O error on device sdc, logical block 14
[ 2346.411829] Buffer I/O error on device sdc, logical block 15
[ 2526.810657] sd 2:0:2:0: timing out command, waited 180s
[ 2526.810824] sd 2:0:2:0: [sdc] Unhandled error code
[ 2526.810827] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2526.810831] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 2526.810841] end_request: I/O error, dev sdc, sector 0
[ 2526.810978] Buffer I/O error on device sdc, logical block 0
[ 2526.811096] Buffer I/O error on device sdc, logical block 1
[ 2526.811184] Buffer I/O error on device sdc, logical block 2
[ 2526.811271] Buffer I/O error on device sdc, logical block 3
[ 2526.811391]  unable to read partition table
[ 2526.811899] sd 2:0:2:0: [sdc] Cache data unavailable
[ 2526.811901] sd 2:0:2:0: [sdc] Assuming drive cache: write through
[ 2526.811998] sd 2:0:2:0: [sdc] Attached SCSI disk
[ 2706.830303] VMware vmxnet3 virtual NIC driver - version 1.0.5.0-k-NAPI
[ 2706.833247] VMware PVSCSI driver - version 1.0.1.0-k
[ 2706.836002] VMware vmxnet virtual NIC driver
[ 2706.845581] PM: Starting manual resume from disk
[ 2706.845584] PM: Resume from partition 8:5
[ 2706.845586] PM: Checking hibernation image.
[ 2706.845921] PM: Error -22 checking image file
[ 2706.845923] PM: Resume from disk failed.
[ 2706.858105] kjournald starting.  Commit interval 5 seconds
[ 2706.858117] EXT3-fs: mounted filesystem with ordered data mode.
[ 2707.043045] udev[390]: starting version 164
[ 2707.070265] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[ 2707.070276] ACPI: Power Button [PWRF]
[ 2707.075278] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 2707.077419] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 2707.093012] [415]: VMCI: shared components initialized.
[ 2707.093068] Probing for vmci/PCI.
[ 2707.093106]   alloc irq_desc for 16 on node -1
[ 2707.093107]   alloc kstat_irqs on node -1
[ 2707.093117] vmci 0000:00:07.7: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[ 2707.093196] Found vmci/PCI at 0x1080, irq 16.
[ 2707.093226] VMCI: using capabilities 0xc.
[ 2707.093327] [415]: VMCI: Host capability check passed.
[ 2707.093471]   alloc irq_desc for 56 on node -1
[ 2707.093472]   alloc kstat_irqs on node -1
[ 2707.093524] vmci 0000:00:07.7: irq 56 for MSI/MSI-X
[ 2707.093525]   alloc irq_desc for 57 on node -1
[ 2707.093526]   alloc kstat_irqs on node -1
[ 2707.093546] vmci 0000:00:07.7: irq 57 for MSI/MSI-X
[ 2707.093717] Registered vmci device.
[ 2707.093760] [415]: VMCI: Using guest personality
[ 2707.093762] [415]: VMCI: host components initialized.
[ 2707.093785] [415]: VMCI: Module registered (name=vmci,major=10,minor=59).
[ 2707.093786] [415]: VMCI: Using host personality
[ 2707.093787] [415]: VMCI: Module (name=vmci) is initialized
[ 2707.102777] processor LNXCPU:00: registered as cooling_device0
[ 2707.103353] processor LNXCPU:01: registered as cooling_device1
[ 2707.103522] piix4_smbus 0000:00:07.3: Host SMBus controller not enabled!
[ 2707.104218] processor LNXCPU:02: registered as cooling_device2
[ 2707.105603] processor LNXCPU:03: registered as cooling_device3
[ 2707.107367] input: PC Speaker as /devices/platform/pcspkr/input/input3
[ 2707.117231] ACPI: AC Adapter [ACAD] (on-line)
[ 2707.148586] parport_pc 00:09: reported by Plug and Play ACPI
[ 2707.148765] parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
[ 2707.193924] input: ImPS/2 Generic Wheel Mouse as /devices/platform/i8042/serio1/input/input4
[ 2707.199564] Error: Driver 'pcspkr' is already registered, aborting...
[ 2707.209963] sd 2:0:2:0: timing out command, waited 180s
[ 2707.210018] sd 2:0:2:0: [sdc] Unhandled error code
[ 2707.210020] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2707.210022] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 2707.210029] end_request: I/O error, dev sdc, sector 0
[ 2707.210077] Buffer I/O error on device sdc, logical block 0
[ 2707.210129] Buffer I/O error on device sdc, logical block 1
[ 2707.210179] Buffer I/O error on device sdc, logical block 2
[ 2707.210230] Buffer I/O error on device sdc, logical block 3
[ 3787.606904] sd 2:0:2:0: [sdc] Unhandled error code
[ 3787.606907] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 3787.606910] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 3787.606921] end_request: I/O error, dev sdc, sector 0
[ 3787.607032] Buffer I/O error on device sdc, logical block 0
[ 3787.607145] Buffer I/O error on device sdc, logical block 1
[ 3787.607256] Buffer I/O error on device sdc, logical block 2
[ 3787.607367] Buffer I/O error on device sdc, logical block 3
[ 4868.003267] sd 2:0:2:0: timing out command, waited 1080s
[ 4868.003392] sd 2:0:2:0: [sdc] Unhandled error code
[ 4868.003394] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 4868.003397] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 4868.003407] end_request: I/O error, dev sdc, sector 0
[ 4868.003495] Buffer I/O error on device sdc, logical block 0
[ 4868.003595] Buffer I/O error on device sdc, logical block 1
[ 4868.003685] Buffer I/O error on device sdc, logical block 2
[ 4868.003774] Buffer I/O error on device sdc, logical block 3
[ 5948.399889] sd 2:0:2:0: timing out command, waited 1080s
[ 5948.400005] sd 2:0:2:0: [sdc] Unhandled error code
[ 5948.400012] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 5948.400019] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 02 00
[ 5948.400028] end_request: I/O error, dev sdc, sector 0
[ 5948.400110] Buffer I/O error on device sdc, logical block 0
[ 5948.400209] sd 2:0:2:0: timing out command, waited 1080s
[ 5948.400294] sd 2:0:2:0: [sdc] Unhandled error code
[ 5948.400295] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 5948.400298] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 02 00 00 06 00
[ 5948.400304] end_request: I/O error, dev sdc, sector 2
[ 5948.400384] Buffer I/O error on device sdc, logical block 1
[ 5948.400473] Buffer I/O error on device sdc, logical block 2
[ 5948.400562] Buffer I/O error on device sdc, logical block 3
[ 7028.796518] sd 2:0:2:0: timing out command, waited 1080s
[ 7028.796633] sd 2:0:2:0: [sdc] Unhandled error code
[ 7028.796640] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 7028.796648] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 02 00
[ 7028.796657] end_request: I/O error, dev sdc, sector 0
[ 7028.796746] Buffer I/O error on device sdc, logical block 0
[ 7028.796864] sd 2:0:2:0: timing out command, waited 1080s
[ 7028.796948] sd 2:0:2:0: [sdc] Unhandled error code
[ 7028.796949] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 7028.796952] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 02 00 00 06 00
[ 7028.796958] end_request: I/O error, dev sdc, sector 2
[ 7028.797037] Buffer I/O error on device sdc, logical block 1
[ 7028.797122] Buffer I/O error on device sdc, logical block 2
[ 7028.797207] Buffer I/O error on device sdc, logical block 3
share|improve this question
    
I still have the /dev/sdc, but no longer have /dev/sdc1. –  tdbui22 Apr 22 '13 at 15:18
1  
Does dmesg show any error messages related to the drive? Hangs like this might be signs of a hardware problem; missing partition is also suspicious, because resize2fs does not itself touch the partition table. –  Sergey Vlasov Apr 22 '13 at 15:31
    
Also make sure that /dev/sdc is really the drive you think it is — sdX device assignments can change, e.g., if some device was not detected, or in some cases they may change every boot if you have multiple controllers detected in parallel. –  Sergey Vlasov Apr 22 '13 at 15:34
    
@SergeyVlasov I've added the dmesg output to my question. I also forgot to mention that this is a esxi vm drive. It is definitely the correct drive as well. –  tdbui22 Apr 22 '13 at 15:51
    
Do you have admin access to the ESXi server, and on what kind of storage is that virtual drive located? Also, was the VM just rebooted, or stopped completely and restarted? –  Sergey Vlasov Apr 22 '13 at 16:55
show 4 more comments

1 Answer

First, resize2fs may take several hours or even more than a day to run, depending on the size of your partition.

If you interrupted it while it was running, that's a very good recipe for losing all your data. All of the filesystem data structures are probably in an inconsistent state.

Now, your drive has errors at the very beginning of the disk, where the partition table is stored.


At this point, what you should do is replace the disk and restore your backups. Since you don't have backups, you should update your CV. If this server was very important, the company might go out of business.

To avoid situations like this in future, always ensure that you have backups and that your backups are fully functional, that you can successfully restore with them.


So, aside from updating your CV and making backups, this is what you need to do today:

  1. You can attempt to use dd_rescue to copy the disk, skipping any errors, to another disk, and then work on rescuing data from the copy.

    Don't expect to get too much out of this, though.

  2. If you can't rescue anything from the disk, send it to a professional data recovery company.

share|improve this answer
    
Unfortunately I do not have backup and in serious crisis at the moment. I just tried dd_rescue and it just freezes. Checking ps and it is in sleep state. Everything that touches this drive goes to sleep state. –  tdbui22 Apr 22 '13 at 17:45
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.