How to recover resize2fs failure

Question

I was resizing my hard drive last night and was not successful.

My system and drives are local on an ESXi 5.1 vm.

I'm running Debian 6 x64 and have a 2TB mount that I was resizing. It was about 1.8T that I was resizing to the full 2TB.

I ran e2fsck on /dev/sdc1 and it had no issues. I than ran resize2fs and let it run for a few hours. After checking ps, it showed the state in sleep (D+) with no activity. I couldn't kill the process so I rebooted and now I cannot do anything with the drive.

Now I'm missing the /dev/sdc1 drive and I cannot do anything that will read that drive; fdisk, cfdisk, e2fsck, resize2fs all freeze attempting to do anything related to this drive.

What can I do to recover this drive? I unfortunately do not have a backup of this drive. I just want to be able to mount the drive to recover my data or at least read the data on the drive somehow.

dmesg output

[  722.816205] sd 2:0:2:0: timing out command, waited 180s
[  722.816329] sd 2:0:2:0: [sdc] Unhandled error code
[  722.816336] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[  722.816343] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[  722.816352] end_request: I/O error, dev sdc, sector 0
[  722.816443] Buffer I/O error on device sdc, logical block 0
[  722.816534] Buffer I/O error on device sdc, logical block 1
[  722.816619] Buffer I/O error on device sdc, logical block 2
[  722.816703] Buffer I/O error on device sdc, logical block 3
[  903.215677] sd 2:0:2:0: timing out command, waited 180s
[  903.215802] sd 2:0:2:0: [sdc] Unhandled error code
[  903.215805] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[  903.215808] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[  903.215817] end_request: I/O error, dev sdc, sector 0
[  903.215943] Buffer I/O error on device sdc, logical block 0
[  903.216037] Buffer I/O error on device sdc, logical block 1
[  903.216124] Buffer I/O error on device sdc, logical block 2
[  903.216211] Buffer I/O error on device sdc, logical block 3
[  903.216337] ldm_validate_partition_table(): Disk read failed.
[ 1083.615097] sd 2:0:2:0: timing out command, waited 180s
[ 1083.615216] sd 2:0:2:0: [sdc] Unhandled error code
[ 1083.615218] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1083.615222] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1083.615231] end_request: I/O error, dev sdc, sector 0
[ 1083.615316] Buffer I/O error on device sdc, logical block 0
[ 1083.615411] Buffer I/O error on device sdc, logical block 1
[ 1083.615496] Buffer I/O error on device sdc, logical block 2
[ 1083.615580] Buffer I/O error on device sdc, logical block 3
[ 1264.014510] sd 2:0:2:0: timing out command, waited 180s
[ 1264.014667] sd 2:0:2:0: [sdc] Unhandled error code
[ 1264.014669] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1264.014672] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1264.014680] end_request: I/O error, dev sdc, sector 0
[ 1264.014766] Buffer I/O error on device sdc, logical block 0
[ 1264.014855] Buffer I/O error on device sdc, logical block 1
[ 1264.014939] Buffer I/O error on device sdc, logical block 2
[ 1264.015022] Buffer I/O error on device sdc, logical block 3
[ 1444.413968] sd 2:0:2:0: timing out command, waited 180s
[ 1444.414098] sd 2:0:2:0: [sdc] Unhandled error code
[ 1444.414105] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1444.414113] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1444.414124] end_request: I/O error, dev sdc, sector 0
[ 1444.414246] Buffer I/O error on device sdc, logical block 0
[ 1444.414363] Buffer I/O error on device sdc, logical block 1
[ 1444.414450] Buffer I/O error on device sdc, logical block 2
[ 1444.414536] Buffer I/O error on device sdc, logical block 3
[ 1624.813488] sd 2:0:2:0: timing out command, waited 180s
[ 1624.813596] sd 2:0:2:0: [sdc] Unhandled error code
[ 1624.813603] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1624.813611] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1624.813620] end_request: I/O error, dev sdc, sector 0
[ 1624.813699] Buffer I/O error on device sdc, logical block 0
[ 1624.813786] Buffer I/O error on device sdc, logical block 1
[ 1624.813870] Buffer I/O error on device sdc, logical block 2
[ 1624.813953] Buffer I/O error on device sdc, logical block 3
[ 1624.814071] Dev sdc: unable to read RDB block 0
[ 1805.212836] sd 2:0:2:0: timing out command, waited 180s
[ 1805.212951] sd 2:0:2:0: [sdc] Unhandled error code
[ 1805.212953] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1805.212956] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1805.212963] end_request: I/O error, dev sdc, sector 0
[ 1805.213052] Buffer I/O error on device sdc, logical block 0
[ 1805.213143] Buffer I/O error on device sdc, logical block 1
[ 1805.213230] Buffer I/O error on device sdc, logical block 2
[ 1805.213317] Buffer I/O error on device sdc, logical block 3
[ 1985.612363] sd 2:0:2:0: timing out command, waited 180s
[ 1985.612556] sd 2:0:2:0: [sdc] Unhandled error code
[ 1985.612558] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 1985.612561] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 1985.612568] end_request: I/O error, dev sdc, sector 0
[ 1985.612691] Buffer I/O error on device sdc, logical block 0
[ 1985.612785] Buffer I/O error on device sdc, logical block 1
[ 1985.612878] Buffer I/O error on device sdc, logical block 2
[ 1985.612966] Buffer I/O error on device sdc, logical block 3
[ 2166.011692] sd 2:0:2:0: timing out command, waited 180s
[ 2166.011806] sd 2:0:2:0: [sdc] Unhandled error code
[ 2166.011808] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2166.011811] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 18 00 00 08 00
[ 2166.011818] end_request: I/O error, dev sdc, sector 24
[ 2166.011899] Buffer I/O error on device sdc, logical block 12
[ 2166.011998] Buffer I/O error on device sdc, logical block 13
[ 2166.012083] Buffer I/O error on device sdc, logical block 14
[ 2166.012168] Buffer I/O error on device sdc, logical block 15
[ 2346.411194] sd 2:0:2:0: timing out command, waited 180s
[ 2346.411377] sd 2:0:2:0: [sdc] Unhandled error code
[ 2346.411384] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2346.411392] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 18 00 00 08 00
[ 2346.411403] end_request: I/O error, dev sdc, sector 24
[ 2346.411505] Buffer I/O error on device sdc, logical block 12
[ 2346.411628] Buffer I/O error on device sdc, logical block 13
[ 2346.411730] Buffer I/O error on device sdc, logical block 14
[ 2346.411829] Buffer I/O error on device sdc, logical block 15
[ 2526.810657] sd 2:0:2:0: timing out command, waited 180s
[ 2526.810824] sd 2:0:2:0: [sdc] Unhandled error code
[ 2526.810827] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2526.810831] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 2526.810841] end_request: I/O error, dev sdc, sector 0
[ 2526.810978] Buffer I/O error on device sdc, logical block 0
[ 2526.811096] Buffer I/O error on device sdc, logical block 1
[ 2526.811184] Buffer I/O error on device sdc, logical block 2
[ 2526.811271] Buffer I/O error on device sdc, logical block 3
[ 2526.811391]  unable to read partition table
[ 2526.811899] sd 2:0:2:0: [sdc] Cache data unavailable
[ 2526.811901] sd 2:0:2:0: [sdc] Assuming drive cache: write through
[ 2526.811998] sd 2:0:2:0: [sdc] Attached SCSI disk
[ 2706.830303] VMware vmxnet3 virtual NIC driver - version 1.0.5.0-k-NAPI
[ 2706.833247] VMware PVSCSI driver - version 1.0.1.0-k
[ 2706.836002] VMware vmxnet virtual NIC driver
[ 2706.845581] PM: Starting manual resume from disk
[ 2706.845584] PM: Resume from partition 8:5
[ 2706.845586] PM: Checking hibernation image.
[ 2706.845921] PM: Error -22 checking image file
[ 2706.845923] PM: Resume from disk failed.
[ 2706.858105] kjournald starting.  Commit interval 5 seconds
[ 2706.858117] EXT3-fs: mounted filesystem with ordered data mode.
[ 2707.043045] udev[390]: starting version 164
[ 2707.070265] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[ 2707.070276] ACPI: Power Button [PWRF]
[ 2707.075278] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 2707.077419] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 2707.093012] [415]: VMCI: shared components initialized.
[ 2707.093068] Probing for vmci/PCI.
[ 2707.093106]   alloc irq_desc for 16 on node -1
[ 2707.093107]   alloc kstat_irqs on node -1
[ 2707.093117] vmci 0000:00:07.7: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[ 2707.093196] Found vmci/PCI at 0x1080, irq 16.
[ 2707.093226] VMCI: using capabilities 0xc.
[ 2707.093327] [415]: VMCI: Host capability check passed.
[ 2707.093471]   alloc irq_desc for 56 on node -1
[ 2707.093472]   alloc kstat_irqs on node -1
[ 2707.093524] vmci 0000:00:07.7: irq 56 for MSI/MSI-X
[ 2707.093525]   alloc irq_desc for 57 on node -1
[ 2707.093526]   alloc kstat_irqs on node -1
[ 2707.093546] vmci 0000:00:07.7: irq 57 for MSI/MSI-X
[ 2707.093717] Registered vmci device.
[ 2707.093760] [415]: VMCI: Using guest personality
[ 2707.093762] [415]: VMCI: host components initialized.
[ 2707.093785] [415]: VMCI: Module registered (name=vmci,major=10,minor=59).
[ 2707.093786] [415]: VMCI: Using host personality
[ 2707.093787] [415]: VMCI: Module (name=vmci) is initialized
[ 2707.102777] processor LNXCPU:00: registered as cooling_device0
[ 2707.103353] processor LNXCPU:01: registered as cooling_device1
[ 2707.103522] piix4_smbus 0000:00:07.3: Host SMBus controller not enabled!
[ 2707.104218] processor LNXCPU:02: registered as cooling_device2
[ 2707.105603] processor LNXCPU:03: registered as cooling_device3
[ 2707.107367] input: PC Speaker as /devices/platform/pcspkr/input/input3
[ 2707.117231] ACPI: AC Adapter [ACAD] (on-line)
[ 2707.148586] parport_pc 00:09: reported by Plug and Play ACPI
[ 2707.148765] parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
[ 2707.193924] input: ImPS/2 Generic Wheel Mouse as /devices/platform/i8042/serio1/input/input4
[ 2707.199564] Error: Driver 'pcspkr' is already registered, aborting...
[ 2707.209963] sd 2:0:2:0: timing out command, waited 180s
[ 2707.210018] sd 2:0:2:0: [sdc] Unhandled error code
[ 2707.210020] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 2707.210022] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 2707.210029] end_request: I/O error, dev sdc, sector 0
[ 2707.210077] Buffer I/O error on device sdc, logical block 0
[ 2707.210129] Buffer I/O error on device sdc, logical block 1
[ 2707.210179] Buffer I/O error on device sdc, logical block 2
[ 2707.210230] Buffer I/O error on device sdc, logical block 3
[ 3787.606904] sd 2:0:2:0: [sdc] Unhandled error code
[ 3787.606907] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 3787.606910] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 3787.606921] end_request: I/O error, dev sdc, sector 0
[ 3787.607032] Buffer I/O error on device sdc, logical block 0
[ 3787.607145] Buffer I/O error on device sdc, logical block 1
[ 3787.607256] Buffer I/O error on device sdc, logical block 2
[ 3787.607367] Buffer I/O error on device sdc, logical block 3
[ 4868.003267] sd 2:0:2:0: timing out command, waited 1080s
[ 4868.003392] sd 2:0:2:0: [sdc] Unhandled error code
[ 4868.003394] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 4868.003397] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[ 4868.003407] end_request: I/O error, dev sdc, sector 0
[ 4868.003495] Buffer I/O error on device sdc, logical block 0
[ 4868.003595] Buffer I/O error on device sdc, logical block 1
[ 4868.003685] Buffer I/O error on device sdc, logical block 2
[ 4868.003774] Buffer I/O error on device sdc, logical block 3
[ 5948.399889] sd 2:0:2:0: timing out command, waited 1080s
[ 5948.400005] sd 2:0:2:0: [sdc] Unhandled error code
[ 5948.400012] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 5948.400019] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 02 00
[ 5948.400028] end_request: I/O error, dev sdc, sector 0
[ 5948.400110] Buffer I/O error on device sdc, logical block 0
[ 5948.400209] sd 2:0:2:0: timing out command, waited 1080s
[ 5948.400294] sd 2:0:2:0: [sdc] Unhandled error code
[ 5948.400295] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 5948.400298] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 02 00 00 06 00
[ 5948.400304] end_request: I/O error, dev sdc, sector 2
[ 5948.400384] Buffer I/O error on device sdc, logical block 1
[ 5948.400473] Buffer I/O error on device sdc, logical block 2
[ 5948.400562] Buffer I/O error on device sdc, logical block 3
[ 7028.796518] sd 2:0:2:0: timing out command, waited 1080s
[ 7028.796633] sd 2:0:2:0: [sdc] Unhandled error code
[ 7028.796640] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 7028.796648] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 02 00
[ 7028.796657] end_request: I/O error, dev sdc, sector 0
[ 7028.796746] Buffer I/O error on device sdc, logical block 0
[ 7028.796864] sd 2:0:2:0: timing out command, waited 1080s
[ 7028.796948] sd 2:0:2:0: [sdc] Unhandled error code
[ 7028.796949] sd 2:0:2:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 7028.796952] sd 2:0:2:0: [sdc] CDB: Read(10): 28 00 00 00 00 02 00 00 06 00
[ 7028.796958] end_request: I/O error, dev sdc, sector 2
[ 7028.797037] Buffer I/O error on device sdc, logical block 1
[ 7028.797122] Buffer I/O error on device sdc, logical block 2
[ 7028.797207] Buffer I/O error on device sdc, logical block 3

Answer 1

First, resize2fs may take several hours or even more than a day to run, depending on the size of your partition.

If you interrupted it while it was running, that's a very good recipe for losing all your data. All of the filesystem data structures are probably in an inconsistent state.

Now, your drive has errors at the very beginning of the disk, where the partition table is stored.

---

At this point, what you should do is replace the disk and restore your backups. Since you don't have backups, you should update your CV. If this server was very important, the company might go out of business.

To avoid situations like this in future, always ensure that you have backups and that your backups are fully functional, that you can successfully restore with them.

---

So, aside from updating your CV and making backups, this is what you need to do today:

1. You can attempt to use dd_rescue to copy the disk, skipping any errors, to another disk, and then work on rescuing data from the copy.

   Don't expect to get too much out of this, though.

2. If you can't rescue anything from the disk, send it to a professional data recovery company.

Answer 2

I know this is years later and not likely helpful to the OP, but I wanted to add another perspective on this for others coming across this post.

I've had resize2fs interrupted more than once (i.e. power failures), and after a quick fsck have not had any issues or data loss. It certainly sounds plausible to lose large amounts of data this way or even corrupt the entire partition since this is affecting the filesystem itself, I've just never personally witnessed it before.

However, the errors the OP posted combined with failed attempts to run dd_rescue (which doesn't care about corruption at the filesystem level) points to a hardware failure, (or in the case of a VM, a problem with the container file or perhaps the underlying hardware). While coincidental, this sounds independent of the failed resize2fs attempt.

I would first make sure the underlying hardware is good. If all checks out there, make a copy of the VM image, then rebuild the copy (the specific method for that will depend on the image format). Finally, boot the copy, running fsck as well, and see if that resolves the issue.

Answer 3

I also thought my resize2fs had failed because the process was not on top of top any more and there was of course no output on the command line from the command itself - since I did not run it as resize2fs -p.

So I checked my system and found it hanging on I/O:

I further found with pgrep resize2fs that the process is still alive and with dmesg I found output like this:

[26517.498831] ata13.00: status: { DRDY ERR }
[26517.498837] ata13.00: error: { UNC }
[26517.500149] ata13.00: configured for UDMA/133
[26517.500214] sd 12:0:0:0: [sdi] tag#24 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[26517.500219] sd 12:0:0:0: [sdi] tag#24 Sense Key : Medium Error [current] 
[26517.500223] sd 12:0:0:0: [sdi] tag#24 Add. Sense: Unrecovered read error - auto reallocate failed
[26517.500228] sd 12:0:0:0: [sdi] tag#24 CDB: Read(16) 88 00 00 00 00 00 cc 59 5d b8 00 00 00 08 00 00
[26517.500232] print_req_error: I/O error, dev sdi, sector 3428408760
[26517.500263] ata13: EH complete

So, anybody who reads this, maybe one of your drives is just failing and the command is still running. In my case it was resolved after a while by:

[26768.067369] raid5_end_read_request: 95 callbacks suppressed
[26768.067372] md/raid:md127: read error corrected (8 sectors at 3428408944 on sdi)
[26768.068351] md/raid:md127: read error corrected (8 sectors at 3428409008 on sdi)
[26768.068357] md/raid:md127: read error corrected (8 sectors at 3428409016 on sdi)
[26768.079144] md/raid:md127: read error corrected (8 sectors at 3428408744 on sdi)
[26768.079150] md/raid:md127: read error corrected (8 sectors at 3428408752 on sdi)
[26768.079152] md/raid:md127: read error corrected (8 sectors at 3428408760 on sdi)
[26768.079153] md/raid:md127: read error corrected (8 sectors at 3428408768 on sdi)
[26768.079155] md/raid:md127: read error corrected (8 sectors at 3428408776 on sdi)
[26768.079156] md/raid:md127: read error corrected (8 sectors at 3428408784 on sdi)
[26768.079158] md/raid:md127: read error corrected (8 sectors at 3428408792 on sdi)