0

I have an ESXi 5.1 server (free version) with two 500G drives mounted and assigned to one datastore. I am not able to delete or move certain files in the datastore. I don't have vMotion.

This is not related to: VMWare ESX 3.5 Input/Output Error

Neither is it related to this VMWare KB

The output may be similar but the circumstances are different.

I've tried expanding the storage by adding another hard drive, so there are currently 3 500G drives assigned to the datastore, and there is a lot of space available to it.

The following is a listing of the datastore folder

-rw-r--r--    1 root     root          27 Feb  5  2015 DC-9f914f3e.hlog
-rw-------    1 root     root       16.0G Aug  5 15:20 DC-9f914f3e.vswp
-rw-------    1 root     root      995.6G Aug  5 16:01 DC-flat.vmdk
-rwxr-xr-x    1 root     root        3.5K Aug  3 05:36 DC.bakvmx
-rw-------    1 root     root        8.5K Aug  3 05:37 DC.nvram
-rw-------    1 root     root         505 Aug  3 05:36 DC.vmdk
-rw-r--r--    1 root     root           0 Jan 27  2014 DC.vmsd
-rwxr-xr-x    1 root     root        3.6K Aug  3 05:36 DC.vmx
-rw-------    1 root     root           0 Aug  3 05:36 DC.vmx.backup
-rw-r--r--    1 root     root         269 Aug  3 05:36 DC.vmxf
-rw-r--r--    1 root     root           0 Aug  6 08:19 DC2.vmsd
-rw-r--r--    1 root     root        3.5K Aug  6 12:47 DC2.vmx
-rw-r--r--    1 root     root         270 Aug  7 10:51 DC2.vmxf
-rw-------    1 root     root      500.0G Aug  6 12:51 DC_1-flat.vmdk
-rw-------    1 root     root         506 Aug  6 10:50 DC_1.vmdk
-rw-------    1 root     root        3.7M Jun 23 08:20 vmmcores-5.gz
-rw-------    1 root     root        2.9M Jun 23 08:41 vmmcores-6.gz
-rw-------    1 root     root        2.9M Jun 23 08:50 vmmcores-7.gz
-rw-------    1 root     root        2.3M Jun 23 08:56 vmmcores-8.gz
-rw-------    1 root     root        3.0M Jun 23 10:44 vmmcores-9.gz
-rw-r--r--    1 root     root      214.1K Aug  4 08:55 vmware-106.log
-rw-------    1 root     root       94.0M Aug  3 05:36 vmx-DC-2677100350-1.vswp
-rw-------    1 root     root       91.0M Oct 30  2014 vmx-DC-2677100350-2.vswp

I'm attempting to delete DC_1-flat.vmdk, or the DC.vmx.backup file which has a zero size. It cannot be deleted. I get the following output:

 rm: can't remove 'DC_1-flat.vmdk': Input/output error

The other files can be moved or deleted, but not these files.

I've tried everything, and I'm not sure what to do. The main vmdk has the same problem. The VM is not starting up because a similar Input/Output error is produced.

I've run the following check and the following output was produced:

# vmkfstools -v 0 -t 0 DC.vmdk
FileIOErrno2Result: Unexpected errno=5, Input/output error
OBJLIB-FILEBE : FileBEOpen: can't open './DC-flat.vmdk' : Input/output error (327683).
DISKLIB-VMFS  : "./DC-flat.vmdk" : failed to open (Input/output error): ObjLib_Open failed. Type 3
DISKLIB-LINK  : "DC.vmdk" : failed to open (Input/output error).
DISKLIB-CHAIN : "DC.vmdk" : failed to open (Input/output error).
DISKLIB-LIB   : Failed to open 'DC.vmdk' with flags 0xe Input/output error (327689).
Failed to open virtual disk: Input/output error 327689
AIOMGR-S : stat o=1 r=3 w=0 i=0 br=49152 bw=0
AIOMGR-U : stat o=1 r=0 w=0 i=0 br=0 bw=0
OBJLIB-LIB : ObjLib cleanup done.

I've checked the hard drive for errors using the following article

And the S.M.A.R.T. health check on the hard drives seem to check out okay.

  • 2
    Sounds like bad disks. Can you verify that all disks are checking out okay? Do you monitor with anything? – TryTryAgain Aug 7 '15 at 17:38
  • What do you mean with two 500G drives mounted in one datastore? – GregL Aug 7 '15 at 18:41
  • I don't monitor with anything really. What could I use to check the disks? I tried some of the vmware disk utilities but they are apparently not necessary since vmware does an automatic health check. They checked out ok, but I do believe that it could be a disk fault. I just need to know how to identify it. – Xenno Aug 7 '15 at 19:49
  • Two 500g hard drives assigned to a single datatype, making 1TB available to the vm's – Xenno Aug 9 '15 at 7:54
  • I found the problem. One of the hard drives has a hardware fault. I scanned it using the manufacturer's disk utility and found a few bad sectors on the hard drive. I need to attempt to repair those sectors. I guess loss of data would be inevitable. – Xenno Aug 11 '15 at 10:46
0

TryTryAgain answered the question in a comment. The problem was the bad disks. One of the drives had a few bad sectors that could not be repaired.

We took the drive a data recovery centre and they cloned the drive to a new hard drive with the data as is. After installing it back into the server, and re-installing ESXi, the datastore was back online and the VM was restored. Thankfully, no data loss experienced.

  • You should configure in RAID as this would eliminate that problem, rather than having them all striped. Remember: just because smart tool says PASSED, that merely means hat it was successful in checking the smart data and not that the drive is fine. You should pay particular attention to offline uncorrectable being anything other than 0, as well as bad sectors. – Timothy Frew Apr 28 '18 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.