VMWare ESX 5.1 Input/Output Error when copying or deleting a file

Question

I have an ESXi 5.1 server (free version) with two 500G drives mounted and assigned to one datastore. I am not able to delete or move certain files in the datastore. I don't have vMotion.

This is not related to: VMWare ESX 3.5 Input/Output Error

Neither is it related to this VMWare KB

The output may be similar but the circumstances are different.

I've tried expanding the storage by adding another hard drive, so there are currently 3 500G drives assigned to the datastore, and there is a lot of space available to it.

The following is a listing of the datastore folder

-rw-r--r--    1 root     root          27 Feb  5  2015 DC-9f914f3e.hlog
-rw-------    1 root     root       16.0G Aug  5 15:20 DC-9f914f3e.vswp
-rw-------    1 root     root      995.6G Aug  5 16:01 DC-flat.vmdk
-rwxr-xr-x    1 root     root        3.5K Aug  3 05:36 DC.bakvmx
-rw-------    1 root     root        8.5K Aug  3 05:37 DC.nvram
-rw-------    1 root     root         505 Aug  3 05:36 DC.vmdk
-rw-r--r--    1 root     root           0 Jan 27  2014 DC.vmsd
-rwxr-xr-x    1 root     root        3.6K Aug  3 05:36 DC.vmx
-rw-------    1 root     root           0 Aug  3 05:36 DC.vmx.backup
-rw-r--r--    1 root     root         269 Aug  3 05:36 DC.vmxf
-rw-r--r--    1 root     root           0 Aug  6 08:19 DC2.vmsd
-rw-r--r--    1 root     root        3.5K Aug  6 12:47 DC2.vmx
-rw-r--r--    1 root     root         270 Aug  7 10:51 DC2.vmxf
-rw-------    1 root     root      500.0G Aug  6 12:51 DC_1-flat.vmdk
-rw-------    1 root     root         506 Aug  6 10:50 DC_1.vmdk
-rw-------    1 root     root        3.7M Jun 23 08:20 vmmcores-5.gz
-rw-------    1 root     root        2.9M Jun 23 08:41 vmmcores-6.gz
-rw-------    1 root     root        2.9M Jun 23 08:50 vmmcores-7.gz
-rw-------    1 root     root        2.3M Jun 23 08:56 vmmcores-8.gz
-rw-------    1 root     root        3.0M Jun 23 10:44 vmmcores-9.gz
-rw-r--r--    1 root     root      214.1K Aug  4 08:55 vmware-106.log
-rw-------    1 root     root       94.0M Aug  3 05:36 vmx-DC-2677100350-1.vswp
-rw-------    1 root     root       91.0M Oct 30  2014 vmx-DC-2677100350-2.vswp

I'm attempting to delete DC_1-flat.vmdk, or the DC.vmx.backup file which has a zero size. It cannot be deleted. I get the following output:

 rm: can't remove 'DC_1-flat.vmdk': Input/output error

The other files can be moved or deleted, but not these files.

I've tried everything, and I'm not sure what to do. The main vmdk has the same problem. The VM is not starting up because a similar Input/Output error is produced.

I've run the following check and the following output was produced:

# vmkfstools -v 0 -t 0 DC.vmdk
FileIOErrno2Result: Unexpected errno=5, Input/output error
OBJLIB-FILEBE : FileBEOpen: can't open './DC-flat.vmdk' : Input/output error (327683).
DISKLIB-VMFS  : "./DC-flat.vmdk" : failed to open (Input/output error): ObjLib_Open failed. Type 3
DISKLIB-LINK  : "DC.vmdk" : failed to open (Input/output error).
DISKLIB-CHAIN : "DC.vmdk" : failed to open (Input/output error).
DISKLIB-LIB   : Failed to open 'DC.vmdk' with flags 0xe Input/output error (327689).
Failed to open virtual disk: Input/output error 327689
AIOMGR-S : stat o=1 r=3 w=0 i=0 br=49152 bw=0
AIOMGR-U : stat o=1 r=0 w=0 i=0 br=0 bw=0
OBJLIB-LIB : ObjLib cleanup done.

I've checked the hard drive for errors using the following article

And the S.M.A.R.T. health check on the hard drives seem to check out okay.

Answer 1

TryTryAgain answered the question in a comment. The problem was the bad disks. One of the drives had a few bad sectors that could not be repaired.

We took the drive a data recovery centre and they cloned the drive to a new hard drive with the data as is. After installing it back into the server, and re-installing ESXi, the datastore was back online and the VM was restored. Thankfully, no data loss experienced.