Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I was uploading a file to a datastore on ESXi 5 host, when something went wrong. I rebooted the host, but now cannot delete the file. (I renamed it to BADFILE). As you can see from the commands below, I've done all the right things...but just can't delete it! Any ideas?

Note this file is on the ESXi host - not used by a VM guest. lsof shows no processes using the file.

/vmfs/volumes/4ed306ea-ac7ce83d-0559-001731f8a1de/ISO Images # touch BADFILE 
touch: BADFILE: Device or resource busy
/vmfs/volumes/4ed306ea-ac7ce83d-0559-001731f8a1de/ISO Images # vmkfstools -D BADFILE 
Lock [type 10c00001 offset 10063872 v 215, hb offset 3579904
gen 133, mode 2, owner 00000000-00000000-0000-000000000000 mtime 480
num 1 gblnum 0 gblgen 0 gblbrk 0]
RO Owner[0] HB Offset 3579904 511c5a3a-8cabaab1-ae4c-001731f8a1de
Addr <4, 3, 170>, gen 206, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 3695179776, nb 441 tbz 0, cow 0, newSinceEpoch 441, zla 3, bs 8388608
/vmfs/volumes/4ed306ea-ac7ce83d-0559-001731f8a1de/ISO Images # tail /var/log/vmkernel.log 
len 3695179776, nb 441 tbz 0, cow 0, newSinceEpoch 441 zla 3, bs 8388608
2013-02-14T04:15:13.635Z cpu0:5826)FS3: 173: <END BADFILE>
2013-02-14T04:16:57.856Z cpu0:2105)FS3Misc: 1465: Long VMFS rsv time on 'MIRROR1' (held for 331 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2013-02-14T04:23:34.691Z cpu1:6190)FS3: 171: <START BADFILE>
2013-02-14T04:23:34.691Z cpu1:6190)Lock [type 10c00001 offset 10063872 v 215, hb offset 3579904
gen 133, mode 2, owner 00000000-00000000-0000-000000000000 mtime 480
num 1 gblnum 0 gblgen 0 gblbrk 0]
2013-02-14T04:23:34.692Z cpu1:6190)Addr <4, 3, 170>, gen 206, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600
len 3695179776, nb 441 tbz 0, cow 0, newSinceEpoch 441 zla 3, bs 8388608
2013-02-14T04:23:34.692Z cpu1:6190)FS3: 173: <END BADFILE>
/vmfs/volumes/4ed306ea-ac7ce83d-0559-001731f8a1de/ISO Images # lsof | grep BADFILE 
/vmfs/volumes/4ed306ea-ac7ce83d-0559-001731f8a1de/ISO Images # vmkvsitools lsof | grep BADFILE 
/vmfs/volumes/4ed306ea-ac7ce83d-0559-001731f8a1de/ISO Images # ls
BADFILE                      Windows_Server_2012_x64.iso
/vmfs/volumes/4ed306ea-ac7ce83d-0559-001731f8a1de/ISO Images # rm -f BADFILE 
rm: can't remove 'BADFILE': Device or resource busy
share|improve this question
    
have you try to stopped all services on host and then delete the file ?, or try to disconnect the storage from other host if possible, then stopped all services and delete the file –  Rikih Feb 14 '13 at 5:35
add comment

3 Answers

The file was likely locked by an individual VM. Being an .ISO file, was it possibly attached to one of the running virtual machines?

You resolved this by entering maintenance mode (thus shutting down all of the running VM's). The same thing could have been accomplished without downtime by opening the virtual machines' settings and removing the .ISO from the CDROM device field.

share|improve this answer
add comment

The file is locked, you have to try to release the lock. The vmkernel log indicates that the MAC address of the owner identifier is all zeros, the last 12 zeros preceding mtime.

gen 133, mode 2, owner 00000000-00000000-0000-000000000000 mtime 480

According to this document from Vmware, if the owner identifier is all zeroes, it is possible that it is a Service Console-based lock, an NFS lock, or a lock generated by another system or product that can use or read VMFS file systems, or a snapshot file of the virtual machine.. In such circumstances, you might try restarting the management agents. One way of doing this is:

 ./sbin/services.sh restart
share|improve this answer
    
I tried to restart the services as you suggested - no effect. I even rebooted the host. Still can't delete it. –  Generation D Systems Feb 14 '13 at 5:14
add comment

I finally put the server into Maintenance Mode and then could delete it.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.