I have written a buggy program that has accidentally created about 30M files under /tmp. (The bug was introduced some weeks ago, and it was creating a couple of subdirectories per second.) I could rename /tmp to /tmp2, and now I need to delete the files. The system is FreeBSD 10, the root filesystem is zfs.

Meanwhile one of the drives in the mirror went wrong, and I have replaced it. The drive has two 120GB SSD disks.

Here is the question: replacing the hard drive and resilvering the whole array took less than an hour. Deleting files /tmp2 is another story. I have written another program to remove the files, and it can only delete 30-70 subdirectories per second. It will take 2-4 days to delete all files.

How is it possible that resilvering the whole array takes an hour, but deleting from the disk takes 4 days? Why do I have so bad performance? 70 deletions/second seems very very bad performance.

I could delete the inode for /tmp2 manually, but that will not free up the space, right?

Could this be a problem with zfs, or the hard drives or what?

  • 1
    I'm not a zfs expert, so I can't speak to your performance tuning or what you might do to improve it (that would also take a lot of information and would probably best be done directly by an expert). However, I can say that resilvering happens at the block level, while your deletions happen at the filesystem level. The filesystem will have mostly overhead when deleting a bagillion inode buffers like that. – Spooler Sep 5 '16 at 6:36
  • Please post your df -h and zpool list and zfs list. – ewwhite Sep 5 '16 at 11:49
  • 5
    Written another program: rm -rf /tmp2 will not do the job? – Thorbjørn Ravn Andersen Sep 5 '16 at 14:32
  • 2
    Could you not just reboot? /tmp should be a tmpfs filesystem and is stored in memory. – Blender Sep 5 '16 at 20:44
up vote 31 down vote accepted

Deletes in ZFS are expensive. Even more so if you have deduplication enabled on the filesystem (since dereferencing deduped files is expensive). Snapshots could complicate matters too.

You may be better off deleting the /tmp directory instead of the data contained within.

If /tmp is a ZFS filesystem, delete it and create again.

  • 1
    @nagylzs In that case I would suggest making it a separate ZFS file system. Then you can move the current /tmp out of the way, move a new /tmp into place, and delete the files at the system's leisure. Result: minimal downtime plus a slight performance degredation (mitigatable with ionice, assuming FreeBSD has it) while the delete is running. – Michael Kjörling Sep 5 '16 at 11:28
  • 9
    I was wrong. It was a separate filesystem. Here is what worked: reboot to single user mode, then do "zfs delete zroot/tmp ; zfs create zroot/tmp; chmod 41777 /tmp " – nagylzs Sep 5 '16 at 16:28
  • 6
    It was 5 minutes total downtime. Fantastic! :-) – nagylzs Sep 5 '16 at 16:30
  • 1
    Well, that also speaks to the concern I had, that deleting fikes never frees up space because of snapshots. But tmp will be set up to not make automatic periodic snapshots, right? – JDługosz Sep 5 '16 at 20:32
  • 1
    Actually this was: zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp ; chmod 1777 /zroot/tmp ; zfs set mountpoint=/tmp zroot/tmp ; I'm not sure how to turn off auto snapshots though. There is "zfs set com.sun:auto-snapshot=false" but that works on solaris only, I think. – nagylzs Sep 6 '16 at 10:11

How is it possible that resilvering the whole array takes an hour, but deleting from the disk takes 4 days?

Consider an office building.

Removing all of the computers and furniture and fixings from all the offices on all the floors takes a long time, but leaves the offices immediately usable by another client.

Demolishing the whole building with RDX is a whole lot quicker, but the next client is quite likely to complain about how drafty the place is.

  • 5
    ZFS is not an office building :) – developerbmw Sep 5 '16 at 23:22
  • 9
    @developerbmw there is also not actually a file or folder on there either but we need metaphorical concepts to understand what is going on. – JamesRyan Sep 6 '16 at 14:37
  • 1
    @JamesRyan yep it's actually a nice analogy... I was just being stupid – developerbmw Sep 6 '16 at 22:50

There's a number of things going on here.

First, all modern disk technologies are optimised for bulk transfers. If you need to move 100MB of data, they'll do it much faster if they're in one contiguous block instead of scattered all over the place. SSDs help a lot here, but even they prefer data in contiguous blocks.

Second, resilvering is pretty optimal as far as disk operations goes. You read a massive contiguous chunk of data from one disk, do some fast CPU ops on it, then rewrite it in another big contiguous chunk to another disk. If power fails partway through, no big deal - you'll just ignore any data with bad checksums and carry on as per normal.

Third, deleting a file is really slow. ZFS is particularly bad, but practically all filesystems are slow to delete. They must modify a large number of different chunks of data on the disk and time it correctly (i.e. wait) so the filesystem is not damaged if power fails.

How is it possible that resilvering the whole array takes an hour, but deleting from the disk takes 4 days?

Resilvering is something that disks are really fast at, and deletion is something that disks are slow at. Per megabyte of disk, you only have to do a little bit of resilvering. You might have a thousand files in that space which need to be deleted.

70 deletions/second seems very very bad performance

It depends. I would not be surprised by this. You haven't mentioned what type of SSD you're using. Modern Intel and Samsung SSDs are pretty good at this sort of operation (read-modify-write) and will perform better. Cheaper/older SSDs (e.g. Corsair) will be slow. The number of I/O operations per second (IOPS) is the determining factor here.

ZFS is particularly slow to delete things. Normally, it will perform deletions in the background so you don't see the delay. If you're doing a huge number of them it can't hide it and must delay you.


Appendix: why are deletions slow?

  • Deleting a file requires a several steps. The file metadata must be marked as 'deleted', and eventually it must be reclaimed so the space can be reused. ZFS is a 'log structured filesystem' which performs best if you only ever create things, never delete them. The log structure means that if you delete something, there's a gap in the log and so other data must be rearranged (defragmented) to fill the gap. This is invisible to the user but generally slow.
  • The changes must be made in such a way that if power were to fail partway through, the filesystem remains consistent. Often, this means waiting until the disk confirms that data really is on the media; for an SSD, that can take a long time (hundreds of milliseconds). The net effect of this is that there is a lot more bookkeeping (i.e. disk I/O operations).
  • All of the changes are small. Instead of reading, writing and erasing whole flash blocks (or cylinders for a magnetic disk) you need to modify a little bit of one. To do this, the hardware must read in a whole block or cylinder, modify it in memory, then write it out to the media again. This takes a long time.
  • I don't know about ZFS, but some file-systems allow you to unlink a directory with contents, but have those contents just removed later during a garbage collection/defrag/cleanup phase. Does ZFS have any utilities to do such a lazy deletion perhaps? It will not actually speed up the OP's delete but would likely make it less problematic if it happens implicitly during housekeeping. – Vality Sep 6 '16 at 19:33

How is it possible that resilvering the whole array takes an hour, but deleting from the disk takes 4 days?

It is possible because the two operations work on different layers of the file system stack. Resilvering can run low-level and does not actually need to look at individual files, copying large chunks of data at a time.

Why do I have so bad performance? 70 deletions/second seems very very bad performance.

It does have to do a lot of bookkeeping...

I could delete the inode for /tmp2 manually, but that will not free up the space, right?

I don't know for ZFS, but if it could automatically recover from that, it would likely, in the end, do the same operations you are already doing, in the background.

Could this be a problem with zfs, or the hard drives or what?

Does zfs scrub say anything?

Deleting lots of files is never really a fast operation.

In order to delete a file on any filesystem, you need to read the file index, remove (or mark as deleted) the file entry in the index, remove any other metadata associated with the file, and mark the space allocated for the file as unused. This has to be done individually for each file to be deleted, which means deleting lots of files requires lots of small I/Os. To do this in a manner which ensures data integrity in the event of power failure adds even more overhead.

Even without the peculiarities ZFS introduces, deleting 30 million files typically means over a hundred million separate I/O operations. This will take a long time even with a fast SSD. As others have mentioned, the design of ZFS further compounds this issue.

Ian Howson gives a good answer on why it is slow.

If you delete files in parallel you may see an increase in speed due to the deletion may use the same blocks and thus can save rewriting the same block many times.

So try:

find /tmp -print0 | parallel -j100 -0 -n100 rm

and see if that performs better than your 70 deletes per second.

Very simple if you invert your thinking.

  1. Get a second drive (you seem to have this already)

  2. Copy everything from drive A to drive B with rsync, excluding the /tmp directory. Rsync will be slower than a block copy.

  3. Reboot, using drive B as the new boot volume

  4. Reformat drive A.

This will also defragment your drive and give you a fresh directory (fine, defrag is not so important with an SSD but linearizing your files never hurt anything)

  • First of all copy everything except /tmp? So including /dev and /proc? Secondly, sound a bit kludgy to me, especially on a production server. – Hennes Sep 5 '16 at 11:48
  • I'm assuming he's smart enough to exclude non-files, mounted volumes, and the virtual-memory folder, most of which cannot be guessed here. Or do it from a maintenance boot where none of those things matter. – peter Sep 5 '16 at 13:23
  • I think you could also zfs send/recv (block-level copy) all other file systems except the root file system (where /tmp is located in this case) and copy the remaining data on the root file system manually (excluding /tmp of course). – user121391 Sep 5 '16 at 14:49
  • 2
    That will lose the snapshots and bypass some of the reliability features. Misses the point of using zfs. – JDługosz Sep 5 '16 at 20:34
  • 1
    @JDługosz valid points, but only relevant if the user cares. Sort of like "my backups are corrupted, how to repair?" -> "Do you need any backup files?" -> "No." -> "Reformat". – peter Sep 6 '16 at 14:48

You have 30million entries in an unsorted list. You scan the list for the entry you want to remove and you remove it. Now you have only 29,999,999 entries in your unsorted list. If they are all in /tmp, why not just reboot?


Edited to reflect the information in the comments: Statement of problem: Removing most, but not all, of the 30M+ incorrectly created files in /tmp is taking a long time.
Problem 1) Best way to remove large numbers of unwanted files from /tmp.
Problem 2) Understanding why it is so slow to delete files.

Solution 1) - /tmp is reset to empty at boot by most *nix distributions. FreeBSD however, is not one of them.
Step 1 - copy interesting files somewhere else.
Step 2 - As root

 $ grep -i tmp /etc/rc.conf  
 clear_tmp_enable="YES" # Clear /tmp at startup.  

Step 3 - reboot.
Step 4 - change clear_tmp_enable back to "No".
Unwanted files are now gone as ZFS on FreeBSD has the feature that "Destroying a dataset is much quicker than deleting all of the files that reside on the dataset, as it does not involve scanning all of the files and updating all of the corresponding metadata." so all it has to do at boot time is reset the metadata for the /tmp dataset. This is very quick.

Solution 2) Why is it so slow? ZFS is a wonderful file system which includes such features as constant time directory access. This works well if you know what you are doing, but the evidence suggests that the OP is not a ZFS expert. The OP has not indicated how they were attempting to remove the files, but at a guess, I would say they used a variation on "find regex -exec rm {} \;". This works well with small numbers but does not scale because there are three serial operations going on 1) get the list of available files (returns 30 million files in hash order), 2) use regex to pick the next file to be deleted, 3) tell the OS to find and remove that file from a list of 30 million. Even if ZFS returns a list from memory and if 'find' caches it, the regex still has to identify the next file to be processed from the list and then tell the OS to update its metadata to reflect that change and then update the list so it isn't processed again.

  • 1
    I think you misunderstood the question. I needed to remove most of the files. That is, 30M+ files. – nagylzs Sep 6 '16 at 13:31
  • @nagylzs /tmp is cleared on reboot. If you want to delete most, then you only want to keep some, ie less then half, so copy out the ones you want to keep and then reboot to get rid of the rest. The reason your deletions are so slow is that having large numbers of files in a directory results in a large unsorted list that need to be processed to find the the file to be operated on, which takes time. The only problem here is PEBCAK. – Paul Smith Sep 6 '16 at 14:27
  • Zfs directories are unsorted? I thought zfs specifically handled large directories well. – JDługosz Sep 6 '16 at 16:41
  • Well, /tmp is not cleared, only X related files. At least on FreeBSD. It cannot be cleared anyway on boot, because it would take days for the rc script to delete normally. – nagylzs Sep 6 '16 at 17:26
  • @JDlugosz - ZFS is much better then most, but inode lists (which is all directories are) are unsorted. – Paul Smith Sep 7 '16 at 14:36

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.