Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've been using LVM snapshots for backing up MySQL databases. FLUSH TABLES WITH READ LOCK is issued and then lvcreate --snapshot --size 4G etc. Because the database is live while the snapshot is active the snap_percent (amount of snapshot storage being used to track deltas with the original state of the filesystem at the time the snapshot was taken) starts increasing. This snap_percent is monitored day-over-day and the --size is increased in the event it reaches 80%.

My question is whether or not there is an equivalent statistic or property in ZFS for determining how much space is being consumed by a snapshot as a percentage of space remaining in the pool? Obviously I don't need to pass a --size parameter to zfs snapshot but how can I determine whether the clone based upon that snapshot is nearing the limits of the pool.

Hopefully that makes sense, it sure sounds like a convoluted question now that I read.

share|improve this question
The space used by a snapshot is not part of the space remaining in the pool so asking for a percentage looks meaningless. It will be infinite if the pool is full. Can you clarify what you are asking with examples and expected values ? –  jlliagre Sep 14 '13 at 15:05
Do not confuse clones and snapshots. Snapshots are simply point in time holds on blocks - the only space they use is delta afterwards. CLONES, which are based off a snapshot but are now writeable, show up as new datasets equivalent to the parent of the snapshot -- eg: they'll be a new zvol or a new filesystem - not a snapshot themselves (in fact, you can then take snapshots of the clone, after you make it). Like snapshots, clone space usage is delta -- but in this case, it's delta it causes itself since it's writeable as opposed to delta between 'then & now' as with a snapshot. –  Nex7 Sep 14 '13 at 15:28
@Nex7 I couldn't tell who you were talking to, but yes I'm aware of the difference hence my statement "but how can I determine whether the clone based upon that snapshot is nearing the limits of the pool." –  Joe Sep 14 '13 at 15:33
First, I tend to just use 'zpool list' to get an idea for actual disk usage. I do this because zpool list doesn't make allowances for reservation/refreservations and such, and is just telling you the % of the disks actually used. If that's not too high, I don't have to to any further accounting work. :) If it is, and I need to know, then it starts to be time to look at zfs list -o name,used,refer,avail,quota,refquota,reserv,refreserv,usedbysnapshot,usedbydatas‌​et,usedbychildren,usedbyrefreservation or something similar, and reading through it carefully. –  Nex7 Sep 14 '13 at 15:39
In the case of a clone, you need to determine both how much space the clone itself has eaten up in delta from its origin snapshot -- and separately, you need to look and see how much delta the snapshot is holding due to changes in its parent. These are separate, I believe, and you'll need to look at both. Also a word of advice: when playing around with this, some people get confused running their own little test scenarios, because they don't realize the accounting is only updated during txg's - give it 10-30 seconds after committing any action to actually be accurately portrayed. –  Nex7 Sep 14 '13 at 15:43

2 Answers 2

ZFS snapshot space is reflected in the filesystem's consumption. You can derive what you're asking for by monitoring the most appropriate fields below.

In the end, you'll watch your filesystem's "avail" space... See how "used"+"avail" is less than "size"?:

root@deore:~# df -h /volumes/vol1/LA_Specialty
Filesystem             size   used  avail capacity  Mounted on
vol1/LA_Specialty      800G   391G   254G    61%    /volumes/vol1/LA_Specialty

I've filtered the output of zfs get all pool/filesystem below to show the relevant properties. Below, I have an 800GB filesystem (quota) where 545GB is used. 391GB is referenced, meaning that's the size of the real data. 154GB is used by snapshots.

root@deore:/volumes# zfs get all vol1/LA_Specialty
NAME               PROPERTY              VALUE                       SOURCE
vol1/LA_Specialty  type                  filesystem                  -
vol1/LA_Specialty  creation              Sat Sep 24 18:44 2011       -
vol1/LA_Specialty  used                  545G                        -
vol1/LA_Specialty  available             255G                        -
vol1/LA_Specialty  referenced            391G                        -
vol1/LA_Specialty  compressratio         2.96x                       -
vol1/LA_Specialty  quota                 800G                        local
vol1/LA_Specialty  reservation           none                        default
vol1/LA_Specialty  recordsize            16K                         local
vol1/LA_Specialty  mountpoint            /volumes/vol1/LA_Specialty  inherited from vol1
vol1/LA_Specialty  usedbysnapshots       154G                        -
vol1/LA_Specialty  usedbydataset         391G                        -
vol1/LA_Specialty  usedbychildren        0                           -
vol1/LA_Specialty  usedbyrefreservation  0                           -

Then looking at the snapshots... It's possible to see the individual size of the snapshots and the total data size that they reference.

root@deore:/volumes# zfs list -t snapshot      
NAME                                               USED  AVAIL  REFER  MOUNTPOINT
vol1/LA_Specialty@snap-daily-1-2013-09-07-020003  57.6G      -   389G  -
vol1/LA_Specialty@snap-daily-1-2013-09-08-020003  1.95G      -   391G  -
vol1/LA_Specialty@snap-daily-1-2013-09-09-020008  3.42G      -   392G  -
vol1/LA_Specialty@snap-daily-1-2013-09-10-020003  3.05G      -   391G  -
vol1/LA_Specialty@snap-daily-1-2013-09-11-020003  2.81G      -   391G  -
vol1/LA_Specialty@snap-daily-1-2013-09-12-020004  2.65G      -   391G  -
vol1/LA_Specialty@snap-daily-1-2013-09-13-020003  2.70G      -   391G  -
vol1/LA_Specialty@snap-daily-1-2013-09-14-020003    25K      -   391G  -
vol1/LA_Specialty@snap-daily-1-latest               25K      -   391G  -

And a du listing of the snapshot directory...

root@deore:/volumes/vol1/LA_Specialty/.zfs/snapshot# du -skh *
 389G   snap-daily-1-2013-09-07-020003
 391G   snap-daily-1-2013-09-08-020003
 392G   snap-daily-1-2013-09-09-020008
 391G   snap-daily-1-2013-09-10-020003
 391G   snap-daily-1-2013-09-11-020003
 391G   snap-daily-1-2013-09-12-020004
 391G   snap-daily-1-2013-09-13-020003
 391G   snap-daily-1-2013-09-14-020003
 391G   snap-daily-1-latest
share|improve this answer
Thanks for the detailed response - I kept playing around and reading Oracle's ZFS documentation and cobbled together zpool list -o capacity,size mysql to get output like CAP SIZE 27% 31.8G. What do you think about monitoring that capacity percentage while the clone is active? –  Joe Sep 14 '13 at 15:03
@ewwhite's answer is to your question about monitoring snapshot delta/disk usage. A snapshot in ZFS is not a clone. A snapshot in ZFS does not act like a snapshot in LVM, either, and should not be thought of the same way. As for monitoring it, yes, you should keep an eye on overall pool usage. If your origin dataset has a lot of delta, the snapshots created on it will hold ever more blocks, eating up ever more space. If you start finding yourself approaching 80% on pool capacity, start destroying snapshots. Don't let a zpool get over 80% used if you can help it. –  Nex7 Sep 14 '13 at 15:34

ZFS snapshots have a lot of hidden data in them. Generally I would refer you to

zfs list -ro space

Which shows an output similar to:

rootpool/export/home                 6.37G  11.7G     2.80G   8.87G              0          0
rootpool/export/home@weekly.2            -   134M         -       -              -          -
rootpool/export/home@weekly.1            -   320M         -       -              -          -
rootpool/export/home@weekly.0            -   251M         -       -              -          -
rootpool/export/home@daily.6             -  1.02M         -       -              -          -
rootpool/export/home@daily.5             -  1.04M         -       -              -          -
rootpool/export/home@daily.4             -   850K         -       -              -          -
rootpool/export/home@daily.3             -   747K         -       -              -          -
rootpool/export/home@daily.2             -   326K         -       -              -          -
rootpool/export/home@daily.1             -   454K         -       -              -          -
rootpool/export/home@daily.0             -   319K         -       -              -          -

This will tell you that I am using a TOTAL of 11.7G on this particular dataset and that 2.8G is used by snaps and 8.87 is used by the actual filesystem (active data). However, the USED size next to each snapshot is very misleading.

If you add up all of the numbers in the used column for the snapshot you will see that they do not come anywhere near the USEDSNAP total. This is because the USED value is how much unique space each snapshot holds.

For example:

If I have a pool named "newpool" and it has 2 1G files (fileA and fileB):

 newpool                    11.0G    2.0G     0.00G   2.0G              0          0

Now I snap that:

 newpool                    11.0G    2.0G     0.00G   2.0G              0          0
 newpool@snap1              11.0G    0.0G     0.00G   2.0G              0          0

Now I delete 1 of the 1G files (fileA):

 newpool                    11.0G    2.0G     1.00G   1.0G              0          0
 newpool@snap1                  -    1.0G         -      -              -          -

Now I create a new 1G file (fileC):

 newpool                    10.0G    3.0G     1.00G   2.0G              0          0
 newpool@snap1                  -    1.0G         -      -              -          -

Now I snap it again

 newpool                    10.0G    3.0G     1.00G   2.0G              0          0
 newpool@snap1                  -    1.0G         -      -              -          -
 newpool@snap2                  -    0.0G         -      -              -          -

Now I delete fileB (which is in both snapshots):

 newpool                    10.0G    3.0G     2.00G   1.0G              0          0
 newpool@snap1                  -    1.0G         -      -              -          -
 newpool@snap2                  -    0.0G         -      -              -          -

Notice how the snapshot USED column did not reflect the change? That's because fileB was referenced by both snapshots and since it is not unique it is not shown in the USED count for any particular snapshot. The USEDSNAP column reflects that the space has been used by the snapshots, but it doesn't associate it to any particular one.

Now if you were to remove snap1:

 newpool                    11.0G    2.0G     1.00G   1.0G              0          0
 newpool@snap2                  -    1.0G         -      -              -          -

snap2 now shows that it has 1.0G used because that data is now unique to that snapshot.

The USED column will show you how much space you can reclaim if you delete that individual snapshot, but doesn't show you truly how much space that snapshot is reserving.

So now that I have said all of that -

If you are planning on only keeping one snapshot of any particular dataset then the zfs list -ro space command should give you what you are looking for.

If you are going to have multiple snapshots at the same time, this data can be misleading. Don't do what comes natural and assume the USED column means anything when dealing with multiple snapshots. Also, du is a poor choice on the snapshot directories since that just shows you what is referenced by the snapshot, not what space the snapshot is actually using.

The zfs manpage goes through some of this, but it isn't great at showing those relationships.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.