up vote 0 down vote favorite
2
share [g+] share [fb]

We two OpenSolaris 2009.06 NFS file servers. Each user home directory has a separate filesystem. Home directories are also grouped into separate filesystems. The hourly service auto-snapshot was creating and rolling hourly snapshots successfully for almost a year now. There would be 24 snapshots at any given time.

The Problem

A large sub-tree of filesystems is getting only 1 snapshot in the .zfs/snapshot directory instead of the normal 24. These broken filesystems are getting the latest hourly snapshot but nothing else.

How to find what causes this?

Attempts

There seems to be no difference between the working and the broken filesystems that could explain the problem.

  • I compared ZFS properties, filesystem permissions and ownerships - those are the same.
  • When enabling the Time Slider via GUI Dialog there is an Advanced option that says "Reduce backups when disk capacity is less than 80%" but we are using much less than 80%.
  • Looked at svccfg listprop - a lot of info but nothing there seems to hint at the cause of the problem.

Hierarchy

All filesystems have 24 snapshots, except the zpoola/labi filesystems. Here is the output of zfs list which my comments:

NAME                         USED  AVAIL  REFER 
zpoola                      2.76T  1016G  36.8G 
zpoola/000-reserved-10g     10.0G  1.00T  29.5K 
zpoola/labi                 2.20T  1016G   208G  <--- BAD: only 1 snap
zpoola/labi/userx           84.5M  1016G  84.4M  <--- BAD: only 1 sanp
(153 sub-filsystems)                             <--- 153 BAD: only 1 sanp
...
zpoola/labj                  530G  1016G    40K 
zpoola/labj/usery           37.3G  1016G  36.8G 
(6 sub-filsystems)
...
zpoola/labk                 3.54M  1016G    32K 
zpoola/labk/userz            598K  1016G    26K 
(11 sub-filsystems)
...
zpoolb                      1.80T  1.96T  86.5K 
zpoolb/000-reserved-10g     10.0G  1.97T  29.5K 
zpoolb/labl                  380G  1.96T    41K 
zpoolb/labl/userx            198M  1.96T   197M 
(10 sub-filsystems)
...
zpoolc                      3.09T   109G  1.93T 
zpoolc/000-reserved-10g     7.00G   119G  29.5K 
zpoolc/projectx              143G   109G   143G 
(2 sub-filsystems)
...
link|improve this question
72% accept rate
What does zfs list show the used/available space to be? – Chris S♦ Dec 9 '10 at 15:31
@Chris, I added the output of zfs list above. – Aleksandr Levchuk Dec 9 '10 at 20:08
It must be something with the auto-snapshot service. I don't see anything that looks funny here, might be a bug. I don't know the service well enough to diagnose it. Sorry. – Chris S♦ Dec 10 '10 at 14:17
I added the counts of sub-filesystems. I wonder if the fact that labi has so many children has to do with the problem. Maybe a simple test can be suggested. – Aleksandr Levchuk Dec 13 '10 at 20:13
Does everything work if you generate snapshots by hand? That is, using "zfs snapshot", rather than using Time Slider or other tools? – larsks Dec 13 '10 at 20:41
show 1 more comment
feedback

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
or
required, but never shown
discard

Browse other questions tagged or ask your own question.