Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'd like to create a backup script that uses LVM snapshots during backup. This script will run on several Linux machines. I'd like to avoid creating snapshot that is too small and gets full during backup. I have only volume size, percentage of used and free space on this volume as input parameters. I don't have average disk I/O speed and amount of data changed during backup on this volume. Is there a way to calculate optimal snapshot size for a volume based on this conditions? The available free space to create snapshots is limited.

share|improve this question
    
Are you using several snapshots at the same time? Else, can't you just use all free space in the group? –  Johan Lundberg Aug 22 '13 at 11:41
add comment

2 Answers 2

up vote 3 down vote accepted

Remember that you can resize snapshots on the fly, e.g. with lvextend. So you can give them a sensible initial size and then grow them whenever they get too ful.

This can even be done automatically: Using dmeventd and setting this in lvm.conf:

    # 'snapshot_autoextend_threshold' and 'snapshot_autoextend_percent' define
    # how to handle automatic snapshot extension. The former defines when the
    # snapshot should be extended: when its space usage exceeds this many
    # percent. The latter defines how much extra space should be allocated for
    # the snapshot, in percent of its current size.
    #
    # For example, if you set snapshot_autoextend_threshold to 70 and
    # snapshot_autoextend_percent to 20, whenever a snapshot exceeds 70% usage,
    # it will be extended by another 20%. For a 1G snapshot, using up 700M will
    # trigger a resize to 1.2G. When the usage exceeds 840M, the snapshot will
    # be extended to 1.44G, and so on.
    #
    # Setting snapshot_autoextend_threshold to 100 disables automatic
    # extensions. The minimum value is 50 (A setting below 50 will be treated
    # as 50).

    snapshot_autoextend_threshold = 50
    snapshot_autoextend_percent = 50

Autoextending does not work instantly, it takes a few seconds for dmeventd to react ... and 50% fill grade and 50% growth is pretty harsh, but for testing with a very small snapshot (and thus rapidly filling the snapshot with data) they are needed.

# lvcreate -n TEST_LV -L 1G /dev/base_vg
  Logical volume "TEST_LV" created
# mke2fs -t ext4 /dev/base_vg/TEST_LV
mke2fs 1.42.5 (29-Jul-2012)
[...]
Writing superblocks and filesystem accounting information: done
# mount /dev/base_vg/TEST_LV /mnt

no need to be root to write files

# cd /mnt
# chown USER .
#

$ for i in 1 2 3 4 5 6 7 8 9 10 11 12 ; do 
    dd if=/dev/urandom bs=1024k count=10 > /mnt/File$i
  done
$

# lvcreate -n TEST_LV-SNAP -s /dev/base_vg/TEST_LV -L 25M
  Rounding up size to full physical extent 28.00 MiB
  Logical volume "TEST_LV-SNAP" created
# lvs /dev/base_vg/TEST_LV-SNAP; \
  while true; do 
      lvs /dev/base_vg/TEST_LV-SNAP | 
          grep -v Origin
      sleep 1
  done | uniq 

while this is running, start

$ for i in 1 2 3 4 5 6 7 8 9 10 11 12 ; do
      dd if=/dev/urandom bs=1024k count=10 > /mnt/File$i
      sleep 10
  done

The sleep in the writing is needed to let dmeventd catch up --- IIRC it only checks every 10 seconds.

Back to our output:

  LV                        VG      Attr     LSize  Pool Origin           Data%  Move Log Copy%  Convert
  TEST_LV-SNAP base_vg swi-a-s- 28.00m      TEST_LV   0.04                        
  TEST_LV-SNAP base_vg swi-a-s- 28.00m      TEST_LV   0.04                        
  TEST_LV-SNAP base_vg swi-a-s- 28.00m      TEST_LV  35.90                        
  TEST_LV-SNAP base_vg swi-a-s- 28.00m      TEST_LV  36.01                        
  TEST_LV-SNAP base_vg swi-a-s- 28.00m      TEST_LV  71.86                        
  TEST_LV-SNAP base_vg swi-a-s- 44.00m      TEST_LV  45.82                        
  TEST_LV-SNAP base_vg swi-a-s- 44.00m      TEST_LV  68.63                        
  TEST_LV-SNAP base_vg swi-a-s- 68.00m      TEST_LV  44.46                        
  TEST_LV-SNAP base_vg swi-a-s- 68.00m      TEST_LV  59.22                        
  TEST_LV-SNAP base_vg swi-a-s- 104.00m      TEST_LV  38.75                        
  TEST_LV-SNAP base_vg swi-a-s- 104.00m      TEST_LV  48.40                        
  TEST_LV-SNAP base_vg swi-a-s- 104.00m      TEST_LV  48.43                        
  TEST_LV-SNAP base_vg swi-a-s- 156.00m      TEST_LV  38.74                        
  TEST_LV-SNAP base_vg swi-a-s- 156.00m      TEST_LV  45.17                        
  TEST_LV-SNAP base_vg swi-a-s- 156.00m      TEST_LV  45.19                        
  TEST_LV-SNAP base_vg swi-a-s- 156.00m      TEST_LV  51.63                        
  TEST_LV-SNAP base_vg swi-a-s- 156.00m      TEST_LV  51.65                        
  TEST_LV-SNAP base_vg swi-a-s- 236.00m      TEST_LV  34.14                        
  TEST_LV-SNAP base_vg swi-a-s- 236.00m      TEST_LV  38.39                        
  TEST_LV-SNAP base_vg swi-a-s- 236.00m      TEST_LV  38.40                        
  TEST_LV-SNAP base_vg swi-a-s- 236.00m      TEST_LV  42.66                        
  TEST_LV-SNAP base_vg swi-a-s- 236.00m      TEST_LV  42.67                        
  TEST_LV-SNAP base_vg swi-a-s- 236.00m      TEST_LV  46.92                        
  TEST_LV-SNAP base_vg swi-a-s- 236.00m      TEST_LV  46.94                        
  TEST_LV-SNAP base_vg swi-a-s- 236.00m      TEST_LV  51.19                        
  TEST_LV-SNAP base_vg swi-a-s- 236.00m      TEST_LV  51.20                        
  TEST_LV-SNAP base_vg swi-a-s- 356.00m      TEST_LV  33.94                        

Watch it grow ...

share|improve this answer
    
Great answer! I had no idea about dmeventd before. Is anything required to ensure that it is running, or is it just there as part of LVM? –  Dan Apr 19 '13 at 21:44
    
@Dan Thanks -- first post ever here. :-) On Debian I had to install the dmeventd package, which is neither recommended nor suggested by the package system --- considering that devemapper events may be (IIRC) RAID events too and that this auto-extension feature (also available for thinly provisioned storage, I understand) is only a small part of lvm, this may be understandable. –  Wolfgang Apr 23 '13 at 20:06
    
@Wolfgang Thanks, I did not thought about resizing snapshots on the fly. –  user168584 Apr 29 '13 at 15:58
    
Is there a solution for Debian Squeeze, apart from upgrading all of LVM to the stable/unstable versions? (I've actually implemented a Python script with similar functionality, but polling LVM seems to result in high iowait.) –  Andrew May 13 '13 at 14:24
add comment

No, if you don't know how much data is going to be written while the snapshot is active, you can't possibly determine the 'optimal' size for it. This is the only piece of information that is relevant to finding the right size.

share|improve this answer
    
I think total volume size will also be relevant. –  user168584 Apr 9 '13 at 2:16
    
The total volume size is not relevant. The snapshot is a record of CHANGES made since the snapshot was created. serverfault.com/a/41036/143703 –  yoonix Apr 11 '13 at 22:16
    
I agree that the size of a snapshot will depend on how much data was changed on volume, but the snapshot size will never be bigger than total volume size. For example if you create a snapshot that is equal to the volume size, it will never be full no mater how much data will be changed on the original volume. –  user168584 Apr 12 '13 at 17:51
    
If needing double the space for all of your logical volumes is the 'optimal' size, more power to you and your budget. –  yoonix Apr 12 '13 at 23:14
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.