7

I have a 64 GB Linux disk image with ~50 GB of unused space across the partitions. The file is sparse, so it only takes ~14 GB on disk.

But if I dd the image, it writes the full 64 GB, which takes quite a while.

Is there any way I can do the equivalent of dd if=os.img of=/dev/sdb with this image, without having to write 50 GB of zeros?

Is there any tool that is smart enough to do this i.e. an imaging tool that has an awareness of the EXT4 filesystem?

1
  • 1
    In general, not writing sparse blocks as zeros to the disk can produce errors: The zeros in sparse regions of os.img can be important. For example, a filesystem that uses a bitmap to keep track of free blocks may have a full bock of zero bytes in this bitmap. This block of zero bytes must go onto the disk. If the disk has other data from previous use in this location and the block is skipped when writing as it is sparse in os.img, the filesystem will be inconsistent and may use some blocks twice, for different files. Feb 23 at 15:59

5 Answers 5

6

dd can handle this. You need to add conv=sparse to the command line.

From the man page:

              try to seek rather than write the output for NUL input blocks
5
  • 1
    I should have mentioned that I did consider conv=sparse but don't think it works in this case. I could be wrong. When dd is creating a file inside a filesystem it can safely seek because EXT4 supports sparse files (i.e. it will treat that seeked-over space as zeros). But when it's writing to a disk device, it's going to seek over random parts of the disk device which could be zeros or junk data, causing corruption. It seems like I would have to dd if=/dev/zero of=/dev/device first to ensure I have all zeros, which would defeat the purpose. Seems like an FS-aware tool is req. Sep 4, 2019 at 20:20
  • @hackerattacker2019 What are you writing the image to? Sep 4, 2019 at 21:58
  • I'm writing the image to a block device (an SSD) e.g. /dev/sdb Sep 4, 2019 at 22:13
  • 1
    If it's really an SSD then it doesn't matter because you'll have trimmed the device first. Sep 4, 2019 at 22:14
  • I am zeroing the drive with ATA security erase, it does read all zeros in my tests. So this does seem like it should work but I'm slightly uneasy ;-) Thanks! Sep 6, 2019 at 17:02
3

I would consider using mksquashfs to create a read-only copy instead of making a dd of whole volume. This additionally compresses and de-duplicates the data.

Please note that you have to evaluate - it's usability depends on your specific use case.

3

It is awkward since you have a whole disk image but you could:

  1. use your partitioning program of choice to list the partitions in os.img and create them on /dev/sdb
  2. restore each partition with e2image. e.g., e2image -aro 1048576 os.img /dev/sdb1

This does not handle the MBR if there is one. /bin/dd if=os.img of=/dev/sdb bs=512 count=1 will do that.

Note: the e2image man page at linux.die.net is out of date. From a bionic install:

SYNOPSIS
   e2image [ -r|Q ] [ -f ] device image-file
   e2image -I device image-file
   e2image -ra [ -cfnp ] [ -o src_offset ] [ -O dest_offset ] src_fs [ dest_fs ]
2
  • The e2image utility does seem like exactly what I'm looking for. Will have to give this a try. Thank you! Sep 6, 2019 at 17:04
  • This is the answer that should be accepted. And because e2image is filesystem aware it won't copy non-zero-filled free blocks, unlike some other tools like ddrescue
    – crass
    Mar 19 at 3:07
1

TLDR: Use partclone.

Skipping the sparse blocks is only a good idea if either you can be sure that the disk is filled with zero bytes, e.g. because you just wiped it, or if you know that os.img only has sparse blocks (holes) where the contents do not matter. The latter is not easy to know. An essential block of zero bytes can, for example, come from a filesystem that stores a non-sparse file containing a block of zero bytes. The file os.img may have a hole at this block from copying, running a tool such as fallocate --dig-holes to save space, or automatic de-duplication where the de-duplication source is an existing sparse file.

More likely, you want to write the blocks that are relevant to the filesystem(s) in os.img and only these blocks. You can do this with partclone.

  • If os.img has a single filesystem you can directly use it as a source, pipe the partclone backup to a restore process that writes to the disk partition. (partclone has an option --dev-to-dev but the man page does not clarify whether this skips unused blocks.)
  • If there are partitions on os.img, you can use losetup --find --show --partscan os.img to access them and then use partclone on each of them.
0

You can use bmaptool for exactly that purpose. bmaptool has a few advantages over dd, see their web page. You run it in two steps:

  1. Create the bmap from the file; the bmap lists blocks which are occupied.
  2. Write the file using the generated bmap.
# Create bmap
bmaptool create -o os.img.bmap os.img

# Burn to disk (equivalent to dd if=os.img of=/dev/sdb)
bmaptool copy --bmap os.img.bmap /dev/sdb

I am using this right now with a HD dump taken with ddrescue -S (which creates a sparse dump). The dump is 230 GB, but as sparse file it only uses 23 GB, and bmaptool writes the dump in a bit more than one minute.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.