How to write a sparse Linux (EXT4) disk image without writing gigabytes of zeros?

Question

I have a 64 GB Linux disk image with ~50 GB of unused space across the partitions. The file is sparse, so it only takes ~14 GB on disk.

But if I dd the image, it writes the full 64 GB, which takes quite a while.

Is there any way I can do the equivalent of dd if=os.img of=/dev/sdb with this image, without having to write 50 GB of zeros?

Is there any tool that is smart enough to do this i.e. an imaging tool that has an awareness of the EXT4 filesystem?

Answer 1

dd can handle this. You need to add conv=sparse to the command line.

From the man page:

              try to seek rather than write the output for NUL input blocks

Answer 2

I would consider using mksquashfs to create a read-only copy instead of making a dd of whole volume. This additionally compresses and de-duplicates the data.

Please note that you have to evaluate - it's usability depends on your specific use case.

Answer 3

It is awkward since you have a whole disk image but you could:

1. use your partitioning program of choice to list the partitions in os.img and create them on /dev/sdb
2. restore each partition with e2image. e.g., e2image -aro 1048576 os.img /dev/sdb1

This does not handle the MBR if there is one. /bin/dd if=os.img of=/dev/sdb bs=512 count=1 will do that.

Note: the e2image man page at linux.die.net is out of date. From a bionic install:

SYNOPSIS
   e2image [ -r|Q ] [ -f ] device image-file
   e2image -I device image-file
   e2image -ra [ -cfnp ] [ -o src_offset ] [ -O dest_offset ] src_fs [ dest_fs ]

Answer 4

TLDR: Use partclone.

Skipping the sparse blocks is only a good idea if either you can be sure that the disk is filled with zero bytes, e.g. because you just wiped it, or if you know that os.img only has sparse blocks (holes) where the contents do not matter. The latter is not easy to know. An essential block of zero bytes can, for example, come from a filesystem that stores a non-sparse file containing a block of zero bytes. The file os.img may have a hole at this block from copying, running a tool such as fallocate --dig-holes to save space, or automatic de-duplication where the de-duplication source is an existing sparse file.

More likely, you want to write the blocks that are relevant to the filesystem(s) in os.img and only these blocks. You can do this with partclone.

• If os.img has a single filesystem you can directly use it as a source, pipe the partclone backup to a restore process that writes to the disk partition. (partclone has an option --dev-to-dev but the man page does not clarify whether this skips unused blocks.)
• If there are partitions on os.img, you can use losetup --find --show --partscan os.img to access them and then use partclone on each of them.

Answer 5

You can use bmaptool for exactly that purpose. bmaptool has a few advantages over dd, see their web page. You run it in two steps:

1. Create the bmap from the file; the bmap lists blocks which are occupied.
2. Write the file using the generated bmap.

# Create bmap
bmaptool create -o os.img.bmap os.img

# Burn to disk (equivalent to dd if=os.img of=/dev/sdb)
bmaptool copy --bmap os.img.bmap /dev/sdb

I am using this right now with a HD dump taken with ddrescue -S (which creates a sparse dump). The dump is 230 GB, but as sparse file it only uses 23 GB, and bmaptool writes the dump in a bit more than one minute.