38

Maybe this will sound like dumb question but the way i'm trying to do it doesn't work.

I'm on livecd, drive is unmounted, etc.

When i do backup this way

sudo dd if=/dev/sda2 of=/media/disk/sda2-backup-10august09.ext3 bs=64k

...normally it would work but i don't have enough space on external hd i'm copying to (it ALMOST fits into it). So I wanted to compress this way

 sudo dd if=/dev/sda2 | gzip > /media/disk/sda2-backup-10august09.gz

...but i got permissions denied. I don't understand.

7
  • 2
    Don't. This is not a backup. Check the 'dump' and 'restore' commands.
    – Juliano
    Aug 10, 2009 at 16:07
  • 2
    Juliano, what do you mean by 'this is not backup'?
    – Phil
    Aug 10, 2009 at 21:26
  • 6
    This is not a backup because backups are serious, well-structured and uses proper tools intended to create backups. You are just making a copy of the raw data of a partition. To restore this data, you will need another partition with the same geometry, which is not guaranteed. Also, if you damage a single block of your archive (superblock, inode tables, root directory, etc), you risk losing all your data. With a proper backup this wouldn't happen.
    – Juliano
    Aug 11, 2009 at 2:09
  • 11
    "To restore this data, you will need another partition with the same geometry, which is not guaranteed" Why would he need that, can't he mount the partition image on a loopback device? Aug 12, 2009 at 13:54
  • 1
    ...what would you recommend instead?
    – Phil
    Aug 13, 2009 at 17:16

6 Answers 6

53

Do you have access to the sda2-backup...gz file? Sudo only works with the command after it, and doesn't apply to the redirection. If you want it to apply to the redirection, then run the shell as root so all the children process are root as well:

sudo bash -c "dd if=/dev/sda2 | gzip > /media/disk/sda2-backup-10august09.gz"

Alternatively, you could mount the disk with the uid / gid mount options (assuming ext3) so you have write permissions as whatever user you are. Or, use root to create a folder in /media/disk which you have permissions for.

Other Information that might help you:

  • The block size only really matters for speed for the most part. The default is 512 bytes which you want to keep for the MBR and floppy disks. Larger sizes to a point should speed up the operations, think of it as analogous to a buffer. Here is a link to someone who did some speed benchmarks with different block sizes. But you should do your own testing, as performance is influenced by many factors. Take also a look at the other answer by andreas
  • If you want to accomplish this over the network with ssh and netcat so space may not be as big of an issue, see this serverfault question.
  • Do you really need an image of the partition, there might be better backup strategies?
  • dd is a very dangerous command, use of instead of if and you end up overwriting what you are trying to backup!! Notice how the keys o and i are next to each other? So be very very very careful.
4
  • i'll try this. how do i also make it bs=64k? (and do i have to?)
    – Phil
    Aug 10, 2009 at 15:40
  • The bs=64k only makes the transfer go faster because dd will be reading blocks of 64k each instead of the default block size of (I don't remember).
    – chris
    Aug 10, 2009 at 16:44
  • What chris said, and if you want to include it put it after dd and before the pipe symbol ( | ) as it is an argument to dd. Aug 10, 2009 at 16:49
  • 1
    I also occasionally will use "sudo tee $file > /dev/null" in a pipeline to allow writing to a file that my user account doesn't have access too. Sep 30, 2015 at 4:25
8

In the first case, dd is running as root. In the second case, dd is running as root but gzip is running as you.

Change the permissions on /media/disk, give yourself a root shell, or run the gzip as root too.

7

In addition, you can replace gzip with bzip2 --best for much better compression:

sudo dd if=/dev/sda2 | bzip2 --best > /media/disk/$(date +%Y%m%d_%H%M%S)_sda2-backup.bz2
6
  • 7
    At a cost of lots of time. See changelog.complete.org/archives/… "How to think about compression" for more details.
    – Bill Weiss
    Aug 10, 2009 at 21:03
  • @BillWeiss: Thanks for your comment, very interesting read!
    – andreas
    Oct 18, 2013 at 9:06
  • compression : lzma > bzip2 > gzip .. speed: gzip > bzip2 > lzma . Unless you are publishing the disk image on internet, there is not much benefit for the time , CPU power and memory you are spending for a better compression.
    – user203492
    Oct 19, 2015 at 7:23
  • 2
    And nowadays zstd is pretty good option.
    – Smar
    Mar 4, 2021 at 8:12
  • 1
    As @Smar mentioned, zstd is probably a really good option for you. It's multi-threaded, tunable, trainable (using dictionaries, if you can expect your data to be similar on each run), and gets better compression ratios in the same compute time as gzip, especially for streaming operations like this. Jan 22, 2023 at 6:21
6
sudo dd if=/dev/sda1 bs=32M | 7z a -si  /data/$(date +%Y%m%d_%H%M%S)_sda1-backup.tar.7z

7z utilizes all CPU cores. Also, adding bs=32M or with some other non-default values may significantly speed up the process.

Test results:

root@pentagon:~# dd if=/dev/sda1 | bzip2 -c > /data/$(date +%Y%m%d_%H%M%S)_pentagon-backup-sda1.bz2
12288000+0 records in
12288000+0 records out
6291456000 bytes (6.3 GB) copied, 2033.77 s, 3.1 MB/s
root@pentagon:~# dd if=/dev/sda1 bs=32M | 7z a -si  /data/$(date +%Y%m%d_%H%M%S)_pentagon-backup-sda1.tar.7z

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C,Utf16=off,HugeFiles=on,64 bits,4 CPUs x64)

Creating archive: /data/20210818_104748_pentagon-backup-sda1.tar.7z

Items to compress: 1

5917M + [Content]187+1 records in
187+1 records out
6291456000 bytes (6.3 GB) copied, 1393.34 s, 4.5 MB/s
                   
Files read from disk: 1
Archive size: 818956969 bytes (782 MiB)
Everything is Ok

Almost 2 times faster.

root@pentagon:~# ls -Alh /data
....
-rw-r--r-- 1 root root            1.2G Aug 18 10:40 20210818_100651_pentagon-backup-sda1.bz2
-rw-r--r-- 1 root root            782M Aug 18 11:11 20210818_104748_pentagon-backup-sda1.tar.7z
....

And, almost 2 times smaller.

Credits to Igor Pavlov for that.

1
  • Nice, any idea how to do it the other direction? Is 7z x -so file.tar.7z | sudo dd of=/dev/sda1 bs=32M safe to do? (there are two possible extraction x and e, not sure if it is important here)
    – tobiasBora
    Oct 20, 2022 at 12:41
1

This is what I do:

dd if=/dev/nvme0n1p3 | xz -z9eT16 > nvme0n1p3.img.xz

The above assumes that the source partition is nvme0n1p3 and the system has 16 hardware threads (as in an 8C/16T processor); replace these values as appropriate. Note that xz -z9e requires about 1 GB of memory per thread, so you might want to use fewer threads.

To further shrink the resulting image, you can shrink the underlying filesystem (e.g. resize2fs -M /dev/nvme0n1p3), then dd only the actual length of the filesystem (e.g. bs=1G count=20 for 20 GiB). If you do this, be sure to resize it back once you're done backing it up or restoring the image. If the source drive is an SSD, mounting it temporarily to run fstrim prior to taking the image should zero-fill unused areas of the filesystem, making it easier to compress as well.

Shrinking the filesystem also allows you to decompress it for loop-mounting without needing as large a drive, so you can access the files in the image and test the validity of the backup without needing to restore it.

0

For example I made below backup my proxmox VE system itself.

If you want a verbose mode you need to install pv before

sudo apt-get install pv

If you haven't gzip you need to install too:

sudo apt-get install gzip

And finally, make the compress iso file in fast mode, type this:

dd if=/dev/sde | pv | gzip -1 - | dd of=/mnt/pve/backup-drive/backup_file_datetimestamp.gz

where:

if=/dev/sde - my whole sdd drive with proxmox

of=/mnt/pve/backup-drive/backup_file_datetimestamp.gz - my backup drive in proxmox VE system

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .