8

I started by creating 16 empty files of exactly 1 billion bytes:

for i in {1..16}; do dd if=/dev/zero of=/mnt/temp/block$i bs=1000000 count=1000 &> /dev/null; done

Then I created larger and larger RAIDZ2 volumes over the files, forcing ashift=12 to simulate a 4K sector drive, e.g.

zpool create tank raidz2 -o ashift=12 /mnt/temp/block1 /mnt/temp/block2...

and then compared using df -B1 to see the actual size.

Filesystem  1B-blocks
tank        12787777536

My results:

+-------+-------------+-------------+------------+------------+
| disks |  expected   |   actual    |  overhead  | efficiency |
+-------+-------------+-------------+------------+------------+
|     3 |  1000000000 |   951975936 |   48024064 | 95.2       |
|     4 |  2000000000 |  1883766784 |  116233216 | 94.2       |
|     5 |  3000000000 |  2892234752 |  107765248 | 96.4       |
|     6 |  4000000000 |  3892969472 |  107030528 | 97.3       |
|     7 |  5000000000 |  4530896896 |  469103104 | 90.6       |
|     8 |  6000000000 |  5541068800 |  458931200 | 92.4       |
|     9 |  7000000000 |  6691618816 |  308381184 | 95.6       |
|    10 |  8000000000 |  7446331392 |  553668608 | 93.1       |
|    11 |  9000000000 |  8201175040 |  798824960 | 91.1       |
|    12 | 10000000000 |  8905555968 | 1094444032 | 89.1       |
|    13 | 11000000000 | 10403577856 |  596422144 | 94.6       |
|    14 | 12000000000 | 11162222592 |  837777408 | 93.0       |
|    15 | 13000000000 | 12029263872 |  970736128 | 92.5       |
|    16 | 14000000000 | 12787908608 | 1212091392 | 91.3       |
+-------+-------------+-------------+------------+------------+

As a chart:

chart of efficiency

  1. Are my results correct, or have I left something out?
  2. If they're correct, why? Where is the space going?
  3. Can I do anything to improve efficiency?
  4. Is there a formula to calculate efficiency?
4
  • Why? Just why?!?
    – ewwhite
    Commented Apr 29, 2014 at 9:58
  • @ewwhite - There's up to 11% less space than what I expected. If you had, say, 12x1TB drives, you'd expect to have 10TB, but have less than 9TB.
    – steveh7
    Commented Apr 29, 2014 at 20:20
  • No, it's a good question. I don't know the answer. I have large differences in zpool usage between ashift 9 and 12 and the same datasets.
    – ewwhite
    Commented Apr 29, 2014 at 20:22
  • Can you re-run these with ashift values ranging from 0 to 12 and plot in 3D?
    – Andrew
    Commented May 7, 2014 at 2:28

1 Answer 1

4

The glib answer, because RAIDZ(2) sucks on 4k drives.

  • Issue #548: Highly inefficient use of space observed when using raidz2 with ashift=12
  • Issue #1807: zvol on RAIDZ2 takes up double the expected space #1807

It has to do with the parity calculations gobbing up a boatload of additional space, especially with smaller files. Try doing the same test against files that are under 50MB if you want to see some real inefficiencies.

2
  • This answer is not actual, the bugs are fixed...
    – Arman
    Commented Feb 27, 2015 at 14:39
  • @Arman If you read the comments on 548 it explains why this is the case. The 'bug' in 548 isn't fixed it's just closed; it's not considered a bug. Commented Feb 28, 2015 at 20:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .