Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I started by creating 16 empty files of exactly 1 billion bytes:

for i in {1..16}; do dd if=/dev/zero of=/mnt/temp/block$i bs=1000000 count=1000 &> /dev/null; done

Then I created larger and larger RAIDZ2 volumes over the files, forcing ashift=12 to simulate a 4K sector drive, e.g.

zpool create tank raidz2 -o ashift=12 /mnt/temp/block1 /mnt/temp/block2...

and then compared using df -B1 to see the actual size.

Filesystem  1B-blocks
tank        12787777536

My results:

+-------+-------------+-------------+------------+------------+
| disks |  expected   |   actual    |  overhead  | efficiency |
+-------+-------------+-------------+------------+------------+
|     3 |  1000000000 |   951975936 |   48024064 | 95.2       |
|     4 |  2000000000 |  1883766784 |  116233216 | 94.2       |
|     5 |  3000000000 |  2892234752 |  107765248 | 96.4       |
|     6 |  4000000000 |  3892969472 |  107030528 | 97.3       |
|     7 |  5000000000 |  4530896896 |  469103104 | 90.6       |
|     8 |  6000000000 |  5541068800 |  458931200 | 92.4       |
|     9 |  7000000000 |  6691618816 |  308381184 | 95.6       |
|    10 |  8000000000 |  7446331392 |  553668608 | 93.1       |
|    11 |  9000000000 |  8201175040 |  798824960 | 91.1       |
|    12 | 10000000000 |  8905555968 | 1094444032 | 89.1       |
|    13 | 11000000000 | 10403577856 |  596422144 | 94.6       |
|    14 | 12000000000 | 11162222592 |  837777408 | 93.0       |
|    15 | 13000000000 | 12029263872 |  970736128 | 92.5       |
|    16 | 14000000000 | 12787908608 | 1212091392 | 91.3       |
+-------+-------------+-------------+------------+------------+

As a chart:

chart of efficiency

  1. Are my results correct, or have I left something out?
  2. If they're correct, why? Where is the space going?
  3. Can I do anything to improve efficiency?
  4. Is there a formula to calculate efficiency?
share|improve this question
    
Why? Just why?!? –  ewwhite Apr 29 '14 at 9:58
    
@ewwhite - There's up to 11% less space than what I expected. If you had, say, 12x1TB drives, you'd expect to have 10TB, but have less than 9TB. –  steveh7 Apr 29 '14 at 20:20
    
No, it's a good question. I don't know the answer. I have large differences in zpool usage between ashift 9 and 12 and the same datasets. –  ewwhite Apr 29 '14 at 20:22
    
Can you re-run these with ashift values ranging from 0 to 12 and plot in 3D? –  Andrew May 7 '14 at 2:28

1 Answer 1

The glib answer, because RAIDZ(2) sucks on 4k drives.

  • Issue #548: Highly inefficient use of space observed when using raidz2 with ashift=12
  • Issue #1807: zvol on RAIDZ2 takes up double the expected space #1807

It has to do with the parity calculations gobbing up a boatload of additional space, especially with smaller files. Try doing the same test against files that are under 50MB if you want to see some real inefficiencies.

share|improve this answer
    
This answer is not actual, the bugs are fixed... –  Arman Feb 27 at 14:39
    
@Arman If you read the comments on 548 it explains why this is the case. The 'bug' in 548 isn't fixed it's just closed; it's not considered a bug. –  RobbieCrash Feb 28 at 20:22

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.