Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm really confused by this. To do some experiments with GlusterFS I created two virtual machines with VirtualBox, running Ubuntu 12.04, each with 10GB of storage. I wrote a script that created lots of little files in lots of folders. Each file was 100k of random data generated by:

dd if=/dev/zero of=#{name} bs=1 count=0 seek=100K

When copying the files from one machine to the other, the recipient run out of space. Which should have been impossible, since they are both the same size and I didn't store anything in them. Trying to figure out what was wrong I found a really confusing situation. This is all on the source machine.

According to df, I'm only using 26% of the space:

root@revisionist:~# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/revisionist-root  9.2G  2.3G  6.5G  26% /
udev                          237M  4.0K  237M   1% /dev
tmpfs                          50M  292K   49M   1% /run
none                          5.0M     0  5.0M   0% /run/lock
none                          246M     0  246M   0% /run/shm
/dev/sda1                     228M   48M  169M  22% /boot

according to du, the whole structure of files I created takes 772M:

root@revisionist:~# du -h files/
...
4.0K    files/3x/ey/hs
8.0K    files/3x/ey
508K    files/3x
772M    files/

which is clearly wrong, as just that folder contains a single 100K file:

root@revisionist:~# ls -lh files/3x/ey/hs
total 0
-rw-r--r-- 1 root root 100K May 19 11:30 3xeyhst4lnkzg5abzeasiw.txt

If I target that directory and file with du, the data is clearly wrong:

root@revisionist:~# du -h files/3x/ey/hs/
4.0K    files/3x/ey/hs/
root@revisionist:~# du -h files/3x/ey/hs/3xeyhst4lnkzg5abzeasiw.txt
0   files/3x/ey/hs/3xeyhst4lnkzg5abzeasiw.txt

This led me to find the option --apparent-size which is documented as:

--apparent-size

print apparent sizes, rather than disk usage; although the apparent size is usually smaller, it may be larger due to holes in (`sparse') files, internal fragmentation, indirect blocks, and the like

and indeed this looks better:

root@revisionist:~# du -h --apparent-size files/3x/ey/hs/3xeyhst4lnkzg5abzeasiw.txt
100K    files/3x/ey/hs/3xeyhst4lnkzg5abzeasiw.txt
root@revisionist:~# du -h --apparent-size files/3x/ey/hs
104K    files/3x/ey/hs

but when I run it for the whole structure I end up with a surprising result:

root@revisionist:~# du -h --apparent-size files
...
104K    files/3x/ey/hs
108K    files/3x/ey
6.8M    files/3x
11G files

How can it be 11G? What's going on? Just rounding up? but then why does it fail to fit on the other machine?

share|improve this question
    
Have you taken into account hard links? –  Mehrdad May 19 at 23:49

1 Answer 1

up vote 4 down vote accepted

If you're looking for the total size of the directory, you should use -s.

See what du -sh files gives you?

Also, how exactly did you create the files, and do they really have content? The differences between the reported sizes might mean the files are sparse.

share|improve this answer
    
This is how I created each file: dd if=/dev/zero of=#{name} bs=1 count=0 seek=100K –  J. Pablo Fernández May 19 at 22:13
1  
@J.PabloFernández Right, so those files you created are all sparse, due to count=0 and seek=100K. There's no data in the files; you're not actually reading any input data from /dev/zero to be written to the file. The total apparent size of your directory tree will be the apparent size of all of the sparse files plus the real size of the directories they're in (this is where that extra gig comes from), while the real size on disk of the tree is just the directories; no real space is being used by the files. –  Shane Madden May 19 at 22:20
    
Ah... right... I was going to use /dev/random and I forgot to change it. Thanks Shane. –  J. Pablo Fernández May 19 at 22:24
1  
@J.PabloFernández Changing to /dev/random alone won't change the results, as you'll need a non-zero count for any data to be read or written - keep that in mind! Also, your files directory is massive - by any chance was the script previously dumping all the files in the same directory? That'll never shrink, delete and re-create it to get that size back down. –  Shane Madden May 19 at 22:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.