2

I'm really confused by this. To do some experiments with GlusterFS I created two virtual machines with VirtualBox, running Ubuntu 12.04, each with 10GB of storage. I wrote a script that created lots of little files in lots of folders. Each file was 100k of random data generated by:

dd if=/dev/zero of=#{name} bs=1 count=0 seek=100K

When copying the files from one machine to the other, the recipient run out of space. Which should have been impossible, since they are both the same size and I didn't store anything in them. Trying to figure out what was wrong I found a really confusing situation. This is all on the source machine.

According to df, I'm only using 26% of the space:

root@revisionist:~# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/revisionist-root  9.2G  2.3G  6.5G  26% /
udev                          237M  4.0K  237M   1% /dev
tmpfs                          50M  292K   49M   1% /run
none                          5.0M     0  5.0M   0% /run/lock
none                          246M     0  246M   0% /run/shm
/dev/sda1                     228M   48M  169M  22% /boot

according to du, the whole structure of files I created takes 772M:

root@revisionist:~# du -h files/
...
4.0K    files/3x/ey/hs
8.0K    files/3x/ey
508K    files/3x
772M    files/

which is clearly wrong, as just that folder contains a single 100K file:

root@revisionist:~# ls -lh files/3x/ey/hs
total 0
-rw-r--r-- 1 root root 100K May 19 11:30 3xeyhst4lnkzg5abzeasiw.txt

If I target that directory and file with du, the data is clearly wrong:

root@revisionist:~# du -h files/3x/ey/hs/
4.0K    files/3x/ey/hs/
root@revisionist:~# du -h files/3x/ey/hs/3xeyhst4lnkzg5abzeasiw.txt
0   files/3x/ey/hs/3xeyhst4lnkzg5abzeasiw.txt

This led me to find the option --apparent-size which is documented as:

--apparent-size

print apparent sizes, rather than disk usage; although the apparent size is usually smaller, it may be larger due to holes in (`sparse') files, internal fragmentation, indirect blocks, and the like

and indeed this looks better:

root@revisionist:~# du -h --apparent-size files/3x/ey/hs/3xeyhst4lnkzg5abzeasiw.txt
100K    files/3x/ey/hs/3xeyhst4lnkzg5abzeasiw.txt
root@revisionist:~# du -h --apparent-size files/3x/ey/hs
104K    files/3x/ey/hs

but when I run it for the whole structure I end up with a surprising result:

root@revisionist:~# du -h --apparent-size files
...
104K    files/3x/ey/hs
108K    files/3x/ey
6.8M    files/3x
11G files

How can it be 11G? What's going on? Just rounding up? but then why does it fail to fit on the other machine?

1
  • Have you taken into account hard links?
    – user541686
    May 19, 2014 at 23:49

1 Answer 1

4

If you're looking for the total size of the directory, you should use -s.

See what du -sh files gives you?

Also, how exactly did you create the files, and do they really have content? The differences between the reported sizes might mean the files are sparse.

4
  • This is how I created each file: dd if=/dev/zero of=#{name} bs=1 count=0 seek=100K May 19, 2014 at 22:13
  • 1
    @J.PabloFernández Right, so those files you created are all sparse, due to count=0 and seek=100K. There's no data in the files; you're not actually reading any input data from /dev/zero to be written to the file. The total apparent size of your directory tree will be the apparent size of all of the sparse files plus the real size of the directories they're in (this is where that extra gig comes from), while the real size on disk of the tree is just the directories; no real space is being used by the files. May 19, 2014 at 22:20
  • Ah... right... I was going to use /dev/random and I forgot to change it. Thanks Shane. May 19, 2014 at 22:24
  • 1
    @J.PabloFernández Changing to /dev/random alone won't change the results, as you'll need a non-zero count for any data to be read or written - keep that in mind! Also, your files directory is massive - by any chance was the script previously dumping all the files in the same directory? That'll never shrink, delete and re-create it to get that size back down. May 19, 2014 at 22:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .