Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

What's the reason behind the difference in reported file sizes?

[root@localhost]# ls -lah sendlog
-rw-rw-r-- 1 mail mail 1.3T Aug 15 17:30 sendlog

[root@localhost]# du -m sendlog
24M  sendlog

This came to our attention when a server's backup kept failing for quota issues, so it wasn't only "ls" which was seeing this wrong size.

Terms like "sparse files" and "block assignment" are coming to mind, but I'm not sure why it would happen or the real reason behind it. Obviously there is a difference in the ways the two commands check size, am I right always trusting du?

FYI, this should be a pretty standard mail log file.

share|improve this question
    
Could it be that the file is still being written to? From a Windows perspective, if I am writing to a large file, the Explorer view will show 0 bytes, whereas a tool like Contig will show the true size. –  user3914 Sep 10 '09 at 8:55
1  
Could you do show us output of df -k, ls -l sendlog (without -h) and stat sendlog ? –  radius Sep 10 '09 at 9:21
1  
And the filesystem type please. –  radius Sep 10 '09 at 9:21
    
The client has now since removed the file in order to make the backups work again, so I can't get any more information. I shall however keep an eye out for it again. –  Coops Sep 11 '09 at 12:53

4 Answers 4

up vote 6 down vote accepted

The difference between the values is as follows.

From the manual of stat(2)

struct stat {
    // snip
    off_t     st_size;    /* total size, in bytes */
    // snip
    blkcnt_t  st_blocks;  /* number of blocks allocated */
    // snip
};

The st_blocks field indicates the number of blocks allocated to the file, 512-byte units. (This may be smaller than st_size/512, for example, when the file has holes.)

The size as reported by ls is st_size, the size as reported by du is st_blocks * 512

The value reported by du is the number of bytes used by the file on the filesystem/disk, and the value reported by ls is the actual size/length of the file when you interact with it. (In addition to operating with on-disk usage, du also only counts hardlilnked files once)

Which value is the "right one" depends on context. If you're after disk-usage du is correct, if you're wondering how many bytes is in the file, ls/st_size is correct.

In addition, you can by using various options get i.e. du (--apparent-size) to use the size reported by st_size or you can get ls (-s) to report the number of blocks used.

Your assumption regarding your logfile beeing a sparse file sounds plausible, however, the reason why I don't know.

share|improve this answer

Just as Kjetil explained, you have a sparse file. Blocks of blank data inside the file are not allocated to disk until you actually write to those blocks. How that happened in a log file is a mystery. You have to check your audit logs from the last time sendlog had a correct size to the time where it had this huge hole. Perhaps the answer is in the log file itself.

Perhaps someone did that intentionally to cause havoc in your system. Or it was some software error.

You can create your own terabyte-sized file easily with:

dd if=/dev/zero of='OMG_Thats_a_1_terabyte_file!!.dat' seek=1T bs=1 count=1

That file will allocate only a few kilobytes of disk space in any current Linux version with a filesystem with support for sparse files.

Your backup solution needs a replacement. Any serious backup system nowadays handle sparse files efficiently. Even the simplest solution using GNU tar support it (-S or --sparse option).

share|improve this answer

Maybe your du doesn't support such large numbers?

share|improve this answer
    
Do you really think that he has a log file of 1.3TB ? .. –  radius Sep 10 '09 at 9:19
    
It's only a 120GB HDD :) –  Coops Sep 10 '09 at 14:28

Your filesystem could be corrupted (or disk has some physical problems). You should do fsck ASAP (on unmounted partition) and see what happens with those numbers afterwards.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.