I am using Dirvish on a Ubuntu server system for backing up a hd to an external usb 3.0 drive. Until a few days ago, everything worked fine, but now every backup fails with "no space left on device (28)" and "file system full". Unfortunately it is not that simple: There is > 500 GB free on the device.
rsync: write "/mnt/backupsys/shd/gesichert1/20130223_213242/tree/<SomeFilename1>.eDJiD9": No space left on device (28) rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: write "/mnt/backupsys/shd/gesichert1/20130223_213242/tree/<SomeFilename2>.RHuUAJ": No space left on device (28) rsync: write "/mnt/backupsys/shd/gesichert1/20130223_213242/tree/<SomeFilename3>.9tVK8Z": No space left on device (28) rsync: write "/mnt/backupsys/shd/gesichert1/20130223_213242/tree/<SomeFilename4>.t3ARSV": No space left on device (28) [... some more files ...] rsync: connection unexpectedly closed (2712185 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
the log looks pretty much as usual until it hits:
<SomeFilename1> <SomeFilename2> <SomeFilename3> <SomeFilename4> <PartOfAFilename>filesystem full write error, filesystem probably full broken pipe RESULTS: warnings = 0, errors = 1
But, as said above, there is lots of space on the device:
df -h /dev/sdg1 2.7T 2.0T 623G 77% /mnt/backupsys/shd
and also there are lots of inodes left:
df -i /dev/sdg1 183148544 2810146 180338398 2% /mnt/backupsys/shd
The device is mounted as rw:
mount /dev/sdg1 on /mnt/backupsys/shd type ext3 (rw)
The process is running as root.
I was about to say that I haven't changed anything but that's not quite true: I have switched on acl for the drive I am backing up:
/dev/md0 on /mnt/md0 type ext4 (rw,acl)
Could that be the problem? If yes, how? root still has full access to the files.
I just checked the temp directories:
- /tmp contains only a .webmin folder that is empty
- /var/tmp is empty
the file system where these directories reside has plenty of free space and inodes:
df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 289G 55G 220G 20% / df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 19202048 167644 19034404 1% /
The directories are quite large, but not > 2 GB. The one where the backup fails is not even one of the largest, it contains 7530 files.
One information which I did not consider relevant when posting this question:
The day before the backups started to fail I had activated acls on the file systems that were backed up. I assume now that this triggered Dirvish (or rsync) to think all the files had changed so the list of files that were to copied rather than hard linked was very large. This could possibly mean that some buffers were too small.
Today a full backup to an empty disk worked flawlessly. I'll try an incremental backup next. This will show whether activating acls was the cause for the problem.