Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm really having a hard time to process these results on my Ubuntu server. I have a major problem with my JBoss server where I get FileNotFoundExceptions along with "No space left on device" errors when JBoss tries to create/move file:

java.io.FileNotFoundException: /data/ftp/statistik/processed/chma10_20140528092555212.xml (No space left on device)

. And I thought "maybe I'm out of disk space", and used df command to figure out how much I have left:

root@ubuntu1:/# df -h
Filsystem            Storlek Anvnt Tillg Anv% Monterat på
/dev/mapper/ubuntu1-root
                       36G   13G   21G  38% /
none                  2,0G  192K  2,0G   1% /dev
none                  2,0G     0  2,0G   0% /dev/shm
none                  2,0G   64K  2,0G   1% /var/run
none                  2,0G     0  2,0G   0% /var/lock
/dev/sda1             228M   23M  193M  11% /boot
/dev/mapper/vgdata-lvdata
                       79G  9,2G   66G  13% /data

And as you can see, I have plenty of space left. And I also checked if I'm out of i-nodes:

root@ubuntu1:/# df -i
Filsystem            Inoder    IAnv   IFria IAnv% Monterat på
/dev/mapper/ubuntu1-root
                     2346512   61992 2284520    3% /
none                  505380     773  504607    1% /dev
none                  507383       1  507382    1% /dev/shm
none                  507383      30  507353    1% /var/run
none                  507383       2  507381    1% /var/lock
/dev/sda1             124496     230  124266    1% /boot
/dev/mapper/vgdata-lvdata
                     10486784  233945 10252839    3% /data

But then i used du:

root@ubuntu1:/# du -s -h /*
7,5M    /bin
23M     /boot
19G     /data
192K    /dev
11G     /eniro
5,3M    /etc
112K    /home
0       /initrd.img
183M    /lib
0       /lib64
16K     /lost+found
12K     /media
4,0K    /mnt
4,0K    /opt
du: kan inte komma åt "/proc/20452/task/20452/fd/3": Filen eller katalogen finns inte
du: kan inte komma åt "/proc/20452/task/20452/fdinfo/3": Filen eller katalogen finns inte
du: kan inte komma åt "/proc/20452/fd/3": Filen eller katalogen finns inte
du: kan inte komma åt "/proc/20452/fdinfo/3": Filen eller katalogen finns inte
0       /proc
18M     /root
8,2M    /sbin
4,0K    /selinux
8,0K    /srv
0       /sys
40K     /tmp
691M    /usr
1,2G    /var
0       /vmlinuz

Notice that /data and /eniro are 30G combined! How is it possible? Do I have a memory leak somewhere? Or is it something else?

----- EDIT 1 -----

Ok, I figured out that /data has its own mount so it's not possible to combine /data and /eniro because they aren't on the same mount. But how come it says 9,2G on the first command when it says 19G on the third on directory /data?

----- EDIT 2 -----

@Pedro Sousa:

root@ubuntu1:# du -h --max-depth=1 /
0       /sys
192K    /dev
4,0K    /opt
40K     /tmp
18M     /root
8,2M    /sbin
du: kan inte komma åt "/proc/24487/task/24487/fd/3": Filen eller katalogen finns inte
du: kan inte komma åt "/proc/24487/task/24487/fdinfo/3": Filen eller katalogen finns inte
du: kan inte komma åt "/proc/24487/fd/3": Filen eller katalogen finns inte
du: kan inte komma åt "/proc/24487/fdinfo/3": Filen eller katalogen finns inte
0       /proc
19G     /data
183M    /lib
11G     /eniro
12K     /media
112K    /home
1,2G    /var
16K     /lost+found
23M     /boot
4,0K    /mnt
4,0K    /selinux
691M    /usr
7,5M    /bin
8,0K    /srv
5,3M    /etc
31G     /




root@ubuntu1:# sudo cat /etc/mtab
/dev/mapper/EnirostatTest1-root / ext4 rw,errors=remount-ro 0 0
proc /proc proc rw,noexec,nosuid,nodev 0 0
none /sys sysfs rw,noexec,nosuid,nodev 0 0
fusectl /sys/fs/fuse/connections fusectl rw 0 0
none /sys/kernel/debug debugfs rw 0 0
none /sys/kernel/security securityfs rw 0 0
none /dev devtmpfs rw,mode=0755 0 0
none /dev/pts devpts rw,noexec,nosuid,gid=5,mode=0620 0 0
none /dev/shm tmpfs rw,nosuid,nodev 0 0
none /var/run tmpfs rw,nosuid,mode=0755 0 0
none /var/lock tmpfs rw,noexec,nosuid,nodev 0 0
/dev/sda1 /boot ext2 rw 0 0
/dev/mapper/vgdata-lvdata /data ext4 rw,errors=remount-ro 0 0
/data/ftp/recordings /data/export/recordings none rw,bind 0 0
rpc_pipefs /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
nfsd /proc/fs/nfsd nfsd rw 0 0





root@ubuntu1:# sudo fdisk -l

Disk /dev/sda: 42,9 GB, 42949672960 byte
255 huvuden, 63 sektorer/spår, 5221 cylindrar
Enheter = cylindrar av 16065 · 512 = 8225280 byte
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Diskidentifierare: 0x0008762a

    Enhet Start     Början        Slut     Block    Id  System
/dev/sda1   *           1          32      248832   83  Linux
Partition 1 slutar inte på cylindergräns.
/dev/sda2              32        5222    41691137    5  Utökad
/dev/sda5              32        5222    41691136   8e  Linux LVM

Disk /dev/sdb: 107,4 GB, 107374182400 byte
255 huvuden, 63 sektorer/spår, 13054 cylindrar
Enheter = cylindrar av 16065 · 512 = 8225280 byte
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Diskidentifierare: 0x58aa8283

    Enhet Start     Början        Slut     Block    Id  System
/dev/sdb1               1       13054   104856223+  83  Linux

Disk /dev/dm-0: 85,9 GB, 85903540224 byte
255 huvuden, 63 sektorer/spår, 10443 cylindrar
Enheter = cylindrar av 16065 · 512 = 8225280 byte
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Diskidentifierare: 0x00000000

Disk /dev/dm-0 innehåller inte en giltig partitionstabell

Disk /dev/dm-1: 38,4 GB, 38394658816 byte
255 huvuden, 63 sektorer/spår, 4667 cylindrar
Enheter = cylindrar av 16065 · 512 = 8225280 byte
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Diskidentifierare: 0x00000000

Disk /dev/dm-1 innehåller inte en giltig partitionstabell

Disk /dev/dm-2: 4294 MB, 4294967296 byte
255 huvuden, 63 sektorer/spår, 522 cylindrar
Enheter = cylindrar av 16065 · 512 = 8225280 byte
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Diskidentifierare: 0x00000000

Disk /dev/dm-2 innehåller inte en giltig partitionstabell






root@ubuntu1:# ls -alh /data
totalt 30K
drwxr-xr-x  6 root root 1,0K 2013-10-01 08:17 .
drwxr-xr-x 23 root root 4,0K 2014-06-05 10:36 ..
drwxr-xr-x  2 stat stat  11K 2014-06-04 23:00 eniro_ftp
drwxrwxrwx  3 root root 1,0K 2013-05-13 14:21 export
drwxrwxrwx  5 root root 1,0K 2013-10-01 08:07 ftp
drwx------  2 root root  12K 2013-05-11 16:30 lost+found
share|improve this question
    
you seem to be using LVM or some partition manager... maybe it's messing with the results? (sorry i'm not very helpful) –  pataluc Jun 5 '14 at 10:52
    
This looks more like a debugging java issues than a sysamin issue perhaps Stack Overflow would be a better place for it? –  Iain Jun 5 '14 at 10:53
    
@Iain I think you are mistaken. There is nothing in the question pointing at a java issue. –  kasperd Jun 5 '14 at 11:17
    
What file system are you using on /data? I have seen a similar symptom in the past, which I at the time suspected to be a btrfs bug. –  kasperd Jun 5 '14 at 11:19
    
@kasperd the error message is a bit of a give away as is lots of disk space. –  Iain Jun 5 '14 at 11:19

1 Answer 1

up vote 1 down vote accepted

I noticed an error when I executed dmesg:

 EXT4-fs warning (device dm-0): ext4_dx_add_entry:1535: Directory index full!

It indicates that the directory that JBoss is trying to move/create files in, has an index that is no longer editable. Out of curiosity, I executed ls -l /data/ftp/statistik/processed/ | wc -l in the directory where I thought should only hold a couple of files consisted of 193074 files. And I moved half of them to another folder and the error didn't appear anymore. So the conclusion is that I had too many files in one directory which lead to its index to be non editable

share|improve this answer
    
A common workaround is to create subdirectories (e.g. a-z) and place the files in one subdirecoty, maybe based on a hash of the file or the filename itself. –  Gerald Schneider Jun 5 '14 at 12:54
    
I'm working on a bash script that takes all files older then 1 month and wrapping them in a tarball, and save the tarball in a different subdirectory –  ClydeFrog Jun 5 '14 at 13:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.