1

I am using a CentOS 7 machine as a BackupPC server to pull in backups from my two frontend servers once per night. Last night the backups failed so I checked the server to see where the problem is. During my SSH session I was confronted with error messages regarding my quota/resource limit was exceeded. I couldn't restart any service, I couldn't create a file or folder etc.

[admin@host ~]$ df
Dateisystem    1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
/dev/vzfs      314572800 9919068 304653732    4% /
devtmpfs         2097152       0   2097152    0% /dev
tmpfs            2097152       0   2097152    0% /dev/shm
tmpfs            2097152     128   2097024    1% /run
tmpfs            2097152       0   2097152    0% /sys/fs/cgroup 

Well, plenty of space left on my drive. Let's check the inodes

[admin@host ~]$ df -i
    Dateisystem                  Inodes             IBenutzt      IFrei IUse% Eingehängt auf
    /dev/vzfs      18446744070451413169 18446744069415392088 1036021081  100% /
    devtmpfs                     524288                   58     524230    1% /dev
    tmpfs                        524288                    1     524287    1% /dev/shm
    tmpfs                        524288                  152     524136    1% /run
    tmpfs                        524288                   10     524278    1% /sys/fs/cgroup

That's a crazy number but I am using reiserfs which is not using inodes/adjusts inodes dynamically. (correct me if I am wrong)

[admin@host ~]$ df -T
Dateisystem    Typ      1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
/dev/vzfs      reiserfs 314572800 9919068 304653732    4% /
devtmpfs       devtmpfs   2097152       0   2097152    0% /dev
tmpfs          tmpfs      2097152       0   2097152    0% /dev/shm
tmpfs          tmpfs      2097152     128   2097024    1% /run
tmpfs          tmpfs      2097152       0   2097152    0% /sys/fs/cgroup

To get the system somehow running again I deleted 4 backups and re-run df -i

[admin@host ~]$ df -i
Dateisystem    Inodes IBenutzt  IFrei IUse% Eingehängt auf
/dev/vzfs      800000   792553   7447  100% /
devtmpfs       524288       58 524230    1% /dev
tmpfs          524288        1 524287    1% /dev/shm
tmpfs          524288      150 524138    1% /run
tmpfs          524288       10 524278    1% /sys/fs/cgroup

Pretty cool, everything is working again and I can create files/folders and start services again. So, I started another backup while observing the inodes count and once the magic number of 800000 is reached it immediately explodes to this crazy count above and the quota exceeded error msgs are back. However my backup doesn't include any folders or files that large.

Now my question(s): Is this raise of inodes number expected behavior of reiserfs? And why is my system running out of quota/resource space? Is there any relation?

Here is my quota output:

[admin@host ~]$ quota
Dateisystemquotas für user admin (uid 1000): kein
[admin@host ~]$ quota -g
Dateisystemquotas für group wheel (gid 10): kein
Dateisystemquotas für group admin (gid 1000): kein
  • 1
    This looks like a problem with the /dev/vzfs device or a limitation/quota on that rather on the reiserfs or some sort of interplay. – Thomas Feb 24 '17 at 11:21
  • 2
    You really shouldn't use ReiserFS anymore, and certainly not on a new(ish) deployment with CentOS 7. Better options exist. – Sven Feb 24 '17 at 11:23
  • @Sven Even when using this CentOS 7 system with mainly with BackupPC? I thought reiserfs would be the best option here... -.- – travisbotello Feb 24 '17 at 11:32
  • 1
    No, ReiserFS is essentially dead and isn't developed anymore. XFS is a good alternative, and ZFS is particularly nice for backups due to its snapshot functionality (though I don't know if/how you can make this useful with BackupPC). The same should be true for btrfs but I've never used this and am not fully aware about its current status on CentOS. – Sven Feb 24 '17 at 11:37
  • @Thomas Any ideas on how to debug/identify those limitations/problems? – travisbotello Feb 24 '17 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.