0

I am very inexperienced in Linux server administration and I got suck... I have a VM running under CentOS centos-release-7-6.1810.2.el7.centos.x86_64. On this VM I am running a test installation of an ActiveMQ-InfluxDB-Grafana stack.

Now my system ran out of disk space. "df" shows the follwing:

/dev/mapper/vg_main-root      16766976 15605268   1161708  94% /
devtmpfs                       8116348        0   8116348   0% /dev
tmpfs                          8133376        0   8133376   0% /dev/shm
tmpfs                          8133376   778780   7354596  10% /run
tmpfs                          8133376        0   8133376   0% /sys/fs/cgroup
/dev/sda1                      1007896   189228    767468  20% /boot
tmpfs                          1626676       12   1626664   1% /run/user/42
tmpfs                          1626676        4   1626672   1% /run/user/0
/dev/mapper/vg_main-influxdb  61856248    32992  61823256   1% /var/lib/influxdb/data

This is the state after having deleted all the InfluxDB-files, which consumed 20G on volume-group vg_main_influxdb. I cleaned as well temporary files on vg_main-root, which brought down the disk usage from 100% to the 94% you see here.

Command vgs shows:

  VG      #PV #LV #SN Attr   VSize   VFree
  vg_main   1   3   0 wz--n- <79.02g    0 

furthermore we have

[root@vmla032 /]# du -h --max-depth=1
184M    ./boot
0       ./dev
0       ./proc
761M    ./run
0       ./sys
42M     ./etc
120M    ./root
1.1G    ./var
36K     ./tmp
4.2G    ./usr
360K    ./home
0       ./mnt
747M    ./opt
0       ./srv
7.0G    .

Result of pvdisplay

[root@vmla032 /]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               vg_main
  PV Size               79.02 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              20229
  Free PE               0
  Allocated PE          20229

Another symptom is that vg_main-root gets filled with data by some process and runs out of disk space. For example log data seems to go there as disk space got freed whe I deleted these files.

How do I get more space? How do I find out which processes clog vg_main-root?

1

Welcome to the community!

Your volume group vg_main has two logical volumes in it. They are called influxdb and root. Since they are on the same volume group and the one you want to grow is root, this is actually pretty easy. You can use the lvresize tool to shrink the influxdb volume so you can add that to the root volume. You will required downtime though because we have to take influxdb offline.

Here is an example of what you might do after stopping any service dependent on the influxdb mount.

sudo -i #(become root)
umount /var/lib/influxdb/data
lvresize --resizefs --size -10G /dev/mapper/vg_main-influxdb
lvresize --resizefs --size +10G /dev/mapper/vg_main-root
mount /var/lib/influxdb/data

Then you should be able to check df -h for the results and start your services back up.

|improve this answer|||||
  • @flickery: your answer cured the problem, but only in a symptomatic way. Still I am not clear, why vg_main-root runs out of disk space. Where does the data come from? Why can I increase free space on root when deleting e.g. log files. How do I see which locations/files consume space in root? – WolfiG Mar 27 at 9:21
  • 1
    That's a huge topic. Install ncdu (or use the du command which is harder to work with) and it'll help show you where your largest amount of storage is taken up. Usually /var/log is where you're going to deal with that and it may mean you need to configure some log rotation. There is a program called logrotate which you can configure to rotate logs. One thing to consider is that even after you delete a file, if it is being held open it isn't actually deleted yet. Perhaps a process is writing to it consistently (often true of logs). Use lsof to find these or restart the processes. – flickerfly Mar 27 at 17:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.