Centos 7 - fstab does not mount some partitions, but mount /dev/sd* does

Question

i have a Virtual Machine with Centos 7, on which i want to mount some ext4 partitions. Physical disks are actually hard disks provided by vSphere. All disks are located on the same NAS - so the three working ones (a,c,d) are virtually identical to the problematic ones (e,f,g)

content of fstab:

UUID=b6ebbca4-71d0-4d2e-bc1a-e465e5190698   /boot           ext4    defaults        1 2
UUID=2c3ab9f5-60a6-4a79-ada3-84737eef7748   /           ext4    defaults        1 1
UUID=e130758c-5108-44de-bbd8-7e003c9072bc   swap            swap    defaults        0 0
UUID=decbbdb6-8362-41ef-aa72-83066c172913   /home           ext4    defaults        1 2
UUID=5717b613-a9f4-43c9-95d2-cfbbb891bd19   /apps_home          ext4    defaults        1 2
UUID=e24df090-2dda-404c-8944-a28bd37d6c5e   /apps/var/progress  ext4    defaults        1 2
UUID=5f254c77-a91d-4255-8315-9325ddb7a9d8   /apps/var/standard  ext4    defaults        1 2
UUID=746c70c1-002a-4249-a06f-df393a99252c   /apps/var/custom    ext4    defaults        1 2

lsblk -o '+UUID,FSTYPE' output:

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT         UUID                                 FSTYPE
sda       8:0    0   50G  0 disk
├─sda1    8:1    0    1G  0 part /boot              b6ebbca4-71d0-4d2e-bc1a-e465e5190698 ext4
└─sda2    8:2    0   49G  0 part /                  2c3ab9f5-60a6-4a79-ada3-84737eef7748 ext4
sdb       8:16   0   40G  0 disk
└─sdb1    8:17   0   40G  0 part [SWAP]             e130758c-5108-44de-bbd8-7e003c9072bc swap
sdc       8:32   0   50G  0 disk
└─sdc1    8:33   0   50G  0 part /home              decbbdb6-8362-41ef-aa72-83066c172913 ext4
sdd       8:48   0  600G  0 disk
└─sdd1    8:49   0  600G  0 part /apps_home              5717b613-a9f4-43c9-95d2-cfbbb891bd19 ext4
sde       8:64   0  250G  0 disk
└─sde1    8:65   0  250G  0 part /apps/var/progress e24df090-2dda-404c-8944-a28bd37d6c5e ext4
sdf       8:80   0    1T  0 disk
└─sdf1    8:81   0 1024G  0 part /apps/var/standard 5f254c77-a91d-4255-8315-9325ddb7a9d8 ext4
sdg       8:96   0    2T  0 disk
└─sdg1    8:97   0    2T  0 part /apps/var/custom   746c70c1-002a-4249-a06f-df393a99252c ext4
sr0      11:0    1 1024M  0 rom

The issue is that fstab is not mounting the three last partitions (that are supposed to be mounted under /apps/var/progress /apps/var/standard and /apps/var/custom respectively) consistently after reboots. It does mount one of them from time to time (it is totally random if any of these three is mounted, and it is always only one or none that is being mounted). The other partitions are being mounted consistently without any issues.

The mount -a option is not doing anything, however $mount /dev/sde1 (or dev/sdf1 or dev/sdg1) work like a charm. Similarly, mounting by using the command $mount /apps/var/progress works without any issues as too.

For now i just used cron to mount these three partitions after every reboot but in the long run i would like to know the root cause for this strange behaviour.

I tried replacing the UUID's with /dev/sd* names, i tried putting quotation marks.

mount always shows that the partitions are mounted on right catalogues, however umount reports that the partition is not mounted at the moment.

I noticed that the three problematic partitions had ext3 filesystems previously, and were somehow converted into ext4. Also, for these three partitions blkid shows the PARTUUID in addition to UUID (i think they were previously used with centos 6)

Answer 1

In summary from the discussion in the comments:

Your problem was essentially that you used symlinks in your mount point paths and at boot the system was not able to follow those properly to recognise the result as "nested mounts". Therefor systemd did not mount your file-systems in a proper sequential order to handle that dependancy.

• You have a mount point /apps_home

• You have a symlink /apps --> /apps_home/apps

• And you have also try to mount volumes on /apps/var/progress /apps/var/custom and /apps/var/custom

The problem is that the mount points /apps/var/[custom|progress|standard] don't exist until /apps_home is mounted.

Solution:

Leave the symlink but mount your file-systems on the actual directory paths of the symlink target: i.e. convert your fstab entries to:

UUID=5717b613-a9f4-43c9-95d2-cfbbb891bd19   /apps_home                    ext4 defaults 1 2
UUID=e24df090-2dda-404c-8944-a28bd37d6c5e   /apps_home/apps/var/progress  ext4 defaults 1 2
UUID=5f254c77-a91d-4255-8315-9325ddb7a9d8   /apps_home/apps/var/standard  ext4 defaults 1 2
UUID=746c70c1-002a-4249-a06f-df393a99252c   /apps_home/apps/var/custom    ext4 defaults 1 2

systemd-fstab-generato will generate the needed mount unit files and systemd.mount will implicitly add the correct dependancies:

If a mount unit is beneath another mount unit in the file system hierarchy, both a requirement dependency and an ordering dependency between both units are created automatically.

Alternative: remove the entries from /etc/fstab and create your own mount unit files and manually configure the requirement and ordering dependancies to ensure that /apps/var/progress /apps/var/custom and /apps/var/custom don't get mounted before /apps_home.

Answer 2

mount always shows that the partitions are mounted on right catalogues, however umount reports that the partition is not mounted at the moment.

From the discussion in the comments, I believe that your system presents that behavior due to /etc/mtab being a regular file instead of a symbolic link that targets to /proc/self/mount. I suggest you to restore the symlink, as documented in this Red Hat solution:

In Red Hat Enterprise Linux 7, /etc/mtab is no longer a flat file, it is instead a symlink to /proc/self/mounts. If a service for some reason uses the sed command to access or modify /etc/mtab, it is possible that it will remove the symlink, and create a flat file. This in turns causes the server not to fully boot normally it drops to emergency mode, df output shows everything mounted, but if /proc/mounts is checked there will be nothing but / is mounted.