48

When installing Linux VMs in a virtualized environment (ESXi in my case), are there any compelling reasons to partition the disks (when using ext4) rather than just adding separate disks for each mount point?

The only one I can see is that it makes it somewhat easier to see if there's data present on a disk with e.g. fdisk.

On the other hand, I can see some good reasons for not using partitions (for other than /boot, obviously).

  • Much easier to extend disks. It's just to increase disk size for the VM (typically in VCenter), then rescan the device in VM, and resize the file system online.
  • No more issues with aligning partitions with underlying LUNs.

I have not found much on this topic around. Have I missed something important?

Improve this question
4
  • 17
    Oh and I just wanted to comment how impressed myself and some of the other 'high-rep' users of SF were with your first question. We get accused sometimes of beating up new guys but it's really just that a lot of new users don't read what we're about and what we're not - so I thought I should just say thanks for asking an appropriate question in a well written and considered way :)
    – Chopper3
    Oct 2, 2014 at 9:15
  • 5
    I have two remarks though: 1) VMWare is not a product but a company. VMWare ESXi would be a product. 2) I would edit this question to be about virtualized environments in general, as this is equally relevant for e.g. KVM, Xen and HyperV.
    – Sven
    Oct 2, 2014 at 9:21
  • 1
    Thanks. And I have edited the wording to be a bit more general.
    – savoche
    Oct 2, 2014 at 11:20
  • @savoche you should mark an answer.
    – ewwhite
    Nov 15, 2014 at 14:35

10 Answers 10

Reset to default
31

This is an interesting question...

I don't think there's a definitive answer, but I can give some historical context on how best-practices surrounding this topic may have changed over time.

I've had to support thousands of Linux VMs deployed in various forms across VMware environments since 2007. My approach to deployment has evolved, and I've had the unique (sometimes unfortunate) experience of inheriting and refactoring systems built by other engineers.

The old days...

Back in the day (2007), my early VMware systems were partitioned just like my bare metal systems. On the VMware side, I was using split 2GB thick files to comprise the VM's data, and didn't even think about the notion of multiple VMDKs, because I was just happy that virtualization could even work!

Virtual Infrastructure...

By ESX 3.5 and the early ESX/ESXi 4.x releases (2009-2011), I was using Linux, partitioned as normal atop monolithic Thick provisioned VMDK files. Having to preallocate storage forced me to think about Linux design in a similar manner as I would with real hardware. I was creating 36GB, 72GB, 146GB VMDK's for the operating system, partitioning the usual /, /boot, /usr, /var, /tmp, then adding another VMDK for the "data" or "growth" partition (whether that be /home, /opt or something application-specific). Again, the sweet-spot in physical hard disk sizes during this era was 146GB, and since preallocation was a requirement (unless using NFS), I needed to be conservative with space.

The advent of thin provisioning

VMware developed better features around Thin provisioning in later ESXi 4.x releases, and this changed how I began to install new systems. With the full feature set being added in 5.0/5.1, a new type of flexibility allowed more creative designs. Mind you, this was keeping pace with increased capabilities on virtual machines, in terms of how many vCPUS and how much RAM could be committed to individual VMs. More types of servers and applications could be virtualized than in the past. This is right as computing environments were starting to go completely virtual.

LVM is awful...

By the time full hot-add functionality at the VM level was in place and common (2011-2012), I was working with a firm that strove to maintain uptime for their clients' VMs at any cost (stupid). So this included online VMware CPU/RAM increases and risky LVM disk resizing on existing VMDKs. Most Linux systems in this environment were single VMDK setups with ext3 partitions on top of LVM. This was terrible because the LVM layer added complexity and unnecessary risk to operations. Running out of space in /usr, for instance, could result in a chain of bad decisions that eventually meant restoring a system from backups... This was partially process and culture-related, but still...

Partition snobbery...

I took this opportunity to try to change this. I'm a bit of a partition-snob in Linux and feel that filesystems should be separated for monitoring and operational needs. I also dislike LVM, especially with VMware and the ability to do what you're asking about. So I expanded the addition of VMDK files to partitions that could potentially grow. /opt, /var, /home could get their own virtual machine files if needed. And those would be raw disks. Sometimes this was an easier method to expand particular undersized partition on the fly.

Obamacare...

With the onboarding of a very high-profile client, I was tasked with the design of the Linux VM reference template that would be used to create their extremely visible application environment. The security requirements of the application required a unique set of mounts, so worked with the developers to try to cram the non-growth partitions onto one VMDK, and then add separate VMDKs for each mount that had growth potential or had specific requirements (encryption, auditing, etc.) So, in the end, these VMs were comprised of 5 or more VMDKs, but provided the best flexibility for future resizing and protection of data.

What I do today...

Today, my general design for Linux and traditional filesystems is OS on one thin VMDK (partitioned), and discrete VMDKs for anything else. I'll hot-add as necessary. For advanced filesystems like ZFS, it's one VMDK for the OS, and another VMDK that serves as a ZFS zpool and can be resized, carved into additional ZFS filesystems, etc.

Improve this answer
13
  • 2
    Oh geez, thanks for making me feel super old. To me 2007 is still "almost current". :-)
    – Brian Knoblauch
    Oct 3, 2014 at 18:14
  • 1
    Extra VMDKs being added as a mountpoint don't get partitioned.
    – ewwhite
    Oct 4, 2014 at 11:48
  • 1
    @JakovSosic This isn't the place, dude.
    – ewwhite
    Oct 4, 2014 at 18:20
  • 2
    Every technology has it's limits, and nothing on that page supports your claim that LVM is awful. I would recommend that you modify that part of your answer, because it's more of a FUD then beneficial info. PS. sorry if any of my comments sounded harsh, I use to write in between doing actual work so I don't often think about how my words can sound to others.
    – Jakov Sosic
    Oct 4, 2014 at 18:34
  • 5
    "Back in the day" was 2007? I was a complimentary license recipient at IBM in 1999 when version 1 shipped. I'm a VM dinosaur :D (waves @BrianKnoblauch). Per your LVM comments, sounds like you are judging it in context of Linux. LVM mature technology in commercial UNIX land for years before Linux. If you had managed top-end Solaris/Sparc/EMC Symmetrix, Linux was like a step down (and still is in many ways). In the days of small disks, LVM made multi-terabyte databases manageable. I've never had the problems you describe, which really sound like people problems, though I can certainly relate.
    – mrjoltcola
    Oct 6, 2014 at 5:08
7

You're right in many ways, I can see the argument - there is one issue that could prove tricky though. If you use Resource Pools (and I know I don't, hateful things) then VMs can get more IO time if they have more disks - in extreme resource constrained situations a VM with two disks could get twice as much IO resources as one with a single disk. This may well not be an issue to you but I thought I'd point it out.

Edit - oh and it would make snapping slightly slower too, but again that might not be an issue.

Improve this answer
7

When I worked in infrastructure at a particular "large virtualization software company," we often needed to increase the size of a vm's filesystem. We used ext3/4 at the time.

Increasing the virtual disk is very easy, picking up the new device size in a live OS is relatively easy (poke around in /sys), resizing the ext3/4 filesystem live was easy, but what always seemed impossible (to do live) was resizing the partition.

You had to use gparted or rewrite/resize the partition table using fdisk -- but it was always locked by the kernel and required a reboot to get the kernel to pick up the new layout (partprobe didn't do it either.)

I moved many systems to LVM and resizing filesystems became an easy, almost pleasant, experience!

  • Increase the virtual disk image outside the VM
  • In the VM,
    • Poke /sys to rescan the disk metrics (echo "1" > /sys/class/scsi_device//device/rescan)
    • pvresize /dev/sdX (resize the physical volume in LVM)
    • lvresize --extents +100%FREE /dev/VG/lvolXX (resize the logical volume in LVM)
    • resize2fs (resize the filesystem)

All of this could be done safely on a live system -- and no reboot required!

Why not a bare disk? It makes me nervous - I don't feel that bare disks are widely accepted enough yet, but I think we're on the verge of much wider acceptance. There was a thread on the btrfs mailing list related to this:

http://www.spinics.net/lists/linux-btrfs/msg24730.html

But a bare disk would just need the rescan and resize2fs.

So, in summary, yeah, avoid partition tables if you can.

Improve this answer
5
  • 1
    You do not need a reboot to let the Kernel re-read the partition table. But you would need to unmount the file system(s) on the resized device (which is tricky if it is the / partition). Other than that partition tables are rather serving documentary purposes - everybody and his uncle would run an fdisk -l (or the corresponding equivalent) to see what an unknown disk is about. If it is unpartitioned, it easily could be mistaken for "empty" and overwritten. This is the reason why I always create a partition table for disks. LVM is evil, though.
    – the-wabbit
    Oct 3, 2014 at 11:50
  • That is not my experience on these specific VMs, although it has worked in the past on others. Unmounting the fs didn't free up the lock. Maybe it was just Centos5, I dunno. I was stumped. In a partition world, LVM is awesome. In the new btrfs/zfs world, it is obsolete. IMHO, of course.
    – rrauenza
    Oct 3, 2014 at 15:37
  • It took a while for me to realize you were actually using lvm inside the VM... Is there a reason you don't use LVM on the host and simply give the guest an lv to use as disk? The steps for resizing would be: resize volume in host, rescan on guest, resize2fs on guest.
    – GnP
    Oct 6, 2014 at 20:42
  • Yes, inside the vm. Since this is under esx the virtual disk has to be a vmdk file. Yes, theoretically we could have used a raw disk in the guest.
    – rrauenza
    Oct 7, 2014 at 2:17
  • Using a bare disk is so much simpler - removes 2 steps out of 5, with no need to know LVM. Resizing an FS in LVM has been risky though it's getting better: LVM dangers and caveats.
    – RichVel
    Jun 17, 2015 at 13:21
1

While your question as written is about VMWare (ESXi), I'd like to add a situation where I switched back to using partition tables after having the same idea on KVM.

It turned out that if you have LVM volumes as disks for VMs and create an LVM volume group inside the VM without using partitions (using the whole virtual disk as PV), this VG will be visible outside the VM on the host machine. This is not the case if you use partitions as PV.

Granted, it's a corner case but worth considering if you need such a setup.

Improve this answer
2
  • Why would you need a VG on an LV inside the VM? (note I'm rather new to LVM, I'm not judging your way, just trying to grasp the use of such a setup)
    – GnP
    Oct 6, 2014 at 20:45
  • You could use LVM filter on host to filter out nested LVs.
    – Mircea Vutcovici
    Oct 28, 2017 at 18:03
1

Whether it's better to do this or not depends on your system.

There are pros and cons of each setup.

However, the main advantages of a single drive are as follows:

  1. Simplicity: A single drive has a single file, which can be easily distributed and replicated.
  2. Cues to the Host OS: A single file will be treated as a single block of data, and hence the host OS will know that sequences of guest machine access will all be in that one file. This can be achieved on some host OS configurations by simply placing all the drive images in the same file, but it won't necessarily be the case.

However, there are advantages to multi-drive.

  1. Bare metal affinity / manual location: With a single drive you are locked to a single bare-metal affinity of the drive.
  2. Size limitations: If your system has limits on the size of the drive or on files, you could hit them on very big systems.
  3. Read-Only volumes for security: This is the big advantage. If your master volume for the OS is read only on the VM side it provides major security advantages, essentially locking out the ability of programs inside the VM from editing the base OS of the guest. Using a separate data drive allows you to create read-only drives, that can be booted read-write for maintenance and updates without only cleanroom template data, preventing modification of vital OS directories from within the server altogether.
Improve this answer
1
  • Multi-drive also lets you have (on ESXi at least) some disk files in independent mode. That way you can avoid including e.g. temporary data in snaps and snap based backups.
    – savoche
    Oct 2, 2014 at 18:52
1

there is another option: mount the application data on NFS volumes. You require good filers (not all NFS implementations are the same).

When the NFS volumes fill up, expand the volume, the linux client will see the extra space right away.

Your application and vendor must support having its data on NFS, and you need a careful NAS design but so you do with every storage solution for your virtualized environment.

Another bonus point for this approach is that if your storage vendor has snapshotting/cloning technology (like zfs or Netapp) backing the data up and creating test/dev environments is really easy.

Improve this answer
0

The reason why you still need to do partition the disk for some the Linux distributions is due to the fact that there is a bootloader and all the legacy pieces to go with it, i.e. emulated BIOS. This makes it harder to resize a disk and many would end-up using LVM or other similar non-sense.

One can simply make a filesystem on the entire volume and mount it on /, which will work with a very custom (or customisable/non-opinionated) Linux distribution. The last time I tried this with Ubuntu 12.04, the installer didn't know how to handle it as it must install their stupid partition table an all the jazz. This is one the problem of general purpose distributions in the virtualised world.

On the other had, one can actually put partitioning to a less traditional use, for example ChromeOS and CoreOS have two read-only root partitions for system upgrades.

Improve this answer
0

One reason that hasn't been mentioned so far is that in some infrastructures like Google Compute, disk IO performance linearly increases with the size of the disk. In other words, one big partitioned drive will have better IO performance than multiple small drives.

Note that this is not generally the case though. As mentioned by Chopper3 most often multiple drives will have better IO performance. Ultimately if all your virtual drives are mapped to a single physical drive, there should be no difference.

Improve this answer
0

In my experience better approach is to use 1 VMDK for OS and I usually partition it in a following way:

/dev/sda1 - /boot - 256M
/dev/sda2 - swap  - ~4GB
/dev/sda3 - /     - ~8GB

I've found 8GBs to be enough for /, because I usually install minimal Linux distribution (~800MB) + software I need. Logs also go to that partition, but if set up correctly (logrotate for a week), and shipped elsewhere (syslog/elasticsearch) they usually don't pose a treat to fill up the partition.

Data is added as another VMDK, and I usually format filesystem directly over bare disk (eg. /dev/sdb). This allows me to resize the volume in VmWare, and resize it directly in VM without need to repartition/umount/reboot.

Improve this answer
2
  • I like how you've specifically partitioned your swap after the /boot, something I only recently figured out (2008 or so). Keeping even one old bloated kernel image around causes modest /boot parts to stretch out, and feeding sda2 to /boot often gives it enough space. Having it where it is means no relocation of the PV holding root, and that saves a tricky operation that sometimes needs to be done remotely. :-)
    – user2066657
    Jun 16, 2018 at 13:58
  • I start my swap partition at +1G, my /boot is usually 768M, and the rest is mounted as /mnt. /mnt has various subdirs, like usb1, usb2, backup, storage, music, etc. The value comes when you try to copy to an external disk that is no longer plugged in - instead of dumping untold gigabytes into your root partition (possibly in an endless loop), you quickly fill the 256M mount partition and error out in seconds. And the extra space at the end of /boot thingy the last guy said. :)
    – Engineer
    Jul 20, 2023 at 9:08
0

I partition for two reasons:

  1. Documentation - I once had a "trained" EMC administrator steal LUNs right out from under me because they were undocumented and seemed to him to be unallocated, and in the middle of the night, was paged for an Oracle database that suddenly went offline. He had re-provisioned my LUNs for another volume for a unrelated app. Since then I am paranoid about documentation.
  2. Over-provisioning my disk. With platters it keeps data off of the slower cylinders and with SSD, it extends the lifetime / MTBF.
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .