14

I have seen on numerous virtualization services (Azure) and products (vmware,kvm,hyperv) I/O and system stalls under heavy I/O workload.

My questions are:

  • Is it ever sane to use a Virtualized solution when performing I/O heavy workloads?
  • What are the best practices around this sort of stuff?
  • What causes these problems, are there well known system bottlenecks, or is just a question of excessive contention?
5
  • Relevant IBM research paper. I'd also consider using containers, e.g. Docker. From the paper: "Containers can also eliminate the distinction between IaaS and “bare metal” non-virtualized servers [...] since they offer the control and isolation of VMs with the performance of bare metal."
    – oleksii
    Jun 10, 2015 at 12:49
  • 1
    1. What research have you done? We expect you to do a significant amount of research before asking. Showing your research helps others, and helps people give you better answers. See serverfault.com/help/how-to-ask and the help center. 2. This question is quite broad, possibly too broad for this site. Standard advice is: one question per question. 3. This site is not a good fit for questions that call for opinion. Can you ask something more specific and more objectively answerable than "best practices around this sort of stuff"?
    – D.W.
    Jun 10, 2015 at 20:42
  • 1. 10 years of running various virtual machines, kvn, xen, virtualbox, amazon, azure, hyperv. I've found the performance to be generally disastrous but there is so much hype and venture capital floating around that current IT orthodoxy is that everything must be virtualized. Wondering if it were just me, or if other ops professionals are feeling a degree of skepticism too.
    – bryan hunt
    Jun 10, 2015 at 21:42
  • 2
    If you found the performance to be GENERALLY disastrous then - after 10 years of working with virtualization - it is time to accept defeat in being an admin and look for something you can do, when the rest of the world has no problems doing that. Seriously, I think you dramatize here. Or never realizes that virtualization needs buying a decent IO subsystem. I always find if horrific when people load 10+ virtual machines - on a desktop disc and complain about IO speed.
    – TomTom
    Jun 11, 2015 at 5:33
  • It's more of an observation. I don't always have a say in the solution that is chosen. I'm ops. I just keep whatever gets thrown at me up and running.
    – bryan hunt
    Jun 11, 2015 at 13:22

3 Answers 3

19

Is it ever sane to use a Virtualized solution when performing I/O heavy workloads?

Yep, very sane indeed, in fact for most organisations now virtual is the default and doing things on physical boxes is the very much the exception. We have over 100k VMs of all forms and many of them are >40k IOPS with no issue at all.

What are the best practices around this sort of stuff?

The key thing here isn't whether it's virtualised or not - it's understanding your IO needs well and matching the virtual storage resources. It's that simple, if you know what you need/want and have the budget to match that with your storage systems then the virtualisation layer really plays little or no part - unless you're REALLY pushing things of course (I'm talking tens/hundred of millions of IOPs).

What causes these problems, are there well known system bottlenecks, or is just a question of excessive contention?

Lack of understanding or trying to do too much with too little storage resources, that's what normally causes people problems.

10

Is it ever sane to use a Virtualized solution when performing I/O heavy workloads?

Does a database server regularly pulling 1gb/second random IO count? Have one here.

Or a virtual file server delivering up to 600mb/second to a HPC cluster. That one is running off 8 Velicoraptors in a Raid 10, dedicated.

What are the best practices around this sort of stuff?

Provide plenty of IO. I think this SQL VM has around 8 or 10 dedicated SSD.

What causes these problems, are there well known system bottlenecks,

People not doing basic math. If the IO subsystem is not capable of handling the load, it also will not do so under virtualization. Need a LOT of IO - then provide a dedicated storage subsystem of appropriate size.

1
  • 10
    "Provide plenty of UI" - you probably meant IO
    – oleksii
    Jun 10, 2015 at 12:43
2

Besides the basic math & concept that you still need the same IOs as non-virtualised, there is also QOS/prioritisation. Most virtualisation platforms offer at least a basic support for this, will help out a lot to prevent the misbehaving dev VM stalling your prod DB.

1
  • 1
    Yeah. Need 5000IOPS then a hard disc with 350 will not cut it. Load more vms on the same disc and no, it will not magically work. I am always astonished how many people ignore basic math.
    – TomTom
    Jun 10, 2015 at 12:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .