1

I'm pretty sure we aren't the only site using jumbo frames (~9k), right? Well, for those of you that are doing it as well, what are you doing about virtualization? Namely:

  • Xen doesn't support packets over 1500 bytes on bridged interfaces. Assigning a real interface to each VM might work, but is a non-starter for me.
  • KVM will do it if I futz around with the source. Otherwise I can get up to 4k packets. Messing with the source isn't something I really want to do (good-bye upstream patches without rebuilding!)
  • VMWare doesn't mention it either way. Their VSphere pricing turns me off, but maybe I can just do ESX(,i)?

I'm not using jumbo packets for iSCSI or NFS. I'm really moving a ton of data between nodes, and upping my MTUs has helped with speeds there. My platform is CentOS 5.x, and I'd prefer to stay with that, but I suppose other options are possible? You tell me!

Anyone doing something clever I'm not thinking of?

[Edit]

Why do I want this? Well, my existing machines all use MTUs of 9000, and the place where that happens is in our clustering layer. If I add a new machine that doesn't speak jumbo packets it can't join the cluster, and it doesn't work. So while I would love to revisit the issue of "do we actually need jumbo packets?", that's a much bigger project than just bringing a new machine online. New machines have to be able to talk to the cluster. Right now that means deploying on bare hardware, and that sucks.

Improve this question

3 Answers 3

Reset to default
2

For ESXi 4 standard Virtual Switches you have to do this from a CLI. If you use the (unsupported) pseudo-console mode or the (supported) VMA the relevant command is:

esxcfg-vswitch -m 9000 vSwitch0

Replace vSwitch0 with the relevant Virtual Switch ID's and repeat as necessary for all vSwitches that you need to enable for 9K Jumbo frames.

In larger (much larger) environments where you are using Distributed Virtual Switches you can change the MTU from the vSphere Client GUI.

Improve this answer
2
  • Are you doing this in a production environment? If so, how's it working?
    – Bill Weiss
    Jun 4, 2010 at 20:28
  • I'm using it in a few places for iSCSI primarily. Works fine when there are no issues with other infrastructure but it can be painful to get right with some switch vendors. I haven't done enough strict testing to be certain that performance is significantly improved over standard frame sizes but it definitely doesn't make things worse on the systems where I've set it up. For the Guest OS's you'll have to make sure you are using one of the enhanced VMXNet NIC's - Scott Lowe has a good article on this here: blog.scottlowe.org/2008/12/15/enabling-enhanced-vmxnet
    – Helvick
    Jun 4, 2010 at 21:11
0

In my experience, jumbo frames are really far from being usable. The offloading technology is a mess, especially the stuff b-com provides, and the switches can't support it well enough.

I for VMs especially, I'd stick with normal MTU sizes, and improve speeds by using mode-4 bonding or switching to 10G or even infiniband.

Having said that, afaik kvm's virtio_net drivers aren't really limited in speed, so in spite of being 1G, they can easily go beyond, given the bandwidth.

Improve this answer
1
  • I appreciate your input, but as I said in the question, I'm using it now and it works great. The switches do it, speeds are great, etc. I'll address why I want to do it in my question, maybe that will make what I'm asking more obvious.
    – Bill Weiss
    Jun 4, 2010 at 20:06
0

Not a direct answer as such but if you're moving lots of data between multiple nodes have you considered Infiniband? it's great for that kind of thing.

Improve this answer
1
  • We're cheap, and gigabit ethernet is too :) No, we haven't looked into Infiniband. There's been some discussion about 10GE, but it hasn't gone anywhere.
    – Bill Weiss
    Jun 4, 2010 at 21:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .