6

I installed Ubuntu 16.04 on vmware esxi 6.7 with 32 core cpu for this machine. Linux only run 8 queue for tx and rx and it cant use all cpus in heavy load.

# ethtool -S ens192 | grep Queue
 Tx Queue#: 0
 Tx Queue#: 1
 Tx Queue#: 2
 Tx Queue#: 3
 Tx Queue#: 4
 Tx Queue#: 5
 Tx Queue#: 6
 Tx Queue#: 7
 Rx Queue#: 0
 Rx Queue#: 1
 Rx Queue#: 2
 Rx Queue#: 3
 Rx Queue#: 4
 Rx Queue#: 5
 Rx Queue#: 6
 Rx Queue#: 7

I config modprob

options vmxnet3 num_rqs=32 num_tqs=32

But Kernel reject options in boot

# dmesg  | grep num_
[    2.626483] vmxnet3: unknown parameter 'num_rqs' ignored
[    2.627238] vmxnet3: unknown parameter 'num_tqs' ignored

This is lspci output

lspci -k
03:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
    DeviceName: Ethernet0
    Subsystem: VMware VMXNET3 Ethernet Controller
    Kernel driver in use: vmxnet3
    Kernel modules: vmxnet3
0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
    DeviceName: Ethernet1
    Subsystem: VMware VMXNET3 Ethernet Controller
    Kernel driver in use: vmxnet3
    Kernel modules: vmxnet3

and driver

# lspci -n | grep "03:00\|0b:00.0"
03:00.0 0200: 15ad:07b0 (rev 01)
0b:00.0 0200: 15ad:07b0 (rev 01)

how can i increase these queues?

3
  • Slightly off-topic, but using that many vCPUs on ESXi (and VM's in general) result is great performance loss due to CPU_WAIT. – Orphans Oct 2 '20 at 13:24
  • Is your kernel up to date? The current kernel for 16.04 LTS is 4.4.0-190-generic. – Michael Hampton Oct 2 '20 at 15:57
  • My kernel is 4.15.0-118-generic – Behrad Eslamifar Oct 3 '20 at 8:27
7

You can change the number of queues at runtime. So, check the limit of queues with ethtool --show-channels ens192, and then change the number with ethtool --set-channels ens192 rx N tx N. Note: the way above isn't applicable to the vmxnet3 driver due by lack of this feature support.

UPDATE: after inspecting of source code of the vmxnet3_driver I've found, that maximum queue count is 8 (https://elixir.bootlin.com/linux/latest/source/drivers/net/vmxnet3/vmxnet3_int.h#L306). The number of queues depends on the cpu count (https://elixir.bootlin.com/linux/latest/source/drivers/net/vmxnet3/vmxnet3_drv.c#L3398). But it cannot exceed the hardcoded value, obviously.

UPDATE2: I've checked the source of vmware-tools. The driver from it has other limits: up to 32 queues (https://github.com/vmware/open-vm-tools/blob/master/open-vm-tools/modules/shared/vmxnet/vmxnet3_defs.h#L636). So you can try to use it.

5
  • # ethtool --show-channels ens192 Channel parameters for ens192: Cannot get device channel parameters : Operation not supported – Behrad Eslamifar Oct 2 '20 at 13:48
  • operation not support – Behrad Eslamifar Oct 2 '20 at 13:49
  • i tested open-vm-tools and vmware-install.pl , both of them limited to 8 and dont detect my real cpu number – Behrad Eslamifar Oct 2 '20 at 14:33
  • 1
    Unfortunately, you don't have ability to use more queues than hardcoded in the driver source. Obviously, you can change this limit and rebuild the module, but it can cause other issues. – Anton Danilov Oct 2 '20 at 14:37
  • The hardcoded limit would probably correspond to the number of IRQs, which would place a hard limit at 32. – Simon Richter Oct 2 '20 at 23:44
1

I asked this problem in open-vm-tools mailinglist

you'll need newer ESXi version (7.0), and virtual machine using hardware version 17 to have vmxnet3 device that supports more than 8 queues.

Then you'll need newer guest driver to be able to use the feature - currently driver in the Linux kernel does not support extended queues.

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.