4

I'd like to change IRQ SMP affinity for reasons, outlined in this question: CPU0 is swamped with eth1 interrupts

But I can't — I see Input/output error when I try to write to /proc/irq/*/smp_affinity.

Please point me to the HOWTO on the matter. (A formal reference on /proc/irq/*/ would be cool as well.)

Gory details:

Note that this is a VM (PV domU) inside an Ubuntu-based Xen XCP host.

$ uname -a
Linux MYHOST 2.6.38-15-virtual #59-Ubuntu SMP Fri Apr 27 16:40:18 UTC 2012 i686 i686 i386 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 11.04
Release:    11.04
Codename:   natty

$ sudo cat /proc/irq/*/smp_affinity
01
01
01
01
01
80
80
80
80
80
80
40
40
40
40
40
40
20
20
20
20
20
20
10
10
10
10
10
10
08
08
08
08
08
08
04
04
04
04
04
04
02
02
02
02
02
02
01
01
01
01
01
01

Update. The error details:

$ N=$(grep -c processor /proc/cpuinfo)
$ echo $N
8

$ printf %x $((2**N-1))
ff

$ printf %x $((2**N-1)) | sudo tee /proc/irq/*/smp_affinity
fftee: /proc/irq/288/smp_affinity: Input/output error
tee: /proc/irq/289/smp_affinity: Input/output error
tee: /proc/irq/290/smp_affinity: Input/output error
tee: /proc/irq/291/smp_affinity: Input/output error
tee: /proc/irq/292/smp_affinity: Input/output error
tee: /proc/irq/293/smp_affinity: Input/output error
tee: /proc/irq/294/smp_affinity: Input/output error
tee: /proc/irq/295/smp_affinity: Input/output error
tee: /proc/irq/296/smp_affinity: Input/output error
tee: /proc/irq/297/smp_affinity: Input/output error
tee: /proc/irq/298/smp_affinity: Input/output error
tee: /proc/irq/299/smp_affinity: Input/output error
tee: /proc/irq/300/smp_affinity: Input/output error
tee: /proc/irq/301/smp_affinity: Input/output error
tee: /proc/irq/302/smp_affinity: Input/output error
tee: /proc/irq/303/smp_affinity: Input/output error
tee: /proc/irq/304/smp_affinity: Input/output error
tee: /proc/irq/305/smp_affinity: Input/output error
tee: /proc/irq/306/smp_affinity: Input/output error
tee: /proc/irq/307/smp_affinity: Input/output error
tee: /proc/irq/308/smp_affinity: Input/output error
tee: /proc/irq/309/smp_affinity: Input/output error
tee: /proc/irq/310/smp_affinity: Input/output error
tee: /proc/irq/311/smp_affinity: Input/output error
tee: /proc/irq/312/smp_affinity: Input/output error
tee: /proc/irq/313/smp_affinity: Input/output error
tee: /proc/irq/314/smp_affinity: Input/output error
tee: /proc/irq/315/smp_affinity: Input/output error
tee: /proc/irq/316/smp_affinity: Input/output error
tee: /proc/irq/317/smp_affinity: Input/output error
tee: /proc/irq/318/smp_affinity: Input/output error
tee: /proc/irq/319/smp_affinity: Input/output error
tee: /proc/irq/320/smp_affinity: Input/output error
tee: /proc/irq/321/smp_affinity: Input/output error
tee: /proc/irq/322/smp_affinity: Input/output error
tee: /proc/irq/323/smp_affinity: Input/output error
tee: /proc/irq/324/smp_affinity: Input/output error
tee: /proc/irq/325/smp_affinity: Input/output error
tee: /proc/irq/326/smp_affinity: Input/output error
tee: /proc/irq/327/smp_affinity: Input/output error
tee: /proc/irq/328/smp_affinity: Input/output error
tee: /proc/irq/329/smp_affinity: Input/output error
tee: /proc/irq/330/smp_affinity: Input/output error
tee: /proc/irq/331/smp_affinity: Input/output error
tee: /proc/irq/332/smp_affinity: Input/output error
tee: /proc/irq/333/smp_affinity: Input/output error
tee: /proc/irq/334/smp_affinity: Input/output error
tee: /proc/irq/335/smp_affinity: Input/output error

Update. irqbalance is running:

$ sudo service irqbalance status
irqbalance start/running, process 560
7
  • So you're running this inside a Xen virtual machine? Nov 9, 2012 at 5:51
  • Correct. I'll update the question. Nov 9, 2012 at 5:52
  • Note that irqbalance is running. Maybe I'm just doing it wrong? Nov 9, 2012 at 5:57
  • Is this a PV or HVM DomU?
    – Nils
    Nov 11, 2012 at 21:58
  • PV. I'll update the question. Nov 12, 2012 at 14:58

3 Answers 3

2
+50

I do not think that moving interrupts to different CPUs - especially for handling network events - would increase the performance.

The contrary will happen, since the network-code can not be held in a specific CPU any longer.

So as long as you do not experience dropped packets on your network interface, I would say - this is quite normal behaviour for a network that serves many packets.

You need to lower the number of interrupts - moving them around will not help (on the contrary, as I tried to outline).

I see two possible solutions:

  • Raise your MTU-size to lower the number of interrupts
  • PIN the vCPU 0 of the DomU to a dedicated CPU on the Dom0 (that is not used by any other VM or the Dom0).

Update 2012-12-17: Since you asked for authoritative links - I tried to ask a general question When not to use virtualisation - I think this is one of the cases, where you are hitting general VM limits. One of the answers to the question contains a different approach: Use containers, instead of virtualization.

I hope this helps...

6
  • 1
    Thanks, but, would it help or not is outside of the scope of this question. Your help is very welcome here: serverfault.com/questions/446173/… or here: serverfault.com/questions/446081/uneven-cpu-core-utilization Nov 12, 2012 at 14:59
  • @AlexanderGladysh Your core question is how to improve performance. Changing affinity (in the DomU) won`t help. I updated my answer accordingly.
    – Nils
    Nov 13, 2012 at 21:16
  • thank you, I'll look into that. But I'd like to see an authoritative answer on why affinity configuration does not work. It worries me that everyone seems to assume that it is possible, but I can't do that. Nov 13, 2012 at 21:59
  • @AlexanderGladysh I doubt that affinity should work on a vCPU in a PV XEN DomU. I even read somewhere that irq-balance does not make any sense there.
    – Nils
    Nov 13, 2012 at 22:14
  • 1
    @AlexanderGladysh See my updated answer. It is not easy to find in the internet, so I asked here on SF...
    – Nils
    Dec 17, 2012 at 15:28
8

There is a file called Documentation/IRQ-affinity.txt in the Linux source code.

/proc/irq/IRQ#/smp_affinity specifies which target CPUs are permitted
for a given IRQ source. It's a bitmask of allowed CPUs. It's not allowed
to turn off all CPUs, and if an IRQ controller does not support IRQ
affinity then the value will not change from the default 0xffffffff.

The catch here is that the bitmask is in hex. So, if you have N CPUs,

N=$(grep -c processor /proc/cpuinfo)

to enable all IRQs for all CPUs, where you have N CPUs you can

printf %x $((2**N-1)) | sudo tee /proc/irq/*/smp_affinity

Kernel 3.0 and later has a file called smp_affinity_list. This file takes a comma separated list of CPUs or CPU ranges. Valid examples: 0, 2,3,5-7, 0-7. The previous command is equivalent to:

echo 0-$((N-1)) | sudo tee /proc/irq/*/smp_affinity_list
4
  • 2
    Does not work, see update to the question. Nov 9, 2012 at 5:44
  • +1 for that link to kernel's doc ;)
    – laconbass
    Oct 19, 2016 at 20:42
  • 1
    Does not work for me either, I am running on a real machine with no virtualization and the files are not writable. May 12, 2017 at 21:55
  • A bunch of the files just say "Input/output error" and refuse to change CPU core
    – Hubro
    Feb 4, 2021 at 22:43
0

But I'd like to see an authoritative answer on why affinity configuration does not work. It worries me that everyone seems to assume that it is possible, but I can't do that.

Just on this specific question - and I have no idea whether Xen affects this answer - you get the Input/output error message when no device driver currently has a handler installed for this interrupt.

If a driver installs a handler on '105', for example, then directory /proc/irq/105 is created, and contains an smp_affinity, among other things. You can now write to smp_affinity. If you close the device, the directory remains, with the smp_affinity, but you can no longer write to it. Open the device (via open, or whatever eventually calls request_irq in the driver) and you can now write to the file. Only tested on RHEL/SL 6.

If you don't know whether or not the device is currently open, check the directory. It will contain the device name if it's open.

1
  • I have a driver running for 113 and I am still not able to write to smp_affinity. Mar 23, 2023 at 9:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .