Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'd like to change IRQ SMP affinity for reasons, outlined in this question: CPU0 is swamped with eth1 interrupts

But I can't — I see Input/output error when I try to write to /proc/irq/*/smp_affinity.

Please point me to the HOWTO on the matter. (A formal reference on /proc/irq/*/ would be cool as well.)

Gory details:

Note that this is a VM (PV domU) inside an Ubuntu-based Xen XCP host.

$ uname -a
Linux MYHOST 2.6.38-15-virtual #59-Ubuntu SMP Fri Apr 27 16:40:18 UTC 2012 i686 i686 i386 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 11.04
Release:    11.04
Codename:   natty

$ sudo cat /proc/irq/*/smp_affinity
01
01
01
01
01
80
80
80
80
80
80
40
40
40
40
40
40
20
20
20
20
20
20
10
10
10
10
10
10
08
08
08
08
08
08
04
04
04
04
04
04
02
02
02
02
02
02
01
01
01
01
01
01

Update. The error details:

$ N=$(grep -c processor /proc/cpuinfo)
$ echo $N
8

$ printf %x $((2**N-1))
ff

$ printf %x $((2**N-1)) | sudo tee /proc/irq/*/smp_affinity
fftee: /proc/irq/288/smp_affinity: Input/output error
tee: /proc/irq/289/smp_affinity: Input/output error
tee: /proc/irq/290/smp_affinity: Input/output error
tee: /proc/irq/291/smp_affinity: Input/output error
tee: /proc/irq/292/smp_affinity: Input/output error
tee: /proc/irq/293/smp_affinity: Input/output error
tee: /proc/irq/294/smp_affinity: Input/output error
tee: /proc/irq/295/smp_affinity: Input/output error
tee: /proc/irq/296/smp_affinity: Input/output error
tee: /proc/irq/297/smp_affinity: Input/output error
tee: /proc/irq/298/smp_affinity: Input/output error
tee: /proc/irq/299/smp_affinity: Input/output error
tee: /proc/irq/300/smp_affinity: Input/output error
tee: /proc/irq/301/smp_affinity: Input/output error
tee: /proc/irq/302/smp_affinity: Input/output error
tee: /proc/irq/303/smp_affinity: Input/output error
tee: /proc/irq/304/smp_affinity: Input/output error
tee: /proc/irq/305/smp_affinity: Input/output error
tee: /proc/irq/306/smp_affinity: Input/output error
tee: /proc/irq/307/smp_affinity: Input/output error
tee: /proc/irq/308/smp_affinity: Input/output error
tee: /proc/irq/309/smp_affinity: Input/output error
tee: /proc/irq/310/smp_affinity: Input/output error
tee: /proc/irq/311/smp_affinity: Input/output error
tee: /proc/irq/312/smp_affinity: Input/output error
tee: /proc/irq/313/smp_affinity: Input/output error
tee: /proc/irq/314/smp_affinity: Input/output error
tee: /proc/irq/315/smp_affinity: Input/output error
tee: /proc/irq/316/smp_affinity: Input/output error
tee: /proc/irq/317/smp_affinity: Input/output error
tee: /proc/irq/318/smp_affinity: Input/output error
tee: /proc/irq/319/smp_affinity: Input/output error
tee: /proc/irq/320/smp_affinity: Input/output error
tee: /proc/irq/321/smp_affinity: Input/output error
tee: /proc/irq/322/smp_affinity: Input/output error
tee: /proc/irq/323/smp_affinity: Input/output error
tee: /proc/irq/324/smp_affinity: Input/output error
tee: /proc/irq/325/smp_affinity: Input/output error
tee: /proc/irq/326/smp_affinity: Input/output error
tee: /proc/irq/327/smp_affinity: Input/output error
tee: /proc/irq/328/smp_affinity: Input/output error
tee: /proc/irq/329/smp_affinity: Input/output error
tee: /proc/irq/330/smp_affinity: Input/output error
tee: /proc/irq/331/smp_affinity: Input/output error
tee: /proc/irq/332/smp_affinity: Input/output error
tee: /proc/irq/333/smp_affinity: Input/output error
tee: /proc/irq/334/smp_affinity: Input/output error
tee: /proc/irq/335/smp_affinity: Input/output error

Update. irqbalance is running:

$ sudo service irqbalance status
irqbalance start/running, process 560
share|improve this question
    
So you're running this inside a Xen virtual machine? –  Michael Hampton Nov 9 '12 at 5:51
    
Correct. I'll update the question. –  Alexander Gladysh Nov 9 '12 at 5:52
    
Note that irqbalance is running. Maybe I'm just doing it wrong? –  Alexander Gladysh Nov 9 '12 at 5:57
    
Is this a PV or HVM DomU? –  Nils Nov 11 '12 at 21:58
    
PV. I'll update the question. –  Alexander Gladysh Nov 12 '12 at 14:58

2 Answers 2

up vote 2 down vote accepted
+50

I do not think that moving interrupts to different CPUs - especially for handling network events - would increase the performance.

The contrary will happen, since the network-code can not be held in a specific CPU any longer.

So as long as you do not experience dropped packets on your network interface, I would say - this is quite normal behaviour for a network that serves many packets.

You need to lower the number of interrupts - moving them around will not help (on the contrary, as I tried to outline).

I see two possible solutions:

  • Raise your MTU-size to lower the number of interrupts
  • PIN the vCPU 0 of the DomU to a dedicated CPU on the Dom0 (that is not used by any other VM or the Dom0).

Update 2012-12-17: Since you asked for authoritative links - I tried to ask a general question When not to use virtualisation - I think this is one of the cases, where you are hitting general VM limits. One of the answers to the question contains a different approach: Use containers, instead of virtualization.

I hope this helps...

share|improve this answer
    
Thanks, but, would it help or not is outside of the scope of this question. Your help is very welcome here: serverfault.com/questions/446173/… or here: serverfault.com/questions/446081/uneven-cpu-core-utilization –  Alexander Gladysh Nov 12 '12 at 14:59
    
@AlexanderGladysh Your core question is how to improve performance. Changing affinity (in the DomU) won`t help. I updated my answer accordingly. –  Nils Nov 13 '12 at 21:16
    
thank you, I'll look into that. But I'd like to see an authoritative answer on why affinity configuration does not work. It worries me that everyone seems to assume that it is possible, but I can't do that. –  Alexander Gladysh Nov 13 '12 at 21:59
    
@AlexanderGladysh I doubt that affinity should work on a vCPU in a PV XEN DomU. I even read somewhere that irq-balance does not make any sense there. –  Nils Nov 13 '12 at 22:14
1  
@AlexanderGladysh See my updated answer. It is not easy to find in the internet, so I asked here on SF... –  Nils Dec 17 '12 at 15:28

There is a file called Documentation/IRQ-affinity.txt in the Linux source code.

/proc/irq/IRQ#/smp_affinity specifies which target CPUs are permitted
for a given IRQ source. It's a bitmask of allowed CPUs. It's not allowed
to turn off all CPUs, and if an IRQ controller does not support IRQ
affinity then the value will not change from the default 0xffffffff.

The catch here is that the bitmask is in hex. So, if you have N CPUs,

N=$(grep -c processor /proc/cpuinfo)

to enable all IRQs for all CPUs, where you have N CPUs you can

printf %x $((2**N-1)) | sudo tee /proc/irq/*/smp_affinity

Kernel 3.0 and later has a file called smp_affinity_list. This file takes a comma separated list of CPUs or CPU ranges. Valid examples: 0, 2,3,5-7, 0-7. The previous command is equivalent to:

echo 0-$((N-1)) | sudo tee /proc/irq/*/smp_affinity_list
share|improve this answer
    
Does not work, see update to the question. –  Alexander Gladysh Nov 9 '12 at 5:44

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.