0

On my host (RHEL 6.3, 2.6.x kernel) , eth2 generates irq # 60-65. I validated it via /proc/interrupts.

The /proc/irq/61-65/smp_affinity shows following bitsets for 5 queues

  1. 1000

  2. 0040

  3. 0400

  4. 0100

  5. 0010

I am reading them as CPU Core Ids:-

  1. 12 (for bitmask 1000)
  2. 6 (for bitmask 0040)
  3. 10 (for bitmask 0400)
  4. 2 (for bitmask 0004)
  5. 8 (for bitmask 0100)
  6. 4 (for bitmask 0010)

Am i reading them correct?

I am running a network application, which receives 10k messages per second on eth2. Oddly enough, i dont see any of the above CPU Core Ids busy, rather only odd numbered cpu core ids are busy. I am unable to co-relate smp_affinity with cpu activity.

Also, /sys/class/net/eth2/queues/rx-[0-4]/rps_cpus is 0000. Does it mean RPS is disabled?

UPDATE:1 In addition to above, irqbalance daemon is running. Can that override smp_affinity?

UPDATE:2 irqbalance --debug shows following

package 0: containing cpu 0, 2, 4, 6, 8, 10, 12 package 1: containing cpu 1, 3, 5, 7, 9, 11, 13, 15

How does irqbalance uses package information for numa based systems. running lstopo on my system yields Socket 0 Node 0 : Cores 0, 2, 4, 6 Node 1: Cores 8, 10, 12, 14

Socket 1 Node 2: Core 1, 3, 5, 7 Node 3: Core 9, 11, 13, 15

Looking at numa topology and packages in irqbalance, it seems like irqs will be spread across Nodes, but limited to a Socket. Not sure why.

Also, using lspci, i can see eth2 is using pci bus 4 and Node 0 is on pci bus 4. So most normal thing in my mind, would be to spread eth 2 interrupts over Node 0, rather it seems to be spread over Socket 1.

2
  • Which distribution, version and kernel? – ewwhite Apr 24 '13 at 4:44
  • rhel 6.3 (2.6.x kernel) – Jimm Apr 24 '13 at 4:58
1

I am reading them as CPU Core Ids:- … Am i reading them correct?

Since basically those numbers are just powers of 2 for each proc number, you can use this snippet which uses bc to do log2 calculation to get N from powers:

for N in 1000 40 400 100 10; do
    echo -n "$N is "
    echo "ibase=16; l($N) / l(2)" | bc -l | sed -e 's!\.000.*!!g'
done

1000 is 12
40 is 6
400 is 10
100 is 8
10 is 4

UPDATE: In addition to above, irqbalance daemon is running. Can that override smp_affinity?

Sure, in case you want manual control you'd need to disable irqbalance.

4
  • does irqbalance limit itself to smp_affinity or can it override it? – Jimm Apr 24 '13 at 4:59
  • unclear question, @Jimm. – poige Apr 24 '13 at 5:14
  • does irqbalance spread load over CPUs specified by smp_affinity or does irqbalance uses its own configuration and does not refer to smp_affinity – Jimm Apr 24 '13 at 5:17
  • Read the description code.google.com/p/irqbalance — smp_affinity is a knob it pushes – poige Apr 24 '13 at 5:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.