Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have used cpuset to shield some cpus for exclusive use by some realtime threads.

Displaying the cpuset config with the test app RealtimeTest1 running and its tasks moved into the cpusets:

$ cset set --list -r
cset: 
         Name         CPUs-X    MEMs-X Tasks Subs Path
 ------------   ---------- - ------- - ----- ---- ----------
         root         0-23 y     0-1 y   279    2 /
       system 0,2,4,6,8,10 n       0 n   202    0 /system
       shield 1,3,5,7,9,11 n       1 n     0    2 /shield
RealtimeTest1      1,3,5,7 n       1 n     0    4 /shield/RealtimeTest1
      thread1            3 n       1 n     1    0 /shield/RealtimeTest1/thread1
      thread2            5 n       1 n     1    0 /shield/RealtimeTest1/thread2
         main            1 n       1 n     1    0 /shield/RealtimeTest1/main

I can interrogate the cpuset filesystem to show that my tasks are supposedly pinned to the cpus I requested:

/cpusets/shield/RealtimeTest1 $ for i in `find -name tasks`; do echo $i; cat $i; echo "------------"; done

./thread1/tasks
17651 
------------
./main/tasks
17649 
------------
./thread2/tasks
17654
------------

Further, if I use sched_getaffinity, it reports what cpuset does - that thread1 is on cpu 3 and thread2 is on cpu 5.

However, if I run top -p 17649 -H with f,j to bring up the last used cpu, it shows that thread 1 is running on thread 2's cpu, and main thread is running on a cpu in the system cpuset

(Note that thread 17654 is running FIFO, hence thread 17651 is blocked)

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND                                                                                                                                                                                             
17654 root      -2   0 54080  35m 7064 R  100  0.4   5:00.77 3 RealtimeTest                                                                                                                                                                                         
17649 root      20   0 54080  35m 7064 S    0  0.4   0:00.05 2 RealtimeTest                                                                                                                                                                                         
17651 root      20   0 54080  35m 7064 R    0  0.4   0:00.00 3 RealtimeTest    

Also, looking at /proc/17649/task to find the last_cpu each of its tasks ran on:

/proc/17649/task $  for i in `ls -1`; do cat $i/stat | awk '{print $1 " is on " $(NF - 5)}'; done
17649 is on 2
17651 is on 3
17654 is on 3

So cpuset and sched_getaffinity reports one thing, but reality is another

I would say that cpuset is not working?

My machine configuration is:

$ cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 1
$ uname -a
Linux foobar 2.6.32.12-0.7-default #1 SMP 2010-05-20 11:14:20 +0200 x86_64 x86_64 x86_64 GNU/Linux

Update:

Additionally I am parsing /proc/pid/task/tid/stat and calling sched_getcpu() before and after the cset --move call, and I'm also doing a sched_yield() after the pinning to try get the thread to move...

This is an example output:

13:12:56 INFO before pinning thread 17508 reports lastCpu=0, currCpu=1
13:12:56 INFO pinning thread 17508 to cpu 3 (/shield/RealtimeTest1/thread1)
13:12:56 INFO SetAffinity cset response:
    cset: moving following pidspec: 17508
    cset: moving 1 userspace tasks to /shield/RealtimeTest1/thread1
    cset: done
13:12:56 INFO after pinning thread 17508 reports lastCpu=0, currCpu=1
13:12:56 INFO after sch_yld thread 17508 reports lastCpu=0, currCpu=1

So the thread is not moving to its new cpuset immediately, or even after a sched_yield

Could this be a SLES 11 / SP 1 issue?

share|improve this question
    
Unrelated: is Hyperthreading enabled? –  ewwhite Jun 27 '12 at 2:51
    
@ewwhite It is, although I don't specify any of the ht cpus in my cpuset configuration. –  Steve Lorimer Jun 27 '12 at 3:01
add comment

1 Answer

up vote 0 down vote accepted

I have reran the tests on a SLES 11 / SP 2 box, and the pinning works.

As such, I'm going to mark this as an answer, which is: This is an issue related to SP 1

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.