isolcpus – binding not working – Managing your servers can streamline the performance of your team by allowing them to complete complex tasks faster. Plus, it can enable them to detect problems early on before they get out of hand and compromise your business. As a result, the risk of experiencing operational setbacks is drastically lower.
But the only way to make the most of your server management is to perform it correctly. And to help you do so, this article will share nine tips on improving your server management and fix some problem about linux, cpu-usage, , , .
I am using isolcpus to isolate cores. I would like to bind specific threads to cores, but it is not working. The threads are moved to different cores after I bind them.
Cores 13, 14, and 15 are isolated:
$ cat /proc/cmdline ro root=/dev/mapper/vg0-root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg0/swaprd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=137M@0M rd_NO_DM KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg0/root rhgb quiet audit=0 intel_idle.max_cstate=0 console=tty0 console=ttyS1,115200 printk.time=1 processor.max_cstate=1 idle=poll biosdevname=0 isolcpus=13-15
top -H -p
pgrep -u prusr12 Ser -d 1 shows this: 5017 and 5018 should have been bound to 14 and 15 and 5014 and 5016 should have been on 13.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 5017 prusr12 20 0 1312m 1.1g 1.1g R 99.9 0.9 9:53.93 5 Server-3.10. 5018 prusr12 20 0 1312m 1.1g 1.1g R 99.9 0.9 10:08.88 7 Server-3.10. 5014 prusr12 20 0 1312m 1.1g 1.1g S 0.0 0.9 0:00.40 2 Server-3.10. 5016 prusr12 20 0 1312m 1.1g 1.1g S 0.0 0.9 0:01.04 4 Server-3.10.
The command line is this:
sg devuser "taskset -c 13 /releases/3.10.0/bin/Server-3.10.0 -n X -e DEV -p DEFAULT > /logs/ServerDevPR_DEFAULT.out 2>&1 &"
There are 4 threads in the process. I want the main thread to start on 13, hence taskset -c 13. Then two threads are spawed and will bind them to 14 and 15. I see that the threads were bound to 14 and 15, but then they were moved to other cores. pthread_setaffinity_np() is being used to bind the threads to cores.
Log after I bind the threads to 14 and 15:
CpuSet returned by pthread_getaffinity_np() contained:CPU 14
CpuSet returned by pthread_getaffinity_np() contained:CPU 15
$ uname -a Linux host123 2.6.32-573.12.1.el6.x86_64 #1 SMP Mon Nov 23 12:55:32 EST 2015 x86_64 x86_64 x86_64 GNU/Linux $ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 1 Core(s) per socket: 8 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Stepping: 2 CPU MHz: 3199.847 BogoMIPS: 6399.06 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 20480K NUMA node0 CPU(s): 0-7 NUMA node1 CPU(s): 8-15
What could be going wrong? Thanks for your time.
Since I’ve worked with CPU isolation in the past, I created a tool to help me since I found out the hard way that it was surprisingly complex to get it right. The tool is available here:
partrt script that you should look at and get inspired by.
One caveat though: It might not work well together with systemd out of the box, I have simply not tested this.
But as for your specific question, my guess is that what you need is to set
0, to disallow SMP load balancing. Now this will disable this for everything, which might not be what you want.
My tool (
partrt) tries to divide the CPUs into two sets: One set which you want to work as your used to and the other set which tries to isolate each CPU within the set.