Re: 2.6.24.2-rt2

From: Jan Kiszka
Date: Tue Feb 26 2008 - 05:30:35 EST


Steven Rostedt wrote:
> We are pleased to announce the 2.6.24.2-rt2 tree, which can be
> downloaded from the location:
>
> http://rt.et.redhat.com/download/
>
> Information on the RT patch can be found at:
>
> http://rt.wiki.kernel.org/index.php/Main_Page
>
> Changes since 2.6.24-rt1
>
> - ported to 2.6.24.2
>
> - *** New ftrace utility ***
> The old latency_tracer has now been replaced with the cleaned up
> version that is being prepared for mainline.
>
> - compiler warning fix (Shi Weihua)

This important fix is still missing:

http://lkml.org/lkml/2008/2/5/295


At this chance: We still see the same unbalanced sched-other load on our
NUMA box as Gernot once reported [1]:

top - 11:19:20 up 4 min, 1 user, load average: 29.52, 9.54, 3.37
Tasks: 502 total, 41 running, 461 sleeping, 0 stopped, 0 zombie
Cpu0 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu12 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu13 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu14 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu15 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65513284k total, 1032032k used, 64481252k free, 6444k buffers
Swap: 3204896k total, 0k used, 3204896k free, 37312k cached

PR PID NI VIRT SHR S %CPU %MEM TIME+ COMMAND
20 5603 0 705m 464 R 100 0.3 1:18.19 load-balance-te
20 5604 0 705m 464 R 100 0.3 1:18.16 load-balance-te
20 5605 0 705m 464 R 100 0.3 1:18.18 load-balance-te
20 5608 0 705m 464 R 100 0.3 1:18.18 load-balance-te
20 5611 0 705m 464 R 25 0.3 0:19.58 load-balance-te
20 5620 0 705m 464 R 25 0.3 0:19.54 load-balance-te
20 5606 0 705m 464 R 25 0.3 0:19.56 load-balance-te
20 5616 0 705m 464 R 25 0.3 0:19.54 load-balance-te
20 5607 0 705m 464 R 20 0.3 0:15.64 load-balance-te
20 5609 0 705m 464 R 20 0.3 0:15.66 load-balance-te
20 5614 0 705m 464 R 20 0.3 0:15.68 load-balance-te
20 5617 0 705m 464 R 20 0.3 0:15.64 load-balance-te
20 5619 0 705m 464 R 20 0.3 0:15.64 load-balance-te
20 5610 0 705m 464 R 17 0.3 0:13.10 load-balance-te
20 5618 0 705m 464 R 17 0.3 0:13.04 load-balance-te
20 5621 0 705m 464 R 17 0.3 0:13.02 load-balance-te
20 5622 0 705m 464 R 17 0.3 0:13.02 load-balance-te
20 5623 0 705m 464 R 17 0.3 0:13.06 load-balance-te
20 5624 0 705m 464 R 17 0.3 0:13.02 load-balance-te
20 5615 0 705m 464 R 5 0.3 0:03.76 load-balance-te
20 5633 0 705m 464 R 5 0.3 0:03.72 load-balance-te
20 5634 0 705m 464 R 5 0.3 0:03.82 load-balance-te
20 5635 0 705m 464 R 5 0.3 0:03.74 load-balance-te
20 5636 0 705m 464 R 5 0.3 0:03.72 load-balance-te
20 5638 0 705m 464 R 5 0.3 0:03.72 load-balance-te
20 5640 0 705m 464 R 5 0.3 0:03.72 load-balance-te
20 5612 0 705m 464 R 5 0.3 0:03.78 load-balance-te
20 5613 0 705m 464 R 5 0.3 0:03.78 load-balance-te
20 5625 0 705m 464 R 5 0.3 0:03.74 load-balance-te
20 5626 0 705m 464 R 5 0.3 0:03.74 load-balance-te
20 5627 0 705m 464 R 5 0.3 0:03.82 load-balance-te
20 5628 0 705m 464 R 5 0.3 0:03.72 load-balance-te
20 5629 0 705m 464 R 5 0.3 0:03.72 load-balance-te
20 5630 0 705m 464 R 5 0.3 0:03.72 load-balance-te
20 5631 0 705m 464 R 5 0.3 0:03.72 load-balance-te
20 5632 0 705m 464 R 5 0.3 0:03.72 load-balance-te
20 5637 0 705m 464 R 5 0.3 0:03.72 load-balance-te
20 5639 0 705m 464 R 5 0.3 0:03.74 load-balance-te
20 5641 0 705m 464 R 5 0.3 0:03.70 load-balance-te
20 5642 0 705m 464 R 5 0.3 0:03.80 load-balance-te

I just accidentally compiled the kernel with NR_CPUS=8 first, and then
the load was balanced. Weird.

Some time ago I tried to dig into this but didn't get far due to the
limited bisectability of -rt (was at least true for .23-rtX) and limited
time. But based on this search, I suspect now that the issue is
introduced via some patch somewhere at the end of the series. If anyone
has a good pointer what to check, I would try to look into this again.

Jan

[1] http://thread.gmane.org/gmane.linux.rt.user/1640/focus=1746

--
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/