Re: [ck] Re: [PATCH] implement nice support across physical cpus on SMP

From: Markus Törnqvist
Date: Mon May 09 2005 - 13:57:56 EST

On Mon, May 09, 2005 at 09:47:05PM +1000, Con Kolivas wrote:
>
>Thanks for feedback.

For once I can give something back, it seems; thus it's my pleasure.

>> top - 11:09:24 up 15:30, 2 users, load average: 4.99, 3.55, 1.63
>> PID USER PR NI VIRT SHR S %CPU %MEM TIME+ #C COMMAND
>> 3917 mjt 25 0 5660 936 R 99.6 0.0 6:11.35 0 load_base.sh
>> 3918 mjt 24 0 5660 936 R 99.6 0.0 6:11.34 3 load_base.sh
>> 3916 mjt 39 0 5660 936 R 65.7 0.0 3:28.95 2 load_base.sh
>> 3919 mjt 39 19 5660 936 R 7.0 0.0 0:23.54 1 load_base.sh
>> 3920 mjt 39 19 5660 936 R 3.0 0.0 0:10.33 2 load_base.sh
>
>These runs don't look absolutely "ideal" as one nice 19 task is bound to cpu1
>however since you're running hyperthreading it would seem the SMT nice code
>is keeping that under check anyway (0:23 vs 6:11)

So let no one touch the SMT code as long as it works...

>These runs pretty much confirm what I found to happen. My test machine for
>this was also 4x. I can't see how the code would behave differently on 2x.

Who on whichever list one is subscribed to or not would care to
replicate these results on 2x and report?

Thank you.

>Perhaps if I make the prio_bias multiplied instead of added to the cpu load
>it will be less affected by SCHED_LOAD_SCALE. The attached patch was
>confirmed during testing to also provide smp distribution according to nice
>on 4x. Carlos I know your machine is in production so you testing may not be
>easy for you. Please try this on top if you have time.

I have no idea about SCHED_LOAD_SCALE, I'm afraid, but I will give
this latest patch a run while I'm at it.

The load.sh is the same I posted before

$ ./load.sh 5
top - 19:41:33 up 9 min, 2 users, load average: 0.40, 0.10, 0.03
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ #C COMMAND
2798 mjt 25 0 5660 1140 936 R 99.9 0.0 0:02.04 2 load_base.sh
2799 mjt 25 0 5660 1140 936 R 99.9 0.0 0:02.04 3 load_base.sh
2797 mjt 25 0 5660 1140 936 R 51.8 0.0 0:01.16 0 load_base.sh
2801 mjt 39 19 5660 1140 936 R 7.0 0.0 0:00.12 1 load_base.sh
2800 mjt 39 19 5660 1140 936 R 3.0 0.0 0:00.05 0 load_base.sh

top - 19:42:20 up 10 min, 2 users, load average: 2.83, 0.78, 0.26
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ #C COMMAND
2798 mjt 35 0 5660 1140 936 R 99.5 0.0 0:48.55 2 load_base.sh
2799 mjt 35 0 5660 1140 936 R 99.5 0.0 0:48.55 3 load_base.sh
2797 mjt 34 0 5660 1140 936 R 61.7 0.0 0:27.43 0 load_base.sh
2801 mjt 39 19 5660 1140 936 R 6.0 0.0 0:03.11 1 load_base.sh
2800 mjt 39 19 5660 1140 936 R 3.0 0.0 0:01.35 0 load_base.sh

top - 19:43:00 up 10 min, 2 users, load average: 3.88, 1.31, 0.46
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ #C COMMAND
2798 mjt 24 0 5660 1140 936 R 99.9 0.0 1:29.13 2 load_base.sh
2799 mjt 24 0 5660 1140 936 R 99.5 0.0 1:29.12 3 load_base.sh
2797 mjt 24 0 5660 1140 936 R 49.8 0.0 0:50.19 0 load_base.sh
2801 mjt 39 19 5660 1140 936 R 7.0 0.0 0:05.76 1 load_base.sh
2800 mjt 39 19 5660 1140 936 R 3.0 0.0 0:02.48 0 load_base.sh

$ ./load.sh 7
top - 19:43:49 up 11 min, 2 users, load average: 4.98, 1.97, 0.73
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ #C COMMAND
2807 mjt 21 0 5660 1140 936 R 99.9 0.0 0:27.73 0 load_base.sh
2804 mjt 34 0 5660 1140 936 R 99.6 0.0 0:23.46 1 load_base.sh
2808 mjt 20 0 5660 1140 936 R 99.6 0.0 0:25.51 2 load_base.sh
2805 mjt 39 0 5660 1140 936 R 39.8 0.0 0:08.12 3 load_base.sh
2806 mjt 33 0 5660 1140 936 R 37.9 0.0 0:12.46 3 load_base.sh
2788 mjt 20 0 5168 1092 832 R 1.0 0.0 0:00.56 3 top
2809 mjt 39 19 5660 1140 936 R 1.0 0.0 0:00.41 3 load_base.sh
2810 mjt 39 19 5660 1144 936 R 1.0 0.0 0:00.40 3 load_base.sh

top - 19:44:20 up 12 min, 2 users, load average: 5.78, 2.45, 0.92
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ #C COMMAND
2807 mjt 22 0 5660 1140 936 R 99.9 0.0 0:56.56 0 load_base.sh
2804 mjt 35 0 5660 1140 936 R 99.5 0.0 0:52.29 1 load_base.sh
2808 mjt 21 0 5660 1140 936 R 99.5 0.0 0:54.34 2 load_base.sh
2805 mjt 35 0 5660 1140 936 R 33.8 0.0 0:15.99 3 load_base.sh
2806 mjt 39 0 5660 1140 936 R 21.9 0.0 0:20.22 3 load_base.sh
2788 mjt 20 0 5168 1092 832 R 1.0 0.0 0:00.65 3 top
2809 mjt 39 19 5660 1140 936 R 1.0 0.0 0:00.80 3 load_base.sh
2810 mjt 39 19 5660 1144 936 R 1.0 0.0 0:00.79 3 load_base.sh

top - 19:45:00 up 12 min, 2 users, load average: 6.37, 3.02, 1.18
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ #C COMMAND
2804 mjt 28 0 5660 1140 936 R 99.9 0.0 1:32.02 1 load_base.sh
2807 mjt 35 0 5660 1140 936 R 99.9 0.0 1:36.29 0 load_base.sh
2808 mjt 33 0 5660 1140 936 R 99.5 0.0 1:34.07 2 load_base.sh
2806 mjt 27 0 5660 1140 936 R 30.9 0.0 0:31.09 3 load_base.sh
2805 mjt 39 0 5660 1140 936 R 24.9 0.0 0:26.81 3 load_base.sh
2809 mjt 39 19 5660 1140 936 R 1.0 0.0 0:01.34 3 load_base.sh
2810 mjt 39 19 5660 1144 936 R 1.0 0.0 0:01.33 3 load_base.sh

Then I decided to do something crazier and renice some pids, to see
what happens...

top - 19:45:45 up 13 min, 2 users, load average: 6.70, 3.58, 1.45
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ #C COMMAND
2807 mjt 37 0 5660 1140 936 R 99.9 0.0 2:21.19 0 load_base.sh
2804 mjt 30 0 5660 1140 936 R 99.5 0.0 2:16.92 1 load_base.sh
2806 mjt 36 0 5660 1140 936 R 41.8 0.0 0:43.73 2 load_base.sh
2805 mjt 39 0 5660 1140 936 R 21.9 0.0 0:39.13 2 load_base.sh
2809 mjt 39 19 5660 1140 936 R 6.0 0.0 0:02.10 3 load_base.sh
2808 mjt 39 10 5660 1140 936 R 2.0 0.0 2:16.01 2 load_base.sh
2810 mjt 39 19 5660 1144 936 R 2.0 0.0 0:01.96 2 load_base.sh

top - 19:46:20 up 14 min, 2 users, load average: 6.83, 3.95, 1.66
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ #C COMMAND
2804 mjt 35 0 5660 1140 936 R 99.6 0.0 2:52.38 1 load_base.sh
2805 mjt 37 0 5660 1140 936 R 99.6 0.0 0:54.29 3 load_base.sh
2807 mjt 21 0 5660 1140 936 R 99.6 0.0 2:56.65 0 load_base.sh
2806 mjt 21 0 5660 1140 936 R 23.9 0.0 0:53.67 2 load_base.sh
2808 mjt 39 10 5660 1140 936 R 21.9 0.0 2:34.49 2 load_base.sh
2809 mjt 39 19 5660 1140 936 R 2.0 0.0 0:02.66 2 load_base.sh
2810 mjt 39 19 5660 1144 936 R 2.0 0.0 0:02.57 2 load_base.sh

top - 19:47:00 up 14 min, 2 users, load average: 6.91, 4.33, 1.88
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ #C COMMAND
2805 mjt 32 0 5660 1140 936 R 99.9 0.0 1:34.05 3 load_base.sh
2807 mjt 37 0 5660 1140 936 R 99.9 0.0 3:36.42 0 load_base.sh
2804 mjt 30 0 5660 1140 936 R 99.6 0.0 3:32.15 1 load_base.sh
2806 mjt 36 0 5660 1140 936 R 40.8 0.0 1:07.04 2 load_base.sh
2808 mjt 39 10 5660 1140 936 R 21.9 0.0 2:41.09 2 load_base.sh
2809 mjt 39 19 5660 1140 936 R 2.0 0.0 0:03.32 2 load_base.sh
2810 mjt 39 19 5660 1144 936 R 1.0 0.0 0:03.23 2 load_base.sh

And sudo renice before I call it a day

top - 19:48:27 up 16 min, 2 users, load average: 7.21, 5.05, 2.34
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ #C COMMAND
2806 mjt 28 0 5660 1140 936 R 99.9 0.0 1:57.55 2 load_base.sh
2807 mjt 27 0 5660 1140 936 R 99.9 0.0 5:00.99 0 load_base.sh
2810 mjt 26 -10 5660 1144 936 R 96.3 0.0 0:05.56 3 load_base.sh
2804 mjt 39 0 5660 1140 936 R 24.8 0.0 4:57.88 1 load_base.sh
2808 mjt 36 10 5660 1140 936 R 9.5 0.0 2:56.90 1 load_base.sh
2805 mjt 39 0 5660 1140 936 R 1.0 0.0 2:29.56 1 load_base.sh

top - 19:49:10 up 16 min, 2 users, load average: 7.65, 5.46, 2.61
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ #C COMMAND
2806 mjt 24 0 5660 1140 936 R 99.9 0.0 2:39.96 2 load_base.sh
2807 mjt 23 0 5660 1140 936 R 99.9 0.0 5:43.40 0 load_base.sh
2810 mjt 26 -10 5660 1144 936 R 99.9 0.0 0:45.87 1 load_base.sh
2805 mjt 39 0 5660 1140 936 R 17.5 0.0 2:36.59 3 load_base.sh
2808 mjt 39 10 5660 1140 936 R 8.7 0.0 2:59.79 3 load_base.sh
2804 mjt 27 0 5660 1140 936 R 7.1 0.0 5:03.47 3 load_base.sh

top - 19:49:45 up 17 min, 2 users, load average: 7.36, 5.63, 2.77
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ #C COMMAND
2806 mjt 31 0 5660 1140 936 R 99.9 0.0 3:15.66 2 load_base.sh
2807 mjt 29 0 5660 1140 936 R 99.9 0.0 6:19.10 0 load_base.sh
2810 mjt 26 -10 5660 1144 936 R 99.9 0.0 1:16.74 3 load_base.sh
2804 mjt 39 0 5660 1140 936 R 17.5 0.0 5:09.32 1 load_base.sh
2805 mjt 39 0 5660 1140 936 R 17.5 0.0 2:45.17 1 load_base.sh
2808 mjt 39 10 5660 1140 936 R 8.7 0.0 3:02.87 1 load_base.sh

Seems good enough under this very fabricated stress, hopefully someone
can tell me a good practical application with processes of different
nices going all over the place, so I can try something else.

But these processes are still clinging a bit to cpu 1 here, that's probably
another SMT feature.
Who tests this on SMP without SMT? Anyone? You! Over there!

This box will go into production soon and then I can maybe get some
glimpses of what happens in practice, and that's about it.
And it'll probably run everything with default values, except a light
mysql -5.

Thanks!

--
mjt

Attachment: signature.asc
Description: Digital signature