Re: [PATCH] sched/numa: fix choosing isolated CPUs when task_numa_migrate()

[Srikar Dronamraju]

Hi Yi Wang,

> When trying to migrate to a CPU in task_numa_migrate(), we invoke
> task_numa_find_cpu() to choose a spot, in which function we skip
> the CPU which is not in cpus_allowed, but forgot to concern the
> isolated CPUs, and this may cause the task would run on the isolcpus.
>
> This patch fixes this issue by checking the load_balance_mask.
>
> Signed-off-by: Yi Wang <wang.yi59@xxxxxxxxxx>
> Reviewed-by: Yi Liu <liu.yi24@xxxxxxxxxx>
> ---
>  kernel/sched/fair.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>

I had proposed something similar
http://lkml.kernel.org/r/1491326848-5748-1-git-send-email-srikar@xxxxxxxxxxxxxxxxxx

but at that time, Peter felt we should fix it at different level.
http://lkml.kernel.org/r/20170406073659.y6ubqriyshax4v4m@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and I think he is right.

I have system with 32 cpus, have passed isolcpus as a kernel parameter.

$ grep -o "isolcpus=[,,1-9]*" /proc/cmdline
isolcpus=1,5,9,13
$ grep -i cpus_allowed /proc/$$/status
Cpus_allowed:   ffffdddd
Cpus_allowed_list:      0,2-4,6-8,10-12,14-31


So for a task running on top_cpuset will not have isolcpus as part of the
cpus_allowed.

However if a said task were to call sched_setaffinity, then there is every
likely hood of the cpus being passed being a mix of isolcpus and
nonisolcpus.

For example 
perf bench numa mem --no-data_rand_walk -p 4 -t 8 -G 0 -P 3072 -T 0 -l 50 -c -s 1000

$ for i in $(pgrep -f perf); do  grep -i cpus_allowed_list  /proc/$i/task/*/status ; done | head -n 10
Cpus_allowed_list:      0,2-4,6-8,10-12,14-31
/proc/2107/task/2107/status:Cpus_allowed_list:  0-31
/proc/2107/task/2196/status:Cpus_allowed_list:  0-31
/proc/2107/task/2197/status:Cpus_allowed_list:  0-31
/proc/2107/task/2198/status:Cpus_allowed_list:  0-31
/proc/2107/task/2199/status:Cpus_allowed_list:  0-31
/proc/2107/task/2200/status:Cpus_allowed_list:  0-31
/proc/2107/task/2201/status:Cpus_allowed_list:  0-31
/proc/2107/task/2202/status:Cpus_allowed_list:  0-31
/proc/2107/task/2203/status:Cpus_allowed_list:  0-31

So the cpus_allowed has a mix of isolcpus and nonisolcpus.

While this patch fixes the problem, there is a risk of missing other places
like update_numa_stats also iterates and accounts for tasks stats running on
isolcpus.

I will send a patch with a slightly different approach.
Request you to review and verify the same.

--
Thanks and Regards
Srikar Dronamraju