low performance of kernel build with quite a few simultaneous jobs

[sat]

Hi,

I found that kernel build speeds are very low with quite a few `n`  for `make -j<n>`,  from
one to the number of logical CPUs(LCPUs) on my EPYC server.

The speeds of `make -j<n>` look fine with `n` roughly from 1 to 24, and from 64 to 96,
but are slow with other `n`s (please refer to the attached images). I suspect that there is
a task scheduling problem about CPUs with complicated topology like EPYC . Plus, it can't be
explained only by the node distance problem because this behavior doesn't disappear with
enabling memory interleave.

Could you guess any reason of this strange behavior? I ask you because you wrote the patch for
better scheduling of recent AMD CPUs, commit 051f3ca02e46432c0965e8948f00c07d8a2f09c0
("sched/topology: Introduce NUMA identity node sched domain, ").

Here is the detailed information.

** Hardware Engironment

- server: Super Micro AS-1023US-TR4
- CPU:  EPYC 7451 x 2 (48 cores、96 threads)
- RAM: MEM-DR432L-HL01-ER26 (32GB * 16, totally 512GB)

** Software Environment

- OS: Ubuntu 18.04
- kernel: v4.18.0
- build target kernel: v4.18.0

** Test load

Run the following commands.

```
make defconfig
for ((i=1:1<`# of online LCPUs``;i++)) ; do
   make clean && ( time make -j$i ) >result.txt
done
```

I measured the above mentioned data for the following settings.

- all: 96 LCPUS: online whole LCPUs
- 1sock: 48 LCPUs: online only the LCPUS in one sockets
- 1die: 12 LCPUs: ... in one die
- 1ccx: 6 LCPUs: ... in one ccx

** result

Please refer to the attached files.

- 4.18.0-no-interleave.png: Setting no interleave on UEFI

  node distances got by `numactl -H` is as follows.

```
node   0   1   2   3   4   5   6   7
  0:  10  16  16  16  32  32  32  32
  1:  16  10  16  16  32  32  32  32
  2:  16  16  10  16  32  32  32  32
  3:  16  16  16  10  32  32  32  32
  4:  32  32  32  32  10  16  16  16
  5:  32  32  32  32  16  10  16  16
  6:  32  32  32  32  16  16  10  16
  7:  32  32  32  32  16  16  16  10
```

- 4.18.0-hw-interleave.png: Setting interleave on UEFI.
 
 node distance is all 10 in this case.

Here x axis is `n` for `make -j<n>` and y axis is "real" value of time command for
above mentioned kernel build over `n`.

In the attached images, all cases besides 1ccx looks strange.

Thanks,
Satoru Takeuchi

Attachment: 4.18.0-hw-interleave.png
Description: 4.18.0-hw-interleave.png

Attachment: 4.18.0-no-interleave.png
Description: 4.18.0-no-interleave.png