Re: [RFC][PATCH v1 3/3] cpufreq: intel_pstate: Set asymmetric CPU capacity on hybrid systems

From: Ricardo Neri
Date: Thu May 02 2024 - 23:26:03 EST


On Thu, May 02, 2024 at 12:42:54PM +0200, Dietmar Eggemann wrote:
> On 25/04/2024 21:06, Rafael J. Wysocki wrote:
> > From: Rafael J. Wysocki <rafael.j.wysocki@xxxxxxxxx>
> >
> > Make intel_pstate use the HWP_HIGHEST_PERF values from
> > MSR_HWP_CAPABILITIES to set asymmetric CPU capacity information
> > via the previously introduced arch_set_cpu_capacity() on hybrid
> > systems without SMT.
>
> Are there such systems around? My i7-13700K has P-cores (CPU0..CPU15)
> with SMT.

We have been experimenting with nosmt in the kernel command line.

>
> > Setting asymmetric CPU capacity is generally necessary to allow the
> > scheduler to compute task sizes in a consistent way across all CPUs
> > in a system where they differ by capacity. That, in turn, should help
> > to improve task placement and load balancing decisions. It is also
> > necessary for the schedutil cpufreq governor to operate as expected
> > on hybrid systems where tasks migrate between CPUs of different
> > capacities.
> >
> > The underlying observation is that intel_pstate already uses
> > MSR_HWP_CAPABILITIES to get CPU performance information which is
> > exposed by it via sysfs and CPU performance scaling is based on it.
> > Thus using this information for setting asymmetric CPU capacity is
> > consistent with what the driver has been doing already. Moreover,
> > HWP_HIGHEST_PERF reflects the maximum capacity of a given CPU including
> > both the instructions-per-cycle (IPC) factor and the maximum turbo
> > frequency and the units in which that value is expressed are the same
> > for all CPUs in the system, so the maximum capacity ratio between two
> > CPUs can be obtained by computing the ratio of their HWP_HIGHEST_PERF
> > values. Of course, in principle that capacity ratio need not be
> > directly applicable at lower frequencies, so using it for providing the
> > asymmetric CPU capacity information to the scheduler is a rough
> > approximation, but it is as good as it gets. Also, measurements
> > indicate that this approximation is not too bad in practice.
>
> So cpu_capacity has a direct mapping to itmt prio. cpu_capacity is itmt
> prio with max itmt prio scaled to 1024.

ITMT enables asym_packing in the load balancer. Since it only cares about
which CPU has higher priority, scaling to 1024 is not necessary.

>
> Running it on i7-13700K (while allowing SMT) gives:
>
> root@gulliver:~# dmesg | grep sched_set_itmt_core_prio
> [ 3.957826] sched_set_itmt_core_prio() cpu=0 prio=68
> [ 3.990401] sched_set_itmt_core_prio() cpu=1 prio=68
> [ 4.015551] sched_set_itmt_core_prio() cpu=2 prio=68
> [ 4.040720] sched_set_itmt_core_prio() cpu=3 prio=68
> [ 4.065871] sched_set_itmt_core_prio() cpu=4 prio=68
> [ 4.091018] sched_set_itmt_core_prio() cpu=5 prio=68
> [ 4.116175] sched_set_itmt_core_prio() cpu=6 prio=68
> [ 4.141374] sched_set_itmt_core_prio() cpu=7 prio=68
> [ 4.166543] sched_set_itmt_core_prio() cpu=8 prio=69
> [ 4.196289] sched_set_itmt_core_prio() cpu=9 prio=69
> [ 4.214964] sched_set_itmt_core_prio() cpu=10 prio=69
> [ 4.239281] sched_set_itmt_core_prio() cpu=11 prio=69
> [ 4.263438] sched_set_itmt_core_prio() cpu=12 prio=68
> [ 4.283790] sched_set_itmt_core_prio() cpu=13 prio=68
> [ 4.308905] sched_set_itmt_core_prio() cpu=14 prio=68
> [ 4.331751] sched_set_itmt_core_prio() cpu=15 prio=68
> [ 4.356002] sched_set_itmt_core_prio() cpu=16 prio=42
> [ 4.381639] sched_set_itmt_core_prio() cpu=17 prio=42
> [ 4.395175] sched_set_itmt_core_prio() cpu=18 prio=42
> [ 4.425625] sched_set_itmt_core_prio() cpu=19 prio=42
> [ 4.449670] sched_set_itmt_core_prio() cpu=20 prio=42
> [ 4.479681] sched_set_itmt_core_prio() cpu=21 prio=42
> [ 4.506319] sched_set_itmt_core_prio() cpu=22 prio=42
> [ 4.523774] sched_set_itmt_core_prio() cpu=23 prio=42
>
> root@gulliver:~# dmesg | grep hybrid_set_cpu_capacity
> [ 4.450883] hybrid_set_cpu_capacity() cpu=0 cap=1009
> [ 4.455846] hybrid_set_cpu_capacity() cpu=1 cap=1009
> [ 4.460806] hybrid_set_cpu_capacity() cpu=2 cap=1009
> [ 4.465766] hybrid_set_cpu_capacity() cpu=3 cap=1009
> [ 4.470730] hybrid_set_cpu_capacity() cpu=4 cap=1009
> [ 4.475699] hybrid_set_cpu_capacity() cpu=5 cap=1009
> [ 4.480664] hybrid_set_cpu_capacity() cpu=6 cap=1009
> [ 4.485626] hybrid_set_cpu_capacity() cpu=7 cap=1009
> [ 4.490588] hybrid_set_cpu_capacity() cpu=9 cap=1024
> [ 4.495550] hybrid_set_cpu_capacity() cpu=10 cap=1024
> [ 4.500598] hybrid_set_cpu_capacity() cpu=11 cap=1024
> [ 4.505649] hybrid_set_cpu_capacity() cpu=12 cap=1009
> [ 4.510701] hybrid_set_cpu_capacity() cpu=13 cap=1009
> [ 4.515749] hybrid_set_cpu_capacity() cpu=14 cap=1009
> [ 4.520802] hybrid_set_cpu_capacity() cpu=15 cap=1009
> [ 4.525846] hybrid_set_cpu_capacity() cpu=16 cap=623
> [ 4.530810] hybrid_set_cpu_capacity() cpu=17 cap=623
> [ 4.535772] hybrid_set_cpu_capacity() cpu=18 cap=623
> [ 4.540732] hybrid_set_cpu_capacity() cpu=19 cap=623
> [ 4.545690] hybrid_set_cpu_capacity() cpu=20 cap=623
> [ 4.550651] hybrid_set_cpu_capacity() cpu=21 cap=623
> [ 4.555612] hybrid_set_cpu_capacity() cpu=22 cap=623
> [ 4.560571] hybrid_set_cpu_capacity() cpu=23 cap=623
>
> > If the given system is hybrid and non-SMT, the new code disables ITMT
> > support in the scheduler (because it may get in the way of asymmetric CPU
> > capacity code in the scheduler that automatically gets enabled by setting
> > asymmetric CPU capacity) after initializing all online CPUs and finds
> > the one with the maximum HWP_HIGHEST_PERF value. Next, it computes the
> > capacity number for each (online) CPU by dividing the product of its
> > HWP_HIGHEST_PERF and SCHED_CAPACITY_SCALE by the maximum HWP_HIGHEST_PERF.
>
> SO either CAS at wakeup and in load_balance or SIS at wakeup and ITMT in
> load balance.

May I know what CAS and SIS stand for?

Thanks and BR,
Ricardo