Re: [RFC][PATCH v1 3/3] cpufreq: intel_pstate: Set asymmetric CPU capacity on hybrid systems

[Ricardo Neri]

On Thu, May 02, 2024 at 12:42:54PM +0200, Dietmar Eggemann wrote:
> On 25/04/2024 21:06, Rafael J. Wysocki wrote:
> > From: Rafael J. Wysocki <rafael.j.wysocki@xxxxxxxxx>
> > 
> > Make intel_pstate use the HWP_HIGHEST_PERF values from
> > MSR_HWP_CAPABILITIES to set asymmetric CPU capacity information
> > via the previously introduced arch_set_cpu_capacity() on hybrid
> > systems without SMT.
> 
> Are there such systems around? My i7-13700K has P-cores (CPU0..CPU15)
> with SMT.

We have been experimenting with nosmt in the kernel command line.

> 
> > Setting asymmetric CPU capacity is generally necessary to allow the
> > scheduler to compute task sizes in a consistent way across all CPUs
> > in a system where they differ by capacity.  That, in turn, should help
> > to improve task placement and load balancing decisions.  It is also
> > necessary for the schedutil cpufreq governor to operate as expected
> > on hybrid systems where tasks migrate between CPUs of different
> > capacities.
> > 
> > The underlying observation is that intel_pstate already uses
> > MSR_HWP_CAPABILITIES to get CPU performance information which is
> > exposed by it via sysfs and CPU performance scaling is based on it.
> > Thus using this information for setting asymmetric CPU capacity is
> > consistent with what the driver has been doing already.  Moreover,
> > HWP_HIGHEST_PERF reflects the maximum capacity of a given CPU including
> > both the instructions-per-cycle (IPC) factor and the maximum turbo
> > frequency and the units in which that value is expressed are the same
> > for all CPUs in the system, so the maximum capacity ratio between two
> > CPUs can be obtained by computing the ratio of their HWP_HIGHEST_PERF
> > values.  Of course, in principle that capacity ratio need not be
> > directly applicable at lower frequencies, so using it for providing the
> > asymmetric CPU capacity information to the scheduler is a rough
> > approximation, but it is as good as it gets.  Also, measurements
> > indicate that this approximation is not too bad in practice.
> 
> So cpu_capacity has a direct mapping to itmt prio. cpu_capacity is itmt
> prio with max itmt prio scaled to 1024.

ITMT enables asym_packing in the load balancer. Since it only cares about
which CPU has higher priority, scaling to 1024 is not necessary.

> 
> Running it on i7-13700K (while allowing SMT) gives:
> 
> root@gulliver:~# dmesg | grep sched_set_itmt_core_prio
> [    3.957826] sched_set_itmt_core_prio() cpu=0 prio=68
> [    3.990401] sched_set_itmt_core_prio() cpu=1 prio=68
> [    4.015551] sched_set_itmt_core_prio() cpu=2 prio=68
> [    4.040720] sched_set_itmt_core_prio() cpu=3 prio=68
> [    4.065871] sched_set_itmt_core_prio() cpu=4 prio=68
> [    4.091018] sched_set_itmt_core_prio() cpu=5 prio=68
> [    4.116175] sched_set_itmt_core_prio() cpu=6 prio=68
> [    4.141374] sched_set_itmt_core_prio() cpu=7 prio=68
> [    4.166543] sched_set_itmt_core_prio() cpu=8 prio=69
> [    4.196289] sched_set_itmt_core_prio() cpu=9 prio=69
> [    4.214964] sched_set_itmt_core_prio() cpu=10 prio=69
> [    4.239281] sched_set_itmt_core_prio() cpu=11 prio=69
> [    4.263438] sched_set_itmt_core_prio() cpu=12 prio=68
> [    4.283790] sched_set_itmt_core_prio() cpu=13 prio=68
> [    4.308905] sched_set_itmt_core_prio() cpu=14 prio=68
> [    4.331751] sched_set_itmt_core_prio() cpu=15 prio=68
> [    4.356002] sched_set_itmt_core_prio() cpu=16 prio=42
> [    4.381639] sched_set_itmt_core_prio() cpu=17 prio=42
> [    4.395175] sched_set_itmt_core_prio() cpu=18 prio=42
> [    4.425625] sched_set_itmt_core_prio() cpu=19 prio=42
> [    4.449670] sched_set_itmt_core_prio() cpu=20 prio=42
> [    4.479681] sched_set_itmt_core_prio() cpu=21 prio=42
> [    4.506319] sched_set_itmt_core_prio() cpu=22 prio=42
> [    4.523774] sched_set_itmt_core_prio() cpu=23 prio=42
> 
> root@gulliver:~# dmesg | grep hybrid_set_cpu_capacity
> [    4.450883] hybrid_set_cpu_capacity() cpu=0 cap=1009
> [    4.455846] hybrid_set_cpu_capacity() cpu=1 cap=1009
> [    4.460806] hybrid_set_cpu_capacity() cpu=2 cap=1009
> [    4.465766] hybrid_set_cpu_capacity() cpu=3 cap=1009
> [    4.470730] hybrid_set_cpu_capacity() cpu=4 cap=1009
> [    4.475699] hybrid_set_cpu_capacity() cpu=5 cap=1009
> [    4.480664] hybrid_set_cpu_capacity() cpu=6 cap=1009
> [    4.485626] hybrid_set_cpu_capacity() cpu=7 cap=1009
> [    4.490588] hybrid_set_cpu_capacity() cpu=9 cap=1024
> [    4.495550] hybrid_set_cpu_capacity() cpu=10 cap=1024
> [    4.500598] hybrid_set_cpu_capacity() cpu=11 cap=1024
> [    4.505649] hybrid_set_cpu_capacity() cpu=12 cap=1009
> [    4.510701] hybrid_set_cpu_capacity() cpu=13 cap=1009
> [    4.515749] hybrid_set_cpu_capacity() cpu=14 cap=1009
> [    4.520802] hybrid_set_cpu_capacity() cpu=15 cap=1009
> [    4.525846] hybrid_set_cpu_capacity() cpu=16 cap=623
> [    4.530810] hybrid_set_cpu_capacity() cpu=17 cap=623
> [    4.535772] hybrid_set_cpu_capacity() cpu=18 cap=623
> [    4.540732] hybrid_set_cpu_capacity() cpu=19 cap=623
> [    4.545690] hybrid_set_cpu_capacity() cpu=20 cap=623
> [    4.550651] hybrid_set_cpu_capacity() cpu=21 cap=623
> [    4.555612] hybrid_set_cpu_capacity() cpu=22 cap=623
> [    4.560571] hybrid_set_cpu_capacity() cpu=23 cap=623
> 
> > If the given system is hybrid and non-SMT, the new code disables ITMT
> > support in the scheduler (because it may get in the way of asymmetric CPU
> > capacity code in the scheduler that automatically gets enabled by setting
> > asymmetric CPU capacity) after initializing all online CPUs and finds
> > the one with the maximum HWP_HIGHEST_PERF value.  Next, it computes the
> > capacity number for each (online) CPU by dividing the product of its
> > HWP_HIGHEST_PERF and SCHED_CAPACITY_SCALE by the maximum HWP_HIGHEST_PERF.
> 
> SO either CAS at wakeup and in load_balance or SIS at wakeup and ITMT in
> load balance.

May I know what CAS and SIS stand for?

Thanks and BR,
Ricardo