Re: Bug report: UDP ~20% degradation

[Tariq Toukan]

On 12/02/2023 13:50, Tariq Toukan wrote:
> On 08/02/2023 16:12, Vincent Guittot wrote:
> > Hi Tariq,
> >
> > On Wed, 8 Feb 2023 at 12:09, Tariq Toukan <tariqt@xxxxxxxxxx> wrote:
> > > Hi all,
> > >
> > > Our performance verification team spotted a degradation of up to ~20% in
> > > UDP performance, for a specific combination of parameters.
> > >
> > > Our matrix covers several parameters values, like:
> > > IP version: 4/6
> > > MTU: 1500/9000
> > > Msg size: 64/1452/8952 (only when applicable while avoiding ip
> > > fragmentation).
> > > Num of streams: 1/8/16/24.
> > > Num of directions: unidir/bidir.
> > >
> > > Surprisingly, the issue exists only with this specific combination:
> > > 8 streams,
> > > MTU 9000,
> > > Msg size 8952,
> > > both ipv4/6,
> > > bidir.
> > > (in unidir it repros only with ipv4)
> > >
> > > The reproduction is consistent on all the different setups we tested 
> > > with.
> > >
> > > Bisect [2] was done between these two points, v5.19 (Good), and v6.0-rc1
> > > (Bad), with ConnectX-6DX NIC.
> > >
> > > c82a69629c53eda5233f13fc11c3c01585ef48a2 is the first bad commit [1].
> > >
> > > We couldn't come up with a good explanation how this patch causes this
> > > issue. We also looked for related changes in the networking/UDP stack,
> > > but nothing looked suspicious.
> > >
> > > Maybe someone here can help with this.
> > > We can provide more details or do further tests/experiments to progress
> > > with the debug.
> >
> > Could you share more details about your system and the cpu topology ?
>
> output for 'lscpu':
>
> Architecture:                    x86_64
> CPU op-mode(s):                  32-bit, 64-bit
> Address sizes:                   40 bits physical, 57 bits virtual
> Byte Order:                      Little Endian
> CPU(s):                          24
> On-line CPU(s) list:             0-23
> Vendor ID:                       GenuineIntel
> BIOS Vendor ID:                  QEMU
> Model name:                      Intel(R) Xeon(R) Platinum 8380 CPU @ 
> 2.30GHz
> BIOS Model name:                 pc-q35-5.0
> CPU family:                      6
> Model:                           106
> Thread(s) per core:              1
> Core(s) per socket:              1
> Socket(s):                       24
> Stepping:                        6
> BogoMIPS:                        4589.21
> Flags:                           fpu vme de pse tsc msr pae mce cx8 apic 
> sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx 
> pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology 
> cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 
> sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand 
> hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd 
> ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid 
> ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f 
> avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni 
> avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves wbnoinvd arat 
> avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni 
> avx512_bitalg avx512_vpopcntdq rdpid md_clear arch_capabilities
> Virtualization:                  VT-x
> Hypervisor vendor:               KVM
> Virtualization type:             full
> L1d cache:                       768 KiB (24 instances)
> L1i cache:                       768 KiB (24 instances)
> L2 cache:                        96 MiB (24 instances)
> L3 cache:                        384 MiB (24 instances)
> NUMA node(s):                    1
> NUMA node0 CPU(s):               0-23
> Vulnerability Itlb multihit:     Not affected
> Vulnerability L1tf:              Not affected
> Vulnerability Mds:               Not affected
> Vulnerability Meltdown:          Not affected
> Vulnerability Mmio stale data:   Vulnerable: Clear CPU buffers 
> attempted, no microcode; SMT Host state unknown
> Vulnerability Retbleed:          Not affected
> Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass 
> disabled via prctl
> Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers 
> and __user pointer sanitization
> Vulnerability Spectre v2:        Vulnerable: eIBRS with unprivileged eBPF
> Vulnerability Srbds:             Not affected
> Vulnerability Tsx async abort:   Not affected
>
> > The commit  c82a69629c53 migrates a task on an idle cpu when the task
> > is the only one running on local cpu but the time spent by this local
> > cpu under interrupt or RT context becomes significant (10%-17%)
> > I can imagine that 16/24 stream overload your system so load_balance
> > doesn't end up in this case and the cpus are busy with several
> > threads. On the other hand, 1 stream is small enough to keep your
> > system lightly loaded but 8 streams make your system significantly
> > loaded to trigger the reduced capacity case but still not overloaded.
>
> I see. Makes sense.
> 1. How do you check this theory? Any suggested tests/experiments?
> 2. How do you suggest this degradation should be fixed?

Hi,
A kind reminder.