Re: [PATCH] PCI: vmd: Do not disable MSI-X remapping in VMD 28C0 controller

[Xinghui Li]

Jonathan Derrick <jonathan.derrick@xxxxxxxxx> 于2023年1月10日周二 05:00写道：
>
> As the bypass mode seems to affect performance greatly depending on the specific configuration,
> it may make sense to use a moduleparam to control it
>
We found that each pcie port can mount four drives. If we only test 2
or 1 dirve of one pcie port,
the performance of the drive performance will be normal. Also, we
observed the interruptions in different modes.
bypass:
.....
2022-12-28-11-39-14: 1224       181665   IR-PCI-MSI 201850948-edge      nvme0q68
2022-12-28-11-39-14: 1179       180115   IR-PCI-MSI 201850945-edge      nvme0q65
2022-12-28-11-39-14:  RES        26743   Rescheduling interrupts
2022-12-28-11-39-17: irqtop - IRQ : 3029, TOTAL : 2100315228, CPU :
192, ACTIVE CPU : 192
disable:
......
2022-12-28-12-05-56: 1714       169797   IR-PCI-MSI 14155850-edge      nvme1q74
2022-12-28-12-05-56: 1701       168753   IR-PCI-MSI 14155849-edge      nvme1q73
2022-12-28-12-05-56:  LOC       163697   Local timer interrupts
2022-12-28-12-05-56:  TLB         5465   TLB shootdowns
2022-12-28-12-06-00: irqtop - IRQ : 3029, TOTAL : 2179022106, CPU :
192, ACTIVE CPU : 192
remapping:
022-12-28-11-25-38:  283       325568   IR-PCI-MSI 24651790-edge      vmd3
2022-12-28-11-25-38:  140       267899   IR-PCI-MSI 13117447-edge      vmd1
2022-12-28-11-25-38:  183       265978   IR-PCI-MSI 13117490-edge      vmd1
......
2022-12-28-11-25-42: irqtop - IRQ : 2109, TOTAL : 2377172002, CPU :
192, ACTIVE CPU : 192

>From the result it is not difficult to find, in remapping mode the
interruptions come from vmd.
While in other modes, interrupts come from nvme devices. Besides, we
found the port mounting
4 dirves total interruptions is much fewer than the port mounting 2 or 1 drive.
NVME 8 and 9 mount in one port, other port mount 4 dirves.

2022-12-28-11-39-14: 2582       494635   IR-PCI-MSI 470810698-edge      nvme9q74
2022-12-28-11-39-14: 2579       489972   IR-PCI-MSI 470810697-edge      nvme9q73
2022-12-28-11-39-14: 2573       480024   IR-PCI-MSI 470810695-edge      nvme9q71
2022-12-28-11-39-14: 2544       312967   IR-PCI-MSI 470286401-edge      nvme8q65
2022-12-28-11-39-14: 2556       312229   IR-PCI-MSI 470286405-edge      nvme8q69
2022-12-28-11-39-14: 2547       310013   IR-PCI-MSI 470286402-edge      nvme8q66
2022-12-28-11-39-14: 2550       308993   IR-PCI-MSI 470286403-edge      nvme8q67
2022-12-28-11-39-14: 2559       308794   IR-PCI-MSI 470286406-edge      nvme8q70
......
2022-12-28-11-39-14: 1296       185773   IR-PCI-MSI 202375243-edge      nvme1q75
2022-12-28-11-39-14: 1209       185646   IR-PCI-MSI 201850947-edge      nvme0q67
2022-12-28-11-39-14: 1831       184151   IR-PCI-MSI 203423828-edge      nvme3q84
2022-12-28-11-39-14: 1254       182313   IR-PCI-MSI 201850950-edge      nvme0q70
2022-12-28-11-39-14: 1224       181665   IR-PCI-MSI 201850948-edge      nvme0q68
2022-12-28-11-39-14: 1179       180115   IR-PCI-MSI 201850945-edge      nvme0q65
> I'd vote for it being in VMD mode (non-bypass) by default.
I speculate that the vmd controller equalizes the interrupt load and
acts like a buffer,
which improves the performance of nvme. I am not sure about my
analysis. So, I'd like
to discuss it with the community.