Re: [PATCH 4/4] PCI/sysfs: Allow userspace to query and set device reset mechanism

[Leon Romanovsky]

On Thu, Mar 18, 2021 at 07:52:52PM +0530, Amey Narkhede wrote:
> On 21/03/18 11:09AM, Leon Romanovsky wrote:
> > On Wed, Mar 17, 2021 at 11:31:40AM -0600, Alex Williamson wrote:
> > > On Wed, 17 Mar 2021 15:58:40 +0200
> > > Leon Romanovsky <leon@xxxxxxxxxx> wrote:
> > >
> > > > On Wed, Mar 17, 2021 at 06:47:18PM +0530, Amey Narkhede wrote:
> > > > > On 21/03/17 01:47PM, Leon Romanovsky wrote:
> > > > > > On Wed, Mar 17, 2021 at 04:53:09PM +0530, Amey Narkhede wrote:
> > > > > > > On 21/03/17 01:02PM, Leon Romanovsky wrote:
> > > > > > > > On Wed, Mar 17, 2021 at 03:54:47PM +0530, Amey Narkhede wrote:
> > > > > > > > > On 21/03/17 06:20AM, Leon Romanovsky wrote:
> > > > > > > > > > On Mon, Mar 15, 2021 at 06:32:32PM +0000, Raphael Norwitz wrote:
> > > > > > > > > > > On Mon, Mar 15, 2021 at 10:29:50AM -0600, Alex Williamson wrote:
> > > > > > > > > > > > On Mon, 15 Mar 2021 21:03:41 +0530
> > > > > > > > > > > > Amey Narkhede <ameynarkhede03@xxxxxxxxx> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > On 21/03/15 05:07PM, Leon Romanovsky wrote:
> > > > > > > > > > > > > > On Mon, Mar 15, 2021 at 08:34:09AM -0600, Alex Williamson wrote:
> > > > > > > > > > > > > > > On Mon, 15 Mar 2021 14:52:26 +0100
> > > > > > > > > > > > > > > Pali Rohár <pali@xxxxxxxxxx> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Monday 15 March 2021 19:13:23 Amey Narkhede wrote:
> > > > > > > > > > > > > > > > > slot reset (pci_dev_reset_slot_function) and secondary bus
> > > > > > > > > > > > > > > > > reset(pci_parent_bus_reset) which I think are hot reset and
> > > > > > > > > > > > > > > > > warm reset respectively.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > No. PCI secondary bus reset = PCIe Hot Reset. Slot reset is just another
> > > > > > > > > > > > > > > > type of reset, which is currently implemented only for PCIe hot plug
> > > > > > > > > > > > > > > > bridges and for PowerPC PowerNV platform and it just call PCI secondary
> > > > > > > > > > > > > > > > bus reset with some other hook. PCIe Warm Reset does not have API in
> > > > > > > > > > > > > > > > kernel and therefore drivers do not export this type of reset via any
> > > > > > > > > > > > > > > > kernel function (yet).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Warm reset is beyond the scope of this series, but could be implemented
> > > > > > > > > > > > > > > in a compatible way to fit within the pci_reset_fn_methods[] array
> > > > > > > > > > > > > > > defined here.  Note that with this series the resets available through
> > > > > > > > > > > > > > > pci_reset_function() and the per device reset attribute is sysfs remain
> > > > > > > > > > > > > > > exactly the same as they are currently.  The bus and slot reset
> > > > > > > > > > > > > > > methods used here are limited to devices where only a single function is
> > > > > > > > > > > > > > > affected by the reset, therefore it is not like the patch you proposed
> > > > > > > > > > > > > > > which performed a reset irrespective of the downstream devices.  This
> > > > > > > > > > > > > > > series only enables selection of the existing methods.  Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Alex,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I asked the patch author here [1], but didn't get any response, maybe
> > > > > > > > > > > > > > you can answer me. What is the use case scenario for this functionality?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1] https://lore.kernel.org/lkml/YE389lAqjJSeTolM@unreal/
> > > > > > > > > > > > > >
> > > > > > > > > > > > > Sorry for not responding immediately. There were some buggy wifi cards
> > > > > > > > > > > > > which needed FLR explicitly not sure if that behavior is fixed in
> > > > > > > > > > > > > drivers. Also there is use a case at Nutanix but the engineer who
> > > > > > > > > > > > > is involved is on PTO that is why I did not respond immediately as
> > > > > > > > > > > > > I don't know the details yet.
> > > > > > > > > > > >
> > > > > > > > > > > > And more generally, devices continue to have reset issues and we
> > > > > > > > > > > > impose a fixed priority in our ordering.  We can and probably should
> > > > > > > > > > > > continue to quirk devices when we find broken resets so that we have
> > > > > > > > > > > > the best default behavior, but it's currently not easy for an end user
> > > > > > > > > > > > to experiment, ie. this reset works, that one doesn't.  We might also
> > > > > > > > > > > > have platform issues where a given reset works better on a certain
> > > > > > > > > > > > platform.  Exposing a way to test these things might lead to better
> > > > > > > > > > > > quirks.  In the case I think Pali was looking for, they wanted a
> > > > > > > > > > > > mechanism to force a bus reset, if this was in reference to a single
> > > > > > > > > > > > function device, this could be accomplished by setting a priority for
> > > > > > > > > > > > that mechanism, which would translate to not only the sysfs reset
> > > > > > > > > > > > attribute, but also the reset mechanism used by vfio-pci.  Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Alex
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > To confirm from our end - we have seen many such instances where default
> > > > > > > > > > > reset methods have not worked well on our platform. Debugging these
> > > > > > > > > > > issues is painful in practice, and this interface would make it far
> > > > > > > > > > > easier.
> > > > > > > > > > >
> > > > > > > > > > > Having an interface like this would also help us better communicate the
> > > > > > > > > > > issues we find with upstream. Allowing others to more easily test our
> > > > > > > > > > > (or other entities') findings should give better visibility into
> > > > > > > > > > > which issues apply to the device in general and which are platform
> > > > > > > > > > > specific. In disambiguating the former from the latter, we should be
> > > > > > > > > > > able to better quirk devices for everyone, and in the latter cases, this
> > > > > > > > > > > interface allows for a safer and more elegant solution than any of the
> > > > > > > > > > > current alternatives.
> > > > > > > > > >
> > > > > > > > > > So to summarize, we are talking about test and debug interface to
> > > > > > > > > > overcome HW bugs, am I right?
> > > > > > > > > >
> > > > > > > > > > My personal experience shows that once the easy workaround exists
> > > > > > > > > > (and write to generally available sysfs is very simple), the vendors
> > > > > > > > > > and users desire for proper fix decreases drastically. IMHO, we will
> > > > > > > > > > see increase of copy/paste in SO and blog posts, but reduce in quirks.
> > > > > > > > > >
> > > > > > > > > > My 2-cents.
> > > > > > > > > >
> > > > > > > > > I agree with your point but at least it gives the userspace ability
> > > > > > > > > to use broken device until bug is fixed in upstream.
> > > > > > > >
> > > > > > > > As I said, I don't expect many fixes once "userspace" will be able to
> > > > > > > > use cheap workaround. There is no incentive to fix it.
> > >
> > > We can increase the annoyance factor of using a modified set of reset
> > > methods, but ultimately we can only control what goes into our kernel,
> > > other kernels might take v1 of this series and incorporate it
> > > regardless of what happens here.
> > >
> > > > > > > > > This is also applicable for obscure devices without upstream
> > > > > > > > > drivers for example custom FPGA based devices.
> > > > > > > >
> > > > > > > > This is not relevant to upstream kernel. Those vendors ship everything
> > > > > > > > custom, they don't need upstream, we don't need them :)
> > > > > > > >
> > > > > > > By custom I meant hobbyists who could tinker with their custom FPGA.
> > > > > >
> > > > > > I invite such hobbyists to send patches and include their FPGA in
> > > > > > upstream kernel.
> > >
> > > This is potentially another good use case, how receptive are we going
> > > to be to an FPGA design that botches a reset.  Do they have a valid
> > > device ID for us to base a quirk on, are they just squatting on one, or
> > > using the default from a library.  Maybe the next bitstream will
> > > resolve it, maybe without any external indication.  IOW, what would the
> > > quality level be for that quirk versus using this as a workaround,
> > > where the user probably wouldn't mind a kernel nag?
> >
> > It is worth to solve it when the need arises.
> >
> > >
> > > > > > > > > Another main application which I forgot to mention is virtualization
> > > > > > > > > where vmm wants to reset the device when the guest is reset,
> > > > > > > > > to emulate machine reboot as closely as possible.
> > > > > > > >
> > > > > > > > It can work in very narrow case, because reset will cause to device
> > > > > > > > reprobe and most likely the driver will be different from the one that
> > > > > > > > started reset. I can imagine that net devices will lose their state and
> > > > > > > > config after such reset too.
> > > > > > > >
> > > > > > > Not sure if I got that 100% right. The pci_reset_function() function
> > > > > > > saves and restores device state over the reset.
> > > > > >
> > > > > > I'm talking about netdev state, but whatever given the existence of
> > > > > > sysfs reset knob.
> > > > > >
> > > > > > >
> > > > > > > > IMHO, it will be saner for everyone if virtualization don't try such resets.
> > >
> > > That would cause a massive regression in device assignment support.  As
> > > with other sysfs attributes, triggering them alongside a running driver
> > > is probably not going to end well.  However, pci_reset_function() is
> > > extremely useful for stopping devices and returning them to a default
> > > state, when either rebooting a VM or returning the device to the host.
> > > The device is not removed and re-probed when this occurs, vfio-pci is
> > > able to hold onto the device across these actions.  Sure, don't reset a
> > > netdev device when it's in use, that's not what these are used for.
> > >
> > > > > > > The exists reset sysfs attribute was added for exactly this case
> > > > > > > though.
> > > > > >
> > > > > > I didn't know the rationale behind that file till you said and I
> > > > > > googled libvirt discussion, so ok. Do you propose that libvirt
> > > > > > will manage database of devices and their working reset types?
> > > > > >
> > > > > I don't have much idea about internals of libvirt but why would
> > > > > it need to manage database of working reset types? It could just
> > > > > read new reset_methods attribute to get the list of supported reset
> > > > > methods.
> > > >
> > > > Because the idea of this patch is to read all supported reset types and
> > > > allow to the user to chose the working one. The user will do it with
> > > > help from StackOverflow, but libvirt will need to have some sort of
> > > > database, otherwise it won't be different from simple "echo 1 > reset"
> > > > which will iterate over all supported resets anyway.
> > >
> > > AFAIK, libvirt no longer attempts to do resets itself, or is at least
> > > moving in that direction.  vfio-pci will reset as device when they're
> > > opened by a user (when available) or triggered via the API.
> >
> > <...>
> >
> > > > The difference here is that this is a workaround to solve bugs that
> > > > should be fixed in the kernel.
> > >
> > > If we want to discourage using this as a primary means to resolve reset
> > > issues on a device then we can create log warnings any time it's used.
> > > Downstreams that really want this functionality are going to take this
> > > patch from the list whether we accept it or not.  As above, it seems
> > > there are valid use cases.  Even with mainstream vfio in QEMU, I go
> > > through some hoops trying to determine if I can do a secondary bus
> > > reset rather than a PM reset because it's not specified anywhere what a
> > > "soft reset" means for any given device.  This sort of interface could
> > > make it easier to apply a system policy that a pci_reset_function()
> > > should always perform a secondary bus reset if the only other option is
> > > a PM reset.  Maybe that policy mostly makes sense for a VM use case, so
> > > we'd want one policy by default and another when the device is used for
> > > this functionality.  How could we accomplish that with a quirk?  Thanks,
> >
> > I'm lost here, does vfio-pci use sysfs interface or internal to the kernel API?
> >
> > If it is latter then we don't really need sysfs, if not, we still need
> > some sort of DB to create second policy, because "supported != working".
> > What am I missing?
> >
> > Thanks
> >
> Can you explain bit more about why supported != working?

It is written in the commit message of this patch.
https://lore.kernel.org/lkml/20210312173452.3855-1-ameynarkhede03@xxxxxxxxx/
"This feature aims to allow greater control of a device for use cases
as device assignment, where specific device or platform issues may
interact poorly with a given reset method, and for which device specific
quirks have not been developed."

You wrote it and also repeated it a couple of times during the discussion.

If device can understand that specific reset doesn't work, it won't
perform it in first place.

Thanks