RE: [EXT] Re: [PATCH v16 7/7] PCI: endpoint: pci-epf-vntb: fix sparse build warning at ntb->reg

[Frank Li]

> -----Original Message-----
> From: Bjorn Helgaas <helgaas@xxxxxxxxxx>
> Sent: Tuesday, December 13, 2022 6:27 PM
> To: Frank Li <frank.li@xxxxxxx>
> Cc: mani@xxxxxxxxxx; allenbh@xxxxxxxxx; bhelgaas@xxxxxxxxxx;
> dave.jiang@xxxxxxxxx; imx@xxxxxxxxxxxxxxx; jdmason@xxxxxxxx;
> kw@xxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; linux-pci@xxxxxxxxxxxxxxx;
> lpieralisi@xxxxxxxxxx; ntb@xxxxxxxxxxxxxxx
> Subject: [EXT] Re: [PATCH v16 7/7] PCI: endpoint: pci-epf-vntb: fix sparse
> build warning at ntb->reg
> 
> Caution: EXT Email
> 
> On Wed, Nov 02, 2022 at 10:10:14AM -0400, Frank Li wrote:
> > From: Frank Li <frank.li@xxxxxxx>
> >
> >   pci-epf-vntb.c:1128:33: sparse:     expected void [noderef] __iomem
> *base
> >   pci-epf-vntb.c:1128:33: sparse:     got struct epf_ntb_ctrl *reg
> >
> > Add __iomem type convert in vntb_epf_peer_spad_read() and
> > vntb_epf_peer_spad_write().
> 
> I don't understand all the bits and pieces here, but I'm a little
> dubious about adding all these "(void __iomem *)"casts.  There are
> very few of them in drivers/pci/, and I doubt this driver is so unique
> that it needs them.
> 
> > @@ -1121,7 +1121,7 @@ static u32 vntb_epf_spad_read(struct ntb_dev
> *ndev, int idx)
> >       struct epf_ntb *ntb = ntb_ndev(ndev);
> >       int off = ntb->reg->spad_offset, ct = ntb->reg->spad_count *
> sizeof(u32);
> >       u32 val;
> > -     void __iomem *base = ntb->reg;
> > +     void __iomem *base = (void __iomem *)ntb->reg;
> >
> >       val = readl(base + off + ct + idx * sizeof(u32));
> >       return val;
> > @@ -1132,7 +1132,7 @@ static int vntb_epf_spad_write(struct ntb_dev
> *ndev, int idx, u32 val)
> >       struct epf_ntb *ntb = ntb_ndev(ndev);
> >       struct epf_ntb_ctrl *ctrl = ntb->reg;
> >       int off = ctrl->spad_offset, ct = ctrl->spad_count * sizeof(u32);
> > -     void __iomem *base = ntb->reg;
> > +     void __iomem *base = (void __iomem *)ntb->reg;
> >
> >       writel(val, base + off + ct + idx * sizeof(u32));
> 
> These things look gratuitously different to begin with:
> 
>   int off = ntb->reg->spad_offset, ct = ntb->reg->spad_count * sizeof(u32);
>   int off = ctrl->spad_offset, ct = ctrl->spad_count * sizeof(u32);
> 
> They're doing the same thing, and they should do it the same way.
> 
> Since db_data[] and db_offset[] are never referenced except to be
> initialized to zero,

Db_data and db_offset map to PCI host bar0,  so PCI host will read it.
It is generally used as MSI physical address and data, which PCI host
do doorbell by write these. I have followed patch, which under review.
Irq MSI platform msi change lots, I need more time to study such change.

Default use software polling.  Even though it is zero, pci host driver still use it
to calculate doorbell register offset.    

> I'm guessing the point of vntb_epf_spad_read()
> and vntb_epf_spad_write() is to read/write things in those arrays?

No, it is separated region. 

> 
> You access other things in ntb->reg directly by dereferencing a
> pointer, e.g.,
> 
>   ntb->reg->link_status |= LINK_STATUS_UP;
>   addr = ntb->reg->addr;
>   ctrl->command_status = COMMAND_STATUS_OK;
> 
> Why don't you just compute the appropriate *index* and access the
> array directly instead of using readl() and writel()?

Good question. 
NTB transfer layer treat it as register, so it need keep write\read order. 
1. write data to buffer,
2. update header point. 

1 and 2 must be keep order.  NTB transfer layer have not added memory
barrier. Need use writel to guarantee order. 

About ntb->reg,  actually I think it should use readl also,  but I port these code
from pci_epf_ntb.c and pci_epf_test.c.

PCI host side use writel, so write is ordered. 

But I think reg->* 's order can't be guaranteed at ARM platform.   Reg->addr may
get order value when check reg->command. 

At least a rmb() need after reg->command.  This code are almost run once at
Begging,  so no problem happen. 

> 
> Bjorn