RE: vPASID capability for VF

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@xxxxxxxxxx>
> Sent: Saturday, May 13, 2023 5:02 AM
> 
> On Fri, May 12, 2023 at 02:59:40AM +0000, Tian, Kevin wrote:
> > > From: Baolu Lu <baolu.lu@xxxxxxxxxxxxxxx>
> > > Sent: Thursday, May 11, 2023 7:34 PM
> > >
> > > On 5/11/23 3:27 PM, Tian, Kevin wrote:
> > > >> From: Alex Williamson<alex.williamson@xxxxxxxxxx>
> > > >> Sent: Thursday, May 11, 2023 1:25 AM
> > > >>
> > > >> On Tue, 9 May 2023 08:34:53 +0000
> > > >> "Tian, Kevin"<kevin.tian@xxxxxxxxx>  wrote:
> > > >>
> > > >>> According to PCIe spec (7.8.9 PASID Extended Capability Structure):
> > > >>>
> > > >>>    The PASID configuration of the single non-VF Function representing
> > > >>>    the device is also used by all VFs in the device. A PF is permitted
> > > >>>    to implement the PASID capability, but VFs must not implement it.
> > > >>>
> > > >>> To enable PASID on VF then one open is where to locate the PASID
> > > >>> capability in VF's vconfig space. vfio-pci doesn't know which offset
> > > >>> may contain VF specific config registers. Finding such offset must
> > > >>> come from a device specific knowledge.
> > > >> Backup for a moment, VFs are governed by the PASID capability on the
> > > >> PF.  The PASID capability exposes control registers that imply the
> > > >> ability to manage various feature enable bits.  The VF owner does not
> > > >> have privileges to manipulate those bits.  For example, the PASID
> Enable
> > > >> bit should restrict the endpoint from sending TLPs with a PASID prefix,
> > > >> but this can only be changed at the PF level for all associated VFs.
> > > >>
> > > >> The protocol specified in 7.8.9.3 defines this enable bit as RW.  How do
> > > >> we virtualize that?  Either it's virtualized to be read-only and we
> > > >> violate the spec or we allow it to be read-write and it has no effect,
> > > >> which violates the spec.
> > > >>
> > > > Currently the PASID cap is enabled by default when a device is probed
> > > > by iommu driver. Leaving it enabled in PF while guest wants it disabled
> > > > in VF is harmless. W/o proper setup in iommu side the VF cannot
> > > > do real work with PASID.
> > >
> > > [sorry for partial reply]
> > >
> > > I am attempting to move PASID enabling/disabling out of the iommu
> > > driver and give its control to the device driver. One puzzle thing is
> > > that PCI spec requires PASID control bits not changed once the ATS is
> > > is enabled. So I also need to add iommu interfaces to enable/disable
> > > ATS on devices.
> > >
> > > By default, the ATS is enabled by the iommu subsystem to utilize the
> > > device TLB, the device driver needs to disable it before change the
> > > PASID control bits and re-enable it afterwards.
> > >
> >
> > ATS is also relied on by PRS. Not sure how that will be affected when
> > ATS is dynamically turned on/off. and PRS is not tied to PASID.
> >
> > Jason, is it still a strong requirement to have driver-opt model for
> > pasid enabling? iirc it's first raised in a discussion for an AMD GPU
> > scenario [1]
> 
> It is sounding worse and worse as we go along..
> 
> AMD and ARM both have the issue that the iommu settings and domain
> types depend on if the driver intends to use PASID or not. There is
> some small performance win if PASID is not used and the iommu driver
> knows it is not used.
> 
> We also get into some trouble with groups, I think, where it will be
> hard for the iommu driver to know which mode to use when creating a
> domain..
> 
> But, if the PASID control for a VF is on the PF then I think it is
> hopeless. The iommu or PCI layers need to make these decisions and
> drivers have to live with it. No PASID support unless the iommu turned
> it on.
> 
> This still suggests there would be some driver call to the iommu side
> to check that PASID is setup.
> 

yes, that still makes sense.