RE: vPASID capability for VF

[Tian, Kevin]

> From: Baolu Lu <baolu.lu@xxxxxxxxxxxxxxx>
> Sent: Thursday, May 11, 2023 7:34 PM
> 
> On 5/11/23 3:27 PM, Tian, Kevin wrote:
> >> From: Alex Williamson<alex.williamson@xxxxxxxxxx>
> >> Sent: Thursday, May 11, 2023 1:25 AM
> >>
> >> On Tue, 9 May 2023 08:34:53 +0000
> >> "Tian, Kevin"<kevin.tian@xxxxxxxxx>  wrote:
> >>
> >>> According to PCIe spec (7.8.9 PASID Extended Capability Structure):
> >>>
> >>>    The PASID configuration of the single non-VF Function representing
> >>>    the device is also used by all VFs in the device. A PF is permitted
> >>>    to implement the PASID capability, but VFs must not implement it.
> >>>
> >>> To enable PASID on VF then one open is where to locate the PASID
> >>> capability in VF's vconfig space. vfio-pci doesn't know which offset
> >>> may contain VF specific config registers. Finding such offset must
> >>> come from a device specific knowledge.
> >> Backup for a moment, VFs are governed by the PASID capability on the
> >> PF.  The PASID capability exposes control registers that imply the
> >> ability to manage various feature enable bits.  The VF owner does not
> >> have privileges to manipulate those bits.  For example, the PASID Enable
> >> bit should restrict the endpoint from sending TLPs with a PASID prefix,
> >> but this can only be changed at the PF level for all associated VFs.
> >>
> >> The protocol specified in 7.8.9.3 defines this enable bit as RW.  How do
> >> we virtualize that?  Either it's virtualized to be read-only and we
> >> violate the spec or we allow it to be read-write and it has no effect,
> >> which violates the spec.
> >>
> > Currently the PASID cap is enabled by default when a device is probed
> > by iommu driver. Leaving it enabled in PF while guest wants it disabled
> > in VF is harmless. W/o proper setup in iommu side the VF cannot
> > do real work with PASID.
> 
> [sorry for partial reply]
> 
> I am attempting to move PASID enabling/disabling out of the iommu
> driver and give its control to the device driver. One puzzle thing is
> that PCI spec requires PASID control bits not changed once the ATS is
> is enabled. So I also need to add iommu interfaces to enable/disable
> ATS on devices.
> 
> By default, the ATS is enabled by the iommu subsystem to utilize the
> device TLB, the device driver needs to disable it before change the
> PASID control bits and re-enable it afterwards.
> 

ATS is also relied on by PRS. Not sure how that will be affected when
ATS is dynamically turned on/off. and PRS is not tied to PASID.

Jason, is it still a strong requirement to have driver-opt model for
pasid enabling? iirc it's first raised in a discussion for an AMD GPU
scenario [1]

[1] https://lore.kernel.org/linux-pci/38a7baf4-9b3b-154b-f764-fa8cfa600858@xxxxxxxxxxxxxxx/