RE: vPASID capability for VF

[Tian, Kevin]

> From: Tian, Kevin
> Sent: Friday, May 12, 2023 10:53 AM
> 
> > From: Alex Williamson <alex.williamson@xxxxxxxxxx>
> > Sent: Thursday, May 11, 2023 11:45 PM
> >
> > On Thu, 11 May 2023 07:27:27 +0000
> > "Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote:
> >
> > > > From: Alex Williamson <alex.williamson@xxxxxxxxxx>
> > > > Sent: Thursday, May 11, 2023 1:25 AM
> > > >
> > > > On Tue, 9 May 2023 08:34:53 +0000
> > > > "Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote:
> > > >
> > > > > According to PCIe spec (7.8.9 PASID Extended Capability Structure):
> > > > >
> > > > >   The PASID configuration of the single non-VF Function representing
> > > > >   the device is also used by all VFs in the device. A PF is permitted
> > > > >   to implement the PASID capability, but VFs must not implement it.
> > > > >
> > > > > To enable PASID on VF then one open is where to locate the PASID
> > > > > capability in VF's vconfig space. vfio-pci doesn't know which offset
> > > > > may contain VF specific config registers. Finding such offset must
> > > > > come from a device specific knowledge.
> > > >
> > > > Backup for a moment, VFs are governed by the PASID capability on the
> > > > PF.  The PASID capability exposes control registers that imply the
> > > > ability to manage various feature enable bits.  The VF owner does not
> > > > have privileges to manipulate those bits.  For example, the PASID
> Enable
> > > > bit should restrict the endpoint from sending TLPs with a PASID prefix,
> > > > but this can only be changed at the PF level for all associated VFs.
> > > >
> > > > The protocol specified in 7.8.9.3 defines this enable bit as RW.  How do
> > > > we virtualize that?  Either it's virtualized to be read-only and we
> > > > violate the spec or we allow it to be read-write and it has no effect,
> > > > which violates the spec.
> > > >
> > >
> > > Currently the PASID cap is enabled by default when a device is probed
> > > by iommu driver. Leaving it enabled in PF while guest wants it disabled
> > > in VF is harmless. W/o proper setup in iommu side the VF cannot
> > > do real work with PASID.
> > >
> > > From this angle fully virtualizing it in software looks good to me.
> >
> > So you're suggesting that the IOMMU setup for the VF to make use of
> > PASID would not occur until or unless PASID Enable is set in the
> > virtualized VF PASID capability and that support would be torn down
> > when PASID Enable is cleared?
> 
> No that is not the case. The IOMMU setup is initiated by vIOMMU
> and orthogonal to the PASID cap virtualization.
> 
> Following the current IOMMU behavior as Baolu described the guest
> will always enable vPASID in the vIOMMU driver.
> 
> Even if the guest implements an driver-opt model for vPASID enabling,
> in typical case the guest driver will not request vIOMMU setup for VF
> PASIDs if it doesn't intend to enable vPASID cap. In this case the
> physical IOMMU is left blocking VF PASIDs hence leaving PF PASID enabled
> doesn't hurt.
> 
> If an insane guest driver does try to enable vIOMMU PASID (so VF PASIDs
> are allowed in physical IOMMU) while leaving vPASID disabled in VF,
> I'm not sure what would be the actual problem leaving PF PASID enabled.
> The guest driver kind of wants to fool itself by already setting up the
> fabric to allow VF PASID but then block PASID in VF itself?
> 
> >
> > This is still not strictly in adherence with the definition of the
> > PASID Enable bit which specifies that this bit controls whether the
> > endpoint is able to send or receive TLPs with the PASID prefix, which
> > clearly virtualization interacting with the IOMMU to block or allow
> > PASIDs from the VF RID cannot change.  Is it sufficient?
> >
> > For example we can't use the vPASID capability to make any guarantees
> > about in-flight PASID TLPs when sequencing IOMMU operations since we
> > can't actually prevent VFs using PASID so long as PASID Enable is set
> > on the PF.
> 
> IOMMU cares about in-flight PASID TLPs only when it's unblocked.
> 
> If it's already blocked then it doesn't matter whether VF is sending PASID
> TLP or not.
> 
> btw think about the current situation. Even if vfio-pci doesn't expose
> PASID cap today, it's physically enabled by iommu driver already. Then
> the guest is already able to program the device to send PASID TLP's.
> 
> fully virtualizing vPASID cap just aligns with this fact. 😊
> 

Hi, Alex,

If you are OK with above explanation we can continue discussing how
to expose the PASID cap for VF.

At the start I listed several options to quirk the offset in the kernel.

Jason suggested that the kernel should not expose the cap unconditionally.

Then I proposed it could be done via a device feature and leave the
offset to be quirked in VMM. [1] Is it a reasonable way to go?

Thanks
Kevin

[1]  https://lore.kernel.org/lkml/BN9PR11MB5276AE43183A3AA6AB806A398C749@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/