RE: vPASID capability for VF

[Tian, Kevin]

Gentle ping in case it's buried in your mailbox. 😊

we'd like to settle it down soon as the conclusion may even affect PF (e.g. if all
agree that device feature is the right interface to go).

> From: Tian, Kevin
> Sent: Wednesday, May 17, 2023 1:22 PM
> 
> > From: Tian, Kevin
> > Sent: Friday, May 12, 2023 10:53 AM
> >
> > > From: Alex Williamson <alex.williamson@xxxxxxxxxx>
> > > Sent: Thursday, May 11, 2023 11:45 PM
> > >
> > > On Thu, 11 May 2023 07:27:27 +0000
> > > "Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote:
> > >
> > > > > From: Alex Williamson <alex.williamson@xxxxxxxxxx>
> > > > > Sent: Thursday, May 11, 2023 1:25 AM
> > > > >
> > > > > On Tue, 9 May 2023 08:34:53 +0000
> > > > > "Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote:
> > > > >
> > > > > > According to PCIe spec (7.8.9 PASID Extended Capability Structure):
> > > > > >
> > > > > >   The PASID configuration of the single non-VF Function representing
> > > > > >   the device is also used by all VFs in the device. A PF is permitted
> > > > > >   to implement the PASID capability, but VFs must not implement it.
> > > > > >
> > > > > > To enable PASID on VF then one open is where to locate the PASID
> > > > > > capability in VF's vconfig space. vfio-pci doesn't know which offset
> > > > > > may contain VF specific config registers. Finding such offset must
> > > > > > come from a device specific knowledge.
> > > > >
> > > > > Backup for a moment, VFs are governed by the PASID capability on
> the
> > > > > PF.  The PASID capability exposes control registers that imply the
> > > > > ability to manage various feature enable bits.  The VF owner does not
> > > > > have privileges to manipulate those bits.  For example, the PASID
> > Enable
> > > > > bit should restrict the endpoint from sending TLPs with a PASID prefix,
> > > > > but this can only be changed at the PF level for all associated VFs.
> > > > >
> > > > > The protocol specified in 7.8.9.3 defines this enable bit as RW.  How
> do
> > > > > we virtualize that?  Either it's virtualized to be read-only and we
> > > > > violate the spec or we allow it to be read-write and it has no effect,
> > > > > which violates the spec.
> > > > >
> > > >
> > > > Currently the PASID cap is enabled by default when a device is probed
> > > > by iommu driver. Leaving it enabled in PF while guest wants it disabled
> > > > in VF is harmless. W/o proper setup in iommu side the VF cannot
> > > > do real work with PASID.
> > > >
> > > > From this angle fully virtualizing it in software looks good to me.
> > >
> > > So you're suggesting that the IOMMU setup for the VF to make use of
> > > PASID would not occur until or unless PASID Enable is set in the
> > > virtualized VF PASID capability and that support would be torn down
> > > when PASID Enable is cleared?
> >
> > No that is not the case. The IOMMU setup is initiated by vIOMMU
> > and orthogonal to the PASID cap virtualization.
> >
> > Following the current IOMMU behavior as Baolu described the guest
> > will always enable vPASID in the vIOMMU driver.
> >
> > Even if the guest implements an driver-opt model for vPASID enabling,
> > in typical case the guest driver will not request vIOMMU setup for VF
> > PASIDs if it doesn't intend to enable vPASID cap. In this case the
> > physical IOMMU is left blocking VF PASIDs hence leaving PF PASID enabled
> > doesn't hurt.
> >
> > If an insane guest driver does try to enable vIOMMU PASID (so VF PASIDs
> > are allowed in physical IOMMU) while leaving vPASID disabled in VF,
> > I'm not sure what would be the actual problem leaving PF PASID enabled.
> > The guest driver kind of wants to fool itself by already setting up the
> > fabric to allow VF PASID but then block PASID in VF itself?
> >
> > >
> > > This is still not strictly in adherence with the definition of the
> > > PASID Enable bit which specifies that this bit controls whether the
> > > endpoint is able to send or receive TLPs with the PASID prefix, which
> > > clearly virtualization interacting with the IOMMU to block or allow
> > > PASIDs from the VF RID cannot change.  Is it sufficient?
> > >
> > > For example we can't use the vPASID capability to make any guarantees
> > > about in-flight PASID TLPs when sequencing IOMMU operations since we
> > > can't actually prevent VFs using PASID so long as PASID Enable is set
> > > on the PF.
> >
> > IOMMU cares about in-flight PASID TLPs only when it's unblocked.
> >
> > If it's already blocked then it doesn't matter whether VF is sending PASID
> > TLP or not.
> >
> > btw think about the current situation. Even if vfio-pci doesn't expose
> > PASID cap today, it's physically enabled by iommu driver already. Then
> > the guest is already able to program the device to send PASID TLP's.
> >
> > fully virtualizing vPASID cap just aligns with this fact. 😊
> >
> 
> Hi, Alex,
> 
> If you are OK with above explanation we can continue discussing how
> to expose the PASID cap for VF.
> 
> At the start I listed several options to quirk the offset in the kernel.
> 
> Jason suggested that the kernel should not expose the cap unconditionally.
> 
> Then I proposed it could be done via a device feature and leave the
> offset to be quirked in VMM. [1] Is it a reasonable way to go?
> 
> Thanks
> Kevin
> 
> [1]
> https://lore.kernel.org/lkml/BN9PR11MB5276AE43183A3AA6AB806A398C74
> 9@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/