Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs

From: Alex Williamson
Date: Tue Apr 14 2020 - 20:36:21 EST


On Tue, 14 Apr 2020 23:57:33 +0000
"Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote:

> > From: Alex Williamson <alex.williamson@xxxxxxxxxx>
> > Sent: Tuesday, April 14, 2020 11:24 PM
> >
> > On Tue, 14 Apr 2020 03:42:42 +0000
> > "Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote:
> >
> > > > From: Alex Williamson <alex.williamson@xxxxxxxxxx>
> > > > Sent: Tuesday, April 14, 2020 11:29 AM
> > > >
> > > > On Tue, 14 Apr 2020 02:40:58 +0000
> > > > "Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote:
> > > >
> > > > > > From: Alex Williamson <alex.williamson@xxxxxxxxxx>
> > > > > > Sent: Tuesday, April 14, 2020 3:21 AM
> > > > > >
> > > > > > On Mon, 13 Apr 2020 08:05:33 +0000
> > > > > > "Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote:
> > > > > >
> > > > > > > > From: Tian, Kevin
> > > > > > > > Sent: Monday, April 13, 2020 3:55 PM
> > > > > > > >
> > > > > > > > > From: Raj, Ashok <ashok.raj@xxxxxxxxxxxxxxx>
> > > > > > > > > Sent: Monday, April 13, 2020 11:11 AM
> > > > > > > > >
> > > > > > > > > On Wed, Apr 08, 2020 at 10:19:40AM -0600, Alex Williamson
> > wrote:
> > > > > > > > > > On Tue, 7 Apr 2020 21:00:21 -0700
> > > > > > > > > > "Raj, Ashok" <ashok.raj@xxxxxxxxx> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Alex
> > > > > > > > > > >
> > > > > > > > > > > + Bjorn
> > > > > > > > > >
> > > > > > > > > > + Don
> > > > > > > > > >
> > > > > > > > > > > FWIW I can't understand why PCI SIG went different ways
> > with
> > > > ATS,
> > > > > > > > > > > where its enumerated on PF and VF. But for PASID and PRI its
> > > > only
> > > > > > > > > > > in PF.
> > > > > > > > > > >
> > > > > > > > > > > I'm checking with our internal SIG reps to followup on that.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Apr 07, 2020 at 09:58:01AM -0600, Alex Williamson
> > > > wrote:
> > > > > > > > > > > > > Is there vendor guarantee that hidden registers will locate
> > at
> > > > the
> > > > > > > > > > > > > same offset between PF and VF config space?
> > > > > > > > > > > >
> > > > > > > > > > > > I'm not sure if the spec really precludes hidden registers,
> > but
> > > > the
> > > > > > > > > > > > fact that these registers are explicitly outside of the
> > capability
> > > > > > > > > > > > chain implies they're only intended for device specific use,
> > so
> > > > I'd
> > > > > > say
> > > > > > > > > > > > there are no guarantees about anything related to these
> > > > registers.
> > > > > > > > > > >
> > > > > > > > > > > As you had suggested in the other thread, we could consider
> > > > > > > > > > > using the same offset as in PF, but even that's a better guess
> > > > > > > > > > > still not reliable.
> > > > > > > > > > >
> > > > > > > > > > > The other option is to maybe extend driver ops in the PF to
> > > > expose
> > > > > > > > > > > where the offsets should be. Sort of adding the quirk in the
> > > > > > > > > > > implementation.
> > > > > > > > > > >
> > > > > > > > > > > I'm not sure how prevalent are PASID and PRI in VF devices. If
> > > > SIG is
> > > > > > > > > resisting
> > > > > > > > > > > making VF's first class citizen, we might ask them to add
> > some
> > > > > > verbiage
> > > > > > > > > > > to suggest leave the same offsets as PF open to help
> > emulation
> > > > > > software.
> > > > > > > > > >
> > > > > > > > > > Even if we know where to expose these capabilities on the VF,
> > it's
> > > > not
> > > > > > > > > > clear to me how we can actually virtualize the capability itself.
> > If
> > > > > > > > > > the spec defines, for example, an enable bit as r/w then
> > software
> > > > that
> > > > > > > > > > interacts with that register expects the bit is settable. There's
> > no
> > > > > > > > > > protocol for "try to set the bit and re-read it to see if the
> > hardware
> > > > > > > > > > accepted it". Therefore a capability with a fixed enable bit
> > > > > > > > > > representing the state of the PF, not settable by the VF, is
> > > > > > > > > > disingenuous to the spec.
> > > > > > > > >
> > > > > > > > > I think we are all in violent agreement. A lot of times the pci spec
> > > > gets
> > > > > > > > > defined several years ahead of real products and no one
> > > > remembers
> > > > > > > > > the justification on why they restricted things the way they did.
> > > > > > > > >
> > > > > > > > > Maybe someone early product wasn't quite exposing these
> > features
> > > > to
> > > > > > the
> > > > > > > > > VF
> > > > > > > > > and hence the spec is bug compatible :-)
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > If what we're trying to do is expose that PASID and PRI are
> > enabled
> > > > on
> > > > > > > > > > the PF to a VF driver, maybe duplicating the PF capabilities on
> > the
> > > > VF
> > > > > > > > > > without the ability to control it is not the right approach.
> > Maybe
> > > > we
> > > > > > > > >
> > > > > > > > > As long as the capability enable is only provided when the PF has
> > > > > > enabled
> > > > > > > > > the feature. Then it seems the hardware seems to do the right
> > thing.
> > > > > > > > >
> > > > > > > > > Assume we expose PASID/PRI only when PF has enabled it. It will
> > be
> > > > the
> > > > > > > > > case since the PF driver needs to exist, and IOMMU would have
> > set
> > > > the
> > > > > > > > > PASID/PRI/ATS on PF.
> > > > > > > > >
> > > > > > > > > If the emulation is purely spoofing the capability. Once vIOMMU
> > > > driver
> > > > > > > > > enables PASID, the context entries for the VF are completely
> > > > > > independent
> > > > > > > > > from the PF context entries.
> > > > > > > > >
> > > > > > > > > vIOMMU would enable PASID, and we just spoof the PASID
> > > > capability.
> > > > > > > > >
> > > > > > > > > If vIOMMU or guest for some reason does disable_pasid(), then
> > the
> > > > > > > > > vIOMMU driver can disaable PASID on the VF context entries. So
> > the
> > > > VF
> > > > > > > > > although the capability is blanket enabled on PF, IOMMU
> > gaurantees
> > > > > > the
> > > > > > > > > transactions are blocked.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > In the interim, it seems like the intent of the virtual capability
> > > > > > > > > can be honored via help from the IOMMU for the controlling
> > aspect..
> > > > > > > > >
> > > > > > > > > Did i miss anything?
> > > > > > > >
> > > > > > > > Above works for emulating the enable bit (under the assumption
> > that
> > > > > > > > PF driver won't disable pasid when vf is assigned). However, there
> > are
> > > > > > > > also "Execute permission enable" and "Privileged mode enable"
> > bits in
> > > > > > > > PASID control registers. I don't know how those bits could be
> > cleanly
> > > > > > > > emulated when the guest writes a value different from PF's...
> > > > > > >
> > > > > > > sent too quick. the IOMMU also includes control bits for allowing/
> > > > > > > blocking execute requests and supervisor requests. We can rely on
> > > > > > > IOMMU to block those requests to emulate the disabled cases of
> > > > > > > all three control bits in the pasid cap.
> > > > > >
> > > > > >
> > > > > > So if the emulation of the PASID capability takes into account the
> > > > > > IOMMU configuration to back that emulation, shouldn't we do that
> > > > > > emulation in the hypervisor, ie. QEMU, rather than the kernel vfio
> > > > > > layer? Thanks,
> > > > > >
> > > > > > Alex
> > > > >
> > > > > We need enforce it in physical IOMMU, to ensure that even the
> > > > > VF may send requests which violate the guest expectation those
> > > > > requests are always blocked by IOMMU. Kernel vfio identifies
> > > > > such need when emulating the pasid cap and then forward the
> > > > > request to host iommu driver.
> > > >
> > > > Implementing this in the kernel would be necessary if we needed to
> > > > protect from the guest device doing something bad to the host or
> > > > other devices. Making sure the physical IOMMU is configured to meet
> > > > guest expectations doesn't sound like it necessarily falls into that
> > > > category. We do that on a regular basis to program the DMA mappings.
> > > > Tell me more about why the hypervisor can't handle this piece of
> > > > guest/host synchronization on top of all the other things it
> > > > synchronizes to make a VM. Thanks,
> > > >
> > >
> > > I care more about "execution permission" and "privileged mode".
> > > It might be dangerous when the guest disallows the VF from sending
> >
> > "Dangerous" how? We're generally ok with the user managing their own
> > consistency, it's when the user can affect other users/devices that we
> > require vfio in the kernel to actively manage something. There's a very
> > different scope to the vfio-pci kernel module implementing a fake
> > capability and trying to make it behave indistinguishably from the real
> > capability versus a userspace driver piecing together an emulation
> > that's good enough for their purposes. Thanks,
> >
>
> How could emulation fix this gap when the VF DMAs don't go through
> the vIOMMU? What you explained all makes sense before talking about
> the emulation of PASID capability, i.e. vfio only cares about isolation
> between assigned devices. However now vfio exposes a capability
> which is shared by PF/VF while pure software emulation may break
> the guest expectation, and now the only viable mitigation is to get
> the help from physical IOMMU. then why cannot vfio include such
> mitigation in its emulation of the PASID capability?

DMA never actually goes "through" the vIOMMU. I'm not suggesting that
vfio doesn't participate some how, but I don't know that emulating a
capability that doesn't exist and involves policy should be done in the
kernel, versus providing userspace with an interface to control what
they need to implement that emulation. Thanks,

Alex