RE: [PATCH v8 0/5] Share sva domains with all devices bound to a mm

[Zhang, Tina]

Hi,

> -----Original Message-----
> From: Jason Gunthorpe <jgg@xxxxxxxx>
> Sent: Wednesday, October 18, 2023 12:42 AM
> To: Zhang, Tina <tina.zhang@xxxxxxxxx>
> Cc: iommu@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; David Woodhouse
> <dwmw2@xxxxxxxxxxxxx>; Lu Baolu <baolu.lu@xxxxxxxxxxxxxxx>; Joerg
> Roedel <joro@xxxxxxxxxx>; Will Deacon <will@xxxxxxxxxx>; Robin Murphy
> <robin.murphy@xxxxxxx>; Tian, Kevin <kevin.tian@xxxxxxxxx>; Nicolin Chen
> <nicolinc@xxxxxxxxxx>; Michael Shavit <mshavit@xxxxxxxxxx>; Vasant
> Hegde <vasant.hegde@xxxxxxx>
> Subject: Re: [PATCH v8 0/5] Share sva domains with all devices bound to a
> mm
> 
> On Tue, Oct 17, 2023 at 08:47:57AM +0800, Tina Zhang wrote:
> > This series is to share sva(shared virtual addressing) domains with
> > all devices bound to one mm.
> >
> > Problem
> > -------
> > In the current iommu core code, sva domain is allocated per IOMMU
> > group, when device driver is binding a process address space to a
> > device (which is handled in iommu_sva_bind_device()). If one than more
> > device is bound to the same process address space, there must be more
> > than one sva domain instance, with each device having one. In other
> > words, the sva domain doesn't share between those devices bound to the
> > same process address space, and that leads to two problems:
> > 1) device driver has to duplicate sva domains with enqcmd, as those
> > sva domains have the same PASID and are relevant to one virtual address
> space.
> > This makes the sva domain handling complex in device drivers.
> > 2) IOMMU driver cannot get sufficient info of the IOMMUs that have
> > devices behind them bound to the same virtual address space, when
> > handling mmu_notifier_ops callbacks. As a result, IOMMU IOTLB
> > invalidation is performed per device instead of per IOMMU, and that
> > may lead to superfluous IOTLB invalidation issue, especially in a
> > virtualization environment where all devices may be behind one virtual
> IOMMU.
> >
> > Solution
> > --------
> > This patch-set tries to fix those two problems by allowing sharing sva
> > domains with all devices bound to a mm. To achieve this, a new
> > structure pointer is introduced to mm to replace the old PASID field,
> > which can keep the info of PASID as well as the corresponding shared sva
> domains.
> > Besides, function iommu_sva_bind_device() is updated to ensure a new
> > sva domain can only be allocated when the old ones cannot work for the
> IOMMU.
> > With these changes, a device driver can expect one sva domain could
> > work for per PASID instance(e.g., enqcmd PASID instance), and
> > therefore may get rid of handling sva domain duplication. Besides,
> > IOMMU driver (e.g., intel vt-d driver) can get sufficient info (e.g.,
> > the info of the IOMMUs having their devices bound to one virtual
> > address space) when handling mmu_notifier_ops callbacks, to remove the
> redundant IOTLB invalidations.
> >
> > Arguably there shouldn't be more than one sva_domain with the same
> > PASID, and in any sane configuration there should be only 1 type of
> > IOMMU driver that needs only 1 SVA domain. However, in reality, IOMMUs
> > on one platform may not be identical to each other. Thus, attaching a
> > sva domain that has been successfully bound to device A behind a IOMMU
> > A, to device B behind IOMMU B may get failed due to the difference
> > between IOMMU A and IOMMU B. In this case, a new sva domain with the
> > same PASID needs to be allocated to work with IOMMU B. That's why we
> > need a list to keep sva domains of one PASID. For the platform where
> > IOMMUs are compatible to each other, there should be one sva domain in
> the list.
> >
> > v8:
> >  - CC more people
> >  - CC iommu@xxxxxxxxxxxxxxx mailing list.
> >    When sending version 7, some issue happened in my CC list and that
> caused
> >    version 7 wasn't sent to iommu@xxxxxxxxxxxxxxx.
> >  - Rebase to v6.6-rc6 and make a few format changes.
> 
> You should based it on Joerg's tree so he can take it without conflcits.
> 
> The conflicts are trivial though (Take Michael's version and switch
> mm->pasid with mm_get_enqcmd_pasid(mm))
> 
> It looks fine, please lets get it in this cycle, the ARM and AMD SVA series
> depend on it.
The V9 will be based on the next branch of Joerg's tree.

Like Baolu mentioned, besides ARM and AMD SVA series, we also have a VT-d series waiting for it.

Regards,
-Tina
> 
> Jason