Re: [RFC v2 0/4] vfio/hisilicon: add acc live migration driver

[Joao Martins]

On 2/2/22 17:03, Jason Gunthorpe wrote:
> On Wed, Feb 02, 2022 at 04:10:06PM +0000, Shameerali Kolothum Thodi wrote:
>>> From: Jason Gunthorpe [mailto:jgg@xxxxxxxxxx]
>>> Sent: 02 February 2022 15:40
>>> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@xxxxxxxxxx>
>>> Cc: kvm@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
>>> linux-crypto@xxxxxxxxxxxxxxx; alex.williamson@xxxxxxxxxx;
>>> mgurtovoy@xxxxxxxxxx; Linuxarm <linuxarm@xxxxxxxxxx>; liulongfang
>>> <liulongfang@xxxxxxxxxx>; Zengtao (B) <prime.zeng@xxxxxxxxxxxxx>;
>>> yuzenghui <yuzenghui@xxxxxxxxxx>; Jonathan Cameron
>>> <jonathan.cameron@xxxxxxxxxx>; Wangzhou (B) <wangzhou1@xxxxxxxxxxxxx>
>>> Subject: Re: [RFC v2 0/4] vfio/hisilicon: add acc live migration driver
>>>
>>> On Wed, Feb 02, 2022 at 02:34:52PM +0000, Shameerali Kolothum Thodi
>>> wrote:
>>>
>>>>> There are few topics to consider:
>>>>>  - Which of the three feature sets (STOP_COPY, P2P and PRECOPY) make
>>>>>    sense for this driver?
>>>>
>>>> I think it will be STOP_COPY only for now. We might have PRECOPY
>>>> feature once we have the SMMUv3 HTTU support in future.
>>>
>>> HTTU is the dirty tracking feature? To be clear VFIO migration support for
>>> PRECOPY has nothing to do with IOMMU based dirty page tracking.
>>
>> Yes, it is based on the IOMMU hardware dirty bit management support.
>> A RFC was posted sometime back,
>> https://lore.kernel.org/kvm/20210507103608.39440-1-zhukeqian1@xxxxxxxxxx/
> 
> Yes, I saw that. I was hoping to have a discussion on this soon about

Sorry to snatch the thread, I was thinking in doing the same this week,
as I was playing around that area, and wanted to ask the community... but
since you mentioned it :D

> how to integrate that with the iommufd work, which I hope will allow
> that series, and the other IOMMU drivers that can support this to be
> merged..

The iommu-fd thread wasn't particularly obvious on how dirty tracking is done
there, but TBH I am not up to speed on iommu-fd yet so I missed something
obvious for sure. When you say 'integrate that with the iommufd' can you
expand on that?

Did you meant to use interface in the link, or perhaps VFIO would use an iommufd
/internally/ but still export the same UAPI as VFIO dirty tracking ioctls() (even if it's
not that efficient with a lot of bitmap copying). And optionally give a iommu_fd for the
VMM to scan iommu pagetables itself and see what was marked dirty or not?

My over-simplistic/naive view was that the proposal in the link above sounded a lot
simpler. While iommu-fd had more longevity for many other usecases outside dirty tracking, no?

I have a PoC-ish using the interface in the link, with AMD IOMMU dirty bit supported
(including Qemu emulated amd-iommu for folks lacking the hardware). Albeit the
eager-spliting + collapsing of IOMMU hugepages is not yet done there, and I wanted to play
around the emulated intel-iommu SLADS from specs looks quite similar. Happy to join
existing effort anyways.