RE: [PATCH for-next v4 0/8] On-Demand Paging on SoftRoCE

[Daisuke Matsuda (Fujitsu)]

On Thu, April 20, 2023 1:07 AM Pearson, Robert B wrote:
> 
> The work queue patch has been submitted and is waiting for some action. -- Bob

Hi,
Could you tell me which is it? I am willing to review it.

This seems to be your latest work queue patch:
https://lore.kernel.org/all/TYCPR01MB8455A2D0B3303FD90B3BB6F1E58B9@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/
I cannot find any one newer on the mailing list nor on the Patchwork.

Daisuke

> 
> -----Original Message-----
> From: Daisuke Matsuda <matsuda-daisuke@xxxxxxxxxxx>
> Sent: Wednesday, April 19, 2023 12:52 AM
> To: linux-rdma@xxxxxxxxxxxxxxx; leonro@xxxxxxxxxx; jgg@xxxxxxxxxx; zyjzyj2000@xxxxxxxxx
> Cc: linux-kernel@xxxxxxxxxxxxxxx; rpearsonhpe@xxxxxxxxx; yangx.jy@xxxxxxxxxxx; lizhijian@xxxxxxxxxxx; Daisuke
> Matsuda <matsuda-daisuke@xxxxxxxxxxx>
> Subject: [PATCH for-next v4 0/8] On-Demand Paging on SoftRoCE
> 
> This patch series implements the On-Demand Paging feature on SoftRoCE(rxe) driver, which has been available only in
> mlx5 driver[1] so far.
> 
> The first patch of this series is provided for testing purpose, and it should be dropped in the end. It converts triple tasklets
> to use workqueue in order to let them sleep during page-fault. Bob Pearson says he will post the patch to do this, and I
> think we can adopt that. The other patches in this series are, I believe, completed works.
> 
> I omitted some contents like the motive behind this series for simplicity.
> Please see the cover letter of v3 for more details[2].
> 
> [Overview]
> When applications register a memory region(MR), RDMA drivers normally pin pages in the MR so that physical addresses
> are never changed during RDMA communication. This requires the MR to fit in physical memory and inevitably leads to
> memory pressure. On the other hand, On-Demand Paging
> (ODP) allows applications to register MRs without pinning pages. They are paged-in when the driver requires and
> paged-out when the OS reclaims. As a result, it is possible to register a large MR that does not fit in physical memory
> without taking up so much physical memory.
> 
> [How does ODP work?]
> "struct ib_umem_odp" is used to manage pages. It is created for each ODP-enabled MR on its registration. This struct
> holds a pair of arrays
> (dma_list/pfn_list) that serve as a driver page table. DMA addresses and PFNs are stored in the driver page table. They
> are updated on page-in and page-out, both of which use the common interfaces in the ib_uverbs layer.
> 
> Page-in can occur when requester, responder or completer access an MR in order to process RDMA operations. If they
> find that the pages being accessed are not present on physical memory or requisite permissions are not set on the pages,
> they provoke page fault to make the pages present with proper permissions and at the same time update the driver page
> table.
> After confirming the presence of the pages, they execute memory access such as read, write or atomic operations.
> 
> Page-out is triggered by page reclaim or filesystem events (e.g. metadata update of a file that is being used as an MR).
> When creating an ODP-enabled MR, the driver registers an MMU notifier callback. When the kernel issues a page
> invalidation notification, the callback is provoked to unmap DMA addresses and update the driver page table. After that,
> the kernel releases the pages.
> 
> [Supported operations]
> All traditional operations are supported on RC connection. The new Atomic write[3] and RDMA Flush[4] operations are
> not included in this patchset. I will post them later after this patchset is merged. On UD connection, Send, Recv, and
> SRQ-Recv are supported.
> 
> [How to test ODP?]
> There are only a few resources available for testing. pyverbs testcases in rdma-core and perftest[5] are recommendable
> ones. Other than them, the ibv_rc_pingpong command can also used for testing. Note that you may have to build perftest
> from upstream because older versions do not handle ODP capabilities correctly.
> 
> The tree is available from github:
> https://github.com/daimatsuda/linux/tree/odp_v4
> While this series is based on commit f605f26ea196, the tree includes an additional bugfix, which is yet to be merged as of
> today (Apr 19th, 2023).
> https://lore.kernel.org/linux-rdma/20230418090642.1849358-1-matsuda-daisuke@xxxxxxxxxxx/
> 
> [Future work]
> My next work is to enable the new Atomic write[3] and RDMA Flush[4] operations with ODP. After that, I am going to
> implement the prefetch feature. It allows applications to trigger page fault using
> ibv_advise_mr(3) to optimize performance. Some existing software like librpma[6] use this feature. Additionally, I think we
> can also add the implicit ODP feature in the future.
> 
> [1] [RFC 00/20] On demand paging
> https://www.spinics.net/lists/linux-rdma/msg18906.html
> 
> [2] [PATCH for-next v3 0/7] On-Demand Paging on SoftRoCE
> https://lore.kernel.org/lkml/cover.1671772917.git.matsuda-daisuke@xxxxxxxxxxx/
> 
> [3] [PATCH v7 0/8] RDMA/rxe: Add atomic write operation
> https://lore.kernel.org/linux-rdma/1669905432-14-1-git-send-email-yangx.jy@xxxxxxxxxxx/
> 
> [4] [for-next PATCH 00/10] RDMA/rxe: Add RDMA FLUSH operation
> https://lore.kernel.org/lkml/20221206130201.30986-1-lizhijian@xxxxxxxxxxx/
> 
> [5] linux-rdma/perftest: Infiniband Verbs Performance Tests https://github.com/linux-rdma/perftest
> 
> [6] librpma: Remote Persistent Memory Access Library https://github.com/pmem/rpma
> 
> v3->v4:
>  1) Re-designed functions that access MRs to use the MR xarray.
>  2) Rebased onto the latest jgg-for-next tree.
> 
> v2->v3:
>  1) Removed a patch that changes the common ib_uverbs layer.
>  2) Re-implemented patches for conversion to workqueue.
>  3) Fixed compile errors (happened when CONFIG_INFINIBAND_ON_DEMAND_PAGING=n).
>  4) Fixed some functions that returned incorrect errors.
>  5) Temporarily disabled ODP for RDMA Flush and Atomic Write.
> 
> v1->v2:
>  1) Fixed a crash issue reported by Haris Iqbal.
>  2) Tried to make lock patters clearer as pointed out by Romanovsky.
>  3) Minor clean ups and fixes.
> 
> Daisuke Matsuda (8):
>   RDMA/rxe: Tentative workqueue implementation
>   RDMA/rxe: Always schedule works before accessing user MRs
>   RDMA/rxe: Make MR functions accessible from other rxe source code
>   RDMA/rxe: Move resp_states definition to rxe_verbs.h
>   RDMA/rxe: Add page invalidation support
>   RDMA/rxe: Allow registering MRs for On-Demand Paging
>   RDMA/rxe: Add support for Send/Recv/Write/Read with ODP
>   RDMA/rxe: Add support for the traditional Atomic operations with ODP
> 
>  drivers/infiniband/sw/rxe/Makefile    |   2 +
>  drivers/infiniband/sw/rxe/rxe.c       |  27 ++-
>  drivers/infiniband/sw/rxe/rxe.h       |  37 ---
>  drivers/infiniband/sw/rxe/rxe_comp.c  |  12 +-
>  drivers/infiniband/sw/rxe/rxe_loc.h   |  49 +++-
>  drivers/infiniband/sw/rxe/rxe_mr.c    |  27 +--
>  drivers/infiniband/sw/rxe/rxe_odp.c   | 311 ++++++++++++++++++++++++++
>  drivers/infiniband/sw/rxe/rxe_recv.c  |   4 +-
>  drivers/infiniband/sw/rxe/rxe_resp.c  |  32 ++-  drivers/infiniband/sw/rxe/rxe_task.c  |  84 ++++---
>  drivers/infiniband/sw/rxe/rxe_task.h  |   6 +-
>  drivers/infiniband/sw/rxe/rxe_verbs.c |   5 +-
>  drivers/infiniband/sw/rxe/rxe_verbs.h |  39 ++++
>  13 files changed, 535 insertions(+), 100 deletions(-)  create mode 100644 drivers/infiniband/sw/rxe/rxe_odp.c
> 
> base-commit: f605f26ea196a3b49bea249330cbd18dba61a33e
> 
> --
> 2.39.1