Re: [PATCH RFC V2 0/6] rxe_map_mr_sg() fix cleanup and refactor

[Zhijian Li (Fujitsu)]

On 09/11/2023 15:16, Greg Sword wrote:
> On Thu, Nov 9, 2023 at 10:25 AM Zhijian Li (Fujitsu)
> <lizhijian@xxxxxxxxxxx> wrote:
>>
>>
>>
>> On 06/11/2023 21:58, Zhu Yanjun wrote:
>>> 在 2023/11/6 12:07, Zhijian Li (Fujitsu) 写道:
>>>>
>>>>
>>>> On 03/11/2023 21:00, Zhu Yanjun wrote:
>>>>> 在 2023/11/3 17:55, Li Zhijian 写道:
>>>>>> I don't collect the Reviewed-by to the patch1-2 this time, since i
>>>>>> think we can make it better.
>>>>>>
>>>>>> Patch1-2: Fix kernel panic[1] and benifit to make srp work again.
>>>>>>              Almost nothing change from V1.
>>>>>> Patch3-5: cleanups # newly add
>>>>>> Patch6: make RXE support PAGE_SIZE aligned mr # newly add, but not fully tested
>>>>>>
>>>>>> My bad arm64 mechine offten hangs when doing blktests even though i use the
>>>>>> default siw driver.
>>>>>>
>>>>>> - nvme and ULPs(rtrs, iser) always registers 4K mr still don't supported yet.
>>>>>
>>>>> Zhijian
>>>>>
>>>>> Please read carefully the whole discussion about this problem. You will find a lot of valuable suggestions, especially suggestions from Jason.
>>>>
>>>> Okay, i will read it again. If you can tell me which thread, that would be better.
>>>>
>>>>
>>>>>
>>>>>    From the whole discussion, it seems that the root cause is very clear.
>>>>> We need to fix this prolem. Please do not send this kind of commits again.
>>>>>
>>>>
>>>> Let's think about what's our goal first.
>>>>
>>>> - 1) Fix the panic[1] and only support PAGE_SIZE MR
>>>> - 2) support PAGE_SIZE aligned MR
>>>> - 3) support any page_size MR.
>>>>
>>>> I'm sorry i'm not familiar with the linux MM subsystem. It seem it's safe/correct to access
>>>> address/memory across pages start from the return of kmap_loca_page(page).
>>>> In other words, 2) is already native supported, right?
>>>
>>> Yes. Please read the comments from Jason, Leon and Bart. They shared a lot of good advice.
>>
>> I read the whole discussion again, but I believed i still missed a lost.
>>
>>
>>>  From them, we can know the root cause and how to fix this problem.
>>
>> I don't think i misunderstood the root cause:
>> RXE splits memory into PAGE_SIZE units in the xarray. As a result, when we extract an address from the xarray,
>> we should not access address beyond a PAGE_SIZE window.
>>
>> IIUC, then how to fix it?
>> - I'm not going to "removing page_size set", it's out of this patch scope.
>>     Feel free to do the cleanup separately.
>> - I'm not going to fix the NVMe/rtrs etc problems in this patch set when 64K page is enabled.
>>     But RXE will tell its callers explicitly "RXE don't don't support such page_size"
>> - I didn't state RXE supports PAGE_SIZE aligned page_size MR before refactoring rxe_map_mr_sg(),
>>     because I worry about it was not correct to access address beyond the PAGE_SIZE window.
>>
>> What I should do next?
>> Just state "RXE support PAGE_SIZE aligned MR" ? Then patches become
>> RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE aligned MR
>> RDMA/rxe: set RXE_PAGE_SIZE_CAP to starting from PAGE_SIZE
>>
> 
> What do you take rdma maillist for? Your bugzilla, jira? or your dev
> program launch? Or your play ground?

May i know which bug you are concerning. Actually i always cannot get your point.





> 
>> Or just keep we have done in the V1
>>
>> Thanks
>>
>>
>>>
>>> Good Luck.
>>>
>>> Zhu Yanjun
>>>
>>>>
>>>> I get totally confused now.
>>>>
>>>>
>>>>
>>>>> Zhu Yanjun
>>>>>
>>>>>>
>>>>>> [1] https://lore.kernel.org/all/CAHj4cs9XRqE25jyVw9rj9YugffLn5+f=1znaBEnu1usLOciD+g@xxxxxxxxxxxxxx/T/
>>>>>>
>>>>>> Li Zhijian (6):
>>>>>>      RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE mr
>>>>>>      RDMA/rxe: set RXE_PAGE_SIZE_CAP to PAGE_SIZE
>>>>>>      RDMA/rxe: remove unused rxe_mr.page_shift
>>>>>>      RDMA/rxe: Use PAGE_SIZE and PAGE_SHIFT to extract address from
>>>>>>        page_list
>>>>>>      RDMA/rxe: cleanup rxe_mr.{page_size,page_shift}
>>>>>>      RDMA/rxe: Support PAGE_SIZE aligned MR
>>>>>>
>>>>>>     drivers/infiniband/sw/rxe/rxe_mr.c    | 80 ++++++++++++++++-----------
>>>>>>     drivers/infiniband/sw/rxe/rxe_param.h |  2 +-
>>>>>>     drivers/infiniband/sw/rxe/rxe_verbs.h |  9 ---
>>>>>>     3 files changed, 48 insertions(+), 43 deletions(-)
>>>>>>
>>>