Re: Re: Re: Re: Re: Re: [PATCH] RDMA/siw: Fix compiler warnings on 32-bit due to u64/pointer abuse

[Bernard Metzler]

-----"Jason Gunthorpe" <jgg@xxxxxxxx> wrote: -----

>To: "Bernard Metzler" <BMT@xxxxxxxxxxxxxx>
>From: "Jason Gunthorpe" <jgg@xxxxxxxx>
>Date: 08/19/2019 08:00PM
>Cc: "Geert Uytterhoeven" <geert@xxxxxxxxxxxxxx>, "Doug Ledford"
><dledford@xxxxxxxxxx>, linux-rdma@xxxxxxxxxxxxxxx,
>linux-kernel@xxxxxxxxxxxxxxx
>Subject: [EXTERNAL] Re: Re: Re: Re: Re: [PATCH] RDMA/siw: Fix
>compiler warnings on 32-bit due to u64/pointer abuse
>
>On Mon, Aug 19, 2019 at 05:39:04PM +0000, Bernard Metzler wrote:
>> 
>> >To: "Bernard Metzler" <BMT@xxxxxxxxxxxxxx>
>> >From: "Jason Gunthorpe" <jgg@xxxxxxxx>
>> >Date: 08/19/2019 06:35PM
>> >Cc: "Geert Uytterhoeven" <geert@xxxxxxxxxxxxxx>, "Doug Ledford"
>> ><dledford@xxxxxxxxxx>, linux-rdma@xxxxxxxxxxxxxxx,
>> >linux-kernel@xxxxxxxxxxxxxxx
>> >Subject: [EXTERNAL] Re: Re: Re: Re: Re: [PATCH] RDMA/siw: Fix
>> >compiler warnings on 32-bit due to u64/pointer abuse
>> >
>> >On Mon, Aug 19, 2019 at 04:29:11PM +0000, Bernard Metzler wrote:
>> >> 
>> >> >To: "Bernard Metzler" <BMT@xxxxxxxxxxxxxx>
>> >> >From: "Jason Gunthorpe" <jgg@xxxxxxxx>
>> >> >Date: 08/19/2019 06:05PM
>> >> >Cc: "Geert Uytterhoeven" <geert@xxxxxxxxxxxxxx>, "Doug Ledford"
>> >> ><dledford@xxxxxxxxxx>, linux-rdma@xxxxxxxxxxxxxxx,
>> >> >linux-kernel@xxxxxxxxxxxxxxx
>> >> >Subject: [EXTERNAL] Re: Re: Re: Re: [PATCH] RDMA/siw: Fix
>compiler
>> >> >warnings on 32-bit due to u64/pointer abuse
>> >> >
>> >> >On Mon, Aug 19, 2019 at 03:54:56PM +0000, Bernard Metzler
>wrote:
>> >> >
>> >> >> Absolutely. But these addresses are conveyed through the
>> >> >> API as unsigned 64 during post_send(), and land in the siw
>> >> >> send queue as is. During send queue processing, these
>addresses
>> >> >> must be interpreted according to its context and transformed
>> >> >> (casted) back to the callers intention. I frankly do not
>> >> >> know what we can do differently... The representation of
>> >> >> all addresses as unsigned 64 is given. Sorry for the
>confusion.
>> >> >
>> >> >send work does not have pointers in it, so I'm confused what
>this
>> >is
>> >> >about. Does siw allow userspace to stick an ordinary pointer
>for
>> >the
>> >> >SG list?
>> >> 
>> >> Right a user references a buffer by address and local key it
>> >> got during reservation of that buffer. The user can provide any
>> >> VA between start of that buffer and registered length. 
>> >
>> >Oh gross, it overloads the IOVA in the WR with a kernel void * ??
>> 
>> Oh no. The user library writes the buffer address into
>> the 64bit address field of the WR. This is nothing siw
>> has invented.
>
>No HW provider sticks pointers into the WR ring.

Now siw is a SW only provider. It sits on top of TCP
kernel sockets. siw translates any local application buffer
reference it gets back into a kvec or page pointer (transmit
from), or a virtual address (receive into). This is what the 
TCP interface wants.

In fact, siw cares about physical addresses only since the RDMA
(kernel level) user may care about it. It translates those back
into something the TCP interface can consume. 
>
>It is either an iova & lkey pair, or SGE information is inlined into
>the WR ring.
>
In siw, the reference to any type of memory is kept uninterpreted
in the send/receive queue until it gets accessed by a data
transfer. The information on what type of memory is being referenced
is deducted from the local memory key. As said, this step is
being executed only when the actual buffer is to be touched.
All it needs before that translation is to keep the 32bit key +
length and the up to 64bit address in a work queue element within
the send queue.
lkey lookup and memory translation + access validation happens
after the work queue element left the send/receive queue and a
local copy of it is being processed by the kernel driver
during RX or TX operations.

Inline data is implemented similar to how HW providers do
it - user data are copied immediately into the WR array.

>Never, ever, a user or kernel pointer.
>
>The closest we get to a kernel pointer is with the local dma lkey &
>iova == physical memory address.
>
>> >Why does siw_pbl_get_buffer not return a void *??
>>
>> 
>> I think, in fact, it should be dma_addr_t, since this is
>> what PBL's are described with. Makes sense?
>
>You mean because siw uses dma_virt_ops and can translate a dma_addr_t
>back to a pfn? Yes, that would make alot more sense.
>
>If all conversions went explicitly from a iova & lkey -> dma_addr_t
>-> void * in
>the kmap then I'd be a lot happier
>
>Jason
>
>