Re: [PATCH rdma-next v1 1/1] RDMA/mana_ib: Fix bug in creation of dma regions

[Jason Gunthorpe]

On Thu, Feb 08, 2024 at 06:53:05PM +0000, Konstantin Taranov wrote:
> > From: Long Li <longli@xxxxxxxxxxxxx>
> > Sent: Thursday, 8 February 2024 19:43
> > To: Konstantin Taranov <kotaranov@xxxxxxxxxxxxxxxxxxx>; Konstantin
> > Taranov <kotaranov@xxxxxxxxxxxxx>; sharmaajay@xxxxxxxxxxxxx;
> > jgg@xxxxxxxx; leon@xxxxxxxxxx
> > Cc: linux-rdma@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx
> > Subject: RE: [PATCH rdma-next v1 1/1] RDMA/mana_ib: Fix bug in creation of
> > dma regions
> > 
> > >
> > >  	/* Hardware requires dma region to align to chosen page size */
> > > -	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, 0);
> > > +	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, virt);
> > >  	if (!page_sz) {
> > >  		ibdev_dbg(&dev->ib_dev, "failed to find page size.\n");
> > >  		return -ENOMEM;
> > >  	}
> > 
> > How about doing:
> > page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, force_zero_offset
> > ? 0 : virt);
> > 
> > Will this work? This can get rid of the following while loop.
> > 
> 
> I do not think so. I mentioned once, that it was failing for me with existing code
> with the 4K-aligned addresses and 8K pages. In this case, we miscalculate the 
> number of pages. So, we think that it is one 8K page, but it is in fact two.

That is a confusing statement.. What is "we" here?

ib_umem_dma_offset() is not always guaranteed to be zero, with a 0
iova. With higher order pages the offset can be within the page, it
generates

  offset = IOVA % pgsz

There are a couple places that do want the offset to be fixed to zero
and have the loop, at this point it would be good to consolidate them
into some common ib_umem_find_best_pgsz_zero_offset() or something.

> > > +
> > > +	if (force_zero_offset) {
> > > +		while (ib_umem_dma_offset(umem, page_sz) && page_sz >
> > > PAGE_SIZE)
> > > +			page_sz /= 2;
> > > +		if (ib_umem_dma_offset(umem, page_sz) != 0) {
> > > +			ibdev_dbg(&dev->ib_dev, "failed to find page size to
> > > force zero offset.\n");
> > > +			return -ENOMEM;
> > > +		}
> > > +	}
> > > +

Yes this doesn't look quite right..

It should flow from the HW capability, the helper you call should be
tightly linked to what the HW can do.

ib_umem_find_best_pgsz() is used for MRs that have the usual
  offset = IOVA % pgsz

We've always created other helpers for other restrictions.

So you should move your "force_zero_offset" into another helper and
describe exactly how the HW works to support the calculation

It is odd to have the offset loop and be using
ib_umem_find_best_pgsz() with some iova, usually you'd use
ib_umem_find_best_pgoff() in those cases, see the other callers.

Jason