Re: [PATCH v1 1/1] ARM: Select DMA_DIRECT_REMAP to fix restricted DMA
From: Jim Quinlan
Date: Fri Sep 29 2023 - 17:13:18 EST
On Fri, Sep 29, 2023 at 3:52 PM Arnd Bergmann <arnd@xxxxxxxx> wrote:
>
> On Fri, Sep 29, 2023, at 15:24, Jim Quinlan wrote:
> > On Thu, Sep 28, 2023 at 11:17 AM Arnd Bergmann <arnd@xxxxxxxx> wrote:
> >> On Thu, Sep 28, 2023, at 10:00, Jim Quinlan wrote:
> >
> > Our RC is definitely not coherent with the ARM/ARM64 caches.
>
> Ok, thanks for the confirmation.
>
> >> It's unlikely but not impossible, as the driver has some
> >> unusual constructs, using a lot of coherent mappings that
> >> might otherwise be streaming mappings, and relying on
> >> dma_sync_single_for_device(..., DMA_BIDIRECTIONAL) for other
> >> data, but without the corresponding dma_sync_single_for_cpu().
> >> If all the testing happens on x86, this might easily lead
> >> to a bug that only shows up on non-coherent systems but
> >> is never seen during testing.
> >>
> >> If the problem is not the "dma-coherent" property, can you
> >> double-check if using a different PCIe device works, or narrow
> >> down which specific buffer you saw get corrupted?
> >
> > I've done some testing, below are the results. The new two devices, a
> > USB controller
> > and an M2 NVMe stick, behave the same as iwlwifi.
> >
> > Note that I'm not advocating that "select DMA_DIRECT_REMAP" is the
> > anser, I'm just showing that it fixes my examples.
>
> Ok, so I think we can stop looking at the device drivers for
> bugs then.
>
> > VER PCI-DEV <--------- RESTRICTED DMA --------->
> > ARM64 ARM ARM64 ARM ARM+DMA_DIRECT_REMAP
> > 5.15 iwlwifi P P P F P
> > 5.15 nvme P P P F P
> > 5.15 usb P P P F P
> >
> > 6.1 iwlwifi P P P F P
> > 6.1 nvme P P P F P
> > 6.1 usb P P P F P
> >
> > Upstrm iwlwifi P P F F F
> > Upstrm nvme P P F F F
> > Upstrm usb P P F F F
> > ARM64 ARM ARM64 ARM ARM+DMA_DIRECT_REMAP
> > VER PCI-DEV <--------- RESTRICTED DMA --------->
> >
> > LEGEND:
> > P := pass, driver probe and some functional test performed
> > F := fail, usually when probe is called; impossible to do
> > functional test
> > Upstrm := 633b47cb009d "Merge tag 'scsi-fixes' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi"
> >
> > iwlwifi := 7260 Wifi 8086:08b1
> > nvme := 1e95:1007
> > usb := Supahub, 1912:0014
>
> Thanks for the thorough testing, that looks very useful, even though
> I don't have an answer immediately. Maybe Robin can see something
> here that I don't.
>
> It's particularly interesting how arm64 only started failing
> on fairly recent kernels, so if nothing else helps you could
> always try bisecting the history between 6.1 and 633b47cb009d,
> hoping that the commit that broke it points us to the arm32
> problem.
Yes I plan to do that.
There's also one other datapoint I have but have only tried it on 6.1
and 5.15 ARM builds.
I can set a board to run its PCIe HW in EP mode, and then connect it
to a board running its
PCIe HW in RC mode.
On the RC side, I've written a small PCIe host driver [1]. On the EP
board, I just run this shell
command when I get the prompt:
while true ; do
for i in `seq 0 15` ; do
devmem $(printf "0x%x" $((0x50a000000 + ($i << 16))))
done
echo
sleep 1
done
You will have to take my word that I've configured the PCIe window on
the EP board at 0x5_0a000000
to correspond to the restricted memory region on the RC side (0x4a000000).
With ARM+restricted DMA I see something like this as the output [2].
The values should appear
all at once or at least in order -- as they do on ARM64 -- but they do
not for ARM.
FWIW,
Jim Quinlan
Broadcom STB/CM
[1]
#define NUM_DMA_REGIONS 16
for (i = 0; i < NUM_DMA_REGIONS; i++)
va[i] = dma_alloc_coherent(dev, size, &dma_addr, flags);
for (i = 0; i < NUM_DMA_REGIONS; i++)
memset(va[i], 0, size/sizeof(u32));
printk("\n================ START READING AT EP ======================\n");
msleep(4000);
for (i = 0; i < NUM_DMA_REGIONS; i++) {
pu32 = va[i];
for (j = 0; j < size/sizeof(u32); j += 64) {
uint32_t x = pu32[j];
mdelay(3);
pu32[j] = 0x12340000 + (i << 12) + j + (x & 0x0f00);
wmb();
}
}
[2]
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x12342000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x12340000
0x00000000
0x12342000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x12340000
0x00000000
0x12342000
0x12343000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x12340000
0x00000000
0x12342000
0x12343000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x12340000
0x12341000
0x12342000
0x12343000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x12340000
0x12341000
0x12342000
0x12343000
0x00000000
0x00000000
0x00000000
0x12347000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x12340000
0x12341000
0x12342000
0x12343000
0x00000000
0x00000000
0x00000000
0x12347000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
^C
>
>
> The only change I see in that time frame that seems related
> is 7bd6680b47fa ("Revert "Revert "arm64: dma: Drop cache
> invalidation from arch_dma_prep_coherent()"""), so you could
> start by reverting that. However, it's probably something
> else since this is for the coherent mappings, not the
> streaming ones.
>
> Arnd
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature