Re: [PATCH v2] arm64: dts: rockchip: rk356x: Fix PCIe register map and ranges
From: Ondřej Jirman
Date: Sat Oct 22 2022 - 10:36:32 EST
On Sat, Oct 22, 2022 at 08:19:57AM -0400, Peter Geis wrote:
> On Fri, Oct 21, 2022 at 4:52 PM Mark Kettenis <mark.kettenis@xxxxxxxxx> wrote:
>
> [...]
>
> Now this is interesting. I've been reading up on PCIe ranges and what
> is necessary for things to work properly, and I found this interesting
> article from ARM:
> https://developer.arm.com/documentation/102337/0000/Programmers-model/Memory-maps/AP-system-memory-map/PCIe-MMIO-and-ECAM-memory-regions
Thanks for the research and the link. :)
> TLDR: We need a low region (below 4g) and a high region.
>
> From other articles I've gleaned that the config / io should probably
> also be in the low range. As such I believe the other patch that was
> sent to me may be the correct way to go. If both of you would try the
> following reg / ranges:
>
> reg = <0x3 0xc0000000 0x0 0x00400000>,
> <0x0 0xfe260000 0x0 0x00010000>,
> <0x0 0xf4000000 0x0 0x00100000>;
>
> ranges = <0x01000000 0x0 0xf4100000 0x0 0xf4100000 0x0 0x00100000>,
> <0x02000000 0x0 0xf4200000 0x0 0xf4200000 0x0 0x01e00000>,
> <0x03000000 0x0 0x40000000 0x3 0x00000000 0x0 0x40000000>;
Tested, and it works.
One thing to note though is that this results in Linux allocating address space
for all my devices in the 32-bit range, so while it works for me, the address
space above 0x3_0000_0000 was not tested in my experiments, yet.
I'll try this with some other pcie devices, too, after I get my other quartz64
+ pcie bridge setup to work. Hopefully some combo of them will hit the 64-bit
MEM range.
See:
00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: 1000-2fff [size=8K] [16-bit]
Memory behind bridge: f4200000-f44fffff [size=3M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Expansion ROM at f4500000 [virtual] [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Secondary PCI Express
Capabilities: [160] L1 PM Substates
Capabilities: [170] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
lspci: Unable to load libkmod resources: error -2
01:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode])
Subsystem: ASMedia Technology Inc. Device 118f
Flags: bus master, fast devsel, latency 0, IRQ 69
Bus: primary=01, secondary=02, subordinate=06, sec-latency=0
I/O behind bridge: 1000-2fff [size=8K] [16-bit]
Memory behind bridge: f4200000-f44fffff [size=3M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Upstream Port, MSI 00
Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 118f
Capabilities: [100] Virtual Channel
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Vendor Specific Information: ID=0000 Rev=0 Len=c00 <?>
Kernel driver in use: pcieport
02:01.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode])
Subsystem: ASMedia Technology Inc. Device 118f
Flags: bus master, fast devsel, latency 0, IRQ 70
Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: f4200000-f42fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Downstream Port (Slot+), MSI 00
Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 118f
Capabilities: [100] Virtual Channel
Capabilities: [200] Advanced Error Reporting
Kernel driver in use: pcieport
02:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode])
Subsystem: ASMedia Technology Inc. Device 118f
Flags: bus master, fast devsel, latency 0, IRQ 71
Bus: primary=02, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: 1000-1fff [size=4K] [16-bit]
Memory behind bridge: f4300000-f43fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Downstream Port (Slot+), MSI 00
Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 118f
Capabilities: [100] Virtual Channel
Capabilities: [200] Advanced Error Reporting
Kernel driver in use: pcieport
02:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode])
Subsystem: ASMedia Technology Inc. Device 118f
Flags: bus master, fast devsel, latency 0, IRQ 72
Bus: primary=02, secondary=05, subordinate=05, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: [disabled] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Downstream Port (Slot+), MSI 00
Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 118f
Capabilities: [100] Virtual Channel
Capabilities: [200] Advanced Error Reporting
Kernel driver in use: pcieport
02:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode])
Subsystem: ASMedia Technology Inc. Device 118f
Flags: bus master, fast devsel, latency 0, IRQ 73
Bus: primary=02, secondary=06, subordinate=06, sec-latency=0
I/O behind bridge: 2000-2fff [size=4K] [16-bit]
Memory behind bridge: f4400000-f44fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Downstream Port (Slot+), MSI 00
Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 118f
Capabilities: [100] Virtual Channel
Capabilities: [200] Advanced Error Reporting
Kernel driver in use: pcieport
03:00.0 Non-Volatile memory controller: Phison Electronics Corporation PS5013 E13 NVMe Controller (rev 01) (prog-if 02 [NVM Express])
Subsystem: Phison Electronics Corporation PS5013 E13 NVMe Controller
Flags: bus master, fast devsel, latency 0, NUMA node 0
Memory at f4200000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [80] Express Endpoint, MSI 00
Capabilities: [d0] MSI-X: Enable+ Count=9 Masked-
Capabilities: [e0] MSI: Enable- Count=1/8 Maskable+ 64bit+
Capabilities: [f8] Power Management version 3
Capabilities: [100] Latency Tolerance Reporting
Capabilities: [110] L1 PM Substates
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Secondary PCI Express
Kernel driver in use: nvme
04:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8822CE 802.11ac PCIe Wireless Network Adapter
Subsystem: Hewlett-Packard Company Device 85f7
Flags: bus master, fast devsel, latency 0, IRQ 75
I/O ports at 1000 [size=256]
Memory at f4300000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Device Serial Number 00-e0-4c-ff-fe-c8-22-01
Capabilities: [158] Latency Tolerance Reporting
Capabilities: [160] L1 PM Substates
Kernel driver in use: rtw_8822ce
06:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8852AE 802.11ax PCIe Wireless Network Adapter
Subsystem: Hewlett-Packard Company Device 88e1
Flags: bus master, fast devsel, latency 0, IRQ 76
I/O ports at 2000 [size=256]
Memory at f4400000 (64-bit, non-prefetchable) [size=1M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Device Serial Number 00-e0-4c-ff-fe-88-52-01
Capabilities: [158] Latency Tolerance Reporting
Capabilities: [160] L1 PM Substates
Kernel driver in use: rtw89_8852ae
kind regards,
o.
> Very Respectfully,
> Peter Geis
>
> >
> > Now admittedly, this is with OpenBSD running on EDK2 UEFI firmware
> > from
> >
> > https://github.com/jaredmcneill/quartz64_uefi
> >
> > that I modified to pass through the device tree and modify the ranges
> > as above. But the way my OpenBSD driver sets up the address
> > translation windows matches what the mainline Linux driver does.
> >
> > I picked the ranges above to match the EDK2 configuration. But it is
> > a setup that maximizes the 32-bit mmio window.
> >
> > Cheers,
> >
> > Mark
> >
> > > > I still haven't tested this with other cards yet, and another patch
> > > > that does similar work I've tested successfully as well with NVMe
> > > > drives. I'll have to get back to you on the results of greater
> > > > testing.
> > > >
> > > > Very Respectfully,
> > > > Peter Geis
> > > >
> > > > >
> > > > > kind regards,
> > > > > o.
> > > > >
> > > > > > Very Respectfully,
> > > > > > Peter Geis
> > >
> > > _______________________________________________
> > > linux-arm-kernel mailing list
> > > linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
> > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel