Re: [PATCH v5 20/20] PCI: dwc: Add Baikal-T1 PCIe controller support

[Serge Semin]

On Mon, Sep 26, 2022 at 04:31:28PM +0200, Christoph Hellwig wrote:
> On Mon, Sep 26, 2022 at 03:49:24PM +0300, Serge Semin wrote:
> > @Christoph, @Marek, @Bjorn, @Rob could you please join to the
> > DMA-mask related discussion. @Lorenzo can't decide which driver should
> > initialize the device DMA-mask.
> 

> The driver that does the actual DMA mapping or allocation functions
> need to set it.  But even with your comments on the questions I'm
> still confused what struct device you are even talking about.  Can
> you explain this a bit better?

We are talking about the DW PCIe Root Port controller with DW eDMA engine
embedded. It' simplified structure can be represented as follows:

         +---------------+     +--------+
         | System memory |     | CPU(s) |
         +---------------+     +--------+
                ^  |              |  ^
                | ... System bus ... |
               ... |              | ...
                |  v              v  |
 +------------+------+--------+----------+------+
 | DW PCIe RP | AXI-m|        | AXI-s/DBI|      |
 |            +------+        +----------+      |
 |                ^              ^     |        |
 |         +------+----+         |    CSRs      |
 |         v           v         v              |
 |     +-------+  +---------+ +----------+      |
 |     | eDMA  |  | in-iATU | | out-iATU |      |
 |     +-------+  +---------+ +----------+      |
 |         ^           ^           ^            |
 |         +--------+--+---+-------+            |
 +------------------| PIPE |--------------------+
                    +------+
                      | ^
                      v |
                   PCIe bus

The DW PCIe controller device is instantiated as a platform device
defined in the system DT source file. The device is probed by the
DW PCIe low-level driver, which after the platform-specific setups
initiates the generic DW PCIe host-controller registration. On the way
of that procedure the DW PCIe core tries to auto-detect the DW eDMA
engine availability. If the engine is found, the DW eDMA probe method
is called in order to register the DMA-engine device. After that the
PCIe host bridge is registered. Both the PCIe host-bridge and
DMA-engine devices will have the DW PCIe platform device as parent.

Getting back to the sketch above. Here is a short description of the
content:
1. DW eDMA is capable of performing the data transfers from/to System
memory to/from PCIe bus memory.
2. in-iATU is the Inbound Address Translation Unit, which is
responsible for the PCIe bus peripheral devices to access the system
memory. The "dma-ranges" DT-property is used to initialize the
PCIe<->Sys memory mapping. (@William note the In-iATU setup doesn't
affect the eDMA transfers.)
3. out-iATU is responsible for the CPU(s) to access the PCIe bus
peripheral devices memory/cfg-space.

So eDMA and in-iATU are using the same AXI-master interface to access
the system memory. Thus the DMAable memory capability is the same for
both of them (Though in-iATU may have some specific mapping based on
the "dma-ranges" DT-property setup). Neither DW eDMA nor DW PCIe Root
Port CSRs region have any register to auto-detect the AXI-m interface
address bus width. It's selected during the IP-core synthesize and is
platform-specific. The question is: "What driver/code is supposed to
set the DMA-mask of the DW PCIe platform device?" Seeing the parental
platform device is used to perform the memory-mapping for both DW eDMA
clients and PCIe-bus peripheral device drivers, and seeing the AXI-m
interface parameters aren't auto-detectable and are platform-specific,
the only place it should be done in is the DW PCIe low-level device
driver. I don't really see any alternative... What is your opinion?

-Sergey