Re: Question about reserved_regions w/ Intel IOMMU

[Robin Murphy]

On 2023-06-08 18:10, Alexander Duyck wrote:
> On Thu, Jun 8, 2023 at 8:40 AM Ashok Raj <ashok_raj@xxxxxxxxxxxxxxx> wrote:
> > On Thu, Jun 08, 2023 at 07:33:31AM -0700, Alexander Duyck wrote:
> > > On Wed, Jun 7, 2023 at 8:05 PM Baolu Lu <baolu.lu@xxxxxxxxxxxxxxx> wrote:
> > > > On 6/8/23 7:03 AM, Alexander Duyck wrote:
> > > > > On Wed, Jun 7, 2023 at 3:40 PM Alexander Duyck
> > > > > <alexander.duyck@xxxxxxxxx> wrote:
> > > > > > I am running into a DMA issue that appears to be a conflict between
> > > > > > ACS and IOMMU. As per the documentation I can find, the IOMMU is
> > > > > > supposed to create reserved regions for MSI and the memory window
> > > > > > behind the root port. However looking at reserved_regions I am not
> > > > > > seeing that. I only see the reservation for the MSI.
> > > > > >
> > > > > > So for example with an enabled NIC and iommu enabled w/o passthru I am seeing:
> > > > > > # cat /sys/bus/pci/devices/0000\:83\:00.0/iommu_group/reserved_regions
> > > > > > 0x00000000fee00000 0x00000000feefffff msi
> > > > > >
> > > > > > Shouldn't there also be a memory window for the region behind the root
> > > > > > port to prevent any possible peer-to-peer access?
> > > > >
> > > > > Since the iommu portion of the email bounced I figured I would fix
> > > > > that and provide some additional info.
> > > > >
> > > > > I added some instrumentation to the kernel to dump the resources found
> > > > > in iova_reserve_pci_windows. From what I can tell it is finding the
> > > > > correct resources for the Memory and Prefetchable regions behind the
> > > > > root port. It seems to be calling reserve_iova which is successfully
> > > > > allocating an iova to reserve the region.
> > > > >
> > > > > However still no luck on why it isn't showing up in reserved_regions.
> > > >
> > > > Perhaps I can ask the opposite question, why it should show up in
> > > > reserve_regions? Why does the iommu subsystem block any possible peer-
> > > > to-peer DMA access? Isn't that a decision of the device driver.
> > > >
> > > > The iova_reserve_pci_windows() you've seen is for kernel DMA interfaces
> > > > which is not related to peer-to-peer accesses.
> > >
> > > The problem is if the IOVA overlaps with the physical addresses of
> > > other devices that can be routed to via ACS redirect. As such if ACS
> > > redirect is enabled a host IOVA could be directed to another device on
> > > the switch instead. To prevent that we need to reserve those addresses
> > > to avoid address space collisions.
>
> Our test case is just to perform DMA to/from the host on one device on
> a switch and what we are seeing is that when we hit an IOVA that
> matches up with the physical address of the neighboring devices BAR0
> then we are seeing an AER followed by a hot reset.
>
> > Any untranslated address from a device must be forwarded to the IOMMU when
> > ACS is enabled correct?I guess if you want true p2p, then you would need
> > to map so that the hpa turns into the peer address.. but its always a round
> > trip to IOMMU.
>
> This assumes all parts are doing the Request Redirect "correctly". In
> our case there is a PCIe switch we are trying to debug and we have a
> few working theories. One concern I have is that the switch may be
> throwing an ACS violation for us using an address that matches a
> neighboring device instead of redirecting it to the upstream port. If
> we pull the switch and just run on the root complex the issue seems to
> be resolved so I started poking into the code which led me to the
> documentation pointing out what is supposed to be reserved based on
> the root complex and MSI regions.
>
> As a part of going down that rabbit hole I realized that the
> reserved_regions seems to only list the MSI reservation. However after
> digging a bit deeper it seems like there is code to reserve the memory
> behind the root complex in the IOVA but it doesn't look like that is
> visible anywhere and is the piece I am currently trying to sort out.
> What I am working on is trying to figure out if the system that is
> failing is actually reserving that memory region in the IOVA, or if
> that is somehow not happening in our test setup.

How old's the kernel? Before 5.11, intel-iommu wasn't hooked up to 
iommu-dma so didn't do quite the same thing - it only reserved whatever 
specific PCI memory resources existed at boot, rather than the whole 
window as iommu-dma does. Either way, ftrace on reserve_iova() (or just 
whack a print in there) should suffice to see what's happened.

Robin.