IO_PAGE_FAULTS with igb or igbvf on AMD IOMMU system
From: Joerg Roedel
Date: Tue Jun 19 2012 - 06:20:55 EST
Hi,
I am trying to use an Intel 82576 NIC on an AMD IOMMU system with
SR-IOV. When I load the igb module with max_vfs=1 to enable a virtual
function I get IO_PAGE_FAULTS from the virtual functions. The relevant
part of dmesg is:
[ 45.788134] igb: Intel(R) Gigabit Ethernet Network Driver - version 3.4.7-k
[ 45.795090] igb: Copyright (c) 2007-2012 Intel Corporation.
[ 45.801049] igb 0000:02:00.0: irq 80 for MSI/MSI-X
[ 45.801056] igb 0000:02:00.0: irq 81 for MSI/MSI-X
[ 45.801061] igb 0000:02:00.0: irq 82 for MSI/MSI-X
[ 45.801067] igb 0000:02:00.0: irq 83 for MSI/MSI-X
[ 45.901445] pci 0000:02:10.0: [8086:10ca] type 00 class 0x020000
[ 45.901585] AMD-Vi: New device 0000:02:10.0
[ 45.906486] igb 0000:02:00.0: 1 VFs allocated
[ 45.937918] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.0.1-k
[ 45.945751] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 46.071749] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 46.078605] igb 0000:02:00.0: eth5: (PCIe:2.5Gb/s:Width x4) 00:1b:21:49:2e:cc
[ 46.085804] igb 0000:02:00.0: eth5: PBA No: E43709-003
[ 46.090946] igb 0000:02:00.0: Using MSI-X interrupts. 2 rx queue(s), 1 tx queue(s)
[ 46.098870] igb 0000:02:00.1: irq 84 for MSI/MSI-X
[ 46.098876] igb 0000:02:00.1: irq 85 for MSI/MSI-X
[ 46.098881] igb 0000:02:00.1: irq 86 for MSI/MSI-X
[ 46.098886] igb 0000:02:00.1: irq 87 for MSI/MSI-X
[ 46.104262] AMD-Vi: Using protection domain 23 for device 0000:02:00.0
[ 46.172988] IPv6: ADDRCONF(NETDEV_UP): eth5: link is not ready
[ 46.202875] pci 0000:02:10.1: [8086:10ca] type 00 class 0x020000
[ 46.203013] AMD-Vi: New device 0000:02:10.1
[ 46.207935] igb 0000:02:00.1: 1 VFs allocated
[ 46.373149] igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 46.380019] igb 0000:02:00.1: eth1: (PCIe:2.5Gb/s:Width x4) 00:1b:21:49:2e:cd
[ 46.387213] igb 0000:02:00.1: eth6: PBA No: E43709-003
[ 46.392347] igb 0000:02:00.1: Using MSI-X interrupts. 2 rx queue(s), 1 tx queue(s)
[ 46.400072] igbvf 0000:02:10.0: enabling device (0000 -> 0002)
[ 46.405977] igbvf 0000:02:10.0: irq 88 for MSI/MSI-X
[ 46.405983] igbvf 0000:02:10.0: irq 89 for MSI/MSI-X
[ 46.405988] igbvf 0000:02:10.0: irq 90 for MSI/MSI-X
[ 46.411492] AMD-Vi: Using protection domain 24 for device 0000:02:00.1
[ 46.480625] IPv6: ADDRCONF(NETDEV_UP): eth6: link is not ready
[ 46.486980] igbvf 0000:02:10.0: Intel(R) 82576 Virtual Function
[ 46.492895] igbvf 0000:02:10.0: Address: ce:5e:41:2f:36:ce
[ 46.498510] igbvf 0000:02:10.1: enabling device (0000 -> 0002)
[ 46.504394] igbvf 0000:02:10.1: irq 91 for MSI/MSI-X
[ 46.504400] igbvf 0000:02:10.1: irq 92 for MSI/MSI-X
[ 46.504405] igbvf 0000:02:10.1: irq 93 for MSI/MSI-X
[ 46.527012] igbvf 0000:02:10.1: Intel(R) 82576 Virtual Function
[ 46.532931] igbvf 0000:02:10.1: Address: 52:3e:8f:47:60:da
[ 46.573209] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:10.0 domain=0x0000 address=0x000000021e170000 flags=0x0050]
[ 46.575620] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 46.589607] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:10.0 domain=0x0000 address=0x000000021e170040 flags=0x0050]
[ 46.600186] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:10.0 domain=0x0000 address=0x000000021e170080 flags=0x0050]
[ 46.610763] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:10.0 domain=0x0000 address=0x000000021e1700c0 flags=0x0050]
[ 46.669940] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
The devices (physical and virtual) are not operational after this. I
think this is a problem in the igb or igbvf driver. The addresses
reported in the IO_PAGE_FAULTS are system-ram addresses are were not
handed out by the AMD IOMMU driver (the driver only hands out DMA
handles below 4GB). Also the reported domain is 0 which means that the
driver for that device has not yet issued _any_ call to the DMA-API. But
the device is doing an DMA write request (as seen in the flags). Any
ideas? Also let me know if you need any additional information.
Thanks,
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/