Re: [PATCH] iommu/vt-d: use domain instead of cache fetching

From: Alex Williamson
Date: Wed Jan 10 2018 - 18:51:09 EST


On Wed, 10 Jan 2018 13:51:37 +0800
Peter Xu <peterx@xxxxxxxxxx> wrote:

> after commit a1ddcbe93010 ("iommu/vt-d: Pass dmar_domain directly into
> iommu_flush_iotlb_psi", 2015-08-12), we have domain pointer as parameter
> to iommu_flush_iotlb_psi(), so no need to fetch it from cache again.
>
> More importantly, a NULL reference pointer bug is reported on RHEL7 (and
> it can be reproduced on some old upstream kernels too, e.g., v4.13) by
> unplugging an 40g nic from a VM (hard to test unplug on real host, but
> it should be the same):
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1531367
>
> [ 24.391863] pciehp 0000:00:03.0:pcie004: Slot(0): Attention button pressed
> [ 24.393442] pciehp 0000:00:03.0:pcie004: Slot(0): Powering off due to button press
> [ 29.721068] i40evf 0000:01:00.0: Unable to send opcode 2 to PF, err I40E_ERR_QUEUE_EMPTY, aq_err OK
> [ 29.783557] iommu: Removing device 0000:01:00.0 from group 3
> [ 29.784662] BUG: unable to handle kernel NULL pointer dereference at 0000000000000304
> [ 29.785817] IP: iommu_flush_iotlb_psi+0xcf/0x120
> [ 29.786486] PGD 0
> [ 29.786487] P4D 0
> [ 29.786812]
> [ 29.787390] Oops: 0000 [#1] SMP
> [ 29.787876] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_ng
> [ 29.795371] CPU: 0 PID: 156 Comm: kworker/0:2 Not tainted 4.13.0 #14
> [ 29.796366] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.11.0-1.el7 04/01/2014
> [ 29.797593] Workqueue: pciehp-0 pciehp_power_thread
> [ 29.798328] task: ffff94f5745b4a00 task.stack: ffffb326805ac000
> [ 29.799178] RIP: 0010:iommu_flush_iotlb_psi+0xcf/0x120
> [ 29.799919] RSP: 0018:ffffb326805afbd0 EFLAGS: 00010086
> [ 29.800666] RAX: ffff94f5bc56e800 RBX: 0000000000000000 RCX: 0000000200000025
> [ 29.801667] RDX: ffff94f5bc56e000 RSI: 0000000000000082 RDI: 0000000000000000
> [ 29.802755] RBP: ffffb326805afbf8 R08: 0000000000000000 R09: ffff94f5bc86bbf0
> [ 29.803772] R10: ffffb326805afba8 R11: 00000000000ffdc4 R12: ffff94f5bc86a400
> [ 29.804789] R13: 0000000000000000 R14: 00000000ffdc4000 R15: 0000000000000000
> [ 29.805792] FS: 0000000000000000(0000) GS:ffff94f5bfc00000(0000) knlGS:0000000000000000
> [ 29.806923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 29.807736] CR2: 0000000000000304 CR3: 000000003499d000 CR4: 00000000000006f0
> [ 29.808747] Call Trace:
> [ 29.809156] flush_unmaps_timeout+0x126/0x1c0
> [ 29.809800] domain_exit+0xd6/0x100
> [ 29.810322] device_notifier+0x6b/0x70
> [ 29.810902] notifier_call_chain+0x4a/0x70
> [ 29.812822] __blocking_notifier_call_chain+0x47/0x60
> [ 29.814499] blocking_notifier_call_chain+0x16/0x20
> [ 29.816137] device_del+0x233/0x320
> [ 29.817588] pci_remove_bus_device+0x6f/0x110
> [ 29.819133] pci_stop_and_remove_bus_device+0x1a/0x20
> [ 29.820817] pciehp_unconfigure_device+0x7a/0x1d0
> [ 29.822434] pciehp_disable_slot+0x52/0xe0
> [ 29.823931] pciehp_power_thread+0x8a/0xa0
> [ 29.825411] process_one_work+0x18c/0x3a0
> [ 29.826875] worker_thread+0x4e/0x3b0
> [ 29.828263] kthread+0x109/0x140
> [ 29.829564] ? process_one_work+0x3a0/0x3a0
> [ 29.831081] ? kthread_park+0x60/0x60
> [ 29.832464] ret_from_fork+0x25/0x30
> [ 29.833794] Code: 85 ed 74 0b 5b 41 5c 41 5d 41 5e 41 5f 5d c3 49 8b 54 24 60 44 89 f8 0f b6 c4 48 8b 04 c2 48 85 c0 74 49 45 0f b6 ff 4a 8b 3c f8 <80> bf
> [ 29.838514] RIP: iommu_flush_iotlb_psi+0xcf/0x120 RSP: ffffb326805afbd0
> [ 29.840362] CR2: 0000000000000304
> [ 29.841716] ---[ end trace b10ec0d6900868d3 ]---
>
> This patch fixes that problem if applied to v4.13 kernel.
>
> The bug does not exist on latest upstream kernel since it's fixed as a
> side effect of commit 13cf01744608 ("iommu/vt-d: Make use of iova
> deferred flushing", 2017-08-15). But IMHO it's still good to have this
> patch upstream.
>
> CC: Alex Williamson <alex.williamson@xxxxxxxxxx>
> Signed-off-by: Peter Xu <peterx@xxxxxxxxxx>
> ---
> drivers/iommu/intel-iommu.c | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> index 4a2de34895ec..869f37d1f3b7 100644
> --- a/drivers/iommu/intel-iommu.c
> +++ b/drivers/iommu/intel-iommu.c
> @@ -1601,8 +1601,7 @@ static void iommu_flush_iotlb_psi(struct intel_iommu *iommu,
> * flush. However, device IOTLB doesn't need to be flushed in this case.
> */
> if (!cap_caching_mode(iommu->cap) || !map)
> - iommu_flush_dev_iotlb(get_iommu_domain(iommu, did),
> - addr, mask);
> + iommu_flush_dev_iotlb(domain, addr, mask);
> }
>
> static void iommu_flush_iova(struct iova_domain *iovad)

Seems like an oversight of a1ddcbe93010 to leave this redundant lookup,
so it's certainly still relevant to current mainline kernels even if
it's just an efficiency improvement rather than a bug fix. Perhaps we
should include a Fixes tag to link it back to where it's easily fixed.

Fixes: a1ddcbe93010 ("iommu/vt-d: Pass dmar_domain directly into iommu_flush_iotlb_psi")
Reviewed-by: Alex Williamson <alex.williamson@xxxxxxxxxx>