Re: [Issue] mlx5_core probe failed with error -17 when rescanning devices

From: Leon Romanovsky
Date: Fri Aug 20 2021 - 09:54:10 EST

Next message: Randy Dunlap: "Re: HDA codec problem"
Previous message: Krzysztof Kozlowski: "Re: [PATCH] dt-bindings: memory: convert Samsung Exynos DMC to dtschema"
In reply to: Yicong Yang: "[Issue] mlx5_core probe failed with error -17 when rescanning devices"
Next in thread: Yicong Yang: "Re: [Issue] mlx5_core probe failed with error -17 when rescanning devices"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Thu, Aug 19, 2021 at 08:46:32PM +0800, Yicong Yang wrote:
> Hi,
>
> I met an issue of mlx5_core on v5.14-rc6. After performing hot reset of root port
> where mellanox connectX-4 locates, I remove the root port and rescan the pci bus,
> but the driver cannot probe the device. The hirarchy of the devices:
>
> # lspci -tv
> [...]
> +-[0000:80]-+-00.0-[81]--+-00.0 Intel Corporation I350 Gigabit Network Connection
> | | +-00.1 Intel Corporation I350 Gigabit Network Connection
> | | +-00.2 Intel Corporation I350 Gigabit Network Connection
> | | \-00.3 Intel Corporation I350 Gigabit Network Connection
> | +-04.0-[82]--+-00.0 Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
> | | \-00.1 Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
> [...]
> # lspci -vx -s 82:00.0
> 82:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
> Subsystem: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
> Flags: fast devsel, IRQ 42, NUMA node 2, IOMMU group 13
> Memory at 280000000000 (64-bit, prefetchable) [size=32M]
> Expansion ROM at f0700000 [disabled] [size=1M]
> Capabilities: [60] Express Endpoint, MSI 00
> Capabilities: [48] Vital Product Data
> Capabilities: [9c] MSI-X: Enable- Count=64 Masked-
> Capabilities: [c0] Vendor Specific Information: Len=18 <?>
> Capabilities: [40] Power Management version 3
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
> Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
> Capabilities: [1c0] Secondary PCI Express
> Capabilities: [230] Access Control Services
> 00: b3 15 15 10 42 01 10 00 00 00 00 02 08 00 80 00
> 10: 0c 00 00 00 00 28 00 00 00 00 00 00 00 00 00 00
> 20: 00 00 00 00 00 00 00 00 00 00 00 00 b3 15 9c 00
> 30: 00 00 10 f0 60 00 00 00 00 00 00 00 ff 01 00 00
>
> Here is the error log:
>
> [...]
> [13904.335445] mlx5_core 0000:82:00.1: Adding to iommu group 29
> [13904.341460] mlx5_core 0000:82:00.1: firmware version: 14.22.1002
> [13904.347479] mlx5_core 0000:82:00.1: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
> [13904.880505] mlx5_core 0000:82:00.1: E-Switch: Total vports 6, per vport: max uc(1024) max mc(16384)
> [13904.893763] mlx5_core 0000:82:00.1: Port module event: module 1, Cable plugged
> [13904.917246] sysfs: cannot create duplicate filename '/bus/auxiliary/devices/mlx5_core.eth.2'
> [13904.925656] CPU: 64 PID: 957 Comm: kworker/64:2 Tainted: G W 5.13.0-rc3-bisect+ #37
> [13904.934573] Hardware name: Huawei TaiShan 200 (Model 2280)/BC82AMDD, BIOS 2280-V2 CS V5.B170.01 06/30/2021
> [13904.944181] Workqueue: events work_for_cpu_fn
> [13904.948522] Call trace:
> [13904.950949] dump_backtrace+0x0/0x19c
> [13904.954598] show_stack+0x24/0x30
> [13904.957899] dump_stack+0xc8/0x104
> [13904.961287] sysfs_warn_dup+0x70/0x90
> [13904.964936] sysfs_do_create_link_sd+0xf8/0x100
> [13904.969448] sysfs_create_link+0x2c/0x50
> [13904.973355] bus_add_device+0x74/0x120
> [13904.977088] device_add+0x2f4/0x840
> [13904.980562] __auxiliary_device_add+0x4c/0xb0
> [13904.984901] add_adev+0x9c/0xf0
> [13904.988030] mlx5_rescan_drivers_locked.part.0+0x154/0x1c0
> [13904.993491] mlx5_register_device+0x80/0xe0
> [13904.997656] mlx5_init_one+0x224/0x4dc
> [13905.001389] probe_one+0x1dc/0x4cc
> [13905.004776] local_pci_probe+0x4c/0xc0
> [13905.008500] work_for_cpu_fn+0x28/0x40
> [13905.012232] process_one_work+0x1dc/0x48c
> [13905.016223] worker_thread+0x2e8/0x464
> [13905.019955] kthread+0x168/0x16c
> [13905.023171] ret_from_fork+0x10/0x18
> [13905.026764] auxiliary mlx5_core.eth.2: adding auxiliary device failed!: -17
> [13905.033698] mlx5_core 0000:82:00.1: add_drivers:424:(pid 957): Device[0] (eth) failed to load
> [13907.274212] mlx5_core 0000:82:00.1: E-Switch: cleanup
> [13907.896435] mlx5_core 0000:82:00.1: probe_one:1484:(pid 957): mlx5_init_one failed with error code -17
> [13907.905855] mlx5_core: probe of 0000:82:00.1 failed with error -17
>
> The test script I used:
>
> #!/bin/bash
> setpci -s 80:04.0 0x3e.b=0x43 # perform hot reset by setting secondary bus reset bit
> setpci -s 80:04.0 0x3e.b=0x3 # clear secondary bus reset bit
> echo 1 > /sys/bus/pci/devices/0000:80:04.0/remove
> echo 1 > /sys/bus/pci/rescan
>
> There is no such issue on v5.13-rc4. So I did a git bisect and here is log:
>
> [...]
> # good: [c7d2ef5dd4b03ed0ee1d13bc0c55f9cf62d49bd6] net/packet: annotate accesses to po->bind
> git bisect good c7d2ef5dd4b03ed0ee1d13bc0c55f9cf62d49bd6
> # bad: [bc39f6792ede3a830b1893c9133636b9f6991e59] Merge tag 'mlx5-fixes-2021-06-16' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux
> git bisect bad bc39f6792ede3a830b1893c9133636b9f6991e59
> # good: [94a4b8414d3e91104873007b659252f855ee344a] net/mlx5: Fix error path for set HCA defaults
> git bisect good 94a4b8414d3e91104873007b659252f855ee344a
> # good: [65fb7d109abe3a1a9f1c2d3ba7e1249bc978d5f0] net/mlx5: DR, Fix STEv1 incorrect L3 decapsulation padding
> git bisect good 65fb7d109abe3a1a9f1c2d3ba7e1249bc978d5f0
> # bad: [0232fc2ddcf4ffe01069fd1aa07922652120f44a] net/mlx5: Reset mkey index on creation
> git bisect bad 0232fc2ddcf4ffe01069fd1aa07922652120f44a
> # bad: [a5ae8fc9058e37437c8c1f82b3d412b4abd1b9e6] net/mlx5e: Don't create devices during unload flow
> git bisect bad a5ae8fc9058e37437c8c1f82b3d412b4abd1b9e6
> # first bad commit: [a5ae8fc9058e37437c8c1f82b3d412b4abd1b9e6] net/mlx5e: Don't create devices during unload flow
>
> So after reverting a5ae8fc9058e ("net/mlx5e: Don't create devices during unload flow") the issue
> is resolved. Seems the devlink file is not removed properly and the unremoved file fails the
> probe process. I don't know about the driver so I hope somebody can get this fixed.

Thank you for your report,

Your command "echo 1 > /sys/bus/pci/devices/0000:80:04.0/remove" was
supposed to remove all auxiliary devices, but because of commit
a5ae8fc9058e ("net/mlx5e: Don't create devices during unload flow"),
the deletion was skipped.

Can you please try this patch instead of revert?

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/dev.c b/drivers/net/ethernet/mellanox/mlx5/core/dev.c
index ff6b03dc7e32..e8093c4e09d4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/dev.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/dev.c
@@ -450,7 +450,7 @@ int mlx5_register_device(struct mlx5_core_dev *dev)
void mlx5_unregister_device(struct mlx5_core_dev *dev)
{
mutex_lock(&mlx5_intf_mutex);
- dev->priv.flags |= MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV;
+ dev->priv.flags = MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV;
mlx5_rescan_drivers_locked(dev);
mutex_unlock(&mlx5_intf_mutex);
}

>
> Thanks,
> Yicong
>
>

Next message: Randy Dunlap: "Re: HDA codec problem"
Previous message: Krzysztof Kozlowski: "Re: [PATCH] dt-bindings: memory: convert Samsung Exynos DMC to dtschema"
In reply to: Yicong Yang: "[Issue] mlx5_core probe failed with error -17 when rescanning devices"
Next in thread: Yicong Yang: "Re: [Issue] mlx5_core probe failed with error -17 when rescanning devices"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]