v3.9 - CPU hotplug and microcode earlier loading hits a mutexdeadlock (x86_cpu_hotplug_driver_mutex)

From: Konrad Rzeszutek Wilk
Date: Mon May 06 2013 - 09:00:04 EST



Hey,

As I was fixing the PV HVM's broken CPU hotplug mechanism I discovered
a deadlock in the microcode and generic code.

I am not sure if the ACPI generic mechanism would expose this, but
looking at the flow (arch_register_cpu, then expecting user-space to call
cpu_up), it should trigger this.

Anyhow, this can easily be triggered if a new CPU is added and
from user-space do:

echo 1 > /sys/devices/system/cpu/cpu3/online

on a newly appeared CPU. The deadlock is that the "store_online" in
drivers/base/cpu.c takes the cpu_hotplug_driver_lock() lock, then
calls "cpu_up". "cpu_up" eventually ends up calling "save_mc_for_early"
which also takes the cpu_hotplug_driver_lock() lock.

And here is that kernel thinks of it:

smpboot: Stack at about ffff880075c39f44
smpboot: CPU3: has booted.
microcode: CPU3 sig=0x206a7, pf=0x2, revision=0x25

=============================================
[ INFO: possible recursive locking detected ]
3.9.0upstream-10129-g167af0e #1 Not tainted
---------------------------------------------
sh/2487 is trying to acquire lock:
(x86_cpu_hotplug_driver_mutex){+.+.+.}, at: [<ffffffff81075512>] cpu_hotplug_driver_lock+0x12/0x20

but task is already holding lock:
(x86_cpu_hotplug_driver_mutex){+.+.+.}, at: [<ffffffff81075512>] cpu_hotplug_driver_lock+0x12/0x20

other info that might help us debug this:
Possible unsafe locking scenario:

CPU0
----
lock(x86_cpu_hotplug_driver_mutex);
lock(x86_cpu_hotplug_driver_mutex);

*** DEADLOCK ***

May be due to missing lock nesting notation

6 locks held by sh/2487:
#0: (sb_writers#5){.+.+.+}, at: [<ffffffff811ca48d>] vfs_write+0x17d/0x190
#1: (&buffer->mutex){+.+.+.}, at: [<ffffffff812464ef>] sysfs_write_file+0x3f/0x160
#2: (s_active#20){.+.+.+}, at: [<ffffffff81246578>] sysfs_write_file+0xc8/0x160
#3: (x86_cpu_hotplug_driver_mutex){+.+.+.}, at: [<ffffffff81075512>] cpu_hotplug_driver_lock+0x12/0x20
#4: (cpu_add_remove_lock){+.+.+.}, at: [<ffffffff810961c2>] cpu_maps_update_begin+0x12/0x20
#5: (cpu_hotplug.lock){+.+.+.}, at: [<ffffffff810962a7>] cpu_hotplug_begin+0x27/0x60

stack backtrace:
CPU: 1 PID: 2487 Comm: sh Not tainted 3.9.0upstream-10129-g167af0e #1
Hardware name: Xen HVM domU, BIOS 4.3-unstable 05/03/2013
ffffffff8229b710 ffff880064e75538 ffffffff816fe47e ffff880064e75608
ffffffff81100cb6 ffff880064e75560 ffff88006670e290 ffff880064e75588
ffffffff00000000 ffff88006670e9b8 21710c800c83a1f5 ffff880064e75598
Call Trace:
[<ffffffff816fe47e>] dump_stack+0x19/0x1b
[<ffffffff81100cb6>] __lock_acquire+0x726/0x1890
[<ffffffff8110b352>] ? is_module_text_address+0x22/0x40
[<ffffffff810bbff8>] ? __kernel_text_address+0x58/0x80
[<ffffffff81101eca>] lock_acquire+0xaa/0x190
[<ffffffff81075512>] ? cpu_hotplug_driver_lock+0x12/0x20
[<ffffffff816fee1e>] __mutex_lock_common+0x5e/0x450
[<ffffffff81075512>] ? cpu_hotplug_driver_lock+0x12/0x20
[<ffffffff810d4225>] ? sched_clock_local+0x25/0x90
[<ffffffff81075512>] ? cpu_hotplug_driver_lock+0x12/0x20
[<ffffffff816ff340>] mutex_lock_nested+0x40/0x50
[<ffffffff81075512>] cpu_hotplug_driver_lock+0x12/0x20
[<ffffffff81080757>] save_mc_for_early+0x27/0xf0
[<ffffffff810ffd30>] ? mark_held_locks+0x90/0x150
[<ffffffff81176a5d>] ? get_page_from_freelist+0x46d/0x8e0
[<ffffffff8110029d>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff81056a39>] ? sched_clock+0x9/0x10
[<ffffffff81177275>] ? __alloc_pages_nodemask+0x165/0xa30
[<ffffffff810d4225>] ? sched_clock_local+0x25/0x90
[<ffffffff810d4348>] ? sched_clock_cpu+0xb8/0x110
[<ffffffff810fb88d>] ? trace_hardirqs_off+0xd/0x10
[<ffffffff811a5e79>] ? vmap_page_range_noflush+0x279/0x370
[<ffffffff811a5f9d>] ? map_vm_area+0x2d/0x50
[<ffffffff811a7dce>] ? __vmalloc_node_range+0x18e/0x260
[<ffffffff810812a8>] ? generic_load_microcode+0xb8/0x1c0
[<ffffffff8108135c>] generic_load_microcode+0x16c/0x1c0
[<ffffffff8110917e>] ? generic_exec_single+0x7e/0xb0
[<ffffffff81081470>] ? request_microcode_user+0x20/0x20
[<ffffffff8108142f>] request_microcode_fw+0x7f/0xa0
[<ffffffff813356ab>] ? kobject_uevent+0xb/0x10
[<ffffffff81081004>] microcode_init_cpu+0xf4/0x110
[<ffffffff816f6b8c>] mc_cpu_callback+0x5b/0xb3
[<ffffffff81706d7c>] notifier_call_chain+0x5c/0x120
[<ffffffff810c6359>] __raw_notifier_call_chain+0x9/0x10
[<ffffffff8109616b>] __cpu_notify+0x1b/0x30
[<ffffffff816f72e1>] _cpu_up+0x103/0x14b
[<ffffffff816f7404>] cpu_up+0xdb/0xee
[<ffffffff816eda0a>] store_online+0xba/0x120
[<ffffffff8145f08b>] dev_attr_store+0x1b/0x20
[<ffffffff81246591>] sysfs_write_file+0xe1/0x160
[<ffffffff811ca3ef>] vfs_write+0xdf/0x190
[<ffffffff811ca96d>] SyS_write+0x5d/0xa0
[<ffffffff8133f4fe>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[<ffffffff8170b7a9>] system_call_fastpath+0x16/0x1b

Thoughts?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/