Re: Please add to stable: module: don't unlink the module untilwe've removed all exposure.

[Ben Greear]

On 06/05/2013 12:15 AM, Tejun Heo wrote:
> Hello,
>
> On Wed, Jun 05, 2013 at 01:47:43PM +0930, Rusty Russell wrote:
> > > I have some printk debugging in (see bottom of email) and was using a serial console, so things
> > > were probably running a bit slower than on most systems.  Here is trace
> > > from my kernel with local patches and not so much debugging enabled
> > > (this is NOT a clean upstream kernel, though I reproduced the same thing
> > > with a clean upstream 3.9.4 kernel plus your module unlink patch yesterday).
> >
> > Tejun CC'd.  We can't be running two stop machines in parallel, since
> > there's a mutex (and there's also one in the module code).
> >
> > > __stop_machine, num-threads: 4, fn: __try_stop_module  data: ffff8801c6ae7f28
> > > cpu: 0 loops: 1 jiffies: 4299011449  timeout: 4299011448 curstate: 0  smdata->state: 1  thread_ack: 4
> > > cpu: 1 loops: 1 jiffies: 4299011449  timeout: 4299011448 curstate: 0  smdata->state: 1  thread_ack: 4
> > > cpu: 2 loops: 1 jiffies: 4299011449  timeout: 4299011448 curstate: 0  smdata->state: 1  thread_ack: 3
> > > cpu: 3 loops: 1 jiffies: 4299011449  timeout: 4299011448 curstate: 0  smdata->state: 1  thread_ack: 2
> > > __stop_machine, num-threads: 4, fn: __unlink_module  data: ffffffffa0aeeab0
> > > cpu: 0 loops: 1 jiffies: 4299011501  timeout: 4299011500 curstate: 0  smdata->state: 1  thread_ack: 4
> > > cpu: 1 loops: 1 jiffies: 4299011501  timeout: 4299011500 curstate: 0  smdata->state: 1  thread_ack: 4
> > > cpu: 3 loops: 1 jiffies: 4299011501  timeout: 4299011500 curstate: 0  smdata->state: 1  thread_ack: 3
> > > ath: wiphy0: Failed to stop TX DMA, queues=0x005!
> > > cpu: 2 loops: 1 jiffies: 4299011501  timeout: 4299011500 curstate: 0  smdata->state: 1  thread_ack: 2
>
> A bit confused.  I looked at the code again and it still seems
> properly interlocked.  Can't see how the above scenario could happen.
> Actually, I'm not even sure, what I'm looking at.  Is the above from
> the lockup?  It'd be helpful if we can get more traces from the locked
> up state.  Shouldn't be hard to detect.  Dumb timeout based detection
> should work fine.  How easily can you reproduce the issue?

I can easily reproduce the problem.  First, my machine info:

dual-core i7, 4 cpu threads total.
8 GB RAM
64-bit kernel, Fedora-17 64-bit OS
2 ath9k NICs, each with 200 virtual stations configured.
  I don't think the virtual stations matter to much, as we reproduce easily
  enough with fewer, but I think that maybe the extra IRQ load helps trigger
  the bug.
I added lots of printk debugging, which makes it easier to hit the
lockup.

There is an 'ath' message in the log above, but most traces have nothing related
to ath in them, so I think that thing above is not related.  (That is a common
ath error message for years, and at least on my hardware, does not appear to cause any
serious problems).

Here below is another trace with even more debugging printouts.  After a while,
it stopped even printing out cpu-hung messages..not sure if that is an important clue or
not.


One pattern I notice repeating for at least most of the hangs is that all but one
CPU thread has irqs disabled and is in state 2.  But, there will be one thread
in state 1 that still has IRQs enabled and it is reported to be in soft-lockup
instead of hard-lockup.  In 'sysrq l' it always shows some IRQ processing,
but typically that of the sysrq itself.  I added printk that would always
print if the thread notices that smdata->state != curstate, and the soft-lockup
thread (cpu 2 below) never shows that message.

I thought it might be because it was reading stale smdata->state, so I changed
that to atomic_t hoping that would mitigate that.  I also tried adding smp_rmb()
below the cpu_relax().  Neither had any affect, so I am left assuming that the
thread instead is stuck handling IRQs and never gets out of the IRQ handler.

Maybe since I have 2 real cores, and 3 processes busy-spinning on their CPU cores,
the remaining process can just never handle all the IRQs and get back to the
cpu shutdown state machine?  The various soft-hang stacks below show at least slightly
different stacks, so I assume that thread is doing at least something.


__stop_machine, num-threads: 4, fn: __try_stop_module(ffffffff810e86da)  data: ffff8802202abf28  smdata: ffff8802202abe58
set_state, cpu: 3  state: 0 newstate: 1 smdata: ffff8802202abe58  fn: ffffffff810e86da
cpu: 1 loops: 1 jiffies: 4294739830  timeout: 4294739829 curstate: 0  smdata->state: 1  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 3 loops: 1 jiffies: 4294739830  timeout: 4294739829 curstate: 0  smdata->state: 1  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 3 loops: 1 jiffies: 4294739830 curstate: 0  smdata->state: 1  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 2 loops: 1 jiffies: 4294739830  timeout: 4294739829 curstate: 0  smdata->state: 1  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 0 loops: 1 jiffies: 4294739830  timeout: 4294739829 curstate: 0  smdata->state: 1  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 3 ack_state jiffies: 4294739830 smdata->state: 1  thread_ack: 4 smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 2 loops: 1 jiffies: 4294739830 curstate: 0  smdata->state: 1  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 0 loops: 1 jiffies: 4294739830 curstate: 0  smdata->state: 1  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 3 ack_state end jiffies: 4294739830 smdata->state: 1  thread_ack: 3  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 2 ack_state jiffies: 4294739830 smdata->state: 1  thread_ack: 3 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 0 ack_state jiffies: 4294739830 smdata->state: 1  thread_ack: 3 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 2 ack_state end jiffies: 4294739830 smdata->state: 1  thread_ack: 2  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 0 ack_state end jiffies: 4294739830 smdata->state: 1  thread_ack: 1  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 1 loops: 1 jiffies: 4294739981 curstate: 0  smdata->state: 1  thread_ack: 1  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 1 ack_state jiffies: 4294739993 smdata->state: 1  thread_ack: 1 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 1  increment state, smdata: ffff8802202abe58 fn: ffffffff810e86da
set_state, cpu: 1  state: 1 newstate: 2 smdata: ffff8802202abe58  fn: ffffffff810e86da
cpu: 1 ack_state end jiffies: 4294740017 smdata->state: 2  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 3 loops: 24479339 jiffies: 4294740017 curstate: 1  smdata->state: 2  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 0 loops: 23433125 jiffies: 4294740017 curstate: 1  smdata->state: 2  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 2 loops: 23440505 jiffies: 4294740017 curstate: 1  smdata->state: 2  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 3 ack_state jiffies: 4294740017 smdata->state: 2  thread_ack: 4 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 0 ack_state jiffies: 4294740017 smdata->state: 2  thread_ack: 4 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 2 ack_state jiffies: 4294740017 smdata->state: 2  thread_ack: 4 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 3 ack_state end jiffies: 4294740017 smdata->state: 2  thread_ack: 3  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 0 ack_state end jiffies: 4294740017 smdata->state: 2  thread_ack: 2  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 2 ack_state end jiffies: 4294740017 smdata->state: 2  thread_ack: 1  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 1 loops: 2 jiffies: 4294740017 curstate: 1  smdata->state: 2  thread_ack: 1  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 1 ack_state jiffies: 4294740017 smdata->state: 2  thread_ack: 1 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 1  increment state, smdata: ffff8802202abe58 fn: ffffffff810e86da
set_state, cpu: 1  state: 2 newstate: 3 smdata: ffff8802202abe58  fn: ffffffff810e86da
cpu: 1 ack_state end jiffies: 4294740017 smdata->state: 3  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 0 loops: 42466689 jiffies: 4294740017 curstate: 2  smdata->state: 3  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 2 loops: 42473881 jiffies: 4294740017 curstate: 2  smdata->state: 3  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 3 loops: 43825073 jiffies: 4294740017 curstate: 2  smdata->state: 3  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
calling fn: cpu: 0 loops: 42466689 curstate: 3  smdata->state: 3  thread_ack: 4  smdata: ffff8802202abe58  fn: ffffffff810e86da
cpu: 2 ack_state jiffies: 4294740017 smdata->state: 3  thread_ack: 4 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 3 ack_state jiffies: 4294740017 smdata->state: 3  thread_ack: 4 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 0 ack_state jiffies: 4294740017 smdata->state: 3  thread_ack: 4 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 2 ack_state end jiffies: 4294740017 smdata->state: 3  thread_ack: 3  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 3 ack_state end jiffies: 4294740017 smdata->state: 3  thread_ack: 2  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 0 ack_state end jiffies: 4294740017 smdata->state: 3  thread_ack: 1  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 1 loops: 3 jiffies: 4294740017 curstate: 2  smdata->state: 3  thread_ack: 1  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 1 ack_state jiffies: 4294740017 smdata->state: 3  thread_ack: 1 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 1  increment state, smdata: ffff8802202abe58 fn: ffffffff810e86da
set_state, cpu: 1  state: 3 newstate: 4 smdata: ffff8802202abe58  fn: ffffffff810e86da
cpu: 1 ack_state end jiffies: 4294740017 smdata->state: 4  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 2 loops: 62146520 jiffies: 4294740017 curstate: 3  smdata->state: 4  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 0 loops: 62138953 jiffies: 4294740017 curstate: 3  smdata->state: 4  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 3 loops: 64625424 jiffies: 4294740017 curstate: 3  smdata->state: 4  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 2 ack_state jiffies: 4294740017 smdata->state: 4  thread_ack: 4 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 0 ack_state jiffies: 4294740017 smdata->state: 4  thread_ack: 4 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 3 ack_state jiffies: 4294740017 smdata->state: 4  thread_ack: 4 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 2 ack_state end jiffies: 4294740017 smdata->state: 4  thread_ack: 3  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 0 ack_state end jiffies: 4294740017 smdata->state: 4  thread_ack: 2  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 3 ack_state end jiffies: 4294740017 smdata->state: 4  thread_ack: 1  smdata: ffff8802202abe58 fn: ffffffff810e86da
state-change: cpu: 1 loops: 4 jiffies: 4294740435 curstate: 3  smdata->state: 4  thread_ack: 1  smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 1 ack_state jiffies: 4294740447 smdata->state: 4  thread_ack: 1 smdata: ffff8802202abe58 fn: ffffffff810e86da
cpu: 1  increment state, smdata: ffff8802202abe58 fn: ffffffff810e86da
set_state, cpu: 1  state: 4 newstate: 5 smdata: ffff8802202abe58  fn: ffffffff810e86da
cpu: 1 ack_state end jiffies: 4294740471 smdata->state: 5  thread_ack: 4  smdata: ffff8802202abe58 fn: ffffffff810e86da
__stop_machine, num-threads: 4, fn: __unlink_module(ffffffff810e817b)  data: ffffffffa0ac5ab0  smdata: ffff8802202abe18
set_state, cpu: 0  state: 0 newstate: 1 smdata: ffff8802202abe18  fn: ffffffff810e817b
cpu: 0 loops: 1 jiffies: 4294740500  timeout: 4294740499 curstate: 0  smdata->state: 1  thread_ack: 4  smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 3 loops: 1 jiffies: 4294740500  timeout: 4294740499 curstate: 0  smdata->state: 1  thread_ack: 4  smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 1 loops: 1 jiffies: 4294740500  timeout: 4294740499 curstate: 0  smdata->state: 1  thread_ack: 4  smdata: ffff8802202abe18 fn: ffffffff810e817b
state-change: cpu: 3 loops: 1 jiffies: 4294740500 curstate: 0  smdata->state: 1  thread_ack: 4  smdata: ffff8802202abe18 fn: ffffffff810e817b
state-change: cpu: 1 loops: 1 jiffies: 4294740500 curstate: 0  smdata->state: 1  thread_ack: 4  smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 3 ack_state jiffies: 4294740500 smdata->state: 1  thread_ack: 4 smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 1 ack_state jiffies: 4294740500 smdata->state: 1  thread_ack: 4 smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 3 ack_state end jiffies: 4294740500 smdata->state: 1  thread_ack: 3  smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 1 ack_state end jiffies: 4294740500 smdata->state: 1  thread_ack: 2  smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 2 loops: 1 jiffies: 4294740501  timeout: 4294740500 curstate: 0  smdata->state: 1  thread_ack: 2  smdata: ffff8802202abe18 fn: ffffffff810e817b
state-change: cpu: 2 loops: 1 jiffies: 4294740501 curstate: 0  smdata->state: 1  thread_ack: 2  smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 2 ack_state jiffies: 4294740501 smdata->state: 1  thread_ack: 2 smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 2 ack_state end jiffies: 4294740501 smdata->state: 1  thread_ack: 1  smdata: ffff8802202abe18 fn: ffffffff810e817b
state-change: cpu: 0 loops: 1 jiffies: 4294740651 curstate: 0  smdata->state: 1  thread_ack: 1  smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 0 ack_state jiffies: 4294740664 smdata->state: 1  thread_ack: 1 smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 0  increment state, smdata: ffff8802202abe18 fn: ffffffff810e817b
set_state, cpu: 0  state: 1 newstate: 2 smdata: ffff8802202abe18  fn: ffffffff810e817b
cpu: 0 ack_state end jiffies: 4294740688 smdata->state: 2  thread_ack: 4  smdata: ffff8802202abe18 fn: ffffffff810e817b
state-change: cpu: 3 loops: 23659570 jiffies: 4294740688 curstate: 1  smdata->state: 2  thread_ack: 4  smdata: ffff8802202abe18 fn: ffffffff810e817b
state-change: cpu: 1 loops: 23670374 jiffies: 4294740688 curstate: 1  smdata->state: 2  thread_ack: 4  smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 3 ack_state jiffies: 4294740688 smdata->state: 2  thread_ack: 4 smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 1 ack_state jiffies: 4294740688 smdata->state: 2  thread_ack: 4 smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 3 ack_state end jiffies: 4294740688 smdata->state: 2  thread_ack: 3  smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 1 ack_state end jiffies: 4294740688 smdata->state: 2  thread_ack: 2  smdata: ffff8802202abe18 fn: ffffffff810e817b
state-change: cpu: 0 loops: 2 jiffies: 4294740688 curstate: 1  smdata->state: 2  thread_ack: 2  smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 0 ack_state jiffies: 4294740688 smdata->state: 2  thread_ack: 2 smdata: ffff8802202abe18 fn: ffffffff810e817b
cpu: 0 ack_state end jiffies: 4294740688 smdata->state: 2  thread_ack: 1  smdata: ffff8802202abe18 fn: ffffffff810e817b
------------[ cut here ]------------
WARNING: at /home/greearb/git/linux-3.9.dev.y/kernel/watchdog.c:245 watchdog_overflow_callback+0x9b/0xa6()
Hardware name: To be filled by O.E.M.
Watchdog detected hard LOCKUP on cpu 0
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
Pid: 8, comm: migration/0 Tainted: G         C O 3.9.4+ #70
Call Trace:
 <NMI>  [<ffffffff810963e1>] warn_slowpath_common+0x85/0x9f
 [<ffffffff8109649e>] warn_slowpath_fmt+0x46/0x48
 [<ffffffff810c4dfb>] ? sched_clock_cpu+0x44/0xce
 [<ffffffff81103cec>] watchdog_overflow_callback+0x9b/0xa6
 [<ffffffff811336cb>] __perf_event_overflow+0x137/0x1cb
 [<ffffffff8101db3f>] ? x86_perf_event_set_period+0x107/0x113
 [<ffffffff81133c1a>] perf_event_overflow+0x14/0x16
 [<ffffffff810230dc>] intel_pmu_handle_irq+0x2b0/0x32d
 [<ffffffff815cbdd1>] perf_event_nmi_handler+0x19/0x1b
 [<ffffffff815cb64a>] nmi_handle+0x55/0x7e
 [<ffffffff815cb71b>] do_nmi+0xa8/0x2db
 [<ffffffff815cadb1>] end_repeat_nmi+0x1e/0x2e
 [<ffffffff810f9945>] ? stop_machine_cpu_stop+0x88/0x267
 [<ffffffff810f9945>] ? stop_machine_cpu_stop+0x88/0x267
 [<ffffffff810f9945>] ? stop_machine_cpu_stop+0x88/0x267
 <<EOE>>  [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
---[ end trace 9767454fd3f66a7f ]---
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109daab>]  [<ffffffff8109daab>] __do_softirq+0x93/0x23c
RSP: 0018:ffff88022bd03ef8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: 0000000000000246 RCX: ffff88022bd0e506
RDX: ffff880222218010 RSI: ffff88022bd0e4f0 RDI: 0000000000000002
RBP: ffff88022bd03f78 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e68
R13: ffffffff815d15dd R14: ffff88022bd03f78 R15: ffff880222218000
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000038
 ffff880200000006 000000111671cb12 000000111671cb12 ffff88022bd0db00
Call Trace:
 <IRQ>
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 89 45 90 49 8d 44 24 10 4c 89 65 b0 65 8b 14 25 20 b0 00 00 48 89 45 88 89 55 ac 65 c7 04 25 40 13 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 50 
a0 8
------------[ cut here ]------------
WARNING: at /home/greearb/git/linux-3.9.dev.y/kernel/watchdog.c:245 watchdog_overflow_callback+0x9b/0xa6()
Hardware name: To be filled by O.E.M.
Watchdog detected hard LOCKUP on cpu 3
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
Pid: 29, comm: migration/3 Tainted: G        WC O 3.9.4+ #70
Call Trace:
 <NMI>  [<ffffffff810963e1>] warn_slowpath_common+0x85/0x9f
 [<ffffffff8109649e>] warn_slowpath_fmt+0x46/0x48
 [<ffffffff810c4dfb>] ? sched_clock_cpu+0x44/0xce
 [<ffffffff81103cec>] watchdog_overflow_callback+0x9b/0xa6
 [<ffffffff811336cb>] __perf_event_overflow+0x137/0x1cb
 [<ffffffff8101db3f>] ? x86_perf_event_set_period+0x107/0x113
 [<ffffffff81133c1a>] perf_event_overflow+0x14/0x16
 [<ffffffff810230dc>] intel_pmu_handle_irq+0x2b0/0x32d
 [<ffffffff815cbdd1>] perf_event_nmi_handler+0x19/0x1b
 [<ffffffff815cb64a>] nmi_handle+0x55/0x7e
 [<ffffffff815cb71b>] do_nmi+0xa8/0x2db
 [<ffffffff815cadb1>] end_repeat_nmi+0x1e/0x2e
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 <<EOE>>  [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
---[ end trace 9767454fd3f66a80 ]---
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109d6b7>]  [<ffffffff8109d6b7>] tasklet_action+0x61/0xcc
RSP: 0018:ffff88022bd03ed8  EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffff880222185dc0 RCX: ffff88022bd0e506
RDX: ffff8802186f2fa8 RSI: ffff88022bd0e4f0 RDI: ffffffff81a050b0
RBP: ffff88022bd03ee8 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e48
R13: ffffffff815d15dd R14: ffff88022bd03ee8 R15: ffff8802186f2fb0
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffffffff81a050b0 ffff880222218000 ffff88022bd03f78 ffffffff8109db1f
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000030
Call Trace:
 <IRQ>
 [<ffffffff8109db1f>] __do_softirq+0x107/0x23c
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: da 00 00 65 48 89 04 25 88 e0 00 00 fb 66 66 90 66 66 90 eb 77 48 8b 1a 4c 8d 62 08 f0 0f ba 6a 08 01 19 c0 85 c0 75 2d 8b 42 10 <85> c0 75 20 f0 41 0f ba 
34 2
------------[ cut here ]------------
WARNING: at /home/greearb/git/linux-3.9.dev.y/kernel/watchdog.c:245 watchdog_overflow_callback+0x9b/0xa6()
Hardware name: To be filled by O.E.M.
Watchdog detected hard LOCKUP on cpu 1
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
Pid: 17, comm: migration/1 Tainted: G        WC O 3.9.4+ #70
Call Trace:
 <NMI>  [<ffffffff810963e1>] warn_slowpath_common+0x85/0x9f
 [<ffffffff8109649e>] warn_slowpath_fmt+0x46/0x48
 [<ffffffff810c4dfb>] ? sched_clock_cpu+0x44/0xce
 [<ffffffff81103cec>] watchdog_overflow_callback+0x9b/0xa6
 [<ffffffff811336cb>] __perf_event_overflow+0x137/0x1cb
 [<ffffffff8101db3f>] ? x86_perf_event_set_period+0x107/0x113
 [<ffffffff81133c1a>] perf_event_overflow+0x14/0x16
 [<ffffffff810230dc>] intel_pmu_handle_irq+0x2b0/0x32d
 [<ffffffff815cbdd1>] perf_event_nmi_handler+0x19/0x1b
 [<ffffffff815cb64a>] nmi_handle+0x55/0x7e
 [<ffffffff815cb71b>] do_nmi+0xa8/0x2db
 [<ffffffff815cadb1>] end_repeat_nmi+0x1e/0x2e
 [<ffffffff810f99a8>] ? stop_machine_cpu_stop+0xeb/0x267
 [<ffffffff810f99a8>] ? stop_machine_cpu_stop+0xeb/0x267
 [<ffffffff810f99a8>] ? stop_machine_cpu_stop+0xeb/0x267
 <<EOE>>  [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff815ca1f4>] ? _raw_spin_unlock_irq+0x10/0x36
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
---[ end trace 9767454fd3f66a81 ]---
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109daab>]  [<ffffffff8109daab>] __do_softirq+0x93/0x23c
RSP: 0018:ffff88022bd03ef8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: 0000000000000246 RCX: ffff88022bd0e506
RDX: ffff880222218010 RSI: ffff88022bd0e4f0 RDI: 0000000000000002
RBP: ffff88022bd03f78 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e68
R13: ffffffff815d15dd R14: ffff88022bd03f78 R15: ffff880222218000
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000038
 ffff880200000006 000000111671cb12 000000111671cb12 ffff88022bd0db00
Call Trace:
 <IRQ>
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 89 45 90 49 8d 44 24 10 4c 89 65 b0 65 8b 14 25 20 b0 00 00 48 89 45 88 89 55 ac 65 c7 04 25 40 13 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 50 
a0 8
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109d712>]  [<ffffffff8109d712>] tasklet_action+0xbc/0xcc
RSP: 0018:ffff88022bd03ed8  EFLAGS: 00000202
RAX: 0000000000000040 RBX: ffff880222185dc0 RCX: ffff88022bd0e506
RDX: ffff8802186f2fa8 RSI: ffff88022bd0e4f0 RDI: 0000000000000006
RBP: ffff88022bd03ee8 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e48
R13: ffffffff815d15dd R14: ffff88022bd03ee8 R15: ffff8802186f2fb0
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffffffff81a050b0 ffff880222218000 ffff88022bd03f78 ffffffff8109db1f
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000030
Call Trace:
 <IRQ>
 [<ffffffff8109db1f>] __do_softirq+0x107/0x23c
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 90 bf 06 00 00 00 48 c7 02 00 00 00 00 65 48 8b 04 25 88 e0 00 00 48 89 10 65 48 89 14 25 88 e0 00 00 e8 1c fe ff ff fb 66 66 90 <66> 66 90 48 89 da 48 85 
d2 7
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109d6e1>]  [<ffffffff8109d6e1>] tasklet_action+0x8b/0xcc
RSP: 0018:ffff88022bd03ed8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff880222185dc0 RCX: ffff88022bd0e506
RDX: ffff8802186f2fa8 RSI: ffff88022bd0e4f0 RDI: ffffffff81a050b0
RBP: ffff88022bd03ee8 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e48
R13: ffffffff815d15dd R14: ffff88022bd03ee8 R15: ffff8802186f2fb0
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffffffff81a050b0 ffff880222218000 ffff88022bd03f78 ffffffff8109db1f
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000030
Call Trace:
 <IRQ>
 [<ffffffff8109db1f>] __do_softirq+0x107/0x23c
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 10 85 c0 75 20 f0 41 0f ba 34 24 00 19 c0 85 c0 75 04 0f 0b eb fe 48 8b 7a 20 ff 52 18 f0 41 80 24 24 fd eb 3a f0 41 80 24 24 fd <fa> 66 66 90 66 66 90 bf 
06 0
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109d6e1>]  [<ffffffff8109d6e1>] tasklet_action+0x8b/0xcc
RSP: 0018:ffff88022bd03ed8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff880222185dc0 RCX: ffff88022bd0e506
RDX: ffff8802186f2fa8 RSI: ffff88022bd0e4f0 RDI: ffffffff81a050b0
RBP: ffff88022bd03ee8 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e48
R13: ffffffff815d15dd R14: ffff88022bd03ee8 R15: ffff8802186f2fb0
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffffffff81a050b0 ffff880222218000 ffff88022bd03f78 ffffffff8109db1f
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000030
Call Trace:
 <IRQ>
 [<ffffffff8109db1f>] __do_softirq+0x107/0x23c
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 10 85 c0 75 20 f0 41 0f ba 34 24 00 19 c0 85 c0 75 04 0f 0b eb fe 48 8b 7a 20 ff 52 18 f0 41 80 24 24 fd eb 3a f0 41 80 24 24 fd <fa> 66 66 90 66 66 90 bf 
06 0
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109dac6>]  [<ffffffff8109dac6>] __do_softirq+0xae/0x23c
RSP: 0018:ffff88022bd03ef8  EFLAGS: 00000282
RAX: 0000000000000000 RBX: 0000000000000246 RCX: ffff88022bd0e506
RDX: ffff880222218010 RSI: ffff88022bd0e4f0 RDI: 0000000000000002
RBP: ffff88022bd03f78 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e68
R13: ffffffff815d15dd R14: ffff88022bd03f78 R15: ffff880222218000
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000028
 ffff880200000006 000000111671cb12 000000111671cb12 ffff88022bd0db00
Call Trace:
 <IRQ>
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 65 c7 04 25 40 13 01 00 00 00 00 00 fb 66 66 90 66 66 90 48 c7 c3 80 50 a0 81 48 c7 45 b8 00 00 00 00 65 4c 8b 2c 25 48 c6 00 00 <41> f6 c7 01 0f 84 ba 00 
00 0
BUG: soft lockup - CPU#2 stuck for 23s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109d69c>]  [<ffffffff8109d69c>] tasklet_action+0x46/0xcc
RSP: 0018:ffff88022bd03ed8  EFLAGS: 00000282
RAX: ffff88022bd0e080 RBX: ffff880222185dc0 RCX: ffff88022bd0e506
RDX: ffff8802186f2fa8 RSI: ffff88022bd0e4f0 RDI: ffffffff81a050b0
RBP: ffff88022bd03ee8 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e48
R13: ffffffff815d15dd R14: ffff88022bd03ee8 R15: ffff880222218000
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffffffff81a050b0 ffff880222218000 ffff88022bd03f78 ffffffff8109db1f
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000030
Call Trace:
 <IRQ>
 [<ffffffff8109db1f>] __do_softirq+0x107/0x23c
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 48 8b 14 25 80 e0 00 00 65 48 c7 04 25 80 e0 00 00 00 00 00 00 65 48 03 04 25 d8 da 00 00 65 48 89 04 25 88 e0 00 00 fb 66 66 90 <66> 66 90 eb 77 48 8b 1a 
4c 8
BUG: soft lockup - CPU#2 stuck for 23s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109d712>]  [<ffffffff8109d712>] tasklet_action+0xbc/0xcc
RSP: 0018:ffff88022bd03ed8  EFLAGS: 00000202
RAX: 0000000000000040 RBX: ffff880222185dc0 RCX: ffff88022bd0e506
RDX: ffff8802186f2fa8 RSI: ffff88022bd0e4f0 RDI: 0000000000000006
RBP: ffff88022bd03ee8 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e48
R13: ffffffff815d15dd R14: ffff88022bd03ee8 R15: ffff8802186f2fb0
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffffffff81a050b0 ffff880222218000 ffff88022bd03f78 ffffffff8109db1f
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000030
Call Trace:
 <IRQ>
 [<ffffffff8109db1f>] __do_softirq+0x107/0x23c
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 90 bf 06 00 00 00 48 c7 02 00 00 00 00 65 48 8b 04 25 88 e0 00 00 48 89 10 65 48 89 14 25 88 e0 00 00 e8 1c fe ff ff fb 66 66 90 <66> 66 90 48 89 da 48 85 
d2 7
BUG: soft lockup - CPU#2 stuck for 23s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109d712>]  [<ffffffff8109d712>] tasklet_action+0xbc/0xcc
RSP: 0018:ffff88022bd03ed8  EFLAGS: 00000202
RAX: 0000000000000040 RBX: ffff880222185dc0 RCX: ffff88022bd0e506
RDX: ffff8802186f2fa8 RSI: ffff88022bd0e4f0 RDI: 0000000000000006
RBP: ffff88022bd03ee8 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e48
R13: ffffffff815d15dd R14: ffff88022bd03ee8 R15: ffff8802186f2fb0
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffffffff81a050b0 ffff880222218000 ffff88022bd03f78 ffffffff8109db1f
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000030
Call Trace:
 <IRQ>
 [<ffffffff8109db1f>] __do_softirq+0x107/0x23c
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 90 bf 06 00 00 00 48 c7 02 00 00 00 00 65 48 8b 04 25 88 e0 00 00 48 89 10 65 48 89 14 25 88 e0 00 00 e8 1c fe ff ff fb 66 66 90 <66> 66 90 48 89 da 48 85 
d2 7
BUG: soft lockup - CPU#2 stuck for 23s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109d6ae>]  [<ffffffff8109d6ae>] tasklet_action+0x58/0xcc
RSP: 0018:ffff88022bd03ed8  EFLAGS: 00000282
RAX: ffff88022bd0e080 RBX: ffff880222185dc0 RCX: ffff88022bd0e506
RDX: ffff8802186f2fa8 RSI: ffff88022bd0e4f0 RDI: ffffffff81a050b0
RBP: ffff88022bd03ee8 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e48
R13: ffffffff815d15dd R14: ffff88022bd03ee8 R15: ffff8802186f2fb0
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffffffff81a050b0 ffff880222218000 ffff88022bd03f78 ffffffff8109db1f
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000030
Call Trace:
 <IRQ>
 [<ffffffff8109db1f>] __do_softirq+0x107/0x23c
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 00 00 00 65 48 03 04 25 d8 da 00 00 65 48 89 04 25 88 e0 00 00 fb 66 66 90 66 66 90 eb 77 48 8b 1a 4c 8d 62 08 f0 0f ba 6a 08 01 <19> c0 85 c0 75 2d 8b 42 
10 8
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8110a66f>]  [<ffffffff8110a66f>] rcu_bh_qs+0x9/0x22
RSP: 0018:ffff88022bd03ee8  EFLAGS: 00000246
RAX: 0000000000000040 RBX: 0000000000000246 RCX: ffff88022bd0e506
RDX: 0000000000000000 RSI: ffff88022bd0e4f0 RDI: 0000000000000002
RBP: ffff88022bd03ee8 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e58
R13: ffffffff815d15dd R14: ffff88022bd03ee8 R15: ffff880222218000
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffff88022bd03f78 ffffffff8109db8a ffff88022bd03f08 ffff880222218010
 ffff880222219fd8 04208040810b79ef 00000000fffc8ad1 000000022bd0e150
 ffff880222218000 0000000000000030 ffff880200000006 000000111671cb12
Call Trace:
 <IRQ>
 [<ffffffff8109db8a>] __do_softirq+0x172/0x23c
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 55 48 89 e5 66 66 66 66 90 48 c7 c0 f0 e4 00 00 48 63 ff 48 8b 14 fd 50 b5 ad 81 c6 44 10 10 01 c9 c3 55 48 89 e5 66 66 66 66 90 <48> c7 c0 30 e6 00 00 48 
63 f
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109dac6>]  [<ffffffff8109dac6>] __do_softirq+0xae/0x23c
RSP: 0018:ffff88022bd03ef8  EFLAGS: 00000296
RAX: 0000000000000000 RBX: 0000000000000246 RCX: ffff88022bd0e506
RDX: ffff880222218010 RSI: ffff88022bd0e4f0 RDI: 0000000000000002
RBP: ffff88022bd03f78 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e68
R13: ffffffff815d15dd R14: ffff88022bd03f78 R15: ffff880222218000
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000020
 ffff880200000006 000000111671cb12 000000111671cb12 ffff88022bd0db00
Call Trace:
 <IRQ>
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 65 c7 04 25 40 13 01 00 00 00 00 00 fb 66 66 90 66 66 90 48 c7 c3 80 50 a0 81 48 c7 45 b8 00 00 00 00 65 4c 8b 2c 25 48 c6 00 00 <41> f6 c7 01 0f 84 ba 00 
00 0
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8110a66f>]  [<ffffffff8110a66f>] rcu_bh_qs+0x9/0x22
RSP: 0018:ffff88022bd03ee8  EFLAGS: 00000246
RAX: 0000000000000040 RBX: 0000000000000246 RCX: ffff88022bd0e506
RDX: 0000000000000000 RSI: ffff88022bd0e4f0 RDI: 0000000000000002
RBP: ffff88022bd03ee8 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e58
R13: ffffffff815d15dd R14: ffff88022bd03ee8 R15: ffff880222218000
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffff88022bd03f78 ffffffff8109db8a ffff88022bd03f08 ffff880222218010
 ffff880222219fd8 04208040810b79ef 00000000fffc8ad1 000000022bd0e150
 ffff880222218000 0000000000000030 ffff880200000006 000000111671cb12
Call Trace:
 <IRQ>
 [<ffffffff8109db8a>] __do_softirq+0x172/0x23c
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 55 48 89 e5 66 66 66 66 90 48 c7 c0 f0 e4 00 00 48 63 ff 48 8b 14 fd 50 b5 ad 81 c6 44 10 10 01 c9 c3 55 48 89 e5 66 66 66 66 90 <48> c7 c0 30 e6 00 00 48 
63 f
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109d6ae>]  [<ffffffff8109d6ae>] tasklet_action+0x58/0xcc
RSP: 0018:ffff88022bd03ed8  EFLAGS: 00000282
RAX: ffff88022bd0e080 RBX: ffff880222185dc0 RCX: ffff88022bd0e506
RDX: ffff8802186f2fa8 RSI: ffff88022bd0e4f0 RDI: ffffffff81a050b0
RBP: ffff88022bd03ee8 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e48
R13: ffffffff815d15dd R14: ffff88022bd03ee8 R15: ffff8802186f2fb0
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffffffff81a050b0 ffff880222218000 ffff88022bd03f78 ffffffff8109db1f
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000030
Call Trace:
 <IRQ>
 [<ffffffff8109db1f>] __do_softirq+0x107/0x23c
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 00 00 00 65 48 03 04 25 d8 da 00 00 65 48 89 04 25 88 e0 00 00 fb 66 66 90 66 66 90 eb 77 48 8b 1a 4c 8d 62 08 f0 0f ba 6a 08 01 <19> c0 85 c0 75 2d 8b 42 
10 8
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109daab>]  [<ffffffff8109daab>] __do_softirq+0x93/0x23c
RSP: 0018:ffff88022bd03ef8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: 0000000000000246 RCX: ffff88022bd0e506
RDX: ffff880222218010 RSI: ffff88022bd0e4f0 RDI: 0000000000000002
RBP: ffff88022bd03f78 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e68
R13: ffffffff815d15dd R14: ffff88022bd03f78 R15: ffff880222218000
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000038
 ffff880200000006 000000111671cb12 000000111671cb12 ffff88022bd0db00
Call Trace:
 <IRQ>
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 89 45 90 49 8d 44 24 10 4c 89 65 b0 65 8b 14 25 20 b0 00 00 48 89 45 88 89 55 ac 65 c7 04 25 40 13 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 50 
a0 8
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109d6e1>]  [<ffffffff8109d6e1>] tasklet_action+0x8b/0xcc
RSP: 0018:ffff88022bd03ed8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff880222185dc0 RCX: ffff88022bd0e506
RDX: ffff8802186f2fa8 RSI: ffff88022bd0e4f0 RDI: ffffffff81a050b0
RBP: ffff88022bd03ee8 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e48
R13: ffffffff815d15dd R14: ffff88022bd03ee8 R15: ffff8802186f2fb0
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffffffff81a050b0 ffff880222218000 ffff88022bd03f78 ffffffff8109db1f
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000030
Call Trace:
 <IRQ>
 [<ffffffff8109db1f>] __do_softirq+0x107/0x23c
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 10 85 c0 75 20 f0 41 0f ba 34 24 00 19 c0 85 c0 75 04 0f 0b eb fe 48 8b 7a 20 ff 52 18 f0 41 80 24 24 fd eb 3a f0 41 80 24 24 fd <fa> 66 66 90 66 66 90 bf 
06 0
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109d712>]  [<ffffffff8109d712>] tasklet_action+0xbc/0xcc
RSP: 0018:ffff88022bd03ed8  EFLAGS: 00000202
RAX: 0000000000000040 RBX: ffff880222185dc0 RCX: ffff88022bd0e506
RDX: ffff8802186f2fa8 RSI: ffff88022bd0e4f0 RDI: 0000000000000006
RBP: ffff88022bd03ee8 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e48
R13: ffffffff815d15dd R14: ffff88022bd03ee8 R15: ffff8802186f2fb0
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffffffff81a050b0 ffff880222218000 ffff88022bd03f78 ffffffff8109db1f
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000030
Call Trace:
 <IRQ>
 [<ffffffff8109db1f>] __do_softirq+0x107/0x23c
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 90 bf 06 00 00 00 48 c7 02 00 00 00 00 65 48 8b 04 25 88 e0 00 00 48 89 10 65 48 89 14 25 88 e0 00 00 e8 1c fe ff ff fb 66 66 90 <66> 66 90 48 89 da 48 85 
d2 7
BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:23]
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
CPU 2
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109daab>]  [<ffffffff8109daab>] __do_softirq+0x93/0x23c
RSP: 0018:ffff88022bd03ef8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: 0000000000000246 RCX: ffff88022bd0e506
RDX: ffff880222218010 RSI: ffff88022bd0e4f0 RDI: 0000000000000002
RBP: ffff88022bd03f78 R08: ffff88022bd0e770 R09: 0000000000000001
R10: ffff88022bd13de0 R11: 0000000000000002 R12: ffff88022bd03e68
R13: ffffffff815d15dd R14: ffff88022bd03f78 R15: ffff880222218000
FS:  0000000000000000(0000) GS:ffff88022bd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f22cc3e7000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 23, threadinfo ffff880222218000, task ffff880222210000)
Stack:
 ffff88022bd03f08 ffff880222218010 ffff880222219fd8 04208040810b79ef
 00000000fffc8ad1 000000022bd0e150 ffff880222218000 0000000000000038
 ffff880200000006 000000111671cb12 000000111671cb12 ffff88022bd0db00
Call Trace:
 <IRQ>
 [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>
 [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
Code: 89 45 90 49 8d 44 24 10 4c 89 65 b0 65 8b 14 25 20 b0 00 00 48 89 45 88 89 55 ac 65 c7 04 25 40 13 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 48 c7 c3 80 50 
a0 8
------------[ cut here ]------------
WARNING: at /home/greearb/git/linux-3.9.dev.y/kernel/watchdog.c:245 watchdog_overflow_callback+0x9b/0xa6()
Hardware name: To be filled by O.E.M.
Watchdog detected hard LOCKUP on cpu 2
Modules linked in: nf_nat_ipv4 nf_nat veth 8021q garp stp mrp llc wanlink(O) pktgen lockd sunrpc coretemp hwmon mperf intel_powerclamp snd_hda_codec_realtek kvm 
gpio]
Pid: 23, comm: migration/2 Tainted: G        WC O 3.9.4+ #70
Call Trace:
 <NMI>  [<ffffffff810963e1>] warn_slowpath_common+0x85/0x9f
 [<ffffffff8109649e>] warn_slowpath_fmt+0x46/0x48
 [<ffffffff810c4dfb>] ? sched_clock_cpu+0x44/0xce
 [<ffffffff81103cec>] watchdog_overflow_callback+0x9b/0xa6
 [<ffffffff811336cb>] __perf_event_overflow+0x137/0x1cb
 [<ffffffff8101db3f>] ? x86_perf_event_set_period+0x107/0x113
 [<ffffffff81133c1a>] perf_event_overflow+0x14/0x16
 [<ffffffff810230dc>] intel_pmu_handle_irq+0x2b0/0x32d
 [<ffffffff815cbdd1>] perf_event_nmi_handler+0x19/0x1b
 [<ffffffff815cb64a>] nmi_handle+0x55/0x7e
 [<ffffffff815cb71b>] do_nmi+0xa8/0x2db
 [<ffffffff815cadb1>] end_repeat_nmi+0x1e/0x2e
 [<ffffffff8109dad4>] ? __do_softirq+0xbc/0x23c
 [<ffffffff8109dad4>] ? __do_softirq+0xbc/0x23c
 [<ffffffff8109dad4>] ? __do_softirq+0xbc/0x23c
 <<EOE>>  <IRQ>  [<ffffffff8109dce6>] irq_exit+0x4b/0xa8
 [<ffffffff815d26ff>] smp_apic_timer_interrupt+0x8b/0x99
 [<ffffffff815d15dd>] apic_timer_interrupt+0x6d/0x80
 <EOI>  [<ffffffff810f99a4>] ? stop_machine_cpu_stop+0xe7/0x267
 [<ffffffff810f9afd>] ? stop_machine_cpu_stop+0x240/0x267
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810e817b>] ? copy_module_from_fd+0xe7/0xe7
 [<ffffffff810f98bd>] ? stop_one_cpu_nowait+0x30/0x30
 [<ffffffff810f9624>] cpu_stopper_thread+0xbd/0x176
 [<ffffffff815c91e5>] ? __schedule+0x59f/0x5e7
 [<ffffffff810bb434>] smpboot_thread_fn+0x217/0x21f
 [<ffffffff810bb21d>] ? test_ti_thread_flag.clone.0+0x11/0x11
 [<ffffffff810b4a09>] kthread+0xb5/0xbd
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
 [<ffffffff815d092c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810b4954>] ? kthread_freezable_should_stop+0x60/0x60
---[ end trace 9767454fd3f66a82 ]---

 CTRL-A Z for help |115200 8N1 | NOR | Minicom 2.5    | VT102 | Online 15:23 




> Thanks.


--
Ben Greear <greearb@xxxxxxxxxxxxxxx>
Candela Technologies Inc  http://www.candelatech.com

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/