Re: [PATCH 3.10] cgroup: break infinite loop in __css_tryget()

From: Roman Gushchin
Date: Wed Dec 03 2014 - 07:27:55 EST


At Tue, 2 Dec 2014 14:35:38 -0500,
Tejun Heo wrote:
>
> On Tue, Dec 02, 2014 at 09:40:45PM +0300, Roman Gushchin wrote:
> > Hi, Tejun!
> >
> > 02.12.2014, 19:56, "Tejun Heo" <tj@xxxxxxxxxx>:
> > > Hello, Roman.
> > >
> > > On Fri, Nov 28, 2014 at 07:47:54PM +0300, Roman Gushchin wrote:
> > >>  If cgroup_destroy_locked() sets the css refcount to a negative value,
> > >>  we get an infinite loop in __css_tryget().
> > >>
> > >>  In this case css_refcnt() returns modified (non-negative value), so
> > >>  both (t == v) and (t < 0) conditions are always false.
> > >
> > > I don't follow.  The count is biased and modified by unbiasing iff the
> > > value is negative.  Here, @v is the unbiased value and @t is the
> > > verbatim value.  If @v is different from @t due to unbiasing, @t must
> > > be negative satisfying the second condition and returning NULL, no?
> >
> > Yep. I missed that we compare t with 0 (not v).
>
> Hmm... what's up with the rcu stall message then?

It should be something else)
Actually, I have a bunch of stacktraces like the one below.
I have them at least on 3.10.53 and 3.10.58.
It seems like it's a rare race: <0.1% machines are stalling per day.

--
[1190308.537478] INFO: rcu_sched self-detected stall on CPU { 10} (t=15001 jiffies g=30275281 c=30275280 q=133479)
[1190308.538833] sending NMI to all CPUs:
[1190308.538839] NMI backtrace for cpu 0
[1190308.538844] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.58-18 #14
[1190308.538846] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012
[1190308.538848] task: ffffffff81a13440 ti: ffffffff81a00000 task.ti: ffffffff81a00000
[1190308.538861] RIP: 0010:[<ffffffff8135e53c>] [<ffffffff8135e53c>] intel_idle+0xcc/0x140
[1190308.538868] RSP: 0018:ffffffff81a01d98 EFLAGS: 00000046
[1190308.538870] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[1190308.538872] RDX: 0000000000000000 RSI: ffffffff81a01fd8 RDI: 0000000000000000
[1190308.538873] RBP: ffffffff81a01dc8 R08: 00000000000000b1 R09: 0000000000002be4
[1190308.538875] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004
[1190308.538876] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004
[1190308.538879] FS: 0000000000000000(0000) GS:ffff88085fc00000(0000) knlGS:0000000000000000
[1190308.538881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1190308.538883] CR2: 00007f1d4d27aae0 CR3: 0000000001a0c000 CR4: 00000000000407f0
[1190308.538885] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1190308.538887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1190308.538888] Stack:
[1190308.538889] ffffffff81a01dc8 000000008109b864 ffffe8f7dfc00400 00043b9213aa19b8
[1190308.538893] ffffffff81a82318 ffffffff81a821a0 ffffffff81a01e28 ffffffff814ebf0f
[1190308.538896] 0000000000000000 00000000003cb8b9 0000000000000000 00000000003cb8b9
[1190308.538899] Call Trace:
[1190308.538906] [<ffffffff814ebf0f>] cpuidle_enter_state+0x4f/0xe0
[1190308.538909] [<ffffffff814ec060>] cpuidle_idle_call+0xc0/0x220
[1190308.538914] [<ffffffff8101c25e>] arch_cpu_idle+0xe/0x30
[1190308.538919] [<ffffffff8109ab78>] cpu_startup_entry+0xe8/0x2b0
[1190308.538923] [<ffffffff81607137>] rest_init+0x77/0x80
[1190308.538928] [<ffffffff81af0efb>] start_kernel+0x402/0x40f
[1190308.538931] [<ffffffff81af08f9>] ? repair_env_string+0x5a/0x5a
[1190308.538934] [<ffffffff81af0120>] ? early_idt_handlers+0x120/0x120
[1190308.538938] [<ffffffff81af05f8>] x86_64_start_reservations+0x2a/0x2c
[1190308.538941] [<ffffffff81af06ed>] x86_64_start_kernel+0xf3/0x102
[1190308.538942] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4
[1190308.538976] NMI backtrace for cpu 16
[1190308.538980] CPU: 16 PID: 0 Comm: swapper/16 Not tainted 3.10.58-18 #14
[1190308.538981] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012
[1190308.538983] task: ffff88085462ba00 ti: ffff88085463c000 task.ti: ffff88085463c000
[1190308.538984] RIP: 0010:[<ffffffff8135e53c>] [<ffffffff8135e53c>] intel_idle+0xcc/0x140
[1190308.538987] RSP: 0018:ffff88085463dda8 EFLAGS: 00000046
[1190308.538988] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[1190308.538989] RDX: 0000000000000000 RSI: ffff88085463dfd8 RDI: 0000000000000010
[1190308.538990] RBP: ffff88085463ddd8 R08: 0000000000000005 R09: 0000000000000009
[1190308.538992] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004
[1190308.538993] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004
[1190308.538994] FS: 0000000000000000(0000) GS:ffff88085fd00000(0000) knlGS:0000000000000000
[1190308.538996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1190308.538997] CR2: 00007f5883684a70 CR3: 0000000001a0c000 CR4: 00000000000407e0
[1190308.538998] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1190308.538999] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1190308.539000] Stack:
[1190308.539001] ffff88085463ddd8 000000108109b864 ffffe8f7dfd00400 00043b9213a9dc35
[1190308.539003] ffffffff81a82318 ffffffff81a821a0 ffff88085463de38 ffffffff814ebf0f
[1190308.539005] 0000000000000000 00000000003cf622 0000000000000000 00000000003cf622
[1190308.539007] Call Trace:
[1190308.539010] [<ffffffff814ebf0f>] cpuidle_enter_state+0x4f/0xe0
[1190308.539013] [<ffffffff814ec060>] cpuidle_idle_call+0xc0/0x220
[1190308.539015] [<ffffffff8101c25e>] arch_cpu_idle+0xe/0x30
[1190308.539017] [<ffffffff8109ab78>] cpu_startup_entry+0xe8/0x2b0
[1190308.539021] [<ffffffff81612bef>] start_secondary+0x204/0x20b
[1190308.539023] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4
[1190308.539045] NMI backtrace for cpu 21
[1190308.539050] CPU: 21 PID: 9465 Comm: vmtouch Not tainted 3.10.58-18 #14
[1190308.539052] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012
[1190308.539054] task: ffff880695666cc0 ti: ffff8808525e2000 task.ti: ffff8808525e2000
[1190308.539056] RIP: 0010:[<ffffffff810b6014>] [<ffffffff810b6014>] __css_tryget+0x24/0x50
[1190308.539061] RSP: 0000:ffff8808525e37e8 EFLAGS: 00000a87
[1190308.539063] RAX: 0000000080000000 RBX: ffff8810531f6800 RCX: ffff8810531f6808
[1190308.539065] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8810531f6800
[1190308.539067] RBP: ffff8808525e37e8 R08: 0000000000000001 R09: 00000000000001ff
[1190308.539069] R10: ffff8806946f6d80 R11: 0000000000000400 R12: 0000000000000001
[1190308.539070] R13: 0000000000000020 R14: 0000000000000001 R15: 00000000000000d0
[1190308.539073] FS: 00007fc619bea700(0000) GS:ffff88085fda0000(0000) knlGS:0000000000000000
[1190308.539075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1190308.539077] CR2: 00007fc60e5c8000 CR3: 000000085365b000 CR4: 00000000000407e0
[1190308.539079] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1190308.539081] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1190308.539082] Stack:
[1190308.539083] ffff8808525e3908 ffffffff811705e8 ffff8808525e3858 ffff880600011280
[1190308.539087] ffff8808525e3818 ffffffff81115385 ffff8808525e3898 ffffffff811156be
[1190308.539090] ffff8808525e2000 ffff8808525e3898 000000d000000002 00000000000000d0
[1190308.539093] Call Trace:
[1190308.539099] [<ffffffff811705e8>] __mem_cgroup_try_charge+0x2c8/0xb20
[1190308.539103] [<ffffffff81115385>] ? mempool_alloc_slab+0x15/0x20
[1190308.539106] [<ffffffff811156be>] ? mempool_alloc+0x5e/0x160
[1190308.539111] [<ffffffff811ae531>] ? __bio_add_page.part.15+0x101/0x240
[1190308.539114] [<ffffffff81171417>] mem_cgroup_charge_common+0x57/0xa0
[1190308.539118] [<ffffffff81171f70>] mem_cgroup_cache_charge+0xb0/0xf0
[1190308.539121] [<ffffffff811130aa>] add_to_page_cache_locked+0x4a/0x190
[1190308.539126] [<ffffffff8112feff>] ? __inc_zone_state+0x5f/0x70
[1190308.539129] [<ffffffff81113211>] add_to_page_cache_lru+0x21/0x50
[1190308.539132] [<ffffffff811b5b9a>] mpage_readpages+0xba/0x130
[1190308.539136] [<ffffffff81215d80>] ? ext4_get_block_write+0x20/0x20
[1190308.539138] [<ffffffff81215d80>] ? ext4_get_block_write+0x20/0x20
[1190308.539142] [<ffffffff81156e9a>] ? alloc_pages_current+0xba/0x170
[1190308.539145] [<ffffffff812121b5>] ext4_readpages+0x45/0x60
[1190308.539149] [<ffffffff8111f0df>] __do_page_cache_readahead+0x1af/0x260
[1190308.539153] [<ffffffff8111f4f1>] ra_submit+0x21/0x30
[1190308.539155] [<ffffffff81114f47>] filemap_fault+0x297/0x500
[1190308.539159] [<ffffffff81137fb2>] __do_fault+0x72/0x520
[1190308.539162] [<ffffffff8113b1a6>] handle_pte_fault+0xf6/0xa60
[1190308.539165] [<ffffffff81156e9a>] ? alloc_pages_current+0xba/0x170
[1190308.539169] [<ffffffff8104b026>] ? pte_alloc_one+0x36/0x40
[1190308.539172] [<ffffffff8162786e>] ? _raw_spin_lock+0xe/0x20
[1190308.539175] [<ffffffff81138e39>] ? __pte_alloc+0xa9/0x160
[1190308.539179] [<ffffffff8113ce21>] handle_mm_fault+0x271/0x390
[1190308.539181] [<ffffffff81112f87>] ? find_get_page+0x37/0x90
[1190308.539186] [<ffffffff8162ba2f>] __do_page_fault+0x19f/0x540
[1190308.539189] [<ffffffff8111c3e7>] ? __free_pages+0x47/0x70
[1190308.539192] [<ffffffff8111c5e5>] ? free_pages+0x45/0x50
[1190308.539195] [<ffffffff8113ec18>] ? SyS_mincore+0x6b8/0x740
[1190308.539198] [<ffffffff8162bdde>] do_page_fault+0xe/0x10
[1190308.539201] [<ffffffff81628218>] page_fault+0x28/0x30
[1190308.539203] Code: 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8d 4f 08 48 89 e5 8b 57 08 8d 82 00 00 00 80 85 d2 0f 48 d0 8d 72 01 89 d0 f0 0f b1 31 <39> c2 75 07 b8 01 00 00 00 5d c3 85 c0 78 04 f3 90 eb d6 31 c0
[1190308.539237] NMI backtrace for cpu 5
[1190308.539240] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 3.10.58-18 #14
[1190308.539242] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012
[1190308.539244] task: ffff88085449e580 ti: ffff880854616000 task.ti: ffff880854616000
[1190308.539246] RIP: 0010:[<ffffffff8135e53c>] [<ffffffff8135e53c>] intel_idle+0xcc/0x140
[1190308.539251] RSP: 0018:ffff880854617da8 EFLAGS: 00000046
[1190308.539253] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[1190308.539255] RDX: 0000000000000000 RSI: ffff880854617fd8 RDI: 0000000000000005
[1190308.539257] RBP: ffff880854617dd8 R08: 000000000000098d R09: 000000000009d6c9
[1190308.539258] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004
[1190308.539260] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004
[1190308.539263] FS: 0000000000000000(0000) GS:ffff88085fca0000(0000) knlGS:0000000000000000
[1190308.539265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1190308.539267] CR2: 00007fb8fc195000 CR3: 0000000001a0c000 CR4: 00000000000407e0
[1190308.539269] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1190308.539271] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1190308.539272] Stack:
[1190308.539274] ffff880854617dd8 000000058109b864 ffffe8f7dfca0400 00043b9213a9eb82
[1190308.539277] ffffffff81a82318 ffffffff81a821a0 ffff880854617e38 ffffffff814ebf0f
[1190308.539280] 0000000000000000 00000000003cea7e 0000000000000000 00000000003cea7e
[1190308.539284] Call Trace:
[1190308.539288] [<ffffffff814ebf0f>] cpuidle_enter_state+0x4f/0xe0
[1190308.539292] [<ffffffff814ec060>] cpuidle_idle_call+0xc0/0x220
[1190308.539296] [<ffffffff8101c25e>] arch_cpu_idle+0xe/0x30
[1190308.539300] [<ffffffff8109ab78>] cpu_startup_entry+0xe8/0x2b0
[1190308.539305] [<ffffffff81612bef>] start_secondary+0x204/0x20b
[1190308.539306] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4
[1190308.539343] NMI backtrace for cpu 27
[1190308.539349] CPU: 27 PID: 0 Comm: swapper/27 Not tainted 3.10.58-18 #14
[1190308.539352] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012
[1190308.539354] task: ffff880854650e80 ti: ffff88085465a000 task.ti: ffff88085465a000
[1190308.539356] RIP: 0010:[<ffffffff8135e53c>] [<ffffffff8135e53c>] intel_idle+0xcc/0x140
[1190308.539363] RSP: 0018:ffff88085465bda8 EFLAGS: 00000046
[1190308.539365] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[1190308.539367] RDX: 0000000000000000 RSI: ffff88085465bfd8 RDI: 000000000000001b
[1190308.539369] RBP: ffff88085465bdd8 R08: 0000000000ad01f5 R09: 000004f3ad35ef89
[1190308.539371] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004
[1190308.539373] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004
[1190308.539375] FS: 0000000000000000(0000) GS:ffff88107fd60000(0000) knlGS:0000000000000000
[1190308.539377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1190308.539379] CR2: 00007fbfc2926420 CR3: 0000000001a0c000 CR4: 00000000000407e0
[1190308.539381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1190308.539383] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1190308.539384] Stack:
[1190308.539386] ffff88085465bdd8 0000001b8109b864 ffffe8ffffd60400 00043b90957bc0f4
[1190308.539389] ffffffff81a82318 ffffffff81a821a0 ffff88085465be38 ffffffff814ebf0f
[1190308.539393] 00000000000000d0 00000000188d53e3 00000000000000d0 00000000188d53e3
[1190308.539396] Call Trace:
[1190308.539402] [<ffffffff814ebf0f>] cpuidle_enter_state+0x4f/0xe0
[1190308.539406] [<ffffffff814ec060>] cpuidle_idle_call+0xc0/0x220
[1190308.539411] [<ffffffff8101c25e>] arch_cpu_idle+0xe/0x30
[1190308.539416] [<ffffffff8109ab78>] cpu_startup_entry+0xe8/0x2b0
[1190308.539422] [<ffffffff81612bef>] start_secondary+0x204/0x20b
[1190308.539423] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4
[1190308.539457] NMI backtrace for cpu 11
[1190308.539460] CPU: 11 PID: 9466 Comm: vmtouch Not tainted 3.10.58-18 #14
[1190308.539462] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012
[1190308.539464] task: ffff880031e42b80 ti: ffff880853768000 task.ti: ffff880853768000
[1190308.539466] RIP: 0010:[<ffffffff810b6027>] [<ffffffff810b6027>] __css_tryget+0x37/0x50
[1190308.539471] RSP: 0000:ffff8808537697e8 EFLAGS: 00000286
[1190308.539475] RAX: 0000000080000000 RBX: ffff8810531f6800 RCX: ffff8810531f6808
[1190308.539479] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8810531f6800
[1190308.539483] RBP: ffff8808537697e8 R08: 0000000000000001 R09: 00000000000001ff
[1190308.539487] R10: ffff8810519b4600 R11: 0000000000000400 R12: 0000000000000001
[1190308.539491] R13: 0000000000000020 R14: 0000000000000001 R15: 00000000000000d0
[1190308.539497] FS: 00007f20e541f700(0000) GS:ffff88107fc60000(0000) knlGS:0000000000000000
[1190308.539502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1190308.539507] CR2: 00007f20e4cb7000 CR3: 0000000850f21000 CR4: 00000000000407e0
[1190308.539513] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1190308.539518] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1190308.539522] Stack:
[1190308.539526] ffff880853769908 ffffffff811705e8 ffff880853769858 ffff880031e42b80
[1190308.539541] ffff880853769818 ffffffff81115385 ffff880853769898 ffffffff811ae7d8
[1190308.539545] ffff880853768000 ffff880853769898 000000d0519b49c0 00000000000000d0
[1190308.539548] Call Trace:
[1190308.539553] [<ffffffff811705e8>] __mem_cgroup_try_charge+0x2c8/0xb20
[1190308.539557] [<ffffffff81115385>] ? mempool_alloc_slab+0x15/0x20
[1190308.539562] [<ffffffff811ae7d8>] ? bvec_alloc+0x68/0x100
[1190308.539566] [<ffffffff811ae531>] ? __bio_add_page.part.15+0x101/0x240
[1190308.539569] [<ffffffff81171417>] mem_cgroup_charge_common+0x57/0xa0
[1190308.539572] [<ffffffff81171f70>] mem_cgroup_cache_charge+0xb0/0xf0
[1190308.539575] [<ffffffff811130aa>] add_to_page_cache_locked+0x4a/0x190
[1190308.539578] [<ffffffff81113211>] add_to_page_cache_lru+0x21/0x50
[1190308.539582] [<ffffffff811b5b9a>] mpage_readpages+0xba/0x130
[1190308.539586] [<ffffffff81215d80>] ? ext4_get_block_write+0x20/0x20
[1190308.539589] [<ffffffff81215d80>] ? ext4_get_block_write+0x20/0x20
[1190308.539594] [<ffffffff81156e9a>] ? alloc_pages_current+0xba/0x170
[1190308.539597] [<ffffffff812121b5>] ext4_readpages+0x45/0x60
[1190308.539602] [<ffffffff8111f0df>] __do_page_cache_readahead+0x1af/0x260
[1190308.539606] [<ffffffff8111f4f1>] ra_submit+0x21/0x30
[1190308.539608] [<ffffffff81114f47>] filemap_fault+0x297/0x500
[1190308.539612] [<ffffffff81137fb2>] __do_fault+0x72/0x520
[1190308.539615] [<ffffffff8113b1a6>] handle_pte_fault+0xf6/0xa60
[1190308.539618] [<ffffffff81156e9a>] ? alloc_pages_current+0xba/0x170
[1190308.539623] [<ffffffff8112fb09>] ? inc_zone_page_state+0x59/0xa0
[1190308.539627] [<ffffffff8104b026>] ? pte_alloc_one+0x36/0x40
[1190308.539631] [<ffffffff8162786e>] ? _raw_spin_lock+0xe/0x20
[1190308.539635] [<ffffffff81138e39>] ? __pte_alloc+0xa9/0x160
[1190308.539639] [<ffffffff8116d1f4>] ? __mem_cgroup_count_vm_event+0x44/0x60
[1190308.539642] [<ffffffff8113ce21>] handle_mm_fault+0x271/0x390
[1190308.539647] [<ffffffff8162ba2f>] __do_page_fault+0x19f/0x540
[1190308.539650] [<ffffffff8108d5fd>] ? set_next_entity+0x3d/0x80
[1190308.539654] [<ffffffff810125bd>] ? __switch_to+0x16d/0x4a0
[1190308.539659] [<ffffffff8108657a>] ? finish_task_switch+0x4a/0xf0
[1190308.539665] [<ffffffff81625cbd>] ? __schedule+0x2cd/0x770
[1190308.539669] [<ffffffff8162bdde>] do_page_fault+0xe/0x10
[1190308.539672] [<ffffffff81628218>] page_fault+0x28/0x30
[1190308.539674] Code: e5 8b 57 08 8d 82 00 00 00 80 85 d2 0f 48 d0 8d 72 01 89 d0 f0 0f b1 31 39 c2 75 07 b8 01 00 00 00 5d c3 85 c0 78 04 f3 90 eb d6 <31> c0 5d 66 0f 1f 44 00 00 c3 66 66 66 66 66 66 2e 0f 1f 84 00
[1190308.539707] NMI backtrace for cpu 10
[1190308.539710] CPU: 10 PID: 9464 Comm: vmtouch Not tainted 3.10.58-18 #14
[1190308.539712] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012
[1190308.539715] task: ffff8800641cde40 ti: ffff880852ec8000 task.ti: ffff880852ec8000
[1190308.539716] RIP: 0010:[<ffffffff8131591d>] [<ffffffff8131591d>] __bitmap_empty+0x6d/0x80
[1190308.539723] RSP: 0000:ffff88107fc43db8 EFLAGS: 00000006
[1190308.539725] RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000002
[1190308.539727] RDX: 0000000000000009 RSI: 0000000000000080 RDI: ffffffff81ad06e0
[1190308.539729] RBP: ffff88107fc43db8 R08: ffff88107fc4dc50 R09: 0000000000000080
[1190308.539731] R10: 000000000000b788 R11: 000000000000b787 R12: ffffffff81a4bfc0
[1190308.539732] R13: ffffffff81a4c080 R14: 000000000000000a R15: ffffffff81ad0c00
[1190308.539735] FS: 00007fe5c385b700(0000) GS:ffff88107fc40000(0000) knlGS:0000000000000000
[1190308.539737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1190308.539739] CR2: 00007fe5a8fcc000 CR3: 0000001051cc8000 CR4: 00000000000407e0
[1190308.539741] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1190308.539743] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1190308.539744] Stack:
[1190308.539745] ffff88107fc43dd8 ffffffff810392e0 0000000000020967 ffff88107fc4e760
[1190308.539749] ffff88107fc43e48 ffffffff810d6e92 000000000000000a ffff88107fc53d80
[1190308.539752] 0000000000000096 0000000000000000 000000000000000a ffff880852ec8000
[1190308.539755] Call Trace:
[1190308.539757] <IRQ>
[1190308.539758] [<ffffffff810392e0>] arch_trigger_all_cpu_backtrace+0x90/0xb0
[1190308.539774] [<ffffffff810d6e92>] rcu_check_callbacks+0x212/0x680
[1190308.539785] [<ffffffff810667e8>] update_process_times+0x48/0x80
[1190308.539797] [<ffffffff810a3d2e>] tick_sched_timer+0x6e/0xa0
[1190308.539808] [<ffffffff8107e9d8>] __run_hrtimer+0x68/0x1c0
[1190308.539819] [<ffffffff810a3cc0>] ? tick_nohz_handler+0xd0/0xd0
[1190308.539827] [<ffffffff8105def9>] ? __do_softirq+0x149/0x270
[1190308.539836] [<ffffffff8107f247>] hrtimer_interrupt+0x107/0x270
[1190308.539846] [<ffffffff8163199c>] ? call_softirq+0x1c/0x30
[1190308.539850] [<ffffffff81632309>] smp_apic_timer_interrupt+0x69/0x99
[1190308.539853] [<ffffffff816311dd>] apic_timer_interrupt+0x6d/0x80
[1190308.539854] <EOI>
[1190308.539855] [<ffffffff810b6014>] ? __css_tryget+0x24/0x50
[1190308.539861] [<ffffffff811705e8>] __mem_cgroup_try_charge+0x2c8/0xb20
[1190308.539864] [<ffffffff81115385>] ? mempool_alloc_slab+0x15/0x20
[1190308.539867] [<ffffffff811ae7d8>] ? bvec_alloc+0x68/0x100
[1190308.539871] [<ffffffff811ae531>] ? __bio_add_page.part.15+0x101/0x240
[1190308.539874] [<ffffffff81171417>] mem_cgroup_charge_common+0x57/0xa0
[1190308.539878] [<ffffffff81171f70>] mem_cgroup_cache_charge+0xb0/0xf0
[1190308.539881] [<ffffffff811130aa>] add_to_page_cache_locked+0x4a/0x190
[1190308.539884] [<ffffffff81113211>] add_to_page_cache_lru+0x21/0x50
[1190308.539887] [<ffffffff811b5b9a>] mpage_readpages+0xba/0x130
[1190308.539890] [<ffffffff81215d80>] ? ext4_get_block_write+0x20/0x20
[1190308.539893] [<ffffffff81215d80>] ? ext4_get_block_write+0x20/0x20
[1190308.539896] [<ffffffff81156e9a>] ? alloc_pages_current+0xba/0x170
[1190308.539899] [<ffffffff812121b5>] ext4_readpages+0x45/0x60
[1190308.539902] [<ffffffff8111f0df>] __do_page_cache_readahead+0x1af/0x260
[1190308.539906] [<ffffffff8111f4f1>] ra_submit+0x21/0x30
[1190308.539908] [<ffffffff81114f47>] filemap_fault+0x297/0x500
[1190308.539911] [<ffffffff81137fb2>] __do_fault+0x72/0x520
[1190308.539915] [<ffffffff8113b1a6>] handle_pte_fault+0xf6/0xa60
[1190308.539917] [<ffffffff81156e9a>] ? alloc_pages_current+0xba/0x170
[1190308.539921] [<ffffffff8104b026>] ? pte_alloc_one+0x36/0x40
[1190308.539923] [<ffffffff8162786e>] ? _raw_spin_lock+0xe/0x20
[1190308.539926] [<ffffffff81138e39>] ? __pte_alloc+0xa9/0x160
[1190308.539929] [<ffffffff8113ce21>] handle_mm_fault+0x271/0x390
[1190308.539933] [<ffffffff8162ba2f>] __do_page_fault+0x19f/0x540
[1190308.539936] [<ffffffff8111c3e7>] ? __free_pages+0x47/0x70
[1190308.539939] [<ffffffff8111c5e5>] ? free_pages+0x45/0x50
[1190308.539942] [<ffffffff8113ec18>] ? SyS_mincore+0x6b8/0x740
[1190308.539946] [<ffffffff8162bdde>] do_page_fault+0xe/0x10
[1190308.539948] [<ffffffff81628218>] page_fault+0x28/0x30
[1190308.539950] Code: 74 29 89 f0 48 63 d2 c1 f8 1f c1 e8 1a 8d 0c 06 83 e1 3f 29 c1 b8 01 00 00 00 48 d3 e0 48 83 e8 01 48 85 04 d7 0f 94 c0 0f b6 c0 <5d> c3 90 31 c0 5d c3 31 d2 eb c1 0f 1f 84 00 00 00 00 00 8d 4e
[1190308.539984] NMI backtrace for cpu 26
[1190308.539987] CPU: 26 PID: 0 Comm: swapper/26 Not tainted 3.10.58-18 #14
[1190308.539988] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012
[1190308.539991] task: ffff880854650740 ti: ffff880854658000 task.ti: ffff880854658000
[1190308.539992] RIP: 0010:[<ffffffff8135e53c>] [<ffffffff8135e53c>] intel_idle+0xcc/0x140
[1190308.539996] RSP: 0018:ffff880854659da8 EFLAGS: 00000046
[1190308.539998] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[1190308.539999] RDX: 0000000000000000 RSI: ffff880854659fd8 RDI: 000000000000001a
[1190308.540001] RBP: ffff880854659dd8 R08: 0000000000000009 R09: 0000000000000010
[1190308.540003] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004
[1190308.540005] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004
[1190308.540007] FS: 0000000000000000(0000) GS:ffff88107fd40000(0000) knlGS:0000000000000000
[1190308.540009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1190308.540011] CR2: 00007f058131f420 CR3: 0000000001a0c000 CR4: 00000000000407e0
[1190308.540012] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1190308.540014] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1190308.540015] Stack:
[1190308.540017] ffff880854659dd8 0000001a8109b864 ffffe8ffffd40400 00043b9213aa04a4
[1190308.540020] ffffffff81a82318 ffffffff81a821a0 ffff880854659e38 ffffffff814ebf0f
[1190308.540023] 000000000000006a 00000000162c0a55 000000000000006a 00000000162c0a55
[1190308.540026] Call Trace:
[1190308.540036] [<ffffffff814ebf0f>] cpuidle_enter_state+0x4f/0xe0
[1190308.540047] [<ffffffff814ec060>] cpuidle_idle_call+0xc0/0x220
[1190308.540057] [<ffffffff8101c25e>] arch_cpu_idle+0xe/0x30
[1190308.540068] [<ffffffff8109ab78>] cpu_startup_entry+0xe8/0x2b0
[1190308.540079] [<ffffffff81612bef>] start_secondary+0x204/0x20b
[1190308.540085] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4
[1190308.540123] NMI backtrace for cpu 12
[1190308.540128] CPU: 12 PID: 0 Comm: swapper/12 Not tainted 3.10.58-18 #14
[1190308.540130] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012
[1190308.540132] task: ffff880854629d00 ti: ffff880854634000 task.ti: ffff880854634000
[1190308.540134] RIP: 0010:[<ffffffff8135e53c>] [<ffffffff8135e53c>] intel_idle+0xcc/0x140
[1190308.540139] RSP: 0018:ffff880854635da8 EFLAGS: 00000046
[1190308.540141] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
[1190308.540142] RDX: 0000000000000000 RSI: ffff880854635fd8 RDI: 000000000000000c
[1190308.540144] RBP: ffff880854635dd8 R08: 00000000000000ad R09: 0000000000001900
[1190308.540146] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000002
[1190308.540148] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000002
[1190308.540150] FS: 0000000000000000(0000) GS:ffff88107fc80000(0000) knlGS:0000000000000000
[1190308.540152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1190308.540154] CR2: 00007f1e5fe60000 CR3: 0000000001a0c000 CR4: 00000000000407e0
[1190308.540156] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1190308.540158] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1190308.540159] Stack:
[1190308.540161] ffff880854635dd8 0000000c8109b864 ffffe8ffffc80400 00043b9213aa3dbd
[1190308.540164] ffffffff81a82268 ffffffff81a821a0 ffff880854635e38 ffffffff814ebf0f
[1190308.540167] 0000000000000000 00000000003c958a 0000000000000000 00000000003c958a
[1190308.540170] Call Trace:
[1190308.540175] [<ffffffff814ebf0f>] cpuidle_enter_state+0x4f/0xe0
[1190308.540178] [<ffffffff814ec060>] cpuidle_idle_call+0xc0/0x220
[1190308.540182] [<ffffffff8101c25e>] arch_cpu_idle+0xe/0x30
[1190308.540186] [<ffffffff8109ab78>] cpu_startup_entry+0xe8/0x2b0
[1190308.540189] [<ffffffff81612bef>] start_secondary+0x204/0x20b
[1190308.540191] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4
[1190308.540225] NMI backtrace for cpu 28
[1190308.540228] CPU: 28 PID: 0 Comm: swapper/28 Not tainted 3.10.58-18 #14
[1190308.540229] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012
[1190308.540230] task: ffff8808546515c0 ti: ffff88085465c000 task.ti: ffff88085465c000
[1190308.540232] RIP: 0010:[<ffffffff8135e53c>] [<ffffffff8135e53c>] intel_idle+0xcc/0x140
[1190308.540234] RSP: 0018:ffff88085465dda8 EFLAGS: 00000046
[1190308.540236] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[1190308.540237] RDX: 0000000000000000 RSI: ffff88085465dfd8 RDI: 000000000000001c
[1190308.540238] RBP: ffff88085465ddd8 R08: 000000000000000d R09: 0000000000000004
[1190308.540239] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004
[1190308.540240] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004
[1190308.540242] FS: 0000000000000000(0000) GS:ffff88107fd80000(0000) knlGS:0000000000000000
[1190308.540243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1190308.540244] CR2: 00007f28a020ae60 CR3: 0000000001a0c000 CR4: 00000000000407e0
[1190308.540245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1190308.540247] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1190308.540247] Stack:
[1190308.540248] ffff88085465ddd8 0000001c8109b864 ffffe8ffffd80400 00043b9213a9e5d1
[1190308.540250] ffffffff81a82318 ffffffff81a821a0 ffff88085465de38 ffffffff814ebf0f
[1190308.540252] 0000000000000000 00000000003cedb6 0000000000000000 00000000003cedb6
[1190308.540254] Call Trace:
[1190308.540257] [<ffffffff814ebf0f>] cpuidle_enter_state+0x4f/0xe0
[1190308.540260] [<ffffffff814ec060>] cpuidle_idle_call+0xc0/0x220
[1190308.540262] [<ffffffff8101c25e>] arch_cpu_idle+0xe/0x30
[1190308.540264] [<ffffffff8109ab78>] cpu_startup_entry+0xe8/0x2b0
[1190308.540273] [<ffffffff81612bef>] start_secondary+0x204/0x20b
[1190308.540277] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4
[1190308.540341] NMI backtrace for cpu 8
[1190308.540345] CPU: 8 PID: 0 Comm: swapper/8 Not tainted 3.10.58-18 #14
[1190308.540347] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012
[1190308.540349] task: ffff880854628000 ti: ffff88085461c000 task.ti: ffff88085461c000
[1190308.540351] RIP: 0010:[<ffffffff8135e53c>] [<ffffffff8135e53c>] intel_idle+0xcc/0x140
[1190308.540356] RSP: 0018:ffff88085461dda8 EFLAGS: 00000046
[1190308.540357] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[1190308.540359] RDX: 0000000000000000 RSI: ffff88085461dfd8 RDI: 0000000000000008
[1190308.540361] RBP: ffff88085461ddd8 R08: 0000000000000005 R09: 0000000000000000
[1190308.540363] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004
[1190308.540364] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004
[1190308.540367] FS: 0000000000000000(0000) GS:ffff88107fc00000(0000) knlGS:0000000000000000
[1190308.540369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1190308.540371] CR2: 0000000000f4f190 CR3: 0000000001a0c000 CR4: 00000000000407e0
[1190308.540373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1190308.540374] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1190308.540376] Stack:
[1190308.540377] ffff88085461ddd8 000000088109b864 ffffe8ffffc00400 00043b9213aa205f
[1190308.540381] ffffffff81a82318 ffffffff81a821a0 ffff88085461de38 ffffffff814ebf0f
[1190308.540384] 0000000000000000 00000000003cb29e 0000000000000000 00000000003cb29e
[1190308.540387] Call Trace:
[1190308.540391] [<ffffffff814ebf0f>] cpuidle_enter_state+0x4f/0xe0
[1190308.540395] [<ffffffff814ec060>] cpuidle_idle_call+0xc0/0x220
[1190308.540398] [<ffffffff8101c25e>] arch_cpu_idle+0xe/0x30
[1190308.540402] [<ffffffff8109ab78>] cpu_startup_entry+0xe8/0x2b0
[1190308.540406] [<ffffffff81612bef>] start_secondary+0x204/0x20b
[1190308.540407] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4
[1190308.540441] NMI backtrace for cpu 24
[1190308.540444] CPU: 24 PID: 0 Comm: swapper/24 Not tainted 3.10.58-18 #14
[1190308.540446] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012
[1190308.540448] task: ffff88085462f400 ti: ffff88085464c000 task.ti: ffff88085464c000
[1190308.540449] RIP: 0010:[<ffffffff8135e53c>] [<ffffffff8135e53c>] intel_idle+0xcc/0x140
[1190308.540451] RSP: 0018:ffff88085464dda8 EFLAGS: 00000046
[1190308.540453] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[1190308.540454] RDX: 0000000000000000 RSI: ffff88085464dfd8 RDI: 0000000000000018
[1190308.540455] RBP: ffff88085464ddd8 R08: 0000000000000005 R09: 0000000000000009
[1190308.540456] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004
[1190308.540457] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004
[1190308.540459] FS: 0000000000000000(0000) GS:ffff88107fd00000(0000) knlGS:0000000000000000
[1190308.540460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1190308.540461] CR2: 00007fd28b8f1420 CR3: 0000000001a0c000 CR4: 00000000000407e0
[1190308.540462] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1190308.540464] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1190308.540464] Stack:
[1190308.540465] ffff88085464ddd8 000000188109b864 ffffe8ffffd00400 00043b9213aa133c
[1190308.540467] ffffffff81a82318 ffffffff81a821a0 ffff88085464de38 ffffffff814ebf0f
[1190308.540469] 0000000000000000 00000000003cc497 0000000000000000 00000000003cc497
[1190308.540471] Call Trace:
[1190308.540474] [<ffffffff814ebf0f>] cpuidle_enter_state+0x4f/0xe0
[1190308.540477] [<ffffffff814ec060>] cpuidle_idle_call+0xc0/0x220
[1190308.540479] [<ffffffff8101c25e>] arch_cpu_idle+0xe/0x30
[1190308.540481] [<ffffffff8109ab78>] cpu_startup_entry+0xe8/0x2b0
[1190308.540484] [<ffffffff81612bef>] start_secondary+0x204/0x20b
[1190308.540485] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4
[1190308.540552] NMI backtrace for cpu 1
[1190308.540563] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.10.58-18 #14
[1190308.540572] Hardware name: Aquarius Aquarius Server/X9DRFF(-7), BIOS 1.0b 10/26/2012
[1190308.540575] task: ffff88085449c880 ti: ffff88085460e000 task.ti: ffff88085460e000
[1190308.540576] RIP: 0010:[<ffffffff8135e53c>] [<ffffffff8135e53c>] intel_idle+0xcc/0x140
[1190308.540588] RSP: 0018:ffff88085460fda8 EFLAGS: 00000046
[1190308.540590] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[1190308.540592] RDX: 0000000000000000 RSI: ffff88085460ffd8 RDI: 0000000000000001
[1190308.540594] RBP: ffff88085460fdd8 R08: 000000000000090d R09: 000000000010fbc1
[1190308.540596] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000004
[1190308.540597] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000004
[1190308.540600] FS: 0000000000000000(0000) GS:ffff88085fc20000(0000) knlGS:0000000000000000
[1190308.540602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1190308.540604] CR2: 0000000000ef8708 CR3: 0000000001a0c000 CR4: 00000000000407e0
[1190308.540606] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1190308.540608] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1190308.540609] Stack:
[1190308.540610] ffff88085460fdd8 000000018109b864 ffffe8f7dfc20400 00043b9213aa1c05
[1190308.540614] ffffffff81a82318 ffffffff81a821a0 ffff88085460fe38 ffffffff814ebf0f
[1190308.540617] 0000000000000000 00000000003cb960 0000000000000000 00000000003cb960
[1190308.540620] Call Trace:
[1190308.540625] [<ffffffff814ebf0f>] cpuidle_enter_state+0x4f/0xe0
[1190308.540628] [<ffffffff814ec060>] cpuidle_idle_call+0xc0/0x220
[1190308.540632] [<ffffffff8101c25e>] arch_cpu_idle+0xe/0x30
[1190308.540635] [<ffffffff8109ab78>] cpu_startup_entry+0xe8/0x2b0
[1190308.540639] [<ffffffff81612bef>] start_secondary+0x204/0x20b
[1190308.540641] Code: 48 8b 34 25 f0 c6 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d ee 3f 72 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 fe 33 d4

--

Regards,
Roman
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/