Re: [BUG] random kernel crashes after THP rework on s390 (maybe also on PowerPC and ARM)
From: Gerald Schaefer
Date: Tue Feb 16 2016 - 11:25:00 EST
On Mon, 15 Feb 2016 23:35:26 +0200
"Kirill A. Shutemov" <kirill@xxxxxxxxxxxxx> wrote:
> On Mon, Feb 15, 2016 at 07:37:02PM +0100, Gerald Schaefer wrote:
> > On Mon, 15 Feb 2016 13:31:59 +0200
> > "Kirill A. Shutemov" <kirill@xxxxxxxxxxxxx> wrote:
> >
> > > On Sat, Feb 13, 2016 at 12:58:31PM +0100, Sebastian Ott wrote:
> > > >
> > > > On Sat, 13 Feb 2016, Kirill A. Shutemov wrote:
> > > > > Could you check if revert of fecffad25458 helps?
> > > >
> > > > I reverted fecffad25458 on top of 721675fcf277cf - it oopsed with:
> > > >
> > > > Â 1851.721062! Unable to handle kernel pointer dereference in virtual kernel address space
> > > > Â 1851.721075! failing address: 0000000000000000 TEID: 0000000000000483
> > > > Â 1851.721078! Fault in home space mode while using kernel ASCE.
> > > > Â 1851.721085! AS:0000000000d5c007 R3:00000000ffff0007 S:00000000ffffa800 P:000000000000003d
> > > > Â 1851.721128! Oops: 0004 ilc:3 Â#1! PREEMPT SMP DEBUG_PAGEALLOC
> > > > Â 1851.721135! Modules linked in: bridge stp llc btrfs mlx4_ib mlx4_en ib_sa ib_mad vxlan xor ip6_udp_tunnel ib_core udp_tunnel ptp pps_core ib_addr ghash_s390raid6_pq prng ecb aes_s390 mlx4_core des_s390 des_generic genwqe_card sha512_s390 sha256_s390 sha1_s390 sha_common crc_itu_t dm_mod scm_block vhost_net tun vhost eadm_sch macvtap macvlan kvm autofs4
> > > > Â 1851.721183! CPU: 7 PID: 256422 Comm: bash Not tainted 4.5.0-rc3-00058-g07923d7-dirty #178
> > > > Â 1851.721186! task: 000000007fbfd290 ti: 000000008c604000 task.ti: 000000008c604000
> > > > Â 1851.721189! Krnl PSW : 0704d00180000000 000000000045d3b8 (__rb_erase_color+0x280/0x308)
> > > > Â 1851.721200! R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 EA:3
> > > > Krnl GPRS: 0000000000000001 0000000000000020 0000000000000000 00000000bd07eff1
> > > > Â 1851.721205! 000000000027ca10 0000000000000000 0000000083e45898 0000000077b61198
> > > > Â 1851.721207! 000000007ce1a490 00000000bd07eff0 000000007ce1a548 000000000027ca10
> > > > Â 1851.721210! 00000000bd07c350 00000000bd07eff0 000000008c607aa8 000000008c607a68
> > > > Â 1851.721221! Krnl Code: 000000000045d3aa: e3c0d0080024 stg %%r12,8(%%r13)
> > > > 000000000045d3b0: b9040039 lgr %%r3,%%r9
> > > > #000000000045d3b4: a53b0001 oill %%r3,1
> > > > >000000000045d3b8: e33010000024 stg %%r3,0(%%r1)
> > > > 000000000045d3be: ec28000e007c cgij %%r2,0,8,45d3da
> > > > 000000000045d3c4: e34020000004 lg %%r4,0(%%r2)
> > > > 000000000045d3ca: b904001c lgr %%r1,%%r12
> > > > 000000000045d3ce: ec143f3f0056 rosbg %%r1,%%r4,63,63,0
> > > > Â 1851.721269! Call Trace:
> > > > Â 1851.721273! (Â<0000000083e45898>! 0x83e45898)
> > > > Â 1851.721279! Â<000000000029342a>! unlink_anon_vmas+0x9a/0x1d8
> > > > Â 1851.721282! Â<0000000000283f34>! free_pgtables+0xcc/0x148
> > > > Â 1851.721285! Â<000000000028c376>! exit_mmap+0xd6/0x300
> > > > Â 1851.721289! Â<0000000000134db8>! mmput+0x90/0x118
> > > > Â 1851.721294! Â<00000000002d76bc>! flush_old_exec+0x5d4/0x700
> > > > Â 1851.721298! Â<00000000003369f4>! load_elf_binary+0x2f4/0x13e8
> > > > Â 1851.721301! Â<00000000002d6e4a>! search_binary_handler+0x9a/0x1f8
> > > > Â 1851.721304! Â<00000000002d8970>! do_execveat_common.isra.32+0x668/0x9a0
> > > > Â 1851.721307! Â<00000000002d8cec>! do_execve+0x44/0x58
> > > > Â 1851.721310! Â<00000000002d8f92>! SyS_execve+0x3a/0x48
> > > > Â 1851.721315! Â<00000000006fb096>! system_call+0xd6/0x258
> > > > Â 1851.721317! Â<000003ff997436d6>! 0x3ff997436d6
> > > > Â 1851.721319! INFO: lockdep is turned off.
> > > > Â 1851.721321! Last Breaking-Event-Address:
> > > > Â 1851.721323! Â<000000000045d31a>! __rb_erase_color+0x1e2/0x308
> > > > Â 1851.721327!
> > > > Â 1851.721329! ---Â end trace 0d80041ac00cfae2 !---
> > > >
> > > >
> > > > >
> > > > > And could you share how crashes looks like? I haven't seen backtraces yet.
> > > > >
> > > >
> > > > Sure. I didn't because they really looked random to me. Most of the time
> > > > in rcu or list debugging but I thought these have just been the messenger
> > > > observing a corruption first. Anyhow, here is an older one that might look
> > > > interesting:
> > > >
> > > > [ 59.851421] list_del corruption. next->prev should be 000000006e1eb000, but was 0000000000000400
> > >
> > > This kinda interesting: 0x400 is TAIL_MAPPING.. Hm..
> > >
> > > Could you check if you see the problem on commit 1c290f642101 and its
> > > immediate parent?
> > >
> >
> > How should the page->mapping poison end up as next->prev in the list of
> > pre-allocated THP splitting page tables?
>
> May be pgtable was casted to struct page or something. I don't know.
>
> > Also, commit 1c290f642101 is before the THP rework, at least the
> > non-bisectable part, so we should expect not to see the problem there.
>
> Just to make sure: commit 122afea9626a is fine, commit 61f5d698cc97
> crashes. Correct?
>
> > 0x400 is also the value of an empty pte on s390, and the thp_deposit/withdraw
> > listheads are placed inside the pre-allocated pagetables instead of page->lru,
> > because we have 2K pagetables on s390 and cannot use struct page == pgtable_t.
>
> 0x400 from empty pte makes more sense than TAIL_MAPPING. But I guess it
> worth changing TAIL_MAPPING to some other value to make sure.
Right, but we cannot trigger this list corruption symptom reliably, in fact
I didn't hit it at all during the last runs, and previous crash logs also
showed list corruptions with other values than 0x400, which may hint towards
concurrent pagetable freeing and re-use, given that our THP splitting pagetable
listhead is located inside the pre-allocated pagetables.
>
> > So, for example, two concurrent withdraws could produce such a list
> > corruption, because the first withdraw will overwrite the listhead at the
> > beginning of the pagetable with 2 empty ptes.
> >
> > Has anything changed regarding the general THP deposit/withdraw logic?
>
> I don't see any changes in this area.
>
> To eliminate one more variable, I would propose to disable split pmd lock
> for testing and check if it makes difference.
Disabling ARCH_ENABLE_SPLIT_PMD_PTLOCK didn't make any difference, other
than maybe a little reduction in "randomness" of the crashes, but that
may be pure coincidence. Out of about 10 runs, I always ended up with either
ODEBUG "WARNING: at lib/debugobjects.c:263" and subsequent "kernel BUG at
mm/slub.c:3629", or "bad swap file / page map" with subsequent "kernel BUG
at kernel/cred.c:142", see below for the full traces.
>
> Is there any chance that I'll be able to trigger the bug using QEMU?
> Does anybody have an QEMU image I can use?
>
I have no image, but trying to reproduce this under virtualization may
help to trigger this also on other architectures. After ruling out IPI
vs. fast_gup I do not really see why this should be arch-specific, and
it wouldn't be the first time that we hit subtle races first on s390, due
to our virtualized environment (my test case is make -j20 with 10 CPUs and
4GB of memory, no swap).
Here are the full traces from the runs w/o split pmd lock:
1)
[ 2584.391880] cc1 (71885) used greatest stack depth: 10496 bytes left
[ 2951.268250] ld (147667) used greatest stack depth: 10472 bytes left
[ 2972.530753] swap_free: Bad swap file entry 1000000000000000
[ 2972.530763] BUG: Bad page map in process cc1 pte:00000420 pmd:6cfd3000
[ 2972.530766] addr:0000000080d00000 vm_flags:00000875 anon_vma: (null) mapping:000000005dc6ac70 index
:d00
[ 2972.530776] file:cc1 fault:ext4_filemap_fault mmap:ext4_file_mmap readpage:ext4_readpage
[ 2972.530781] CPU: 6 PID: 152043 Comm: cc1 Not tainted 4.5.0-rc4-00014-g1926e54-dirty #70
[ 2972.530784] 0000000071947a60 0000000071947af0 0000000000000002 0000000000000000
0000000071947b90 0000000071947b08 0000000071947b08 0000000000113d38
0000000000000000 0000000000b70df4 0000000000b4f348 000000000000000b
0000000071947b50 0000000071947af0 0000000000000000 0000000000000000
07000000c3763ae8 0000000000113d38 0000000071947af0 0000000071947b50
[ 2972.530811] Call Trace:
[ 2972.530818] ([<0000000000113c3c>] show_trace+0x12c/0x150)
[ 2972.530821] [<0000000000113cee>] show_stack+0x8e/0xf0
[ 2972.530826] [<000000000068b8ec>] dump_stack+0x9c/0xe0
[ 2972.530830] [<00000000002bbeda>] print_bad_pte+0x222/0x238
[ 2972.530833] [<00000000002beb92>] zap_pte_range+0x442/0x790
[ 2972.530835] [<00000000002bf2c6>] unmap_single_vma+0x3e6/0x400
[ 2972.530837] [<00000000002c0f46>] unmap_vmas+0x8e/0xc8
[ 2972.530840] [<00000000002c9a56>] exit_mmap+0xc6/0x300
[ 2972.530844] [<0000000000138b10>] mmput+0xa0/0x128
[ 2972.530847] [<000000000013fcb4>] do_exit+0x42c/0xd60
[ 2972.530849] [<00000000001406f0>] do_group_exit+0x98/0xe0
[ 2972.530851] [<0000000000140768>] __wake_up_parent+0x0/0x28
[ 2972.530855] [<0000000000910f2e>] system_call+0xd6/0x270
[ 2972.530883] [<000003ff89b43698>] 0x3ff89b43698
[ 2972.530886] 1 lock held by cc1/152043:
[ 2972.530887] #0: (&(ptlock_ptr(page))->rlock){+.+.-.}, at: [<00000000002be7f6>] zap_pte_range+0xa6/0x790
[ 2972.530897] Disabling lock debugging due to kernel taint
[ 2972.533069] BUG: Bad rss-counter state mm:00000000719d0e00 idx:2 val:-1
[ 5899.109157] ------------[ cut here ]------------
[ 5899.109166] kernel BUG at kernel/cred.c:142!
[ 5899.109211] illegal operation: 0001 ilc:1 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 5899.109217] Modules linked in: nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc mlx4_ib ib_sa ib_mad mlx4_en ib_core vxlan udp_tunnel ptp ib_addr pps_core ghash_s390 prng ecb aes_s390 des_s390 des_generic sha512_s390 sha256_s390 sha1_s390 sha_common mlx4_core eadm_sch nfsd vhost_net tun vhost macvtap auth_rpcgss macvlan kvm oid_registry nfs_acl lockd grace sunrpc dm_multipath dm_mod autofs4
[ 5899.109279] CPU: 1 PID: 12 Comm: ksoftirqd/1 Tainted: G B 4.5.0-rc4-00014-g1926e54-dirty #70
[ 5899.109283] task: 00000000d09e2a48 ti: 00000000d09f4000 task.ti: 00000000d09f4000
[ 5899.109286] Krnl PSW : 0704c00180000000 00000000001651aa (__put_cred+0x22/0x68)
[ 5899.109296] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 EA:3
Krnl GPRS: 0000000000000002 0000000000000020 000000007431f000 00000000c38e3400
[ 5899.109301] 000000000032aaf8 0000000000000002 0000000000000000 000000000000000a
[ 5899.109304] 0000000000000000 000000000032aac0 0000000000000008 00000000749ad000
[ 5899.109306] 00000000c38e3400 000000007431f000 000000000032ab2e 00000000d09f7bf0
[ 5899.109316] Krnl Code: 000000000016519c: 58102004 l %%r1,4(%%r2)
00000000001651a0: ec180005007e cij %%r1,0,8,1651aa
#00000000001651a6: a7f40001 brc 15,1651a8
>00000000001651aa: e3e020080024 stg %%r14,8(%%r2)
00000000001651b0: c01944656144 iilf %%r1,1147494724
00000000001651b6: 50102010 st %%r1,16(%%r2)
00000000001651ba: e31003100004 lg %%r1,784
00000000001651c0: e32018300020 cg %%r2,2096(%%r1)
[ 5899.109371] Call Trace:
[ 5899.109376] ([<000000000032aaf8>] file_free_rcu+0x38/0x88)
[ 5899.109381] [<00000000001c5ddc>] rcu_process_callbacks+0x5fc/0x9f0
[ 5899.109385] [<0000000000141794>] __do_softirq+0x25c/0x570
[ 5899.109387] [<0000000000141ae6>] run_ksoftirqd+0x3e/0xa0
[ 5899.109391] [<0000000000167bee>] smpboot_thread_fn+0x30e/0x360
[ 5899.109394] [<0000000000162f4a>] kthread+0x112/0x128
[ 5899.109398] [<00000000009110fa>] kernel_thread_starter+0x6/0xc
[ 5899.109401] [<00000000009110f4>] kernel_thread_starter+0x0/0xc
[ 5899.109403] INFO: lockdep is turned off.
[ 5899.109405] Last Breaking-Event-Address:
[ 5899.109407] [<00000000001651a6>] __put_cred+0x1e/0x68
[ 5899.109411]
[ 5899.109414] Kernel panic - not syncing: Fatal exception in interrupt
2)
[ 7790.934295] ODEBUG: active_state not available (active state 0) object type: rcu_head hint: (null)
[ 7790.934356] ------------[ cut here ]------------
[ 7790.934359] WARNING: at lib/debugobjects.c:263
[ 7790.934361] Modules linked in: nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack mlx4_ib ib_sa ipt_REJECT mlx4_en ib_mad nf_reject_ipv4 ib_core vxlan udp_tunnel ptp xt_tcpudp ib_addr pps_core iptable_filter ip_tables x_tables bridge stp llc ghash_s390 prng ecb aes_s390 des_s390 des_generic sha512_s390 sha256_s390 mlx4_core sha1_s390 sha_common eadm_sch vhost_net nfsd tun vhost macvtap macvlan auth_rpcgss kvm oid_registry nfs_acl lockd grace sunrpc dm_multipath dm_mod autofs4
[ 7790.934417] CPU: 8 PID: 40 Comm: ksoftirqd/8 Not tainted 4.5.0-rc4-00014-g1926e54-dirty #149
[ 7790.934420] task: 00000000e2955490 ti: 00000000e2958000 task.ti: 00000000e2958000
[ 7790.934422] Krnl PSW : 0404c00180000000 000000000071c340 (debug_print_object+0xb0/0xd0)
[ 7790.934431] R:0 T:1 IO:0 EX:0 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 EA:3
Krnl GPRS: 0000000001e6e3c7 00000000e2955490 0000000000000060 00000000e2958000
[ 7790.934435] 000000000071c33c 0000000000000000 0000000000b975e8 0000000001f2b008
[ 7790.934437] 07000000001d7e24 0000000000000000 0000000001f2b010 0000000000bea6b8
[ 7790.934440] 0000000000e241f8 00000000e295bc38 000000000071c33c 00000000e295bb38
[ 7790.934449] Krnl Code: 000000000071c330: c41f00bf6a14 strl %%r1,1f09758
000000000071c336: c0e5ffdbd64d brasl %%r14,296fd0
#000000000071c33c: a7f40001 brc 15,71c33e
>000000000071c340: c41d0036e746 lrl %%r1,df91cc
000000000071c346: e340f0e80004 lg %%r4,232(%%r15)
000000000071c34c: a71a0001 ahi %%r1,1
000000000071c350: eb6ff0a80004 lmg %%r6,%%r15,168(%%r15)
000000000071c356: c41f0036e73b strl %%r1,df91cc
[ 7790.934493] Call Trace:
[ 7790.934495] ([<000000000071c33c>] debug_print_object+0xac/0xd0)
[ 7790.934498] [<000000000071d704>] debug_object_active_state+0x164/0x178
[ 7790.934504] [<00000000001d7da4>] rcu_process_callbacks+0x57c/0xa00
[ 7790.934508] [<00000000001487ec>] __do_softirq+0x26c/0x580
[ 7790.934510] [<0000000000148b50>] run_ksoftirqd+0x50/0xb0
[ 7790.934515] [<0000000000172b28>] smpboot_thread_fn+0x320/0x378
[ 7790.934517] [<000000000016d21c>] kthread+0x124/0x138
[ 7790.934521] [<00000000009a1d72>] kernel_thread_starter+0x6/0xc
[ 7790.934524] [<00000000009a1d6c>] kernel_thread_starter+0x0/0xc
[ 7790.934526] 1 lock held by ksoftirqd/8/40:
[ 7790.934528] #0: (&obj_hash[i].lock){-.-.-.}, at: [<000000000071d64c>] debug_object_active_state+0xac/0x178
[ 7790.934535] Last Breaking-Event-Address:
[ 7790.934537] [<000000000071c33c>] debug_print_object+0xac/0xd0
[ 7790.934539] ---[ end trace b583bfd967a78637 ]---
[ 7790.934543] ODEBUG: deactivate not available (active state 0) object type: rcu_head hint: (null)
[ 7790.934551] ------------[ cut here ]------------
[ 7790.934553] WARNING: at lib/debugobjects.c:263
[ 7790.934555] Modules linked in: nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack mlx4_ib ib_sa ipt_REJECT mlx4_en ib_mad nf_reject_ipv4 ib_core vxlan udp_tunnel ptp xt_tcpudp ib_addr pps_core iptable_filter ip_tables x_tables bridge stp llc ghash_s390 prng ecb aes_s390 des_s390 des_generic sha512_s390 sha256_s390 mlx4_core sha1_s390 sha_common eadm_sch vhost_net nfsd tun vhost macvtap macvlan auth_rpcgss kvm oid_registry nfs_acl lockd grace sunrpc dm_multipath dm_mod autofs4
[ 7790.934599] CPU: 8 PID: 40 Comm: ksoftirqd/8 Tainted: G W 4.5.0-rc4-00014-g1926e54-dirty #149
[ 7790.934601] task: 00000000e2955490 ti: 00000000e2958000 task.ti: 00000000e2958000
[ 7790.934603] Krnl PSW : 0404c00180000000 000000000071c340 (debug_print_object+0xb0/0xd0)
[ 7790.934608] R:0 T:1 IO:0 EX:0 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 EA:3
Krnl GPRS: 0000000001e6e3c7 00000000e2955490 000000000000005e 00000000e2958000
[ 7790.934612] 000000000071c33c 0000000000000000 0000000000b975e8 000000000000000a
[ 7790.934614] 0000000004bcd020 0700000001f2b010 0000000001f2b010 0000000000ba5d0a
[ 7790.934617] 0000000000e241f8 00000000e295bc48 000000000071c33c 00000000e295bb48
[ 7790.934622] Krnl Code: 000000000071c330: c41f00bf6a14 strl %%r1,1f09758
000000000071c336: c0e5ffdbd64d brasl %%r14,296fd0
#000000000071c33c: a7f40001 brc 15,71c33e
>000000000071c340: c41d0036e746 lrl %%r1,df91cc
000000000071c346: e340f0e80004 lg %%r4,232(%%r15)
000000000071c34c: a71a0001 ahi %%r1,1
000000000071c350: eb6ff0a80004 lmg %%r6,%%r15,168(%%r15)
000000000071c356: c41f0036e73b strl %%r1,df91cc
[ 7790.934639] Call Trace:
[ 7790.934641] ([<000000000071c33c>] debug_print_object+0xac/0xd0)
[ 7790.934644] [<000000000071d0a8>] debug_object_deactivate+0x170/0x188
[ 7790.934646] [<00000000001d7db6>] rcu_process_callbacks+0x58e/0xa00
[ 7790.934648] [<00000000001487ec>] __do_softirq+0x26c/0x580
[ 7790.934651] [<0000000000148b50>] run_ksoftirqd+0x50/0xb0
[ 7790.934653] [<0000000000172b28>] smpboot_thread_fn+0x320/0x378
[ 7790.934655] [<000000000016d21c>] kthread+0x124/0x138
[ 7790.934657] [<00000000009a1d72>] kernel_thread_starter+0x6/0xc
[ 7790.934659] [<00000000009a1d6c>] kernel_thread_starter+0x0/0xc
[ 7790.934661] 1 lock held by ksoftirqd/8/40:
[ 7790.934663] #0: (&obj_hash[i].lock){-.-.-.}, at: [<000000000071cfdc>] debug_object_deactivate+0xa4/0x188
[ 7790.934669] Last Breaking-Event-Address:
[ 7790.934671] [<000000000071c33c>] debug_print_object+0xac/0xd0
[ 7790.934673] ---[ end trace b583bfd967a78638 ]---
[ 7790.934680] ------------[ cut here ]------------
[ 7790.934682] kernel BUG at mm/slub.c:3629!
[ 7790.934707] illegal operation: 0001 ilc:1 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 7790.934715] Modules linked in: nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack mlx4_ib ib_sa ipt_REJECT mlx4_en ib_mad nf_reject_ipv4 ib_core vxlan udp_tunnel ptp xt_tcpudp ib_addr pps_core iptable_filter ip_tables x_tables bridge stp llc ghash_s390 prng ecb aes_s390 des_s390 des_generic sha512_s390 sha256_s390 mlx4_core sha1_s390 sha_common eadm_sch vhost_net nfsd tun vhost macvtap macvlan auth_rpcgss kvm oid_registry nfs_acl lockd grace sunrpc dm_multipath dm_mod autofs4
[ 7790.934789] CPU: 8 PID: 40 Comm: ksoftirqd/8 Tainted: G W 4.5.0-rc4-00014-g1926e54-dirty #149
[ 7790.934791] task: 00000000e2955490 ti: 00000000e2958000 task.ti: 00000000e2958000
[ 7790.934794] Krnl PSW : 0704c00180000000 000000000032295a (kfree+0x3f2/0x428)
[ 7790.934801] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 EA:3
Krnl GPRS: 0000000000000000 0000000000000100 0000000000000100 0000000000e24260
[ 7790.934806] 00000000001d1c82 0000000000000000 0000000000000000 000000000000000a
[ 7790.934809] 0000000000000001 00000000001d7e0a 0000000000000006 000003d10012f340
[ 7790.934812] 0000000004bcd000 0000000000f0433c 000000000032267a 00000000e295bbb0
[ 7790.934818] Krnl Code: 000000000032294c: c0e50033bef6 brasl %%r14,99a738
0000000000322952: a7f4feba brc 15,3226c6
#0000000000322956: a7f40001 brc 15,322958
>000000000032295a: e310b0060090 llgc %%r1,6(%%r11)
0000000000322960: a7110040 tmll %%r1,64
0000000000322964: a774fee9 brc 7,322736
0000000000322968: a7f4feeb brc 15,32273e
000000000032296c: c0e50033be32 brasl %%r14,99a5d0
[ 7790.934838] Call Trace:
[ 7790.934841] ([<000000000032267a>] kfree+0x112/0x428)
[ 7790.934844] [<00000000001d7e0a>] rcu_process_callbacks+0x5e2/0xa00
[ 7790.934847] [<00000000001487ec>] __do_softirq+0x26c/0x580
[ 7790.934850] [<0000000000148b50>] run_ksoftirqd+0x50/0xb0
[ 7790.934854] [<0000000000172b28>] smpboot_thread_fn+0x320/0x378
[ 7790.934856] [<000000000016d21c>] kthread+0x124/0x138
[ 7790.934859] [<00000000009a1d72>] kernel_thread_starter+0x6/0xc
[ 7790.934862] [<00000000009a1d6c>] kernel_thread_starter+0x0/0xc
[ 7790.934864] INFO: lockdep is turned off.
[ 7790.934866] Last Breaking-Event-Address:
[ 7790.934869] [<0000000000322956>] kfree+0x3ee/0x428
[ 7790.934873]
[ 7790.934876] Kernel panic - not syncing: Fatal exception in interrupt