Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crasheswith linux-3.1-rc2
From: Justin Piszcz
Date: Wed Aug 17 2011 - 17:53:11 EST
On Wed, 17 Aug 2011, Arnaud Lacombe wrote:
Hi,
On Wed, Aug 17, 2011 at 4:45 PM, Justin Piszcz <jpiszcz@xxxxxxxxxxxxxxx> wrote:
On Wed, 17 Aug 2011, Jeff Layton wrote:
The crash is happening in the bowels of the slab allocator.
Specifically, it looks like it's hitting this:
/*
* The slab was either on partial or free list so
* there must be at least one object available for
* allocation.
*/
BUG_ON(slabp->inuse >= cachep->num);
...which looks like maybe the accounting of in-use objects is off. This
really sounds like some sort of memory corruption. I've not been able
to reproduce this so far, but I also had someone report panic here that
might be related:
https://bugzilla.redhat.com/show_bug.cgi?id=731278
One thing that might be helpful is turning on page poisoning and
redoing this test, that might make it crash sooner and point out the
source of the corruption.
Even better would be a bisect to track down the cause...
Hi Jeff,
root@acerlw:/usr/src/linux# grep CONFIG_PAGE_POISONING .config
root@acerlw:/usr/src/linux# ls -l ../linux
lrwxrwxrwx 1 root root 13 Aug 17 14:41 ../linux -> linux-3.1-rc2/
root@acerlw:/usr/src/linux#
In what kernel is that feature available, or, how do I enable it?
It is selected by "Kernel hacking" -> "Debug page memory allocations",
provided your arch support pagealloc debug.
- Arnaud
Hi,
Thanks, a larger dump below with that option enabled:
[ 478.103032] general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC
[ 478.103049] CPU 1
[ 478.103052] Modules linked in: bnep rfcomm bluetooth speedstep_lib cryptd aes_x86_64 aes_generic configfs ohci_hcd ssb ath9k mac80211 uvcvideo ath9k_common ath9k_hw ath videodev mmc_core video edac_core k10temp edac_mce_amd v4l2_compat_ioctl32 i2c_piix4 battery cfg80211 ac pcmcia shpchp pci_hotplug wmi pcmcia_core rfkill
[ 478.103107]
[ 478.103113] Pid: 3978, comm: echo Not tainted 3.1.0-rc2 #3 Acer Aspire 7551 /Aspire 7551
[ 478.103126] RIP: 0010:[<ffffffff8134e839>] [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[ 478.103144] RSP: 0018:ffff88012e0f1e88 EFLAGS: 00010282
[ 478.103150] RAX: ffff88013b65d740 RBX: 000000000000002a RCX: ffff88012e0f1f48
[ 478.103155] RDX: ffffffff8199c18c RSI: ffff88013b7da490 RDI: 9440ffff88013273
[ 478.103161] RBP: ffff88012e0f1e88 R08: 00007fcc4da01700 R09: ffff88013b7da490
[ 478.103166] R10: 0000000000000000 R11: 0000000000000246 R12: 9440ffff88013273
[ 478.103172] R13: 00007fcc4da0e000 R14: ffff8801388e7bc0 R15: ffff8801388e7bc0
[ 478.103179] FS: 00007fcc4da01700(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000
[ 478.103185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 478.103190] CR2: 00007fcc4d538380 CR3: 000000013277c000 CR4: 00000000000006e0
[ 478.103195] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 478.103201] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 478.103207] Process echo (pid: 3978, threadinfo ffff88012e0f0000, task ffff88012e15a150)
[ 478.103212] Stack:
[ 478.103215] ffff88012e0f1ef8 ffffffff8134f17b 0000000000000022 00000007fcc4da0e
[ 478.103227] ffff88012e0f1f18 ffffffff8109f2c7 000000002308a472 ffff88013a9df800
[ 478.103237] 0000000000000003 000000000000002a 00007fcc4da0e000 ffff88012e0f1f48
[ 478.103246] Call Trace:
[ 478.103255] [<ffffffff8134f17b>] tty_write+0x3b/0x290
[ 478.103266] [<ffffffff8109f2c7>] ? do_mmap_pgoff+0x357/0x370
[ 478.103274] [<ffffffff810b6e6a>] vfs_write+0xaa/0x160
[ 478.103280] [<ffffffff810b7155>] sys_write+0x45/0x90
[ 478.103290] [<ffffffff8164debb>] system_call_fastpath+0x16/0x1b
[ 478.103295] Code: 00 00 00 00 00 48 89 df e8 c5 f0 d5 ff 48 8b 5d f0 4c 8b 65 f8 c9 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 85 ff 48 89 e5 74 0c
[ 478.103336] 3f 01 54 00 00 75 2b 31 c0 5d c3 8b 76 44 48 89 d1 48 c7 c7
[ 478.103357] RIP [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[ 478.103366] RSP <ffff88012e0f1e88>
[ 478.103372] ---[ end trace df8e9f10dc5e941d ]---
[ 478.103700] general protection fault: 0000 [#2] SMP DEBUG_PAGEALLOC
[ 478.103711] CPU 0
[ 478.103715] Modules linked in: bnep rfcomm bluetooth speedstep_lib cryptd aes_x86_64 aes_generic configfs ohci_hcd ssb ath9k mac80211 uvcvideo ath9k_common ath9k_hw ath videodev mmc_core video edac_core k10temp edac_mce_amd v4l2_compat_ioctl32 i2c_piix4 battery cfg80211 ac pcmcia shpchp pci_hotplug wmi pcmcia_core rfkill
[ 478.103766]
[ 478.103772] Pid: 3933, comm: atd Tainted: G D 3.1.0-rc2 #3 Acer Aspire 7551 /Aspire 7551
[ 478.103785] RIP: 0010:[<ffffffff8134e839>] [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[ 478.103803] RSP: 0018:ffff880139749e88 EFLAGS: 00010282
[ 478.103808] RAX: ffff88013b65d740 RBX: 0000000000000013 RCX: ffff880139749f48
[ 478.103814] RDX: ffffffff8199c18c RSI: ffff88013b7da490 RDI: 9440ffff88013273
[ 478.103820] RBP: ffff880139749e88 R08: 0000000000000000 R09: ffff88013b7da490
[ 478.103825] R10: 0000000000000000 R11: 0000000000000246 R12: 9440ffff88013273
[ 478.103831] R13: 00007fff04275cb0 R14: ffff8801388e7bc0 R15: ffff8801388e7bc0
[ 478.103838] FS: 00007fd613e52700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
[ 478.103844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 478.103849] CR2: 00007fd613a051d5 CR3: 000000013a034000 CR4: 00000000000006f0
[ 478.103854] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 478.103859] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 478.103866] Process atd (pid: 3933, threadinfo ffff880139748000, task ffff88013f2c3910)
[ 478.103870] Stack:
[ 478.103873] ffff880139749ef8 ffffffff8134f17b 0000000000000f8a 0000000000000000
[ 478.103885] ffffffff810349fd 00007fff04275cec 0000000000000004 0000000000000000
[ 478.103894] 0000000000000000 0000000000000013 00007fff04275cb0 ffff880139749f48[ 478.103903] Call Trace:
[ 478.103913] [<ffffffff8134f17b>] tty_write+0x3b/0x290
[ 478.103924] [<ffffffff810349fd>] ? do_fork+0x13d/0x210
[ 478.103932] [<ffffffff810b6e6a>] vfs_write+0xaa/0x160
[ 478.103938] [<ffffffff810b7155>] sys_write+0x45/0x90
[ 478.103948] [<ffffffff8164debb>] system_call_fastpath+0x16/0x1b
[ 478.103953] Code: 00 00 00 00 00 48 89 df e8 c5 f0 d5 ff 48 8b 5d f0 4c 8b 65 f8 c9 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 85 ff 48 89 e5 74 0c
[ 478.103995] 3f 01 54 00 00 75 2b 31 c0 5d c3 8b 76 44 48 89 d1 48 c7 c7
[ 478.104016] RIP [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[ 478.104025] RSP <ffff880139749e88>
[ 478.104072] ---[ end trace df8e9f10dc5e941e ]---
[ 478.104333] general protection fault: 0000 [#3] SMP DEBUG_PAGEALLOC
[ 478.104352] CPU 0
[ 478.104357] Modules linked in: bnep rfcomm bluetooth speedstep_lib cryptd aes_x86_64 aes_generic configfs ohci_hcd ssb ath9k mac80211 uvcvideo ath9k_common ath9k_hw ath videodev mmc_core video edac_core k10temp edac_mce_amd v4l2_compat_ioctl32 i2c_piix4 battery cfg80211 ac pcmcia shpchp pci_hotplug wmi pcmcia_core rfkill
[ 478.104405]
[ 478.104410] Pid: 3933, comm: atd Tainted: G D 3.1.0-rc2 #3 Acer Aspire 7551 /Aspire 7551
[ 478.104422] RIP: 0010:[<ffffffff8134e839>] [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[ 478.104434] RSP: 0018:ffff880139749b28 EFLAGS: 00010282
[ 478.104439] RAX: ffff88013f344382 RBX: 9440ffff88013273 RCX: 0000000000000000
[ 478.104444] RDX: ffffffff8199c20d RSI: ffff88013b7da490 RDI: 9440ffff88013273
[ 478.104450] RBP: ffff880139749b28 R08: 0000000000000000 R09: 0000000000000000
[ 478.104455] R10: ffff8801388e7bd0 R11: 0000000000000001 R12: 0000000000000008
[ 478.104460] R13: ffff8801388e7bc0 R14: ffff88013b65d740 R15: ffff88013b7da490
[ 478.104467] FS: 00007fd613e52700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
[ 478.104473] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 478.104478] CR2: 00007fd613a051d5 CR3: 0000000001c1d000 CR4: 00000000000006f0
[ 478.104483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 478.104488] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 478.104494] Process atd (pid: 3933, threadinfo ffff880139748000, task ffff88013f2c3910)
[ 478.104498] Stack:
[ 478.104501] ffff880139749bd8 ffffffff8134fe31 ffff880139749b48 ffffffff810d259a
[ 478.104512] ffff880139749b98 ffffffff810b85df ffff88012e08b288 ffff88013a154390
[ 478.104520] 0000000000000000 ffff8801388e7bd0 ffff88013b7da490 0000000800000001
[ 478.104529] Call Trace:
[ 478.104538] [<ffffffff8134fe31>] tty_release+0x41/0x550
[ 478.104546] [<ffffffff810d259a>] ? mntput+0x1a/0x30
[ 478.104554] [<ffffffff810b85df>] ? fput+0x15f/0x200
[ 478.104561] [<ffffffff810b8552>] fput+0xd2/0x200
[ 478.104570] [<ffffffff810b50c1>] filp_close+0x61/0x90
[ 478.104578] [<ffffffff810384bf>] put_files_struct+0x7f/0xe0
[ 478.104585] [<ffffffff810385c4>] exit_files+0x44/0x50
[ 478.104591] [<ffffffff81038bc4>] do_exit+0x5f4/0x790
[ 478.104600] [<ffffffff81036e94>] ? kmsg_dump+0x44/0xe0
[ 478.104609] [<ffffffff81004f25>] oops_end+0x75/0xa0
[ 478.104615] [<ffffffff81005093>] die+0x53/0x80
[ 478.104623] [<ffffffff81002854>] do_general_protection+0x154/0x160
[ 478.104631] [<ffffffff8164da7f>] general_protection+0x1f/0x30
[ 478.104641] [<ffffffff8134e839>] ? tty_paranoia_check+0x9/0x70
[ 478.104649] [<ffffffff8134f17b>] tty_write+0x3b/0x290
[ 478.104657] [<ffffffff810349fd>] ? do_fork+0x13d/0x210
[ 478.104664] [<ffffffff810b6e6a>] vfs_write+0xaa/0x160
[ 478.104670] [<ffffffff810b7155>] sys_write+0x45/0x90
[ 478.104679] [<ffffffff8164debb>] system_call_fastpath+0x16/0x1b
[ 478.104683] Code: 00 00 00 00 00 48 89 df e8 c5 f0 d5 ff 48 8b 5d f0 4c 8b 65 f8 c9 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 85 ff 48 89 e5 74 0c
[ 478.104724] 3f 01 54 00 00 75 2b 31 c0 5d c3 8b 76 44 48 89 d1 48 c7 c7
[ 478.104744] RIP [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[ 478.104753] RSP <ffff880139749b28>
[ 478.104757] ---[ end trace df8e9f10dc5e941f ]---
[ 478.104761] Fixing recursive fault but reboot is needed!
[ 478.152105] general protection fault: 0000 [#4] SMP DEBUG_PAGEALLOC
[ 478.152117] CPU 1
[ 478.152120] Modules linked in: bnep rfcomm bluetooth speedstep_lib cryptd aes_x86_64 aes_generic configfs ohci_hcd ssb ath9k mac80211 uvcvideo ath9k_common ath9k_hw ath videodev mmc_core video edac_core k10temp edac_mce_amd v4l2_compat_ioctl32 i2c_piix4 battery cfg80211 ac pcmcia shpchp pci_hotplug wmi pcmcia_core rfkill
[ 478.152171] [ 478.152177] Pid: 3936, comm: danted Tainted: G D 3.1.0-rc2 #3 Acer Aspire 7551 /Aspire 7551
[ 478.152190] RIP: 0010:[<ffffffff8134e839>] [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[ 478.152208] RSP: 0018:ffff880138abdd18 EFLAGS: 00010286
[ 478.152213] RAX: ffff88013f344300 RBX: 88012e1400003000 RCX: 0000000000000000
[ 478.152219] RDX: ffffffff8199c20d RSI: ffff88013b5732f0 RDI: 88012e1400003000
[ 478.152224] RBP: ffff880138abdd18 R08: 0000000000000000 R09: 0000000000000000
[ 478.152229] R10: ffff8801388e7150 R11: 0000000000000001 R12: 0000000000000008
[ 478.152235] R13: ffff8801388e7140 R14: ffff88012fa19d80 R15: ffff88013b5732f0
[ 478.152241] FS: 00007fe3e8099700(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000
[ 478.152247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 478.152253] CR2: 00007fe3e7bad520 CR3: 0000000001c1d000 CR4: 00000000000006e0
[ 478.152258] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 478.152263] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 478.152269] Process danted (pid: 3936, threadinfo ffff880138abc000, task ffff88012e0c1810)
[ 478.152274] Stack:
[ 478.152277] ffff880138abddc8 ffffffff8134fe31 ffff880138abdd38 ffffffff810d259a
[ 478.152288] ffff880138abdd88 ffffffff810b85df ffff8801326ea3d8 ffff8801325a5250
[ 478.152297] 0000000000000000 ffff8801388e7150 ffff88013b5732f0 0000000800000001
[ 478.152307] Call Trace:
[ 478.152317] [<ffffffff8134fe31>] tty_release+0x41/0x550
[ 478.152326] [<ffffffff810d259a>] ? mntput+0x1a/0x30
[ 478.152334] [<ffffffff810b85df>] ? fput+0x15f/0x200
[ 478.152341] [<ffffffff810b8552>] fput+0xd2/0x200
[ 478.152350] [<ffffffff810b50c1>] filp_close+0x61/0x90
[ 478.152358] [<ffffffff810384bf>] put_files_struct+0x7f/0xe0
[ 478.152365] [<ffffffff810385c4>] exit_files+0x44/0x50
[ 478.152372] [<ffffffff81038bc4>] do_exit+0x5f4/0x790
[ 478.152380] [<ffffffff810baf46>] ? vfs_stat+0x16/0x20
[ 478.152387] [<ffffffff810bb285>] ? sys_newstat+0x15/0x30
[ 478.152394] [<ffffffff810b7040>] ? vfs_read+0x120/0x160
[ 478.152402] [<ffffffff8103908f>] do_group_exit+0x3f/0xa0
[ 478.152409] [<ffffffff81039102>] sys_exit_group+0x12/0x20
[ 478.152418] [<ffffffff8164debb>] system_call_fastpath+0x16/0x1b
[ 478.152423] Code: 00 00 00 00 00 48 89 df e8 c5 f0 d5 ff 48 8b 5d f0 4c 8b 65 f8 c9 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 85 ff 48 89 e5 74 0c
[ 478.152464] 3f 01 54 00 00 75 2b 31 c0 5d c3 8b 76 44 48 89 d1 48 c7 c7
[ 478.152485] RIP [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[ 478.152495] RSP <ffff880138abdd18>
[ 478.152501] ---[ end trace df8e9f10dc5e9420 ]---
[ 478.152505] Fixing recursive fault but reboot is needed!
Justin.