Re: Hang in XFS reclaim on 3.7.0-rc3

[Torsten Kaiser]

On Mon, Nov 19, 2012 at 12:51 AM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
> On Sun, Nov 18, 2012 at 04:29:22PM +0100, Torsten Kaiser wrote:
>> On Sun, Nov 18, 2012 at 11:24 AM, Torsten Kaiser
>> <just.for.lkml@xxxxxxxxxxxxxx> wrote:
>> > On Tue, Oct 30, 2012 at 9:37 PM, Torsten Kaiser
>> > <just.for.lkml@xxxxxxxxxxxxxx> wrote:
>> >> I will keep LOCKDEP enabled on that system, and if there really is
>> >> another splat, I will report back here. But I rather doubt that this
>> >> will be needed.
>> >
>> > After the patch, I did not see this problem again, but today I found
>> > another LOCKDEP report that also looks XFS related.
>> > I found it twice in the logs, and as both were slightly different, I
>> > will attach both versions.
>>
>> > Nov  6 21:57:09 thoregon kernel: [ 9941.104353] 3.7.0-rc4 #1 Not tainted
>> > Nov  6 21:57:09 thoregon kernel: [ 9941.104355] inconsistent
>> > {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage.
>> > Nov  6 21:57:09 thoregon kernel: [ 9941.104430]        CPU0
>> > Nov  6 21:57:09 thoregon kernel: [ 9941.104431]        ----
>> > Nov  6 21:57:09 thoregon kernel: [ 9941.104432]   lock(&(&ip->i_lock)->mr_lock);
>> > Nov  6 21:57:09 thoregon kernel: [ 9941.104433]   <Interrupt>
>> > Nov  6 21:57:09 thoregon kernel: [ 9941.104434]
>> > lock(&(&ip->i_lock)->mr_lock);
>> > Nov  6 21:57:09 thoregon kernel: [ 9941.104435]
>> > Nov  6 21:57:09 thoregon kernel: [ 9941.104435]  *** DEADLOCK ***
>>
>> Sorry! Copied the wrong report. Your fix only landed in -rc5, so my
>> vanilla -rc4 did (also) report the old problem again.
>> And I copy&pasted that report instead of the second appearance of the
>> new problem.
>
> Can you repost it with line wrapping turned off? The output simply
> becomes unreadable when it wraps....
>
> Yeah, I know I can put it back together, but I've got better things
> to do with my time than stitch a couple of hundred lines of debug
> back into a readable format....

Sorry about that, but I can't find any option to turn that off in Gmail.

I have added the reports as attachment, I hope thats OK for you.

Thanks for looking into this.

Torsten
[110926.972477] 
[110926.972482] =========================================================
[110926.972484] [ INFO: possible irq lock inversion dependency detected ]
[110926.972486] 3.7.0-rc4 #1 Not tainted
[110926.972487] ---------------------------------------------------------
[110926.972489] kswapd0/725 just changed the state of lock:
[110926.972490]  (sb_internal){.+.+.?}, at: [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50
[110926.972499] but this lock took another, RECLAIM_FS-unsafe lock in the past:
[110926.972500]  (&(&ip->i_lock)->mr_lock/1){+.+.+.}
[110926.972500] 
[110926.972500] and interrupts could create inverse lock ordering between them.
[110926.972500] 
[110926.972503] 
[110926.972503] other info that might help us debug this:
[110926.972504]  Possible interrupt unsafe locking scenario:
[110926.972504] 
[110926.972505]        CPU0                    CPU1
[110926.972506]        ----                    ----
[110926.972507]   lock(&(&ip->i_lock)->mr_lock/1);
[110926.972509]                                local_irq_disable();
[110926.972509]                                lock(sb_internal);
[110926.972511]                                lock(&(&ip->i_lock)->mr_lock/1);
[110926.972512]   <Interrupt>
[110926.972513]     lock(sb_internal);
[110926.972514] 
[110926.972514]  *** DEADLOCK ***
[110926.972514] 
[110926.972516] 2 locks held by kswapd0/725:
[110926.972517]  #0:  (shrinker_rwsem){++++..}, at: [<ffffffff810bbd22>] shrink_slab+0x32/0x1f0
[110926.972522]  #1:  (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a8e>] grab_super_passive+0x3e/0x90
[110926.972527] 
[110926.972527] the shortest dependencies between 2nd lock and 1st lock:
[110926.972533]  -> (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 58117 {
[110926.972536]     HARDIRQ-ON-W at:
[110926.972537]                       [<ffffffff8107f091>] __lock_acquire+0x631/0x1c00
[110926.972540]                       [<ffffffff81080b55>] lock_acquire+0x55/0x70
[110926.972542]                       [<ffffffff8106126a>] down_write_nested+0x4a/0x70
[110926.972545]                       [<ffffffff811e8164>] xfs_ilock+0x84/0xb0
[110926.972548]                       [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0
[110926.972550]                       [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0
[110926.972552]                       [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10
[110926.972554]                       [<ffffffff81100332>] vfs_create+0x72/0xc0
[110926.972556]                       [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80
[110926.972558]                       [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490
[110926.972560]                       [<ffffffff8110184d>] do_filp_open+0x3d/0xa0
[110926.972562]                       [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0
[110926.972565]                       [<ffffffff810f223c>] sys_open+0x1c/0x20
[110926.972567]                       [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b
[110926.972570]     SOFTIRQ-ON-W at:
[110926.972571]                       [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00
[110926.972573]                       [<ffffffff81080b55>] lock_acquire+0x55/0x70
[110926.972574]                       [<ffffffff8106126a>] down_write_nested+0x4a/0x70
[110926.972576]                       [<ffffffff811e8164>] xfs_ilock+0x84/0xb0
[110926.972578]                       [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0
[110926.972580]                       [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0
[110926.972581]                       [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10
[110926.972583]                       [<ffffffff81100332>] vfs_create+0x72/0xc0
[110926.972585]                       [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80
[110926.972587]                       [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490
[110926.972589]                       [<ffffffff8110184d>] do_filp_open+0x3d/0xa0
[110926.972591]                       [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0
[110926.972593]                       [<ffffffff810f223c>] sys_open+0x1c/0x20
[110926.972595]                       [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b
[110926.972597]     RECLAIM_FS-ON-W at:
[110926.972598]                          [<ffffffff8108137e>] mark_held_locks+0x7e/0x130
[110926.972600]                          [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0
[110926.972601]                          [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0
[110926.972603]                          [<ffffffff810dba31>] vm_map_ram+0x271/0x770
[110926.972606]                          [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0
[110926.972609]                          [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130
[110926.972610]                          [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0
[110926.972613]                          [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0
[110926.972616]                          [<ffffffff8121c25e>] xfs_ialloc_ag_alloc+0x1be/0x560
[110926.972618]                          [<ffffffff8121da65>] xfs_dialloc+0x185/0x2a0
[110926.972619]                          [<ffffffff8121f198>] xfs_ialloc+0x58/0x650
[110926.972621]                          [<ffffffff811f3985>] xfs_dir_ialloc+0x65/0x270
[110926.972623]                          [<ffffffff811f536c>] xfs_create+0x3ac/0x5a0
[110926.972624]                          [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0
[110926.972626]                          [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10
[110926.972628]                          [<ffffffff81100332>] vfs_create+0x72/0xc0
[110926.972630]                          [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80
[110926.972632]                          [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490
[110926.972634]                          [<ffffffff8110184d>] do_filp_open+0x3d/0xa0
[110926.972636]                          [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0
[110926.972638]                          [<ffffffff810f223c>] sys_open+0x1c/0x20
[110926.972640]                          [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b
[110926.972642]     INITIAL USE at:
[110926.972642]                      [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00
[110926.972644]                      [<ffffffff81080b55>] lock_acquire+0x55/0x70
[110926.972646]                      [<ffffffff8106126a>] down_write_nested+0x4a/0x70
[110926.972648]                      [<ffffffff811e8164>] xfs_ilock+0x84/0xb0
[110926.972650]                      [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0
[110926.972651]                      [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0
[110926.972653]                      [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10
[110926.972655]                      [<ffffffff81100332>] vfs_create+0x72/0xc0
[110926.972657]                      [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80
[110926.972659]                      [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490
[110926.972661]                      [<ffffffff8110184d>] do_filp_open+0x3d/0xa0
[110926.972663]                      [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0
[110926.972664]                      [<ffffffff810f223c>] sys_open+0x1c/0x20
[110926.972666]                      [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b
[110926.972668]   }
[110926.972669]   ... key      at: [<ffffffff825b4b81>] __key.41355+0x1/0x8
[110926.972672]   ... acquired at:
[110926.972672]    [<ffffffff81080b55>] lock_acquire+0x55/0x70
[110926.972674]    [<ffffffff8106126a>] down_write_nested+0x4a/0x70
[110926.972676]    [<ffffffff811e8164>] xfs_ilock+0x84/0xb0
[110926.972678]    [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0
[110926.972679]    [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0
[110926.972681]    [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10
[110926.972683]    [<ffffffff81100332>] vfs_create+0x72/0xc0
[110926.972684]    [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80
[110926.972686]    [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490
[110926.972688]    [<ffffffff8110184d>] do_filp_open+0x3d/0xa0
[110926.972690]    [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0
[110926.972692]    [<ffffffff810f223c>] sys_open+0x1c/0x20
[110926.972694]    [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b
[110926.972696] 
[110926.972696] -> (sb_internal){.+.+.?} ops: 1710064 {
[110926.972699]    HARDIRQ-ON-R at:
[110926.972700]                     [<ffffffff8107ef6a>] __lock_acquire+0x50a/0x1c00
[110926.972702]                     [<ffffffff81080b55>] lock_acquire+0x55/0x70
[110926.972704]                     [<ffffffff810f452b>] __sb_start_write+0xab/0x190
[110926.972705]                     [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50
[110926.972707]                     [<ffffffff811f5147>] xfs_create+0x187/0x5a0
[110926.972709]                     [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0
[110926.972711]                     [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10
[110926.972712]                     [<ffffffff81100332>] vfs_create+0x72/0xc0
[110926.972714]                     [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80
[110926.972716]                     [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490
[110926.972718]                     [<ffffffff8110184d>] do_filp_open+0x3d/0xa0
[110926.972720]                     [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0
[110926.972722]                     [<ffffffff810f223c>] sys_open+0x1c/0x20
[110926.972724]                     [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b
[110926.972726]    SOFTIRQ-ON-R at:
[110926.972727]                     [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00
[110926.972728]                     [<ffffffff81080b55>] lock_acquire+0x55/0x70
[110926.972730]                     [<ffffffff810f452b>] __sb_start_write+0xab/0x190
[110926.972732]                     [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50
[110926.972734]                     [<ffffffff811f5147>] xfs_create+0x187/0x5a0
[110926.972735]                     [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0
[110926.972737]                     [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10
[110926.972739]                     [<ffffffff81100332>] vfs_create+0x72/0xc0
[110926.972741]                     [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80
[110926.972743]                     [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490
[110926.972745]                     [<ffffffff8110184d>] do_filp_open+0x3d/0xa0
[110926.972747]                     [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0
[110926.972749]                     [<ffffffff810f223c>] sys_open+0x1c/0x20
[110926.972750]                     [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b
[110926.972752]    IN-RECLAIM_FS-R at:
[110926.972753]                        [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00
[110926.972755]                        [<ffffffff81080b55>] lock_acquire+0x55/0x70
[110926.972757]                        [<ffffffff810f452b>] __sb_start_write+0xab/0x190
[110926.972758]                        [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50
[110926.972760]                        [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250
[110926.972762]                        [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480
[110926.972763]                        [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80
[110926.972765]                        [<ffffffff8110cb8f>] evict+0xaf/0x1b0
[110926.972768]                        [<ffffffff8110d219>] dispose_list+0x39/0x50
[110926.972770]                        [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340
[110926.972772]                        [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0
[110926.972773]                        [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0
[110926.972775]                        [<ffffffff810be400>] kswapd+0x690/0xa10
[110926.972777]                        [<ffffffff8105c246>] kthread+0xd6/0xe0
[110926.972779]                        [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0
[110926.972781]    RECLAIM_FS-ON-R at:
[110926.972782]                        [<ffffffff8108137e>] mark_held_locks+0x7e/0x130
[110926.972784]                        [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0
[110926.972785]                        [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0
[110926.972787]                        [<ffffffff811f6d3f>] kmem_zone_alloc+0x5f/0xe0
[110926.972789]                        [<ffffffff811f6dd8>] kmem_zone_zalloc+0x18/0x50
[110926.972790]                        [<ffffffff8122b1e2>] _xfs_trans_alloc+0x32/0x90
[110926.972792]                        [<ffffffff8122b278>] xfs_trans_alloc+0x38/0x50
[110926.972794]                        [<ffffffff811f5147>] xfs_create+0x187/0x5a0
[110926.972796]                        [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0
[110926.972797]                        [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10
[110926.972799]                        [<ffffffff81100332>] vfs_create+0x72/0xc0
[110926.972801]                        [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80
[110926.972803]                        [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490
[110926.972805]                        [<ffffffff8110184d>] do_filp_open+0x3d/0xa0
[110926.972807]                        [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0
[110926.972809]                        [<ffffffff810f223c>] sys_open+0x1c/0x20
[110926.972811]                        [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b
[110926.972813]    INITIAL USE at:
[110926.972814]                    [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00
[110926.972815]                    [<ffffffff81080b55>] lock_acquire+0x55/0x70
[110926.972817]                    [<ffffffff810f452b>] __sb_start_write+0xab/0x190
[110926.972819]                    [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50
[110926.972820]                    [<ffffffff811f5147>] xfs_create+0x187/0x5a0
[110926.972822]                    [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0
[110926.972824]                    [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10
[110926.972826]                    [<ffffffff81100332>] vfs_create+0x72/0xc0
[110926.972827]                    [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80
[110926.972829]                    [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490
[110926.972831]                    [<ffffffff8110184d>] do_filp_open+0x3d/0xa0
[110926.972833]                    [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0
[110926.972835]                    [<ffffffff810f223c>] sys_open+0x1c/0x20
[110926.972837]                    [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b
[110926.972839]  }
[110926.972840]  ... key      at: [<ffffffff81c34e40>] xfs_fs_type+0x60/0x80
[110926.972842]  ... acquired at:
[110926.972843]    [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140
[110926.972845]    [<ffffffff8107e900>] mark_lock+0x190/0x2f0
[110926.972846]    [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00
[110926.972848]    [<ffffffff81080b55>] lock_acquire+0x55/0x70
[110926.972850]    [<ffffffff810f452b>] __sb_start_write+0xab/0x190
[110926.972851]    [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50
[110926.972853]    [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250
[110926.972855]    [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480
[110926.972856]    [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80
[110926.972858]    [<ffffffff8110cb8f>] evict+0xaf/0x1b0
[110926.972860]    [<ffffffff8110d219>] dispose_list+0x39/0x50
[110926.972861]    [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340
[110926.972863]    [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0
[110926.972865]    [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0
[110926.972866]    [<ffffffff810be400>] kswapd+0x690/0xa10
[110926.972868]    [<ffffffff8105c246>] kthread+0xd6/0xe0
[110926.972870]    [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0
[110926.972871] 
[110926.972872] 
[110926.972872] stack backtrace:
[110926.972874] Pid: 725, comm: kswapd0 Not tainted 3.7.0-rc4 #1
[110926.972875] Call Trace:
[110926.972878]  [<ffffffff8107de28>] print_irq_inversion_bug.part.37+0x1e8/0x1f0
[110926.972880]  [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140
[110926.972883]  [<ffffffff8107e900>] mark_lock+0x190/0x2f0
[110926.972885]  [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0
[110926.972887]  [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00
[110926.972889]  [<ffffffff81220424>] ? xfs_iext_bno_to_ext+0x84/0x160
[110926.972892]  [<ffffffff8120a0e3>] ? xfs_bmbt_get_all+0x13/0x20
[110926.972895]  [<ffffffff81201104>] ? xfs_bmap_search_multi_extents+0xa4/0x110
[110926.972897]  [<ffffffff81080b55>] lock_acquire+0x55/0x70
[110926.972899]  [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50
[110926.972901]  [<ffffffff810f452b>] __sb_start_write+0xab/0x190
[110926.972903]  [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50
[110926.972905]  [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50
[110926.972907]  [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50
[110926.972908]  [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250
[110926.972910]  [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50
[110926.972912]  [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480
[110926.972914]  [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50
[110926.972916]  [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80
[110926.972918]  [<ffffffff8110cb8f>] evict+0xaf/0x1b0
[110926.972920]  [<ffffffff8110d219>] dispose_list+0x39/0x50
[110926.972922]  [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340
[110926.972924]  [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0
[110926.972926]  [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0
[110926.972928]  [<ffffffff810be400>] kswapd+0x690/0xa10
[110926.972930]  [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
[110926.972932]  [<ffffffff810bdd70>] ? shrink_lruvec+0x540/0x540
[110926.972934]  [<ffffffff8105c246>] kthread+0xd6/0xe0
[110926.972936]  [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50
[110926.972938]  [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
[110926.972940]  [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0
[110926.972942]  [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0

[66571.610863] 
[66571.610869] =========================================================
[66571.610870] [ INFO: possible irq lock inversion dependency detected ]
[66571.610873] 3.7.0-rc5 #1 Not tainted
[66571.610874] ---------------------------------------------------------
[66571.610875] cc1/21330 just changed the state of lock:
[66571.610877]  (sb_internal){.+.+.?}, at: [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50
[66571.610885] but this lock took another, RECLAIM_FS-unsafe lock in the past:
[66571.610886]  (&(&ip->i_lock)->mr_lock/1){+.+.+.}
[66571.610886] 
[66571.610886] and interrupts could create inverse lock ordering between them.
[66571.610886] 
[66571.610890] 
[66571.610890] other info that might help us debug this:
[66571.610891]  Possible interrupt unsafe locking scenario:
[66571.610891] 
[66571.610892]        CPU0                    CPU1
[66571.610893]        ----                    ----
[66571.610894]   lock(&(&ip->i_lock)->mr_lock/1);
[66571.610896]                                local_irq_disable();
[66571.610897]                                lock(sb_internal);
[66571.610898]                                lock(&(&ip->i_lock)->mr_lock/1);
[66571.610900]   <Interrupt>
[66571.610901]     lock(sb_internal);
[66571.610902] 
[66571.610902]  *** DEADLOCK ***
[66571.610902] 
[66571.610904] 3 locks held by cc1/21330:
[66571.610905]  #0:  (&mm->mmap_sem){++++++}, at: [<ffffffff81029d8b>] __do_page_fault+0xfb/0x480
[66571.610910]  #1:  (shrinker_rwsem){++++..}, at: [<ffffffff810bbd02>] shrink_slab+0x32/0x1f0
[66571.610915]  #2:  (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a7e>] grab_super_passive+0x3e/0x90
[66571.610921] 
[66571.610921] the shortest dependencies between 2nd lock and 1st lock:
[66571.610927]  -> (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 169649 {
[66571.610931]     HARDIRQ-ON-W at:
[66571.610932]                       [<ffffffff8107f091>] __lock_acquire+0x631/0x1c00
[66571.610935]                       [<ffffffff81080b55>] lock_acquire+0x55/0x70
[66571.610937]                       [<ffffffff8106126a>] down_write_nested+0x4a/0x70
[66571.610941]                       [<ffffffff811e8164>] xfs_ilock+0x84/0xb0
[66571.610944]                       [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0
[66571.610946]                       [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0
[66571.610948]                       [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10
[66571.610950]                       [<ffffffff81100322>] vfs_create+0x72/0xc0
[66571.610953]                       [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80
[66571.610955]                       [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490
[66571.610957]                       [<ffffffff8110183d>] do_filp_open+0x3d/0xa0
[66571.610959]                       [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0
[66571.610962]                       [<ffffffff810f222c>] sys_open+0x1c/0x20
[66571.610964]                       [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b
[66571.610967]     SOFTIRQ-ON-W at:
[66571.610968]                       [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00
[66571.610970]                       [<ffffffff81080b55>] lock_acquire+0x55/0x70
[66571.610972]                       [<ffffffff8106126a>] down_write_nested+0x4a/0x70
[66571.610974]                       [<ffffffff811e8164>] xfs_ilock+0x84/0xb0
[66571.610976]                       [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0
[66571.610977]                       [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0
[66571.610979]                       [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10
[66571.610981]                       [<ffffffff81100322>] vfs_create+0x72/0xc0
[66571.610983]                       [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80
[66571.610985]                       [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490
[66571.610987]                       [<ffffffff8110183d>] do_filp_open+0x3d/0xa0
[66571.610989]                       [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0
[66571.610991]                       [<ffffffff810f222c>] sys_open+0x1c/0x20
[66571.610993]                       [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b
[66571.610995]     RECLAIM_FS-ON-W at:
[66571.610996]                          [<ffffffff8108137e>] mark_held_locks+0x7e/0x130
[66571.610998]                          [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0
[66571.610999]                          [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0
[66571.611002]                          [<ffffffff810dba21>] vm_map_ram+0x271/0x770
[66571.611004]                          [<ffffffff811e12b6>] _xfs_buf_map_pages+0x46/0xe0
[66571.611008]                          [<ffffffff811e21ca>] xfs_buf_get_map+0x8a/0x130
[66571.611009]                          [<ffffffff81233989>] xfs_trans_get_buf_map+0xa9/0xd0
[66571.611011]                          [<ffffffff8121bc2d>] xfs_ialloc_inode_init+0xcd/0x1d0
[66571.611015]                          [<ffffffff8121c16f>] xfs_ialloc_ag_alloc+0x18f/0x500
[66571.611017]                          [<ffffffff8121d955>] xfs_dialloc+0x185/0x2a0
[66571.611019]                          [<ffffffff8121f068>] xfs_ialloc+0x58/0x650
[66571.611021]                          [<ffffffff811f3995>] xfs_dir_ialloc+0x65/0x270
[66571.611023]                          [<ffffffff811f537c>] xfs_create+0x3ac/0x5a0
[66571.611024]                          [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0
[66571.611026]                          [<ffffffff811ecb61>] xfs_vn_mkdir+0x11/0x20
[66571.611028]                          [<ffffffff8110016f>] vfs_mkdir+0x7f/0xd0
[66571.611030]                          [<ffffffff81101b83>] sys_mkdirat+0x43/0x80
[66571.611032]                          [<ffffffff81101bd4>] sys_mkdir+0x14/0x20
[66571.611034]                          [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b
[66571.611036]     INITIAL USE at:
[66571.611037]                      [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00
[66571.611038]                      [<ffffffff81080b55>] lock_acquire+0x55/0x70
[66571.611040]                      [<ffffffff8106126a>] down_write_nested+0x4a/0x70
[66571.611042]                      [<ffffffff811e8164>] xfs_ilock+0x84/0xb0
[66571.611044]                      [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0
[66571.611046]                      [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0
[66571.611047]                      [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10
[66571.611049]                      [<ffffffff81100322>] vfs_create+0x72/0xc0
[66571.611051]                      [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80
[66571.611053]                      [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490
[66571.611055]                      [<ffffffff8110183d>] do_filp_open+0x3d/0xa0
[66571.611057]                      [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0
[66571.611059]                      [<ffffffff810f222c>] sys_open+0x1c/0x20
[66571.611061]                      [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b
[66571.611063]   }
[66571.611064]   ... key      at: [<ffffffff825b4b81>] __key.41357+0x1/0x8
[66571.611066]   ... acquired at:
[66571.611067]    [<ffffffff81080b55>] lock_acquire+0x55/0x70
[66571.611069]    [<ffffffff8106126a>] down_write_nested+0x4a/0x70
[66571.611071]    [<ffffffff811e8164>] xfs_ilock+0x84/0xb0
[66571.611073]    [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0
[66571.611074]    [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0
[66571.611076]    [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10
[66571.611078]    [<ffffffff81100322>] vfs_create+0x72/0xc0
[66571.611080]    [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80
[66571.611082]    [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490
[66571.611084]    [<ffffffff8110183d>] do_filp_open+0x3d/0xa0
[66571.611086]    [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0
[66571.611088]    [<ffffffff810f222c>] sys_open+0x1c/0x20
[66571.611090]    [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b
[66571.611091] 
[66571.611092] -> (sb_internal){.+.+.?} ops: 1341531 {
[66571.611095]    HARDIRQ-ON-R at:
[66571.611096]                     [<ffffffff8107ef6a>] __lock_acquire+0x50a/0x1c00
[66571.611098]                     [<ffffffff81080b55>] lock_acquire+0x55/0x70
[66571.611100]                     [<ffffffff810f451b>] __sb_start_write+0xab/0x190
[66571.611102]                     [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50
[66571.611104]                     [<ffffffff811f5157>] xfs_create+0x187/0x5a0
[66571.611105]                     [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0
[66571.611107]                     [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10
[66571.611109]                     [<ffffffff81100322>] vfs_create+0x72/0xc0
[66571.611111]                     [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80
[66571.611113]                     [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490
[66571.611115]                     [<ffffffff8110183d>] do_filp_open+0x3d/0xa0
[66571.611117]                     [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0
[66571.611119]                     [<ffffffff810f222c>] sys_open+0x1c/0x20
[66571.611121]                     [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b
[66571.611123]    SOFTIRQ-ON-R at:
[66571.611124]                     [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00
[66571.611126]                     [<ffffffff81080b55>] lock_acquire+0x55/0x70
[66571.611128]                     [<ffffffff810f451b>] __sb_start_write+0xab/0x190
[66571.611130]                     [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50
[66571.611132]                     [<ffffffff811f5157>] xfs_create+0x187/0x5a0
[66571.611133]                     [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0
[66571.611135]                     [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10
[66571.611137]                     [<ffffffff81100322>] vfs_create+0x72/0xc0
[66571.611139]                     [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80
[66571.611141]                     [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490
[66571.611143]                     [<ffffffff8110183d>] do_filp_open+0x3d/0xa0
[66571.611145]                     [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0
[66571.611147]                     [<ffffffff810f222c>] sys_open+0x1c/0x20
[66571.611149]                     [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b
[66571.611151]    IN-RECLAIM_FS-R at:
[66571.611152]                        [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00
[66571.611154]                        [<ffffffff81080b55>] lock_acquire+0x55/0x70
[66571.611156]                        [<ffffffff810f451b>] __sb_start_write+0xab/0x190
[66571.611158]                        [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50
[66571.611159]                        [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250
[66571.611161]                        [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480
[66571.611163]                        [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80
[66571.611165]                        [<ffffffff8110cb7f>] evict+0xaf/0x1b0
[66571.611168]                        [<ffffffff8110d209>] dispose_list+0x39/0x50
[66571.611170]                        [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340
[66571.611172]                        [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0
[66571.611173]                        [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0
[66571.611175]                        [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0
[66571.611177]                        [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800
[66571.611179]                        [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0
[66571.611182]                        [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0
[66571.611184]                        [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480
[66571.611186]                        [<ffffffff8102a139>] do_page_fault+0x9/0x10
[66571.611188]                        [<ffffffff816b287f>] page_fault+0x1f/0x30
[66571.611190]    RECLAIM_FS-ON-R at:
[66571.611191]                        [<ffffffff8108137e>] mark_held_locks+0x7e/0x130
[66571.611193]                        [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0
[66571.611195]                        [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0
[66571.611197]                        [<ffffffff811f6d4f>] kmem_zone_alloc+0x5f/0xe0
[66571.611198]                        [<ffffffff811f6de8>] kmem_zone_zalloc+0x18/0x50
[66571.611200]                        [<ffffffff8122b0b2>] _xfs_trans_alloc+0x32/0x90
[66571.611202]                        [<ffffffff8122b148>] xfs_trans_alloc+0x38/0x50
[66571.611204]                        [<ffffffff811f5157>] xfs_create+0x187/0x5a0
[66571.611205]                        [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0
[66571.611207]                        [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10
[66571.611209]                        [<ffffffff81100322>] vfs_create+0x72/0xc0
[66571.611211]                        [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80
[66571.611213]                        [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490
[66571.611215]                        [<ffffffff8110183d>] do_filp_open+0x3d/0xa0
[66571.611217]                        [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0
[66571.611219]                        [<ffffffff810f222c>] sys_open+0x1c/0x20
[66571.611221]                        [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b
[66571.611223]    INITIAL USE at:
[66571.611224]                    [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00
[66571.611225]                    [<ffffffff81080b55>] lock_acquire+0x55/0x70
[66571.611227]                    [<ffffffff810f451b>] __sb_start_write+0xab/0x190
[66571.611229]                    [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50
[66571.611231]                    [<ffffffff811f5157>] xfs_create+0x187/0x5a0
[66571.611232]                    [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0
[66571.611234]                    [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10
[66571.611236]                    [<ffffffff81100322>] vfs_create+0x72/0xc0
[66571.611238]                    [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80
[66571.611240]                    [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490
[66571.611242]                    [<ffffffff8110183d>] do_filp_open+0x3d/0xa0
[66571.611244]                    [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0
[66571.611246]                    [<ffffffff810f222c>] sys_open+0x1c/0x20
[66571.611248]                    [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b
[66571.611250]  }
[66571.611251]  ... key      at: [<ffffffff81c34e40>] xfs_fs_type+0x60/0x80
[66571.611254]  ... acquired at:
[66571.611254]    [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140
[66571.611256]    [<ffffffff8107e900>] mark_lock+0x190/0x2f0
[66571.611258]    [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00
[66571.611260]    [<ffffffff81080b55>] lock_acquire+0x55/0x70
[66571.611261]    [<ffffffff810f451b>] __sb_start_write+0xab/0x190
[66571.611263]    [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50
[66571.611265]    [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250
[66571.611266]    [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480
[66571.611268]    [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80
[66571.611270]    [<ffffffff8110cb7f>] evict+0xaf/0x1b0
[66571.611271]    [<ffffffff8110d209>] dispose_list+0x39/0x50
[66571.611273]    [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340
[66571.611275]    [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0
[66571.611277]    [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0
[66571.611278]    [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0
[66571.611280]    [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800
[66571.611282]    [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0
[66571.611284]    [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0
[66571.611286]    [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480
[66571.611287]    [<ffffffff8102a139>] do_page_fault+0x9/0x10
[66571.611289]    [<ffffffff816b287f>] page_fault+0x1f/0x30
[66571.611291] 
[66571.611292] 
[66571.611292] stack backtrace:
[66571.611294] Pid: 21330, comm: cc1 Not tainted 3.7.0-rc5 #1
[66571.611295] Call Trace:
[66571.611298]  [<ffffffff8107de28>] print_irq_inversion_bug.part.37+0x1e8/0x1f0
[66571.611300]  [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140
[66571.611303]  [<ffffffff8107e900>] mark_lock+0x190/0x2f0
[66571.611306]  [<ffffffff8150406e>] ? dm_request+0x2e/0x2a0
[66571.611308]  [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0
[66571.611310]  [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00
[66571.611313]  [<ffffffff812202f4>] ? xfs_iext_bno_to_ext+0x84/0x160
[66571.611316]  [<ffffffff8120a023>] ? xfs_bmbt_get_all+0x13/0x20
[66571.611318]  [<ffffffff81200fb4>] ? xfs_bmap_search_multi_extents+0xa4/0x110
[66571.611320]  [<ffffffff81080b55>] lock_acquire+0x55/0x70
[66571.611322]  [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50
[66571.611324]  [<ffffffff810f451b>] __sb_start_write+0xab/0x190
[66571.611326]  [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50
[66571.611328]  [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50
[66571.611330]  [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50
[66571.611332]  [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250
[66571.611334]  [<ffffffff816b204b>] ? _raw_spin_unlock_irq+0x2b/0x50
[66571.611336]  [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480
[66571.611337]  [<ffffffff816b204b>] ? _raw_spin_unlock_irq+0x2b/0x50
[66571.611340]  [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80
[66571.611342]  [<ffffffff8110cb7f>] evict+0xaf/0x1b0
[66571.611344]  [<ffffffff8110d209>] dispose_list+0x39/0x50
[66571.611346]  [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340
[66571.611347]  [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0
[66571.611349]  [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0
[66571.611352]  [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0
[66571.611354]  [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800
[66571.611356]  [<ffffffff810b9e40>] ? lru_deactivate_fn+0x1c0/0x1c0
[66571.611358]  [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0
[66571.611360]  [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0
[66571.611363]  [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480
[66571.611366]  [<ffffffff8129c7ad>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[66571.611368]  [<ffffffff8102a139>] do_page_fault+0x9/0x10
[66571.611370]  [<ffffffff816b287f>] page_fault+0x1f/0x30