Re: Hang in XFS reclaim on 3.7.0-rc3

From: Torsten Kaiser
Date: Tue Nov 20 2012 - 14:45:00 EST


On Tue, Nov 20, 2012 at 12:53 AM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
> [<ffffffff8108137e>] mark_held_locks+0x7e/0x130
> [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0
> [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0
> [<ffffffff810dba31>] vm_map_ram+0x271/0x770
> [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0
> [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130
> [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0
> [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0
>
> We shouldn't be mapping buffers there, there's a patch below to fix
> this. It's probably the source of this report, even though I cannot
> lockdep seems to be off with the fairies...

That patch seems to break my system.
After it started to swap, because I was compiling seamonkey (firefox
turned into the full navigator suite) on a tmpfs, several processes
got stuck and triggered the hung-task-check.
As a kswapd, xfsaild/md4 and flush-9:4 also got stuck, not even a
shutdown worked.

The attached log first contains the hung-task-notices, then the output
from SysRq+W.

After the shutdown got stuck trying to turn of swap, I first tries
SysRq+S, but did not get a 'Done' and on SysRq+U lockdep complained
about an lock imbalance wrt. sb_writer. SysRq+O also did no longer
work, only SysRq+B.

I don't know which one got stuck first, but I'm somewhat suspicious of
the plasma-desktop and the sshd that SysRq+W reported stuck in xfs
reclaim, even if these processes did never trigger the hung task
check.

Torsten

> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@xxxxxxxxxxxxx
>
> xfs: inode allocation should use unmapped buffers.
>
> From: Dave Chinner <dchinner@xxxxxxxxxx>
>
> Inode buffers do not need to be mapped as inodes are read or written
> directly from/to the pages underlying the buffer. This fixes a
> regression introduced by commit 611c994 ("xfs: make XBF_MAPPED the
> default behaviour").
>
> Signed-off-by: Dave Chinner <dchinner@xxxxxxxxxx>
> ---
> fs/xfs/xfs_ialloc.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/fs/xfs/xfs_ialloc.c b/fs/xfs/xfs_ialloc.c
> index 2d6495e..a815412 100644
> --- a/fs/xfs/xfs_ialloc.c
> +++ b/fs/xfs/xfs_ialloc.c
> @@ -200,7 +200,8 @@ xfs_ialloc_inode_init(
> */
> d = XFS_AGB_TO_DADDR(mp, agno, agbno + (j * blks_per_cluster));
> fbuf = xfs_trans_get_buf(tp, mp->m_ddev_targp, d,
> - mp->m_bsize * blks_per_cluster, 0);
> + mp->m_bsize * blks_per_cluster,
> + XBF_UNMAPPED);
> if (!fbuf)
> return ENOMEM;
> /*
Nov 20 19:24:55 thoregon acpid: client 2991[0:0] has disconnected
Nov 20 19:25:07 thoregon login[3002]: pam_unix(login:session): session opened for user root by LOGIN(uid=0)
Nov 20 19:25:07 thoregon login[27176]: ROOT LOGIN on '/dev/tty1'
Nov 20 19:27:23 thoregon kernel: [ 2160.595990] INFO: task kswapd0:725 blocked for more than 120 seconds.
Nov 20 19:27:23 thoregon kernel: [ 2160.595998] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 19:27:23 thoregon kernel: [ 2160.596004] kswapd0 D 0000000000000001 0 725 2 0x00000000
Nov 20 19:27:23 thoregon kernel: [ 2160.596016] ffff8803280d13f8 0000000000000046 ffff880329a0ab80 ffff8803280d1fd8
Nov 20 19:27:23 thoregon kernel: [ 2160.596030] ffff8803280d1fd8 ffff8803280d1fd8 ffff880046b7c880 ffff880329a0ab80
Nov 20 19:27:23 thoregon kernel: [ 2160.596041] ffff8803280d1408 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff
Nov 20 19:27:23 thoregon kernel: [ 2160.596052] Call Trace:
Nov 20 19:27:23 thoregon kernel: [ 2160.596070] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.596080] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80
Nov 20 19:27:23 thoregon kernel: [ 2160.596091] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.596100] [<ffffffff81500753>] bitmap_unplug+0x173/0x180
Nov 20 19:27:23 thoregon kernel: [ 2160.596109] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:27:23 thoregon kernel: [ 2160.596118] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110
Nov 20 19:27:23 thoregon kernel: [ 2160.596130] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240
Nov 20 19:27:23 thoregon kernel: [ 2160.596138] [<ffffffff816b24ca>] ? _raw_spin_unlock_irqrestore+0x3a/0x70
Nov 20 19:27:23 thoregon kernel: [ 2160.596148] [<ffffffff816b15c3>] io_schedule_timeout+0x83/0xf0
Nov 20 19:27:23 thoregon kernel: [ 2160.596156] [<ffffffff810815cd>] ? trace_hardirqs_on+0xd/0x10
Nov 20 19:27:23 thoregon kernel: [ 2160.596166] [<ffffffff810b0e1d>] mempool_alloc+0x12d/0x160
Nov 20 19:27:23 thoregon kernel: [ 2160.596174] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.596184] [<ffffffff811263da>] bvec_alloc_bs+0xda/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.596192] [<ffffffff811264ea>] bio_alloc_bioset+0xea/0x110
Nov 20 19:27:23 thoregon kernel: [ 2160.596201] [<ffffffff81126656>] bio_clone_bioset+0x16/0x40
Nov 20 19:27:23 thoregon kernel: [ 2160.596208] [<ffffffff814f471a>] bio_clone_mddev+0x1a/0x30
Nov 20 19:27:23 thoregon kernel: [ 2160.596217] [<ffffffff814edbb1>] make_request+0x551/0xde0
Nov 20 19:27:23 thoregon kernel: [ 2160.596224] [<ffffffff814ed9e0>] ? make_request+0x380/0xde0
Nov 20 19:27:23 thoregon kernel: [ 2160.596233] [<ffffffff810b0b80>] ? mempool_alloc_slab+0x10/0x20
Nov 20 19:27:23 thoregon kernel: [ 2160.596242] [<ffffffff814f80bb>] md_make_request+0x21b/0x4d0
Nov 20 19:27:23 thoregon kernel: [ 2160.596249] [<ffffffff814f7ef0>] ? md_make_request+0x50/0x4d0
Nov 20 19:27:23 thoregon kernel: [ 2160.596258] [<ffffffff81276e52>] generic_make_request+0xc2/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.596267] [<ffffffff81276ef5>] submit_bio+0x65/0x110
Nov 20 19:27:23 thoregon kernel: [ 2160.596278] [<ffffffff811e07bf>] xfs_submit_ioend_bio.isra.21+0x2f/0x40
Nov 20 19:27:23 thoregon kernel: [ 2160.596286] [<ffffffff811e088e>] xfs_submit_ioend+0xbe/0x110
Nov 20 19:27:23 thoregon kernel: [ 2160.596295] [<ffffffff811e0c91>] xfs_vm_writepage+0x3b1/0x540
Nov 20 19:27:23 thoregon kernel: [ 2160.596305] [<ffffffff810bcd84>] shrink_page_list+0x564/0x890
Nov 20 19:27:23 thoregon kernel: [ 2160.596315] [<ffffffff810bd637>] shrink_inactive_list+0x1d7/0x310
Nov 20 19:27:23 thoregon kernel: [ 2160.596325] [<ffffffff810bdb9d>] shrink_lruvec+0x42d/0x530
Nov 20 19:27:23 thoregon kernel: [ 2160.596334] [<ffffffff810be323>] kswapd+0x683/0xa20
Nov 20 19:27:23 thoregon kernel: [ 2160.596344] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.596352] [<ffffffff810bdca0>] ? shrink_lruvec+0x530/0x530
Nov 20 19:27:23 thoregon kernel: [ 2160.596360] [<ffffffff8105c246>] kthread+0xd6/0xe0
Nov 20 19:27:23 thoregon kernel: [ 2160.596367] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.596376] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:27:23 thoregon kernel: [ 2160.596384] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0
Nov 20 19:27:23 thoregon kernel: [ 2160.596392] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:27:23 thoregon kernel: [ 2160.596397] no locks held by kswapd0/725.
Nov 20 19:27:23 thoregon kernel: [ 2160.596424] INFO: task xfsaild/md4:1742 blocked for more than 120 seconds.
Nov 20 19:27:23 thoregon kernel: [ 2160.596428] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 19:27:23 thoregon kernel: [ 2160.596432] xfsaild/md4 D 0000000000000003 0 1742 2 0x00000000
Nov 20 19:27:23 thoregon kernel: [ 2160.596441] ffff88032438bb68 0000000000000046 ffff880329965700 ffff88032438bfd8
Nov 20 19:27:23 thoregon kernel: [ 2160.596452] ffff88032438bfd8 ffff88032438bfd8 ffff88032827e580 ffff880329965700
Nov 20 19:27:23 thoregon kernel: [ 2160.596462] ffff88032438bb78 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff
Nov 20 19:27:23 thoregon kernel: [ 2160.596473] Call Trace:
Nov 20 19:27:23 thoregon kernel: [ 2160.596483] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.596490] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80
Nov 20 19:27:23 thoregon kernel: [ 2160.596498] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.596507] [<ffffffff81500753>] bitmap_unplug+0x173/0x180
Nov 20 19:27:23 thoregon kernel: [ 2160.596516] [<ffffffff81278c13>] ? blk_finish_plug+0x13/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.596524] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110
Nov 20 19:27:23 thoregon kernel: [ 2160.596533] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240
Nov 20 19:27:23 thoregon kernel: [ 2160.596542] [<ffffffff81278c13>] blk_finish_plug+0x13/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.596550] [<ffffffff811e296a>] __xfs_buf_delwri_submit+0x1ca/0x1e0
Nov 20 19:27:23 thoregon kernel: [ 2160.596558] [<ffffffff811e2ffb>] xfs_buf_delwri_submit_nowait+0x1b/0x20
Nov 20 19:27:23 thoregon kernel: [ 2160.596566] [<ffffffff81233066>] xfsaild+0x226/0x4c0
Nov 20 19:27:23 thoregon kernel: [ 2160.596575] [<ffffffff81065dfa>] ? finish_task_switch+0x3a/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.596583] [<ffffffff81232e40>] ? xfs_trans_ail_cursor_first+0xa0/0xa0
Nov 20 19:27:23 thoregon kernel: [ 2160.596591] [<ffffffff8105c246>] kthread+0xd6/0xe0
Nov 20 19:27:23 thoregon kernel: [ 2160.596598] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.596607] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:27:23 thoregon kernel: [ 2160.596614] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0
Nov 20 19:27:23 thoregon kernel: [ 2160.596622] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:27:23 thoregon kernel: [ 2160.596627] no locks held by xfsaild/md4/1742.
Nov 20 19:27:23 thoregon kernel: [ 2160.596709] INFO: task flush-9:4:30365 blocked for more than 120 seconds.
Nov 20 19:27:23 thoregon kernel: [ 2160.596712] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 19:27:23 thoregon kernel: [ 2160.596716] flush-9:4 D 0000000000000001 0 30365 2 0x00000000
Nov 20 19:27:23 thoregon kernel: [ 2160.596725] ffff8801d8ce1978 0000000000000046 ffff880328238e80 ffff8801d8ce1fd8
Nov 20 19:27:23 thoregon kernel: [ 2160.596735] ffff8801d8ce1fd8 ffff8801d8ce1fd8 ffff8803032cc880 ffff880328238e80
Nov 20 19:27:23 thoregon kernel: [ 2160.596745] ffff8801d8ce1988 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff
Nov 20 19:27:23 thoregon kernel: [ 2160.596756] Call Trace:
Nov 20 19:27:23 thoregon kernel: [ 2160.596765] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.596773] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80
Nov 20 19:27:23 thoregon kernel: [ 2160.596781] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.596788] [<ffffffff81500753>] bitmap_unplug+0x173/0x180
Nov 20 19:27:23 thoregon kernel: [ 2160.596796] [<ffffffff810b6acf>] ? write_cache_pages+0x12f/0x420
Nov 20 19:27:23 thoregon kernel: [ 2160.596804] [<ffffffff810b6700>] ? set_page_dirty_lock+0x60/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.596812] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110
Nov 20 19:27:23 thoregon kernel: [ 2160.596821] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240
Nov 20 19:27:23 thoregon kernel: [ 2160.596830] [<ffffffff81278c13>] blk_finish_plug+0x13/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.596838] [<ffffffff810b6e0a>] generic_writepages+0x4a/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.596847] [<ffffffff811df898>] xfs_vm_writepages+0x48/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.596856] [<ffffffff8111b144>] ? writeback_sb_inodes+0x174/0x410
Nov 20 19:27:23 thoregon kernel: [ 2160.596864] [<ffffffff810b84be>] do_writepages+0x1e/0x30
Nov 20 19:27:23 thoregon kernel: [ 2160.596873] [<ffffffff8111a1ee>] __writeback_single_inode+0x3e/0x120
Nov 20 19:27:23 thoregon kernel: [ 2160.596882] [<ffffffff8111b234>] writeback_sb_inodes+0x264/0x410
Nov 20 19:27:23 thoregon kernel: [ 2160.596891] [<ffffffff8111b476>] __writeback_inodes_wb+0x96/0xc0
Nov 20 19:27:23 thoregon kernel: [ 2160.596900] [<ffffffff8111b673>] wb_writeback+0x1d3/0x1e0
Nov 20 19:27:23 thoregon kernel: [ 2160.596908] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:27:23 thoregon kernel: [ 2160.596916] [<ffffffff8111bb1c>] ? wb_do_writeback+0x7c/0x140
Nov 20 19:27:23 thoregon kernel: [ 2160.596925] [<ffffffff8111bb40>] wb_do_writeback+0xa0/0x140
Nov 20 19:27:23 thoregon kernel: [ 2160.596935] [<ffffffff8111bc52>] bdi_writeback_thread+0x72/0x150
Nov 20 19:27:23 thoregon kernel: [ 2160.596944] [<ffffffff8111bbe0>] ? wb_do_writeback+0x140/0x140
Nov 20 19:27:23 thoregon kernel: [ 2160.596951] [<ffffffff8105c246>] kthread+0xd6/0xe0
Nov 20 19:27:23 thoregon kernel: [ 2160.596958] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.596967] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:27:23 thoregon kernel: [ 2160.596975] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0
Nov 20 19:27:23 thoregon kernel: [ 2160.596983] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:27:23 thoregon kernel: [ 2160.596988] 1 lock held by flush-9:4/30365:
Nov 20 19:27:23 thoregon kernel: [ 2160.596991] #0: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f59de>] grab_super_passive+0x3e/0x90
Nov 20 19:27:23 thoregon kernel: [ 2160.597019] INFO: task x86_64-pc-linux:26516 blocked for more than 120 seconds.
Nov 20 19:27:23 thoregon kernel: [ 2160.597023] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 19:27:23 thoregon kernel: [ 2160.597027] x86_64-pc-linux D 0000000000000000 0 26516 26114 0x00000000
Nov 20 19:27:23 thoregon kernel: [ 2160.597035] ffff8800468dddf8 0000000000000046 ffff880329303a00 ffff8800468ddfd8
Nov 20 19:27:23 thoregon kernel: [ 2160.597045] ffff8800468ddfd8 ffff8800468ddfd8 ffff88032989c880 ffff880329303a00
Nov 20 19:27:23 thoregon kernel: [ 2160.597056] 0000000000000246 ffff8800468dc000 ffff8801b1a1bc68 0000000002405150
Nov 20 19:27:23 thoregon kernel: [ 2160.597066] Call Trace:
Nov 20 19:27:23 thoregon kernel: [ 2160.597076] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.597085] [<ffffffff816b1501>] schedule_preempt_disabled+0x21/0x30
Nov 20 19:27:23 thoregon kernel: [ 2160.597094] [<ffffffff816af3a5>] mutex_lock_nested+0x165/0x330
Nov 20 19:27:23 thoregon kernel: [ 2160.597103] [<ffffffff810ff5e3>] ? do_unlinkat+0xa3/0x1b0
Nov 20 19:27:23 thoregon kernel: [ 2160.597112] [<ffffffff810fefae>] ? filename_lookup.isra.57+0x2e/0x80
Nov 20 19:27:23 thoregon kernel: [ 2160.597121] [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0
Nov 20 19:27:23 thoregon kernel: [ 2160.597129] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:27:23 thoregon kernel: [ 2160.597137] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Nov 20 19:27:23 thoregon kernel: [ 2160.597146] [<ffffffff81101bb1>] sys_unlink+0x11/0x20
Nov 20 19:27:23 thoregon kernel: [ 2160.597154] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:27:23 thoregon kernel: [ 2160.597159] 2 locks held by x86_64-pc-linux/26516:
Nov 20 19:27:23 thoregon kernel: [ 2160.597162] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.597178] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0
Nov 20 19:27:23 thoregon kernel: [ 2160.597198] INFO: task x86_64-pc-linux:26523 blocked for more than 120 seconds.
Nov 20 19:27:23 thoregon kernel: [ 2160.597201] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 19:27:23 thoregon kernel: [ 2160.597205] x86_64-pc-linux D 0000000000000000 0 26523 26114 0x00000000
Nov 20 19:27:23 thoregon kernel: [ 2160.597212] ffff8800c0def858 0000000000000046 ffff88032986c880 ffff8800c0deffd8
Nov 20 19:27:23 thoregon kernel: [ 2160.597223] ffff8800c0deffd8 ffff8800c0deffd8 ffff88032989e580 ffff88032986c880
Nov 20 19:27:23 thoregon kernel: [ 2160.597233] ffff8800c0def928 ffff8801c57e3a30 7fffffffffffffff ffff88032986c880
Nov 20 19:27:23 thoregon kernel: [ 2160.597243] Call Trace:
Nov 20 19:27:23 thoregon kernel: [ 2160.597253] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.597261] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0
Nov 20 19:27:23 thoregon kernel: [ 2160.597269] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.597277] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:27:23 thoregon kernel: [ 2160.597286] [<ffffffff816b057e>] __down_common+0x97/0xe8
Nov 20 19:27:23 thoregon kernel: [ 2160.597294] [<ffffffff816b062e>] __down+0x18/0x1a
Nov 20 19:27:23 thoregon kernel: [ 2160.597301] [<ffffffff8106245b>] down+0x3b/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.597308] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.597315] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200
Nov 20 19:27:23 thoregon kernel: [ 2160.597323] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130
Nov 20 19:27:23 thoregon kernel: [ 2160.597331] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0
Nov 20 19:27:23 thoregon kernel: [ 2160.597339] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460
Nov 20 19:27:23 thoregon kernel: [ 2160.597346] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90
Nov 20 19:27:23 thoregon kernel: [ 2160.597353] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0
Nov 20 19:27:23 thoregon kernel: [ 2160.597361] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180
Nov 20 19:27:23 thoregon kernel: [ 2160.597371] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0
Nov 20 19:27:23 thoregon kernel: [ 2160.597380] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330
Nov 20 19:27:23 thoregon kernel: [ 2160.597389] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0
Nov 20 19:27:23 thoregon kernel: [ 2160.597398] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150
Nov 20 19:27:23 thoregon kernel: [ 2160.597406] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370
Nov 20 19:27:23 thoregon kernel: [ 2160.597416] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330
Nov 20 19:27:23 thoregon kernel: [ 2160.597424] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.597432] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90
Nov 20 19:27:23 thoregon kernel: [ 2160.597441] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.597449] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0
Nov 20 19:27:23 thoregon kernel: [ 2160.597458] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:27:23 thoregon kernel: [ 2160.597466] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Nov 20 19:27:23 thoregon kernel: [ 2160.597475] [<ffffffff81101bb1>] sys_unlink+0x11/0x20
Nov 20 19:27:23 thoregon kernel: [ 2160.597482] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:27:23 thoregon kernel: [ 2160.597487] 6 locks held by x86_64-pc-linux/26523:
Nov 20 19:27:23 thoregon kernel: [ 2160.597490] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.597505] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0
Nov 20 19:27:23 thoregon kernel: [ 2160.597522] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.597538] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.597552] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0
Nov 20 19:27:23 thoregon kernel: [ 2160.597567] #5: (&(&ip->i_lock)->mr_lock/5){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0
Nov 20 19:27:23 thoregon kernel: [ 2160.597584] INFO: task x86_64-pc-linux:26875 blocked for more than 120 seconds.
Nov 20 19:27:23 thoregon kernel: [ 2160.597587] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 19:27:23 thoregon kernel: [ 2160.597591] x86_64-pc-linux D 0000000000000000 0 26875 26114 0x00000000
Nov 20 19:27:23 thoregon kernel: [ 2160.597598] ffff8800c0c5ba08 0000000000000046 ffff8802f0594880 ffff8800c0c5bfd8
Nov 20 19:27:23 thoregon kernel: [ 2160.597609] ffff8800c0c5bfd8 ffff8800c0c5bfd8 ffff88032989c880 ffff8802f0594880
Nov 20 19:27:23 thoregon kernel: [ 2160.597619] 0000000000000250 ffff880205a7aa30 7fffffffffffffff ffff8802f0594880
Nov 20 19:27:23 thoregon kernel: [ 2160.597630] Call Trace:
Nov 20 19:27:23 thoregon kernel: [ 2160.597639] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.597648] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0
Nov 20 19:27:23 thoregon kernel: [ 2160.597655] [<ffffffff811f70b4>] ? kmem_zone_zalloc+0x34/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.597662] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.597670] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:27:23 thoregon kernel: [ 2160.597679] [<ffffffff816b057e>] __down_common+0x97/0xe8
Nov 20 19:27:23 thoregon kernel: [ 2160.597688] [<ffffffff816b062e>] __down+0x18/0x1a
Nov 20 19:27:23 thoregon kernel: [ 2160.597695] [<ffffffff8106245b>] down+0x3b/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.597702] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.597708] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200
Nov 20 19:27:23 thoregon kernel: [ 2160.597716] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130
Nov 20 19:27:23 thoregon kernel: [ 2160.597723] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90
Nov 20 19:27:23 thoregon kernel: [ 2160.597730] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0
Nov 20 19:27:23 thoregon kernel: [ 2160.597739] [<ffffffff8121ce6f>] xfs_read_agi+0x7f/0x110
Nov 20 19:27:23 thoregon kernel: [ 2160.597748] [<ffffffff8121fb55>] xfs_iunlink+0x45/0x150
Nov 20 19:27:23 thoregon kernel: [ 2160.597757] [<ffffffff81041fc1>] ? current_fs_time+0x11/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.597764] [<ffffffff81234352>] ? xfs_trans_ichgtime+0x22/0xa0
Nov 20 19:27:23 thoregon kernel: [ 2160.597772] [<ffffffff811f3ea0>] xfs_droplink+0x50/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.597778] [<ffffffff811f5d77>] xfs_remove+0x287/0x370
Nov 20 19:27:23 thoregon kernel: [ 2160.597788] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330
Nov 20 19:27:23 thoregon kernel: [ 2160.597796] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.597804] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90
Nov 20 19:27:23 thoregon kernel: [ 2160.597813] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.597821] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0
Nov 20 19:27:23 thoregon kernel: [ 2160.597830] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:27:23 thoregon kernel: [ 2160.597837] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Nov 20 19:27:23 thoregon kernel: [ 2160.597846] [<ffffffff81101bb1>] sys_unlink+0x11/0x20
Nov 20 19:27:23 thoregon kernel: [ 2160.597854] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:27:23 thoregon kernel: [ 2160.597859] 6 locks held by x86_64-pc-linux/26875:
Nov 20 19:27:23 thoregon kernel: [ 2160.597862] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.597877] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0
Nov 20 19:27:23 thoregon kernel: [ 2160.597894] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.597909] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.597923] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0
Nov 20 19:27:23 thoregon kernel: [ 2160.597937] #5: (&(&ip->i_lock)->mr_lock/5){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0
Nov 20 19:27:23 thoregon kernel: [ 2160.597955] INFO: task x86_64-pc-linux:26897 blocked for more than 120 seconds.
Nov 20 19:27:23 thoregon kernel: [ 2160.597958] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 19:27:23 thoregon kernel: [ 2160.597961] x86_64-pc-linux D 0000000000000001 0 26897 26114 0x00000000
Nov 20 19:27:23 thoregon kernel: [ 2160.597969] ffff880101959858 0000000000000046 ffff88032934ab80 ffff880101959fd8
Nov 20 19:27:23 thoregon kernel: [ 2160.597980] ffff880101959fd8 ffff880101959fd8 ffff880162a3e580 ffff88032934ab80
Nov 20 19:27:23 thoregon kernel: [ 2160.597990] ffff880101959928 ffff8801ca545030 7fffffffffffffff ffff88032934ab80
Nov 20 19:27:23 thoregon kernel: [ 2160.598000] Call Trace:
Nov 20 19:27:23 thoregon kernel: [ 2160.598010] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.598018] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0
Nov 20 19:27:23 thoregon kernel: [ 2160.598026] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.598034] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:27:23 thoregon kernel: [ 2160.598043] [<ffffffff816b057e>] __down_common+0x97/0xe8
Nov 20 19:27:23 thoregon kernel: [ 2160.598052] [<ffffffff816b062e>] __down+0x18/0x1a
Nov 20 19:27:23 thoregon kernel: [ 2160.598058] [<ffffffff8106245b>] down+0x3b/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.598065] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.598072] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200
Nov 20 19:27:23 thoregon kernel: [ 2160.598079] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130
Nov 20 19:27:23 thoregon kernel: [ 2160.598087] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0
Nov 20 19:27:23 thoregon kernel: [ 2160.598094] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460
Nov 20 19:27:23 thoregon kernel: [ 2160.598102] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90
Nov 20 19:27:23 thoregon kernel: [ 2160.598109] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0
Nov 20 19:27:23 thoregon kernel: [ 2160.598117] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180
Nov 20 19:27:23 thoregon kernel: [ 2160.598126] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0
Nov 20 19:27:23 thoregon kernel: [ 2160.598135] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330
Nov 20 19:27:23 thoregon kernel: [ 2160.598144] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0
Nov 20 19:27:23 thoregon kernel: [ 2160.598152] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150
Nov 20 19:27:23 thoregon kernel: [ 2160.598160] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370
Nov 20 19:27:23 thoregon kernel: [ 2160.598170] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330
Nov 20 19:27:23 thoregon kernel: [ 2160.598178] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.598186] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90
Nov 20 19:27:23 thoregon kernel: [ 2160.598195] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.598203] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0
Nov 20 19:27:23 thoregon kernel: [ 2160.598211] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:27:23 thoregon kernel: [ 2160.598219] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Nov 20 19:27:23 thoregon kernel: [ 2160.598228] [<ffffffff81101bb1>] sys_unlink+0x11/0x20
Nov 20 19:27:23 thoregon kernel: [ 2160.598236] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:27:23 thoregon kernel: [ 2160.598241] 6 locks held by x86_64-pc-linux/26897:
Nov 20 19:27:23 thoregon kernel: [ 2160.598244] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.598258] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0
Nov 20 19:27:23 thoregon kernel: [ 2160.598275] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.598290] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.598304] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0
Nov 20 19:27:23 thoregon kernel: [ 2160.598318] #5: (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffff811e854c>] xfs_ilock_nowait+0xbc/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.598334] INFO: task x86_64-pc-linux:26904 blocked for more than 120 seconds.
Nov 20 19:27:23 thoregon kernel: [ 2160.598337] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 19:27:23 thoregon kernel: [ 2160.598341] x86_64-pc-linux D 0000000000000001 0 26904 26114 0x00000000
Nov 20 19:27:23 thoregon kernel: [ 2160.598348] ffff880066bff858 0000000000000046 ffff880329301d00 ffff880066bfffd8
Nov 20 19:27:23 thoregon kernel: [ 2160.598359] ffff880066bfffd8 ffff880066bfffd8 ffff88013a7ce580 ffff880329301d00
Nov 20 19:27:23 thoregon kernel: [ 2160.598369] ffff880066bff928 ffff8801c572c830 7fffffffffffffff ffff880329301d00
Nov 20 19:27:23 thoregon kernel: [ 2160.598380] Call Trace:
Nov 20 19:27:23 thoregon kernel: [ 2160.598389] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.598398] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0
Nov 20 19:27:23 thoregon kernel: [ 2160.598405] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.598413] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:27:23 thoregon kernel: [ 2160.598422] [<ffffffff816b057e>] __down_common+0x97/0xe8
Nov 20 19:27:23 thoregon kernel: [ 2160.598431] [<ffffffff816b062e>] __down+0x18/0x1a
Nov 20 19:27:23 thoregon kernel: [ 2160.598437] [<ffffffff8106245b>] down+0x3b/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.598444] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.598451] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200
Nov 20 19:27:23 thoregon kernel: [ 2160.598458] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130
Nov 20 19:27:23 thoregon kernel: [ 2160.598466] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0
Nov 20 19:27:23 thoregon kernel: [ 2160.598474] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460
Nov 20 19:27:23 thoregon kernel: [ 2160.598481] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90
Nov 20 19:27:23 thoregon kernel: [ 2160.598488] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0
Nov 20 19:27:23 thoregon kernel: [ 2160.598496] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180
Nov 20 19:27:23 thoregon kernel: [ 2160.598505] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0
Nov 20 19:27:23 thoregon kernel: [ 2160.598515] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330
Nov 20 19:27:23 thoregon kernel: [ 2160.598523] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0
Nov 20 19:27:23 thoregon kernel: [ 2160.598532] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150
Nov 20 19:27:23 thoregon kernel: [ 2160.598540] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370
Nov 20 19:27:23 thoregon kernel: [ 2160.598549] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330
Nov 20 19:27:23 thoregon kernel: [ 2160.598557] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.598565] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90
Nov 20 19:27:23 thoregon kernel: [ 2160.598574] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.598582] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0
Nov 20 19:27:23 thoregon kernel: [ 2160.598591] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:27:23 thoregon kernel: [ 2160.598598] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Nov 20 19:27:23 thoregon kernel: [ 2160.598607] [<ffffffff81101bb1>] sys_unlink+0x11/0x20
Nov 20 19:27:23 thoregon kernel: [ 2160.598615] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:27:23 thoregon kernel: [ 2160.598620] 6 locks held by x86_64-pc-linux/26904:
Nov 20 19:27:23 thoregon kernel: [ 2160.598623] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.598637] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0
Nov 20 19:27:23 thoregon kernel: [ 2160.598654] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.598669] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.598683] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0
Nov 20 19:27:23 thoregon kernel: [ 2160.598697] #5: (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffff811e854c>] xfs_ilock_nowait+0xbc/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.598713] INFO: task x86_64-pc-linux:26907 blocked for more than 120 seconds.
Nov 20 19:27:23 thoregon kernel: [ 2160.598716] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 19:27:23 thoregon kernel: [ 2160.598720] x86_64-pc-linux D 0000000000000000 0 26907 26114 0x00000000
Nov 20 19:27:23 thoregon kernel: [ 2160.598727] ffff8800c0eed858 0000000000000046 ffff8802a7be6580 ffff8800c0eedfd8
Nov 20 19:27:23 thoregon kernel: [ 2160.598738] ffff8800c0eedfd8 ffff8800c0eedfd8 ffff88032989e580 ffff8802a7be6580
Nov 20 19:27:23 thoregon kernel: [ 2160.598748] ffff8800c0eed928 ffff8801c55c7630 7fffffffffffffff ffff8802a7be6580
Nov 20 19:27:23 thoregon kernel: [ 2160.598758] Call Trace:
Nov 20 19:27:23 thoregon kernel: [ 2160.598768] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.598776] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0
Nov 20 19:27:23 thoregon kernel: [ 2160.598784] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.598792] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:27:23 thoregon kernel: [ 2160.598801] [<ffffffff816b057e>] __down_common+0x97/0xe8
Nov 20 19:27:23 thoregon kernel: [ 2160.598809] [<ffffffff816b062e>] __down+0x18/0x1a
Nov 20 19:27:23 thoregon kernel: [ 2160.598816] [<ffffffff8106245b>] down+0x3b/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.598823] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60
Nov 20 19:27:23 thoregon kernel: [ 2160.598830] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200
Nov 20 19:27:23 thoregon kernel: [ 2160.598837] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130
Nov 20 19:27:23 thoregon kernel: [ 2160.598844] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0
Nov 20 19:27:23 thoregon kernel: [ 2160.598852] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460
Nov 20 19:27:23 thoregon kernel: [ 2160.598859] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90
Nov 20 19:27:23 thoregon kernel: [ 2160.598867] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0
Nov 20 19:27:23 thoregon kernel: [ 2160.598874] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180
Nov 20 19:27:23 thoregon kernel: [ 2160.598884] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0
Nov 20 19:27:23 thoregon kernel: [ 2160.598893] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330
Nov 20 19:27:23 thoregon kernel: [ 2160.598901] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0
Nov 20 19:27:23 thoregon kernel: [ 2160.598910] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150
Nov 20 19:27:23 thoregon kernel: [ 2160.598918] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370
Nov 20 19:27:23 thoregon kernel: [ 2160.598927] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330
Nov 20 19:27:23 thoregon kernel: [ 2160.598935] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.598944] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90
Nov 20 19:27:23 thoregon kernel: [ 2160.598952] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.598961] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0
Nov 20 19:27:23 thoregon kernel: [ 2160.598969] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:27:23 thoregon kernel: [ 2160.598977] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Nov 20 19:27:23 thoregon kernel: [ 2160.598986] [<ffffffff81101bb1>] sys_unlink+0x11/0x20
Nov 20 19:27:23 thoregon kernel: [ 2160.598993] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:27:23 thoregon kernel: [ 2160.598998] 6 locks held by x86_64-pc-linux/26907:
Nov 20 19:27:23 thoregon kernel: [ 2160.599001] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.599016] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0
Nov 20 19:27:23 thoregon kernel: [ 2160.599033] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100
Nov 20 19:27:23 thoregon kernel: [ 2160.599048] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50
Nov 20 19:27:23 thoregon kernel: [ 2160.599062] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0
Nov 20 19:27:23 thoregon kernel: [ 2160.599076] #5: (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffff811e854c>] xfs_ilock_nowait+0xbc/0x100
Nov 20 19:27:41 thoregon login[3002]: pam_unix(login:session): session closed for user root
Nov 20 19:27:43 thoregon acpid: client connected from 2991[0:0]
Nov 20 19:27:43 thoregon acpid: 1 client rule loaded
Nov 20 19:29:23 thoregon kernel: [ 2280.601023] INFO: task kswapd0:725 blocked for more than 120 seconds.
Nov 20 19:29:23 thoregon kernel: [ 2280.601030] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 19:29:23 thoregon kernel: [ 2280.601036] kswapd0 D 0000000000000001 0 725 2 0x00000000
Nov 20 19:29:23 thoregon kernel: [ 2280.601048] ffff8803280d13f8 0000000000000046 ffff880329a0ab80 ffff8803280d1fd8
Nov 20 19:29:23 thoregon kernel: [ 2280.601063] ffff8803280d1fd8 ffff8803280d1fd8 ffff880046b7c880 ffff880329a0ab80
Nov 20 19:29:23 thoregon kernel: [ 2280.601073] ffff8803280d1408 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff
Nov 20 19:29:23 thoregon kernel: [ 2280.601084] Call Trace:
Nov 20 19:29:23 thoregon kernel: [ 2280.601101] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:29:23 thoregon kernel: [ 2280.601111] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80
Nov 20 19:29:23 thoregon kernel: [ 2280.601121] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
Nov 20 19:29:23 thoregon kernel: [ 2280.601129] [<ffffffff81500753>] bitmap_unplug+0x173/0x180
Nov 20 19:29:23 thoregon kernel: [ 2280.601138] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:29:23 thoregon kernel: [ 2280.601147] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110
Nov 20 19:29:23 thoregon kernel: [ 2280.601157] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240
Nov 20 19:29:23 thoregon kernel: [ 2280.601165] [<ffffffff816b24ca>] ? _raw_spin_unlock_irqrestore+0x3a/0x70
Nov 20 19:29:23 thoregon kernel: [ 2280.601175] [<ffffffff816b15c3>] io_schedule_timeout+0x83/0xf0
Nov 20 19:29:23 thoregon kernel: [ 2280.601182] [<ffffffff810815cd>] ? trace_hardirqs_on+0xd/0x10
Nov 20 19:29:23 thoregon kernel: [ 2280.601192] [<ffffffff810b0e1d>] mempool_alloc+0x12d/0x160
Nov 20 19:29:23 thoregon kernel: [ 2280.601200] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
Nov 20 19:29:23 thoregon kernel: [ 2280.601209] [<ffffffff811263da>] bvec_alloc_bs+0xda/0x100
Nov 20 19:29:23 thoregon kernel: [ 2280.601217] [<ffffffff811264ea>] bio_alloc_bioset+0xea/0x110
Nov 20 19:29:23 thoregon kernel: [ 2280.601226] [<ffffffff81126656>] bio_clone_bioset+0x16/0x40
Nov 20 19:29:23 thoregon kernel: [ 2280.601233] [<ffffffff814f471a>] bio_clone_mddev+0x1a/0x30
Nov 20 19:29:23 thoregon kernel: [ 2280.601241] [<ffffffff814edbb1>] make_request+0x551/0xde0
Nov 20 19:29:23 thoregon kernel: [ 2280.601249] [<ffffffff814ed9e0>] ? make_request+0x380/0xde0
Nov 20 19:29:23 thoregon kernel: [ 2280.601257] [<ffffffff810b0b80>] ? mempool_alloc_slab+0x10/0x20
Nov 20 19:29:23 thoregon kernel: [ 2280.601266] [<ffffffff814f80bb>] md_make_request+0x21b/0x4d0
Nov 20 19:29:23 thoregon kernel: [ 2280.601273] [<ffffffff814f7ef0>] ? md_make_request+0x50/0x4d0
Nov 20 19:29:23 thoregon kernel: [ 2280.601282] [<ffffffff81276e52>] generic_make_request+0xc2/0x100
Nov 20 19:29:23 thoregon kernel: [ 2280.601291] [<ffffffff81276ef5>] submit_bio+0x65/0x110
Nov 20 19:29:23 thoregon kernel: [ 2280.601301] [<ffffffff811e07bf>] xfs_submit_ioend_bio.isra.21+0x2f/0x40
Nov 20 19:29:23 thoregon kernel: [ 2280.601309] [<ffffffff811e088e>] xfs_submit_ioend+0xbe/0x110
Nov 20 19:29:23 thoregon kernel: [ 2280.601318] [<ffffffff811e0c91>] xfs_vm_writepage+0x3b1/0x540
Nov 20 19:29:23 thoregon kernel: [ 2280.601328] [<ffffffff810bcd84>] shrink_page_list+0x564/0x890
Nov 20 19:29:23 thoregon kernel: [ 2280.601338] [<ffffffff810bd637>] shrink_inactive_list+0x1d7/0x310
Nov 20 19:29:23 thoregon kernel: [ 2280.601347] [<ffffffff810bdb9d>] shrink_lruvec+0x42d/0x530
Nov 20 19:29:23 thoregon kernel: [ 2280.601356] [<ffffffff810be323>] kswapd+0x683/0xa20
Nov 20 19:29:23 thoregon kernel: [ 2280.601365] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
Nov 20 19:29:23 thoregon kernel: [ 2280.601374] [<ffffffff810bdca0>] ? shrink_lruvec+0x530/0x530
Nov 20 19:29:23 thoregon kernel: [ 2280.601381] [<ffffffff8105c246>] kthread+0xd6/0xe0
Nov 20 19:29:23 thoregon kernel: [ 2280.601388] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:29:23 thoregon kernel: [ 2280.601397] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:29:23 thoregon kernel: [ 2280.601404] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0
Nov 20 19:29:23 thoregon kernel: [ 2280.601412] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:29:23 thoregon kernel: [ 2280.601417] no locks held by kswapd0/725.
Nov 20 19:30:01 thoregon cron[27484]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons)
Nov 20 19:35:16 thoregon kernel: [ 2633.536440] SysRq : Show Blocked State
Nov 20 19:35:16 thoregon kernel: [ 2633.536453] task PC stack pid father
Nov 20 19:35:16 thoregon kernel: [ 2633.536485] kswapd0 D 0000000000000001 0 725 2 0x00000000
Nov 20 19:35:16 thoregon kernel: [ 2633.536496] ffff8803280d13f8 0000000000000046 ffff880329a0ab80 ffff8803280d1fd8
Nov 20 19:35:16 thoregon kernel: [ 2633.536507] ffff8803280d1fd8 ffff8803280d1fd8 ffff880046b7c880 ffff880329a0ab80
Nov 20 19:35:16 thoregon kernel: [ 2633.536517] ffff8803280d1408 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff
Nov 20 19:35:16 thoregon kernel: [ 2633.536526] Call Trace:
Nov 20 19:35:16 thoregon kernel: [ 2633.536542] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.536553] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80
Nov 20 19:35:16 thoregon kernel: [ 2633.536563] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.536571] [<ffffffff81500753>] bitmap_unplug+0x173/0x180
Nov 20 19:35:16 thoregon kernel: [ 2633.536581] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.536589] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110
Nov 20 19:35:16 thoregon kernel: [ 2633.536599] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240
Nov 20 19:35:16 thoregon kernel: [ 2633.536607] [<ffffffff816b24ca>] ? _raw_spin_unlock_irqrestore+0x3a/0x70
Nov 20 19:35:16 thoregon kernel: [ 2633.536616] [<ffffffff816b15c3>] io_schedule_timeout+0x83/0xf0
Nov 20 19:35:16 thoregon kernel: [ 2633.536623] [<ffffffff810815cd>] ? trace_hardirqs_on+0xd/0x10
Nov 20 19:35:16 thoregon kernel: [ 2633.536633] [<ffffffff810b0e1d>] mempool_alloc+0x12d/0x160
Nov 20 19:35:16 thoregon kernel: [ 2633.536641] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.536650] [<ffffffff811263da>] bvec_alloc_bs+0xda/0x100
Nov 20 19:35:16 thoregon kernel: [ 2633.536658] [<ffffffff811264ea>] bio_alloc_bioset+0xea/0x110
Nov 20 19:35:16 thoregon kernel: [ 2633.536666] [<ffffffff81126656>] bio_clone_bioset+0x16/0x40
Nov 20 19:35:16 thoregon kernel: [ 2633.536673] [<ffffffff814f471a>] bio_clone_mddev+0x1a/0x30
Nov 20 19:35:16 thoregon kernel: [ 2633.536681] [<ffffffff814edbb1>] make_request+0x551/0xde0
Nov 20 19:35:16 thoregon kernel: [ 2633.536689] [<ffffffff814ed9e0>] ? make_request+0x380/0xde0
Nov 20 19:35:16 thoregon kernel: [ 2633.536697] [<ffffffff810b0b80>] ? mempool_alloc_slab+0x10/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.536706] [<ffffffff814f80bb>] md_make_request+0x21b/0x4d0
Nov 20 19:35:16 thoregon kernel: [ 2633.536712] [<ffffffff814f7ef0>] ? md_make_request+0x50/0x4d0
Nov 20 19:35:16 thoregon kernel: [ 2633.536722] [<ffffffff81276e52>] generic_make_request+0xc2/0x100
Nov 20 19:35:16 thoregon kernel: [ 2633.536730] [<ffffffff81276ef5>] submit_bio+0x65/0x110
Nov 20 19:35:16 thoregon kernel: [ 2633.536740] [<ffffffff811e07bf>] xfs_submit_ioend_bio.isra.21+0x2f/0x40
Nov 20 19:35:16 thoregon kernel: [ 2633.536748] [<ffffffff811e088e>] xfs_submit_ioend+0xbe/0x110
Nov 20 19:35:16 thoregon kernel: [ 2633.536757] [<ffffffff811e0c91>] xfs_vm_writepage+0x3b1/0x540
Nov 20 19:35:16 thoregon kernel: [ 2633.536767] [<ffffffff810bcd84>] shrink_page_list+0x564/0x890
Nov 20 19:35:16 thoregon kernel: [ 2633.536777] [<ffffffff810bd637>] shrink_inactive_list+0x1d7/0x310
Nov 20 19:35:16 thoregon kernel: [ 2633.536785] [<ffffffff810bdb9d>] shrink_lruvec+0x42d/0x530
Nov 20 19:35:16 thoregon kernel: [ 2633.536795] [<ffffffff810be323>] kswapd+0x683/0xa20
Nov 20 19:35:16 thoregon kernel: [ 2633.536804] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.536812] [<ffffffff810bdca0>] ? shrink_lruvec+0x530/0x530
Nov 20 19:35:16 thoregon kernel: [ 2633.536819] [<ffffffff8105c246>] kthread+0xd6/0xe0
Nov 20 19:35:16 thoregon kernel: [ 2633.536826] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.536835] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:35:16 thoregon kernel: [ 2633.536842] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0
Nov 20 19:35:16 thoregon kernel: [ 2633.536849] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:35:16 thoregon kernel: [ 2633.536890] xfsaild/md4 D 0000000000000003 0 1742 2 0x00000000
Nov 20 19:35:16 thoregon kernel: [ 2633.536898] ffff88032438bb68 0000000000000046 ffff880329965700 ffff88032438bfd8
Nov 20 19:35:16 thoregon kernel: [ 2633.536908] ffff88032438bfd8 ffff88032438bfd8 ffff88032827e580 ffff880329965700
Nov 20 19:35:16 thoregon kernel: [ 2633.536917] ffff88032438bb78 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff
Nov 20 19:35:16 thoregon kernel: [ 2633.536926] Call Trace:
Nov 20 19:35:16 thoregon kernel: [ 2633.536935] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.536942] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80
Nov 20 19:35:16 thoregon kernel: [ 2633.536950] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.536957] [<ffffffff81500753>] bitmap_unplug+0x173/0x180
Nov 20 19:35:16 thoregon kernel: [ 2633.536966] [<ffffffff81278c13>] ? blk_finish_plug+0x13/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.536974] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110
Nov 20 19:35:16 thoregon kernel: [ 2633.536982] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240
Nov 20 19:35:16 thoregon kernel: [ 2633.536991] [<ffffffff81278c13>] blk_finish_plug+0x13/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.536998] [<ffffffff811e296a>] __xfs_buf_delwri_submit+0x1ca/0x1e0
Nov 20 19:35:16 thoregon kernel: [ 2633.537006] [<ffffffff811e2ffb>] xfs_buf_delwri_submit_nowait+0x1b/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.537014] [<ffffffff81233066>] xfsaild+0x226/0x4c0
Nov 20 19:35:16 thoregon kernel: [ 2633.537022] [<ffffffff81065dfa>] ? finish_task_switch+0x3a/0x100
Nov 20 19:35:16 thoregon kernel: [ 2633.537030] [<ffffffff81232e40>] ? xfs_trans_ail_cursor_first+0xa0/0xa0
Nov 20 19:35:16 thoregon kernel: [ 2633.537037] [<ffffffff8105c246>] kthread+0xd6/0xe0
Nov 20 19:35:16 thoregon kernel: [ 2633.537044] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.537053] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:35:16 thoregon kernel: [ 2633.537060] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0
Nov 20 19:35:16 thoregon kernel: [ 2633.537067] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:35:16 thoregon kernel: [ 2633.537158] plasma-desktop D 0000000000000001 0 3195 1 0x00000000
Nov 20 19:35:16 thoregon kernel: [ 2633.537165] ffff88030b99d448 0000000000000046 ffff8803266c1d00 ffff88030b99dfd8
Nov 20 19:35:16 thoregon kernel: [ 2633.537174] ffff88030b99dfd8 ffff88030b99dfd8 ffff880046b78e80 ffff8803266c1d00
Nov 20 19:35:16 thoregon kernel: [ 2633.537183] 0000000000000282 ffff88030b99d468 0000000100038f76 0000000100038f74
Nov 20 19:35:16 thoregon kernel: [ 2633.537192] Call Trace:
Nov 20 19:35:16 thoregon kernel: [ 2633.537201] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.537210] [<ffffffff816aed8e>] schedule_timeout+0x12e/0x1c0
Nov 20 19:35:16 thoregon kernel: [ 2633.537219] [<ffffffff81048730>] ? call_timer_fn+0xf0/0xf0
Nov 20 19:35:16 thoregon kernel: [ 2633.537228] [<ffffffff816aee39>] schedule_timeout_uninterruptible+0x19/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.537235] [<ffffffff811f1f99>] xfs_reclaim_inode+0x129/0x340
Nov 20 19:35:16 thoregon kernel: [ 2633.537242] [<ffffffff811f2d6f>] xfs_reclaim_inodes_ag+0x2bf/0x4f0
Nov 20 19:35:16 thoregon kernel: [ 2633.537249] [<ffffffff811f2b90>] ? xfs_reclaim_inodes_ag+0xe0/0x4f0
Nov 20 19:35:16 thoregon kernel: [ 2633.537258] [<ffffffff811f30de>] xfs_reclaim_inodes_nr+0x2e/0x40
Nov 20 19:35:16 thoregon kernel: [ 2633.537267] [<ffffffff811ef9b0>] xfs_fs_free_cached_objects+0x10/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.537274] [<ffffffff810f5b43>] prune_super+0x113/0x1a0
Nov 20 19:35:16 thoregon kernel: [ 2633.537282] [<ffffffff810bbd4e>] shrink_slab+0x11e/0x1f0
Nov 20 19:35:16 thoregon kernel: [ 2633.537290] [<ffffffff810b4fa5>] ? drain_pages+0x95/0xc0
Nov 20 19:35:16 thoregon kernel: [ 2633.537298] [<ffffffff810be8df>] try_to_free_pages+0x21f/0x4e0
Nov 20 19:35:16 thoregon kernel: [ 2633.537306] [<ffffffff810b5eb6>] __alloc_pages_nodemask+0x506/0x800
Nov 20 19:35:16 thoregon kernel: [ 2633.537316] [<ffffffff81570ed3>] ? __alloc_skb+0x83/0x280
Nov 20 19:35:16 thoregon kernel: [ 2633.537323] [<ffffffff810b6232>] __get_free_pages+0x12/0x40
Nov 20 19:35:16 thoregon kernel: [ 2633.537331] [<ffffffff810ebd8c>] __kmalloc_track_caller+0x14c/0x170
Nov 20 19:35:16 thoregon kernel: [ 2633.537339] [<ffffffff815703e6>] __kmalloc_reserve+0x36/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.537346] [<ffffffff81570ed3>] __alloc_skb+0x83/0x280
Nov 20 19:35:16 thoregon kernel: [ 2633.537354] [<ffffffff8105c905>] ? remove_wait_queue+0x55/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.537361] [<ffffffff81569fb1>] sock_alloc_send_pskb+0x251/0x3e0
Nov 20 19:35:16 thoregon kernel: [ 2633.537369] [<ffffffff8156a150>] sock_alloc_send_skb+0x10/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.537376] [<ffffffff81615f21>] unix_stream_sendmsg+0x2e1/0x420
Nov 20 19:35:16 thoregon kernel: [ 2633.537385] [<ffffffff81564900>] ? sock_aio_read+0x30/0x30
Nov 20 19:35:16 thoregon kernel: [ 2633.537393] [<ffffffff81564a0c>] sock_aio_write+0x10c/0x140
Nov 20 19:35:16 thoregon kernel: [ 2633.537403] [<ffffffff810f2eeb>] do_sync_readv_writev+0x9b/0xe0
Nov 20 19:35:16 thoregon kernel: [ 2633.537413] [<ffffffff810f31a8>] do_readv_writev+0xc8/0x1c0
Nov 20 19:35:16 thoregon kernel: [ 2633.537421] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.537428] [<ffffffff8107c899>] ? __lock_is_held+0x59/0x70
Nov 20 19:35:16 thoregon kernel: [ 2633.537438] [<ffffffff8110f892>] ? fget_light+0x42/0x4b0
Nov 20 19:35:16 thoregon kernel: [ 2633.537446] [<ffffffff810f32d7>] vfs_writev+0x37/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.537454] [<ffffffff810f341d>] sys_writev+0x4d/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.537462] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:35:16 thoregon kernel: [ 2633.537518] ssh D 0000000000000000 0 3573 1 0x00000004
Nov 20 19:35:16 thoregon kernel: [ 2633.537525] ffff8802c9f2f508 0000000000000046 ffff8802f04dd700 ffff8802c9f2ffd8
Nov 20 19:35:16 thoregon kernel: [ 2633.537534] ffff8802c9f2ffd8 ffff8802c9f2ffd8 ffffffff81c13420 ffff8802f04dd700
Nov 20 19:35:16 thoregon kernel: [ 2633.537543] 0000000000000282 ffff8802c9f2f528 0000000100038f75 0000000100038f73
Nov 20 19:35:16 thoregon kernel: [ 2633.537552] Call Trace:
Nov 20 19:35:16 thoregon kernel: [ 2633.537561] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.537569] [<ffffffff816aed8e>] schedule_timeout+0x12e/0x1c0
Nov 20 19:35:16 thoregon kernel: [ 2633.537578] [<ffffffff81048730>] ? call_timer_fn+0xf0/0xf0
Nov 20 19:35:16 thoregon kernel: [ 2633.537587] [<ffffffff816aee39>] schedule_timeout_uninterruptible+0x19/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.537594] [<ffffffff811f1f99>] xfs_reclaim_inode+0x129/0x340
Nov 20 19:35:16 thoregon kernel: [ 2633.537601] [<ffffffff811f2d6f>] xfs_reclaim_inodes_ag+0x2bf/0x4f0
Nov 20 19:35:16 thoregon kernel: [ 2633.537607] [<ffffffff811f2b90>] ? xfs_reclaim_inodes_ag+0xe0/0x4f0
Nov 20 19:35:16 thoregon kernel: [ 2633.537616] [<ffffffff811f30de>] xfs_reclaim_inodes_nr+0x2e/0x40
Nov 20 19:35:16 thoregon kernel: [ 2633.537625] [<ffffffff811ef9b0>] xfs_fs_free_cached_objects+0x10/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.537631] [<ffffffff810f5b43>] prune_super+0x113/0x1a0
Nov 20 19:35:16 thoregon kernel: [ 2633.537639] [<ffffffff810bbd4e>] shrink_slab+0x11e/0x1f0
Nov 20 19:35:16 thoregon kernel: [ 2633.537647] [<ffffffff810b4fa5>] ? drain_pages+0x95/0xc0
Nov 20 19:35:16 thoregon kernel: [ 2633.537655] [<ffffffff810be8df>] try_to_free_pages+0x21f/0x4e0
Nov 20 19:35:16 thoregon kernel: [ 2633.537663] [<ffffffff810b5eb6>] __alloc_pages_nodemask+0x506/0x800
Nov 20 19:35:16 thoregon kernel: [ 2633.537670] [<ffffffff81104d10>] ? poll_select_set_timeout+0x90/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.537679] [<ffffffff81570ed3>] ? __alloc_skb+0x83/0x280
Nov 20 19:35:16 thoregon kernel: [ 2633.537686] [<ffffffff810b6232>] __get_free_pages+0x12/0x40
Nov 20 19:35:16 thoregon kernel: [ 2633.537693] [<ffffffff810ebd8c>] __kmalloc_track_caller+0x14c/0x170
Nov 20 19:35:16 thoregon kernel: [ 2633.537701] [<ffffffff815703e6>] __kmalloc_reserve+0x36/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.537708] [<ffffffff81570ed3>] __alloc_skb+0x83/0x280
Nov 20 19:35:16 thoregon kernel: [ 2633.537715] [<ffffffff81569ac3>] ? release_sock+0x183/0x1e0
Nov 20 19:35:16 thoregon kernel: [ 2633.537722] [<ffffffff81569fb1>] sock_alloc_send_pskb+0x251/0x3e0
Nov 20 19:35:16 thoregon kernel: [ 2633.537729] [<ffffffff8156a150>] sock_alloc_send_skb+0x10/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.537736] [<ffffffff81615f21>] unix_stream_sendmsg+0x2e1/0x420
Nov 20 19:35:16 thoregon kernel: [ 2633.537745] [<ffffffff81564a0c>] sock_aio_write+0x10c/0x140
Nov 20 19:35:16 thoregon kernel: [ 2633.537753] [<ffffffff810cb5ae>] ? might_fault+0x4e/0xa0
Nov 20 19:35:16 thoregon kernel: [ 2633.537760] [<ffffffff811059aa>] ? core_sys_select+0x47a/0x4d0
Nov 20 19:35:16 thoregon kernel: [ 2633.537769] [<ffffffff810f238b>] do_sync_write+0x9b/0xe0
Nov 20 19:35:16 thoregon kernel: [ 2633.537778] [<ffffffff810f2a75>] vfs_write+0x145/0x160
Nov 20 19:35:16 thoregon kernel: [ 2633.537787] [<ffffffff810f2ccd>] sys_write+0x4d/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.537794] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:35:16 thoregon kernel: [ 2633.537801] bash D 0000000000000000 0 28378 28334 0x00000000
Nov 20 19:35:16 thoregon kernel: [ 2633.537809] ffff880299f5d5d8 0000000000000046 ffff880326a83a00 ffff880299f5dfd8
Nov 20 19:35:16 thoregon kernel: [ 2633.537818] ffff880299f5dfd8 ffff880299f5dfd8 ffff88032989ba00 ffff880326a83a00
Nov 20 19:35:16 thoregon kernel: [ 2633.537827] ffff880299f5d6a8 ffff880299c50230 7fffffffffffffff ffff880326a83a00
Nov 20 19:35:16 thoregon kernel: [ 2633.537836] Call Trace:
Nov 20 19:35:16 thoregon kernel: [ 2633.537845] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.537853] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0
Nov 20 19:35:16 thoregon kernel: [ 2633.537860] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.537868] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.537876] [<ffffffff816b057e>] __down_common+0x97/0xe8
Nov 20 19:35:16 thoregon kernel: [ 2633.537885] [<ffffffff816b062e>] __down+0x18/0x1a
Nov 20 19:35:16 thoregon kernel: [ 2633.537891] [<ffffffff8106245b>] down+0x3b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.537898] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.537904] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200
Nov 20 19:35:16 thoregon kernel: [ 2633.537911] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130
Nov 20 19:35:16 thoregon kernel: [ 2633.537918] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.537925] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0
Nov 20 19:35:16 thoregon kernel: [ 2633.537933] [<ffffffff811f73fb>] xfs_alloc_read_agfl+0x6b/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.537941] [<ffffffff811f986d>] ? xfs_alloc_read_agf+0xcd/0xf0
Nov 20 19:35:16 thoregon kernel: [ 2633.537948] [<ffffffff810e9a93>] ? kmem_cache_free+0x83/0x150
Nov 20 19:35:16 thoregon kernel: [ 2633.537955] [<ffffffff811f9ac1>] xfs_alloc_fix_freelist+0x231/0x480
Nov 20 19:35:16 thoregon kernel: [ 2633.537962] [<ffffffff8107c899>] ? __lock_is_held+0x59/0x70
Nov 20 19:35:16 thoregon kernel: [ 2633.537971] [<ffffffff811fa279>] xfs_free_extent+0x99/0x120
Nov 20 19:35:16 thoregon kernel: [ 2633.537978] [<ffffffff811f6fff>] ? kmem_zone_alloc+0x5f/0xe0
Nov 20 19:35:16 thoregon kernel: [ 2633.537987] [<ffffffff81206edc>] xfs_bmap_finish+0x15c/0x1a0
Nov 20 19:35:16 thoregon kernel: [ 2633.537996] [<ffffffff8121f9eb>] xfs_itruncate_extents+0xdb/0x200
Nov 20 19:35:16 thoregon kernel: [ 2633.538006] [<ffffffff811ed7d5>] xfs_setattr_size+0x355/0x3b0
Nov 20 19:35:16 thoregon kernel: [ 2633.538014] [<ffffffff811ed85e>] xfs_vn_setattr+0x2e/0x40
Nov 20 19:35:16 thoregon kernel: [ 2633.538022] [<ffffffff8110dfa6>] notify_change+0x186/0x3a0
Nov 20 19:35:16 thoregon kernel: [ 2633.538030] [<ffffffff810f1229>] do_truncate+0x59/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.538039] [<ffffffff810fd777>] ? __inode_permission+0x27/0x80
Nov 20 19:35:16 thoregon kernel: [ 2633.538048] [<ffffffff811008da>] do_last.isra.69+0x60a/0xc80
Nov 20 19:35:16 thoregon kernel: [ 2633.538056] [<ffffffff810fd7e3>] ? inode_permission+0x13/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.538064] [<ffffffff810fd882>] ? link_path_walk+0x62/0x910
Nov 20 19:35:16 thoregon kernel: [ 2633.538073] [<ffffffff81100ffb>] path_openat.isra.70+0xab/0x490
Nov 20 19:35:16 thoregon kernel: [ 2633.538080] [<ffffffff8107c899>] ? __lock_is_held+0x59/0x70
Nov 20 19:35:16 thoregon kernel: [ 2633.538089] [<ffffffff8110179d>] do_filp_open+0x3d/0xa0
Nov 20 19:35:16 thoregon kernel: [ 2633.538097] [<ffffffff8111082b>] ? __alloc_fd+0x16b/0x210
Nov 20 19:35:16 thoregon kernel: [ 2633.538106] [<ffffffff810f2089>] do_sys_open+0xf9/0x1e0
Nov 20 19:35:16 thoregon kernel: [ 2633.538114] [<ffffffff810f218c>] sys_open+0x1c/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.538121] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:35:16 thoregon kernel: [ 2633.538126] flush-9:4 D 0000000000000001 0 30365 2 0x00000000
Nov 20 19:35:16 thoregon kernel: [ 2633.538134] ffff8801d8ce1978 0000000000000046 ffff880328238e80 ffff8801d8ce1fd8
Nov 20 19:35:16 thoregon kernel: [ 2633.538143] ffff8801d8ce1fd8 ffff8801d8ce1fd8 ffff8803032cc880 ffff880328238e80
Nov 20 19:35:16 thoregon kernel: [ 2633.538151] ffff8801d8ce1988 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff
Nov 20 19:35:16 thoregon kernel: [ 2633.538160] Call Trace:
Nov 20 19:35:16 thoregon kernel: [ 2633.538169] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.538176] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80
Nov 20 19:35:16 thoregon kernel: [ 2633.538184] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.538192] [<ffffffff81500753>] bitmap_unplug+0x173/0x180
Nov 20 19:35:16 thoregon kernel: [ 2633.538199] [<ffffffff810b6acf>] ? write_cache_pages+0x12f/0x420
Nov 20 19:35:16 thoregon kernel: [ 2633.538206] [<ffffffff810b6700>] ? set_page_dirty_lock+0x60/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.538214] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110
Nov 20 19:35:16 thoregon kernel: [ 2633.538222] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240
Nov 20 19:35:16 thoregon kernel: [ 2633.538231] [<ffffffff81278c13>] blk_finish_plug+0x13/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.538238] [<ffffffff810b6e0a>] generic_writepages+0x4a/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.538247] [<ffffffff811df898>] xfs_vm_writepages+0x48/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.538256] [<ffffffff8111b144>] ? writeback_sb_inodes+0x174/0x410
Nov 20 19:35:16 thoregon kernel: [ 2633.538264] [<ffffffff810b84be>] do_writepages+0x1e/0x30
Nov 20 19:35:16 thoregon kernel: [ 2633.538272] [<ffffffff8111a1ee>] __writeback_single_inode+0x3e/0x120
Nov 20 19:35:16 thoregon kernel: [ 2633.538280] [<ffffffff8111b234>] writeback_sb_inodes+0x264/0x410
Nov 20 19:35:16 thoregon kernel: [ 2633.538289] [<ffffffff8111b476>] __writeback_inodes_wb+0x96/0xc0
Nov 20 19:35:16 thoregon kernel: [ 2633.538298] [<ffffffff8111b673>] wb_writeback+0x1d3/0x1e0
Nov 20 19:35:16 thoregon kernel: [ 2633.538305] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.538313] [<ffffffff8111bb1c>] ? wb_do_writeback+0x7c/0x140
Nov 20 19:35:16 thoregon kernel: [ 2633.538322] [<ffffffff8111bb40>] wb_do_writeback+0xa0/0x140
Nov 20 19:35:16 thoregon kernel: [ 2633.538331] [<ffffffff8111bc52>] bdi_writeback_thread+0x72/0x150
Nov 20 19:35:16 thoregon kernel: [ 2633.538339] [<ffffffff8111bbe0>] ? wb_do_writeback+0x140/0x140
Nov 20 19:35:16 thoregon kernel: [ 2633.538346] [<ffffffff8105c246>] kthread+0xd6/0xe0
Nov 20 19:35:16 thoregon kernel: [ 2633.538353] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.538361] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:35:16 thoregon kernel: [ 2633.538369] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0
Nov 20 19:35:16 thoregon kernel: [ 2633.538376] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0
Nov 20 19:35:16 thoregon kernel: [ 2633.538384] x86_64-pc-linux D 0000000000000000 0 26516 1 0x00000004
Nov 20 19:35:16 thoregon kernel: [ 2633.538391] ffff8800468dddf8 0000000000000046 ffff880329303a00 ffff8800468ddfd8
Nov 20 19:35:16 thoregon kernel: [ 2633.538400] ffff8800468ddfd8 ffff8800468ddfd8 ffff88032989c880 ffff880329303a00
Nov 20 19:35:16 thoregon kernel: [ 2633.538409] 0000000000000246 ffff8800468dc000 ffff8801b1a1bc68 0000000002405150
Nov 20 19:35:16 thoregon kernel: [ 2633.538418] Call Trace:
Nov 20 19:35:16 thoregon kernel: [ 2633.538427] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.538436] [<ffffffff816b1501>] schedule_preempt_disabled+0x21/0x30
Nov 20 19:35:16 thoregon kernel: [ 2633.538444] [<ffffffff816af3a5>] mutex_lock_nested+0x165/0x330
Nov 20 19:35:16 thoregon kernel: [ 2633.538452] [<ffffffff810ff5e3>] ? do_unlinkat+0xa3/0x1b0
Nov 20 19:35:16 thoregon kernel: [ 2633.538461] [<ffffffff810fefae>] ? filename_lookup.isra.57+0x2e/0x80
Nov 20 19:35:16 thoregon kernel: [ 2633.538469] [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0
Nov 20 19:35:16 thoregon kernel: [ 2633.538477] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.538485] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Nov 20 19:35:16 thoregon kernel: [ 2633.538494] [<ffffffff81101bb1>] sys_unlink+0x11/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.538501] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:35:16 thoregon kernel: [ 2633.538505] x86_64-pc-linux D 0000000000000000 0 26523 1 0x00000004
Nov 20 19:35:16 thoregon kernel: [ 2633.538512] ffff8800c0def858 0000000000000046 ffff88032986c880 ffff8800c0deffd8
Nov 20 19:35:16 thoregon kernel: [ 2633.538521] ffff8800c0deffd8 ffff8800c0deffd8 ffff88032989e580 ffff88032986c880
Nov 20 19:35:16 thoregon kernel: [ 2633.538530] ffff8800c0def928 ffff8801c57e3a30 7fffffffffffffff ffff88032986c880
Nov 20 19:35:16 thoregon kernel: [ 2633.538539] Call Trace:
Nov 20 19:35:16 thoregon kernel: [ 2633.538548] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.538556] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0
Nov 20 19:35:16 thoregon kernel: [ 2633.538563] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.538571] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.538579] [<ffffffff816b057e>] __down_common+0x97/0xe8
Nov 20 19:35:16 thoregon kernel: [ 2633.538588] [<ffffffff816b062e>] __down+0x18/0x1a
Nov 20 19:35:16 thoregon kernel: [ 2633.538594] [<ffffffff8106245b>] down+0x3b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.538601] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.538607] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200
Nov 20 19:35:16 thoregon kernel: [ 2633.538614] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130
Nov 20 19:35:16 thoregon kernel: [ 2633.538621] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0
Nov 20 19:35:16 thoregon kernel: [ 2633.538628] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460
Nov 20 19:35:16 thoregon kernel: [ 2633.538635] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.538642] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0
Nov 20 19:35:16 thoregon kernel: [ 2633.538650] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180
Nov 20 19:35:16 thoregon kernel: [ 2633.538659] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0
Nov 20 19:35:16 thoregon kernel: [ 2633.538668] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330
Nov 20 19:35:16 thoregon kernel: [ 2633.538676] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0
Nov 20 19:35:16 thoregon kernel: [ 2633.538684] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150
Nov 20 19:35:16 thoregon kernel: [ 2633.538692] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370
Nov 20 19:35:16 thoregon kernel: [ 2633.538701] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330
Nov 20 19:35:16 thoregon kernel: [ 2633.538709] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100
Nov 20 19:35:16 thoregon kernel: [ 2633.538717] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.538725] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100
Nov 20 19:35:16 thoregon kernel: [ 2633.538733] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0
Nov 20 19:35:16 thoregon kernel: [ 2633.538741] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.538748] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Nov 20 19:35:16 thoregon kernel: [ 2633.538757] [<ffffffff81101bb1>] sys_unlink+0x11/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.538764] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:35:16 thoregon kernel: [ 2633.538769] x86_64-pc-linux D 0000000000000000 0 26875 1 0x00000004
Nov 20 19:35:16 thoregon kernel: [ 2633.538776] ffff8800c0c5ba08 0000000000000046 ffff8802f0594880 ffff8800c0c5bfd8
Nov 20 19:35:16 thoregon kernel: [ 2633.538785] ffff8800c0c5bfd8 ffff8800c0c5bfd8 ffff88032989c880 ffff8802f0594880
Nov 20 19:35:16 thoregon kernel: [ 2633.538793] 0000000000000250 ffff880205a7aa30 7fffffffffffffff ffff8802f0594880
Nov 20 19:35:16 thoregon kernel: [ 2633.538802] Call Trace:
Nov 20 19:35:16 thoregon kernel: [ 2633.538811] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.538819] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0
Nov 20 19:35:16 thoregon kernel: [ 2633.538826] [<ffffffff811f70b4>] ? kmem_zone_zalloc+0x34/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.538833] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.538841] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.538849] [<ffffffff816b057e>] __down_common+0x97/0xe8
Nov 20 19:35:16 thoregon kernel: [ 2633.538857] [<ffffffff816b062e>] __down+0x18/0x1a
Nov 20 19:35:16 thoregon kernel: [ 2633.538864] [<ffffffff8106245b>] down+0x3b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.538870] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.538877] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200
Nov 20 19:35:16 thoregon kernel: [ 2633.538883] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130
Nov 20 19:35:16 thoregon kernel: [ 2633.538890] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.538897] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0
Nov 20 19:35:16 thoregon kernel: [ 2633.538906] [<ffffffff8121ce6f>] xfs_read_agi+0x7f/0x110
Nov 20 19:35:16 thoregon kernel: [ 2633.538915] [<ffffffff8121fb55>] xfs_iunlink+0x45/0x150
Nov 20 19:35:16 thoregon kernel: [ 2633.538922] [<ffffffff81041fc1>] ? current_fs_time+0x11/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.538929] [<ffffffff81234352>] ? xfs_trans_ichgtime+0x22/0xa0
Nov 20 19:35:16 thoregon kernel: [ 2633.538936] [<ffffffff811f3ea0>] xfs_droplink+0x50/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.538943] [<ffffffff811f5d77>] xfs_remove+0x287/0x370
Nov 20 19:35:16 thoregon kernel: [ 2633.538951] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330
Nov 20 19:35:16 thoregon kernel: [ 2633.538959] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100
Nov 20 19:35:16 thoregon kernel: [ 2633.538967] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.538975] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100
Nov 20 19:35:16 thoregon kernel: [ 2633.538983] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0
Nov 20 19:35:16 thoregon kernel: [ 2633.538991] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.538999] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Nov 20 19:35:16 thoregon kernel: [ 2633.539007] [<ffffffff81101bb1>] sys_unlink+0x11/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.539014] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:35:16 thoregon kernel: [ 2633.539020] x86_64-pc-linux D 0000000000000001 0 26897 1 0x00000004
Nov 20 19:35:16 thoregon kernel: [ 2633.539027] ffff880101959858 0000000000000046 ffff88032934ab80 ffff880101959fd8
Nov 20 19:35:16 thoregon kernel: [ 2633.539036] ffff880101959fd8 ffff880101959fd8 ffff880162a3e580 ffff88032934ab80
Nov 20 19:35:16 thoregon kernel: [ 2633.539045] ffff880101959928 ffff8801ca545030 7fffffffffffffff ffff88032934ab80
Nov 20 19:35:16 thoregon kernel: [ 2633.539053] Call Trace:
Nov 20 19:35:16 thoregon kernel: [ 2633.539062] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.539070] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0
Nov 20 19:35:16 thoregon kernel: [ 2633.539077] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.539085] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.539094] [<ffffffff816b057e>] __down_common+0x97/0xe8
Nov 20 19:35:16 thoregon kernel: [ 2633.539102] [<ffffffff816b062e>] __down+0x18/0x1a
Nov 20 19:35:16 thoregon kernel: [ 2633.539108] [<ffffffff8106245b>] down+0x3b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.539115] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.539121] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200
Nov 20 19:35:16 thoregon kernel: [ 2633.539128] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130
Nov 20 19:35:16 thoregon kernel: [ 2633.539135] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0
Nov 20 19:35:16 thoregon kernel: [ 2633.539142] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460
Nov 20 19:35:16 thoregon kernel: [ 2633.539149] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.539156] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0
Nov 20 19:35:16 thoregon kernel: [ 2633.539164] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180
Nov 20 19:35:16 thoregon kernel: [ 2633.539172] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0
Nov 20 19:35:16 thoregon kernel: [ 2633.539181] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330
Nov 20 19:35:16 thoregon kernel: [ 2633.539189] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0
Nov 20 19:35:16 thoregon kernel: [ 2633.539197] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150
Nov 20 19:35:16 thoregon kernel: [ 2633.539205] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370
Nov 20 19:35:16 thoregon kernel: [ 2633.539214] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330
Nov 20 19:35:16 thoregon kernel: [ 2633.539222] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100
Nov 20 19:35:16 thoregon kernel: [ 2633.539230] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.539238] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100
Nov 20 19:35:16 thoregon kernel: [ 2633.539246] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0
Nov 20 19:35:16 thoregon kernel: [ 2633.539254] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.539261] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Nov 20 19:35:16 thoregon kernel: [ 2633.539270] [<ffffffff81101bb1>] sys_unlink+0x11/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.539277] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:35:16 thoregon kernel: [ 2633.539282] x86_64-pc-linux D 0000000000000001 0 26904 1 0x00000004
Nov 20 19:35:16 thoregon kernel: [ 2633.539290] ffff880066bff858 0000000000000046 ffff880329301d00 ffff880066bfffd8
Nov 20 19:35:16 thoregon kernel: [ 2633.539299] ffff880066bfffd8 ffff880066bfffd8 ffff88013a7ce580 ffff880329301d00
Nov 20 19:35:16 thoregon kernel: [ 2633.539307] ffff880066bff928 ffff8801c572c830 7fffffffffffffff ffff880329301d00
Nov 20 19:35:16 thoregon kernel: [ 2633.539316] Call Trace:
Nov 20 19:35:16 thoregon kernel: [ 2633.539325] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.539333] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0
Nov 20 19:35:16 thoregon kernel: [ 2633.539340] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.539348] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.539356] [<ffffffff816b057e>] __down_common+0x97/0xe8
Nov 20 19:35:16 thoregon kernel: [ 2633.539365] [<ffffffff816b062e>] __down+0x18/0x1a
Nov 20 19:35:16 thoregon kernel: [ 2633.539371] [<ffffffff8106245b>] down+0x3b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.539377] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.539384] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200
Nov 20 19:35:16 thoregon kernel: [ 2633.539390] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130
Nov 20 19:35:16 thoregon kernel: [ 2633.539398] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0
Nov 20 19:35:16 thoregon kernel: [ 2633.539405] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460
Nov 20 19:35:16 thoregon kernel: [ 2633.539412] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.539419] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0
Nov 20 19:35:16 thoregon kernel: [ 2633.539426] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180
Nov 20 19:35:16 thoregon kernel: [ 2633.539435] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0
Nov 20 19:35:16 thoregon kernel: [ 2633.539444] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330
Nov 20 19:35:16 thoregon kernel: [ 2633.539452] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0
Nov 20 19:35:16 thoregon kernel: [ 2633.539460] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150
Nov 20 19:35:16 thoregon kernel: [ 2633.539468] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370
Nov 20 19:35:16 thoregon kernel: [ 2633.539477] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330
Nov 20 19:35:16 thoregon kernel: [ 2633.539484] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100
Nov 20 19:35:16 thoregon kernel: [ 2633.539493] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.539501] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100
Nov 20 19:35:16 thoregon kernel: [ 2633.539509] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0
Nov 20 19:35:16 thoregon kernel: [ 2633.539517] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.539524] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Nov 20 19:35:16 thoregon kernel: [ 2633.539532] [<ffffffff81101bb1>] sys_unlink+0x11/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.539540] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b
Nov 20 19:35:16 thoregon kernel: [ 2633.539544] x86_64-pc-linux D 0000000000000000 0 26907 1 0x00000004
Nov 20 19:35:16 thoregon kernel: [ 2633.539552] ffff8800c0eed858 0000000000000046 ffff8802a7be6580 ffff8800c0eedfd8
Nov 20 19:35:16 thoregon kernel: [ 2633.539560] ffff8800c0eedfd8 ffff8800c0eedfd8 ffff88032989e580 ffff8802a7be6580
Nov 20 19:35:16 thoregon kernel: [ 2633.539569] ffff8800c0eed928 ffff8801c55c7630 7fffffffffffffff ffff8802a7be6580
Nov 20 19:35:16 thoregon kernel: [ 2633.539578] Call Trace:
Nov 20 19:35:16 thoregon kernel: [ 2633.539587] [<ffffffff816b1224>] schedule+0x24/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.539595] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0
Nov 20 19:35:16 thoregon kernel: [ 2633.539602] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.539610] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.539618] [<ffffffff816b057e>] __down_common+0x97/0xe8
Nov 20 19:35:16 thoregon kernel: [ 2633.539627] [<ffffffff816b062e>] __down+0x18/0x1a
Nov 20 19:35:16 thoregon kernel: [ 2633.539633] [<ffffffff8106245b>] down+0x3b/0x50
Nov 20 19:35:16 thoregon kernel: [ 2633.539639] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60
Nov 20 19:35:16 thoregon kernel: [ 2633.539646] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200
Nov 20 19:35:16 thoregon kernel: [ 2633.539652] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130
Nov 20 19:35:16 thoregon kernel: [ 2633.539660] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0
Nov 20 19:35:16 thoregon kernel: [ 2633.539667] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460
Nov 20 19:35:16 thoregon kernel: [ 2633.539674] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.539681] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0
Nov 20 19:35:16 thoregon kernel: [ 2633.539688] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180
Nov 20 19:35:16 thoregon kernel: [ 2633.539697] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0
Nov 20 19:35:16 thoregon kernel: [ 2633.539706] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330
Nov 20 19:35:16 thoregon kernel: [ 2633.539714] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0
Nov 20 19:35:16 thoregon kernel: [ 2633.539722] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150
Nov 20 19:35:16 thoregon kernel: [ 2633.539730] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370
Nov 20 19:35:16 thoregon kernel: [ 2633.539738] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330
Nov 20 19:35:16 thoregon kernel: [ 2633.539746] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100
Nov 20 19:35:16 thoregon kernel: [ 2633.539754] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90
Nov 20 19:35:16 thoregon kernel: [ 2633.539763] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100
Nov 20 19:35:16 thoregon kernel: [ 2633.539771] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0
Nov 20 19:35:16 thoregon kernel: [ 2633.539779] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190
Nov 20 19:35:16 thoregon kernel: [ 2633.539786] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Nov 20 19:35:16 thoregon kernel: [ 2633.539794] [<ffffffff81101bb1>] sys_unlink+0x11/0x20
Nov 20 19:35:16 thoregon kernel: [ 2633.539802] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b