Re: [PATCH 00/17] Backport rt/deadline crash and the ardous story of FUTEX_UNLOCK_PI to 4.4

[Henrik Austad]

On Fri, Nov 09, 2018 at 11:35:31AM +0100, Henrik Austad wrote:
> On Fri, Nov 09, 2018 at 11:07:28AM +0100, Henrik Austad wrote:
> > From: Henrik Austad <haustad@xxxxxxxxx>
> > 
> > Short story:
> 
> Sorry for the spam, it looks like I was not very specific in /which/ 
> version I targeted this to, as well as not providing a full Cc-list for the 
> cover-letter.

Gentle prod. I realize this was sent out just before plumbers and that 
people had pretty packed agendas, so a small nudge to gain a spot closer to 
the top of the inbox :)

This series has now been running on an arm64 system for 9 days without any 
issues and pi_stress showed a dramatic improvement from ~30 seconds and up 
to several ours (it finally deadlocked at 3.9e9 inversions).

I'd greatly appreciate if someone could give the list of patches a quick 
glance to verify that I got all the required patches and then if it could 
be added to 4.4.y.

Thanks!

-Henrik


> The series is targeted at stable v4.4.162.
> 
> Expanding Cc-list to those missing from the first attempt.
> 
> -Henrik
> 
> > The following patches are needed on a 4.4 kernel to avoid
> > Oops in the scheduler when a sched_rr and sched_deadline task contends
> > on the same futex (with PI).
> > 
> > Longer story:
> > 
> > On one of our arm64 systems, we occasionally crash with an Oops in the
> > scheduler with the following backtrace.
> > 
> > [<ffffffc0000ee398>] enqueue_task_dl+0x1f0/0x420
> > [<ffffffc0000d0f14>] activate_task+0x7c/0x90
> > [<ffffffc0000edbdc>] push_dl_task+0x164/0x1c8
> > [<ffffffc0000edc60>] push_dl_tasks+0x20/0x30
> > [<ffffffc0000cc00c>] __balance_callback+0x44/0x68
> > [<ffffffc000d2c018>] __schedule+0x6f0/0x728
> > [<ffffffc000d2c278>] schedule+0x78/0x98
> > [<ffffffc000d2e76c>] __rt_mutex_slowlock+0x9c/0x108
> > [<ffffffc000d2e9d0>] rt_mutex_slowlock+0xd8/0x198
> > [<ffffffc0000f7f28>] rt_mutex_timed_futex_lock+0x30/0x40
> > [<ffffffc00012c1a8>] futex_lock_pi+0x200/0x3b0
> > [<ffffffc00012cf84>] do_futex+0x1c4/0x550
> > [<ffffffc00012d92c>] compat_SyS_futex+0x10c/0x138
> > [<ffffffc00008504c>] __sys_trace_return+0x0/0x4
> > 
> > This seems to be the same bug Xuneli Pang triggered and fixed in
> > e96a7705e7d3 "sched/rtmutex/deadline: Fix a PI crash for deadline
> > tasks". As noted by Peter Zijlstra in the previous attempt, this fix
> > requires a few other patches, most notably the FUTEX_UNLOCK_PI series
> > [1]
> > 
> > Testing this on a dual-core VM I have not been able to reproduce the
> > same crash, but pi_stress (part of the rt-test suite) reveals that
> > vanilla 4.4.162 behaves rather badly with a mix of deadline and
> > sched_(rr|fifo) tasks:
> > 
> > time pi_stress --rr --mlockall --sched id=high,policy=deadline,runtime=100000,deadline=200000,period=200000
> > Starting PI Stress Test
> > Number of thread groups: 1
> > Duration of test run: infinite
> > Number of inversions per group: unlimited
> >      Admin thread SCHED_RR priority 4
> > 1 groups of 3 threads will be created
> >       High thread SCHED_DEADLINE runtime 100000 deadline 200000 period 200000
> >        Med thread SCHED_RR priority 2
> >        Low thread SCHED_RR priority 1
> > Current Inversions: 141627
> > WATCHDOG triggered: group 0 is deadlocked!
> > reporter stopping due to watchdog event
> > Stopping test
> > Terminated
> > 
> > real    0m26.291s
> > user    0m0.148s
> > sys     0m18.819s
> > 
> > With this series applied, the test ran for ~4.5 hours and again for 129
> > minutes (when I remembered to time it) before crashing:
> > 
> > time pi_stress --rr --mlockall --sched id=high,policy=deadline,runtime=100000,deadline=200000,period=200000
> > Starting PI Stress Test
> > Number of thread groups: 1
> > Duration of test run: infinite
> > Number of inversions per group: unlimited
> >      Admin thread SCHED_RR priority 4
> > 1 groups of 3 threads will be created
> >       High thread SCHED_DEADLINE runtime 100000 deadline 200000 period 200000
> >        Med thread SCHED_RR priority 2
> >        Low thread SCHED_RR priority 1
> > Current Inversions: 51985223
> > WATCHDOG triggered: group 0 is deadlocked!
> > reporter stopping due to watchdog event
> > Stopping test
> > Terminated
> > 
> > real    129m38.807s
> > user    0m59.084s
> > sys     109m53.666s
> > 
> > 
> > So clearly not perfect, but a *lot* better.
> > 
> > The same series on our vendor-4.4 kernel moves pi_stress up from ~30
> > seconds before deadlock up to the same level as the VM (the test is
> > still going as of this writing).
> > 
> > I suspect other users of 4.4 would benefit from having these patches
> > backported, so tag them for stable. I assume 4.9 and 4.14 could benefit
> > as well, but I have not had time to look into those.
> > 
> > 1) https://www.mail-archive.com/linux-kernel@xxxxxxxxxxxxxxx/msg1359667.html
> > 
> > Peter Zijlstra (13):
> >   futex: Cleanup variable names for futex_top_waiter()
> >   futex: Use smp_store_release() in mark_wake_futex()
> >   futex: Remove rt_mutex_deadlock_account_*()
> >   futex,rt_mutex: Provide futex specific rt_mutex API
> >   futex: Change locking rules
> >   futex: Cleanup refcounting
> >   futex: Rework inconsistent rt_mutex/futex_q state
> >   futex: Pull rt_mutex_futex_unlock() out from under hb->lock
> >   futex,rt_mutex: Introduce rt_mutex_init_waiter()
> >   futex,rt_mutex: Restructure rt_mutex_finish_proxy_lock()
> >   futex: Rework futex_lock_pi() to use rt_mutex_*_proxy_lock()
> >   futex: Futex_unlock_pi() determinism
> >   futex: Drop hb->lock before enqueueing on the rtmutex
> > 
> > Thomas Gleixner (2):
> >   rtmutex: Make wait_lock irq safe
> >   futex: Rename free_pi_state() to put_pi_state()
> > 
> > Xunlei Pang (2):
> >   rtmutex: Deboost before waking up the top waiter
> >   sched/rtmutex/deadline: Fix a PI crash for deadline tasks
> > 
> >  include/linux/init_task.h       |   1 +
> >  include/linux/sched.h           |   2 +
> >  include/linux/sched/rt.h        |   1 +
> >  kernel/fork.c                   |   1 +
> >  kernel/futex.c                  | 532 ++++++++++++++++++++++++++--------------
> >  kernel/locking/rtmutex-debug.c  |   9 -
> >  kernel/locking/rtmutex-debug.h  |   3 -
> >  kernel/locking/rtmutex.c        | 406 ++++++++++++++++++------------
> >  kernel/locking/rtmutex.h        |   2 -
> >  kernel/locking/rtmutex_common.h |  24 +-
> >  kernel/sched/core.c             |   2 +
> >  11 files changed, 620 insertions(+), 363 deletions(-)
> > 
> > -- 
> > 2.7.4
> > 
> 
> -- 
> Henrik Austad



-- 
Henrik Austad

Attachment: signature.asc
Description: PGP signature