RE: [PATCH v3] PM-runtime: fix deadlock with ktime

[Biju Das]

Hi Vincent,

I have rebased my kernel to "next-20190201".  Still I am seeing dead lock.

Am I missing any patch?

root@ek874:/# echo e61e0000.timer > /sys/devices/system/clocksource/clocksource0/current_clocksource
[  193.869423]
[  193.870963] ============================================
[  193.876292] WARNING: possible recursive locking detected
[  193.881625] 5.0.0-rc4-next-20190201-00007-g731346f #3 Not tainted
[  193.887737] --------------------------------------------
[  193.893066] migration/0/11 is trying to acquire lock:
[  193.898136] (____ptrval____) (tk_core.seq){----}, at: update_pm_runtime_accounting+0x14/0x68
[  193.906632]
[  193.906632] but task is already holding lock:
[  193.912483] (____ptrval____) (tk_core.seq){----}, at: multi_cpu_stop+0x8c/0x140
[  193.919828]
[  193.919828] other info that might help us debug this:
[  193.926377]  Possible unsafe locking scenario:
[  193.926377]
[  193.932314]        CPU0
[  193.934765]        ----
[  193.937216]   lock(tk_core.seq);
[  193.940453]   lock(tk_core.seq);
[  193.943691]
[  193.943691]  *** DEADLOCK ***
[  193.943691]
[  193.949634]  May be due to missing lock nesting notation
[  193.949634]
[  193.956446] 3 locks held by migration/0/11:
[  193.960642]  #0: (____ptrval____) (timekeeper_lock){-.-.}, at: change_clocksource+0x2c/0x118
[  193.969125]  #1: (____ptrval____) (tk_core.seq){----}, at: multi_cpu_stop+0x8c/0x140
[  193.976903]  #2: (____ptrval____) (&(&dev->power.lock)->rlock){....}, at: __pm_runtime_resume+0x40/0x98
[  193.986339]
[  193.986339] stack backtrace:
[  193.990715] CPU: 0 PID: 11 Comm: migration/0 Not tainted 5.0.0-rc4-next-20190201-00007-g731346f #3
[  193.999707] Hardware name: Silicon Linux RZ/G2E evaluation kit EK874 (CAT874 + CAT875) (DT)
[  194.008089] Call trace:
[  194.010553]  dump_backtrace+0x0/0x178
[  194.014227]  show_stack+0x14/0x20
[  194.017562]  dump_stack+0xb0/0xec
[  194.020895]  __lock_acquire+0xfb4/0x1c08
[  194.024832]  lock_acquire+0xd0/0x268
[  194.028420]  ktime_get+0x5c/0x108
[  194.031747]  update_pm_runtime_accounting+0x14/0x68
[  194.036643]  rpm_resume+0x4ec/0x698
[  194.040144]  __pm_runtime_resume+0x50/0x98
[  194.044264]  sh_tmu_enable.part.1+0x24/0x50
[  194.048462]  sh_tmu_clocksource_enable+0x48/0x70
[  194.053097]  change_clocksource+0x84/0x118
[  194.057208]  multi_cpu_stop+0x8c/0x140
[  194.060970]  cpu_stopper_thread+0xac/0x120
[  194.065087]  smpboot_thread_fn+0x1ac/0x2c8
[  194.069198]  kthread+0x128/0x130
[  194.072439]  ret_from_fork+0x10/0x18


Regards,
Biju

> -----Original Message-----
> From: Rafael J. Wysocki <rafael@xxxxxxxxxx>
> Sent: 30 January 2019 21:53
> To: Vincent Guittot <vincent.guittot@xxxxxxxxxx>
> Cc: Linux PM <linux-pm@xxxxxxxxxxxxxxx>; Linux Kernel Mailing List <linux-
> kernel@xxxxxxxxxxxxxxx>; Linux ARM <linux-arm-
> kernel@xxxxxxxxxxxxxxxxxxx>; Linux OMAP Mailing List <linux-
> omap@xxxxxxxxxxxxxxx>; Rafael J. Wysocki <rjw@xxxxxxxxxxxxx>; Ulf
> Hansson <ulf.hansson@xxxxxxxxxx>; Biju Das <biju.das@xxxxxxxxxxxxxx>;
> Geert Uytterhoeven <geert@xxxxxxxxxxxxxx>; Linux-Renesas <linux-
> renesas-soc@xxxxxxxxxxxxxxx>
> Subject: Re: [PATCH v3] PM-runtime: fix deadlock with ktime
>
> On Wed, Jan 30, 2019 at 6:26 PM Vincent Guittot
> <vincent.guittot@xxxxxxxxxx> wrote:
> >
> > A deadlock has been seen when swicthing clocksources which use PM
> runtime.
> > The call path is:
> > change_clocksource
> >     ...
> >     write_seqcount_begin
> >     ...
> >     timekeeping_update
> >         ...
> >         sh_cmt_clocksource_enable
> >             ...
> >             rpm_resume
> >                 pm_runtime_mark_last_busy
> >                     ktime_get
> >                         do
> >                             read_seqcount_begin
> >                         while read_seqcount_retry
> >     ....
> >     write_seqcount_end
> >
> > Although we should be safe because we haven't yet changed the
> > clocksource at that time, we can't because of seqcount protection.
> >
> > Use ktime_get_mono_fast_ns() instead which is lock safe for such case
> >
> > With ktime_get_mono_fast_ns, the timestamp is not guaranteed to be
> > monotonic across an update and as a result can goes backward.
> > According to
> > update_fast_timekeeper() description: "In the worst case, this can
> > result is a slightly wrong timestamp (a few nanoseconds)". For PM
> > runtime autosuspend, this means only that the suspend decision can be
> > slightly sub optimal.
> >
> > Fixes: 8234f6734c5d ("PM-runtime: Switch autosuspend over to using
> > hrtimers")
> > Reported-by: Biju Das <biju.das@xxxxxxxxxxxxxx>
> > Signed-off-by: Vincent Guittot <vincent.guittot@xxxxxxxxxx>
> > ---
> >
> > Hi Rafael,
> >
> > Sorry, I sent the version with the typo mistake that generated the
> > compilation error reported by kbuild-test-robot
> >
> > This version doesn't have the typo.
>
> OK, I've applied this one, thanks!



Renesas Electronics Europe Ltd, Dukes Meadow, Millboard Road, Bourne End, Buckinghamshire, SL8 5FH, UK. Registered in England & Wales under Registered No. 04586709.