Re: [rfc patch v2] rt,nohz_full: fix nohz_full for PREEMPT_RT_FULL

From: Mike Galbraith
Date: Sat Apr 11 2015 - 09:16:14 EST


On Fri, 2015-04-10 at 16:15 +0200, Sebastian Andrzej Siewior wrote:
> * Mike Galbraith | 2015-03-13 05:53:25 [+0100]:
>
> > First of all, a task being ticked and trying to shut the tick down
> > will
> > fail to do so due to having just awakened ksoftirqd, so let
> > ksoftirqd
> > try to do that after SOFTIRQ_TIMER processing. Secondly, should
> > the
> > tick be shut down, we may livelock in hrtimer-cancel() because in -
> > rt
> > a callback may be running. Break the loop, and let
> > tick_nohz_restart()
> > know that the timer is busy so it can bail.
>
> So with this patch
> http://marc.info/?l=linux-rt-users&m=142866940327602&w=2


@@ -117,10 +113,8 @@ bool irq_work_queue(struct irq_work *wor
if (work->flags & IRQ_WORK_HARD_IRQ) {
if (llist_add(&work->llnode, this_cpu_ptr(&hirq_work_list)))
arch_irq_work_raise();
- } else {
- if (llist_add(&work->llnode, this_cpu_ptr(&lazy_list)))
- arch_irq_work_raise();
- }
+ } /* for lazy_list we have the timer irq */

If we don't queue work without IRQ_WORK_HARD_IRQ set, how does it run?

Anyway, box did not livelock, nor did it with that bit reverted.

> you could drop the hrtimer hacks. Is this the remaining part all
> that is
> required or we shorten the following patch a little more?

rt, nohz_full: fix nohz_full for PREEMPT_RT_FULL

A task being ticked and trying to shut the tick down will fail due
to having just awakened ksoftirqd, subtract it from nr_running.

Signed-off-by: Mike Galbraith <umgwanakikbuti@xxxxxxxxx>
---
kernel/sched/core.c | 17 ++++++++++++++++-
kernel/time/tick-sched.c | 5 +++++
2 files changed, 21 insertions(+), 1 deletion(-)

--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -781,14 +781,29 @@ static inline bool got_nohz_idle_kick(vo
#endif /* CONFIG_NO_HZ_COMMON */

#ifdef CONFIG_NO_HZ_FULL
+
+static int ksoftirqd_running(void)
+{
+ struct task_struct *softirqd;
+
+ if (!IS_ENABLED(CONFIG_PREEMPT_RT_FULL))
+ return 0;
+ softirqd = this_cpu_ksoftirqd();
+ if (softirqd && softirqd->on_rq)
+ return 1;
+ return 0;
+}
+
bool sched_can_stop_tick(void)
{
/*
* More than one running task need preemption.
* nr_running update is assumed to be visible
* after IPI is sent from wakers.
+ *
+ * NOTE, RT: if ksoftirqd is awake, subtract it.
*/
- if (this_rq()->nr_running > 1)
+ if (this_rq()->nr_running - ksoftirqd_running() > 1)
return false;

return true;
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -227,7 +227,12 @@ void __tick_nohz_full_check(void)

static void nohz_full_kick_work_func(struct irq_work *work)
{
+ unsigned long flags;
+
+ /* ksoftirqd processes sirqs with interrupts enabled */
+ local_irq_save(flags);
__tick_nohz_full_check();
+ local_irq_restore(flags);
}

static DEFINE_PER_CPU(struct irq_work, nohz_full_kick_work) = {
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/