Re: [RFC/RFT PATCH 1/2] sched/core: Check and schedule ksoftirq

[Peter Zijlstra]

On Mon, Dec 19, 2022 at 12:46:54PM +0100, Sebastian Andrzej Siewior wrote:
> On 2022-12-19 12:33:22 [+0100], Peter Zijlstra wrote:
> > ksoftirq is typically a CFS task while idle injection is required to be
> > a FIFO (typically FIFO-1) task -- so that would require lifting
> > ksoftirqd to FIFO and that has other problems.
> > 
> > I'm guessing the problem case is where idle injection comes in while
> > ksoftirq is running (and preempted), because in that case you cannot run
> > the softirq stuff in-place.
> > 
> > The 'right' thing to do would be to PI boost ksoftirqd in that case I
> > suppose. Perhaps something like so, it would boost ksoftirq when it's
> > running, and otherwise runs the ksoftirqd thing in-situ.
> > 
> > I've not really throught hard about this though, so perhaps I completely
> > wrecked things.
> 
> I don't know why you intend to run ksoftirqd but in general it breaks RT

So the upstream problem is where softirq is punted to ksoftirq (obvious
from hardirq tail) and idle-injection comes in and either:

 - runs before ksoftirq had a chance to run, or
 - preempts ksoftirqd.

In both cases we'll appear to go idle with softirqs pending -- which
triggers a pesky warning, because obviously undesirable.

In the first case we can run 'ksoftirqd' in-context, but then need to
serialize against the actual ksoftirqd. In the second case we need to
serialize against ksoftirqd and ensure it is complete before proceeding
with going 'idle'.

Since idle-injection runs FIFO and ksoftirqd typically does not, some
form of PI is required.

> left and right and we attempt to avoid running ksoftirqd as much as
> possible. If it runs and you go and boost it then it probably gets even
> worse from RT point of view.

So if you never run ksoftirqd, the above problem doesn't happen. If
ksoftirqd *can* run, you still need to deal with it somehow. Boosting
ksoftirqd to whatever priority the idle-injection thread has should not
be worse than anything the idle-injection already does, no?

Specifically, without idle-injection involved the patch as proposed (+-
fixes required to make it compile and work) should be a no-op.

> ksoftirqd runs softirqs from hardirq context. Everything else is handled
> in is handled within local-bh-disable+enable loop. We already have have
> the boost-ksoftird hammer which is the per-CPU BLK called
> local_bh_disable(). In general everything should be moved out of it.
> For timers we have the ktimerd thread which needs clean up.