Re: [RFC 1/2] softirq: Defer net rx/tx processing to ksoftirqd context

[Mike Galbraith]

On Fri, 2018-01-12 at 15:58 +0100, Frederic Weisbecker wrote:
> On Fri, Jan 12, 2018 at 06:23:08AM +0100, Mike Galbraith wrote:
> > On Thu, 2018-01-11 at 12:22 -0800, Linus Torvalds wrote:
> > > On Thu, Jan 11, 2018 at 12:16 PM, Eric Dumazet <edumazet@xxxxxxxxxx> wrote:
> > > >
> > > > Note that when I implemented TCP Small queues, I did experiments between
> > > > using a work queue or a tasklet, and workqueues added unacceptable P99
> > > > latencies, when many user threads are competing with kernel threads.
> > > 
> > > Yes.
> > > 
> > > So I think one solution might be to have a hybrid system, where we do
> > > the softirq's synchronously normally (which is what you really want
> > > for good latency).
> > > 
> > > But then fall down on a threaded model - but that fallback case should
> > > be per-softirq, not global. So if one softirq uses a lot of CPU time,
> > > that shouldn't affect the latency of other softirqs.
> > > 
> > > So maybe we could get rid of the per-cpu ksoftirqd entirely, and
> > > replace it with with per-cpu and per-softirq workqueues?
> > 
> > How would that be better than what RT used to do, and I still do for my
> > RT kernels via boot option, namely split ksoftirqd into per-softirq
> > threads.
> 
> Workqueue are probably more simple. Unless you need to set specific prios
> to your ksoftirqds? Not sure if that's tunable on workqueues.

No, you can't prioritize workqueues, and they spawn threads whenever
they bloody well feel like.

I carry a hack to give users minimal control over kthread/workqueue
priority.  Very handy thing to have, especially if you're doing high
utilization stuff, and would prefer your box actually survive it.

	-Mike