Re: [PATCH -tip V3 0/8] workqueue: break affinity initiatively

[Jens Axboe]

On 1/11/21 12:21 PM, Valentin Schneider wrote:
> On 11/01/21 18:16, Peter Zijlstra wrote:
>> Sadly it appears like io_uring() uses kthread_create_on_cpu() without
>> then having any hotplug crud on, so that needs additinoal frobbing.
>>
> 
> I noticed that as well sometime ago, and I believed then (still do) this
> usage is broken. I don't think usage of kthread_create_on_cpu() outside
> of smpboot makes sense, because without any hotplug step to park the
> thread, its affinity can end up being reset after its dedicated CPU gets
> offlined.
> 
> I'm clueless about io_uring, but if it *actually* has a good reason to
> use some pcpu kthreads (it seems it doesn't have to be on all CPUs?),
> then it needs to register some hotplug step to park them / do something
> sensible on hotplug.

For io_uring, it's purely used by the SQPOLL mode, which sets aside a
kernel thread for submissions so the application doesn't have to do
anything but write new SQE entries to submit. The thread then notices
these, and submits them. There's an option to affinitize that thread
to a single CPU, which often makes sense for setups like that. Think
of it like a strong hint.

Things aren't going to break if this CPU goes away and we end up being
affinitized to some other CPU, though it is suboptimal. So I guess we
might need some notifiers to ensure that we reset the CPU back again
if it's gone offline+online again? I can take a look at that.

-- 
Jens Axboe