Re: [RFC PATCH 0/6] CPU hotplug: Reverse invocation of notifiersduring CPU hotplug

[Srivatsa S. Bhat]

On 07/25/2012 08:27 PM, Alan Stern wrote:
> On Wed, 25 Jul 2012, Srivatsa S. Bhat wrote:
> 
>> Hi,
>>
>> This patchset implements the approach of invoking the CPU hotplug callbacks
>> (notifiers) in one order during CPU online and in the reverse order during CPU
>> offline. The rationale behind this is that services for a CPU are started in a
>> particular order (perhaps, with implicit dependencies between them) while
>> bringing up the CPU, and hence, it makes sense to tear down the services in
>> the opposite order, thereby honoring most of the dependencies automatically
>> (and also correctly). This is explained in more detail in Patch 6.
> 
> This strongly suggests that a notifier chain may be the wrong mechanism
> to use here.  Notifiers provide only limited guarantees about ordering,
> and it's hard to say much about the services a particular chain will
> provide since callbacks can be added from anywhere.
> 

True, the ability to register any random callback from anywhere is still a
problem that we are fighting... The zillions of callbacks that we have today
makes the hotplug process quite entangled.. we can't even roll-back from a
failure easily!

> Instead of adding all this complication to the notifier mechanism, how 
> about using something else for CPU hotplug?
> 

The problem is that today, many different subsystems need to know about CPUs coming
up or going down.. And CPU hotplug is not atomic, it happens in stages, and the
coordination between those subsystems is what actually drives CPU hotplug, in a way.
At present, I think that the best we can do is to redesign the hotplug code such that
the number of callbacks that are needed can be reduced to a minimum amount and then
have good control over what those callbacks do. For example, Thomas Gleixner posted
the park/unpark patchset[1], which not only speeds-up CPU hotplug by avoiding destruction
and creation of per-cpu kthreads on every hotplug operation, but also gets rid of quite
a few notifiers by providing a framework to manage those per-cpu kthreads...

One of the other ideas to improve the hotplug notifier stuff that came up during some
of the discussions was to implement explicit dependency tracking between the notifiers
and perhaps get rid of the priority numbers that are currently being used to provide
some sort of ordering between the callbacks. Links to some of the related discussions
are provided below.

References:
[1]. http://thread.gmane.org/gmane.linux.kernel/1328391/focus=1328391
[2]. https://lkml.org/lkml/2012/4/5/379
[3]. http://thread.gmane.org/gmane.linux.kernel.cross-arch/13656/focus=13678
[4]. http://thread.gmane.org/gmane.linux.kernel.cross-arch/13656/focus=14208
[5]. http://thread.gmane.org/gmane.linux.kernel.cross-arch/13656/focus=14214 

Regards,
Srivatsa S. Bhat

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/