Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

[Paul E. McKenney]

On Thu, Jun 11, 2020 at 12:12:46AM +0200, Frederic Weisbecker wrote:
> On Wed, Jun 10, 2020 at 07:02:10AM -0700, Paul E. McKenney wrote:
> > And just to argue against myself...
> > 
> > Another approach is to maintain explicit multiple states for each
> > ->cblist, perhaps something like this:
> > 
> > 1.	In softirq.  Transition code advances to next.
> > 2.	To no-CB 1.  Either GP or CB kthread for the transitioning
> > 	CPU advances to next.  Note that the fact that the
> > 	transition code runs on the transitioning CPU means that
> > 	the RCU softirq handler doesn't need to be involved.
> > 3.	To no-CB 2.  Either GP or CB kthread for the transitioning
> > 	CPU advances to next.
> 
> Just to clarify, if GP has set NO_CB2 in (2), we want CB to set NO_CB3
> in 3), right? OTOH if CB has set NO_CB2 in (2), we want GP to set NO_CB3
> in (3), right?
> 
> The point being to make sure that both threads acknowledge the transition?

Exactly!

> > 4.	To no-CB 3.  Transitioning code advances to next.
> > 	At this point, the no-CBs setup is fully functional.
> 
> And softirq can stop processing callbacks from that point on.

You got it!

> > 5.	No-CB.  Transitioning code advances to next.
> > 	Again, the fact that the transitioning code is running
> > 	on the transitioning CPU with interrupts disabled means
> > 	that the RCU softirq handler need not be explicitly
> > 	involved.
> > 6.	To softirq 1.  The RCU softirq handler for the transitioning
> > 	CPU advances to next.
> > 	At this point, the RCU softirq handler is processing callbacks.
> > 7.	To softirq 2.  Either GP or CB kthread for the transitioning
> > 	CPU advances to next.
> > 	At this point, the softirq handler is processing callbacks.
> 
> SOFTIRQ2 should be part of what happens in SOFTIRQ1. The transitioning
> CPU sets SOFTIRQ1, which is immediately visible by local softirqs,
> and wakes up CB/GP. CB/GP sets SOFTIRQ2, CB/GP sets SOFTIRQ3 and
> we go back to transitioning code that sets IN_SOFTIRQ.
> 
> Or did I miss something?

I was perhaps being overly paranoid.  You might well be right.

> > 8.	To softirq 3.  Either GP or CB kthread for the transitioning
> > 	CPU advances to next.
> > 	At this point, the no-CBs setup is fully shut down.
> > 9.	To softirq 4.  Transitioning code advances to next,
> > 	which is the first, "In softirq".
> > 	(This one -might- be unnecessary, but...)
> > 
> > All transitions are of course with the ->nocb_lock held.
> > 
> > When there is only one CPU during early boot near rcu_init() time,
> > the transition from "In softirq" to "No-CB" can remain be instantaneous.
> > 
> > This has the advantage of not slowing things down just because there
> > is an RCU callback flood in progress.  It also uses an explicit
> > protocol that should (give or take bugs) maintain full safety both
> > in protection of ->cblist and in dealing with RCU callback floods.
> > 
> > Thoughts?
> 
> Agreed. And I really like that it details the whole process in a very
> explicit way.
> 
> Thanks!

Glad you like it!  And of course please adjust it as needed, up to and
including doing something completely different that works better.  ;-)

							Thanx, Paul