Re: [PATCH v4 1/3] cacheinfo: Add arch specific early level initializer

[Radu Rendec]

On Wed, 2023-05-10 at 12:12 -0700, Ricardo Neri wrote:
> On Wed, Apr 12, 2023 at 02:57:57PM -0400, Radu Rendec wrote:
> > This patch gives architecture specific code the ability to initialize
> > the cache level and allocate cacheinfo memory early, when cache level
> > initialization runs on the primary CPU for all possible CPUs.
[cut]
> > -int detect_cache_attributes(unsigned int cpu)
> > +static inline int init_level_allocate_ci(unsigned int cpu)
> >  {
> > -       int ret;
> > +       unsigned int early_leaves = cache_leaves(cpu);
> >  
> >         /* Since early initialization/allocation of the cacheinfo is allowed
> >          * via fetch_cache_info() and this also gets called as CPU hotplug
> >          * callbacks via cacheinfo_cpu_online, the init/alloc can be skipped
> >          * as it will happen only once (the cacheinfo memory is never freed).
> > -        * Just populate the cacheinfo.
> > +        * Just populate the cacheinfo. However, if the cacheinfo has been
> > +        * allocated early through the arch-specific early_cache_level() call,
> > +        * there is a chance the info is wrong (this can happen on arm64). In
> > +        * that case, call init_cache_level() anyway to give the arch-specific
> > +        * code a chance to make things right.
> >          */
> > -       if (per_cpu_cacheinfo(cpu))
> > -               goto populate_leaves;
> > +       if (per_cpu_cacheinfo(cpu) && !ci_cacheinfo(cpu)->early_ci_levels)
> > +               return 0;
> >  
> >         if (init_cache_level(cpu) || !cache_leaves(cpu))
> >                 return -ENOENT;
> >  
> > -       ret = allocate_cache_info(cpu);
> > +       /*
> > +        * Now that we have properly initialized the cache level info, make
> > +        * sure we don't try to do that again the next time we are called
> > +        * (e.g. as CPU hotplug callbacks).
> > +        */
> > +       ci_cacheinfo(cpu)->early_ci_levels = false;
> > +
> > +       if (cache_leaves(cpu) <= early_leaves)
> > +               return 0;
> > +
> 
> I had posted a patchset[1] for x86 that initializes
> ci_cacheinfo(cpu)->num_leaves during SMP boot.
> 
> This means that early_leaves and a late cache_leaves() are equal but
> per_cpu_cacheinfo(cpu) is never allocated. Currently, x86 does not use
> fetch_cache_info().
> 
> I think that we should check here that per_cpu_cacheinfo() has been allocated to
> take care of the case in which early and late cache leaves remain the same:
> 
> -       if (cache_leaves(cpu) <= early_leaves)
> +       if (cache_leaves(cpu) <= early_leaves && per_cpu_cacheinfo(cpu))
> 
> Otherwise, in v6.4-rc1 + [1] I observe a NULL pointer dereference from
> last_level_cache_is_valid().
> 
> I can post a patch with this fix if it makes sense.
> 
> [1]. https://lore.kernel.org/all/20230424001956.21434-3-ricardo.neri-calderon@xxxxxxxxxxxxxxx/

Hi Ricardo,

Thanks for bringing this to my attention. I need to run some tests on
x86 (I did all that work/testing on arm64) and wrap my head around it.

While I don't see any problem with the fix you're proposing, I'm afraid
it may circle back to the other problem I tried to fix initially. Have
you tested this on an RT kernel by any chance?

I'm thinking that if we end up in init_level_allocate_ci() without the
cacheinfo memory having been allocated earlier, we're up for a "BUG"
splat on RT kernels.

If early_leaves has the right value at that point, the cacheinfo memory
should be allocated early (on the primary CPU), so perhaps there's a
different problem somewhere else.

I'll get back to you as soon as I look at this in more detail but I
just wanted to give you a quick heads-up.

Regards,
Radu