Re: [PATCH v4 1/2] driver core: Introduce device_link_wait_removal()
From: Nuno Sá
Date: Wed Mar 06 2024 - 09:09:16 EST
On Wed, 2024-03-06 at 14:05 +0100, Rafael J. Wysocki wrote:
> On Wed, Mar 6, 2024 at 2:01 PM Nuno Sá <noname.nuno@xxxxxxxxx> wrote:
> >
> > On Wed, 2024-03-06 at 13:43 +0100, Rafael J. Wysocki wrote:
> > > On Wed, Mar 6, 2024 at 10:17 AM Nuno Sá <noname.nuno@xxxxxxxxx> wrote:
> > > >
> > > > On Wed, 2024-03-06 at 09:50 +0100, Herve Codina wrote:
> > > > > The commit 80dd33cf72d1 ("drivers: base: Fix device link removal")
> > > > > introduces a workqueue to release the consumer and supplier devices
> > > > > used
> > > > > in the devlink.
> > > > > In the job queued, devices are release and in turn, when all the
> > > > > references to these devices are dropped, the release function of the
> > > > > device itself is called.
> > > > >
> > > > > Nothing is present to provide some synchronisation with this workqueue
> > > > > in order to ensure that all ongoing releasing operations are done and
> > > > > so, some other operations can be started safely.
> > > > >
> > > > > For instance, in the following sequence:
> > > > > 1) of_platform_depopulate()
> > > > > 2) of_overlay_remove()
> > > > >
> > > > > During the step 1, devices are released and related devlinks are
> > > > > removed
> > > > > (jobs pushed in the workqueue).
> > > > > During the step 2, OF nodes are destroyed but, without any
> > > > > synchronisation with devlink removal jobs, of_overlay_remove() can
> > > > > raise
> > > > > warnings related to missing of_node_put():
> > > > > ERROR: memory leak, expected refcount 1 instead of 2
> > > > >
> > > > > Indeed, the missing of_node_put() call is going to be done, too late,
> > > > > from the workqueue job execution.
> > > > >
> > > > > Introduce device_link_wait_removal() to offer a way to synchronize
> > > > > operations waiting for the end of devlink removals (i.e. end of
> > > > > workqueue jobs).
> > > > > Also, as a flushing operation is done on the workqueue, the workqueue
> > > > > used is moved from a system-wide workqueue to a local one.
> > > > >
> > > > > Fixes: 80dd33cf72d1 ("drivers: base: Fix device link removal")
> > > > > Cc: stable@xxxxxxxxxxxxxxx
> > > > > Signed-off-by: Herve Codina <herve.codina@xxxxxxxxxxx>
> > > > > ---
> > > >
> > > > With the below addressed:
> > > >
> > > > Reviewed-by: Nuno Sa <nuno.sa@xxxxxxxxxx>
> > > >
> > > > > drivers/base/core.c | 26 +++++++++++++++++++++++---
> > > > > include/linux/device.h | 1 +
> > > > > 2 files changed, 24 insertions(+), 3 deletions(-)
> > > > >
> > > > > diff --git a/drivers/base/core.c b/drivers/base/core.c
> > > > > index d5f4e4aac09b..48b28c59c592 100644
> > > > > --- a/drivers/base/core.c
> > > > > +++ b/drivers/base/core.c
> > > > > @@ -44,6 +44,7 @@ static bool fw_devlink_is_permissive(void);
> > > > > static void __fw_devlink_link_to_consumers(struct device *dev);
> > > > > static bool fw_devlink_drv_reg_done;
> > > > > static bool fw_devlink_best_effort;
> > > > > +static struct workqueue_struct *device_link_wq;
> > > > >
> > > > > /**
> > > > > * __fwnode_link_add - Create a link between two fwnode_handles.
> > > > > @@ -532,12 +533,26 @@ static void devlink_dev_release(struct device
> > > > > *dev)
> > > > > /*
> > > > > * It may take a while to complete this work because of the SRCU
> > > > > * synchronization in device_link_release_fn() and if the
> > > > > consumer
> > > > > or
> > > > > - * supplier devices get deleted when it runs, so put it into the
> > > > > "long"
> > > > > - * workqueue.
> > > > > + * supplier devices get deleted when it runs, so put it into the
> > > > > + * dedicated workqueue.
> > > > > */
> > > > > - queue_work(system_long_wq, &link->rm_work);
> > > > > + queue_work(device_link_wq, &link->rm_work);
> > > > > }
> > > > >
> > > > > +/**
> > > > > + * device_link_wait_removal - Wait for ongoing devlink removal jobs
> > > > > to
> > > > > terminate
> > > > > + */
> > > > > +void device_link_wait_removal(void)
> > > > > +{
> > > > > + /*
> > > > > + * devlink removal jobs are queued in the dedicated work queue.
> > > > > + * To be sure that all removal jobs are terminated, ensure that
> > > > > any
> > > > > + * scheduled work has run to completion.
> > > > > + */
> > > > > + flush_workqueue(device_link_wq);
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(device_link_wait_removal);
> > > > > +
> > > > > static struct class devlink_class = {
> > > > > .name = "devlink",
> > > > > .dev_groups = devlink_groups,
> > > > > @@ -4099,9 +4114,14 @@ int __init devices_init(void)
> > > > > sysfs_dev_char_kobj = kobject_create_and_add("char", dev_kobj);
> > > > > if (!sysfs_dev_char_kobj)
> > > > > goto char_kobj_err;
> > > > > + device_link_wq = alloc_workqueue("device_link_wq", 0, 0);
> > > > > + if (!device_link_wq)
> > > > > + goto wq_err;
> > > > >
> > > >
> > > > I can't still agree with this. Why not doing it in devlink_class_init()?
> > > > This is
> > > > devlink specific so it makes complete sense to me.
> > >
> > > If you do that in devlink_class_init() and it fails, you essentially
> > > cause the creation of every device link to fail. IOW, you try to live
> > > without device links and pretend that it is all OK. That won't get
> > > you very far, especially on systems where DT is used.
> > >
> > > Doing it here, if it fails, you prevent the driver model from working
> > > at all (because one of its necessary components is unavailable), which
> > > arguably is a better choice.
> >
> > That makes sense but then the only thing I still don't fully get is why we
> > have
> > a separate devlink_class_init() initcall for registering the devlink class
> > (which can also fail)...
>
> Well, I haven't added it. :-)
>
> > What I take from the above is that we should fail the
> > driver model if one of it's fundamental components fails so I would say we
> > should merge devlink_class_init() with device_init() otherwise it's a bit
> > confusing (at least to me) and gives the idea that it's ok for the driver
> > model
> > to exist without the links (unless I'm missing some other reason for the
> > devlink
> > init function).
>
> +1
>
> Feel free to send a patch along these lines, chances are that it will
> be popular. ;-)
I was actually thinking about that but I think I encountered the reason why we
have it like this... devices_init() is called from driver_init() and there we
have:
..
devices_init();
buses_init();
classes_init();
..
So classes are initialized after devices which means we can't really do
class_register(&devlink_class) from devices_init(). Unless, of course, we re-
order things in driver_init() but that would be a questionable change at the
very least.
So, while I agree with what you've said, I'm still not sure if mixing devlink
stuff between devices_init() and devlink_class_init() is the best thing to do
given that we already have the case where devlink_class_init() can fail while
the driver model is up.
- Nuno Sá