Re: [PATCH] RDMA/device: Fix a race between mad_client and cm_client init

[Ding Hui]

On 2024/1/4 20:37, Jason Gunthorpe wrote:
> On Thu, Jan 04, 2024 at 02:48:14PM +0800, Shifeng Li wrote:
>
> > The root cause is that mad_client and cm_client may init concurrently
> > when devices_rwsem write semaphore is downgraded in enable_device_and_get() like:
>
> That can't be true, the module loader infrastructue ensures those two
> things are sequential.

Please consider the sequence again and notice that:

1. We agree that dependencies ensure mad_client be registered before cm_client.
2. But the mad_client.add() is not invoked in ib_register_client(), since
   there is no DEVICE_REGISTERED device at that time.
   Instead, it will be delayed until the device driver init (e.g. mlx5_core)
   in enable_device_and_get().
3. The ib_cm and mlx5_core can be loaded concurrently, after setting DEVICE_REGISTERED
   and downgrade_write(&devices_rwsem) in enable_device_and_get(), there is a chance
   that cm_client.add() can be invoked before mad_client.add().


        T1(ib_core init)      |      T2(device driver init)        |        T3(ib_cm init)
---------------------------------------------------------------------------------------------------
ib_register_client mad_client
  assign_client_id
    add clients CLIENT_REGISTERED
    (with clients_rwsem write)
  down_read(&devices_rwsem);
  xa_for_each_marked (&devices, DEVICE_REGISTERED)
    nop # no devices
  up_read(&devices_rwsem);

                              ib_register_device
                                enable_device_and_get
                                  down_write(&devices_rwsem);
                                  set DEVICE_REGISTERED
                                  downgrade_write(&devices_rwsem);
                                                                    ib_register_client cm_client
                                                                      assign_client_id
                                                                        add clients CLIENT_REGISTERED
                                                                        (with clients_rwsem write)
                                                                      down_read(&devices_rwsem);
                                                                      xa_for_each_marked (&devices, DEVICE_REGISTERED)
                                                                        add_client_context
                                                                          down_write(&device->client_data_rwsem);
                                                                          get CLIENT_DATA_REGISTERED
                                                                          downgrade_write(&device->client_data_rwsem);
                                                                          cm_client.add
                                                                            cm_add_one
                                                                              ib_register_mad_agent
                                                                                ib_get_mad_port
                                                                                  __ib_get_mad_port return NULL!
                                                                          set CLIENT_DATA_REGISTERED
                                                                          up_read(&device->client_data_rwsem);
                                                                      up_read(&devices_rwsem);
                                down_read(&clients_rwsem);
                                xa_for_each_marked (&clients, CLIENT_REGISTERED)
                                  add_client_context [mad]
                                    mad_client.add
                                  add_client_context [cm]
                                    nop # already CLIENT_DATA_REGISTERED
                                up_read(&clients_rwsem);
                                up_read(&devices_rwsem);

> You are trying to say that the post-client fixup stuff will still see
> the DEVICE_REGISTERED before it reaches the clients_rwsem lock?
>
> That probably just says the clients_rwsem should be obtained before
> changing the DEVICE_STATE too :\


--
Thanks,
- Ding Hui