Re: [PATCH v7 1/4] PCI: Introduce pcim_alloc_irq_vectors()

[Bjorn Helgaas]

On Fri, Jun 11, 2021 at 12:37:22PM +0300, Andy Shevchenko wrote:
> On Thu, Jun 10, 2021 at 05:41:43PM -0500, Bjorn Helgaas wrote:
> > On Mon, Jun 07, 2021 at 11:39:13PM +0800, Dejin Zheng wrote:
> > > Introduce pcim_alloc_irq_vectors(), a device-managed version of
> > > pci_alloc_irq_vectors(). Introducing this function can simplify
> > > the error handling path in many drivers.
> > > 
> > > And use pci_free_irq_vectors() to replace some code in pcim_release(),
> > > they are equivalent, and no functional change. It is more explicit
> > > that pcim_alloc_irq_vectors() is a device-managed function.
> 
> ...
> 
> > > @@ -1989,10 +1989,7 @@ static void pcim_release(struct device *gendev, void *res)
> > >  	struct pci_devres *this = res;
> > >  	int i;
> > >  
> > > -	if (dev->msi_enabled)
> > > -		pci_disable_msi(dev);
> > > -	if (dev->msix_enabled)
> > > -		pci_disable_msix(dev);
> > > +	pci_free_irq_vectors(dev);
> > 
> > If I understand correctly, this hunk is a nice simplification, but
> > actually has nothing to do with making pcim_alloc_irq_vectors().  I
> > have it split to a separate patch in my local tree.  Or am I wrong
> > about that?
> 
> It's a good simplification that had to be done when pci_free_irq_vectors()
> appeared.

Sorry to be pedantic.  You say the simplification "had to be done,"
but AFAICT there was no actual *requirement* for this simplification
to be done since pci_free_irq_vectors() is functionally identical to
the previous code.  I think we should do it because it's a little
simpler, but not because it *fixes* anything.

> But here is the fact that indirectly it's related to the pcim_*()
> APIs, i.e. pcim_alloc_irq_vectors(), because you may noticed this is inside
> pcim_release().

Yes.  For posterity, my notes about the call chain (after applying
this patch):

  pci_alloc_irq_vectors
    pci_alloc_irq_vectors_affinity
      __pci_enable_msix_range                 # MSI-X path
        __pci_enable_msix
          msix_capability_init
            msix_setup_entries
              for (...)
                entry = alloc_msi_entry
                  kzalloc(msi_desc)           <--- alloc
                  kmemdup(msi_desc->affinity) <--- alloc
            dev->msix_enabled = 1             # MSI-X enabled
      __pci_enable_msi_range                  # MSI path
        msi_capability_init
          msi_setup_entry
            alloc_msi_entry                   <--- alloc
          dev->msi_enabled = 1                # MSI enabled

  pcim_release
    pci_free_irq_vectors
      pci_disable_msix                        # MSI-X
        if (!dev->msix_enabled)
          return
        pci_msix_shutdown
          dev->msix_enabled = 0               # MSI-X disabled
        free_msi_irqs
          list_for_each_entry_safe(..., msi_list, ...)
            free_msi_entry
              kfree(msi_desc->affinity)       <--- free
              kfree(msi_desc)                 <--- free
      pci_disable_msi                         # MSI
        if (!dev->msi_enabled)
          return
        pci_msi_shutdown
          dev->msi_enabled = 0                # MSI disabled
        free_msi_irqs                         <--- free

So I *think* (correct me if I'm wrong):

  - If a driver calls pcim_enable_device(), we will call
    pcim_release() when the last reference to the device is dropped.

  - pci_alloc_irq_vectors() allocates msi_desc and irq_affinity_desc
    structures via msix_setup_entries() or msi_setup_entry().

  - pcim_release() will free those msi_desc and irq_affinity_desc
    structures.

  - Even before this series, pcim_release() frees msi_desc and
    irq_affinity_desc structures by calling pci_disable_msi() and
    pci_disable_msix().

  - Calling pci_free_irq_vectors() (or pci_disable_msi() or
    pci_disable_msix()) twice is unnecessary but probably harmless
    because they bail out early.

So this series actually does not fix any problems whatsoever.

It *does* remove unnecessary pci_free_irq_vectors() calls from
i2c-designware-pcidrv.c.

But because pci_alloc_irq_vectors() and related interfaces are
*already* managed as soon as a driver calls pcim_enable_device(),
we can simply remove the pci_free_irq_vectors() without doing anything
else.

I don't think we *should* do anything else.  There are many callers of
pcim_enable_device() that also call pci_alloc_irq_vectors(),
pci_enable_msix_range(), etc.  We don't have pcim_enable_msix_range(),
pcim_enable_msi(), pcim_alloc_irq_vectors_affinity(), etc.  I don't
think it's worth the churn of adding all those and changing all the
callers to use pcim_*() (as in patch 4/4 here).

Browsing the output of this:

  git grep -En "pcim_enable_device|pci_alloc_irq_vectors|pci_enable_msix_|pci_free_irq_vectors|pci_disable_msi"

leads me to believe there are similar calls of pci_free_irq_vectors()
that could be removed here:

  mtip_pci_probe
  sp_pci_probe
  dw_edma_pcie_probe
  hisi_dma_probe
  ioat_pci_probe
  plx_dma_probe
  cci_pci_probe
  hibmc_pci_probe
  ...

and many more, but I got tired of looking.

> > > +/**
> > > + * pcim_alloc_irq_vectors - a device-managed pci_alloc_irq_vectors()
> > > + * @dev:		PCI device to operate on
> > > + * @min_vecs:		minimum number of vectors required (must be >= 1)
> > > + * @max_vecs:		maximum (desired) number of vectors
> > > + * @flags:		flags or quirks for the allocation
> > > + *
> > > + * Return the number of vectors allocated, (which might be smaller than
> > > + * @max_vecs) if successful, or a negative error code on error. If less
> > > + * than @min_vecs interrupt vectors are available for @dev the function
> > > + * will fail with -ENOSPC.
> > > + *
> > > + * It depends on calling pcim_enable_device() to make IRQ resources
> > > + * manageable.
> > > + */
> > > +static inline int
> > > +pcim_alloc_irq_vectors(struct pci_dev *dev, unsigned int min_vecs,
> > > +			unsigned int max_vecs, unsigned int flags)
> > > +{
> > > +	if (!pci_is_managed(dev))
> > > +		return -EINVAL;
> > > +	return pci_alloc_irq_vectors(dev, min_vecs, max_vecs, flags);
> > 
> > This is great, but can you explain how pci_alloc_irq_vectors()
> > magically becomes a managed interface if we've already called
> > pcim_enable_device()?
> > 
> > I certainly believe it does; I'd just like to put a hint in the commit
> > log since my 5 minutes of grepping around didn't make it obvious to
> > me.
> > 
> > I see that pcim_enable_device() sets pdev->is_managed, but I didn't
> > find the connection between that and pci_alloc_irq_vectors().
> 
> One needs to read and understand the code, I agree. The explanation is spread
> between pcim_release() and __pci_enable_msi/x_range().
> 
> The call chain is
> 
> msi_capability_init() / msix_capability_init()
>   ...
>   <- __pci_enable_msi/x_range()
>     <- pci_alloc_irq_vectors_affinity()
>       <- pci_alloc_irq_vectors()
> 
> where device msi_enabled / msix_enabled is set.
> 
> So, it may deserve to be explained in the commit message.
> 
> > > +}
> 
> -- 
> With Best Regards,
> Andy Shevchenko
> 
>