Re: [PATCH 2/2] PCI: pciehp: Disable Data Link Layer State Changed event on suspend

[Mika Westerberg]

Hi Bjorn,

Sorry for the delay - I was on a business trip.

On Mon, Jan 14, 2019 at 06:24:10PM -0600, Bjorn Helgaas wrote:
> On Mon, Jan 07, 2019 at 05:39:59PM +0300, Mika Westerberg wrote:
> > Commit 0e157e528604 ("PCI/PME: Implement runtime PM callbacks") tried to
> > solve an issue where the hierarchy immediately wakes up when it is
> > transitioned into D3cold. It turns out preventing PME propagation in
> > some PCIe hierarchies not supporting D3cold.
> 
> I can't quite parse this last sentence.  Is it missing a word?

No, but I'm not native speaker :)

> > I looked more closely what might cause the immediate wakeup. It happens
> > when the ACPI power resource of the root port is turned off. The AML
> > code associated with the _OFF() method of the ACPI power resource
> > executes PCIe L2/3 ready transition and waits it to complete.
> 
> waits *for* it ...


OK.

> > Right
> > after the L2/3 ready transition is started the root port receives PME
> > from the downstream port.
> > 
> > The simplest hierarchy where this happens looks like this:
> > 
> >   00:1d.0 PCIe Root port
> >     ^
> >     |
> >     v
> >     05:00.0 PCIe switch #1 upstream port
> >       06:01.0 PCIe switch #1 downstream hotplug port
> >         ^
> >         |
> >         v
> >         08:00.0 Pcie switch #2 upstream port
> > 
> > It seems that the PCIe link between the two switches, before
> > PME_Turn_Off/PME_TO_Ack is complete for the whole hierarchy, goes
> > inactive and triggers PME towards the root port bringing it back to D0.
> > Disabling Data Link Layer State Changed event (DLLSCE) prevents the
> > issue and still allows the downstream hotplug port to notice when a
> > device is plugged/unplugged.
> 
> I don't understand the "link goes inactive and triggers PME" part.  Is
> that prescribed by the spec, or is this implementation-specific
> behavior, or even a bug?

It is not directly described in the spec (unfortunately). In general
they leave a lot for the reader to interpret instead of explaining how
the thing is supposed to be programmed in different situations :(

This is the behavior I see when the hierarchy goes into D3cold. I don't
have any equipment that would let me see what exactly happens there. The
above is my guess of what happens.

> Are there spec references that would be useful here?  PCIe r4.0, sec
> 5.2 seems like one starting point.  5.3.1.4.2?  5.3.3.2.1 and
> following sections seem like they should be very relevant, but I
> haven't studied it in detail.

Sure, I can add those references to the changelog.

> > Link: https://bugzilla.kernel.org/show_bug.cgi?id=202103
> > Fixes: 0e157e528604 ("PCI/PME: Implement runtime PM callbacks")
> > Signed-off-by: Mika Westerberg <mika.westerberg@xxxxxxxxxxxxxxx>
> > ---
> >  drivers/pci/hotplug/pciehp_hpc.c | 10 ++++++++--
> >  1 file changed, 8 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/pci/hotplug/pciehp_hpc.c b/drivers/pci/hotplug/pciehp_hpc.c
> > index cd9eae650aa5..6fdaa8d48ebe 100644
> > --- a/drivers/pci/hotplug/pciehp_hpc.c
> > +++ b/drivers/pci/hotplug/pciehp_hpc.c
> > @@ -736,12 +736,18 @@ void pcie_clear_hotplug_events(struct controller *ctrl)
> >  
> >  void pcie_enable_interrupt(struct controller *ctrl)
> >  {
> > -	pcie_write_cmd(ctrl, PCI_EXP_SLTCTL_HPIE, PCI_EXP_SLTCTL_HPIE);
> > +	u16 mask;
> > +
> > +	mask = PCI_EXP_SLTCTL_HPIE | PCI_EXP_SLTCTL_DLLSCE;
> > +	pcie_write_cmd(ctrl, mask, mask);
> >  }
> >  
> >  void pcie_disable_interrupt(struct controller *ctrl)
> >  {
> > -	pcie_write_cmd(ctrl, 0, PCI_EXP_SLTCTL_HPIE);
> > +	u16 mask;
> > +
> 
> I think some sort of comment here would be useful.  It's pretty hard
> for the casual reader to figure out what things need to be masked.

Sure, I'll add a comment.

> > +	mask = PCI_EXP_SLTCTL_HPIE | PCI_EXP_SLTCTL_DLLSCE;
> > +	pcie_write_cmd(ctrl, 0, mask);
> >  }
> >  
> >  /*
> > -- 
> > 2.19.2
> >