Re: [PATCH v2 1/3] usb: notify hcd when USB device suspend or resume

From: Lu, Baolu
Date: Tue May 12 2015 - 22:36:42 EST




On 05/12/2015 11:54 PM, Alan Stern wrote:
On Tue, 12 May 2015, Lu, Baolu wrote:

I'm sorry that I confused you.

FSC is a different thing from what this patch series does.
I know that. The patch series, in its current form, is fine. Now I'm
trying to understand what you originally wanted to do.

Let's see if I understand it correctly:

When the controller goes into suspend, you want the cache to
be pushed into memory so that the cache can be powered down,
thereby saving additional energy.
Not the controller goes into suspend, but USB devices.
Suppose a USB device goes into runtime suspend, and suppose the
hardware pushes the cached information for the endpoints on that device
into memory. The cache will still contain information for other,
non-suspended devices. Consequently the cache _can't_ be powered down
at this time, right? You seem to be saying:

When a USB device goes into suspend, you want the cache to
be pushed into memory so that the cache can be powered down,
thereby saving additional energy.

But that is just wrong! The only time you know the hardware can safely
power-down the cache is when the _controller_ goes into suspend. At
that time you _know_ all the attached devices are suspended, so the
cache isn't needed.

Therefore, when the controller goes into suspend, you want to make sure
that the cache has been pushed into memory. The actual push operation
can occur earlier, during xhci_device_suspend, but the time when the
information's location matters is during xhci_suspend.

In order to talking to USB devices, xHCI has an endpoint state for each
endpoint of a device. When a USB device goes into suspend, xHC driver
could ask the hardware to push the state from cache to the memory.
Thereby saving additional energy. This is the intention of this patch
series.
You don't save any energy when the _device_ goes into suspend. You
save energy only when the _controller_ goes into suspend, because
that's the only time when the cache can be powered down. Right?

Yes, I agree with you.

Hardware could only power down the cache after all data has been
pushed out. That could happen during host suspend as far as I can see.


If the hardware supports FSC, this will happen automatically.

If the hardware doesn't support FSC, the cached data won't get
pushed to memory unless the driver tells the controller to do
so at the time the device is suspended. But this will slow
things down, so the driver should avoid doing it when it's not
needed.

During system suspend you know in advance that the controller
will be suspended. Therefore the driver should push the cache
to memory if the hardware doesn't support FSC. During runtime
suspend you don't know in advance whether the controller will
be suspended, so the driver should not push the cache to
memory.

But what happens in the case where all the devices have gone
into runtime suspend, so the controller also goes into runtime
suspend, and the hardware doesn't support FSC? It seems that
in this case the cache would have to remain powered on.
FSC is not part of this patch series. It was introduced when I tried to
explain the reason why I kept 'msg' parameter in the callback.

If the hardware support FSC, another operation named "save context
operation" will push everything in hardware cache into memory. So when
a USB device is going to suspend and xhci_device_suspend() callback is
being called, software can do an optimization. That is, if "save context
operation" will push everything in hardware cache to memory later,
xhci_device_suspend() could skip asking hardware for cache operation
and let "save context operation" do it later.
That's more or less the same as what I wrote above, except that I said
it would happen automatically if the hardware supports FSC. It isn't
automatic; it requires the driver to issue a "save context operation"
command.

Yes, exactly.


One example of this situation is system suspend. During system suspend,
below process will be done for a USB3 fabric.

1) all USB devices suspend
2) root hub suspend
3) host controller suspend

In 1), xhci_device_suspend() call back will be called for each device
suspend.
In 3) "save context operation" will be executed.

In this case, if FSC is supported, xhci_device_suspend() could skip asking
host hardware for cache operation.
That's basically what I said.

But now why should "msg" matter? It seems that xhci_device_suspend()
should skip asking for the cache operation whenever FSC is supported,
regardless of whether you're talking about runtime suspend or system
suspend.

In runtime case, a USB device could be selected to suspend while host
controller is busy with other USB devices. Since FSC only impacts the
behavior of save context operation, it's probably that a USB device
is selectively suspended while it's state is still kept in cache.

My thinking is that although cache management is hardware implementation
dependent, it's always a good behavior for software to ask for the cache
operation whenever it realizes that caching the state is unnecessary. This
is good not only for power saving, but also for better cache utilization.


Also, it's still not clear what the driver needs to do differently in
order to push out the cached data. You have managed to imply both:

Issuing Stop Endpoint with SP set is mandatory whenever
a device is suspended, and

The SP bit is what tells the controller to push the endpoint
state to memory.

This doesn't make sense.
Software doesn't handle the cache directly. XHC driver just tell hardware
that a device is going to suspend. XHC driver does this through stop
endpoint
command by setting the SP (suspend) bit in the command trb.

During xHC handles stop endpoint command, it will check SP bit, if it's set,
xHC knows that the endpoint state could be pushed out to memory now,
since device is going to suspend. The operation of cache is very hardware
implementation dependent.
Does that mean the driver has no control over whether "stop endpoint
with SP set" will push the cached information into memory? In that
case, how can xhci_device_suspend skip asking the hardware for the
cache operation?

XHCI spec doesn't define what hardware does when it sees SP bit set in
stop endpoint command. An implementation can even be designed without
an internal cache. But whenever cache is used, spec does recommend
and assume that hardware should push cached state into system memory
when SP bit set.

Spec (1.1) has an implementation note on page 253. It says,

If an implementation uses the Scratchpad or non-volatile internal memory
to cache endpoint state rather than Endpoint Contexts, then the Save State
operation can flush that state to the Scratchpad and software does not have
to issue Stop Endpoint Commands as part of the suspend process.


One last question: How much extra time does it take to push the cached
data to memory? Bear in mind that suspending a device is not a rapid
operation; if pushing the data takes no more than a few microseconds, I
think it would be worthwhile to do it always, even when it's not
necessary.
It depends on how much time software spends in xhci_device_suspend().
I don't have data in hand, but I agree with you that if this time is no more
than a few microseconds, we can let do it always.
Above you seem to be saying that there's no way to prevent it! Your
exact words are: "The operation of cache is very hardware
implementation dependent."

I explained my intention of "the operation of cache is very hardware
implementation dependent" above.

From software point of view, the extra time it takes to ask for cache
operation can be measured in xhci_device_suspend(). I can measure
the data later when I complete my tasks in hand. If that data is ignorable
comparing to the suspend time, we can simply do it always.


Alan Stern

Thank you!
-Baolu


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/