Re: [PATCH][RESEND] usb: dwc3: gadget: Handle dequeuing of non queued URB gracefully

From: Lars-Peter Clausen
Date: Tue Mar 10 2020 - 10:07:10 EST


On 3/10/20 2:45 PM, Lars-Peter Clausen wrote:
On 3/10/20 2:22 PM, Ardelean, Alexandru wrote:
On Thu, 2020-01-30 at 14:02 +0200, Felipe Balbi wrote:
[External]


Hi,

Alexandru Ardelean <alexandru.ardelean@xxxxxxxxxx> writes:

From: Lars-Peter Clausen <lars@xxxxxxxxxx>

Trying to dequeue and URB that is currently not queued should be a no-op
and be handled gracefully.

Use the list field of the URB to indicate whether it is queued or not by
setting it to the empty list when it is not queued.

Handling this gracefully allows for race condition free synchronization
between the complete callback being called to to a completed transfer and
trying to call usb_ep_dequeue() at the same time.
We need a little more information here. Can you further explain what
happens and how you caught this?
Apologies for the delay [of this reply].
It's been a while since this patch was created, and it was on a 4.14 kernel.
Lars was trying to fix various crashes with USB DWC3 OTG + some Xilinx patches.
I did not track the status of the OTG stuff upstream. I think it's a lot of
patches in the Xilinx tree.

The context has changed from 4.14 [obviously], and there were many things that
could have influenced things.
I've been trying to RFC some of these patches now.
[ yeah I know: maybe I should have [probably] also added an RFC tag :) ]
Some of the patches [including this one] seemed to make sense, even outside of
the context of the crashes that were happening on 4.14.
Atm, we're at 4.19 and we don't see issues, but we still have this patch.
We may drop it and see what happens.
Â\_(ã)_/Â

But in any case, it does require a bit more re-investigation.
Apologies for the noise that this patch created :)

The race condition is between a gadget calling usb_ep_dequeue() and the driver completing the URB.

Lets say in a thread you have a reference to a in-flight URB and you want to abort the request, e.g. because the application that sent the request has been closed. But concurrently to that the URB is completed by the hardware and the interrupt fires and marks the URB as complete. Your thread is suspended while the interrupt is running, once the interrupt has finished the thread wakes up, still has the reference to the URB, but now it has been completed. The thread still calls usb_ep_dequeue() though and then undefined behavior occurs.


Sorry, one quick correction. I believe the issue actually occurs when you have more than one CPU and the thread is not suspended, while the interrupt is running. In this case it is possible that the IRQ fires the driver marks the URB as complete, then unlocks, the driver lock and calls the complete callback, but before the complete callback runs the other thread calls usb_ep_dequeue(). There is no way to protect against this condition at the gadget level and it needs to be handled in the driver.

Basically

CPU 1 | CPU 2
--------------------------------------------------------------
URB IRQ fires | |
spin_lock(&dwc->lock); | |
| |
Driver handles completed URB | |
frees resources, etc | spin_lock(&gadget->lock) |
| usb_ep_dequeue() |
spin_unlock(&dwc->lock); | |
usb_gadget_giveback_request() | spin_lock(&dwc->lock) |
| |
Call compelte callback | Try to free URB resources |
| again => Undefined behavior |
| |
| spin_lock(&dwc->lock) |
| spin_unlock(&gadget->lock) |
spin_lock(&gadget->lock) | |
| |
Mark URB as compelte in gadget | |
spin_unlock(&gadget->lock) | |
spin_unlock(&dwc->lock) | |
spin_lock(&dwc->lock) | |
finish IRQ | |


The context in which we observed the issue is when using function fs to create a userspace gadget and using aio_cancel() to abort a pending URB. But really any gadget that aborts a transfer before it is completed or before the timeout occurred can run into this issue.

- Lars