Re: [PATCH v2] media: uvcvideo: Implement V4L2_EVENT_FRAME_SYNC

[Nicolas Dufresne]

Le jeudi 09 novembre 2023 à 01:27 +0100, Ricardo Ribalda a écrit :
> Hi Laurent
> 
> On Thu, 9 Nov 2023 at 01:03, Laurent Pinchart
> <laurent.pinchart@xxxxxxxxxxxxxxxx> wrote:
> > 
> > Hi Ricardo,
> > 
> > On Wed, Nov 08, 2023 at 11:46:40PM +0100, Ricardo Ribalda wrote:
> > > On Wed, 8 Nov 2023 at 21:32, <nicolas@xxxxxxxxxxxx> wrote:
> > > > 
> > > > The fact that you interpret the time from FRAME_SYNC to DQBUF (well the
> > > > READ IO notification) as the actual latency is yours of course. It
> > > > assumes that the camera on the other end does not introduce other
> > > 
> > > We want to use this signal to measure how much power is used since we
> > > start receiving the frame until we can use it.
> > > I agree with you that the latency between capture and dqbuf should be
> > > measured using the timestamp. That is not our use case here.
> > > 
> > > > source of latency (or that these are negligible). You are also going to
> > > > introduce a lot of jitter, since it relies on when the OS decides to
> > > > wake up your process.
> > > 
> > > We have measured a jitter of around 2.5 msec, which is acceptable for our needs.
> > > 
> > > > I think my opinion resides in if you can accurately *enough* implement
> > > > what the spec says for FRAME_SYNC then do it, otherwise just don't lie.
> > > 
> > > What the specs says is:
> > > ```
> > > Triggered immediately when the reception of a frame has begun
> > > ```
> > > In my opinion, that is true for usb devices, we are triggering it as
> > > soon as the transfer has started to the eyes of the driver. We cannot
> > > trigger earlier than that.
> > > 
> > > 
> > > > I think for ISO, "after the first chunk" i a small lie, but acceptable.
> > > > But for BULK, the way it was explained is that it will be always very
> > > > close to DQBUF time. and it should not emit FRAME_SYNC for this type of
> > > > UVC device. If it fits other events fine of course, I'm just making a
> > > > judgment on if its fits V4L2_EVENT_FRAME_SYNC or not.
> > > 
> > > nit: I believe that you have swapped iso and bulk on this description
> > 
> > I've confused the USB packet size and the UVC payload size. The latter
> > is typically much bigger for bulk devices than isoc devices, but the
> > former will be in similar order of magnitudes in a large number of
> > cases, but not all cases.
> > 
> > The URB size is the result of the USB packet size and number of packets
> > per URB. The uvcvideo driver currently sets the number of packets per
> > URB to 32 at most (and lowers it if the frame size is small, or if not
> > enough memory can be allocated). This could be increased or made dynamic
> > in the future, as higher speeds typically benefit from larger URB sizes.
> > The packet size differs between bulk and isoc endpoints.
> > 
> > For bulk, the packet size can be up to 512 bytes for USB 2.0 and 1024
> > bytes for USB 3.0, and the device can select a smaller size. The largest
> > URB size (again based on the current implementation of the uvcvideo
> > driver) is thus 32 KiB.
> > 
> > For isochronous the situation is more complicated. The term "packet" as
> > used in the uvcvideo driver actually means all the data transferred in
> > one service interval, thus made of multiple isoc packets. It is heavily
> > dependent on the USB speed, and the device can advertise different
> > supported sizes (which translate directly to the reserved bandwidth for
> > the transfer), with the driver picking the smallest bandwidth large
> > enough for the data rate required by the resolution and frame rate. The
> > theoretical worst case is 1024 bytes per isoc packet * 16 isoc packets
> > per burst * 6 burst per interval * 32 "packets" per URB, equal to 3 MiB.
> > 
> > Even with the largest URB size you have witnessed of ~1 MiB, we will end
> > up lying quite a bit if we consider the URB completion callback for the
> > first URB of the frame as indicating the start of reception.
> > 
> > > > In term of accuracy, if timestamp was passed with the FRAME_SYNC event,
> > > > it would not matter how fast your process the event anymore and greatly
> > > > improve accuracy.
> > > 
> > > +1 to that. If we could easily change the uAPI for FRAME_SYNC that
> > > should definitely be implemented.
> > > 
> > > > > Not to mention that the UVC timestamping requires a bit of love.
> > > > > 
> > > > > @Laurent Pinchart, @Kieran Bingham  any progress reviewing :P :
> > > > > https://patchwork.linuxtv.org/project/linux-media/list/?series=10083
> > > > 
> > > > Thanks for working on this by the way, hope someone will find the time
> > > > to review this. The timestamps should in theory provide a jitter free
> > > 
> > > It already has a couple of Reviewed-by stamped in.... ;)
> > > 
> > > > measurement of the delay Esker is trying to measure, and if it wasn't
> > > > of bugs (and crazy complexity) it would in the worst case match the
> > > > transfer time.
> > > 
> > > Sorry to repeat myself, but just to avoid the confusion: Esker needs
> > > to know how much power is used since we start receiving a frame until
> > > it is available for dqbuf, not de frame latency.
> > 
> > As I think everybody is aware, the earliest notification you get on the
> > CPU side is the *end* of reception of the first URB, which can possibly
> > be significantly later than the start of reception of the frame.
> > 
> > Based on what I understand, the goal is to measure the CPU power
> > consumption related to CPU processing of the frame. If that's the case,
> > there's good and bad news. The good news is that the CPU doesn't process
> > the frame at all until the URB has been received (if you were to measure
> > the power consumption of the USB host controller too, it would be a
> > different story), so the delay shouldn't be a problem. The bad news is
> > that I don't see how the information you're trying to get will help you,
> > as there's plenty of other things unrelated to the uvcvideo driver that
> > can take CPU time while a frame is being received. That may not be any
> > of my business, but from the point of view of the uvcvideo driver, I'm
> > less inclined to accept a possibly significant V4L2_EVENT_FRAME_SYNC lie
> > if the use case ends up making little sense :-)
> 
> Just to add some numbers to add some context:
> 
>  V4L2_EVENT_FRAME_SYNC for BULK and ISO will be delayed from:
> ```
> Triggered immediately when the reception of a frame has begun
> ```
> one urb.
> 
> For bulk devices this is a maximum of 0.05 msec (32KiB/600MBps)

I lack a bit of knowledge on how to scale this to different devices, with
different speed/framesize. My only bulk device is:

https://inogeni.com/product/4k2usb3/

Which is USB 3.0, and have raw (NV12) resolution from 640x480 (max 60pfs) to 4K
(max 30fps). What would the error look like with that ?

> For 1MiB transfer isoc devices (which is the biggest we have seen),
> that is 1.8 msec.
> In both cases, this is smaller than the jitter to process the event
> itself by userspace.
> 
> The time from V4L2_EVENT_FRAME_SYNC to DQBUF is around 30 msec.
> 
> I do not know how much delay is considered acceptable... but if we
> take the delay argument to the extreme, we could say that
> V4L2_EVENT_FRAME_SYNC can never be implemented, because the event will
> always be delayed by something.

We have v4l2_event.timestamp for all events, so the jitter to process the event
by userpace can be removed reliably already.

Nicolas

p.s. missed it earlier

> 
> > 
> > --
> > Regards,
> > 
> > Laurent Pinchart
> 
> 
>