Re: Stateless Encoding uAPI Discussion and Proposal

[Andrzej Pietrasiewicz]

Hi Paul & Hans,

W dniu 9.08.2023 o 16:43, Paul Kocialkowski pisze:
> Hi Hans,
>
> On Wed 26 Jul 23, 10:18, Hans Verkuil wrote:
> > On 11/07/2023 20:18, Nicolas Dufresne wrote:
> > > Le mardi 11 juillet 2023 à 19:12 +0200, Paul Kocialkowski a écrit :
> > > > Hi everyone!
> > > >
> > > > After various discussions following Andrzej's talk at EOSS, feedback from the
> > > > Media Summit (which I could not attend unfortunately) and various direct
> > > > discussions, I have compiled some thoughts and ideas about stateless encoders
> > > > support with various proposals. This is the result of a few years of interest
> > > > in the topic, after working on a PoC for the Hantro H1 using the hantro driver,
> > > > which turned out to have numerous design issues.
> > > >
> > > > I am now working on a H.264 encoder driver for Allwinner platforms (currently
> > > > focusing on the V3/V3s), which already provides some usable bitstream and will
> > > > be published soon.
> > > >
> > > > This is a very long email where I've tried to split things into distinct topics
> > > > and explain a few concepts to make sure everyone is on the same page.
> > > >
> > > > # Bitstream Headers
> > > >
> > > > Stateless encoders typically do not generate all the bitstream headers and
> > > > sometimes no header at all (e.g. Allwinner encoder does not even produce slice
> > > > headers). There's often some hardware block that makes bit-level writing to the
> > > > destination buffer easier (deals with alignment, etc).
> > > >
> > > > The values of the bitstream headers must be in line with how the compressed
> > > > data bitstream is generated and generally follow the codec specification.
> > > > Some encoders might allow configuring all the fields found in the headers,
> > > > others may only allow configuring a few or have specific constraints regarding
> > > > which values are allowed.
> > > >
> > > > As a result, we cannot expect that any given encoder is able to produce frames
> > > > for any set of headers. Reporting related constraints and limitations (beyond
> > > > profile/level) seems quite difficult and error-prone.
> > > >
> > > > So it seems that keeping header generation in-kernel only (close to where the
> > > > hardware is actually configured) is the safest approach.
> > >
> > > This seems to match with what happened with the Hantro VP8 proof of concept. The
> > > encoder does not produce the frame header, but also, it produces 2 encoded
> > > buffers which cannot be made contiguous at the hardware level. This notion of
> > > plane in coded data wasn't something that blended well with the rest of the API
> > > and we didn't want to copy in the kernel while the userspace would also be
> > > forced to copy to align the headers. Our conclusion was that it was best to
> > > generate the headers and copy both segment before delivering to userspace. I
> > > suspect this type of situation will be quite common.
> > >
> > > > # Codec Features
> > > >
> > > > Codecs have many variable features that can be enabled or not and specific
> > > > configuration fields that can take various values. There is usually some
> > > > top-level indication of profile/level that restricts what can be used.
> > > >
> > > > This is a very similar situation to stateful encoding, where codec-specific
> > > > controls are used to report and set profile/level and configure these aspects.
> > > > A particularly nice thing about it is that we can reuse these existing controls
> > > > and add new ones in the future for features that are not yet covered.
> > > >
> > > > This approach feels more flexible than designing new structures with a selected
> > > > set of parameters (that could match the existing controls) for each codec.
> > >
> > > Though, reading more into this emails, we still have a fair amount of controls
> > > to design and add, probably some compound controls too ?
> >
> > I expect that for stateless encoders support for read-only requests will be needed:
> >
> > https://patchwork.linuxtv.org/project/linux-media/list/?series=5647
> >
> > I worked on that in the past together with dynamic control arrays. The dynamic
> > array part was merged, but the read-only request part wasn't (there was never a
> > driver that actually needed it).
> >
> > I don't know if that series still applies, but if there is a need for it then I
> > can rebase it and post an RFCv3.
>
> So if I understand this correctly (from a quick look), this would be to allow
> stateless encoder drivers to attach a particular control value to a specific
> returned frame?
>
> I guess this would be a good match to return statistics about the encoded frame.
> However that would probably be expressed in a hardware-specific way so it
> seems preferable to not expose this to userspace and handle it in-kernel
> instead.
>
> What's really important for userspace to know (in order to do user-side
> rate-control, which we definitely want to support) is the resulting bitstream
> size. This is already available with bytesused.
>
> So all in all I think we're good with the current status of request support.

Yup. I agree. Initially, while working on VP8 encoding we introduced (read-only)
requests on the capture queue, but they turned out not to be useful in this
context and we removed them.

Regards.

Andrzej

> Cheers,
>
> Paul