Re: [PATCH WIP RFC v2 1/2] media: uapi: Add V4L2_CID_{V,H}TOTAL controls

[Sakari Ailus]

Hi Benjamin,

Thank you for the patchset and my apologies for reviewing this so late.

On Thu, Jun 22, 2023 at 09:12:23AM +0200, Benjamin Bara wrote:
> From: Benjamin Bara <benjamin.bara@xxxxxxxxxxx>
> 
> Currently, V4L2_CID_{V,H}BLANK can be used to control the frame duration
> of a stream. However, camera sensors usually have a register for the
> total size (image data + blanking), e.g. {V,H}MAX on the imx290. As the
> user space also typically wants to set a frame size, both sides
> currently have to calculate with values they actually don't care about.
> 
> The dependency between format height and vertical blanking also results
> to a change of the vertical blanking value and limits whenever the
> format of the frame is changed and therefore makes it harder for user
> space to do calculations, e.g. the frame duration.
> V4L2_CID_{V,H}TOTAL do not depend on the format and therefore simplify
> calculations. Additionally, they represent the hardware "better" and
> therefore also simplify calculations on the driver side.
> 
> Add additional documentation about V4L2_CID_HTOTAL to have a control
> which enables the user space to express its intends to the driver. The
> driver can then update its controls and limits accordingly, and can do
> possible re-calculations during mode switches in a defined behaviour.
> This increases compatibility between different camera sensors.
> 
> Signed-off-by: Benjamin Bara <benjamin.bara@xxxxxxxxxxx>
> ---
> v2:
> - add HTOTAL
> - add documentation about expectations
> ---
>  Documentation/driver-api/media/camera-sensor.rst         | 11 +++++++++--
>  .../userspace-api/media/v4l/ext-ctrls-image-source.rst   | 16 ++++++++++++++++
>  drivers/media/v4l2-core/v4l2-ctrls-defs.c                |  2 ++
>  include/uapi/linux/v4l2-controls.h                       |  2 ++
>  4 files changed, 29 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/driver-api/media/camera-sensor.rst b/Documentation/driver-api/media/camera-sensor.rst
> index c7d4891bd24e..3ddeb0533728 100644
> --- a/Documentation/driver-api/media/camera-sensor.rst
> +++ b/Documentation/driver-api/media/camera-sensor.rst
> @@ -85,12 +85,14 @@ a result of the configuration of a number of camera sensor implementation
>  specific parameters. Luckily, these parameters tend to be the same for more or
>  less all modern raw camera sensors.
>  
> -The frame interval is calculated using the following equation::
> +The frame interval is calculated using one of the following equations::
>  
>  	frame interval = (analogue crop width + horizontal blanking) *
>  			 (analogue crop height + vertical blanking) / pixel rate
>  
> -The formula is bus independent and is applicable for raw timing parameters on
> +	frame interval = horizontal total size * vertical total size / pixel rate
> +
> +The formulas are bus independent and are applicable for raw timing parameters on

I think we should continue to have a single formula, but that the values
could be obtained different ways, depending on the driver.

For the user space it should be easier to use the controls you're
proposing but existing drivers will still continue to use the existing
controls.

In particular, you should add something like:

``V4L2_CID_VTOTAL`` value equals to analogue crop height + value of the
``V4L2_CID_VBLANK``. Similarly, ``V4L2_CID_HTOTAL`` value equals to
analogue crop width + value of the ``V4L2_CID_HBLANK``.

New drivers shall and existing drivers should implement ``V4L2_CID_VTOTAL``
and ``V4L2_CID_HTOTAL`` whereas existing drivers shall continue to
implement ``V4L2_CID_VBLANK`` and ``V4L2_CID_HBLANK``.

Implementing both in drivers can be a little bit painful. It'd be good to
add driver support for this and use this as an example (e.g. CCS). I'll see
if I could provide a patch, this will take some time still though.

I wonder what Laurent thinks.

>  large variety of devices beyond camera sensors. Devices that have no analogue
>  crop, use the full source image size, i.e. pixel array size.
>  
> @@ -100,6 +102,11 @@ is pixels and the unit of the ``V4L2_CID_VBLANK`` is lines. The pixel rate in
>  the sensor's **pixel array** is specified by ``V4L2_CID_PIXEL_RATE`` in the same
>  sub-device. The unit of that control is pixels per second.
>  
> +Additionally, the horizontal and vertical total sizes are specified by
> +``V4L2_CID_HTOTAL`` and ``V4L2_CID_VTOTAL``, respectively. The unit of the
> +``V4L2_CID_HTOTAL`` control is pixels and the unit of the ``V4L2_CID_VTOTAL`` is
> +lines, analogous to the blanking.

Add "controls" before the period.

> +
>  Register list based drivers need to implement read-only sub-device nodes for the
>  purpose. Devices that are not register list based need these to configure the
>  device's internal processing pipeline.
> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-image-source.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-image-source.rst
> index 71f23f131f97..5451fa0223cd 100644
> --- a/Documentation/userspace-api/media/v4l/ext-ctrls-image-source.rst
> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-image-source.rst
> @@ -59,6 +59,22 @@ Image Source Control IDs
>      non-sensitive.
>      This control is required for automatic calibration of sensors/cameras.
>  
> +``V4L2_CID_VTOTAL (integer)``
> +    Number of total lines per frame, including data and idle lines (blanking).
> +    The unit of the vertical total size is a line. Every line has length of the
> +    image width plus horizontal blanking at the pixel rate defined by
> +    ``V4L2_CID_PIXEL_RATE`` control in the same sub-device.
> +
> +``V4L2_CID_HTOTAL (integer)``
> +    Number of total pixels per line, including data and idle pixels (blanking).
> +    The unit of the horizontal total size is pixels. The default value of this
> +    control is set to a value which allows a constant total size for every
> +    supported mode of the sensor. The control can be used to indicate the driver
> +    if a high frame rate (value < default) or a high exposure (value > default)
> +    should be achieved. With value = default, a constant frame size across the
> +    different modes is targeted. Note that the actual frame rate depends on the
> +    ``V4L2_CID_PIXEL_RATE`` control, which might vary between mode switches.
> +
>  .. c:type:: v4l2_area
>  
>  .. flat-table:: struct v4l2_area
> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> index 564fedee2c88..34e17e1faa7a 100644
> --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> @@ -1112,6 +1112,8 @@ const char *v4l2_ctrl_get_name(u32 id)
>  	case V4L2_CID_TEST_PATTERN_BLUE:	return "Blue Pixel Value";
>  	case V4L2_CID_TEST_PATTERN_GREENB:	return "Green (Blue) Pixel Value";
>  	case V4L2_CID_NOTIFY_GAINS:		return "Notify Gains";
> +	case V4L2_CID_VTOTAL:			return "Vertical Total Size";
> +	case V4L2_CID_HTOTAL:			return "Horizontal Total Size";
>  
>  	/* Image processing controls */
>  	/* Keep the order of the 'case's the same as in v4l2-controls.h! */
> diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
> index 5e80daa4ffe0..a4bbd91b8aef 100644
> --- a/include/uapi/linux/v4l2-controls.h
> +++ b/include/uapi/linux/v4l2-controls.h
> @@ -1117,6 +1117,8 @@ enum v4l2_jpeg_chroma_subsampling {
>  #define V4L2_CID_TEST_PATTERN_GREENB		(V4L2_CID_IMAGE_SOURCE_CLASS_BASE + 7)
>  #define V4L2_CID_UNIT_CELL_SIZE			(V4L2_CID_IMAGE_SOURCE_CLASS_BASE + 8)
>  #define V4L2_CID_NOTIFY_GAINS			(V4L2_CID_IMAGE_SOURCE_CLASS_BASE + 9)
> +#define V4L2_CID_VTOTAL				(V4L2_CID_IMAGE_SOURCE_CLASS_BASE + 10)
> +#define V4L2_CID_HTOTAL				(V4L2_CID_IMAGE_SOURCE_CLASS_BASE + 11)
>  
>  
>  /* Image processing controls */
> 

-- 
Kind regards,

Sakari Ailus