Re: [PATCH 4/4] drm: xlnx: zynqmp_dpsub: Set live video in format

[Klymenko, Anatoliy]

Hi Tommy,

Thank you for the comments.

> Date: Wed, 17 Jan 2024 17:32:57 +0200
> From: Tomi Valkeinen <tomi.valkeinen@xxxxxxxxxxxxxxxx>
> To: Anatoliy Klymenko <anatoliy.klymenko@xxxxxxx>,
>         laurent.pinchart@xxxxxxxxxxxxxxxx, maarten.lankhorst@xxxxxxxxxxxxxxx,
>         mripard@xxxxxxxxxx, tzimmermann@xxxxxxx, airlied@xxxxxxxxx,
>         daniel@xxxxxxxx, michal.simek@xxxxxxx,
>         dri-devel@xxxxxxxxxxxxxxxxxxxxx, linux-arm-kernel@lists.infradeadorg,
>         linux-kernel@xxxxxxxxxxxxxxx
> Subject: Re: [PATCH 4/4] drm: xlnx: zynqmp_dpsub: Set live video in
>         format
> Message-ID: <dc0c448e-f03d-43ea-b571-3e96d859f89b@xxxxxxxxxxxxxxxx>
> Content-Type: text/plain; charset=UTF-8; format=flowed
> 
> Hi Anatoliy,
> 
> On 13/01/2024 01:42, Anatoliy Klymenko wrote:
> > Live video input format is expected to be set as "bus-format" property
> > in connected remote endpoint.
> > Program live video input format DPSUB registers.
> > Set display layer mode in layer creation context.
> 
> Some comments inline below. But one thing to improve is the commit desc.
> 
> I think this needs more explanation on what's the issue here. So basically
> something like what's the feature in question, why it's not working or what's
> missing, and what does this patch do to get it working.
> 
> And while often it's reasonable to expect some level of understanding of the HW
> in question, it doesn't hurt to give some clarifications on the names used (here the
> "live video input").
> 

Sure, good point, how about something like
"    ZynqMP DPSUB supports 2 modes of operations in regard to video data
    input.

    In the first mode, DPSUB uses DMA engine to pull video data from memory
    buffers. To support this the driver implements CRTC and DRM bridge
    representing DP encoder.

    In the second mode, DPSUB acquires video data pushed from FPGA and
    passes it downstream to DP output. This mode of operation is modeled in
    the driver as a DRM bridge that should be attached to some external
    CRTC. DPSUB supports multiple input data formats. In order to properly
    operate exact media bus format should be negotiated between external
    CRTC and DPSUB bridge. DRM framework provides a mechanism to negotiate
    media bus formats between bridges connected into a chain, but not
    between CRTC and encoder (first bridge in the chain). This change
    mitigates the issue for FPGA based CRTC, which would typically have a
    single fixed media bus format.

    Expect live video input format to be set as "bus-format" property in
    connected remote endpoint.

    Set display layer mode in the layer creation context."?

> > Signed-off-by: Anatoliy Klymenko <anatoliy.klymenko@xxxxxxx>
> > ---
> >   drivers/gpu/drm/xlnx/zynqmp_disp.c      | 109 ++++++++++++++++++++++--
> >   drivers/gpu/drm/xlnx/zynqmp_disp.h      |   3 +-
> >   drivers/gpu/drm/xlnx/zynqmp_disp_regs.h |   8 +-
> >   drivers/gpu/drm/xlnx/zynqmp_dp.c        |   2 +-
> >   drivers/gpu/drm/xlnx/zynqmp_kms.c       |   2 +-
> >   5 files changed, 107 insertions(+), 17 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/xlnx/zynqmp_disp.c
> > b/drivers/gpu/drm/xlnx/zynqmp_disp.c
> > index 8a39b3accce5..83af3ad9cdb5 100644
> > --- a/drivers/gpu/drm/xlnx/zynqmp_disp.c
> > +++ b/drivers/gpu/drm/xlnx/zynqmp_disp.c
> > @@ -20,8 +20,10 @@
> >   #include <linux/dmaengine.h>
> >   #include <linux/module.h>
> >   #include <linux/of.h>
> > +#include <linux/of_graph.h>
> >   #include <linux/platform_device.h>
> >   #include <linux/slab.h>
> > +#include <linux/media-bus-format.h>
> 
> Alphabetical order, please.

Got it - will fix in the next version.

> 
> >   #include "zynqmp_disp.h"
> >   #include "zynqmp_disp_regs.h"
> > @@ -67,12 +69,16 @@
> >   /**
> >    * struct zynqmp_disp_format - Display subsystem format information
> >    * @drm_fmt: DRM format (4CC)
> > + * @bus_fmt: Live video media bus format
> >    * @buf_fmt: AV buffer format
> >    * @swap: Flag to swap R & B for RGB formats, and U & V for YUV formats
> >    * @sf: Scaling factors for color components
> >    */
> >   struct zynqmp_disp_format {
> > -     u32 drm_fmt;
> > +     union {
> > +             u32 drm_fmt;
> > +             u32 bus_fmt;
> > +     };
> >       u32 buf_fmt;
> >       bool swap;
> >       const u32 *sf;
> > @@ -354,6 +360,16 @@ static const struct zynqmp_disp_format
> avbuf_gfx_fmts[] = {
> >       },
> >   };
> >
> > +/* TODO: add support for different formats */ static const struct
> > +zynqmp_disp_format avbuf_live_vid_fmts[] = {
> > +     {
> > +             .bus_fmt        = MEDIA_BUS_FMT_UYVY8_1X16,
> > +             .buf_fmt        = ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_BPC_8 |
> > +                               ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_FMT_YUV422,
> > +             .sf             = scaling_factors_888,
> > +     }
> > +};
> > +
> >   static u32 zynqmp_disp_avbuf_read(struct zynqmp_disp *disp, int reg)
> >   {
> >       return readl(disp->avbuf.base + reg); @@ -369,6 +385,34 @@
> > static bool zynqmp_disp_layer_is_video(const struct zynqmp_disp_layer *layer)
> >       return layer->id == ZYNQMP_DPSUB_LAYER_VID;
> >   }
> >
> > +/**
> > + * zynqmp_disp_avbuf_set_live_format - Set live input format for a
> > +layer
> > + * @disp: Display controller
> > + * @layer: The layer
> > + * @fmt: The format information
> > + *
> > + * Set the live video input format for @layer to @fmt.
> > + */
> > +static void zynqmp_disp_avbuf_set_live_format(struct zynqmp_disp *disp,
> > +                                           struct zynqmp_disp_layer *layer,
> > +                                           const struct
> > +zynqmp_disp_format *fmt) {
> > +     u32 reg, i;
> > +
> > +     reg = zynqmp_disp_layer_is_video(layer)
> > +         ? ZYNQMP_DISP_AV_BUF_LIVE_VID_CONFIG
> > +         : ZYNQMP_DISP_AV_BUF_LIVE_GFX_CONFIG;
> > +     zynqmp_disp_avbuf_write(disp, reg, fmt->buf_fmt);
> > +
> > +     for (i = 0; i < ZYNQMP_DISP_AV_BUF_NUM_SF; ++i) {
> > +             reg = zynqmp_disp_layer_is_video(layer)
> > +                 ? ZYNQMP_DISP_AV_BUF_LIVD_VID_COMP_SF(i)
> > +                 : ZYNQMP_DISP_AV_BUF_LIVD_GFX_COMP_SF(i);
> > +             zynqmp_disp_avbuf_write(disp, reg, fmt->sf[i]);
> > +     }
> > +     layer->disp_fmt = fmt;
> > +}
> > +
> >   /**
> >    * zynqmp_disp_avbuf_set_format - Set the input format for a layer
> >    * @disp: Display controller
> > @@ -902,15 +946,12 @@ u32 *zynqmp_disp_layer_drm_formats(struct
> zynqmp_disp_layer *layer,
> >   /**
> >    * zynqmp_disp_layer_enable - Enable a layer
> >    * @layer: The layer
> > - * @mode: Operating mode of layer
> >    *
> >    * Enable the @layer in the audio/video buffer manager and the blender. DMA
> >    * channels are started separately by zynqmp_disp_layer_update().
> >    */
> > -void zynqmp_disp_layer_enable(struct zynqmp_disp_layer *layer,
> > -                           enum zynqmp_dpsub_layer_mode mode)
> > +void zynqmp_disp_layer_enable(struct zynqmp_disp_layer *layer)
> >   {
> > -     layer->mode = mode;
> >       zynqmp_disp_avbuf_enable_video(layer->disp, layer);
> >       zynqmp_disp_blend_layer_enable(layer->disp, layer);
> >   }
> > @@ -950,11 +991,12 @@ void zynqmp_disp_layer_set_format(struct
> zynqmp_disp_layer *layer,
> >       layer->disp_fmt = zynqmp_disp_layer_find_format(layer, info->format);
> >       layer->drm_fmt = info;
> >
> > -     zynqmp_disp_avbuf_set_format(layer->disp, layer, layer->disp_fmt);
> > -
> > -     if (!layer->disp->dpsub->dma_enabled)
> > +     /* Live format set during layer creation */
> > +     if (layer->mode == ZYNQMP_DPSUB_LAYER_LIVE)
> >               return;
> >
> > +     zynqmp_disp_avbuf_set_format(layer->disp, layer,
> > + layer->disp_fmt);
> > +
> >       /*
> >        * Set pconfig for each DMA channel to indicate they're part of a
> >        * video group.
> > @@ -1083,7 +1125,7 @@ static int zynqmp_disp_layer_request_dma(struct
> zynqmp_disp *disp,
> >       unsigned int i;
> >       int ret;
> >
> > -     if (!disp->dpsub->dma_enabled)
> > +     if (layer->mode == ZYNQMP_DPSUB_LAYER_LIVE)
> >               return 0;
> >
> >       for (i = 0; i < layer->info->num_channels; i++) { @@ -1104,6
> > +1146,43 @@ static int zynqmp_disp_layer_request_dma(struct zynqmp_disp
> *disp,
> >       return 0;
> >   }
> >
> > +/**
> > + * zynqmp_disp_get_live_fmt - Get live video format
> > + * @disp: Display controller
> > + * @layer: Display layer
> > + *
> > + * Parse connected remote endpoint and retrieve configured bus-format
> > + *
> > + * Return: live format pointer on success, NULL otherwise  */ static
> > +const struct zynqmp_disp_format *zynqmp_disp_get_live_fmt(struct
> zynqmp_disp *disp,
> > +                                                              struct
> > +zynqmp_disp_layer *layer) {
> > +     struct device_node *local, *remote, *dpsub = disp->dev->of_node;
> > +     int rc, i;
> > +     u32 fmt;
> > +
> > +     local = of_graph_get_endpoint_by_regs(dpsub, layer->id, -1);
> > +     if (!local)
> > +             return NULL;
> > +
> > +     remote = of_graph_get_remote_endpoint(local);
> 
> The "remote" here is some PL component that provides the live stream?
> I'm not sure if there's a rule for these, but I think usually a driver should only read
> its own properties. I would add 'bus-format' to dp endpoint's DT data.
> 

The point is that bus format limitation is imposed by FPGA IP (AXI-S to Native Video Converter to be precise), and from a topology point of view, it should attribute the corresponding device tree node. Moving this property to the DPSUB node may simplify the driver but would decouple the property from the attributed entity.

> > +     of_node_put(local);
> > +     if (!remote)
> > +             return NULL;
> > +
> > +     rc = of_property_read_u32_index(remote, "bus-format", 0, &fmt);
> 
> Does this require a change to the DT bindings?
> 

This will require an update of DT documentation for a few soft IP drivers in the Xilinx downstream kernel. No changes in the upstream kernel are needed.

> Why is this not of_property_read_u32()?
> 

You are correct - it should be of_property_read_u32(). I will fix it in the next version. Thank you.

> > +     of_node_put(remote);
> > +     if (rc)
> > +             return NULL;
> > +
> > +     for (i = 0; i < ARRAY_SIZE(avbuf_live_vid_fmts); ++i)
> > +             if (avbuf_live_vid_fmts[i].bus_fmt == fmt)
> > +                     return &avbuf_live_vid_fmts[i];
> > +
> > +     return NULL;
> > +}
> > +
> >   /**
> >    * zynqmp_disp_create_layers - Create and initialize all layers
> >    * @disp: Display controller
> > @@ -1130,9 +1209,15 @@ static int zynqmp_disp_create_layers(struct
> > zynqmp_disp *disp)
> >
> >       for (i = 0; i < ARRAY_SIZE(disp->layers); i++) {
> >               struct zynqmp_disp_layer *layer = &disp->layers[i];
> > +             const struct zynqmp_disp_format *disp_fmt;
> >
> >               layer->id = i;
> >               layer->disp = disp;
> > +             /* For now we assume dpsub works in either live or non-live mode for
> both layers.
> > +              * Hybrid mode is not supported yet.
> > +              */
> > +             layer->mode = disp->dpsub->dma_enabled ?
> ZYNQMP_DPSUB_LAYER_NONLIVE
> > +                                                    :
> > + ZYNQMP_DPSUB_LAYER_LIVE;
> >               layer->info = &layer_info[i];
> >
> >               ret = zynqmp_disp_layer_request_dma(disp, layer); @@
> > -1140,6 +1225,12 @@ static int zynqmp_disp_create_layers(struct
> zynqmp_disp *disp)
> >                       goto err;
> >
> >               disp->dpsub->layers[i] = layer;
> > +
> > +             if (layer->mode == ZYNQMP_DPSUB_LAYER_LIVE) {
> > +                     disp_fmt = zynqmp_disp_get_live_fmt(disp, layer);
> > +                     if (disp_fmt)
> > +                             zynqmp_disp_avbuf_set_live_format(disp, layer, disp_fmt);
> > +             }
> >       }
> >
> >       return 0;
> > diff --git a/drivers/gpu/drm/xlnx/zynqmp_disp.h
> > b/drivers/gpu/drm/xlnx/zynqmp_disp.h
> > index 123cffac08be..f3357b2d5c09 100644
> > --- a/drivers/gpu/drm/xlnx/zynqmp_disp.h
> > +++ b/drivers/gpu/drm/xlnx/zynqmp_disp.h
> > @@ -62,8 +62,7 @@ void zynqmp_disp_blend_set_global_alpha(struct
> > zynqmp_disp *disp,
> >
> >   u32 *zynqmp_disp_layer_drm_formats(struct zynqmp_disp_layer *layer,
> >                                  unsigned int *num_formats); -void
> > zynqmp_disp_layer_enable(struct zynqmp_disp_layer *layer,
> > -                           enum zynqmp_dpsub_layer_mode mode);
> > +void zynqmp_disp_layer_enable(struct zynqmp_disp_layer *layer);
> >   void zynqmp_disp_layer_disable(struct zynqmp_disp_layer *layer);
> >   void zynqmp_disp_layer_set_format(struct zynqmp_disp_layer *layer,
> >                                 const struct drm_format_info *info);
> > diff --git a/drivers/gpu/drm/xlnx/zynqmp_disp_regs.h
> > b/drivers/gpu/drm/xlnx/zynqmp_disp_regs.h
> > index f92a006d5070..926e07c255bb 100644
> > --- a/drivers/gpu/drm/xlnx/zynqmp_disp_regs.h
> > +++ b/drivers/gpu/drm/xlnx/zynqmp_disp_regs.h
> > @@ -165,10 +165,10 @@
> >   #define ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_BPC_10               0x2
> >   #define ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_BPC_12               0x3
> >   #define ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_BPC_MASK
> GENMASK(2, 0)
> > -#define ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_FMT_RGB               0x0
> > -#define ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_FMT_YUV444    0x1
> > -#define ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_FMT_YUV422    0x2
> > -#define ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_FMT_YONLY     0x3
> > +#define ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_FMT_RGB               0x00
> > +#define ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_FMT_YUV444    0x10
> > +#define ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_FMT_YUV422    0x20
> > +#define ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_FMT_YONLY     0x30
> 
> What's this about? Were these wrong before? Sounds like a separate patch
> needed for these.
> 

It is an embedded bit shift that corresponds to DPSUB live video / gfx format register layout. Original values are technically correct but would require extra bit shifts to operate with. The current patch is the first instance of actual use of those defines. Do you think it's worth to factor those changes out into a separate patch?

> >   #define ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_FMT_MASK
> GENMASK(5, 4)
> >   #define ZYNQMP_DISP_AV_BUF_LIVE_CONFIG_CB_FIRST             BIT(8)
> >   #define ZYNQMP_DISP_AV_BUF_PALETTE_MEMORY           0x400
> > diff --git a/drivers/gpu/drm/xlnx/zynqmp_dp.c
> > b/drivers/gpu/drm/xlnx/zynqmp_dp.c
> > index 571c5dbc97e5..59616ed1c3d9 100644
> > --- a/drivers/gpu/drm/xlnx/zynqmp_dp.c
> > +++ b/drivers/gpu/drm/xlnx/zynqmp_dp.c
> > @@ -1295,7 +1295,7 @@ static void zynqmp_dp_disp_enable(struct
> zynqmp_dp *dp,
> >       /* TODO: Make the format configurable. */
> >       info = drm_format_info(DRM_FORMAT_YUV422);
> >       zynqmp_disp_layer_set_format(layer, info);
> > -     zynqmp_disp_layer_enable(layer, ZYNQMP_DPSUB_LAYER_LIVE);
> > +     zynqmp_disp_layer_enable(layer);
> >
> >       if (layer_id == ZYNQMP_DPSUB_LAYER_GFX)
> >               zynqmp_disp_blend_set_global_alpha(dp->dpsub->disp,
> > true, 255); diff --git a/drivers/gpu/drm/xlnx/zynqmp_kms.c
> > b/drivers/gpu/drm/xlnx/zynqmp_kms.c
> > index db3bb4afbfc4..43bf416b33d5 100644
> > --- a/drivers/gpu/drm/xlnx/zynqmp_kms.c
> > +++ b/drivers/gpu/drm/xlnx/zynqmp_kms.c
> > @@ -122,7 +122,7 @@ static void
> > zynqmp_dpsub_plane_atomic_update(struct drm_plane *plane,
> >
> >       /* Enable or re-enable the plane if the format has changed. */
> >       if (format_changed)
> > -             zynqmp_disp_layer_enable(layer, ZYNQMP_DPSUB_LAYER_NONLIVE);
> > +             zynqmp_disp_layer_enable(layer);
> >   }
> >
> >   static const struct drm_plane_helper_funcs
> > zynqmp_dpsub_plane_helper_funcs = {
> 
> 

Thank you,
Anatoliy