Re: Re: [PATCH] firmware: qcom_scm: Introduce batching of hyp assign calls

[Elliot Berman]

On Tue, Feb 20, 2024 at 11:24:33AM +0530, Dibakar Singh wrote:
> 
> 
> On 09-Feb-24 11:36 PM, Elliot Berman wrote:
> > On Fri, Feb 09, 2024 at 04:55:36PM +0530, Dibakar Singh wrote:
> > > Expose an API qcom_scm_assign_table to allow client drivers to batch
> > > multiple memory regions in a single hyp assign call.
> > > 
> > > In the existing situation, if our goal is to process an sg_table and
> > > transfer its ownership from the current VM to a different VM, we have a
> > > couple of strategies. The first strategy involves processing the entire
> > > sg_table at once and then transferring the ownership. However, this
> > > method may have an adverse impact on the system because during an SMC
> > > call, the NS interrupts are disabled, and this delay could be
> > > significant when dealing with large sg_tables. To address this issue, we
> > > can adopt a second strategy, which involves processing each sg_list in
> > > the sg_table individually and reassigning memory ownership. Although
> > > this method is slower and potentially impacts performance, it will not
> > > keep the NS interrupts disabled for an extended period.
> > > 
> > > A more efficient strategy is to process the sg_table in batches. This
> > > approach addresses both scenarios by involving memory processing in
> > > batches, thus avoiding prolonged NS interrupt disablement for longer
> > > duration when dealing with large sg_tables. Moreover, since we process
> > > in batches, this method is faster compared to processing each item
> > > individually. The observations on testing both the approaches for
> > > performance is as follows:
> > > 
> > > Allocation Size/            256MB            512MB            1024MB
> > > Algorithm Used           ===========      ===========      ============
> > > 
> > > Processing each sg_list   73708(us)        149289(us)       266964(us)
> > > in sg_table one by one
> > > 
> > > Processing sg_table in    46925(us)         92691(us)       176893(us)
> > > batches
> > > 
> > > This implementation serves as a wrapper around the helper function
> > > __qcom_scm_assign_mem, which takes an sg_list and processes it in
> > > batches. We’ve set the limit to a minimum of 32 sg_list in a batch or a
> > > total batch size of 512 pages. The selection of these numbers is
> > > heuristic, based on the test runs conducted. Opting for a smaller number
> > > would compromise performance, while a larger number would result in
> > > non-secure interrupts being disabled for an extended duration.
> > > 
> > > Co-developed-by: Prakruthi Deepak Heragu <quic_pheragu@xxxxxxxxxxx>
> > > Signed-off-by: Prakruthi Deepak Heragu <quic_pheragu@xxxxxxxxxxx>
> > > Signed-off-by: Dibakar Singh <quic_dibasing@xxxxxxxxxxx>
> > > ---
> > >   drivers/firmware/qcom/qcom_scm.c       | 211 +++++++++++++++++++++++++
> > >   include/linux/firmware/qcom/qcom_scm.h |   7 +
> > >   2 files changed, 218 insertions(+)
> > > 
> > > diff --git a/drivers/firmware/qcom/qcom_scm.c b/drivers/firmware/qcom/qcom_scm.c
> > > index 520de9b5633a..038b96503d65 100644
> > > --- a/drivers/firmware/qcom/qcom_scm.c
> > > +++ b/drivers/firmware/qcom/qcom_scm.c
> > > @@ -21,6 +21,8 @@
> > >   #include <linux/platform_device.h>
> > >   #include <linux/reset-controller.h>
> > >   #include <linux/types.h>
> > > +#include <linux/scatterlist.h>
> > > +#include <linux/slab.h>
> > >   #include "qcom_scm.h"
> > > @@ -1048,6 +1050,215 @@ int qcom_scm_assign_mem(phys_addr_t mem_addr, size_t mem_sz,
> > >   }
> > >   EXPORT_SYMBOL_GPL(qcom_scm_assign_mem);
> > > +/**
> > > + * qcom_scm_assign_mem_batch() - Make a secure call to reassign memory
> > > + *				   ownership of several memory regions
> > > + * @mem_regions:    A buffer describing the set of memory regions that need to
> > > + *		    be reassigned
> > > + * @nr_mem_regions: The number of memory regions that need to be reassigned
> > > + * @srcvms:	    A buffer populated with he vmid(s) for the current set of
> > > + *		    owners
> > > + * @src_sz:	    The size of the srcvms buffer (in bytes)
> > > + * @destvms:	    A buffer populated with the new owners and corresponding
> > > + *		    permission flags.
> > > + * @dest_sz:	    The size of the destvms buffer (in bytes)
> > > + *
> > > + * Return negative errno on failure, 0 on success.
> > > + */
> > > +static int qcom_scm_assign_mem_batch(struct qcom_scm_mem_map_info *mem_regions,
> > > +				     size_t nr_mem_regions, u32 *srcvms,
> > > +				     size_t src_sz,
> > > +				     struct qcom_scm_current_perm_info *destvms,
> > > +				     size_t dest_sz)
> > > +{
> > > +	dma_addr_t mem_dma_addr;
> > > +	size_t mem_regions_sz;
> > > +	int ret = 0, i;
> > > +
> > > +	for (i = 0; i < nr_mem_regions; i++) {
> > > +		mem_regions[i].mem_addr = cpu_to_le64(mem_regions[i].mem_addr);
> > > +		mem_regions[i].mem_size = cpu_to_le64(mem_regions[i].mem_size);
> > > +	}
> > > +
> > > +	mem_regions_sz = nr_mem_regions * sizeof(*mem_regions);
> > > +	mem_dma_addr = dma_map_single(__scm->dev, mem_regions, mem_regions_sz,
> > > +				      DMA_TO_DEVICE);
> > > +	if (dma_mapping_error(__scm->dev, mem_dma_addr)) {
> > > +		dev_err(__scm->dev, "mem_dma_addr mapping failed\n");
> > > +		return -ENOMEM;
> > > +	}
> > > +
> > > +	ret = __qcom_scm_assign_mem(__scm->dev, virt_to_phys(mem_regions),
> > > +				    mem_regions_sz, virt_to_phys(srcvms), src_sz,
> > > +				    virt_to_phys(destvms), dest_sz);
> > > +
> > > +	dma_unmap_single(__scm->dev, mem_dma_addr, mem_regions_sz, DMA_TO_DEVICE);
> > > +	return ret;
> > > +}
> > > +
> > > +/**
> > > + * qcom_scm_prepare_mem_batch() - Prepare batches of memory regions
> > > + * @sg_table:       A scatter list whose memory needs to be reassigned
> > > + * @srcvms:	    A buffer populated with he vmid(s) for the current set of
> > > + *		    owners
> > > + * @nr_src:	    The number of the src_vms buffer
> > > + * @destvms:	    A buffer populated with he vmid(s) for the new owners
> > > + * @destvms_perms:  A buffer populated with the permission flags of new owners
> > > + * @nr_dest:	    The number of the destvms
> > > + * @last_sgl:	    Denotes to the last scatter list element. Used in case of rollback
> > > + * @roll_back:	    Identifies whether we are executing rollback in case of failure
> > > + *
> > > + * Return negative errno on failure, 0 on success.
> > > + */
> > > +static int qcom_scm_prepare_mem_batch(struct sg_table *table,
> > > +				      u32 *srcvms, int nr_src,
> > > +				      int *destvms, int *destvms_perms,
> > > +				      int nr_dest,
> > > +				      struct scatterlist *last_sgl, bool roll_back)
> > > +{
> > > +	struct qcom_scm_current_perm_info *destvms_cp;
> > > +	struct qcom_scm_mem_map_info *mem_regions_buf;
> > > +	struct scatterlist *curr_sgl = table->sgl;
> > > +	dma_addr_t source_dma_addr, dest_dma_addr;
> > > +	size_t batch_iterator;
> > > +	size_t batch_start = 0;
> > > +	size_t destvms_cp_sz;
> > > +	size_t srcvms_cp_sz;
> > > +	size_t batch_size;
> > > +	u32 *srcvms_cp;
> > > +	int ret = 0;
> > > +	int i;
> > > +
> > > +	if (!table || !table->sgl || !srcvms || !nr_src ||
> > > +	    !destvms || !destvms_perms || !nr_dest || !table->nents)
> > > +		return -EINVAL;
> > > +
> > > +	srcvms_cp_sz = sizeof(*srcvms_cp) * nr_src;
> > > +	srcvms_cp = kmemdup(srcvms, srcvms_cp_sz, GFP_KERNEL);
> > > +	if (!srcvms_cp)
> > > +		return -ENOMEM;
> > > +
> > > +	for (i = 0; i < nr_src; i++)
> > > +		srcvms_cp[i] = cpu_to_le32(srcvms_cp[i]);
> > > +
> > > +	source_dma_addr = dma_map_single(__scm->dev, srcvms_cp,
> > > +					 srcvms_cp_sz, DMA_TO_DEVICE);
> > 
> > Please use the new tzmem allocator:
> > 
> > https://lore.kernel.org/all/20240205182810.58382-1-brgl@xxxxxxxx/
> 
> Noted, I will use this allocator in V2 patch.
> 
> > 
> > > +
> > > +	if (dma_mapping_error(__scm->dev, source_dma_addr)) {
> > > +		ret = -ENOMEM;
> > > +		goto out_free_source;
> > > +	}
> > > +
> > > +	destvms_cp_sz = sizeof(*destvms_cp) * nr_dest;
> > > +	destvms_cp = kzalloc(destvms_cp_sz, GFP_KERNEL);
> > > +
> > > +	if (!destvms_cp) {
> > > +		ret = -ENOMEM;
> > > +		goto out_unmap_source;
> > > +	}
> > > +
> > > +	for (i = 0; i < nr_dest; i++) {
> > > +		destvms_cp[i].vmid = cpu_to_le32(destvms[i]);
> > > +		destvms_cp[i].perm = cpu_to_le32(destvms_perms[i]);
> > > +		destvms_cp[i].ctx = 0;
> > > +		destvms_cp[i].ctx_size = 0;
> > > +	}
> > > +
> > > +	dest_dma_addr = dma_map_single(__scm->dev, destvms_cp,
> > > +				       destvms_cp_sz, DMA_TO_DEVICE);
> > > +	if (dma_mapping_error(__scm->dev, dest_dma_addr)) {
> > > +		ret = -ENOMEM;
> > > +		goto out_free_dest;
> > > +	}
> > > +
> > > +	mem_regions_buf = kcalloc(QCOM_SCM_MAX_BATCH_SECTION, sizeof(*mem_regions_buf),
> > > +				  GFP_KERNEL);
> > > +	if (!mem_regions_buf)
> > > +		return -ENOMEM;
> > > +
> > > +	while (batch_start < table->nents) {
> > > +		batch_size = 0;
> > > +		batch_iterator = 0;
> > > +
> > > +		do {
> > > +			mem_regions_buf[batch_iterator].mem_addr = page_to_phys(sg_page(curr_sgl));
> > > +			mem_regions_buf[batch_iterator].mem_size = curr_sgl->length;
> > > +			batch_size += curr_sgl->length;
> > > +			batch_iterator++;
> > > +			if (roll_back && curr_sgl == last_sgl)
> > > +				break;

Is roll_back bool necessary? last_sgl is NULL in case of not
rolling back and I think is good sentinel value when not rolling back.

If last_sgl is NULL, then all sgl entries are assigned. If last_sgl is
not NULL, then all the sgl entries are assigned until last_sgl is
encountered (inclusive).

> > > +			curr_sgl = sg_next(curr_sgl);
> > > +		} while (curr_sgl && batch_iterator < QCOM_SCM_MAX_BATCH_SECTION &&
> > > +				curr_sgl->length + batch_size < QCOM_SCM_MAX_BATCH_SIZE);
> > > +
> > > +		batch_start += batch_iterator;
> > > +
> > > +		ret = qcom_scm_assign_mem_batch(mem_regions_buf, batch_iterator,
> > > +						srcvms_cp, srcvms_cp_sz, destvms_cp, destvms_cp_sz);
> > > +
> > > +		if (ret) {
> > > +			dev_info(__scm->dev, "Failed to assign memory protection, ret = %d\n", ret);
> > > +			last_sgl = curr_sgl;
> > > +			ret = -EADDRNOTAVAIL;
> > 
> > Probably should not be changing the ret value.
> 
> We are adjusting the return value here because we have already printed the
> failed return value earlier. Our goal is to ensure that clients invoking
> this API consistently receive the same failed values, without needing to
> worry about the precise details of the failure.
> 

Clients will need special handling for ENOMEM, ENODEV, or EPROBEDEFER.
Clients can ignore the details of the error if they don't care, but
don't mask the error.

> > 
> > What happens to the memory that has been assigned so far? How do we
> > reclaim that?
> 
> We’ve already added rollback functionality in qcom_scm_assign_table that
> allows batches to be rolled back to the HLOS if there’s a failure.
> 
> > 
> > > +			break;
> > > +		}
> > > +		if (roll_back && curr_sgl == last_sgl)
> > > +			break;
> > > +	}
> > > +	kfree(mem_regions_buf);
> > > +
> > > +	dma_unmap_single(__scm->dev, dest_dma_addr,
> > > +			 destvms_cp_sz, DMA_TO_DEVICE);
> > > +
> > > +out_free_dest:
> > > +	kfree(destvms_cp);
> > > +out_unmap_source:
> > > +	dma_unmap_single(__scm->dev, source_dma_addr,
> > > +			 srcvms_cp_sz, DMA_TO_DEVICE);
> > > +out_free_source:
> > > +	kfree(srcvms_cp);
> > > +	return ret;
> > > +}
> > > +
> > > +/**
> > > + * qcom_scm_assign_table() - Make a call to prepare batches of memory regions
> > > + *			     and reassign memory ownership of several memory regions at once
> > > + * @sg_table:       A scatter list whose memory needs to be reassigned
> > > + * @srcvms:	    A buffer populated with he vmid(s) for the current set of
> > > + *		    owners
> > > + * @nr_src:	    The number of the src_vms buffer
> > > + * @destvms:	    A buffer populated with he vmid(s) for the new owners
> > > + * @destvms_perms:  A buffer populated with the permission flags of new owners
> > > + * @nr_dest:	    The number of the destvms
> > > + *
> > > + * Return negative errno on failure, 0 on success.
> > > + */
> > > +int qcom_scm_assign_table(struct sg_table *table,
> > > +			  u32 *srcvms, int nr_src,
> > > +			  int *destvms, int *destvms_perms,
> > > +			  int nr_dest)
> > > +{
> > > +	struct scatterlist *last_sgl = NULL;
> > > +	int rb_ret = 0;
> > > +	u32 new_dests;
> > > +	int new_perms;
> > > +	int ret = 0;
> > > +
> > > +	ret = qcom_scm_prepare_mem_batch(table, srcvms, nr_src,
> > > +					 destvms, destvms_perms, nr_dest, last_sgl, false);
> > > +
> > > +	if (!ret)
> > > +		goto out;
> > > +	new_dests = QCOM_SCM_VMID_HLOS;
> > 
> > We have the original srcvms. Will it always be HLOS? If so, then do we
> > need to pass srcvms at all?
> 
> No, srcvms can be VMs other than HLOS. However, this API does not expect
> clients to provide srcvms permissions. As a result, when such permissions
> are absent, we proceed by assigning the previously processed batches to
> HLOS.
> 

This is a wrong assumption though, right? You are allowing srcvms to be
something other than HLOS but assuming HLOS VMID is the only one in the
error path.

> > 
> > > +	new_perms = QCOM_SCM_PERM_EXEC | QCOM_SCM_PERM_WRITE | QCOM_SCM_PERM_READ;
> > > +	rb_ret = qcom_scm_prepare_mem_batch(table, destvms, nr_dest, &new_dests,
> > > +					    &new_perms, nr_src, last_sgl, true);
> > > +	WARN_ON_ONCE(rb_ret);
> > > +out:
> > > +	return ret;
> > > +}
> > > +EXPORT_SYMBOL_GPL(qcom_scm_assign_table);
> > 
> > Who uses this function?
> 
> https://lore.kernel.org/all/cover.1700544802.git.quic_vjitta@xxxxxxxxxxx/
> 
> This patch responsible for extending qcom secure heap support which will be
> the user of this API. In the current implementation, we use
> qcom_scm_assign_mem and iterate over each sg_list in the sg_table
> individually to secure the memory. Going forward, we will replace this
> approach with our API.
> 
> > 
> > > +
> > >   /**
> > >    * qcom_scm_ocmem_lock_available() - is OCMEM lock/unlock interface available
> > >    */
> > > diff --git a/include/linux/firmware/qcom/qcom_scm.h b/include/linux/firmware/qcom/qcom_scm.h
> > > index ccaf28846054..abd675c7ef49 100644
> > > --- a/include/linux/firmware/qcom/qcom_scm.h
> > > +++ b/include/linux/firmware/qcom/qcom_scm.h
> > > @@ -8,6 +8,7 @@
> > >   #include <linux/err.h>
> > >   #include <linux/types.h>
> > >   #include <linux/cpumask.h>
> > > +#include <linux/scatterlist.h>
> > >   #include <dt-bindings/firmware/qcom,scm.h>
> > > @@ -15,6 +16,8 @@
> > >   #define QCOM_SCM_CPU_PWR_DOWN_L2_ON	0x0
> > >   #define QCOM_SCM_CPU_PWR_DOWN_L2_OFF	0x1
> > >   #define QCOM_SCM_HDCP_MAX_REQ_CNT	5
> > > +#define QCOM_SCM_MAX_BATCH_SECTION	32
> > > +#define QCOM_SCM_MAX_BATCH_SIZE		SZ_2M

Move the macros to qcom_scm.c

> > >   struct qcom_scm_hdcp_req {
> > >   	u32 addr;
> > > @@ -93,6 +96,10 @@ int qcom_scm_mem_protect_video_var(u32 cp_start, u32 cp_size,
> > >   int qcom_scm_assign_mem(phys_addr_t mem_addr, size_t mem_sz, u64 *src,
> > >   			const struct qcom_scm_vmperm *newvm,
> > >   			unsigned int dest_cnt);
> > > +int qcom_scm_assign_table(struct sg_table *table,
> > > +			  u32 *srcvms, int nr_src,
> > > +			  int *destvms, int *destvms_perms,
> > > +			  int nr_dest);
> > >   bool qcom_scm_ocmem_lock_available(void);
> > >   int qcom_scm_ocmem_lock(enum qcom_scm_ocmem_client id, u32 offset, u32 size,
> > > -- 
> > > 2.34.1
> > > 
>