Re: [PATCH] iommu/amd: Do not flush IRTE when only updating isRun and destination fields

[Alejandro Jimenez]

On 10/17/23 10:42, Suravee Suthikulpanit wrote:
> According to the recent update in the AMD IOMMU spec [1], the IsRun and
> Destination fields of the Interrupt Remapping Table Entry (IRTE) are not
> cached by the IOMMU hardware.
>
> Therefore, do not issue the INVALIDATE_INTERRUPT_TABLE command when
> updating IRTE[IsRun] and IRTE[Destination] when IRTE[GuestMode]=1, which
> should help improve IOMMU AVIC/x2AVIC performance.
>
> References:
> [1] AMD IOMMU Spec Revision (Rev 3.08-PUB)
> (Link: https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/specifications/48882_IOMMU.pdf)
>
> Cc: Joao Martins <joao.m.martins@xxxxxxxxxx>
> Cc: Alejandro Jimenez <alejandro.j.jimenez@xxxxxxxxxx>
> Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@xxxxxxx>

Built on top of v6.6-rc6 tag, and booted on EPYC Genoa host. Launched 380 vCPU guest with AVIC enabled and 16 VFs, and ran rds-ping workload that uses all of the VFs to send traffic. THis workload has been used in the past to stress IOMMU AVIC code paths. No issues found.

Tested-by: Alejandro Jimenez <alejandro.j.jimenez@xxxxxxxxxx>

> ---
>   drivers/iommu/amd/iommu.c | 20 ++++++++++++++++----
>   1 file changed, 16 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
> index 089886485895..d63590563d3e 100644
> --- a/drivers/iommu/amd/iommu.c
> +++ b/drivers/iommu/amd/iommu.c
> @@ -2970,8 +2970,8 @@ static int alloc_irq_index(struct amd_iommu *iommu, u16 devid, int count,
>   	return index;
>   }
>   
> -static int modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index,
> -			  struct irte_ga *irte)
> +static int __modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index,
> +			    struct irte_ga *irte)
>   {
>   	struct irq_remap_table *table;
>   	struct irte_ga *entry;
> @@ -2998,6 +2998,18 @@ static int modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index,
>   
>   	raw_spin_unlock_irqrestore(&table->lock, flags);
>   
> +	return 0;
> +}
> +
> +static int modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index,
> +			  struct irte_ga *irte)
> +{
> +	bool ret;
> +
> +	ret = __modify_irte_ga(iommu, devid, index, irte);
> +	if (ret)
> +		return ret;
> +
>   	iommu_flush_irt_and_complete(iommu, devid);
>   
>   	return 0;
> @@ -3681,8 +3693,8 @@ int amd_iommu_update_ga(int cpu, bool is_run, void *data)
>   	}
>   	entry->lo.fields_vapic.is_run = is_run;
>   
> -	return modify_irte_ga(ir_data->iommu, ir_data->irq_2_irte.devid,
> -			      ir_data->irq_2_irte.index, entry);
> +	return __modify_irte_ga(ir_data->iommu, ir_data->irq_2_irte.devid,
> +				ir_data->irq_2_irte.index, entry);
>   }
>   EXPORT_SYMBOL(amd_iommu_update_ga);
>   #endif