Re: [PATCH v1 13/24] kvm: arm64: Add CPU entry point in nVHE hyp

[Marc Zyngier]

On 2020-11-09 11:32, David Brazdil wrote:
> When nVHE hyp starts interception host's PSCI CPU_ON SMCs, it will need
> to install KVM on the newly booted CPU before returning to the host. 
> Add
> an entry point which expects the same kvm_nvhe_init_params struct as 
> the
> __kvm_hyp_init HVC in the CPU_ON context argument (x0).
>
> The entry point initializes EL2 state with the same init_el2_state 
> macro
> used by the kernel's entry point. It then initializes KVM using the 
> same
> helper function used in the __kvm_hyp_init HVC.
>
> When done, the entry point branches to a function provided in the init
> params.
>
> Signed-off-by: David Brazdil <dbrazdil@xxxxxxxxxx>
> ---
>  arch/arm64/include/asm/kvm_asm.h   |  1 +
>  arch/arm64/kernel/asm-offsets.c    |  1 +
>  arch/arm64/kvm/hyp/nvhe/hyp-init.S | 30 ++++++++++++++++++++++++++++++
>  3 files changed, 32 insertions(+)
>
> diff --git a/arch/arm64/include/asm/kvm_asm.h 
> b/arch/arm64/include/asm/kvm_asm.h
> index 893327d1e449..efb4872bb29f 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -155,6 +155,7 @@ struct kvm_nvhe_init_params {
>  	unsigned long tpidr_el2;
>  	unsigned long hyp_stack_ptr;
>  	unsigned long vector_ptr;
> +	unsigned long psci_cpu_entry_fn;
>  };
>
>  /* Translate a kernel address @ptr into its equivalent linear mapping 
> */
> diff --git a/arch/arm64/kernel/asm-offsets.c 
> b/arch/arm64/kernel/asm-offsets.c
> index 0cbb86135c7c..ffc84e68ad97 100644
> --- a/arch/arm64/kernel/asm-offsets.c
> +++ b/arch/arm64/kernel/asm-offsets.c
> @@ -114,6 +114,7 @@ int main(void)
>    DEFINE(NVHE_INIT_TPIDR_EL2,	offsetof(struct kvm_nvhe_init_params,
> tpidr_el2));
>    DEFINE(NVHE_INIT_STACK_PTR,	offsetof(struct kvm_nvhe_init_params,
> hyp_stack_ptr));
>    DEFINE(NVHE_INIT_VECTOR_PTR,	offsetof(struct kvm_nvhe_init_params,
> vector_ptr));
> +  DEFINE(NVHE_INIT_PSCI_CPU_ENTRY_FN,	offsetof(struct
> kvm_nvhe_init_params, psci_cpu_entry_fn));
>  #endif
>  #ifdef CONFIG_CPU_PM
>    DEFINE(CPU_CTX_SP,		offsetof(struct cpu_suspend_ctx, sp));
> diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S
> b/arch/arm64/kvm/hyp/nvhe/hyp-init.S
> index 1697d25756e9..f999a35b2c8c 100644
> --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S
> +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S
> @@ -6,6 +6,7 @@
>
>  #include <linux/arm-smccc.h>
>  #include <linux/linkage.h>
> +#include <linux/irqchip/arm-gic-v3.h>

This should probably be included from the file that provides 
init_el2_state.

>  #include <asm/alternative.h>
>  #include <asm/assembler.h>
> @@ -159,6 +160,35 @@ alternative_else_nop_endif
>  	ret
>  SYM_CODE_END(___kvm_hyp_init)
>
> +SYM_CODE_START(__kvm_hyp_cpu_entry)
> +	msr	SPsel, #1			// We want to use SP_EL{1,2}
> +
> +	/*
> +	 * Check that the core was booted in EL2. Loop indefinitely if not
> +	 * because it cannot be safely given to the host without installing 
> KVM.
> +	 */
> +	mrs	x1, CurrentEL
> +	cmp	x1, #CurrentEL_EL2
> +	b.ne	.

This is a bit brutal. Consider using a WFE/WFI loop as we have in other
places already (see __secondary_too_slow for example).

> +
> +	/* Initialize EL2 CPU state to sane values. */
> +	mov	x29, x0
> +	init_el2_state nvhe
> +	mov	x0, x29
> +
> +	/*
> +	 * Load hyp VA of C entry function. Must do so before switching on 
> the
> +	 * MMU because the struct pointer is PA and not identity-mapped in 
> hyp.
> +	 */
> +	ldr	x29, [x0, #NVHE_INIT_PSCI_CPU_ENTRY_FN]
> +
> +	/* Enable MMU, set vectors and stack. */
> +	bl	___kvm_hyp_init
> +
> +	/* Leave idmap. */
> +	br	x29

To a point I made against an earlier patch: psci_cpu_entry_fn seems to 
be a HYP
VA, and really needs to be documented as such, because this is pretty 
hard to
follow otherwise.

> +SYM_CODE_END(__kvm_hyp_cpu_entry)
> +
>  SYM_CODE_START(__kvm_handle_stub_hvc)
>  	cmp	x0, #HVC_SOFT_RESTART
>  	b.ne	1f

Thanks,

        M.
--
Jazz is not dead. It just smells funny...