Re: [PATCH 3/4] x86: save r11 into pt_regs->eflags on SYSCALL64 fastpath

[Andy Lutomirski]

On Tue, Feb 24, 2015 at 10:51 AM, Denys Vlasenko <dvlasenk@xxxxxxxxxx> wrote:
> Before this patch, rcx and r11 were saved in pt_regs->rcx
> and pt_regs->r11. Which looks natural, but requires messy
> shuffling to/from iret stack whenever ptrace or e.g. iopl
> wants to modify return address or flags - because that's
> how these registers are used by SYSCALL/SYSRET.
>
> This patch saves rcx and r11 in pt_regs->rip and pt_regs->flags,
> and uses these values for SYSRET64 insn. Shuffling is eliminated.
>
> stub_iopl is no longer needed: pt_regs->flags needs no fixing up.
>
> Testing shows that syscall fast path is ~54.3 ns before
> and after the patch (on 2.7 GHz Sandy Bridge CPU).
>
> Signed-off-by: Denys Vlasenko <dvlasenk@xxxxxxxxxx>
> CC: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
> CC: Steven Rostedt <rostedt@xxxxxxxxxxx>
> CC: Ingo Molnar <mingo@xxxxxxxxxx>
> CC: Borislav Petkov <bp@xxxxxxxxx>
> CC: "H. Peter Anvin" <hpa@xxxxxxxxx>
> CC: Andy Lutomirski <luto@xxxxxxxxxxxxxx>
> CC: Oleg Nesterov <oleg@xxxxxxxxxx>
> CC: Frederic Weisbecker <fweisbec@xxxxxxxxx>
> CC: Alexei Starovoitov <ast@xxxxxxxxxxxx>
> CC: Will Drewry <wad@xxxxxxxxxxxx>
> CC: Kees Cook <keescook@xxxxxxxxxxxx>
> CC: x86@xxxxxxxxxx
> CC: linux-kernel@xxxxxxxxxxxxxxx
> ---
>  arch/x86/include/asm/calling.h   | 20 ++++++++++++++------
>  arch/x86/kernel/entry_64.S       | 33 +++++++++------------------------
>  arch/x86/syscalls/syscall_64.tbl |  2 +-
>  arch/x86/um/sys_call_table_64.c  |  2 +-
>  4 files changed, 25 insertions(+), 32 deletions(-)
>
> diff --git a/arch/x86/include/asm/calling.h b/arch/x86/include/asm/calling.h
> index f1a962f..4b5f7bf 100644
> --- a/arch/x86/include/asm/calling.h
> +++ b/arch/x86/include/asm/calling.h
> @@ -95,9 +95,11 @@ For 32-bit we have the following conventions - kernel is built with
>         CFI_ADJUST_CFA_OFFSET 15*8+\addskip
>         .endm
>
> -       .macro SAVE_C_REGS_HELPER offset=0 rax=1 rcx=1 r8plus=1
> -       .if \r8plus
> +       .macro SAVE_C_REGS_HELPER offset=0 rax=1 rcx=1 r8910=1 r11=1
> +       .if \r11
>         movq_cfi r11, 6*8+\offset
> +       .endif
> +       .if \r8910
>         movq_cfi r10, 7*8+\offset
>         movq_cfi r9,  8*8+\offset
>         movq_cfi r8,  9*8+\offset
> @@ -113,16 +115,19 @@ For 32-bit we have the following conventions - kernel is built with
>         movq_cfi rdi, 14*8+\offset
>         .endm
>         .macro SAVE_C_REGS offset=0
> -       SAVE_C_REGS_HELPER \offset, 1, 1, 1
> +       SAVE_C_REGS_HELPER \offset, 1, 1, 1, 1
>         .endm
>         .macro SAVE_C_REGS_EXCEPT_RAX_RCX offset=0
> -       SAVE_C_REGS_HELPER \offset, 0, 0, 1
> +       SAVE_C_REGS_HELPER \offset, 0, 0, 1, 1
>         .endm
>         .macro SAVE_C_REGS_EXCEPT_R891011
> -       SAVE_C_REGS_HELPER 0, 1, 1, 0
> +       SAVE_C_REGS_HELPER 0, 1, 1, 0, 0
>         .endm
>         .macro SAVE_C_REGS_EXCEPT_RCX_R891011
> -       SAVE_C_REGS_HELPER 0, 1, 0, 0
> +       SAVE_C_REGS_HELPER 0, 1, 0, 0, 0
> +       .endm
> +       .macro SAVE_C_REGS_EXCEPT_RAX_RCX_R11
> +       SAVE_C_REGS_HELPER 0, 0, 0, 1, 0
>         .endm
>

This is unnecessarily difficult to read.  Could you rework it to use
named macro parameters?

>         .macro SAVE_EXTRA_REGS offset=0
> @@ -179,6 +184,9 @@ For 32-bit we have the following conventions - kernel is built with
>         .macro RESTORE_C_REGS_EXCEPT_R11
>         RESTORE_C_REGS_HELPER 1,1,0,1,1
>         .endm
> +       .macro RESTORE_C_REGS_EXCEPT_RCX_R11
> +       RESTORE_C_REGS_HELPER 1,0,0,1,1
> +       .endm

Ditto.

>         .macro RESTORE_RSI_RDI
>         RESTORE_C_REGS_HELPER 0,0,0,0,0
>         .endm
> diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
> index 91af6be..2fd9349 100644
> --- a/arch/x86/kernel/entry_64.S
> +++ b/arch/x86/kernel/entry_64.S
> @@ -121,14 +121,12 @@ ENDPROC(native_usergs_sysret64)
>  #endif
>
>  /*
> - * C code is not supposed to know about undefined top of stack. Every time
> + * C code is not supposed to know that iret frame is not populated. Every time

"that the iret frame," please.  (ÐÑ ÐÐÐÐÑÐÑÐ ÐÐ-ÑÑÑÑÐÐ? :) )

>   * a C function with an pt_regs argument is called from the SYSCALL based
>   * fast path FIXUP_TOP_OF_STACK is needed.
>   * RESTORE_TOP_OF_STACK syncs the syscall state after any possible ptregs
>   * manipulation.
>   */
> -
> -       /* %rsp:at FRAMEEND */
>         .macro FIXUP_TOP_OF_STACK tmp offset=0
>         movq PER_CPU_VAR(old_rsp),\tmp
>         movq \tmp,RSP+\offset(%rsp)
> @@ -136,15 +134,13 @@ ENDPROC(native_usergs_sysret64)
>         movq $__USER_CS,CS+\offset(%rsp)
>         movq RIP+\offset(%rsp),\tmp  /* get rip */
>         movq \tmp,RCX+\offset(%rsp)  /* copy it to rcx as sysret would do */
> -       movq R11+\offset(%rsp),\tmp  /* get eflags */
> -       movq \tmp,EFLAGS+\offset(%rsp)
> +       movq EFLAGS+\offset(%rsp),\tmp /* ditto for rflags->r11 */
> +       movq \tmp,R11+\offset(%rsp)
>         .endm

It occurs to me that both the name of this macro and comment are
wrong.  It's not fixing the *top* of the stack, since it fixes both
rcx and r11.  Oh, well, maybe we'll just delete it eventually.

The patch looks correct.  Can you submit a v2 once I finish reading
the rest of these?  (Also, can you put v2 in the subject?  My email is
going nuts here.)

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/