Re: [PATCH v3] x86/asm: Force native_apic_mem_read to use mov

[Adam Dunlap]

On Fri, Feb 9, 2024 at 7:22 AM Ard Biesheuvel <ardb@xxxxxxxxxx> wrote:
>
> On Thu, 8 Feb 2024 at 16:48, Dave Hansen <dave.hansen@xxxxxxxxx> wrote:
> >
> > On 2/6/24 14:36, Adam Dunlap wrote:
> > ...
> > > In particular, when compiled with clang and run as a SEV-ES or
> > > SEV-SNP guest, the compiler would emit a testl instruction which is
> > > not supported by the SEV-ES emulator
> >
> > What changed?  Why is this a bug that we're only noticing now?  The line
> > of code that's modified here is from 2008.
> >
> > I assume that it's something new in clang, but it'd be great to know
> > that for sure.
> >
>
> Might be the use of LTO in the Google prod[uction]kernel. Adam, can you confirm?

It doesn't look like it's LTO. I disabled the LTO options in .config
and you can see the
problem just in a single object file:

With gcc:

% gdb -batch -ex 'file arch/x86/kernel/apic/ipi.o' -ex 'disassemble
apic_mem_wait_icr_idle'
Dump of assembler code for function apic_mem_wait_icr_idle:
   0x0000000000000260 <+0>:     endbr64
   0x0000000000000264 <+4>:     jmp    0x268 <apic_mem_wait_icr_idle+8>
   0x0000000000000266 <+6>:     pause
   0x0000000000000268 <+8>:     mov    0xffffffffff5fc300,%eax
   0x000000000000026f <+15>:    test   $0x10,%ah
   0x0000000000000272 <+18>:    jne    0x266 <apic_mem_wait_icr_idle+6>
   0x0000000000000274 <+20>:    jmpq   0x279

With clang:

% gdb -batch -ex 'file arch/x86/kernel/apic/ipi.o' -ex 'disassemble
apic_mem_wait_icr_idle'
Dump of assembler code for function apic_mem_wait_icr_idle:
   0x0000000000000250 <+0>:     endbr64
   0x0000000000000254 <+4>:     testl  $0x1000,0xffffffffff5fc300
   0x000000000000025f <+15>:    je     0x270 <apic_mem_wait_icr_idle+32>
   0x0000000000000261 <+17>:    pause
   0x0000000000000263 <+19>:    testl  $0x1000,0xffffffffff5fc300
   0x000000000000026e <+30>:    jne    0x261 <apic_mem_wait_icr_idle+17>
   0x0000000000000270 <+32>:    cs jmpq 0x276

This shows how gcc uses mov to load the apic memory and then testl to
test it, while clang
combines those instructions.

I plugged in the relevant subsection into godbolt [0] and it appears
the assembly changed in
clang 8 (released 2019). I'm not set up to do full compilations with
old clang versions, but
this is the most likely change point.

> this seems like the kind of thing we'll want in -stable in case folks
> are compiling stable kernels with new clangs.

That makes sense. Note that there was another patch accepted recently
that fixed another
clang-with-SEV problem [1], so they should probably be backported to
the same stable
branches since neither is that useful without the other.

[0] https://godbolt.org/z/nq9M9e8ex
[1] https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=1c811d403afd73f04bde82b83b24c754011bd0e8