Re: [PATCH 00/15] Add support for clang LTO

From: Ard Biesheuvel
Date: Fri Nov 03 2017 - 16:23:26 EST


On 3 November 2017 at 19:26, Mark Rutland <mark.rutland@xxxxxxx> wrote:
> On Fri, Nov 03, 2017 at 11:11:45AM -0700, Nick Desaulniers wrote:
>> On Fri, Nov 3, 2017 at 11:09 AM, Mark Rutland <mark.rutland@xxxxxxx> wrote:
>> ently compile
>> > What's the minimum set of patches necessary to work with clang (ignoring
>> > LTO)?
>>
>> If you have a build of clang-5, then just patch 7 in this series to
>> work around the last compiler bug. If you build clang from source
>> from master, ToT, for arm64, then none. :)
>
> Thanks for the pointers.
>
> With patch 7 I can build a defconfig from the arm64 for-next/core
> branch, but it panics (in hyp) on my Juno R1 when initialising hyp.
>
> If I build that branch with the Linaro 17.05 GCC 6.3.1 toolchain, I get
> all the way to userspace.
>
> The same is true (for both compilers) with v4.14-rc7.
>
> I guess that in Google you haven't tested on a platform with EL2
> available?
>
> [ 1.301280] kvm [1]: 8-bit VMID
> [ 1.304416] kvm [1]: IDMAP page: 809e2000
> [ 1.308406] kvm [1]: HYP VA range: 800000000000:ffffffffffff
> [ 1.315077] kvm [1]: Hyp mode initialized successfully
> [ 1.320211] Kernel panic - not syncing: HYP panic:
> [ 1.320211] PS:800003c9 PC:ffff0000089e6fd8 ESR:86000004
> [ 1.320211] FAR:ffff0000089e6fd8 HPFAR:0000000009825000 PAR:0000000000000000
> [ 1.320211] VCPU:000804fc20001221
> [ 1.320211]
> [ 1.341947] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.0-rc7-dirty #3
> [ 1.348675] Hardware name: ARM Juno development board (r1) (DT)
> [ 1.354543] Call trace:
> [ 1.356977] [<ffff000008088ea4>] dump_backtrace+0x0/0x34c
> [ 1.362333] [<ffff000008089208>] show_stack+0x18/0x20
> [ 1.367345] [<ffff0000089c73ec>] dump_stack+0xc4/0xfc
> [ 1.372357] [<ffff0000080c8e1c>] panic+0x138/0x2b4
> [ 1.377109] [<ffff0000080c8ce4>] panic+0x0/0x2b4
> [ 1.381692] SMP: stopping secondary CPUs
> [ 2.571295] SMP: failed to stop secondary CPUs 0-3,5
> [ 2.576214] Kernel Offset: disabled
> [ 2.579670] CPU features: 0x002086
> [ 2.583039] Memory Limit: none
> [ 2.586073] ---[ end Kernel panic - not syncing: HYP panic:
> [ 2.586073] PS:800003c9 PC:ffff0000089e6fd8 ESR:86000004
> [ 2.586073] FAR:ffff0000089e6fd8 HPFAR:0000000009825000 PAR:0000000000000000
> [ 2.586073] VCPU:000804fc20001221
>
> The PC in question is:
>
> [mark@lakrids:~/src/linux]% uselinaro 17.05 aarch64-linux-gnu-addr2line -ife vmlinux ffff0000089e6fd8
> __init_stage2_translation
> /home/mark/src/linux/arch/arm64/kvm/hyp/s2-setup.c:38
>
> AFAICT, the generated assembly is only using PC-relative branches, and
> no absolute relocations:
>
> 0000000000000000 <__init_stage2_translation>:
> 0: d5380709 mrs x9, id_aa64mmfr0_el1
> 4: 12000928 and w8, w9, #0x7
> 8: 7100111f cmp w8, #0x4
> c: d3700928 ubfiz x8, x9, #16, #3
> 10: 54000188 b.hi 40 <__init_stage2_translation+0x40>
> 14: 9000000a adrp x10, 0 <__init_stage2_translation>
> 18: 92400929 and x9, x9, #0x7
> 1c: 9100014a add x10, x10, #0x0
> 20: f869794b ldr x11, [x10,x9,lsl #3]
> 24: 52800509 mov w9, #0x28 // #40
> 28: 321b03e0 orr w0, wzr, #0x20
> 2c: 321b03ea orr w10, wzr, #0x20
> 30: d61f0160 br x11
> 34: 52800480 mov w0, #0x24 // #36
> 38: 321e0bea orr w10, wzr, #0x1c
> 3c: 14000008 b 5c <__init_stage2_translation+0x5c>
> 40: 321c07e9 orr w9, wzr, #0x30
> 44: 14000004 b 54 <__init_stage2_translation+0x54>
> 48: 52800549 mov w9, #0x2a // #42
> 4c: 14000002 b 54 <__init_stage2_translation+0x54>
> 50: 52800589 mov w9, #0x2c // #44
> 54: 321d07ea orr w10, wzr, #0x18
> 58: 2a0903e0 mov w0, w9
> 5c: aa080148 orr x8, x10, x8
> 60: 929957ea mov x10, #0xffffffffffff3540 // #-51904
> 64: d5380729 mrs x9, id_aa64mmfr1_el1
> 68: f2b0000a movk x10, #0x8000, lsl #16
> 6c: d538072b mrs x11, id_aa64mmfr1_el1
> 70: 9148014c add x12, x10, #0x200, lsl #12
> 74: f2400d3f tst x9, #0xf
> 78: 927c0d69 and x9, x11, #0xf0
> 7c: 9a8c014a csel x10, x10, x12, eq
> 80: f100813f cmp x9, #0x20
> 84: aa0a0108 orr x8, x8, x10
> 88: 1a9f17e9 cset w9, eq
> 8c: aa094d08 orr x8, x8, x9, lsl #19
> 90: d51c2148 msr vtcr_el2, x8
> 94: d65f03c0 ret
>
> ... so I guess something is going wrong around kvm_call_hyp() where this
> is called from EL1.
>

Given that PC == FAR here, it seems kvm_call_hyp() is calling this
function at its EL1 virtual address, so I suspect the VHE alternatives
patching is miscompiled here.