[tip: x86/mm] x86/mm: Prevent early boot triple-faults with instrumentation

[tip-bot2 for Borislav Petkov]

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     b64dfcde1ca9cb82e38e573753f0c0db8fb841c2
Gitweb:        https://git.kernel.org/tip/b64dfcde1ca9cb82e38e573753f0c0db8fb841c2
Author:        Borislav Petkov <bp@xxxxxxx>
AuthorDate:    Fri, 17 Dec 2021 16:48:29 +01:00
Committer:     Borislav Petkov <bp@xxxxxxx>
CommitterDate: Wed, 22 Dec 2021 11:51:20 +01:00

x86/mm: Prevent early boot triple-faults with instrumentation

Commit in Fixes added a global TLB flush on the early boot path, after
the kernel switches off of the trampoline page table.

Compiler profiling options enabled with GCOV_PROFILE add additional
measurement code on clang which needs to be initialized prior to
use. The global flush in x86_64_start_kernel() happens before those
initializations can happen, leading to accessing invalid memory.
GCOV_PROFILE builds with gcc are still ok so this is clang-specific.

The second issue this fixes is with KASAN: for a similar reason,
kasan_early_init() needs to have happened before KASAN-instrumented
functions are called.

Therefore, reorder the flush to happen after the KASAN early init
and prevent the compilers from adding profiling instrumentation to
native_write_cr4().

Fixes: f154f290855b ("x86/mm/64: Flush global TLB on boot and AP bringup")
Reported-by: "J. Bruce Fields" <bfields@xxxxxxxxxxxx>
Reported-by: kernel test robot <oliver.sang@xxxxxxxxx>
Signed-off-by: Borislav Petkov <bp@xxxxxxx>
Tested-by: Carel Si <beibei.si@xxxxxxxxx>
Tested-by: "J. Bruce Fields" <bfields@xxxxxxxxxxxx>
Link: https://lore.kernel.org/r/20211209144141.GC25654@xsang-OptiPlex-9020
---
 arch/x86/kernel/cpu/common.c |  2 +-
 arch/x86/kernel/head64.c     | 16 ++++++++++++++--
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 0083464..79b3d67 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -384,7 +384,7 @@ set_register:
 }
 EXPORT_SYMBOL(native_write_cr0);
 
-void native_write_cr4(unsigned long val)
+void __no_profile native_write_cr4(unsigned long val)
 {
 	unsigned long bits_changed = 0;
 
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 75acb60..f5e80a8 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -483,10 +483,12 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
 	/* Kill off the identity-map trampoline */
 	reset_early_page_tables();
 
-	__native_tlb_flush_global(native_read_cr4());
-
 	clear_bss();
 
+	/*
+	 * This needs to happen *before* kasan_early_init() because latter maps stuff
+	 * into that page.
+	 */
 	clear_page(init_top_pgt);
 
 	/*
@@ -498,6 +500,16 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
 
 	kasan_early_init();
 
+	/*
+	 * Flush global TLB entries which could be left over from the trampoline page
+	 * table.
+	 *
+	 * This needs to happen *after* kasan_early_init() as KASAN-enabled .configs
+	 * instrument native_write_cr4() so KASAN must be initialized for that
+	 * instrumentation to work.
+	 */
+	__native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4));
+
 	idt_setup_early_handler();
 
 	copy_bootdata(__va(real_mode_data));