[PATCH v3 00/37] KVM: x86: TLB flushing fixes and enhancements

From: Sean Christopherson
Date: Fri Mar 20 2020 - 17:28:45 EST


VMX TLB flushing cleanup series to fix a variety of bugs, and to avoid
unnecessary TLB flushes on nested VMX transitions.

1) Nested VMX doesn't properly flush all ASIDs/contexts on system events,
e.g. on mmu_notifier invalidate all contexts for L1 *and* L2 need to
be invalidated, but KVM generally only flushes L1 or L2 (or just L1).

2) #1 is largely benign because nested VMX always flushes the new
context on nested VM-Entry/VM-Exit.

High level overview:

a) Fix the main TLB flushing bug with a big hammer.

b) Fix a few other flushing related bugs.

c) Clean up vmx_tlb_flush(), i.e. what was v1 of this series.

d) Reintroduce current-ASID/context flushing to regain some of the
precision that got blasted away by the big hammer in #1.

e) Fix random code paths that unnecessarily trigger TLB flushes on
nested VMX transitions.

f) Stop flushing on every nested VMX transition.

g) Extra cleanup.


v3:
- Fix freeing of roots during INVVPID, I botched things when tweaking
Junaid's original patch.
- Move "last vpid02" logic into nested VMX flushing helper. [Paolo]
- Split "skip tlb flush" and "skip mmu sync" logic during fast roots
switch. [Paolo]
- Unconditionally skip tlb flush during fast roots switch on nested VMX
transitions, i.e. let nested_vmx_transition_tlb_flush() do the work.
This avoids flushing when EPT=0 and VPID=1. [Paolo]
- Do more cr3->pgd conversions in related code that drove me bonkers
when trying to figure out what was broken with VPID. I think this
knocks off the last code that uses "cr3" for variables/functions that
work with cr3 or eptp.

v2:
- Basically a new series.

v1:
- https://patchwork.kernel.org/cover/11394987/

Junaid Shahid (2):
KVM: nVMX: Invalidate all roots when emulating INVVPID without EPT
KVM: x86: Sync SPTEs when injecting page/EPT fault into L1

Sean Christopherson (35):
KVM: VMX: Flush all EPTP/VPID contexts on remote TLB flush
KVM: nVMX: Validate the EPTP when emulating INVEPT(EXTENT_CONTEXT)
KVM: nVMX: Invalidate all EPTP contexts when emulating INVEPT for L1
KVM: x86: Export kvm_propagate_fault() (as
kvm_inject_emulated_page_fault)
KVM: x86: Consolidate logic for injecting page faults to L1
KVM: VMX: Skip global INVVPID fallback if vpid==0 in
vpid_sync_context()
KVM: VMX: Use vpid_sync_context() directly when possible
KVM: VMX: Move vpid_sync_vcpu_addr() down a few lines
KVM: VMX: Handle INVVPID fallback logic in vpid_sync_vcpu_addr()
KVM: VMX: Drop redundant capability checks in low level INVVPID
helpers
KVM: nVMX: Use vpid_sync_vcpu_addr() to emulate INVVPID with address
KVM: x86: Move "flush guest's TLB" logic to separate kvm_x86_ops hook
KVM: VMX: Clean up vmx_flush_tlb_gva()
KVM: x86: Drop @invalidate_gpa param from kvm_x86_ops' tlb_flush()
KVM: SVM: Wire up ->tlb_flush_guest() directly to svm_flush_tlb()
KVM: VMX: Move vmx_flush_tlb() to vmx.c
KVM: nVMX: Move nested_get_vpid02() to vmx/nested.h
KVM: VMX: Introduce vmx_flush_tlb_current()
KVM: SVM: Document the ASID logic in svm_flush_tlb()
KVM: x86: Rename ->tlb_flush() to ->tlb_flush_all()
KVM: nVMX: Add helper to handle TLB flushes on nested VM-Enter/VM-Exit
KVM: x86: Introduce KVM_REQ_TLB_FLUSH_CURRENT to flush current ASID
KVM: x86/mmu: Use KVM_REQ_TLB_FLUSH_CURRENT for MMU specific flushes
KVM: nVMX: Selectively use TLB_FLUSH_CURRENT for nested
VM-Enter/VM-Exit
KVM: nVMX: Reload APIC access page on nested VM-Exit only if necessary
KVM: VMX: Retrieve APIC access page HPA only when necessary
KVM: VMX: Don't reload APIC access page if its control is disabled
KVM: x86/mmu: Move fast_cr3_switch() side effects to
__kvm_mmu_new_cr3()
KVM: x86/mmu: Add separate override for MMU sync during fast CR3
switch
KVM: x86/mmu: Add module param to force TLB flush on root reuse
KVM: nVMX: Skip MMU sync on nested VMX transition when possible
KVM: nVMX: Don't flush TLB on nested VMX transition
KVM: nVMX: Free only the affected contexts when emulating INVEPT
KVM: x86: Replace "cr3" with "pgd" in "new cr3/pgd" related code
KVM: VMX: Clean cr3/pgd handling in vmx_load_mmu_pgd()

arch/x86/include/asm/kvm_host.h | 25 +++-
arch/x86/kvm/mmu/mmu.c | 145 +++++++++----------
arch/x86/kvm/mmu/paging_tmpl.h | 2 +-
arch/x86/kvm/svm.c | 19 ++-
arch/x86/kvm/vmx/nested.c | 249 ++++++++++++++++++++++----------
arch/x86/kvm/vmx/nested.h | 7 +
arch/x86/kvm/vmx/ops.h | 32 ++--
arch/x86/kvm/vmx/vmx.c | 119 ++++++++++++---
arch/x86/kvm/vmx/vmx.h | 19 +--
arch/x86/kvm/x86.c | 67 ++++++---
arch/x86/kvm/x86.h | 6 +
11 files changed, 448 insertions(+), 242 deletions(-)

--
2.24.1