Re: [PATCH 0/3] KVM: arm64: Handle CCSIDR associativity mismatches

From: Akihiko Odaki
Date: Thu Dec 01 2022 - 12:26:18 EST


On 2022/12/01 20:06, Marc Zyngier wrote:
On Thu, 01 Dec 2022 10:49:11 +0000,
Akihiko Odaki <akihiko.odaki@xxxxxxxxxx> wrote:

Thanks for looking into this.

M2 MacBook Air has mismatched CCSIDR associativity bits, which makes the
bits a KVM vCPU sees inconsistent when migrating.

Can you describe the actual discrepancy? Is that an issue between the
two core types? In which case, nothing says that these two cluster
should have the same cache topology.

Yes, the processor has big.LITTLE configuration.

On the processor, the valid CSSELR values are 0 (L1D), 1 (L1I), 3 (L2D). For each CSSELR values, each cluster has:
- 0x700FE03A, 0x203FE01A, 0x70FFE07B
- 0x701FE03A, 0x203FE02A, 0x73FFE07B


It also makes QEMU fail restoring the vCPU registers because QEMU saves
and restores all of the registers including CCSIDRs, and if the vCPU
migrated among physical CPUs between saving and restoring, it tries to
restore CCSIDR values that mismatch with the current physical CPU, which
causes EFAULT.

Well, QEMU will have plenty of other problems, starting with MIDRs,
which always reflect the physical one. In general, KVM isn't well
geared for VMs spanning multiple CPU types. It is improving, but there
is a long way to go.

On M2 MacBook Air, I have seen no other difference in standard ID registers and CCSIDRs are exceptions. Perhaps Apple designed this way so that macOS's Hypervisor can freely migrate vCPU, but I can't assure that without more analysis. This is still enough to migrate vCPU running Linux at least.


Trap CCSIDRs if there are CCSIDR value msimatches, and override the
associativity bits when handling the trap.

TBH, I'd rather we stop reporting this stuff altogether.

There is nothing a correctly written arm64 guest should do with any of
this (this is only useful for set/way CMOs, which non-secure SW should
never issue). It would be a lot better to expose a virtual topology
(one set, one way, one level). It would also save us from the CCSIDRX
silliness.

The only complexity would be to still accept different topologies from
userspace so that we can restore a VM saved before this virtual
topology.

Another (minor) concern is that trapping relevant registers may cost too much. Currently KVM traps CSSELR and CCSIDR accesses with HCR_TID2, but HCR_TID2 also affects CTR_EL0. Although I'm not sure if the register is referred frequently, Arm introduced FEAT_EVT to trap CSSELR and CSSIDR but not CTR_EL0 so there may be some case where trapping CTR_EL0 is not tolerated. Perhaps Arm worried that a userspace application may read CTR_EL0 frequently.

If you think the concern on VM restoration you mentioned and the trapping overhead is tolerable, I'll write a new, much smaller patch accordingly.

Regards,
Akihiko Odaki


Do you mind having a look at this?

Thanks,

M.