Re: [Patch v4 07/13] perf/x86: Add constraint for guest perf metrics event

[Mi, Dapeng]

On 9/28/2023 1:27 AM, Sean Christopherson wrote:
> +Jim, David, and Mingwei
>
> On Wed, Sep 27, 2023, Peter Zijlstra wrote:
> > On Wed, Sep 27, 2023 at 11:31:18AM +0800, Dapeng Mi wrote:
> > > When guest wants to use PERF_METRICS MSR, a virtual metrics event needs
> > > to be created in the perf subsystem so that the guest can have exclusive
> > > ownership of the PERF_METRICS MSR.
> > Urgh, can someone please remind me how all that is supposed to work
> > again? The guest is just a task that wants the event. If the
> > host creates a CPU event, then that gets scheduled with higher priority
> > and the task looses out, no joy.


It looks I used the inaccurate words in the comments. Yes, it's not 
*exclusive* from host's point view.  Currently the perf events created 
by KVM are task-pinned events, they are indeed possible to be preempted 
by CPU-pinned host events which have higher priority. This is a long 
term issue which vPMU encountered. We ever have some internal discussion 
about this issue, but it seems we don't have a good way to solve this 
issue thoroughly in current vPMU framework.

But if there is no such CPU-pinned events which have the highest 
priority on host, KVM perf events can share the HW resource with other 
host events with the way of time-multiplexing.

> > So you cannot guarantee the guest gets anything.
> >
> > That is, I remember we've had this exact problem before, but I keep
> > forgetting how this all is supposed to work. I don't use this virt stuff
> > (and every time I try qemu arguments defeat me and I give up in
> > disgust).
> I don't think it does work, at least not without a very, very carefully crafted
> setup and a host userspace that knows it must not use certain aspects of perf.
> E.g. for PEBS, if the guest virtual counters don't map 1:1 to the "real" counters
> in hardware, KVM+perf simply disables the counter.
>
> And for top-down slots, getting anything remotely accurate requires pinning vCPUs
> 1:1 with pCPUs and enumerating an accurate toplogy to the guest:
>
>    The count is distributed among unhalted logical processors (hyper-threads) who
>    share the same physical core, in processors that support Intel Hyper-Threading
>    Technology.
>
> Jumping the gun a bit (we're in the *super* early stages of scraping together a
> rough PoC), but I think we should effectively put KVM's current vPMU support into
> maintenance-only mode, i.e. stop adding new features unless they are *very* simple
> to enable, and instead pursue an implementation that (a) lets userspace (and/or
> the kernel builder) completely disable host perf (or possibly just host perf usage
> of the hardware PMU) and (b) let KVM passthrough the entire hardware PMU when it
> has been turned off in the host.
>
> I.e. keep KVM's existing best-offset vPMU support, e.g. for setups where the
> platform owner is also the VM ueer (running a Windows VM on a Linux box, hosting
> a Linux VM in ChromeOS, etc...).  But for anything advanced and for hard guarantees,
> e.g. cloud providers that want to expose fully featured vPMU to customers, force
> the platform owner to choose between using perf (or again, perf with hardware PMU)
> in the host, and exposing the hardware PMU to the guest.
>
> Hardware vendors are pushing us in the direction whether we like it or not, e.g.
> SNP and TDX want to disallow profiling the guest from the host, ARM has an
> upcoming PMU model where (IIUC) it can't be virtualized without a passthrough
> approach, Intel's hybrid CPUs are a complete trainwreck unless vCPUs are pinned,
> and virtualizing things like top-down slots, PEBS, and LBRs in the shared model
> requires an absurd amount of complexity throughout the kernel and userspace.
>
> Note, a similar idea was floated and rejected in the past[*], but that failed
> proposal tried to retain host perf+PMU functionality by making the behavior dynamic,
> which I agree would create an awful ABI for the host.  If we make the "knob" a
> Kconfig or kernel param, i.e. require the platform owner to opt-out of using perf
> no later than at boot time, then I think we can provide a sane ABI, keep the
> implementation simple, all without breaking existing users that utilize perf in
> the host to profile guests.
>
> [*] https://lore.kernel.org/all/CALMp9eRBOmwz=mspp0m5Q093K3rMUeAsF3vEL39MGV5Br9wEQQ@xxxxxxxxxxxxxx