Re: [PATCH v2 00/17] x86/resctrl : Support AMD Assignable Bandwidth Monitoring Counters (ABMC)

[James Morse]

Hi Babu,

On 19/01/2024 18:22, Babu Moger wrote:
> These series adds the support for Assignable Bandwidth Monitoring Counters
> (ABMC). It is also called QoS RMID Pinning feature
> 
> The feature details are documented in the  APM listed below [1].
> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
> Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth
> Monitoring (ABMC). The documentation is available at
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> 
> The patches are based on top of commit
> 1ac6b49423e83af2abed9be7fbdf2e491686c66b (tip/master)
> 
> # Introduction
> 
> AMD hardware can support 256 or more RMIDs. However, bandwidth monitoring
> feature only guarantees that RMIDs currently assigned to a processor will
> be tracked by hardware. The counters of any other RMIDs which are no longer
> being tracked will be reset to zero. The MBM event counters return
> "Unavailable" for the RMIDs that are not active.
>
> Users can create 256 or more monitor groups. But there can be only limited
> number of groups that can be give guaranteed monitoring numbers.  With ever
> changing configurations there is no way to definitely know which of these
> groups will be active for certain point of time. Users do not have the
> option to monitor a group or set of groups for certain period of time
> without worrying about RMID being reset in between.
>
> The ABMC feature provides an option to the user to assign an RMID to the
> hardware counter and monitor the bandwidth for a longer duration.
> The assigned RMID will be active until the user unassigns it manually.
> There is no need to worry about counters being reset during this period.
> Additionally, the user can specify a bitmask identifying the specific
> bandwidth types from the given source to track with the counter.

At a high level, if existing software can't use the counters, I'd prefer we move them into
perf. We're currently re-inventing the perf wheel. (this argument doesn't hold for the
llc_occupancy, which is a state not counter!)

But if this lets someone 'pin' the counters for the groups they monitor, then use existing
tools, that seems a good enough argument for doing this.


> Without ABMC enabled, monitoring will work in current mode without
> assignment option.

To check I understand: the counters will get spuriously reset a the whim of the hardware?


> # Linux Implementation
> 
> Linux resctrl subsystem provides the interface to count maximum of two
> memory bandwidth events per group, from a combination of available total
> and local events. Keeping the current interface, users can assign a maximum
> of 2 ABMC counters per group. User will also have the option to assign only
> one counter to the group. If the system runs out of assignable ABMC
> counters, kernel will display an error. Users need to unassign an already
> assigned counter to make space for new assignments.
> 
> 
> # Examples
> 
> a. Check if ABMC support is available
> 	#mount -t resctrl resctrl /sys/fs/resctrl/
> 
> 	#cat /sys/fs/resctrl/info/L3_MON/mon_features 
> 	llc_occupancy
> 	mbm_total_bytes
> 	mbm_total_bytes_config
> 	mbm_local_bytes
> 	mbm_local_bytes_config
> 	mbm_assign_capable ←  Linux kernel detected ABMC feature
> 
> b. Check if ABMC is enabled. By default, ABMC feature is disabled.
>    Monitoring works in legacy monitor mode when ABMC is not enabled.
> 
> 	#cat /sys/fs/resctrl/info/L3_MON/mbm_assign_enable
> 	0
> 
> c. There will be new file "monitor_state" for each monitor group when ABMC
>    feature is supported. However, monitor_state is not available if ABMC is
>    disabled.
> 	#cat /sys/fs/resctrl/monitor_state 
> 	Unsupported
> 	
> d. Read the event mbm_total_bytes and mbm_local_bytes. Without ABMC
>    enabled, monitoring will work in current mode without assignment option.
> 	
> 	# cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
> 	779247936
> 	# cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 
> 	765207488
> 	
> e. Enable ABMC mode.
> 
> 	#echo 1 > /sys/fs/resctrl/info/L3_MON/mbm_assign_enable
>         #cat /sys/fs/resctrl/info/L3_MON/mbm_assign_enable
>         1

Why does this mode need enabling? Can't it be enabled automatically on hardware that
supports it, or enabled implicitly when the first assignment attempt arrives?

I guess this is really needed for a reset - could we implement that instead? This way
there isn't an extra step user-space has to do to make the assignments work.


> f. Read the monitor states. By default, both total and local MBM
> 	events are in "unassign" state.
> 	
> 	#cat /sys/fs/resctrl/monitor_state
> 	total=unassign;local=unassign


> g. Read the event mbm_total_bytes and mbm_local_bytes. In ABMC mode,
>    the MBA events are not available until the user assigns the events
>    explicitly.

How does this fit with "monitoring will work in current mode without assignment option.".
You mentioned the hardware resets the counters when this mode is enabled, does it also
refuse to count until the MSR is programmed?

If so - is there any mileage in auto-assigning the first N RMID to counters when the
groups are created? This way existing user-space tools work until they exceed the limits
of hardware. From that point a counter needs to be unassigned from another group. (we'd
need to make it easy to find which groups have a counter assigned)


> 	#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
> 	Unsupported
> 	
> 	#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 
> 	Unsupported
> 
> h. The event llc_occupancy is not affected by ABMC mode. Users can still
>    read the llc_occupancy.
> 
> 	#cat /sys/fs/resctrl/mon_data/mon_L3_00/llc_occupancy 
> 	557056

{
	MPAM would be the same - because llc_occupancy isn't a counter its a view of the
	state, its possible to multiplex a single llc_occupancy counter behind the scenes
	to provide the value for as many groups as needed. I suspect any other
	architecture would  have the same property.
}

> i. Now assign the total event and read the monitor_state.
> 	
> 	#echo total=assign > /sys/fs/resctrl/monitor_state
> 	#cat /sys/fs/resctrl/monitor_state 
> 	total=assign;local=unassign
> 	
> j. Now that the total event is assigned. Read the total event.
> 	
> 	#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
> 	6136000
> 	
> k. Now assign the local event and read the monitor_state.
> 	
> 	#echo local=assign > /sys/fs/resctrl/monitor_state
> 	#cat /sys/fs/resctrl/monitor_state
> 	total=assign;local=assign
> 
>         Users can also assign both total and local events in one single
> 	command.
> 
> l. Now that both total and local events are assigned, read the events.
> 	
> 	#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
> 	6136000
> 	#cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes
> 	58694

(the bandwidth configuration stuff is the existing BMEC support right?)

>From user-space's perspective MPAM could be made to look the same.

There ought to be some indication to user-space of how many counters it can assign, this
number might be different for different resources. This won't be a problem today, but if
we had 'mbm_total_bytes' on the L2 cache, the number of counters may be different.

MPAM platforms are unlikely to support both 'mbm_total' and 'mbm_local',  I think this is
just a documentation problem to say that mbm_local can't be configured if its not
supported - user-space can't blindly assign both.

If the configuration is changed over time - I bet user-space needs a quick way to find
where the counters are currently assigned - walking the tree to find out is a bit rubbish.
A file that lists the "control_group_name[/mon_group_name]" would help.


Thanks,

James