Re: [PATCH v2 00/17] x86/resctrl : Support AMD Assignable Bandwidth Monitoring Counters (ABMC)

[Peter Newman]

Hi Reinette,

On Thu, Mar 7, 2024 at 12:41 PM Reinette Chatre
<reinette.chatre@xxxxxxxxx> wrote:
>
> Hi Peter,
>
> On 3/7/2024 10:57 AM, Peter Newman wrote:
> > Hi Babu,
> >
> > On Mon, Mar 4, 2024 at 2:24 PM Moger, Babu <bmoger@xxxxxxx> wrote:
> >> Based on our discussion, I am listing few examples here. Let me know if
> >> I missed something.
> >>
> >>    mount  -t resctrl resctrl /sys/fs/resctrl/
> >>
> >> 1. Assign both local and total counters to default group on domain 0 and 1.
> >>     $echo "//00=lt;01=lt" > /sys/fs/resctrl/info/L3_MON/mbm_assign_control
> >>
> >>     $cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
> >>     //00=lt;01=lt
> >>
> >> 2. Assign a total event to mon group inside the default group for both
> >> domain 0 and 1.
> >>
> >>     $mkdir /sys/fs/resctrl/mon_groups/mon_a
> >>     $echo "/mon_a/00+t;01+t" >
> >> /sys/fs/resctrl/info/L3_MON/mbm_assign_control
> >>
> >>     $cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
> >>     //00=lt;01=lt
> >>     /mon_a/00=t;01=t
> >>
> >> 3. Assign a local event to non-default control mon group both domain 0
> >> and 1.
> >>     $mkdir /sys/fs/resctrl/ctrl_a
> >>     $echo "/ctrl_a/00=l;01=l"  >
> >> /sys/fs/resctrl/info/L3_MON/mbm_assign_control
> >>
> >>     $cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
> >>     //00=lt;01=lt
> >>     /mon_a/00=t;01=t
> >>     /ctrl_a/00=l;01=l
> >>
> >> 4. Assign a both counters to mon group inside another control
> >> group(non-default).
> >>     $mkdir /sys/fs/resctrl/ctrl_a/mon_ab/
> >>     $echo "ctrl_a/mon_ab/00=lt;01=lt" >
> >> /sys/fs/resctrl/nfo/L3_MON/mbm_assign_contro
> >>
> >>     $cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
> >>     //00=lt;01=lt
> >>     /mon_a/00=t;01=t
> >>     /ctrl_a/00=l;01=l
> >>     ctrl_a/mon_ab/00=lt;01=lt
> >>
> >> 5. Unassign a counter to mon group inside another control
> >> group(non-default).
> >>     $echo "ctrl_a/mon_ab/00-l;01-l" >
> >> /sys/fs/resctrl/nfo/L3_MON/mbm_assign_control
> >>
> >>    $cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
> >>    //00=lt;01=lt
> >>    /mon_a/00=t;01=t
> >>    /ctrl_a/00=l;01=l
> >>    ctrl_a/mon_ab/00=t;01=t
> >>
> >> 6. Unassign all the counters on a specific group.
> >>     $echo "ctrl_a/mon_ab/00=_" >
> >> /sys/fs/resctrl/nfo/L3_MON/mbm_assign_control
> >>
> >>     $cat /sys/fs/resctrl/info/L3_MON/mbm_assign_control
> >>     //00=lt;01=lt
> >>     /mon_a/00=t;01=t
> >>     /ctrl_a/00=l;01=l
> >>     ctrl_a/mon_ab/00=_;01=_
> >
> > The use case I'm interested in is iterating 32 counters over 256
> > groups[1]. If it's not possible to reassign 32 counters in a single
> > write system call, with just one IPI per domain per batch reassignment
> > operation, then I don't see any advantage over the original proposal
> > with the assignment control file in every group directory. We already
> > had fine-grained control placing assign/unassign nodes throughout the
> > directory hierarchy, with the scope implicit in the directory
> > location.
>
> The intent of this interface is to support modification of several
> groups with a single write. These examples only show impact to a single
> group at a time, but multiple groups can be modified by separating
> configurations with a "\n". I believe Babu was planning to add some
> of these examples in his next iteration since it is not obvious yet.
>
> >
> > The interface I proposed in [1] aims to reduce the per-domain IPIs by
> > a factor of the number of counters, rather than sending off 2 rounds
> > of IPIs to each domain for each monitoring group.
>
> I understood the proposed interface appeared to focus on one use case
> while the goal is to find an interface to support all requirements.
> With this proposed interface it it possible to make large scale changes
> with a single sysfs write.

Ok I see you requested[1] one such example earlier.

>From what I've read, is this what you had in mind of reassigning 32
counters from the first 16 groups to the next?

I had found that it's hard to get a single write() syscall out of a
string containing newlines, so I'm using one explicit call:

write([mbm_assign_control fd],
"/c1/m1/00=_;02=_;03=_;04=_;05=_;06=_;07=_;08=_;09=_;10=_;11=_;12=_;13=_;14=_;15=_\n"
"/c1/m2/00=_;01=_;02=_;03=_;04=_;05=_;06=_;07=_;08=_;09=_;10=_;11=_;12=_;13=_;14=_;15=_\n"
"/c1/m3/00=_;01=_;02=_;03=_;04=_;05=_;06=_;07=_;08=_;09=_;10=_;11=_;12=_;13=_;14=_;15=_\n"
[...]
"/c1/m14/00=_;01=_;02=_;03=_;04=_;05=_;06=_;07=_;08=_;09=_;10=_;11=_;12=_;13=_;14=_;15=_\n"
"/c1/m15/00=_;01=_;02=_;03=_;04=_;05=_;06=_;07=_;08=_;09=_;10=_;11=_;12=_;13=_;14=_;15=_\n"
"/c1/m16/00=lt;01=lt;02=lt;03=lt;04=lt;05=lt;06=lt;07=lt;08=lt;09=lt;10=lt;11=lt;12=lt;13=lt;14=lt;15=lt\n"
"/c1/m17/00=lt;01=lt;02=lt;03=lt;04=lt;05=lt;06=lt;07=lt;08=lt;09=lt;10=lt;11=lt;12=lt;13=lt;14=lt;15=lt\n"
"/c1/m18/00=lt;01=lt;02=lt;03=lt;04=lt;05=lt;06=lt;07=lt;08=lt;09=lt;10=lt;11=lt;12=lt;13=lt;14=lt;15=lt\n"
[...]
"/c1/m30/00=lt;01=lt;02=lt;03=lt;04=lt;05=lt;06=lt;07=lt;08=lt;09=lt;10=lt;11=lt;12=lt;13=lt;14=lt;15=lt\n"
"/c1/m31/00=lt;01=lt;02=lt;03=lt;04=lt;05=lt;06=lt;07=lt;08=lt;09=lt;10=lt;11=lt;12=lt;13=lt;14=lt;15=lt\n",
size);

Thanks!
-Peter

[1] https://lore.kernel.org/lkml/d29a28c8-1180-45ec-bd87-d2e8a8124c42@intelcom/