Re: [RFC PATCH v2 0/3] mm: mempolicy: Multi-tier weighted interleaving

[Gregory Price]

On Thu, Oct 19, 2023 at 02:28:42PM +0800, Huang, Ying wrote:
> Gregory Price <gregory.price@xxxxxxxxxxxx> writes:
> > Were you suggesting that weights should actually be part of
> > drivers/base/node.c?
> 
> Yes.  drivers/base/node.c vs. memory tiers.
>

Then yes I agree this can and probably should be placed there,
especially since I see accessor details are now being exposed at that
level, which can be used to auto-generate weights (assuming HMAT/CDAT
data exposed by devices is actually accurate).

> > Assuming this is your meaning, I agree and I will pivot to this.
> 
> Can you give a not-so-abstract example?  For example, on a system with
> node 0, 1, 2, 3, memory tiers 4 (0, 1), 22 (2, 3), ....  A workload runs
> on CPU of node 0, ...., interleaves memory on node 0, 1, ...  Then
> compare the different behavior (including memory bandwidth) with node
> and memory-tier based solution.

ah, I see.

Example 1: A single-socket system with multiple CXL memory devices
===
CPU Node: node0
CXL Nodes: node1, node2

Bandwidth attributes (in theory):
node0 - 8 channels - ~307GB/s
node1 - x16 link - 64GB/s
node2 - x8 link - 32GB/s

In a system like this, the optimal distribution of memory on an
interleave for maximizing bandwidth is about 76%/16%/8%.

for the sake of simplicity:  --weighted-interleave=0:76,1:16,0:8
but realistically we could make the weights sysfs values in the node

Regardless of the mechanism to engage this, the most effective way to
capture this in the system is by applying weights to nodes, not tiers.
If done in tiers, each node would be assigned to its own tier, making
the mechanism equivalent. So you might as well simplify the whole thing
and chop the memtier component out.

Is this configuration realistic? *shrug* - technically possible. And in
fact most hardware or driver based interleaving mechanisms would not
really be able to manage an interleave region across these nodes, at
least not without placing the x16 driver in x8 mode, or just having the
wrong distribution %'s.



Example 2: A dual-socket system with 1 CXL device per socket
===
CPU Nodes: node0, node1
CXL Nodes: node2, node3 (on sockets 0 and 1 respective)

Bandwidth Attributes (in theory):
nodes 0 & 1 - 8 channels - ~307GB/s ea.
nodes 2 & 3 - x16 link - 64GB/s ea.

This is similar to example #1, but with one difference:  A task running
on node 0 should not treat nodes 0 and 1 the same, nor nodes 2 and 3.
This is because on access to nodes 1 and 3, the cross-socket link (UPI,
or whatever AMD calls it) becomes a bandwidth chokepoint.

So from the perspective of node 0, the "real total" available bandwidth
is about 307GB+64GB+(41.6GB * UPI Links) in the case of intel.  so the
best result you could get is around 307+64+164=535GB/s if you have the
full 4 links.

You'd want to distribute the cross-socket traffic proportional to UPI,
not the total.

This leaves us with weights of:

node0 - 57%
node1 - 26%
node2 - 12%
node3 - 5%

Again, naturally nodes are the place to carry the weights here. In this
scenario, placing it in memory-tiers would require that 1 tier per node
existed.


Example 3: A single-socket system with 2 CXL devices
===
Different than example 1: Both devices are the same.

CPU Node: node0
CXL Nodes: node1, node2

Bandwidth attributes (in theory):
node0 - 8 channels - ~307GB/s
node1/2 - x16 link - 64GB/s

In this case you want the weights to be: 70/15/15% respectively

Caveat: A user may, in fact, use the CXL driver to implement a single
node which naturally interleaves the 2 devices. In this case it's the
same as a 1-socket, 1-device setup which is trivially 1-node-per-tier,
and therefore weights should live with nodes.

In the case of a single memory tier, you could simply make this 70/30.

However, and this the the argument against placing it in the
memory-tier: the user is free to hack-off any of the chosen numa nodes
via mempolicy, which makes the aggregated weight meaningless.

Example:  --weighted-interleave=0,1

Under my current code, if I set the weights to 70/30 in the memory-tiers
code, the result is that node1 inherits the full 30% defined in the
tier, which leads to a suboptimal distribution.  What you actually want
in this case is about 83/17% split.

However, this also presents an issue for placing the weights in the
nodes:  A node weight is meaningless outside the context of the active
context.  If I have 2 nodes and i set their weights to 70/30, and I hack
off node1, i can't have 70% of all memory go to node0, I have to send
100% of the memory to node0 - making the weight functionally
meaningless.

So this would imply a single global weight set on the nodes is ALSO a
bad solution, and instead it would be better to extend set_mempolicy
to have a --weighted-interleave option that reads HMAT/CDAT provided
bandwidth data and generates the weights for the selected nodes as
part of the policy.

The downside of this is that if the HMAT/CDAT data is garbage, the
policy becomes garbage.  To mitigate this, we should consider allowing
userland to override those values explicitly for the purpose of weighted
interleave should the HMAT/CDAT information be garbage/lies.

Another downside to this is that nodemask changes require recalculation
of weights, which may introduce some racey conditions, but that can
probably be managed.


Should we carry weights in node, memtier, or mempolicy?
===
The current analysis suggest carrying it in memory-tier would simply
require memory-tier to make 1 tier per node - which may or may not
be consistent with other goals of the memtier subsystem.

The pros of placing a weight in node is that the "effective" weight in
the context of the entire system can be calculated at the time nodes are
created.  If, at interleave time, the interface required a node+nodemask
then it's probably preferable to forego manual weighting, and simply
calculate based on HMAT/CDAT data.

The downside of placing it in nodes is that mempolicy is free to set the
interleave set to some combination of nodes, and this would prevent any
nodes created after process launch from being used in the interleave set
unless the software detected the hotplug event.  I don't know how much
of a real concern this is, but it is a limitation.

The other option is to add --weighted-interleave, but have mempolicy
generate the weights based on node-provided CDAT/HMAT data (or
overrides), which keep almost everything inside of mempolicy except for
a couple of interfaces to drivers/base/node.c that allow querying of
that data.



Summarize:
===
The weights are actually a function of bandwidth, and can probably be
calculated on the fly - rather than being manually set. However, we may
want to consider allowing the bandwidth attributes being exposed by
CDAT/HMAT to be overridden should the user discover they are functionally
incorrect. (for reference: I have seen this myself, where a device
published 5GB/s but actually achieves 22GB/s).

for reference, they are presently RO:

static DEVICE_ATTR_RO(property)
ACCESS_ATTR(read_bandwidth);
ACCESS_ATTR(read_latency);
ACCESS_ATTR(write_bandwidth);
ACCESS_ATTR(write_latency);


~Gregory