Re: [PATCH RFC 1/4] drivers/base/node: Add demotion_nodes sys infterface
From: Zhijian Li (Fujitsu)
Date: Thu Nov 02 2023 - 01:55:07 EST
On 02/11/2023 13:18, Huang, Ying wrote:
> "Zhijian Li (Fujitsu)" <lizhijian@xxxxxxxxxxx> writes:
>
>>> We have /sys/devices/virtual/memory_tiering/memory_tier*/nodelist
>>> already. A node in a higher tier can demote to any node in the lower
>>> tiers. What's more need to be displayed in nodeX/demotion_nodes?
>>
>> IIRC, they are not the same. memory_tier[number], where the number is shared by
>> the memory using the same memory driver(dax/kmem etc). Not reflect the actual distance
>> across nodes(different distance will be grouped into the same memory_tier).
>> But demotion will only select the nearest nodelist to demote.
>
> In the following patchset, we will use the performance information from
> HMAT to place nodes using the same memory driver into different memory
> tiers.
>
> https://lore.kernel.org/all/20230926060628.265989-1-ying.huang@xxxxxxxxx/
Thanks for your reminder. It seems like I've fallen behind the world by months.
I will rebase on it later if this patch is still needed.
>
> The patch is in mm-stable tree.
>
>> Below is an example, node0 node1 are DRAM, node2 node3 are PMEM, but distance to DRAM nodes
>> are different.
>>
>> # numactl -H
>> available: 4 nodes (0-3)
>> node 0 cpus: 0
>> node 0 size: 964 MB
>> node 0 free: 746 MB
>> node 1 cpus: 1
>> node 1 size: 685 MB
>> node 1 free: 455 MB
>> node 2 cpus:
>> node 2 size: 896 MB
>> node 2 free: 897 MB
>> node 3 cpus:
>> node 3 size: 896 MB
>> node 3 free: 896 MB
>> node distances:
>> node 0 1 2 3
>> 0: 10 20 20 25
>> 1: 20 10 25 20
>> 2: 20 25 10 20
>> 3: 25 20 20 10
>> # cat /sys/devices/system/node/node0/demotion_nodes
>> 2
>
> node 2 is only the preferred demotion target. In fact, memory in node 0
> can be demoted to node 2,3. Please check demote_folio_list() for
> details.
Have I missed something, at least the on master tree, nd->preferred only include the
nearest ones(by specific algorithms), so in above numa topology, nd->preferred of
node0 is node2 only. node0 distance to node3 is 25 greater than to node2(20).
> 1657 int target_nid = next_demotion_node(pgdat->node_id);
So target_nid cannot be node3 IIUC.
(I cooked this patches weeks ago, maybe something has changed, i will also take a deep look later.)
1650 /*
1651 * Take folios on @demote_folios and attempt to demote them to another node.
1652 * Folios which are not demoted are left on @demote_folios.
1653 */
1654 static unsigned int demote_folio_list(struct list_head *demote_folios,
1655 struct pglist_data *pgdat)
1656 {
1657 int target_nid = next_demotion_node(pgdat->node_id);
1658 unsigned int nr_succeeded;
1659 nodemask_t allowed_mask;
1660
1661 struct migration_target_control mtc = {
1662 /*
1663 * Allocate from 'node', or fail quickly and quietly.
1664 * When this happens, 'page' will likely just be discarded
1665 * instead of migrated.
1666 */
1667 .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOWARN |
1668 __GFP_NOMEMALLOC | GFP_NOWAIT,
1669 .nid = target_nid,
1670 .nmask = &allowed_mask
1671 };
1672
1673 if (list_empty(demote_folios))
1674 return 0;
1675
1676 if (target_nid == NUMA_NO_NODE)
1677 return 0;
1678
1679 node_get_allowed_targets(pgdat, &allowed_mask);
1680
1681 /* Demotion ignores all cpuset and mempolicy settings */
1682 migrate_pages(demote_folios, alloc_demote_folio, NULL,
1683 (unsigned long)&mtc, MIGRATE_ASYNC, MR_DEMOTION,
1684 &nr_succeeded);
>
> --
> Best Regards,
> Huang, Ying
>
>> # cat /sys/devices/system/node/node1/demotion_nodes
>> 3
>> # cat /sys/devices/virtual/memory_tiering/memory_tier22/nodelist
>> 2-3
>>
>> Thanks
>> Zhijian
>>
>> (I hate the outlook native reply composition format.)
>> ________________________________________
>> From: Huang, Ying <ying.huang@xxxxxxxxx>
>> Sent: Thursday, November 2, 2023 11:17
>> To: Li, Zhijian/李 智坚
>> Cc: Andrew Morton; Greg Kroah-Hartman; rafael@xxxxxxxxxx; linux-mm@xxxxxxxxx; Gotou, Yasunori/五島 康文; linux-kernel@xxxxxxxxxxxxxxx
>> Subject: Re: [PATCH RFC 1/4] drivers/base/node: Add demotion_nodes sys infterface
>>
>> Li Zhijian <lizhijian@xxxxxxxxxxx> writes:
>>
>>> It shows the demotion target nodes of a node. Export this information to
>>> user directly.
>>>
>>> Below is an example where node0 node1 are DRAM, node3 is a PMEM node.
>>> - Before PMEM is online, no demotion_nodes for node0 and node1.
>>> $ cat /sys/devices/system/node/node0/demotion_nodes
>>> <show nothing>
>>> - After node3 is online as kmem
>>> $ daxctl reconfigure-device --mode=system-ram --no-online dax0.0 && daxctl online-memory dax0.0
>>> [
>>> {
>>> "chardev":"dax0.0",
>>> "size":1054867456,
>>> "target_node":3,
>>> "align":2097152,
>>> "mode":"system-ram",
>>> "online_memblocks":0,
>>> "total_memblocks":7
>>> }
>>> ]
>>> $ cat /sys/devices/system/node/node0/demotion_nodes
>>> 3
>>> $ cat /sys/devices/system/node/node1/demotion_nodes
>>> 3
>>> $ cat /sys/devices/system/node/node3/demotion_nodes
>>> <show nothing>
>>
>> We have /sys/devices/virtual/memory_tiering/memory_tier*/nodelist
>> already. A node in a higher tier can demote to any node in the lower
>> tiers. What's more need to be displayed in nodeX/demotion_nodes?
>>
>> --
>> Best Regards,
>> Huang, Ying
>>
>>> Signed-off-by: Li Zhijian <lizhijian@xxxxxxxxxxx>
>>> ---
>>> drivers/base/node.c | 13 +++++++++++++
>>> include/linux/memory-tiers.h | 6 ++++++
>>> mm/memory-tiers.c | 8 ++++++++
>>> 3 files changed, 27 insertions(+)
>>>
>>> diff --git a/drivers/base/node.c b/drivers/base/node.c
>>> index 493d533f8375..27e8502548a7 100644
>>> --- a/drivers/base/node.c
>>> +++ b/drivers/base/node.c
>>> @@ -7,6 +7,7 @@
>>> #include <linux/init.h>
>>> #include <linux/mm.h>
>>> #include <linux/memory.h>
>>> +#include <linux/memory-tiers.h>
>>> #include <linux/vmstat.h>
>>> #include <linux/notifier.h>
>>> #include <linux/node.h>
>>> @@ -569,11 +570,23 @@ static ssize_t node_read_distance(struct device *dev,
>>> }
>>> static DEVICE_ATTR(distance, 0444, node_read_distance, NULL);
>>>
>>> +static ssize_t demotion_nodes_show(struct device *dev,
>>> + struct device_attribute *attr, char *buf)
>>> +{
>>> + int ret;
>>> + nodemask_t nmask = next_demotion_nodes(dev->id);
>>> +
>>> + ret = sysfs_emit(buf, "%*pbl\n", nodemask_pr_args(&nmask));
>>> + return ret;
>>> +}
>>> +static DEVICE_ATTR_RO(demotion_nodes);
>>> +
>>> static struct attribute *node_dev_attrs[] = {
>>> &dev_attr_meminfo.attr,
>>> &dev_attr_numastat.attr,
>>> &dev_attr_distance.attr,
>>> &dev_attr_vmstat.attr,
>>> + &dev_attr_demotion_nodes.attr,
>>> NULL
>>> };
>>>
>>> diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
>>> index 437441cdf78f..8eb04923f965 100644
>>> --- a/include/linux/memory-tiers.h
>>> +++ b/include/linux/memory-tiers.h
>>> @@ -38,6 +38,7 @@ void init_node_memory_type(int node, struct memory_dev_type *default_type);
>>> void clear_node_memory_type(int node, struct memory_dev_type *memtype);
>>> #ifdef CONFIG_MIGRATION
>>> int next_demotion_node(int node);
>>> +nodemask_t next_demotion_nodes(int node);
>>> void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets);
>>> bool node_is_toptier(int node);
>>> #else
>>> @@ -46,6 +47,11 @@ static inline int next_demotion_node(int node)
>>> return NUMA_NO_NODE;
>>> }
>>>
>>> +static inline next_demotion_nodes next_demotion_nodes(int node)
>>> +{
>>> + return NODE_MASK_NONE;
>>> +}
>>> +
>>> static inline void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
>>> {
>>> *targets = NODE_MASK_NONE;
>>> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
>>> index 37a4f59d9585..90047f37d98a 100644
>>> --- a/mm/memory-tiers.c
>>> +++ b/mm/memory-tiers.c
>>> @@ -282,6 +282,14 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
>>> rcu_read_unlock();
>>> }
>>>
>>> +nodemask_t next_demotion_nodes(int node)
>>> +{
>>> + if (!node_demotion)
>>> + return NODE_MASK_NONE;
>>> +
>>> + return node_demotion[node].preferred;
>>> +}
>>> +
>>> /**
>>> * next_demotion_node() - Get the next node in the demotion path
>>> * @node: The starting node to lookup the next node