RE: [PATCH RFC 3/4] mm/vmstat: rename pgdemote_* to pgdemote_dst_* and add pgdemote_src_*

[Yasunori Gotou (Fujitsu)]

> >> > Hello,
> >> >
> >> >> On 02/11/2023 13:45, Huang, Ying wrote:
> >> >> > Li Zhijian <lizhijian@xxxxxxxxxxx> writes:
> >> >> >
> >> >> >> pgdemote_src_*: pages demoted from this node.
> >> >> >> pgdemote_dst_*: pages demoted to this node.
> >> >> >>
> >> >> >> So that we are able to know their demotion per-node stats by
> >> >> >> checking
> >> this.
> >> >> >>
> >> >> >> In the environment, node0 and node1 are DRAM, node3 is PMEM.
> >> >> >>
> >> >> >> Global stats:
> >> >> >> $ grep -E 'demote' /proc/vmstat pgdemote_src_kswapd 130155
> >> >> >> pgdemote_src_direct 113497 pgdemote_src_khugepaged 0
> >> >> >> pgdemote_dst_kswapd 130155 pgdemote_dst_direct 113497
> >> >> >> pgdemote_dst_khugepaged 0
> >> >> >>
> >> >> >> Per-node stats:
> >> >> >> $ grep demote /sys/devices/system/node/node0/vmstat
> >> >> >> pgdemote_src_kswapd 68454
> >> >> >> pgdemote_src_direct 83431
> >> >> >> pgdemote_src_khugepaged 0
> >> >> >> pgdemote_dst_kswapd 0
> >> >> >> pgdemote_dst_direct 0
> >> >> >> pgdemote_dst_khugepaged 0
> >> >> >>
> >> >> >> $ grep demote /sys/devices/system/node/node1/vmstat
> >> >> >> pgdemote_src_kswapd 185834
> >> >> >> pgdemote_src_direct 30066
> >> >> >> pgdemote_src_khugepaged 0
> >> >> >> pgdemote_dst_kswapd 0
> >> >> >> pgdemote_dst_direct 0
> >> >> >> pgdemote_dst_khugepaged 0
> >> >> >>
> >> >> >> $ grep demote /sys/devices/system/node/node3/vmstat
> >> >> >> pgdemote_src_kswapd 0
> >> >> >> pgdemote_src_direct 0
> >> >> >> pgdemote_src_khugepaged 0
> >> >> >> pgdemote_dst_kswapd 254288
> >> >> >> pgdemote_dst_direct 113497
> >> >> >> pgdemote_dst_khugepaged 0
> >> >> >>
> >> >> >>  From above stats, we know node3 is the demotion destination
> >> >> >> which one the node0 and node1 will demote to.
> >> >> >
> >> >> > Why do we need these information?  Do you have some use case?
> >> >>
> >> >> I recall our customers have mentioned that they want to know how
> >> >> much the memory is demoted to the CXL memory device in a specific
> period.
> >> >
> >> > I'll mention about it more.
> >> >
> >> > I had a conversation with one of our customers. He expressed a
> >> > desire for more detailed profile information to analyze the
> >> > behavior of demotion (and promotion) when his workloads are executed.
> >> > If the results are not satisfactory for his workloads, he wants to
> >> > tune his servers for his workloads with these profiles.
> >> > Additionally, depending on the results, he may want to change his
> >> > server
> >> configuration.
> >> > For example, he may want to buy more expensive DDR memories rather
> >> > than
> >> cheaper CXL memory.
> >> >
> >> > In my impression, our customers seems to think that CXL memory is
> >> > NOT as
> >> reliable as DDR memory yet.
> >> > Therefore, they want to prepare for the new world that CXL will
> >> > bring, and want to have a method for the preparation by profiling
> >> > information as
> >> much as possible.
> >> >
> >> > it this enough for your question?
> >>
> >> I want some more detailed information about how these stats are used?
> >> Why isn't per-node pgdemote_xxx counter enough?
> >
> > I rechecked the customer's original request.
> >
> > - If a memory area is demoted to a CXL memory node, he wanted to
> > analyze how it affects performance  of their workload, such as
> > latency. He wanted to use CXL Node memory usage as basic  information for
> the analysis.
> >
> > - If he notices that demotion occurs well on a server and CXL memories are
> used 85% constantly, he
> >   may want to add DDR DRAM or select some other ways to avoid demotion.
> >   (His image is likely Swap free/used.)
> >   IIRC, demotion target is not spread to all of the CXL memory node, right?
> >   Then, he needs to know how CXL memory is occupied by demoted
> memory.
> >
> > If I misunderstand something, or you have any better idea, please let
> > us know. I'll talk with him again. (It will be next week...)
> 
> 
> To check CXL memory usage, /proc/PID/numa_maps,
> /sys/fs/cgroup/CGROUP/memory.numa_stat, and
> /sys/devices/system/node/nodeN/meminfo can be used for process, cgroup,
> and NUMA node respectively.  Is this enough?

Thank you for your idea
We will investigate your idea and talk with our customer.
Please wait.

Thanks,
---
Yasunori Goto


> 
> --
> Best Regards,
> Huang, Ying
> 
> >> >
> >> >>
> >> >>
> >> >> >>>   	mod_node_page_state(NODE_DATA(target_nid),
> >> >> >>> -		    PGDEMOTE_KSWAPD + reclaimer_offset(),
> >> >> nr_succeeded);
> >> >> >>> +		    PGDEMOTE_DST_KSWAPD + reclaimer_offset(),
> >> >> nr_succeeded);
> >> >>
> >> >> But if the *target_nid* is only indicate the preferred node, this
> >> >> accounting maybe not accurate.
> >> >>
> 
> [snip]