Re: [EXT] Re: [RFC PATCH v2 0/2] Node migration between memory tiers

[Huang, Ying]

Srinivasulu Thanneeru <sthanneeru@xxxxxxxxxx> writes:

> Micron Confidential
>
>
>
> Micron Confidential
>> -----Original Message-----
>> From: Huang, Ying <ying.huang@xxxxxxxxx>
>> Sent: Wednesday, January 3, 2024 11:38 AM
>> To: Srinivasulu Thanneeru <sthanneeru@xxxxxxxxxx>
>> Cc: gregory.price <gregory.price@xxxxxxxxxxxx>; Srinivasulu Opensrc
>> <sthanneeru.opensrc@xxxxxxxxxx>; linux-cxl@xxxxxxxxxxxxxxx; linux-
>> mm@xxxxxxxxx; aneesh.kumar@xxxxxxxxxxxxx; dan.j.williams@xxxxxxxxx;
>> mhocko@xxxxxxxx; tj@xxxxxxxxxx; john@xxxxxxxxxxxxxx; Eishan Mirakhur
>> <emirakhur@xxxxxxxxxx>; Vinicius Tavares Petrucci
>> <vtavarespetr@xxxxxxxxxx>; Ravis OpenSrc <Ravis.OpenSrc@xxxxxxxxxx>;
>> Jonathan.Cameron@xxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; Johannes
>> Weiner <hannes@xxxxxxxxxxx>; Wei Xu <weixugc@xxxxxxxxxx>
>> Subject: Re: [EXT] Re: [RFC PATCH v2 0/2] Node migration between memory
>> tiers
>>
>> CAUTION: EXTERNAL EMAIL. Do not click links or open attachments unless
>> you recognize the sender and were expecting this message.
>>
>>
>> Srinivasulu Thanneeru <sthanneeru@xxxxxxxxxx> writes:
>>
>> > Micron Confidential
>> >
>> > Hi Huang, Ying,
>> >
>> > My apologies for wrong mail reply format, my mail client settings got
>> changed on my PC.
>> > Please find comments bellow inline.
>> >
>> > Regards,
>> > Srini
>> >
>> >
>> > Micron Confidential
>> >> -----Original Message-----
>> >> From: Huang, Ying <ying.huang@xxxxxxxxx>
>> >> Sent: Monday, December 18, 2023 11:26 AM
>> >> To: gregory.price <gregory.price@xxxxxxxxxxxx>
>> >> Cc: Srinivasulu Opensrc <sthanneeru.opensrc@xxxxxxxxxx>; linux-
>> >> cxl@xxxxxxxxxxxxxxx; linux-mm@xxxxxxxxx; Srinivasulu Thanneeru
>> >> <sthanneeru@xxxxxxxxxx>; aneesh.kumar@xxxxxxxxxxxxx;
>> >> dan.j.williams@xxxxxxxxx; mhocko@xxxxxxxx; tj@xxxxxxxxxx;
>> >> john@xxxxxxxxxxxxxx; Eishan Mirakhur <emirakhur@xxxxxxxxxx>; Vinicius
>> >> Tavares Petrucci <vtavarespetr@xxxxxxxxxx>; Ravis OpenSrc
>> >> <Ravis.OpenSrc@xxxxxxxxxx>; Jonathan.Cameron@xxxxxxxxxx; linux-
>> >> kernel@xxxxxxxxxxxxxxx; Johannes Weiner <hannes@xxxxxxxxxxx>; Wei Xu
>> >> <weixugc@xxxxxxxxxx>
>> >> Subject: [EXT] Re: [RFC PATCH v2 0/2] Node migration between memory
>> tiers
>> >>
>> >> CAUTION: EXTERNAL EMAIL. Do not click links or open attachments unless
>> >> you recognize the sender and were expecting this message.
>> >>
>> >>
>> >> Gregory Price <gregory.price@xxxxxxxxxxxx> writes:
>> >>
>> >> > On Fri, Dec 15, 2023 at 01:02:59PM +0800, Huang, Ying wrote:
>> >> >> <sthanneeru.opensrc@xxxxxxxxxx> writes:
>> >> >>
>> >> >> > =============
>> >> >> > Version Notes:
>> >> >> >
>> >> >> > V2 : Changed interface to memtier_override from adistance_offset.
>> >> >> > memtier_override was recommended by
>> >> >> > 1. John Groves <john@xxxxxxxxxxxxxx>
>> >> >> > 2. Ravi Shankar <ravis.opensrc@xxxxxxxxxx>
>> >> >> > 3. Brice Goglin <Brice.Goglin@xxxxxxxx>
>> >> >>
>> >> >> It appears that you ignored my comments for V1 as follows ...
>> >> >>
>> >> >>
>> >>
>> https://lore.k/
>> %2F&data=05%7C02%7Csthanneeru%40micron.com%7C3e5d38eb47be463c2
>> 95c08dc0c229d22%7Cf38a5ecd28134862b11bac1d563c806f%7C0%7C0%7C63
>> 8398590664228240%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
>> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C
>> &sdata=7fPxb1YYR2tZ0v2FB1vlXnMJFcI%2Fr9HT2%2BUD1MNUd%2FI%3D&re
>> served=0
>> >> ernel.org%2Flkml%2F87o7f62vur.fsf%40yhuang6-
>> >>
>> desk2.ccr.corp.intel.com%2F&data=05%7C02%7Csthanneeru%40micron.com
>> >>
>> %7C5e614e5f028342b6b59c08dbff8e3e37%7Cf38a5ecd28134862b11bac1d56
>> >>
>> 3c806f%7C0%7C0%7C638384758666895965%7CUnknown%7CTWFpbGZsb3d
>> >>
>> 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
>> >>
>> D%7C3000%7C%7C%7C&sdata=OpMkYCar%2Fv8uHb7AvXbmaNltnXeTvcNUTi
>> >> bLhwV12Fg%3D&reserved=0
>> >
>> > Thank you, Huang, Ying for pointing to this.
>> >
>> https://lpc.ev/
>> ents%2Fevent%2F16%2Fcontributions%2F1209%2Fattachments%2F1042%2F1
>> 995%2FLive%2520In%2520a%2520World%2520With%2520Multiple%2520Me
>> mory%2520Types.pdf&data=05%7C02%7Csthanneeru%40micron.com%7C3e
>> 5d38eb47be463c295c08dc0c229d22%7Cf38a5ecd28134862b11bac1d563c806
>> f%7C0%7C0%7C638398590664228240%7CUnknown%7CTWFpbGZsb3d8eyJW
>> IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
>> 000%7C%7C%7C&sdata=1fGraxff7%2F1hNaE0an0xEudSKSUvaF3HgClMkmdC7
>> n8%3D&reserved=0
>> >
>> > In the presentation above, the adistance_offsets are per memtype.
>> > We believe that adistance_offset per node is more suitable and flexible.
>> > since we can change it per node. If we keep adistance_offset per memtype,
>> > then we cannot change it for a specific node of a given memtype.
>> >
>> >> >>
>> >>
>> https://lore.k/
>> %2F&data=05%7C02%7Csthanneeru%40micron.com%7C3e5d38eb47be463c2
>> 95c08dc0c229d22%7Cf38a5ecd28134862b11bac1d563c806f%7C0%7C0%7C63
>> 8398590664228240%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
>> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C
>> &sdata=7fPxb1YYR2tZ0v2FB1vlXnMJFcI%2Fr9HT2%2BUD1MNUd%2FI%3D&re
>> served=0
>> >> ernel.org%2Flkml%2F87jzpt2ft5.fsf%40yhuang6-
>> >>
>> desk2.ccr.corp.intel.com%2F&data=05%7C02%7Csthanneeru%40micron.com
>> >>
>> %7C5e614e5f028342b6b59c08dbff8e3e37%7Cf38a5ecd28134862b11bac1d56
>> >>
>> 3c806f%7C0%7C0%7C638384758666895965%7CUnknown%7CTWFpbGZsb3d
>> >>
>> 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
>> >>
>> D%7C3000%7C%7C%7C&sdata=O0%2B6T%2FgU0TicCEYBac%2FAyjOLwAeouh
>> >> D%2BcMI%2BflOsI1M%3D&reserved=0
>> >
>> > Yes, memory_type would be grouping the related memories together as
>> single tier.
>> > We should also have a flexibility to move nodes between tiers, to address
>> the issues.
>> > described in use cases above.
>>
>> We don't pursue absolute flexibility.  We add necessary flexibility
>> only.  Why do you need this kind of flexibility?  Can you provide some
>> use cases where memory_type based "adistance_offset" doesn't work?
>
> - /sys/devices/virtual/memory_type/memory_type/ adistance_offset
> memory_type based "adistance_offset will provide a way to move all nodes of same memory_type (e.g. all cxl nodes)
> to different tier.

We will not put the CXL nodes with different performance metrics in one
memory_type.  If so, do you still need to move one of them?

> Whereas /sys/devices/system/node/node2/memtier_override provide a way migrate a node from one tier to another.
> Considering a case where we would like to move two cxl nodes into two different tiers in future.
> So, I thought it would be good to have flexibility at node level instead of at memory_type.

--
Best Regards,
Huang, Ying