RE: [EXT] Re: [RFC PATCH v2 0/2] Node migration between memory tiers

[Srinivasulu Thanneeru]

> -----Original Message-----
> From: Huang, Ying <ying.huang@xxxxxxxxx>
> Sent: Wednesday, January 3, 2024 2:00 PM
> To: Srinivasulu Thanneeru <sthanneeru@xxxxxxxxxx>
> Cc: gregory.price <gregory.price@xxxxxxxxxxxx>; Srinivasulu Opensrc
> <sthanneeru.opensrc@xxxxxxxxxx>; linux-cxl@xxxxxxxxxxxxxxx; linux-
> mm@xxxxxxxxx; aneesh.kumar@xxxxxxxxxxxxx; dan.j.williams@xxxxxxxxx;
> mhocko@xxxxxxxx; tj@xxxxxxxxxx; john@xxxxxxxxxxxxxx; Eishan Mirakhur
> <emirakhur@xxxxxxxxxx>; Vinicius Tavares Petrucci
> <vtavarespetr@xxxxxxxxxx>; Ravis OpenSrc <Ravis.OpenSrc@xxxxxxxxxx>;
> Jonathan.Cameron@xxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; Johannes
> Weiner <hannes@xxxxxxxxxxx>; Wei Xu <weixugc@xxxxxxxxxx>
> Subject: Re: [EXT] Re: [RFC PATCH v2 0/2] Node migration between memory
> tiers
>
> CAUTION: EXTERNAL EMAIL. Do not click links or open attachments unless
> you recognize the sender and were expecting this message.
>
>
> Srinivasulu Thanneeru <sthanneeru@xxxxxxxxxx> writes:
>
> > Micron Confidential
> >
> >
> >
> > Micron Confidential
> >> -----Original Message-----
> >> From: Huang, Ying <ying.huang@xxxxxxxxx>
> >> Sent: Wednesday, January 3, 2024 11:38 AM
> >> To: Srinivasulu Thanneeru <sthanneeru@xxxxxxxxxx>
> >> Cc: gregory.price <gregory.price@xxxxxxxxxxxx>; Srinivasulu Opensrc
> >> <sthanneeru.opensrc@xxxxxxxxxx>; linux-cxl@xxxxxxxxxxxxxxx; linux-
> >> mm@xxxxxxxxx; aneesh.kumar@xxxxxxxxxxxxx;
> dan.j.williams@xxxxxxxxx;
> >> mhocko@xxxxxxxx; tj@xxxxxxxxxx; john@xxxxxxxxxxxxxx; Eishan Mirakhur
> >> <emirakhur@xxxxxxxxxx>; Vinicius Tavares Petrucci
> >> <vtavarespetr@xxxxxxxxxx>; Ravis OpenSrc
> <Ravis.OpenSrc@xxxxxxxxxx>;
> >> Jonathan.Cameron@xxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; Johannes
> >> Weiner <hannes@xxxxxxxxxxx>; Wei Xu <weixugc@xxxxxxxxxx>
> >> Subject: Re: [EXT] Re: [RFC PATCH v2 0/2] Node migration between
> memory
> >> tiers
> >>
> >> CAUTION: EXTERNAL EMAIL. Do not click links or open attachments unless
> >> you recognize the sender and were expecting this message.
> >>
> >>
> >> Srinivasulu Thanneeru <sthanneeru@xxxxxxxxxx> writes:
> >>
> >> > Micron Confidential
> >> >
> >> > Hi Huang, Ying,
> >> >
> >> > My apologies for wrong mail reply format, my mail client settings got
> >> changed on my PC.
> >> > Please find comments bellow inline.
> >> >
> >> > Regards,
> >> > Srini
> >> >
> >> >
> >> > Micron Confidential
> >> >> -----Original Message-----
> >> >> From: Huang, Ying <ying.huang@xxxxxxxxx>
> >> >> Sent: Monday, December 18, 2023 11:26 AM
> >> >> To: gregory.price <gregory.price@xxxxxxxxxxxx>
> >> >> Cc: Srinivasulu Opensrc <sthanneeru.opensrc@xxxxxxxxxx>; linux-
> >> >> cxl@xxxxxxxxxxxxxxx; linux-mm@xxxxxxxxx; Srinivasulu Thanneeru
> >> >> <sthanneeru@xxxxxxxxxx>; aneesh.kumar@xxxxxxxxxxxxx;
> >> >> dan.j.williams@xxxxxxxxx; mhocko@xxxxxxxx; tj@xxxxxxxxxx;
> >> >> john@xxxxxxxxxxxxxx; Eishan Mirakhur <emirakhur@xxxxxxxxxx>;
> Vinicius
> >> >> Tavares Petrucci <vtavarespetr@xxxxxxxxxx>; Ravis OpenSrc
> >> >> <Ravis.OpenSrc@xxxxxxxxxx>; Jonathan.Cameron@xxxxxxxxxx;
> linux-
> >> >> kernel@xxxxxxxxxxxxxxx; Johannes Weiner <hannes@xxxxxxxxxxx>; Wei
> Xu
> >> >> <weixugc@xxxxxxxxxx>
> >> >> Subject: [EXT] Re: [RFC PATCH v2 0/2] Node migration between memory
> >> tiers
> >> >>
> >> >> CAUTION: EXTERNAL EMAIL. Do not click links or open attachments
> unless
> >> >> you recognize the sender and were expecting this message.
> >> >>
> >> >>
> >> >> Gregory Price <gregory.price@xxxxxxxxxxxx> writes:
> >> >>
> >> >> > On Fri, Dec 15, 2023 at 01:02:59PM +0800, Huang, Ying wrote:
> >> >> >> <sthanneeru.opensrc@xxxxxxxxxx> writes:
> >> >> >>
> >> >> >> > =============
> >> >> >> > Version Notes:
> >> >> >> >
> >> >> >> > V2 : Changed interface to memtier_override from adistance_offset.
> >> >> >> > memtier_override was recommended by
> >> >> >> > 1. John Groves <john@xxxxxxxxxxxxxx>
> >> >> >> > 2. Ravi Shankar <ravis.opensrc@xxxxxxxxxx>
> >> >> >> > 3. Brice Goglin <Brice.Goglin@xxxxxxxx>
> >> >> >>
> >> >> >> It appears that you ignored my comments for V1 as follows ...
> >> >> >>
> >> >> >>
> >> >>
> >>
> https://lore.k/
> %2F&data=05%7C02%7Csthanneeru%40micron.com%7Ce9e04d25ea7540100
> cf308dc0c366eb1%7Cf38a5ecd28134862b11bac1d563c806f%7C0%7C0%7C63
> 8398675187014390%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C
> &sdata=k6J1wxcuHTwR9eoD9Yz137bkn6wt1L9zpf5YaOjoIqA%3D&reserved=0
> >>
> %2F&data=05%7C02%7Csthanneeru%40micron.com%7C3e5d38eb47be463c2
> >>
> 95c08dc0c229d22%7Cf38a5ecd28134862b11bac1d563c806f%7C0%7C0%7C63
> >>
> 8398590664228240%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
> >>
> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C
> >>
> &sdata=7fPxb1YYR2tZ0v2FB1vlXnMJFcI%2Fr9HT2%2BUD1MNUd%2FI%3D&re
> >> served=0
> >> >> ernel.org%2Flkml%2F87o7f62vur.fsf%40yhuang6-
> >> >>
> >>
> desk2.ccr.corp.intel.com%2F&data=05%7C02%7Csthanneeru%40micron.com
> >> >>
> >>
> %7C5e614e5f028342b6b59c08dbff8e3e37%7Cf38a5ecd28134862b11bac1d56
> >> >>
> >>
> 3c806f%7C0%7C0%7C638384758666895965%7CUnknown%7CTWFpbGZsb3d
> >> >>
> >>
> 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> >> >>
> >>
> D%7C3000%7C%7C%7C&sdata=OpMkYCar%2Fv8uHb7AvXbmaNltnXeTvcNUTi
> >> >> bLhwV12Fg%3D&reserved=0
> >> >
> >> > Thank you, Huang, Ying for pointing to this.
> >> >
> >>
> https://lpc.ev/
> %2F&data=05%7C02%7Csthanneeru%40micron.com%7Ce9e04d25ea7540100
> cf308dc0c366eb1%7Cf38a5ecd28134862b11bac1d563c806f%7C0%7C0%7C63
> 8398675187014390%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C
> &sdata=%2F0AW8RYpTIa7%2FiScnkzmmTeAE9TYqjsuWWjTuxBPptk%3D&rese
> rved=0
> >>
> ents%2Fevent%2F16%2Fcontributions%2F1209%2Fattachments%2F1042%2F1
> >>
> 995%2FLive%2520In%2520a%2520World%2520With%2520Multiple%2520Me
> >>
> mory%2520Types.pdf&data=05%7C02%7Csthanneeru%40micron.com%7C3e
> >>
> 5d38eb47be463c295c08dc0c229d22%7Cf38a5ecd28134862b11bac1d563c806
> >>
> f%7C0%7C0%7C638398590664228240%7CUnknown%7CTWFpbGZsb3d8eyJW
> >>
> IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
> >>
> 000%7C%7C%7C&sdata=1fGraxff7%2F1hNaE0an0xEudSKSUvaF3HgClMkmdC7
> >> n8%3D&reserved=0
> >> >
> >> > In the presentation above, the adistance_offsets are per memtype.
> >> > We believe that adistance_offset per node is more suitable and flexible.
> >> > since we can change it per node. If we keep adistance_offset per
> memtype,
> >> > then we cannot change it for a specific node of a given memtype.
> >> >
> >> >> >>
> >> >>
> >>
> https://lore.k/
> %2F&data=05%7C02%7Csthanneeru%40micron.com%7Ce9e04d25ea7540100
> cf308dc0c366eb1%7Cf38a5ecd28134862b11bac1d563c806f%7C0%7C0%7C63
> 8398675187014390%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C
> &sdata=k6J1wxcuHTwR9eoD9Yz137bkn6wt1L9zpf5YaOjoIqA%3D&reserved=0
> >>
> %2F&data=05%7C02%7Csthanneeru%40micron.com%7C3e5d38eb47be463c2
> >>
> 95c08dc0c229d22%7Cf38a5ecd28134862b11bac1d563c806f%7C0%7C0%7C63
> >>
> 8398590664228240%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
> >>
> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C
> >>
> &sdata=7fPxb1YYR2tZ0v2FB1vlXnMJFcI%2Fr9HT2%2BUD1MNUd%2FI%3D&re
> >> served=0
> >> >> ernel.org%2Flkml%2F87jzpt2ft5.fsf%40yhuang6-
> >> >>
> >>
> desk2.ccr.corp.intel.com%2F&data=05%7C02%7Csthanneeru%40micron.com
> >> >>
> >>
> %7C5e614e5f028342b6b59c08dbff8e3e37%7Cf38a5ecd28134862b11bac1d56
> >> >>
> >>
> 3c806f%7C0%7C0%7C638384758666895965%7CUnknown%7CTWFpbGZsb3d
> >> >>
> >>
> 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> >> >>
> >>
> D%7C3000%7C%7C%7C&sdata=O0%2B6T%2FgU0TicCEYBac%2FAyjOLwAeouh
> >> >> D%2BcMI%2BflOsI1M%3D&reserved=0
> >> >
> >> > Yes, memory_type would be grouping the related memories together as
> >> single tier.
> >> > We should also have a flexibility to move nodes between tiers, to
> address
> >> the issues.
> >> > described in use cases above.
> >>
> >> We don't pursue absolute flexibility.  We add necessary flexibility
> >> only.  Why do you need this kind of flexibility?  Can you provide some
> >> use cases where memory_type based "adistance_offset" doesn't work?
> >
> > - /sys/devices/virtual/memory_type/memory_type/ adistance_offset
> > memory_type based "adistance_offset will provide a way to move all nodes
> of same memory_type (e.g. all cxl nodes)
> > to different tier.
>
> We will not put the CXL nodes with different performance metrics in one
> memory_type.  If so, do you still need to move one of them?

>From  https://lpc.events/event/16/contributions/1209/attachments/1042/1995/Live%20In%20a%20World%20With%20Multiple%20Memory%20Types.pdf
abstract_distance_offset: override by users to deal with firmware issue.

say firmware can configure the cxl node into wrong tiers, similar to that it may also configure all cxl nodes into single memtype, hence all these nodes can fall into a single wrong tier.
In this case, per node adistance_offset would be good to have ?

--
Srini
> > Whereas /sys/devices/system/node/node2/memtier_override provide a
> way migrate a node from one tier to another.
> > Considering a case where we would like to move two cxl nodes into two
> different tiers in future.
> > So, I thought it would be good to have flexibility at node level instead of at
> memory_type.
>
> --
> Best Regards,
> Huang, Ying