Re: [PATCH v3 1/2] mm/page_alloc: use ac->high_zoneidx for classzone_idx

From: Baoquan He
Date: Sun Mar 22 2020 - 23:59:05 EST


On 03/23/20 at 12:50pm, Joonsoo Kim wrote:
> Hello, Baoquan.
>
> 2020ë 3ì 20ì (ê) ìí 7:30, Baoquan He <bhe@xxxxxxxxxx>ëì ìì:
> >
> >
> > On 03/20/20 at 05:32pm, js1304@xxxxxxxxx wrote:
> > > From: Joonsoo Kim <iamjoonsoo.kim@xxxxxxx>
> > >
> > > Currently, we use the zone index of preferred_zone which represents
> > > the best matching zone for allocation, as classzone_idx. It has
> > > a problem on NUMA systems when the lowmem reserve protection exists
> > > for some zones on a node that do not exist on other nodes.
> > >
> > > In NUMA system, it can be possible that each node has different populated
> > > zones. For example, node 0 could have DMA/DMA32/NORMAL/MOVABLE zone and
> > > node 1 could have only NORMAL zone. In this setup, allocation request
> > > initiated on node 0 and the one on node 1 would have different
> > > classzone_idx, 3 and 2, respectively, since their preferred_zones are
> > > different. If the allocation is local, there is no problem. However,
> > > if it is handled by the remote node due to memory shortage, the problem
> > > would happen.
> >
> > Hi Joonsoo,
> >
> > Not sure if adding one sentence into above paragraph would be make it
> > easier to understand. Assume you are only talking about the high_zoneidx
> > is MOVABLE_ZONE with calculation of gfp_zone(gfp_mask), since any other
> > case doesn't have this problem. Please correct me if I am wrong.
>
> You're right. This example is for the allocation request with
> gfp_zone(gfp_mask),
> MOVABLE_ZONE.
>
> > In NUMA system, it can be possible that each node has different populated
> > zones. For example, node 0 could have DMA/DMA32/NORMAL/MOVABLE zone and
> > node 1 could have only NORMAL zone. In this setup, if we get high_zoneidx
> > as 3 (namely MOVABLE zone), with gfp_zone(gfp_mask), allocation request
> > initiated on node 0 and the one on node 1 would have different
> > classzone_idx, 3 and 2, respectively, since their preferred_zones are
> > different. If the allocation is local, there is no problem. However,
> > if it is handled by the remote node due to memory shortage, the problem
> > would happen.
>
> I'm okay with your change but I try again to be better. Please check the
> following rewritten commit message and please let me know if it is better
> than before.

Yeah, this new one looks very detailed, I believe anyone interested can
get what's going on. Thanks for doing this.

>
> Thanks.
>
> ------------------------>8-------------------------------
> From 3842d99d54c02c25cc6f6077fd0e97acabc10917 Mon Sep 17 00:00:00 2001
> From: Joonsoo Kim <iamjoonsoo.kim@xxxxxxx>
> Date: Fri, 4 May 2018 09:55:21 +0900
> Subject: [PATCH] mm/page_alloc: use ac->high_zoneidx for classzone_idx
>
> Currently, we use classzone_idx to calculate lowmem reserve proetection
> for an allocation request. This classzone_idx causes a problem
> on NUMA systems when the lowmem reserve protection exists for some zones
> on a node that do not exist on other nodes.
>
> Before further explanation, I should first clarify how to compute
> the classzone_idx and the high_zoneidx.
>
> - ac->high_zoneidx is computed via the arcane gfp_zone(gfp_mask) and
> represents the index of the highest zone the allocation can use
> - classzone_idx was supposed to be the index of the highest zone on
> the local node that the allocation can use, that is actually available
> in the system
>
> Think about following example. Node 0 has 4 populated zone,
> DMA/DMA32/NORMAL/MOVABLE. Node 1 has 1 populated zone, NORMAL. Some zones,
> such as MOVABLE, doesn't exist on node 1 and this makes following
> difference.
>
> Assume that there is an allocation request whose gfp_zone(gfp_mask) is
> the zone, MOVABLE. Then, it's high_zoneidx is 3. If this allocation is
> initiated on node 0, it's classzone_idx is 3 since actually
> available/usable zone on local (node 0) is MOVABLE. If this allocation
> is initiated on node 1, it's classzone_idx is 2 since actually
> available/usable zone on local (node 1) is NORMAL.
>
> You can see that classzone_idx of the allocation request are different
> according to their starting node, even if their high_zoneidx is the same.
>
> Think more about these two allocation requests. If they are processed
> on local, there is no problem. However, if allocation is initiated
> on node 1 are processed on remote, in this example, at the NORMAL zone
> on node 0, due to memory shortage, problem occurs. Their different
> classzone_idx leads to different lowmem reserve and then different
> min watermark. See the following example.
>
> root@ubuntu:/sys/devices/system/memory# cat /proc/zoneinfo
> Node 0, zone DMA
> per-node stats
> ...
> pages free 3965
> min 5
> low 8
> high 11
> spanned 4095
> present 3998
> managed 3977
> protection: (0, 2961, 4928, 5440)
> ...
> Node 0, zone DMA32
> pages free 757955
> min 1129
> low 1887
> high 2645
> spanned 1044480
> present 782303
> managed 758116
> protection: (0, 0, 1967, 2479)
> ...
> Node 0, zone Normal
> pages free 459806
> min 750
> low 1253
> high 1756
> spanned 524288
> present 524288
> managed 503620
> protection: (0, 0, 0, 4096)
> ...
> Node 0, zone Movable
> pages free 130759
> min 195
> low 326
> high 457
> spanned 1966079
> present 131072
> managed 131072
> protection: (0, 0, 0, 0)
> ...
> Node 1, zone DMA
> pages free 0
> min 0
> low 0
> high 0
> spanned 0
> present 0
> managed 0
> protection: (0, 0, 1006, 1006)
> Node 1, zone DMA32
> pages free 0
> min 0
> low 0
> high 0
> spanned 0
> present 0
> managed 0
> protection: (0, 0, 1006, 1006)
> Node 1, zone Normal
> per-node stats
> ...
> pages free 233277
> min 383
> low 640
> high 897
> spanned 262144
> present 262144
> managed 257744
> protection: (0, 0, 0, 0)
> ...
> Node 1, zone Movable
> pages free 0
> min 0
> low 0
> high 0
> spanned 262144
> present 0
> managed 0
> protection: (0, 0, 0, 0)
>
> - static min watermark for the NORMAL zone on node 0 is 750.
> - lowmem reserve for the request with classzone idx 3 at the NORMAL
> on node 0 is 4096.
> - lowmem reserve for the request with classzone idx 2 at the NORMAL
> on node 0 is 0.
>
> So, overall min watermark is:
> allocation initiated on node 0 (classzone_idx 3): 750 + 4096 = 4846
> allocation initiated on node 1 (classzone_idx 2): 750 + 0 = 750
>
> allocation initiated on node 1 will have some precedence than allocation
> initiated on node 0 because min watermark of the former allocation is
> lower than the other. So, allocation initiated on node 1 could succeed
> on node 0 when allocation initiated on node 0 could not, and, this could
> cause too many numa_miss allocation. Then, performance could be
> downgraded.
>
> Recently, there was a regression report about this problem on CMA patches
> since CMA memory are placed in ZONE_MOVABLE by those patches. I checked
> that problem is disappeared with this fix that uses high_zoneidx
> for classzone_idx.
>
> http://lkml.kernel.org/r/20180102063528.GG30397@yexl-desktop
>
> Using high_zoneidx for classzone_idx is more consistent way than previous
> approach because system's memory layout doesn't affect anything to it.
> With this patch, both classzone_idx on above example will be 3 so will
> have the same min watermark.
>
> allocation initiated on node 0: 750 + 4096 = 4846
> allocation initiated on node 1: 750 + 4096 = 4846
>
> One could wonder if there is a side effect that allocation initiated on
> node 1 will use higher bar when allocation is handled on local since
> classzone_idx could be higher than before. It will not happen because
> the zone without managed page doesn't contributes lowmem_reserve at all.
>
> Reported-by: Ye Xiaolong <xiaolong.ye@xxxxxxxxx>
> Tested-by: Ye Xiaolong <xiaolong.ye@xxxxxxxxx>
> Acked-by: Vlastimil Babka <vbabka@xxxxxxx>
> Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@xxxxxxx>
> ---
> mm/internal.h | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/mm/internal.h b/mm/internal.h
> index c39c895..aebaa33 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -119,7 +119,7 @@ struct alloc_context {
> bool spread_dirty_pages;
> };
>
> -#define ac_classzone_idx(ac) zonelist_zone_idx(ac->preferred_zoneref)
> +#define ac_classzone_idx(ac) (ac->high_zoneidx)
>
> /*
> * Locate the struct page for both the matching buddy in our
> --
> 2.7.4
>