Re: [RFC PATCH V1 00/11] riscv: Introduce 64K base page

[Arnd Bergmann]

On Thu, Nov 23, 2023, at 07:56, Xu Lu wrote:
> Some existing architectures like ARM supports base page larger than 4K
> as their MMU supports more page sizes. Thus, besides hugetlb page and
> transparent huge page, there is another way for these architectures to
> enjoy the benefits of fewer TLB misses without worrying about cost of
> splitting and merging huge pages. However, on architectures with only
> 4K MMU, larger base page is unavailable now.
>
> This patch series attempts to break through the limitation of MMU and
> supports larger base page on RISC-V, which only supports 4K page size
> now.
>
> The key idea to implement larger base page based on 4K MMU is to
> decouple the MMU page from the base page in view of kernel mm, which we
> denote as software page. In contrary to software page, we denote the MMU
> page as hardware page. Below is the difference between these two kinds
> of pages.

We have played with this on arm32, but the conclusion is that it's
almost never worth the memory overhead, as most workloads end up
using several times the amount of physical RAM after each small
file in the page cache and any sparse populated anonymous memory
area explodes to up to 16 times the size.

On ppc64, using 64KB pages was way to get around limitations in
their hashed MMU design, which had a much bigger performance impact
because any page table access ends up being a cache miss. On arm64,
there are some CPUs like the Fujitsu A64FX that are really bad at
4KB pages and don't support 16KB pages, so this is the only real
option.

You will see a notable performance benefit in synthetic benchmarks
like speccpu with 64KB pages, or on specific computational
workloads that have large densely packed memory chunks, but for
real workloads, the usual answer is to just use transparent
hugepages for larger mappings and a page size of no more than
16KB for the page cache.

With the work going into using folios in the kernel (see e.g.
https://lwn.net/Articles/932386/), even the workloads that
benefit from 64KB base pages should be better off with 4KB
pages and just using the TLB hints for large folios.

     Arnd