ovl: ovl_fs::creator_cred::usage scalability issues

[Chen Hu]

*Problem*
ovl_permission() checks the underlying inode with the credential of mounter.
The cred, struct ovl_fs::creator_cred, is somewhat global per overlayfs
superblock. Performance degrades when concurrency increases on the cred, to be
specific, on ovl_fs::creator_cred::usage.

This happens when there are massive file access inside container, especially
on SoC with many cores. With Linux 6.6.0-rc2, we run a web workload container
on Intel 4th Xeon Sapphire Rapids which has 56 cores. Perf reports that 5.7%
(2.50% + 1.87% + 1.33%) CPU stall in overlayfs:
Self    Command       Shared Object            Symbol
2.50%   foo           [kernel.vmlinux]         [k] override_creds
1.87%   foo           [kernel.vmlinux]         [k] revert_creds
1.33%   foo           [kernel.vmlinux]         [k] generic_permission

On Soc with more than 100 cores, we can even observe ~30% CPU stalled!

This scalability issue is caused by two factors:
1) Contention on creator_cred::usage
   creator_cred::usage is atomic_t and is inc/dec atomically during every file
   access. So HW acquires the corresponding cache line exclusively. This
   operataiton is expensive and gets worse when contention is heavy.
   Call chain:
      ovl_permission()
      -> ovl_override_creds()
      -> override_creds()
      -> get_new_cred()
      -> atomic_inc(&cred->usage);

      ovl_permission()
      -> revert_creds()
      -> put_cred()
      -> atomic_dec_and_test(&(cred)->usage))

2) False sharing
   `perf c2c` shows false sharing issue between cred::usage and cred::fsuid.
   This is why generic_permission() stalls 1.33% CPU in above perf report.
   ovl_permission() updates cred::usage and it also reads cred::fsuid.
   Unfortunately, they locate in the same cache line and thus false sharing
   occurs. cred::fsuid is read at:
      ovl_permission()
      -> inode_permission()
      -> generic_permission()
      -> acl_permission_check()
      -> current_fsuid()

*Mitigations we tried*
We tried several mitigations but are not sure if it can be a fix or just
workaround / hack. So we report this and want to have some discussions.

Our mitigations aims to eliminate the contention on creator_cred->usage.
Without contention, the false sharing will be tiny and no need to handle. The
mitigations we tested are:
   1) Check underlying inode once in its lifetime. 
   OR
   2) In ovl_permission(), copy global creator_cred to a local variable to
      avoid concurrency.

With any mitigations above, CPU will not stall on overlayfs.

Paste mitigation 1 below.