Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified image filesystem
From: Gao Xiang
Date: Thu Feb 02 2023 - 02:37:35 EST
On 2023/2/2 15:17, Gao Xiang wrote:
On 2023/2/2 14:37, Amir Goldstein wrote:
On Wed, Feb 1, 2023 at 1:22 PM Gao Xiang <hsiangkao@xxxxxxxxxxxxxxxxx> wrote:
On 2023/2/1 18:01, Gao Xiang wrote:
On 2023/2/1 17:46, Alexander Larsson wrote:
...
| uncached(ms)| cached(ms)
----------------------------------|-------------|-----------
composefs (with digest) | 326 | 135
erofs (w/o -T0) | 264 | 172
erofs (w/o -T0) + overlayfs | 651 | 238
squashfs (compressed) | 538 | 211
squashfs (compressed) + overlayfs | 968 | 302
Clearly erofs with sparse files is the best fs now for the ro-fs +
overlay case. But still, we can see that the additional cost of the
overlayfs layer is not negligible.
According to amir this could be helped by a special composefs-like mode
in overlayfs, but its unclear what performance that would reach, and
we're then talking net new development that further complicates the
overlayfs codebase. Its not clear to me which alternative is easier to
develop/maintain.
Also, the difference between cached and uncached here is less than in
my tests. Probably because my test image was larger. With the test
image I use, the results are:
| uncached(ms)| cached(ms)
----------------------------------|-------------|-----------
composefs (with digest) | 681 | 390
erofs (w/o -T0) + overlayfs | 1788 | 532
squashfs (compressed) + overlayfs | 2547 | 443
I gotta say it is weird though that squashfs performed better than
erofs in the cached case. May be worth looking into. The test data I'm
using is available here:
As another wild guess, cached performance is a just vfs-stuff.
I think the performance difference may be due to ACL (since both
composefs and squashfs don't support ACL). I already asked Jingbo
to get more "perf data" to analyze this but he's now busy in another
stuff.
Again, my overall point is quite simple as always, currently
composefs is a read-only filesystem with massive symlink-like files.
It behaves as a subset of all generic read-only filesystems just
for this specific use cases.
In facts there are many options to improve this (much like Amir
said before):
1) improve overlayfs, and then it can be used with any local fs;
2) enhance erofs to support this (even without on-disk change);
3) introduce fs/composefs;
In addition to option 1), option 2) has many benefits as well, since
your manifest files can save real regular files in addition to composefs
model.
(add some words..)
My first response at that time (on Slack) was "kindly request
Giuseppe to ask in the fsdevel mailing list if this new overlay model
and use cases is feasable", if so, I'm much happy to integrate in to
EROFS (in a cooperative way) in several ways:
- just use EROFS symlink layout and open such file in a stacked way;
or (now)
- just identify overlayfs "trusted.overlay.redirect" in EROFS itself
and open file so such image can be both used for EROFS only and
EROFS + overlayfs.
If that happened, then I think the overlayfs "metacopy" option can
also be shown by other fs community people later (since I'm not an
overlay expert), but I'm not sure why they becomes impossible finally
and even not mentioned at all.
Or if you guys really don't want to use EROFS for whatever reasons
(EROFS is completely open-source, used, contributed by many vendors),
you could improve squashfs, ext4, or other exist local fses with this
new use cases (since they don't need any on-disk change as well, for
example, by using some xattr), I don't think it's really hard.
Engineering-wise, merging composefs features into EROFS
would be the simplest option and FWIW, my personal preference.
However, you need to be aware that this will bring into EROFS
vfs considerations, such as s_stack_depth nesting (which AFAICS
is not see incremented composefs?). It's not the end of the world, but this
is no longer plain fs over block game. There's a whole new class of bugs
(that syzbot is very eager to explore) so you need to ask yourself whether
this is a direction you want to lead EROFS towards.
I'd like to make a seperated Kconfig for this. I consider this just because
currently composefs is much similar to EROFS but it doesn't have some ability
to keep real regular file (even some README, VERSION or Changelog in these
images) in its (composefs-called) manifest files. Even its on-disk super block
doesn't have a UUID now [1] and some boot sector for booting or some potential
hybird formats such as tar + EROFS, cpio + EROFS.
I'm not sure if those potential new on-disk features is unneeded even for
future composefs. But if composefs laterly supports such on-disk features,
that makes composefs closer to EROFS even more. I don't see disadvantage to
make these actual on-disk compatible (like ext2 and ext4).
The only difference now is manifest file itself I/O interface -- bio vs file.
but EROFS can be distributed to raw block devices as well, composefs can't.
Also, I'd like to seperate core-EROFS from advanced features (or people who
are interested to work on this are always welcome) and composefs-like model,
if people don't tend to use any EROFS advanced features, it could be disabled
from compiling explicitly.
Apart from that, I still fail to get some thoughts (apart from unprivileged
mounts) how EROFS + overlayfs combination fails on automative real workloads
aside from "ls -lR" (readdir + stat).
And eventually we still need overlayfs for most use cases to do writable
stuffs, anyway, it needs some words to describe why such < 1s difference is
very very important to the real workload as you already mentioned before.
And with overlayfs lazy lookup, I think it can be close to ~100ms or better.
Giuseppe expressed his plans to make use of the composefs method
inside userns one day. It is not a hard dependency, but I believe that
keeping the "RO efficient verifiable image format" functionality (EROFS)
separate from "userns composition of verifiable images" (overlayfs)
may benefit the userns mount goal in the long term.
If that is needed, I'm very happy to get more detailed path of this from
some discussion in LSF/MM/BPF 2023: how we get this (userns) reliably in
practice.
As of code lines, core EROFS on-disk format is quite simple (I don't think
total LOC is a barrier), if you see
fs/erofs/data.c
fs/erofs/namei.c
fs/erofs/dir.c
or
erofs_super_block
erofs_inode_compact
erofs_inode_extended
erofs_dirent
but for example, fs/erofs/super.c which is just used to enable EROFS advanced
features is almost 1000LOC now. But most code is quite trivial, I don't think
these can cause any difference to userns plan.
Thanks,
Gao Xiang
[1] https://lore.kernel.org/r/CAOQ4uxjm7i+uO4o4470ACctsft1m18EiUpxBfCeT-Wyqf1FAYg@xxxxxxxxxxxxxx/
Thanks,
Amir.