Re: [LSF/MM/BPF TOPIC] no tears atomics & LBS

[IBM]

Luis Chamberlain <mcgrof@xxxxxxxxxx> writes:

> At last year's LSFMM we learned through Ted Ts'o about the interest by
> cloud providers in large atomics [0]. It is a good example where cloud
> providers innovated in an area perhaps before storage vendors were
> providing hardware support for such features. An example use case was
> databases. In short, with large atomics databases can disable their own version
> of journaling so to increase TPS. Large atomics lets you  disabling things like
> MySQL innodb_doublewrite. The feature to allow you to disable this and use
> large atomcis is known as torn write prevention [1]. At least for MySQL the
> default page size for the database (used for columns) is 16k, and so enabling
> for example a 16k atomic can allow you to take advantage of this. It was also
> mentioned how PostgreSQL only supports buffered-IO and so it would be desirable
> for a solution to support buffered-IO with large atomics as well. The way
> cloud providers enable torn write protection, is by using direct IO.
>
> John Garry has been working on adding an API for atomic writes, it would
> seem some folks refer to this as the no-tears atomic API. It consists of
> two parts, one for the block layer [2] and another set of changes for
> XFS [3]. It enables Direct IO support with large atomics.  It includes
> a userspace API which lets you peg a FS_XFLAG_ATOMICWRITES flag onto a
> file, and you then create an XFS filesystem using the XFS realtime
> subvolume with with an extent alignment. The current users of this API
> seems to be SCSI, but obviously this can grow to support others. A neat
> feature of this effort is you can have two separate directories with
> separate aligment requirements. There is no generic filesystem solution
> yet.
>
> Meanwhile we're now at a v2 RFC for LBS support [4]. Although the LBS
> effort originally was a completely orthogonal effort to large atomics, it
> would seem there is a direct relationship here now worth discussing.
> In short LBS enables buffered-IO large atomic support if the hardware
> support its.
> We get both alignment constraints gauranteed and now ensure
> we use contigous memory for the IOs for DMA too it is built on using large
> folios. We expect NVMe drives which support support large atomics can
> easily profit from this without any userspace modification other than
> when you create the filesystem.
>
> We reviewed the possible intersection of both efforts at our last LBS cabal
> with LBS interested folks and Martin Peterson and John Garry. It is somewhat
> unclear exactly how to follow up on some aspects of the no-tear API [5]
> but there was agreement about the possible intersection of both efforts,
> and that we should discuss this at LSFMM. The goal would be to try to reach
> consensus on how no-tear API and how LBS could help with those
> interested in leveraging large atomics.
>
> Some things to evaluate or for us to discuss:
>
>  * no-tear API:
>    - allows directories to have separate alignment requirements
>      - this might be useful for folks who want to use large IOs with
>        large atomics for some workloads but smaller IOs for another
>        directory on the same drive. It this a viable option to some
>        users for large atomics with concerns of being forced to use
>        only large writes with LBS?
>    - statx is modified so to display new alignment considerations
>    - atomics are power of 2
>    - there seems to be some interest in supporting no-hardware-accel atomic
>      solution, so a software implemented atomic solution, could someone
>      clarify if that's accurate? How is the double write avoided? What are
>      the use cases? Do databases use that today?
>    - How do we generalize a solution per file? Would extending a min
>      order per file be desirable? Is that even tenable?

I would also be interested in this discussion. For e.g. let's also try
and bring below points in the agenda -

1. Like LBS, for systems with large page size 64k (PowerPC and ARM), we should
already be able to utilize the untorn/atomic writes if they format the
filesystem with a given blocksize (for DIO atleast).
I think we need not even use bigalloc in such case for ext4.
So what does it takes from Linux Filesystems to expose an interface
to user such that they can start utilizing it?
(Now this has a catch that the FS still needs to be formatted with a
given blocksize to utilize untorn writes.)

2. What others think on adding O_ATOMIC interface similar to O_DIRECT
such that applications don't need much changes? We should still have
RWF_ATOMIC for pwrites, but for open/read/write calls an O_ATOMIC will
be useful too.

3. Buffered-io is important for Postgres and I have been looking into it
from the perspective of supporting untorn writes for buffered-io as
well. It will be again easier maybe to start off with 64k pagesize
systems or by utilizing large folio support. This way we have less work
in managing multiple pages which needs to be written atomically.

4. We already have RFC for ext4 multiblock code to support aligned
allocations which can be used to plug in untorn Direct-io write support
to ext4 [1]

[1]: https://lore.kernel.org/linux-ext4/cover.1701339358.git.ojaswin@xxxxxxxxxxxxx/


>
>   * LBS:
>     - stat will return the block size set, so userspace applications
>       using stat / statx will use the larger block size to ensure
>       alignment
>     - a drive with support for a large atomic but supporting smaller
>       logical block sizes will still allow writes to the logical block
>       size. If a block driver has a "preference" (in NVMe this would
>       be the NPWG for the IU) to write above the logical block size,
>       do we want the option to lift the logical block size? In
>       retrospect I don't think this is needed given Jan Kara's patches
>       to prevent allowing writes to to mounted devices [4], that should
>       ensure that if a filesystem takes advantage of a larger physical
>       block size and creates a filesystem with it as the sector size,
>       userspace won't be mucking around with lower IOs to the drive
>       while it is mounted. But, are there any applications which would
>       get the block device logical block size instead for DIO?
>     - LBS is transparent to to userspace applications
>     - We've verified *most* IOs are aligned if you use a 16k block size
>       but a smaller sector size, the lower IOs were verified to come
>       from the XFS buffer cache. If your drive supports a large atomic
>       you can avoid these as you can lift the sector size set as the
>       physical block size will be larger than the logical block size.
>       For NVMe today this is possible for drives with a large
>       NPWG (the IU) and NAWUFP (the large atomic), for example.
>
> Tooling:
>
>   - Both efforts stand to gain from a shared verification set of tools
>     for alignment and atomic use
>   - We have a block layer eBPF alignent tool written by Daniel Gomez [6]
>     however there is lack of interested parties to help review a simpler
>     version of this tool this tool so we merge it [7], we can benefit from more
>     eyeablls from experienced eBPF / block layer folks.
>   - More advanced tools are typically not encouraged, and this leaves us
>     wondering what a better home would be other than side forks
>   - Other than preventing torn writes, do users of the no-tear API care
>     about WAF? While we have one for NVMe for WAF [8] would
>     collaborating on a generic tool be of interest ?
>
> Any other things folks want to get out of this as a session, provided
> there is interest?
>
> [0] https://lwn.net/Articles/932900/
> [1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-twp.html
> [2] https://lore.kernel.org/linux-nvme/20240124113841.31824-1-john.g.garry@xxxxxxxxxx/T/#m4ad28b480a8e12eb51467e17208d98ca50041ff2
> [3] https://lore.kernel.org/all/20240124142645.9334-1-john.g.garry@xxxxxxxxxx/
> [4] https://lore.kernel.org/all/20240213093713.1753368-1-kernel@xxxxxxxxxxxxxxxx/T/#u
> [5] https://lkml.kernel.org/r/20231101173542.23597-1-jack@xxxxxxx
> [6] https://github.com/dagmcr/bcc/tree/blkalgn-dump
> [7] https://github.com/iovisor/bcc/pull/4813
> [8] https://github.com/dagmcr/bcc/tree/nvmeiuwaf
>
>   Luis