So, I'd like to take the approach of a compromise. Here's what I see as
a list of the behaviors a semi-raw device should do in various
circumstances that a raw device would be used if it was available...
First, I'll cover the case of read caching on a buffered device.
I believe this is the easier and more obvious case.
I think that a normal disk device could be used here, with one or
more ioctl's added to alter read caching behavior as follows:
r1. In the case of a shared SCSI device, there needs to be a mechanism
to tell the OS to throw away read cached blocks as invalid, as the
other masters on the SCSI bus may have written new data to the block.
If there is higher level caching going on too, then invalidating
that cache needs to be negotiated at in that layer as well.
r2. In the case of a database or other complex application accessing
the disk, the application should be able to tell the buffer cache
to move specific blocks to the top of the list of memory pages to
be reused, so that access patterns unexpected by the OS don't
cause worst case cache behavior. madvise is probably not enough
here unless it can address specific pages or ranges of pages.
r3. If the application wants to cache read pages itself, it should be
possible for the OS and the application to share the same pages in
cache, possibly via mmap or some similar mechanism. (This is
already implemented, I believe.)
I don't see a need in any of these cases for a pure raw read device.
I think that mechanisms to invalidate or control the aging of read
cache data should be enough.
If there are other read cases I've not considered, please address them.
Now, the case of semi-raw writing is much nastier...
w1. In the case of SCSI device sharing, the blocks being written to
probably will need to be negotiated for by some mechanism at a
higher level and possibly in a different medium (ethernet?
high speed serial? direct host to host SCSI communications?), and
locked once negotiated, to prevent multiple writes or premature
reads of stale data. Once the blocks to be modified are negotiated
for, I don't think it matters when they get actually written to,
except that in some circumstances, the other machine may be waiting
for the write to complete, so it should be flushed ASAP to prevent
deadlocks or performance bottlenecks.
The case of a database write, however, is far more complex, and, at
least to me, it seems like it might actually be easier to implement real
raw writes than to build a queuing system that would be acceptable to a
database. However, if such a system were built, I imagine it would need
at least the following semantics (anyone expert in databases might be
able to point out cases I've missed here...)
w2. Some metadata blocks are critical, and should be written immediately.
For instance, in Solaris 2, the superblock has a 'dirty' flag.
When all dirty filesystem blocks are written to disk the dirty flag
is cleared, even if the filesystem is still mounted. However, as soon
as new dirty blocks are generated (or perhaps just before a partial
write is started), the dirty flag is set again, and the superblock
immediately written back out.
Of course, there may be more than one of these "write immediately"
datablocks, so they would have to be queued or prioritized.
w3. Some kinds of metadata and data can be delayed indefinitely, but
need to be written in a specific order. Perhaps, there are chunks
of this type of data where order is not internally important.
Some form of tagged queuing or prioritized queuing might work
here, allowing ordered markers between internally unordered chunks,
and possibly allowing new data to be added to unwritten chunks.
w4. Of course, it is necessary to have a "write everything now" call,
such as sync().
w5. Going back to shared SCSI devices...
If, for instance, we negotiated for a block we will eventually
write to, and the other system originally decided that it didn't
need the block...and then later, the second system changes its
mind, there needs to be a way to say "OK, write that delayed block
ASAP", again to prevent deadlocks and performance bottlenecks.
The actual negotiation for blocks on shared devices should probably
occur at a higher level than the buffer cache--either user level or
at filesystem level; however, there still needs to be a buffer cache
level mechanism to change the priority of or immediately flush either
single blocks or entire groups of blocks.
If done carefully, these types of behaviors could probably be implemented
with a few ioctl's or system calls on top of the existing buffered disk
devices.
Does this make sense? Would this kind of behavior be an acceptable
compromise between those who don't want raw devices, and those that say
that kernel buffering is not currently acceptable? Are there any
situations I've left out?
Steve
ssd@nevets.oau.org