vbus design points: shm and shm-signals

[Gregory Haskins]

Gregory Haskins wrote:
> Ingo Molnar wrote:
>>
>> We all love faster code and better management interfaces and tons 
>> of your prior patches got accepted by Avi. This time you didnt even 
>> _try_ to improve virtio.
> 
> Im sorry, but you are mistaken:
> 
> http://lkml.indiana.edu/hypermail/linux/kernel/0904.2/02443.html
> 

BTW: One point that I forgot to point out in this most recent thread
that I am particularly proud of here is the design of the vbus
shared-memory model.  Despite some claims to the contrary; not only is
it possible to improve virtio with vbus (as evident by the patch
referenced above)...I specifically designed vbus with virtio
considerations in mind from the start!  In fact, the design is conducive
to accelerating a variety of other models as well.  Read on for details.

Vbus was designed it to be _agnostic_ to the shm algorithm in general.
This allows you to, of course, run ring algorithms (such as virtqueues,
or IOQs), but really any other designs as well, such as shared-tables, etc.

A guest driver sees the following interface:

struct vbus_device_proxy_ops {
	int (*open)(struct vbus_device_proxy *dev, int version, int flags);
	int (*close)(struct vbus_device_proxy *dev, int flags);
	int (*shm)(struct vbus_device_proxy *dev, int id, int prio,
		   void *ptr, size_t len,
		   struct shm_signal_desc *sigdesc, struct shm_signal **signal,
		   int flags);
	int (*call)(struct vbus_device_proxy *dev, u32 func,
		    void *data, size_t len, int flags);
	void (*release)(struct vbus_device_proxy *dev);
};

note the ops->shm() method.  This allows the driver to register some
arbitrary pointer (ptr, len) with the host, optionally embedding a
shm_signal_desc object in the memory.  If "sigdesc" is non-null, the
connector will allocate and return a fully formed shm_signal object in
**signal.

shm-signal (posted in patch 1/6, I believe) is a generalization of the
basic mechanism you need for emitting and catching events via any
shared-memory based design.  It includes interrupt/hypercall mitigation
support which is independent of the actual shm algorithm (e.g. ring,
table, etc).  This means that we can get the event mitigation code (e.g.
disable unecessary interrupts/hypercalls, prevent spurious re-calls,
etc) in once place (and, ideally, correct), and things like the ring
algorithm (or table design, etc) can focus on their value-add, instead
of re-inventing the mitigation code each time.

You can then build your higher-layer algorithms (rings, tables, etc) in
terms of the shm-signal library.  As a matter of fact, if you look at
the patch referenced above, it implements the virtqueue->kick() method
on top of shm-signal.  IOQs follow a similar pattern.  And the RT
enhancements will, for instance, be using a table design for the
scheduler/interrupt control.

In short, vbus is the result of my experience in dealing with issues
like IO in virt.  I thought long and hard about where the issues were
for high-performance, low-latency, software-to-software IO for a
wide-variety of applications and environments.  I then tried to
systematically solve those problems at various key points in the stack
to promote maximum flexibility and reuse of those enhancements.  So we
see things like this generalization of async event mitigation with the
shm/shm-signal design, low-latency "hypercalls", reusable backend models
(which support both a variety of virt, as well as physical system use), etc.

Part of this flexibility means that we do not want to rely on something
like PCI because it's not necessarily available/logical in all
platforms/environments.  So rather than force such environments to fit
into a PCI ABI, we start over again and offer an ABI specific to the
actual goals (high-performance, low-latency, software-to-software IO).

Yes, that means we will possibly take some lumps in the short-term
before these concepts and support for them are ubiquitous (like PCI, USB
are today).  But, like all things, you have to start somewhere.  The PCI
guys didn't try to make PCI look like ISA, and the USB guys didn't try
to make USB look like PCI.  It took a little while for support for such
notions to catch on, but eventually all relevant platforms supported
them.  I am going for the same thing here.  Sometimes, the existing
model just doesn't fit well and you have to re-evaluate.

I hope that this helps clarify some of the design of vbus and why I
believe it to be worth considering.  I will plan on taking this email as
the first entry on the wiki as a "design series" (or something like
that) and update it regularly with other aspects of the design.

Kind Regards,
-Greg

Attachment: signature.asc
Description: OpenPGP digital signature