RE: [PATCH v4 0/4] Implement vdpasim stop operation

[Parav Pandit]

> From: Jason Wang <jasowang@xxxxxxxxxx>
> Sent: Wednesday, June 1, 2022 11:54 PM
> 
> On Thu, Jun 2, 2022 at 10:59 AM Parav Pandit <parav@xxxxxxxxxx> wrote:
> >
> >
> > > From: Jason Wang <jasowang@xxxxxxxxxx>
> > > Sent: Wednesday, June 1, 2022 10:00 PM
> > >
> > > On Thu, Jun 2, 2022 at 2:58 AM Parav Pandit <parav@xxxxxxxxxx> wrote:
> > > >
> > > >
> > > > > From: Jason Wang <jasowang@xxxxxxxxxx>
> > > > > Sent: Tuesday, May 31, 2022 10:42 PM
> > > > >
> > > > > Well, the ability to query the virtqueue state was proposed as
> > > > > another feature (Eugenio, please correct me). This should be
> > > > > sufficient for making virtio-net to be live migrated.
> > > > >
> > > > The device is stopped, it won't answer to this special vq config done
> here.
> > >
> > > This depends on the definition of the stop. Any query to the device
> > > state should be allowed otherwise it's meaningless for us.
> > >
> > > > Programming all of these using cfg registers doesn't scale for
> > > > on-chip
> > > memory and for the speed.
> > >
> > > Well, they are orthogonal and what I want to say is, we should first
> > > define the semantics of stop and state of the virtqueue.
> > >
> > > Such a facility could be accessed by either transport specific
> > > method or admin virtqueue, it totally depends on the hardware
> architecture of the vendor.
> > >
> > I find it hard to believe that a vendor can implement a CVQ but not AQ and
> chose to expose tens of hundreds of registers.
> > But maybe, it fits some specific hw.
> 
> You can have a look at the ifcvf dpdk driver as an example.
> 
Ifcvf is an example of using registers.
It is not an answer why AQ is hard for it. :)
virtio spec has definition of queue now and implementing yet another queue shouldn't be a problem.

So far no one seem to have problem with the additional queue.
So I take it as AQ is ok.

> But another thing that is unrelated to hardware architecture is the nesting
> support. Having admin virtqueue in a nesting environment looks like an
> overkill. Presenting a register in L1 and map it to L0's admin should be good
> enough.
So may be a optimized interface can be added that fits nested env.
At this point in time real users that we heard are interested in non-nested use cases. Let's enable them first.


> 
> >
> > I like to learn the advantages of such method other than simplicity.
> >
> > We can clearly that we are shifting away from such PCI registers with SIOV,
> IMS and other scalable solutions.
> > virtio drifting in reverse direction by introducing more registers as
> transport.
> > I expect it to an optional transport like AQ.
> 
> Actually, I had a proposal of using admin virtqueue as a transport, it's
> designed to be SIOV/IMS capable. And it's not hard to extend it with the
> state/stop support etc.
> 
> >
> > > >
> > > > Next would be to program hundreds of statistics of the 64 VQs
> > > > through a
> > > giant PCI config space register in some busy polling scheme.
> > >
> > > We don't need giant config space, and this method has been
> > > implemented by some vDPA vendors.
> > >
> > There are tens of 64-bit counters per VQs. These needs to programmed on
> destination side.
> > Programming these via registers requires exposing them on the registers.
> > In one of the proposals, I see them being queried via CVQ from the device.
> 
> I didn't see a proposal like this. And I don't think querying general virtio state
> like idx with a device specific CVQ is a good design.
> 
My example was not for the idx. But for VQ statistics that is queried via CVQ.

> >
> > Programming them via cfg registers requires large cfg space or synchronous
> programming until receiving ACK from it.
> > This means one entry at a time...
> >
> > Programming them via CVQ needs replicate and align cmd values etc on all
> device types. All duplicate and hard to maintain.
> >
> >
> > > >
> > > > I can clearly see how all these are inefficient for faster LM.
> > > > We need an efficient AQ to proceed with at minimum.
> > >
> > > I'm fine with admin virtqueue, but the stop and state are orthogonal to
> that.
> > > And using admin virtqueue for stop/state will be more natural if we
> > > use admin virtqueue as a transport.
> > Ok.
> > We should have defined it bit earlier that all vendors can use. :(
> 
> I agree.

I remember few months back, you acked in the weekly meeting that TC has approved the AQ direction.
And we are still in this circle of debating the AQ.