Re: [PATCH 00/15] Coalesced Interrupt Delivery with posted MSI

[Jacob Pan]

Hi Jens,

On Mon, 12 Feb 2024 11:36:42 -0700, Jens Axboe <axboe@xxxxxxxxx> wrote:

> > For the record, here is my set up and performance data for 4 Samsung
> > disks. IOPS increased from 1.6M per disk to 2.1M. One difference I
> > noticed is that IRQ throughput is improved instead of reduction with
> > this patch on my setup. e.g. BEFORE: 185545/sec/vector 
> >      AFTER:  220128  
> 
> I'm surprised at the rates being that low, and if so, why the posted MSI
> makes a difference? Usually what I've seen for IRQ being slower than
> poll is if interrupt delivery is unreasonably slow on that architecture
> of machine. But ~200k/sec isn't that high at all.

Even at ~200k/sec, I am seeing around 75% ratio between posted interrupt
notification and MSIs. i.e. for every 4 MSIs, we save one CPU notification.
That might be where the savings come from.

I was expecting an even or reduction in CPU notifications but more MSI
throughput. Instead, Optane gets less MSIs/sec as your data shows.

Is it possible to get the interrupt coalescing ratio on your set up? ie.
PMN count in cat /proc/interrupts divided by total NVME MSIs.

Here is a summary of my testing on 4 Samsung Gen 5 drives:
test cases		IOPS*1000	ints/sec(MSI)*
=================================================
aio 			6348		182218
io_uring		6895		207932
aio w/ posted MSI	8295		185545
io_uring w/ post MSI	8811		220128
io_uring poll_queue	13000		0
================================================


Thanks,

Jacob