Re: [PATCH v7] NVMe: conversion to blk-mq
From: Keith Busch
Date: Fri Jun 13 2014 - 11:06:18 EST
On Fri, 13 Jun 2014, Jens Axboe wrote:
On 06/12/2014 06:06 PM, Keith Busch wrote:
When cancelling IOs, we have to check if the hwctx has a valid tags
for some reason. I have 32 cores in my system and as many queues, but
It's because unused queues are torn down, to save memory.
blk-mq is only using half of those queues and freed the "tags" for the
rest after they'd been initialized without telling the driver. Why is
blk-mq not making utilizing all my queues?
You have 31 + 1 queues, so only 31 mappable queues. blk-mq symmetrically
distributes these, so you should have a core + thread sibling on 16
queues. And yes, that leaves 15 idle hardware queues for this specific
case. I like the symmetry, it makes it more predictable if things are
spread out evenly.
You'll see performance differences on some workloads that depend on which
cores your process runs and which one services an interrupt. We can play
games with with cores and see what happens on my 32 cpu system. I usually
run 'irqbalance --hint=exact' for best performance, but that doesn't do
anything with blk-mq since the affinity hint is gone.
I ran the following script several times on each version of the
driver. This will pin a sequential read test to cores 0, 8, and 16. The
device is local to NUMA node on cores 0-7 and 16-23; the second test
runs on the remote node and the third on the thread sibling of 0. Results
were averaged, but very consistent anyway. The system was otherwise idle.
# for i in $(seq 0 8 16); do
> let "cpu=1<<$i"
> cpu=`echo $cpu | awk '{printf "%#x\n", $1}'`
> taskset ${cpu} dd if=/dev/nvme0n1 of=/dev/null bs=4k count=1000000 iflag=direct
> done
Here are the performance drops observed with blk-mq with the existing
driver as baseline:
CPU : Drop
....:.....
0 : -6%
8 : -36%
16 : -12%
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/