Re: [PATCH 1/2] nfs: writeback pages wait queue

[Wu Fengguang]

On Sun, Oct 23, 2011 at 11:54:39PM +0800, Wu Fengguang wrote:
> On Fri, Oct 21, 2011 at 12:05:30AM +0800, Wu Fengguang wrote:
> > Trond,
> > 
> > After applying these two patches, the IO-less patchset performances
> > 45% better than the vanilla kernel and the average commit size only
> > decreases by -16% in the common NFS-thresh=1G/nfs-1dd case :)
> 
> To better understand how the NFS writeback wait queue helps, I
> visualized the network traffic over time. Attached are the graphs for
> the vanilla kernel and the one with the IO-less + NFS wait queue
> patches.
> 
> nfs-1dd-4k-32p-32016M-1024M:10-3.1.0-rc8-vanilla+/dstat-bw.png
> nfs-1dd-4k-32p-31951M-1024M:10-3.1.0-rc8-nfs-wq4+/dstat-bw.png
> 
> The obvious difference is, the network traffic become now more
> distributed and the "zero traffic" periods are mostly reduced.
> 
> The other 2dd, 10dd cases have similar results.

To better explore the internal NFS server states and to make it clear
where the performance gain comes from, I collected the dstat data in
the NFS server and visualized the network/disk throughputs over time.
The results are attached for the following test cases. 

nfs-1dd-4k-32p-32016M-1024M:10-3.1.0-rc10-vanilla+
nfs-1dd-4k-32p-31951M-1024M:10-3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+

In the above cases, the vanilla kernel is shown to have roughly
_interleaved_ disk/net activities. After the patches, the disk/net
pipeline is well established, hence the impressive performance improvements.

     3.1.0-rc10-vanilla+  3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+  
------------------------  ------------------------  
                   57.48       +25.5%        72.11  NFS-thresh=1G/nfs-10dd-4k-32p-32768M-1024M:10-X
                   63.86       +27.1%        81.15  NFS-thresh=1G/nfs-1dd-4k-32p-32768M-1024M:10-X
                   47.51       +80.1%        85.54  NFS-thresh=1G/nfs-2dd-4k-32p-32768M-1024M:10-X
                  168.85       +41.4%       238.80  TOTAL write_bw

          3.1.0-vanilla+  3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+
------------------------  ------------------------
                   21.93       +97.0%        43.21  NFS-thresh=100M/nfs-10dd-4k-32p-32768M-100M:10-X
                   55.91       +27.4%        71.23  NFS-thresh=100M/nfs-1dd-4k-32p-32768M-100M:10-X
                   26.73      +129.2%        61.25  NFS-thresh=100M/nfs-2dd-4k-32p-32768M-100M:10-X
                   15.66       +40.3%        21.96  NFS-thresh=10M/nfs-10dd-4k-32p-32768M-10M:10-X
                   29.01        +0.7%        29.20  NFS-thresh=10M/nfs-1dd-4k-32p-32768M-10M:10-X
                   20.52        +7.9%        22.13  NFS-thresh=10M/nfs-2dd-4k-32p-32768M-10M:10-X
                   56.30       +28.1%        72.11  NFS-thresh=1G/nfs-10dd-4k-32p-32768M-1024M:10-X
                   63.33       +28.1%        81.15  NFS-thresh=1G/nfs-1dd-4k-32p-32768M-1024M:10-X
                   51.95       +64.6%        85.54  NFS-thresh=1G/nfs-2dd-4k-32p-32768M-1024M:10-X
                    9.87       -48.2%         5.11  NFS-thresh=1M/nfs-10dd-4k-32p-32768M-1M:10-X
                   14.52       -24.0%        11.03  NFS-thresh=1M/nfs-1dd-4k-32p-32768M-1M:10-X
                   15.69       -67.4%         5.12  NFS-thresh=1M/nfs-2dd-4k-32p-32768M-1M:10-X
                  381.42       +33.5%       509.05  TOTAL write_bw

The vanilla kernel's disk idle periods indicate that the background
flusher work is not running. This is true. The NFS server has 12GB
memory and hence 1.2GB background dirty thresh. So when the client
side has dirty thresh <= 1GB, it won't send enough data to trigger
the background work in NFS server. So the server disk IOs will be
triggered mostly by NFS commits. The patched kernel seems to generate
smoother commit pattern to prevent most disk idles.

To explore how it performs when the NFS server's background writeback
kicks in, I further tried lowering the NFS server's dirty thresholds
to

dirty_bytes = 1GB, background threshold = 512MB
-----------------------------------------------
graphs attached for cases:
        nfs-1dd-1M-32p-32016M-1024M:10-3.1.0-vanilla+
        nfs-1dd-1M-32p-31951M-1024M:10-3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+

          3.1.0-vanilla+  3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+
------------------------  ------------------------
                   25.23       +71.2%        43.21  NFS-thresh=100M/nfs-10dd-1M-32p-32768M-100M:10-X
                   60.21       +15.1%        69.29  NFS-thresh=100M/nfs-1dd-1M-32p-32768M-100M:10-X
                   33.46       +82.8%        61.18  NFS-thresh=100M/nfs-2dd-1M-32p-32768M-100M:10-X
                   21.81        +2.8%        22.42  NFS-thresh=10M/nfs-10dd-1M-32p-32768M-10M:10-X
                   28.71       -11.0%        25.55  NFS-thresh=10M/nfs-1dd-1M-32p-32768M-10M:10-X
                   28.11       -13.0%        24.46  NFS-thresh=10M/nfs-2dd-1M-32p-32768M-10M:10-X
                   54.10       +42.3%        76.97  NFS-thresh=1G/nfs-10dd-1M-32p-32768M-1024M:10-X
                   64.88       +29.1%        83.76  NFS-thresh=1G/nfs-1dd-1M-32p-32768M-1024M:10-X
                   53.64       +56.2%        83.79  NFS-thresh=1G/nfs-2dd-1M-32p-32768M-1024M:10-X
                   13.00       -61.0%         5.06  NFS-thresh=1M/nfs-10dd-1M-32p-32768M-1M:10-X
                   13.53       -33.9%         8.95  NFS-thresh=1M/nfs-1dd-1M-32p-32768M-1M:10-X
                   16.51       -72.5%         4.54  NFS-thresh=1M/nfs-2dd-1M-32p-32768M-1M:10-X
                  413.20       +23.2%       509.18  TOTAL write_bw

dirty_bytes = 100MB, background threshold = 50MB
------------------------------------------------
graphs attached for cases:
        nfs-1dd-100k-32p-32016M-1024M:10-3.1.0-vanilla+
        nfs-1dd-100k-32p-31951M-1024M:10-3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+

          3.1.0-vanilla+  3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+
------------------------  ------------------------
                   24.93       +71.1%        42.67  NFS-thresh=100M/nfs-10dd-100k-32p-32768M-100M:10-X
                   58.35       +23.0%        71.78  NFS-thresh=100M/nfs-1dd-100k-32p-32768M-100M:10-X
                   28.47      +123.0%        63.48  NFS-thresh=100M/nfs-2dd-100k-32p-32768M-100M:10-X
                   18.44       +22.5%        22.60  NFS-thresh=10M/nfs-10dd-100k-32p-32768M-10M:10-X
                   28.75        +0.1%        28.78  NFS-thresh=10M/nfs-1dd-100k-32p-32768M-10M:10-X
                   20.84       +12.0%        23.34  NFS-thresh=10M/nfs-2dd-100k-32p-32768M-10M:10-X
                   71.78        +6.4%        76.38  NFS-thresh=1G/nfs-10dd-100k-32p-32768M-1024M:10-X
                   91.02        -4.7%        86.72  NFS-thresh=1G/nfs-1dd-100k-32p-32768M-1024M:10-X
                   78.84        +9.9%        86.64  NFS-thresh=1G/nfs-2dd-100k-32p-32768M-1024M:10-X
                  421.42       +19.2%       502.39  TOTAL write_bw

The conclusion is,

- it helps the vanilla kernel performance to lower the NFS server's
  dirty threshold

- when NFS server's background writeback is not running, the patches
  will help establish good net/disk pipeline and improve performance;
  otherwise it maintains roughly the same performance

Thanks,
Fengguang

Attachment: dstat-nfss-bw.png
Description: PNG image

Attachment: dstat-nfss-bw.png
Description: PNG image

Attachment: dstat-nfss-bw.png
Description: PNG image

Attachment: dstat-nfss-bw.png
Description: PNG image

Attachment: dstat-nfss-bw.png
Description: PNG image

Attachment: dstat-nfss-bw.png
Description: PNG image