Re: large files unnecessary trashing filesystem cache?

[Bodo Eggert]

On Tue, 18 Oct 2005, Badari Pulavarty wrote:

> On Tue, 2005-10-18 at 23:58 +0200, Bodo Eggert wrote:
> > Badari Pulavarty <pbadari@xxxxxxxxx> wrote:
> > > On Tue, 2005-10-18 at 22:01 +0200, Guido Fiala wrote:
> > 
> > [large files trash cache]
> > 
> > > Is there a reason why those applications couldn't use O_DIRECT ?
> > 
> > The cache trashing will affect all programs handling large files:
> > 
> > mkisofs * > iso
> > dd < /dev/hdx42 | gzip > imagefile
> > perl -pe's/filenamea/filenameb/' < iso | cdrecord - # <- never tried
> > 
> 
> Are these examples which demonstrate the thrashing problem.

You can alyo cat a big file into /dev/null. I made those examples in order 
to demonstrate the problem with using O_DIRECT.

OTOH, I don't realtime stuff on my computer, so I'm not really affected, 
but I'll try to show it anyway.

> > Changing a few programs will only partly cover the problems.
> > 
> > I guess the solution would be using random cache eviction rather than
> > a FIFO. I never took a look the cache mechanism, so I may very well be
> > wrong here.
> 
> Read-only pages should be re-cycled really easily & quickly. I can't
> belive read-only pages are causing you all the trouble.

Just a q&d test:

$ time ls -l $DIR > /dev/null
real    0m0.442s
user    0m0.008s
sys     0m0.024s

$ time ls -l $DIR > /dev/null
real    0m0.077s
user    0m0.008s
sys     0m0.008s

cat $BIGFILES_1.5GB > /Dev/null

$ time ls -l $DIR > /dev/null
real    0m0.270s
user    0m0.008s
sys     0m0.008s

$ time ls -l $DIR > /dev/null
real    0m0.078s
user    0m0.004s
sys     0m0.004s



BTW:
I suggested the random eviction because it will evict pages from large 
files more likely than pages from small files, but I now think it will 
cause the evicted pages to be non-continuous, too, and thereby cause 
rereading them to be slower. I don't know which effect would be worse.

-- 
Is reading in the bathroom considered Multitasking? 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/