Re: Hugepages demand paging V2 [0/8]: Discussion and overview

[Ray Bryant]

Robin Holt wrote:
> On Wed, Oct 27, 2004 at 06:01:12PM -0500, Ray Bryant wrote:
>
> > Christoph Lameter wrote:
> >
> > > On Tue, 26 Oct 2004, Robin Holt wrote:
> > >
> > >
> > >
> > > > Sorry for being a stickler here, but the BTE is really part of the
> > > > I/O Interface portion of the shub.  That portion has a seperate clock
> > > > frequency from the memory controller (unfortunately slower).  The BTE
> > > > can zero at a slightly slower speed than the processor.  It does, as
> > > > you pointed out, not trash the CPU cache.
> > > >
> > > > One other feature of the BTE is it can operate asynchronously from
> > > > the cpu.  This could be used to, during a clock interrupt, schedule
> > > > additional huge page zero filling on multiple nodes at the same time.
> > > > This could result in a huge speed boost on machines that have multiple
> > > > memory only nodes.  That has not been tested thoroughly.  We have done
> > > > considerable testing of the page zero functionality as well as the
> > > > error handling.
> > >
> > >
> > > If the huge patch would support some way of redirecting the clearing of a
> > > huge page then we could:
> > >
> > > 1. set the huge pte to not present so that we get a fault on access
> > > 2. run the bte clearer.
> > > 3. On receiving a huge fault we could check for the bte being finished.
> > >
> > > This would parallelize the clearing of huge pages. But is that really more
> > > efficient? There may be complexity involved in allowing the clearing of
> > > multiple pages and tracking of the clear in progress is additional
> > > overhead.
> > >
> > >
> > > -
> > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > > the body of a message to majordomo@xxxxxxxxxxxxxxx
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > Please read the FAQ at  http://www.tux.org/lkml/
> >
> > I'm personally of the opinion that using the BTE to "speculatively" clear
> > hugetlb pages in advance of when the hugetlb pages are requested is not a 
> > good
> > thing [tm].  One never knows if those pages will ever be requested.  And in
> > the meantime, tasks that need the BTE will be delayed by speculative use.
> > But that is a personal bias  :-), with no data to back it up.
>
>
> I was thinking the bte would be best used in an async mode where the pages
> would be pre-zeroed and available for use if the application needs them.
> If the pre-zeroed list is empty, then use the cpu to zero the page.
>
>
> > AFAIK, it is faster to clear the page with the processor anyway, since the
>
>
> The processor is slightly faster.  I believe the FSB is 200Mhz and the
> II is 100Mhz (150Mhz with no attached IX brick).  Future versions of the
> BTE will possibly have faster access to on node memory than the processor.
>
>
> > processor has a faster clock cycle.  Yes, it destroys the processor cache,
> > but the application has clearly indicated that it wants the page NOW, 
> > please,
> > (because it has faulted on it), and delivering the page to the application
> > as quickly as possible sounds like a good thing.  I'm not sure reloading
>
>
> I am not either.  I just would like to see any design take into consideration
> the possible uses and not design them out.  Nothing more.
>
>
> > the processor cache at this point is a cost we care about, given that the
> > application is likely just starting up anyway.  I figure hugetlb pages are
> > allocated once, stay around a long long time, so I'm not sure optimizing to
> > minimize cache damage is the correct way to go here.
> >
> > The only obvious win is for memory only nodes, that have a BTE and no CPU.
> > It is probably faster to use the local BTE than a remote CPU to clear the 
> > page.
>
>
> Plus, a single CPU could schedule the clearing of pages on multiple
> nodes at the same time.  Imagine a system that has 256 compute nodes
> and 756 memory nodes.  That configuration is theoretically possible with
> todays hardware, but we have never built or sold one.  Looking at that
> configuration gives you an one possible indication of how a pre-zeroing
> mechanism might improve things.
>
> I am not saying that the BTE is the best option, or even a good one.  It
> just looks interesting.  It does bring up some interesting problems with
> repeatability.  Consider the application startup following termination
> of another which used all the huge pages.  The pre-zeroed list will
> be nearly if not completely empty.  The first fault will find the list
> empty and have to zero the page itself.  Hopefully, the second fault will
> find one on the zeroed list and return immediately.  This would cause
> application startup time to feel like it doubled from the previous run.
> Ouch.  That would be very upsetting for our typical customers.

Yep.

> The more memory nodes you have per cpu, the better this number will
> appear.
>
> Sorry for being spineless, but I don't feel very strongly that it will
> be beneficial enough to be desirable.  I am just not sure.  I would
> just hope that it is taken into consideration during the design and,
> as long as it has no negative impact on the design, be left as a
> possibility.
>
> Thanks,
> Robin Holt

As always, Robin, you are being very reasonable.  I think the option
should be kept open as you suggest, since it may help and I agree it
is an interesting approach that might yield big speedups.

--
Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
raybry@xxxxxxx             raybry@xxxxxxxxxxxxx
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/