Re: [PATCHv4] SGI UV: TLB shootdown using broadcast assist unit

[Cliff Wickman]

Hi Nick,

On Thu, Jun 12, 2008 at 10:35:29PM +1000, Nick Piggin wrote:
> On Thursday 12 June 2008 22:23, Cliff Wickman wrote:
> > From: Cliff Wickman <cpw@xxxxxxx>
> >
> > TLB shootdown for SGI UV.
> >
> > v1: 6/2 original
> > v2: 6/3 corrections/improvements per Ingo's review
> > v3: 6/4 split atomic operations off to a separate patch (Jeremy's review)
> > v4: 6/12 include <mach_apic.h> rather than <asm/mach-bigsmp/mach_apic.h>
> >          (fixes a !SMP build problem that Ingo found)
> >          fix the index on uv_table_bases[blade]
> > Now depends on patch:
> >     x86 atomic operations: atomic_or_long atomic_inc_short
> >   which was split off (and improved) at the suggestion of
> >   Jeremy Fitzhardinge <jeremy@xxxxxxxx>
> >
> > This patch provides the ability to flush TLB's in cpu's that are not on
> > the local node.  The hardware mechanism for distributing the flush
> > messages is the UV's "broadcast assist unit".
> >
> > The hook to intercept TLB shootdown requests is a 2-line change to
> > native_flush_tlb_others() (arch/x86/kernel/tlb_64.c).
> >
> > This code has been tested on a hardware simulator. The real hardware
> > is not yet available.
> >
> > The shootdown statistics are provided through /proc/sgi_uv/ptc_statistics.
> > The use of /sys was considered, but would have required the use of
> > many /sys files.  The debugfs was also considered, but these statistics
> > should be available on an ongoing basis, not just for debugging.
> >
> > Issues to be fixed later:
> > - The IRQ for the messaging interrupt is currently hardcoded as 200
> >   (see UV_BAU_MESSAGE).  It should be dynamically assigned in the future.
> > - The use of appropriate udelay()'s is untested, as they are a problem
> >   in the simulator.
> 
> For someone not too familiar with low level x86 (or UV) code, can
> you explain why you are hooking at this point? I mean, what it
> looks like is either a performance improvement, or for some reason
> UV does not support send_IPI_mask out to CPUs "not on the local node".

Yes, a performance improvement.  The UV machine has hardware for
broadcasting messages to a set of nodes (represented in a bit mask).  The
messages will raise interrupts at each of the target nodes and provide
the message - all in one step.
(IPI is supported.  In fact this patch falls back to the IPI method
 if all the cpus on the remote nodes do not respond.)
> 
> If the former, what sort of improvement to you expect / see?

Good question.  The hardware does not exist yet.  But using IPI there
would be one set of packets exchanged to deliver the interrupts and
another set to pull over the flush address, just to start the operation.
I expect the improvement to be significant.

-Cliff
-- 
Cliff Wickman
Silicon Graphics, Inc.
cpw@xxxxxxx
(651) 683-3824
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/