Re: [RFC PATCH 0/4] Gang scheduling in CFS

[Nikunj A Dadhania]

On Wed, 21 Dec 2011 12:43:43 +0200, Avi Kivity <avi@xxxxxxxxxx> wrote:
> On 12/21/2011 12:39 PM, Nikunj A Dadhania wrote:
> > On Mon, 19 Dec 2011 12:23:26 +0100, Ingo Molnar <mingo@xxxxxxx> wrote:
> > > 
> > > * Nikunj A. Dadhania <nikunj@xxxxxxxxxxxxxxxxxx> wrote:
> > > 
> > > So could we please approach this from the benchmarked workload 
> > > angle first? The highest improvement is in ebizzy:
> > > 
> > > >     ebizzy 2vm (improved 15 times, i.e. 1520%)
> > > 
> > > What's behind this huge speedup? Does ebizzy use user-space 
> > > spinlocks perhaps? Could we do something on the user-space side 
> > > to get a similar speedup?
> > > 
> > This is from the perf run on the host:
> >
> > Baseline:
> >
> > 16.22%         qemu-kvm  [kvm_intel]    [k] free_kvm_area
> >  8.27%         qemu-kvm  [kvm]          [k] start_apic_timer
> >  7.53%         qemu-kvm  [kvm]          [k] kvm_put_guest_fpu
> >
> > Gang:
> >
> > 24.44%         qemu-kvm  [kvm_intel]    [k] free_kvm_area
> > 13.42%         qemu-kvm  [kvm]          [k] start_apic_timer 
> >  9.91%         qemu-kvm  [kvm]          [k] kvm_put_guest_fpu
> >
> > Ingo, Avi, I am not getting anything obvious from this. Any ideas?
> >

Here some interesting perf reports from inside the guest:

Baseline:
  29.79%   ebizzy  [kernel.kallsyms]   [k] native_flush_tlb_others
  18.70%   ebizzy  libc-2.12.so        [.] __GI_memcpy
   7.23%   ebizzy  [kernel.kallsyms]   [k] get_page_from_freelist
   5.38%   ebizzy  [kernel.kallsyms]   [k] __do_page_fault
   4.50%   ebizzy  [kernel.kallsyms]   [k] ____pagevec_lru_add
   3.58%   ebizzy  [kernel.kallsyms]   [k] default_send_IPI_mask_logical
   3.26%   ebizzy  [kernel.kallsyms]   [k] native_flush_tlb_single
   2.82%   ebizzy  [kernel.kallsyms]   [k] handle_pte_fault
   2.16%   ebizzy  [kernel.kallsyms]   [k] kunmap_atomic
   2.10%   ebizzy  [kernel.kallsyms]   [k] _spin_unlock_irqrestore
   1.90%   ebizzy  [kernel.kallsyms]   [k] down_read_trylock
   1.65%   ebizzy  [kernel.kallsyms]   [k] __mem_cgroup_commit_charge.clone.4
   1.60%   ebizzy  [kernel.kallsyms]   [k] up_read
   1.24%   ebizzy  [kernel.kallsyms]   [k] __alloc_pages_nodemask

Gang:
  22.53%   ebizzy  libc-2.12.so       [.] __GI_memcpy
   9.73%   ebizzy  [kernel.kallsyms]  [k] ____pagevec_lru_add
   8.22%   ebizzy  [kernel.kallsyms]  [k] get_page_from_freelist
   7.80%   ebizzy  [kernel.kallsyms]  [k] default_send_IPI_mask_logical
   7.68%   ebizzy  [kernel.kallsyms]  [k] native_flush_tlb_others
   6.22%   ebizzy  [kernel.kallsyms]  [k] __do_page_fault
   5.54%   ebizzy  [kernel.kallsyms]  [k] native_flush_tlb_single
   4.44%   ebizzy  [kernel.kallsyms]  [k] _spin_unlock_irqrestore
   2.90%   ebizzy  [kernel.kallsyms]  [k] kunmap_atomic
   2.78%   ebizzy  [kernel.kallsyms]  [k] __mem_cgroup_commit_charge.clone.4
   2.76%   ebizzy  [kernel.kallsyms]  [k] handle_pte_fault
   2.16%   ebizzy  [kernel.kallsyms]  [k] __mem_cgroup_uncharge_common
   1.59%   ebizzy  [kernel.kallsyms]  [k] down_read_trylock
   1.43%   ebizzy  [kernel.kallsyms]  [k] up_read

I see the main difference between both the reports is:
native_flush_tlb_others.

Regards
Nikunj

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/