Re: [bisected] x86 boot still broken on -rc2

[Prarit Bhargava]

On 12/04/2017 07:28 AM, Prarit Bhargava wrote:
> 
> 
> On 12/03/2017 08:28 PM, Jakub Kicinski wrote:
>> Same thing on rc2, bisected down to:
>>
>> commit b4c0a7326f5dc0ef7a64128b0ae7d081f4b2cbd1 (refs/bisect/bad)
>> Author: Prarit Bhargava <prarit@xxxxxxxxxx>
>> Date:   Tue Nov 14 07:42:57 2017 -0500
>>
>>     x86/smpboot: Fix __max_logical_packages estimate
>>     
>>     A system booted with a small number of cores enabled per package
>>     panics because the estimate of __max_logical_packages is too low.
>>     
>>     This occurs when the total number of active cores across all packages is
>>     less than the maximum core count for a single package. e.g.:
>>     
>>       On a 4 package system with 20 cores/package where only 4 cores are
>>       enabled on each package, the value of __max_logical_packages is
>>       calculated as DIV_ROUND_UP(16 / 20) = 1 and not 4.
>>     
>>     Calculate __max_logical_packages after the cpu enumeration has completed.
>>     Use the boot cpu's data to extrapolate the number of packages.
>>     
>>     Signed-off-by: Prarit Bhargava <prarit@xxxxxxxxxx>
>>     Signed-off-by: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
>>     Cc: Tom Lendacky <thomas.lendacky@xxxxxxx>
>>     Cc: Andi Kleen <ak@xxxxxxxxxxxxxxx>
>>     Cc: Christian Borntraeger <borntraeger@xxxxxxxxxx>
>>     Cc: Peter Zijlstra <peterz@xxxxxxxxxxxxx>
>>     Cc: Kan Liang <kan.liang@xxxxxxxxx>
>>     Cc: He Chen <he.chen@xxxxxxxxxxxxxxx>
>>     Cc: Stephane Eranian <eranian@xxxxxxxxxx>
>>     Cc: Dave Hansen <dave.hansen@xxxxxxxxx>
>>     Cc: Piotr Luc <piotr.luc@xxxxxxxxx>
>>     Cc: Andy Lutomirski <luto@xxxxxxxxxx>
>>     Cc: Arvind Yadav <arvind.yadav.cs@xxxxxxxxx>
>>     Cc: Vitaly Kuznetsov <vkuznets@xxxxxxxxxx>
>>     Cc: Borislav Petkov <bp@xxxxxxx>
>>     Cc: Tim Chen <tim.c.chen@xxxxxxxxxxxxxxx>
>>     Cc: Mathias Krause <minipli@xxxxxxxxxxxxxx>
>>     Cc: "Kirill A. Shutemov" <kirill.shutemov@xxxxxxxxxxxxxxx>
>>     Link: https://lkml.kernel.org/r/20171114124257.22013-4-prarit@xxxxxxxxxx
>>
>>
>> On Fri, 1 Dec 2017 16:39:54 -0800, Jakub Kicinski wrote:
>>> Hi!
>>>
>>> I'm hitting these after DaveM pulled rc1 into net-next on my Xeon
>>> E5-2630 v4 box.  It also happens on linux-next.  Did anyone else
>>> experience it?  (.config attached)
>>>
>>> [    5.003771] WARNING: CPU: 14 PID: 1 at ../arch/x86/events/intel/uncore.c:936 uncore_pci_probe+0x285/0x2b0
>>> [    5.007544] Modules linked in:
>>> [    5.007544] CPU: 14 PID: 1 Comm: swapper/0 Not tainted 4.15.0-rc1-perf-00225-gb2a4e0a76b1d #782
>>> [    5.007544] Hardware name: Dell Inc. PowerEdge R730/072T6D, BIOS 2.3.4 11/08/2016
> 
> I have a Dell R730 available for use.  OOC are you booting with the default
> BIOS options?
>

Jakub, I was able to reproduce this on a similar system by DISABLING
hyperthreading in the BIOS.  Doing this on other systems seems to have no
impact.  What is odd about this system when booting is that the
kernel claims that hyperthreading is ENABLED:

x86: Booting SMP configuration:
.... node  #0, CPUs:        #1  #2  #3  #4
.... node  #1, CPUs:    #5  #6  #7  #8  #9
.... node  #0, CPUs:   #10 #11 #12 #13 #14
.... node  #1, CPUs:   #15 #16 #17 #18 #19
smp: Brought up 2 nodes, 20 CPUs
smpboot: Max logical packages: 1

which means that the calculation of logical packages is wrong because

	ncpus = cpu_data(0).booted_cores * smp_num_siblings;
	ncpus = 10 * 2;
	ncpus = 20;

smp_num_siblings is defined as "The number of threads in a core" which
should be 1 if HT/SMT is disabled.

It looks like my patch has exposed a bug in the
smp_num_siblings calculation.   I'm still debugging ...

FWIW, I did test this code on systems by disabling HT/SMT in BIOS on
several systems.  I have tested those systems again and don't see a
problem.  It is something peculiar to a few systems.

P.