Re: [Fdutils] DMA cache consistency bug introduced in 2.6.28

[Pallipadi, Venkatesh]

On Fri, 2010-01-08 at 09:42 -0800, Mark Hounschell wrote:
> On 12/23/2009 03:30 PM, Pallipadi, Venkatesh wrote:
> 
> >>> Can you try this one line patch either on .28 or .32 (with /proc/interrupts
> >>> output).
> >>> This disables hpet2 and lapic timer should then be used on CPU 0. If things
> >>> work with this test patch, we will know that the failure is somehow related
> >>> to HPET usage in MSI mode.
> >>>
> >>> Thanks,
> >>> Venki
> >>>
> >>> Reduce the rating of percpu hpet timer
> >>>
> >>> Signed-off-by: Venkatesh Pallipadi <venkatesh.pallipadi@xxxxxxxxx>
> >>> ---
> >>>  arch/x86/kernel/hpet.c |    2 +-
> >>>  1 files changed, 1 insertions(+), 1 deletions(-)
> >>>
> >>> diff --git a/arch/x86/kernel/hpet.c b/arch/x86/kernel/hpet.c
> >>> index cafb1c6..f89d17a 100644
> >>> --- a/arch/x86/kernel/hpet.c
> >>> +++ b/arch/x86/kernel/hpet.c
> >>> @@ -480,7 +480,7 @@ static void init_one_hpet_msi_clockevent(struct hpet_dev *hdev, int cpu)
> >>>  	hpet_setup_irq(hdev);
> >>>  	evt->irq = hdev->irq;
> >>>  
> >>> -	evt->rating = 110;
> >>> +	evt->rating = 40;
> >>>  	evt->features = CLOCK_EVT_FEAT_ONESHOT;
> >>>  	if (hdev->flags & HPET_DEV_PERI_CAP)
> >>>  		evt->features |= CLOCK_EVT_FEAT_PERIODIC;
> >>
> >> That made it work. Used 2.6.32.2
> >>
> >> cat /proc/interrupts
> >>            CPU0       CPU1       CPU2       CPU3
> >>   0:         82          0          0          1   IO-APIC-edge      timer
> >>   1:          0          0          0         67   IO-APIC-edge      i8042
> >>   3:          0          0          0          6   IO-APIC-edge
> >>   4:          0          0          0          4   IO-APIC-edge
> >>   6:          0          0          0          4   IO-APIC-edge      floppy
> >>   8:          0          0          0          8   IO-APIC-edge      rtc0
> >>   9:          0          0          0          0   IO-APIC-fasteoi   acpi
> >>  12:          0          0         10       1519   IO-APIC-edge      i8042
> >>  14:          0          0         39      10995   IO-APIC-edge
> >> pata_atiixp
> >>  15:          0          0          3        391   IO-APIC-edge
> >> pata_atiixp
> >>  16:          0          0          2        606   IO-APIC-fasteoi
> >> aic79xx, ohci_hcd:usb3, ohci_hcd:usb4, HDA Intel, Digi DBX2, ni-pci-gpib
> >>  17:          0          0          0          3   IO-APIC-fasteoi
> >> ehci_hcd:usb1, parport0, ni-pci-gpib
> >>  18:          0          0         10       2168   IO-APIC-fasteoi
> >> ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7, Digi DBX2, nvidia
> >>  19:          0          0          0        130   IO-APIC-fasteoi
> >> aic7xxx, ehci_hcd:usb2, ttySLG0, eth1
> >>  22:          0          0          8       1151   IO-APIC-fasteoi   ahci
> >>  24:          0          0          0          0  HPET_MSI-edge      hpet2
> >>  29:          0          0          0         48   PCI-MSI-edge
> >> sky2@pci:0000:04:00.0
> >> NMI:          0          0          0          0   Non-maskable interrupts
> >> LOC:      34842      30177      29672      29632   Local timer interrupts
> >> SPU:          0          0          0          0   Spurious interrupts
> >> PMI:          0          0          0          0   Performance monitoring
> >> interrupts
> >> PND:          0          0          0          0   Performance pending work
> >> RES:      17501      20449      16670      11224   Rescheduling interrupts
> >> CAL:      10554       2336       1102       1071   Function call interrupts
> >> TLB:        364        562        753        468   TLB shootdowns
> >> ERR:          0
> >> MIS:          0
> >>
> >>
> >> # fdformat /dev/fd0u1440
> >> Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB.
> >> Formatting ... done
> >> Verifying ... done
> > 
> > Hmmm.. Thats very interesting indeed.
> > 
> > That clearly says that HPET MSI interrupts somehow is causing some
> > caching side effect in the chipset that results in this floppy dma
> > failure.
> > 
> > Here's is what we have until now.
> > IRQ 0 is based on HPET legacy interrupt and HPET device is also capable
> > of MSI on this platform. So we also have a percpu hpet (hpet2 tied to
> > CPU0). percpu hpet was added to avoid the usage of IRQ0+LAPIC broadcast
> > in cases where LAPIC timer will stop working in deep C-state. As we have
> > only one HPET channel free for percpu HPET, we only have hpet2 tied to
> > CPU 0 and other CPUs still have to go through IRQ0+LAPIC broadcast with
> > deep C-state.
> > 
> > One problem here is that percpu hpet should only get used when LAPIC
> > cannot be used (that is when CPU enters deep C-state). Using hpet2 in
> > place of LAPIC timer even when deep C-state is not supported is not
> > right in terms of performance. We need some changes here to fix that
> > [Problem 1].
> > 
> > But, that still does not explain why we are seeing this problem in the
> > first place. I mean, using hpet2 is not optimal, but should not have
> > functionality issues like this. Even fixing [Problem 1] above, we may
> > see this problem on some other platform that supports deep C-state and
> > so has hpet2 enabled for a valid reason.
> > 
> > Also, I am not sure whether the problem also happens if legacy HPET
> > interrupts are used during run time in place of LAPIC timer (May be
> > worth to try this with a simple test patch, let me think about it). In
> > this case, legacy HPET interrupt rightly goes quiet after boot, giving
> > priority to LAPIC timer.
> > 
> > With hpet MSI interrupts, we do a write followed by read of HPET
> > memmapped register to set a HPET channel timeout + read of global HPET
> > timer. This happens on every timer interrupt on CPU 0. And we also have
> > MSI interrupt being delivered to CPU 0. I cannot think of any reason why
> > this can break dma. We can probably try adding some dummy HPET read
> > after dma write, to see if that flushes things properly.
> > 
> 
> Haven't seen any activity on this thread in a while. Just curious, are we
> still working this?
> Is there anything else I can do to help?

Sorry for not following up on this. We have narrowed this down to HPET
MSI and floppy DMA. I still don't know how HPET MSI interrupts are
breaking floppy DMA.

You are seeing the problem on two different systems. Correct? Do you
have any system where this works with HPET MSI enabled?

Couple of options on how we can go about this one:
1) Change the HPET-MSI change to not get activated when there are no
C-states with LAPIC stoppage involved. This will resolve the problem on
the systems you reported as there are no deep C-states. But, I fear that
with the actual problem unresolved, we may hit it in future with this or
some other platform having same issue with CPUs that support deep
C-state.
2) Try this testcase on few other platforms that support HPET-MSI and
deep C-states and check how widespread the problem is and then add a
whitelist-blacklist for HPET MSI usage.

I think, for 2.6.33 option 1 is better. Will work on that and send in
patches for you test.

Thanks,
Venki 
 

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/