Re: [RFC PATCH] clocksource: Suspend the watchdog temporarily when high read lantency detected

[Paul E. McKenney]

On Thu, Dec 22, 2022 at 02:37:24PM +0800, Feng Tang wrote:
> On Wed, Dec 21, 2022 at 10:14:29PM -0800, Paul E. McKenney wrote:
> > On Thu, Dec 22, 2022 at 02:00:42PM +0800, Feng Tang wrote:
> > > On Wed, Dec 21, 2022 at 09:55:15PM -0800, Paul E. McKenney wrote:
> > > > On Wed, Dec 21, 2022 at 10:39:53PM -0500, Waiman Long wrote:
> > > > > On 12/21/22 19:40, Paul E. McKenney wrote:
> > > > > > commit 199dfa2ba23dd0d650b1482a091e2e15457698b7
> > > > > > Author: Paul E. McKenney<paulmck@xxxxxxxxxx>
> > > > > > Date:   Wed Dec 21 16:20:25 2022 -0800
> > > > > > 
> > > > > >      clocksource: Verify HPET and PMTMR when TSC unverified
> > > > > >      On systems with two or fewer sockets, when the boot CPU has CONSTANT_TSC,
> > > > > >      NONSTOP_TSC, and TSC_ADJUST, clocksource watchdog verification of the
> > > > > >      TSC is disabled.  This works well much of the time, but there is the
> > > > > >      occasional system that meets all of these criteria, but which still
> > > > > >      has a TSC that skews significantly from atomic-clock time.  This is
> > > > > >      usually attributed to a firmware or hardware fault.  Yes, the various
> > > > > >      NTP daemons do express their opinions of userspace-to-atomic-clock time
> > > > > >      skew, but they put them in various places, depending on the daemon and
> > > > > >      distro in question.  It would therefore be good for the kernel to have
> > > > > >      some clue that there is a problem.
> > > > > >      The old behavior of marking the TSC unstable is a non-starter because a
> > > > > >      great many workloads simply cannot tolerate the overheads and latencies
> > > > > >      of the various non-TSC clocksources.  In addition, NTP-corrected systems
> > > > > >      often seem to be able to tolerate significant kernel-space time skew as
> > > > > >      long as the userspace time sources are within epsilon of atomic-clock
> > > > > >      time.
> > > > > >      Therefore, when watchdog verification of TSC is disabled, enable it for
> > > > > >      HPET and PMTMR (AKA ACPI PM timer).  This provides the needed in-kernel
> > > > > >      time-skew diagnostic without degrading the system's performance.
> > > > > >      Signed-off-by: Paul E. McKenney<paulmck@xxxxxxxxxx>
> > > > > >      Cc: Thomas Gleixner<tglx@xxxxxxxxxxxxx>
> > > > > >      Cc: Ingo Molnar<mingo@xxxxxxxxxx>
> > > > > >      Cc: Borislav Petkov<bp@xxxxxxxxx>
> > > > > >      Cc: Dave Hansen<dave.hansen@xxxxxxxxxxxxxxx>
> > > > > >      Cc: "H. Peter Anvin"<hpa@xxxxxxxxx>
> > > > > >      Cc: Daniel Lezcano<daniel.lezcano@xxxxxxxxxx>
> > > > > >      Cc: Feng Tang<feng.tang@xxxxxxxxx>
> > > > > >      Cc: Waiman Long <longman@xxxxxxxxxx
> > > > > >      Cc:<x86@xxxxxxxxxx>
> > > > > 
> > > > > As I currently understand, you are trying to use TSC as a watchdog to check
> > > > > against HPET and PMTMR. I do have 2 questions about this patch.
> > > > > 
> > > > > First of all, why you need to use both HPET and PMTMR? Can you just use one
> > > > > of those that are available. Secondly, is it possible to enable this
> > > > > time-skew diagnostic for a limit amount of time instead running
> > > > > indefinitely? The running of the clocksource watchdog itself will still
> > > > > consume a tiny amount of CPU cycles.
> > > > 
> > > > I could certainly do something so that only the first of HPET and PMTMR
> > > > is checked.  Could you give me a quick run-through of the advantages of
> > > > using only one?  I would need to explain that in the commit log.
> > > > 
> > > > Would it make sense to have a kernel boot variable giving the number of
> > > > minutes for which the watchdog was to run, with a default of zero
> > > > meaning "indefinitely"?
> > > 
> > > We've discussed about the "os noise", which customer may really care.
> > > IIUC, this patch intends to test if HPET/PMTIMER HW is broken, so how
> > > about making it run for a number of minutes the default behavior.   
> > 
> > It is also intended to determine if TSC is broken, with NTP drift rates
> > used to determine which timer is at fault.
> > 
> > OK, how about a Kconfig option for the number of minutes, set to whatever
> > you guys tell me?  (Three minutes?  Five minutes?  Something else?)
> > People wanting to run it continuously could then build their kernels
> > with that Kconfig option set to zero.
>  
> I don't have specific preference for 5 or 10 minutes, as long as it
> is a one time deal :) 
> 
> > > Also I've run the patch on a Alderlake system, with a fine acpi pm_timer
> > > and a fake broken pm_timer, and they both works without errors.
> > 
> > Thank you!  Did it correctly identify the fake broken pm_timer as being
> > broken?  If so, may I have your Tested-by?
> 
> On that Alderlake system, HPET will be disabled by kernel, and I
> manually increased the tsc frequency about 1/256 to make pm_timer
> look to have 1/256 deviation. And got dmesg like:
> 
> [    2.738554] clocksource: timekeeping watchdog on CPU3: Marking clocksource 'acpi_pm' as unstable because the skew is too large:
> [    2.738558] clocksource:                       'tsc' wd_nsec: 275956624 wd_now: 13aab38d0d wd_last: 1382c1144d mask: ffffffffffffffff
> [    2.738564] clocksource:                       'acpi_pm' cs_nsec: 277034651 cs_now: 731575 cs_last: 63f3cb mask: ffffff
> [    2.738568] clocksource:                       'tsc' (not 'acpi_pm') is current clocksource.
> 
> The deviation is indeed about 1/256. And pm_timer won't be shown in /sys/:
> 
> /sys/devices/system/clocksource/clocksource0/available_clocksource:tsc 
> /sys/devices/system/clocksource/clocksource0/current_clocksource:tsc
> 
> So feel free to add:
> 
> 	Tested-by: Feng Tang <feng.tang@xxxxxxxxx>

Thank you very much!  I will apply this on my next rebase.

							Thanx, Paul