HEAD is now at 3db7490... 0day head guard for 'devel-roam-i386-201404051825' git checkout 455c6fdbd219161bd09b1165f11699d6d73de11c ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linux-devel:devel-roam-i386-201404051825:455c6fdbd219161bd09b1165f11699d6d73de11c:bisect-linux6 2014-04-07-10:20:43 455c6fdbd219161bd09b1165f11699d6d73de11c compiling Queued build task to /kernel-tests/build-queue/i386-randconfig-r3-0405-455c6fdbd219161bd09b1165f11699d6d73de11c Check for kernel in /kernel/i386-randconfig-r3-0405/455c6fdbd219161bd09b1165f11699d6d73de11c waiting for completion of /kernel-tests/build-queue/i386-randconfig-r3-0405-455c6fdbd219161bd09b1165f11699d6d73de11c waiting for completion of /kernel-tests/build-queue/.i386-randconfig-r3-0405-455c6fdbd219161bd09b1165f11699d6d73de11c kernel: /kernel/i386-randconfig-r3-0405/455c6fdbd219161bd09b1165f11699d6d73de11c/vmlinuz-3.14.0 2014-04-07-10:24:43 detecting boot state 1 6 8 9.. 10.. 15 20 SUCCESS bisect: good commit 455c6fdbd219161bd09b1165f11699d6d73de11c git bisect start 3db74909a7c789e467b126d837f7e0cea4a5c165 455c6fdbd219161bd09b1165f11699d6d73de11c -- /c/kernel-tests/linear-bisect: ["-b", "3db74909a7c789e467b126d837f7e0cea4a5c165", "-g", "455c6fdbd219161bd09b1165f11699d6d73de11c", "/c/kernel-tests/bisect-test-boot-failure.sh", "/kernel-tests/linux6/obj-bisect"] Bisecting: 262 revisions left to test after this (roughly 9 steps) [1301fa9378b394d49525c4a3df8e61a64b11fd0d] Merge 'tchen/timer_debug3' into devel-roam-i386-201404051825 running /c/kernel-tests/bisect-test-boot-failure.sh /kernel-tests/linux6/obj-bisect ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linux-devel:devel-roam-i386-201404051825:1301fa9378b394d49525c4a3df8e61a64b11fd0d:bisect-linux6 2014-04-07-10:30:14 1301fa9378b394d49525c4a3df8e61a64b11fd0d compiling Queued build task to /kernel-tests/build-queue/i386-randconfig-r3-0405-1301fa9378b394d49525c4a3df8e61a64b11fd0d Check for kernel in /kernel/i386-randconfig-r3-0405/1301fa9378b394d49525c4a3df8e61a64b11fd0d waiting for completion of /kernel-tests/build-queue/i386-randconfig-r3-0405-1301fa9378b394d49525c4a3df8e61a64b11fd0d waiting for completion of /kernel-tests/build-queue/.i386-randconfig-r3-0405-1301fa9378b394d49525c4a3df8e61a64b11fd0d kernel: /kernel/i386-randconfig-r3-0405/1301fa9378b394d49525c4a3df8e61a64b11fd0d/vmlinuz-3.14.0-00090-g1301fa9 2014-04-07-10:34:14 detecting boot state . TEST FAILURE [ 5.639341] crc32_combine: 8373 self tests passed [ 5.654305] [ 5.655057] ========================================================= [ 5.656539] [ INFO: possible irq lock inversion dependency detected ] [ 5.657533] 3.14.0-00090-g1301fa9 #2 Not tainted [ 5.657533] --------------------------------------------------------- [ 5.657533] swapper/0/0 just changed the state of lock: [ 5.657533] (watchdog_lock){..-...}, at: [] clocksource_watchdog+0x21/0x450 [ 5.657533] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 5.657533] (cpu_add_remove_lock){+.+.+.} and interrupts could create inverse lock ordering between them. [ 5.657533] [ 5.657533] other info that might help us debug this: [ 5.657533] Possible interrupt unsafe locking scenario: [ 5.657533] [ 5.657533] CPU0 CPU1 [ 5.657533] ---- ---- [ 5.657533] lock(cpu_add_remove_lock); [ 5.657533] local_irq_disable(); [ 5.657533] lock(watchdog_lock); [ 5.657533] lock(cpu_add_remove_lock); [ 5.657533] [ 5.657533] lock(watchdog_lock); [ 5.657533] [ 5.657533] *** DEADLOCK *** [ 5.657533] [ 5.657533] 1 lock held by swapper/0/0: [ 5.657533] #0: ((&watchdog_timer)){+.-...}, at: [] call_timer_fn+0x0/0x170 [ 5.657533] [ 5.657533] the shortest dependencies between 2nd lock and 1st lock: [ 5.657533] -> (cpu_add_remove_lock){+.+.+.} ops: 46 { [ 5.657533] HARDIRQ-ON-W at: [ 5.657533] [] __lock_acquire+0xa05/0x1880 [ 5.657533] [] lock_acquire+0xd2/0x110 [ 5.657533] [] mutex_lock_nested+0x98/0x790 [ 5.657533] [] register_cpu_notifier+0x20/0x70 [ 5.657533] [] page_writeback_init+0x2e/0x56 [ 5.657533] [] start_kernel+0x9f0/0xa51 [ 5.657533] [] i386_start_kernel+0x19e/0x1af [ 5.657533] SOFTIRQ-ON-W at: [ 5.657533] [] __lock_acquire+0xa53/0x1880 [ 5.657533] [] lock_acquire+0xd2/0x110 [ 5.657533] [] mutex_lock_nested+0x98/0x790 [ 5.657533] [] register_cpu_notifier+0x20/0x70 [ 5.657533] [] page_writeback_init+0x2e/0x56 [ 5.657533] [] start_kernel+0x9f0/0xa51 [ 5.657533] [] i386_start_kernel+0x19e/0x1af [ 5.657533] RECLAIM_FS-ON-W at: [ 5.657533] [] mark_held_locks+0xc0/0x1e0 [ 5.657533] [] lockdep_trace_alloc+0x126/0x230 [ 5.657533] [] kmem_cache_alloc+0x3f/0x260 [ 5.657533] [] __smpboot_create_thread+0x55/0x190 [ 5.657533] [] smpboot_create_threads+0x7b/0xc0 [ 5.657533] [] cpu_up+0x12a/0x520 [ 5.657533] [] smp_init+0x18b/0x1f2 [ 5.657533] [] kernel_init_freeable+0xec/0x3f4 [ 5.657533] [] kernel_init+0x19/0x210 [ 5.657533] [] ret_from_kernel_thread+0x1b/0x28 [ 5.657533] INITIAL USE at: [ 5.657533] [] __lock_acquire+0x47c/0x1880 [ 5.657533] [] lock_acquire+0xd2/0x110 [ 5.657533] [] mutex_lock_nested+0x98/0x790 [ 5.657533] [] register_cpu_notifier+0x20/0x70 [ 5.657533] [] kvm_guest_init+0x1bc/0x24d [ 5.657533] [] setup_arch+0x1835/0x18e2 [ 5.657533] [] start_kernel+0x145/0xa51 [ 5.657533] [] i386_start_kernel+0x19e/0x1af [ 5.657533] } [ 5.657533] ... key at: [] cpu_add_remove_lock+0x40/0xf0 [ 5.657533] ... acquired at: [ 5.657533] [] __lock_acquire+0x66b/0x1880 [ 5.657533] [] lock_acquire+0xd2/0x110 [ 5.657533] [] mutex_lock_nested+0x98/0x790 [ 5.657533] [] register_cpu_notifier+0x20/0x70 [ 5.657533] [] clocksource_enqueue_watchdog+0xde/0x230 [ 5.657533] [] __clocksource_register_scale+0xf1/0x140 [ 5.657533] [] tsc_refine_calibration_work+0x1bd/0x2a0 [ 5.657533] [] process_one_work+0x265/0x750 [ 5.657533] [] worker_thread+0x28a/0x6f0 [ 5.657533] [] kthread+0x110/0x150 [ 5.657533] [] ret_from_kernel_thread+0x1b/0x28 [ 5.657533] [ 5.657533] -> (watchdog_lock){..-...} ops: 8 { [ 5.657533] IN-SOFTIRQ-W at: [ 5.657533] [] __lock_acquire+0xbd0/0x1880 [ 5.657533] [] lock_acquire+0xd2/0x110 [ 5.657533] [] _raw_spin_lock+0x67/0x110 [ 5.657533] [] clocksource_watchdog+0x21/0x450 [ 5.657533] [] call_timer_fn+0xa7/0x170 [ 5.657533] [] run_timer_softirq+0x206/0x380 [ 5.657533] [] __do_softirq+0x14c/0x380 [ 5.657533] INITIAL USE at: [ 5.657533] [] __lock_acquire+0x47c/0x1880 [ 5.657533] [] lock_acquire+0xd2/0x110 [ 5.657533] [] _raw_spin_lock_irqsave+0xa5/0x150 [ 5.657533] [] clocksource_enqueue_watchdog+0x22/0x230 [ 5.657533] [] __clocksource_register_scale+0xf1/0x140 [ 5.657533] [] kvmclock_init+0x2cf/0x35b [ 5.657533] [] setup_arch+0x16fd/0x18e2 [ 5.657533] [] start_kernel+0x145/0xa51 [ 5.657533] [] i386_start_kernel+0x19e/0x1af [ 5.657533] } [ 5.657533] ... key at: [] watchdog_lock+0x10/0x40 [ 5.657533] ... acquired at: [ 5.657533] [] check_usage_forwards+0xb9/0x190 [ 5.657533] [] mark_lock+0x69b/0xc10 [ 5.657533] [] __lock_acquire+0xbd0/0x1880 [ 5.657533] [] lock_acquire+0xd2/0x110 [ 5.657533] [] _raw_spin_lock+0x67/0x110 [ 5.842982] crc32c_combine: 8373 self tests passed [ 5.657533] rbtree testing [ 5.657533] [] clocksource_watchdog+0x21/0x450 [ 5.657533] [] call_timer_fn+0xa7/0x170 [ 5.657533] [] run_timer_softirq+0x206/0x380 [ 5.657533] [] __do_softirq+0x14c/0x380 [ 5.657533] [ 5.657533] [ 5.657533] stack backtrace: [ 5.657533] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.0-00090-g1301fa9 #2 [ 5.657533] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 5.657533] c3b000e0 cb011d7c c2891be7 cb011dd8 cb011db8 c10eff09 c2ea735d c2ea76ab [ 5.657533] 00000000 cb011dd8 00000006 c10f01b0 cb011db8 c10eddd3 00000000 00000000 [ 5.657533] c3084c2c c30846e0 c10f01b0 cb011e04 c10f0269 c3084c2c 00000001 c2ea76ab [ 5.657533] Call Trace: [ 5.657533] [] dump_stack+0xb8/0x108 [ 5.657533] [] print_irq_inversion_bug+0x389/0x4a0 [ 5.657533] [] ? check_usage_backwards+0x190/0x190 [ 5.657533] [] ? find_usage_forwards+0x53/0xd0 [ 5.657533] [] ? check_usage_backwards+0x190/0x190 [ 5.657533] [] check_usage_forwards+0xb9/0x190 [ 5.657533] [] mark_lock+0x69b/0xc10 [ 5.657533] [] ? print_context_stack+0x5d/0x160 [ 5.657533] [] __lock_acquire+0xbd0/0x1880 [ 5.657533] [] lock_acquire+0xd2/0x110 [ 5.657533] [] ? clocksource_watchdog+0x21/0x450 [ 5.657533] [] _raw_spin_lock+0x67/0x110 [ 5.657533] [] ? clocksource_watchdog+0x21/0x450 [ 5.657533] [] ? clocksource_cpu_notify+0xd0/0xd0 [ 5.657533] [] clocksource_watchdog+0x21/0x450 [ 5.657533] [] ? process_timeout+0x30/0x30 [ 5.657533] [] ? clocksource_cpu_notify+0xd0/0xd0 [ 5.657533] [] call_timer_fn+0xa7/0x170 [ 5.657533] [] ? process_timeout+0x30/0x30 [ 5.657533] [] run_timer_softirq+0x206/0x380 [ 5.657533] [] ? clocksource_cpu_notify+0xd0/0xd0 [ 5.657533] [] __do_softirq+0x14c/0x380 [ 5.657533] [] ? _local_bh_enable+0xa0/0xa0 [ 5.657533] [] ? irq_exit+0xcb/0xe0 [ 5.657533] [] ? smp_apic_timer_interrupt+0x6b/0x90 [ 5.657533] [] ? apic_timer_interrupt+0x32/0x38 [ 5.657533] [] ? native_safe_halt+0x13/0x30 [ 5.657533] [] ? default_idle+0x29/0x40 [ 5.657533] [] ? arch_cpu_idle+0x6c/0x80 [ 5.657533] [] ? cpu_startup_entry+0x21e/0x390 [ 5.657533] [] ? rest_init+0x15c/0x170 [ 5.657533] [] ? csum_partial_copy_generic+0xf8/0xf8 [ 5.657533] [] ? start_kernel+0xa29/0xa51 [ 5.657533] [] ? repair_env_string+0xd3/0xd3 [ 5.657533] [] ? i386_start_kernel+0x19e/0x1af [ 6.263767] -> 11332 cycles [ 6.321410] augmented rbtree testing -> 24206 cycles [ 7.317286] Console: switching consoles 13-16 to MDA-2 /kernel/i386-randconfig-r3-0405/1301fa9378b394d49525c4a3df8e61a64b11fd0d/dmesg-quantal-inn-46:20140407103431:i386-randconfig-r3-0405:: 0:1:1 all_good:bad:all_bad boots 2014-04-07 10:34:46 REPEAT COUNT: 20 # /kernel-tests/linux6/obj-bisect/.repeat Bisecting: 90 revisions left to test after this (roughly 7 steps) [9fde1e2345f8ac522a84c77156d7e2b8ec4e8d94] 0day base guard for 'devel-roam-i386-201404051825' running /c/kernel-tests/bisect-test-boot-failure.sh /kernel-tests/linux6/obj-bisect ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linux-devel:devel-roam-i386-201404051825:9fde1e2345f8ac522a84c77156d7e2b8ec4e8d94:bisect-linux6 2014-04-07-10:34:47 9fde1e2345f8ac522a84c77156d7e2b8ec4e8d94 compiling Queued build task to /kernel-tests/build-queue/i386-randconfig-r3-0405-9fde1e2345f8ac522a84c77156d7e2b8ec4e8d94 Check for kernel in /kernel/i386-randconfig-r3-0405/9fde1e2345f8ac522a84c77156d7e2b8ec4e8d94 waiting for completion of /kernel-tests/build-queue/i386-randconfig-r3-0405-9fde1e2345f8ac522a84c77156d7e2b8ec4e8d94 waiting for completion of /kernel-tests/build-queue/.i386-randconfig-r3-0405-9fde1e2345f8ac522a84c77156d7e2b8ec4e8d94 kernel: /kernel/i386-randconfig-r3-0405/9fde1e2345f8ac522a84c77156d7e2b8ec4e8d94/vmlinuz-3.14.0-00001-g9fde1e2 2014-04-07-10:38:47 detecting boot state ................ 1.... 2. 3 4 9 17 19....... 20 SUCCESS linear-bisect: bad branch may be tchen/timer_debug3 linear-bisect: handle over to git bisect linear-bisect: git bisect start 1301fa9378b394d49525c4a3df8e61a64b11fd0d 9fde1e2345f8ac522a84c77156d7e2b8ec4e8d94 -- Previous HEAD position was 9fde1e2... 0day base guard for 'devel-roam-i386-201404051825' HEAD is now at fc6d0b0... Merge branch 'lockref' (locked reference counts) Bisecting: 44 revisions left to test after this (roughly 6 steps) [b712c8dae05931a76b6c17a4254f403798e6caef] x86: hpet: Use proper destructor for delayed work linear-bisect: git bisect run /c/kernel-tests/bisect-test-boot-failure.sh /kernel-tests/linux6/obj-bisect running /c/kernel-tests/bisect-test-boot-failure.sh /kernel-tests/linux6/obj-bisect ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linux-devel:devel-roam-i386-201404051825:b712c8dae05931a76b6c17a4254f403798e6caef:bisect-linux6 2014-04-07-10:56:58 b712c8dae05931a76b6c17a4254f403798e6caef compiling Queued build task to /kernel-tests/build-queue/i386-randconfig-r3-0405-b712c8dae05931a76b6c17a4254f403798e6caef Check for kernel in /kernel/i386-randconfig-r3-0405/b712c8dae05931a76b6c17a4254f403798e6caef waiting for completion of /kernel-tests/build-queue/i386-randconfig-r3-0405-b712c8dae05931a76b6c17a4254f403798e6caef waiting for completion of /kernel-tests/build-queue/.i386-randconfig-r3-0405-b712c8dae05931a76b6c17a4254f403798e6caef kernel: /kernel/i386-randconfig-r3-0405/b712c8dae05931a76b6c17a4254f403798e6caef/vmlinuz-3.14.0-rc1-00044-gb712c8d 2014-04-07-10:59:58 detecting boot state . 3 6 11 13 14 15 16 17.. 19 20 SUCCESS Bisecting: 22 revisions left to test after this (roughly 5 steps) [5b108727a416a790e9ef87eecd370f67773838d3] hrtimer: remove 'base' parameter from hrtimer_{enqueue_}reprogram() running /c/kernel-tests/bisect-test-boot-failure.sh /kernel-tests/linux6/obj-bisect ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linux-devel:devel-roam-i386-201404051825:5b108727a416a790e9ef87eecd370f67773838d3:bisect-linux6 2014-04-07-11:06:29 5b108727a416a790e9ef87eecd370f67773838d3 compiling Queued build task to /kernel-tests/build-queue/i386-randconfig-r3-0405-5b108727a416a790e9ef87eecd370f67773838d3 Check for kernel in /kernel/i386-randconfig-r3-0405/5b108727a416a790e9ef87eecd370f67773838d3 waiting for completion of /kernel-tests/build-queue/i386-randconfig-r3-0405-5b108727a416a790e9ef87eecd370f67773838d3 waiting for completion of /kernel-tests/build-queue/.i386-randconfig-r3-0405-5b108727a416a790e9ef87eecd370f67773838d3 kernel: /kernel/i386-randconfig-r3-0405/5b108727a416a790e9ef87eecd370f67773838d3/vmlinuz-3.14.0-rc1-00066-g5b10872 2014-04-07-11:10:30 detecting boot state 1 4 7 14 17 18 19 20 SUCCESS Bisecting: 11 revisions left to test after this (roughly 4 steps) [af57797fadd8c265ce34605153b75c417d3df384] hrtimer: remove {} around a single liner 'for' loop in migrate_hrtimers() running /c/kernel-tests/bisect-test-boot-failure.sh /kernel-tests/linux6/obj-bisect ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linux-devel:devel-roam-i386-201404051825:af57797fadd8c265ce34605153b75c417d3df384:bisect-linux6 2014-04-07-11:14:31 af57797fadd8c265ce34605153b75c417d3df384 compiling Queued build task to /kernel-tests/build-queue/i386-randconfig-r3-0405-af57797fadd8c265ce34605153b75c417d3df384 Check for kernel in /kernel/i386-randconfig-r3-0405/af57797fadd8c265ce34605153b75c417d3df384 waiting for completion of /kernel-tests/build-queue/i386-randconfig-r3-0405-af57797fadd8c265ce34605153b75c417d3df384 waiting for completion of /kernel-tests/build-queue/.i386-randconfig-r3-0405-af57797fadd8c265ce34605153b75c417d3df384 kernel: /kernel/i386-randconfig-r3-0405/af57797fadd8c265ce34605153b75c417d3df384/vmlinuz-3.14.0-rc1-00077-gaf57797 2014-04-07-11:18:31 detecting boot state 6 17 20 SUCCESS Bisecting: 5 revisions left to test after this (roughly 3 steps) [9ffd0897bd721352543c84041daf5dc07e60f559] hrtimer: take lock only once for a cpu_base in hrtimer_run_queues() running /c/kernel-tests/bisect-test-boot-failure.sh /kernel-tests/linux6/obj-bisect ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linux-devel:devel-roam-i386-201404051825:9ffd0897bd721352543c84041daf5dc07e60f559:bisect-linux6 2014-04-07-11:20:02 9ffd0897bd721352543c84041daf5dc07e60f559 compiling Queued build task to /kernel-tests/build-queue/i386-randconfig-r3-0405-9ffd0897bd721352543c84041daf5dc07e60f559 Check for kernel in /kernel/i386-randconfig-r3-0405/9ffd0897bd721352543c84041daf5dc07e60f559 waiting for completion of /kernel-tests/build-queue/i386-randconfig-r3-0405-9ffd0897bd721352543c84041daf5dc07e60f559 waiting for completion of /kernel-tests/build-queue/.i386-randconfig-r3-0405-9ffd0897bd721352543c84041daf5dc07e60f559 kernel: /kernel/i386-randconfig-r3-0405/9ffd0897bd721352543c84041daf5dc07e60f559/vmlinuz-3.14.0-rc1-00083-g9ffd089 2014-04-07-11:23:02 detecting boot state .. 1 6 10 14 18 19 20 SUCCESS Bisecting: 2 revisions left to test after this (roughly 2 steps) [5a8530b7c3b1889861949d4e52b2da82d0aff242] timer: track pinned timers with TIMER_PINNED flag running /c/kernel-tests/bisect-test-boot-failure.sh /kernel-tests/linux6/obj-bisect ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linux-devel:devel-roam-i386-201404051825:5a8530b7c3b1889861949d4e52b2da82d0aff242:bisect-linux6 2014-04-07-11:27:33 5a8530b7c3b1889861949d4e52b2da82d0aff242 compiling Queued build task to /kernel-tests/build-queue/i386-randconfig-r3-0405-5a8530b7c3b1889861949d4e52b2da82d0aff242 Check for kernel in /kernel/i386-randconfig-r3-0405/5a8530b7c3b1889861949d4e52b2da82d0aff242 waiting for completion of /kernel-tests/build-queue/i386-randconfig-r3-0405-5a8530b7c3b1889861949d4e52b2da82d0aff242 waiting for completion of /kernel-tests/build-queue/.i386-randconfig-r3-0405-5a8530b7c3b1889861949d4e52b2da82d0aff242 kernel: /kernel/i386-randconfig-r3-0405/5a8530b7c3b1889861949d4e52b2da82d0aff242/vmlinuz-3.14.0-rc1-00086-g5a8530b 2014-04-07-11:33:33 detecting boot state 3 7 8 10 13 16. 17 18 20 SUCCESS Bisecting: 0 revisions left to test after this (roughly 1 step) [90585a71ecf3285bc73d54e37080c3bb21583594] clocksource: register cpu notifier to remove timer from dying CPU running /c/kernel-tests/bisect-test-boot-failure.sh /kernel-tests/linux6/obj-bisect ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linux-devel:devel-roam-i386-201404051825:90585a71ecf3285bc73d54e37080c3bb21583594:bisect-linux6 2014-04-07-11:38:35 90585a71ecf3285bc73d54e37080c3bb21583594 compiling Queued build task to /kernel-tests/build-queue/i386-randconfig-r3-0405-90585a71ecf3285bc73d54e37080c3bb21583594 Check for kernel in /kernel/i386-randconfig-r3-0405/90585a71ecf3285bc73d54e37080c3bb21583594 waiting for completion of /kernel-tests/build-queue/i386-randconfig-r3-0405-90585a71ecf3285bc73d54e37080c3bb21583594 waiting for completion of /kernel-tests/build-queue/.i386-randconfig-r3-0405-90585a71ecf3285bc73d54e37080c3bb21583594 kernel: /kernel/i386-randconfig-r3-0405/90585a71ecf3285bc73d54e37080c3bb21583594/vmlinuz-3.14.0-rc1-00088-g90585a7 2014-04-07-11:42:35 detecting boot state ... TEST FAILURE [ 3.650334] crc32c_combine: 8373 self tests passed [ 3.651169] rbtree testing [ 3.787010] ========================================================= [ 3.787976] [ INFO: possible irq lock inversion dependency detected ] [ 3.788955] 3.14.0-rc1-00088-g90585a7 #2 Not tainted [ 3.789772] --------------------------------------------------------- [ 3.789969] swapper/0/1 just changed the state of lock: [ 3.789969] (watchdog_lock){..-...}, at: [] clocksource_watchdog+0x21/0x450 [ 3.789969] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 3.789969] (cpu_add_remove_lock){+.+.+.} and interrupts could create inverse lock ordering between them. [ 3.789969] [ 3.789969] other info that might help us debug this: [ 3.789969] Possible interrupt unsafe locking scenario: [ 3.789969] [ 3.789969] CPU0 CPU1 [ 3.789969] ---- ---- [ 3.789969] lock(cpu_add_remove_lock); [ 3.789969] local_irq_disable(); [ 3.789969] lock(watchdog_lock); [ 3.789969] lock(cpu_add_remove_lock); [ 3.789969] [ 3.789969] lock(watchdog_lock); [ 3.789969] [ 3.789969] *** DEADLOCK *** [ 3.789969] [ 3.789969] 1 lock held by swapper/0/1: [ 3.789969] #0: ((&watchdog_timer)){+.-...}, at: [] call_timer_fn+0x0/0x170 [ 3.789969] [ 3.789969] the shortest dependencies between 2nd lock and 1st lock: [ 3.789969] -> (cpu_add_remove_lock){+.+.+.} ops: 16 { [ 3.789969] HARDIRQ-ON-W at: [ 3.789969] [] __lock_acquire+0xa05/0x1880 [ 3.789969] [] lock_acquire+0xd2/0x110 [ 3.789969] [] mutex_lock_nested+0x98/0x790 [ 3.789969] [] register_cpu_notifier+0x20/0x70 [ 3.789969] [] page_writeback_init+0x2e/0x56 [ 3.789969] [] start_kernel+0x9f0/0xa51 [ 3.789969] [] i386_start_kernel+0x19e/0x1af [ 3.789969] SOFTIRQ-ON-W at: [ 3.789969] [] __lock_acquire+0xa53/0x1880 [ 3.789969] [] lock_acquire+0xd2/0x110 [ 3.789969] [] mutex_lock_nested+0x98/0x790 [ 3.789969] [] register_cpu_notifier+0x20/0x70 [ 3.789969] [] page_writeback_init+0x2e/0x56 [ 3.789969] [] start_kernel+0x9f0/0xa51 [ 3.789969] [] i386_start_kernel+0x19e/0x1af [ 3.789969] RECLAIM_FS-ON-W at: [ 3.789969] [] mark_held_locks+0xc0/0x1e0 [ 3.789969] [] lockdep_trace_alloc+0x126/0x230 [ 3.789969] [] kmem_cache_alloc+0x3f/0x260 [ 3.789969] [] __smpboot_create_thread+0x55/0x190 [ 3.789969] [] smpboot_create_threads+0x7b/0xc0 [ 3.789969] [] cpu_up+0x12a/0x520 [ 3.789969] [] smp_init+0x18b/0x1f2 [ 3.789969] [] kernel_init_freeable+0xec/0x3f4 [ 3.789969] [] kernel_init+0x19/0x230 [ 3.789969] [] ret_from_kernel_thread+0x1b/0x28 [ 3.789969] INITIAL USE at: [ 3.789969] [] __lock_acquire+0x47c/0x1880 [ 3.789969] [] lock_acquire+0xd2/0x110 [ 3.789969] [] mutex_lock_nested+0x98/0x790 [ 3.789969] [] register_cpu_notifier+0x20/0x70 [ 3.789969] [] kvm_guest_init+0x1bc/0x24d [ 3.789969] [] setup_arch+0x1835/0x18e2 [ 3.789969] [] start_kernel+0x145/0xa51 [ 3.789969] [] i386_start_kernel+0x19e/0x1af [ 3.789969] } [ 3.789969] ... key at: [] cpu_add_remove_lock+0x40/0xf0 [ 3.789969] ... acquired at: [ 3.789969] [] __lock_acquire+0x66b/0x1880 [ 3.789969] [] lock_acquire+0xd2/0x110 [ 3.789969] [] mutex_lock_nested+0x98/0x790 [ 3.789969] [] register_cpu_notifier+0x20/0x70 [ 3.789969] [] clocksource_enqueue_watchdog+0xde/0x230 [ 3.789969] [] __clocksource_register_scale+0xf1/0x140 [ 3.789969] [] tsc_refine_calibration_work+0x1bd/0x2a0 [ 3.789969] [] process_one_work+0x265/0x740 [ 3.789969] [] worker_thread+0x28a/0x6f0 [ 3.789969] [] kthread+0x110/0x150 [ 3.789969] [] ret_from_kernel_thread+0x1b/0x28 [ 3.789969] [ 3.789969] -> (watchdog_lock){..-...} ops: 8 { [ 3.789969] IN-SOFTIRQ-W at: [ 3.789969] [] __lock_acquire+0xbd0/0x1880 [ 3.789969] [] lock_acquire+0xd2/0x110 [ 3.789969] [] _raw_spin_lock+0x67/0x110 [ 3.789969] [] clocksource_watchdog+0x21/0x450 [ 3.789969] [] call_timer_fn+0xa7/0x170 [ 3.789969] [] run_timer_softirq+0x206/0x380 [ 3.789969] [] __do_softirq+0x14c/0x380 [ 3.789969] INITIAL USE at: [ 3.789969] [] __lock_acquire+0x47c/0x1880 [ 3.789969] [] lock_acquire+0xd2/0x110 [ 3.789969] [] _raw_spin_lock_irqsave+0xa5/0x150 [ 3.789969] [] clocksource_enqueue_watchdog+0x22/0x230 [ 3.789969] [] __clocksource_register_scale+0xf1/0x140 [ 3.789969] [] kvmclock_init+0x2cf/0x35b [ 3.789969] [] setup_arch+0x16fd/0x18e2 [ 3.789969] [] start_kernel+0x145/0xa51 [ 3.789969] [] i386_start_kernel+0x19e/0x1af [ 3.789969] } [ 3.789969] ... key at: [] watchdog_lock+0x10/0x40 [ 3.789969] ... acquired at: [ 3.789969] [] check_usage_forwards+0xb9/0x190 [ 3.789969] [] mark_lock+0x69b/0xc10 [ 3.789969] [] __lock_acquire+0xbd0/0x1880 [ 3.789969] [] lock_acquire+0xd2/0x110 [ 3.789969] [] _raw_spin_lock+0x67/0x110 [ 3.789969] [] clocksource_watchdog+0x21/0x450 [ 3.789969] [] call_timer_fn+0xa7/0x170 [ 3.789969] [] run_timer_softirq+0x206/0x380 [ 3.789969] [] __do_softirq+0x14c/0x380 [ 3.789969] [ 3.789969] [ 3.789969] stack backtrace: [ 3.789969] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.14.0-rc1-00088-g90585a7 #2 [ 3.789969] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 3.789969] c3adc020 d180dd7c c287f340 d180ddd8 d180ddb8 c10ef2b9 c2e845e0 c2e8492e [ 3.789969] 00000001 d180ddd8 00000006 c10ef560 d180ddb8 c10ed183 00000000 00000000 [ 3.789969] d186054c d1860000 c10ef560 d180de04 c10ef619 d186054c 00000001 c2e8492e [ 3.789969] Call Trace: [ 3.789969] [] dump_stack+0xb8/0x108 [ 3.789969] [] print_irq_inversion_bug+0x389/0x4a0 [ 3.789969] [] ? check_usage_backwards+0x190/0x190 [ 3.789969] [] ? find_usage_forwards+0x53/0xd0 [ 3.789969] [] ? check_usage_backwards+0x190/0x190 [ 3.789969] [] check_usage_forwards+0xb9/0x190 [ 3.789969] [] mark_lock+0x69b/0xc10 [ 3.789969] [] ? print_context_stack+0x5d/0x160 [ 3.789969] [] __lock_acquire+0xbd0/0x1880 [ 3.789969] [] lock_acquire+0xd2/0x110 [ 3.789969] [] ? clocksource_watchdog+0x21/0x450 [ 3.789969] [] _raw_spin_lock+0x67/0x110 [ 3.789969] [] ? clocksource_watchdog+0x21/0x450 [ 3.789969] [] ? clocksource_cpu_notify+0xd0/0xd0 [ 3.789969] [] clocksource_watchdog+0x21/0x450 [ 3.789969] [] ? process_timeout+0x30/0x30 [ 3.789969] [] ? clocksource_cpu_notify+0xd0/0xd0 [ 3.789969] [] call_timer_fn+0xa7/0x170 [ 3.789969] [] ? process_timeout+0x30/0x30 [ 3.789969] [] run_timer_softirq+0x206/0x380 [ 3.789969] [] ? clocksource_cpu_notify+0xd0/0xd0 [ 3.789969] [] __do_softirq+0x14c/0x380 [ 3.789969] [] ? _local_bh_enable+0xa0/0xa0 [ 3.789969] [] ? irq_exit+0xcb/0xe0 [ 3.789969] [] ? smp_apic_timer_interrupt+0x6b/0x90 [ 3.789969] [] ? apic_timer_interrupt+0x32/0x38 [ 3.789969] [] ? check+0x7e/0x52d [ 3.789969] [] ? rbtree_test_init+0xa3/0x45f [ 3.789969] [] ? audit_register_class+0x29/0x10a [ 3.789969] [] ? test_atomic64+0xdc0/0xdc0 [ 3.789969] [] ? do_one_initcall+0xf6/0x222 [ 3.789969] [] ? parameq+0x21/0x100 [ 3.789969] [] ? repair_env_string+0x32/0xd3 [ 3.789969] [] ? parse_args+0x2eb/0x5f0 [ 3.789969] [] ? kernel_init_freeable+0x26c/0x3f4 [ 3.789969] [] ? do_early_param+0x131/0x131 [ 3.789969] [] ? kernel_init+0x19/0x230 [ 3.789969] [] ? ret_from_kernel_thread+0x1b/0x28 [ 3.789969] [] ? rest_init+0x170/0x170 [ 4.202425] -> 13739 cycles [ 4.252106] augmented rbtree testing -> 15602 cycles [ 4.943124] Console: switching consoles 13-16 to MDA-2 /kernel/i386-randconfig-r3-0405/90585a71ecf3285bc73d54e37080c3bb21583594/dmesg-quantal-xian-6:20140407114329:i386-randconfig-r3-0405:3.14.0-rc1-00088-g90585a7:2 /kernel/i386-randconfig-r3-0405/90585a71ecf3285bc73d54e37080c3bb21583594/dmesg-quantal-lkp-ib04-21:20140407114350:i386-randconfig-r3-0405:3.14.0-rc1-00088-g90585a7:2 /kernel/i386-randconfig-r3-0405/90585a71ecf3285bc73d54e37080c3bb21583594/dmesg-quantal-lkp-ib04-79:20140407114400:i386-randconfig-r3-0405:3.14.0-rc1-00088-g90585a7:2 0:3:3 all_good:bad:all_bad boots 2014-04-07 11:44:08 REPEAT COUNT: 20 # /kernel-tests/linux6/obj-bisect/.repeat Bisecting: 0 revisions left to test after this (roughly 0 steps) [6378cb51af5f4743db0dcb3cbcf862eac5908754] timer: don't migrate pinned timers running /c/kernel-tests/bisect-test-boot-failure.sh /kernel-tests/linux6/obj-bisect ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linux-devel:devel-roam-i386-201404051825:6378cb51af5f4743db0dcb3cbcf862eac5908754:bisect-linux6 2014-04-07-11:44:09 6378cb51af5f4743db0dcb3cbcf862eac5908754 compiling Queued build task to /kernel-tests/build-queue/i386-randconfig-r3-0405-6378cb51af5f4743db0dcb3cbcf862eac5908754 Check for kernel in /kernel/i386-randconfig-r3-0405/6378cb51af5f4743db0dcb3cbcf862eac5908754 waiting for completion of /kernel-tests/build-queue/i386-randconfig-r3-0405-6378cb51af5f4743db0dcb3cbcf862eac5908754 waiting for completion of /kernel-tests/build-queue/.i386-randconfig-r3-0405-6378cb51af5f4743db0dcb3cbcf862eac5908754 kernel: /kernel/i386-randconfig-r3-0405/6378cb51af5f4743db0dcb3cbcf862eac5908754/vmlinuz-3.14.0-rc1-00087-g6378cb5 2014-04-07-11:48:09 detecting boot state 2 14 16 20 SUCCESS 90585a71ecf3285bc73d54e37080c3bb21583594 is the first bad commit commit 90585a71ecf3285bc73d54e37080c3bb21583594 Author: Viresh Kumar Date: Sat Apr 5 11:43:25 2014 +0530 clocksource: register cpu notifier to remove timer from dying CPU clocksource core is using add_timer_on() to run clocksource_watchdog() on all CPUs one by one. But when a core is brought down, clocksource core doesn't remove this timer from the dying CPU. And in this case timer core gives this (Gives this only with unmerged code, anyway in the current code as well timer core is migrating a pinned timer to other CPUs, which is also wrong: http://www.gossamer-threads.com/lists/linux/kernel/1898117) migrate_timer_list: can't migrate pinned timer: ffffffff81f06a60, timer->function: ffffffff810d7010,deactivating it Modules linked in: CPU: 0 PID: 1932 Comm: 01-cpu-hotplug Not tainted 3.14.0-rc1-00088-gab3c4fd #4 Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 0000000000000009 ffff88001d407c38 ffffffff817237bd ffff88001d407c80 ffff88001d407c70 ffffffff8106a1dd 0000000000000010 ffffffff81f06a60 ffff88001e04d040 ffffffff81e3d4c0 ffff88001e04d030 ffff88001d407cd0 Call Trace: [] dump_stack+0x4d/0x66 [] warn_slowpath_common+0x7d/0xa0 [] warn_slowpath_fmt+0x4c/0x50 [] ? __internal_add_timer+0x113/0x130 [] ? clocksource_watchdog_kthread+0x40/0x40 [] migrate_timer_list+0xdb/0xf0 [] timer_cpu_notify+0xfc/0x1f0 [] notifier_call_chain+0x4c/0x70 [] __raw_notifier_call_chain+0xe/0x10 [] cpu_notify+0x23/0x50 [] cpu_notify_nofail+0xe/0x20 [] _cpu_down+0x1ad/0x2e0 [] cpu_down+0x34/0x50 [] cpu_subsys_offline+0x14/0x20 [] device_offline+0x95/0xc0 [] online_store+0x40/0x90 [] dev_attr_store+0x18/0x30 [] sysfs_kf_write+0x3d/0x50 This patch tries to fix this by registering cpu notifiers from clocksource core, only when we start clocksource-watchdog. And if on the CPU_DEAD notification it is found that dying CPU was the CPU on which this timer is queued on, then it is removed from that CPU and queued to next CPU. Reported-by: Jet Chen Reported-by: Fengguang Wu Signed-off-by: Viresh Kumar :040000 040000 a83bcee9913417082d3e544cfc69ae4e264d2deb 3d05adae5067ea6bffbcd927696e740274958741 M kernel bisect run success HEAD is now at 6378cb5... timer: don't migrate pinned timers ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linux-devel:devel-roam-i386-201404051825:6378cb51af5f4743db0dcb3cbcf862eac5908754:bisect-linux6 2014-04-07-11:50:10 6378cb51af5f4743db0dcb3cbcf862eac5908754 reuse /kernel/i386-randconfig-r3-0405/6378cb51af5f4743db0dcb3cbcf862eac5908754/vmlinuz-3.14.0-rc1-00087-g6378cb5 2014-04-07-11:50:10 detecting boot state .. 3 25 43 46 58 59 60 SUCCESS Previous HEAD position was 6378cb5... timer: don't migrate pinned timers HEAD is now at 3db7490... 0day head guard for 'devel-roam-i386-201404051825' ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linux-devel:devel-roam-i386-201404051825:3db74909a7c789e467b126d837f7e0cea4a5c165:bisect-linux6 TEST FAILURE [ 48.818475] Switched to clocksource tsc [ 49.316248] [ 49.316786] ========================================================= [ 49.317080] [ INFO: possible irq lock inversion dependency detected ] [ 49.317623] 3.14.0-00262-g3db7490 #69 Not tainted [ 49.317892] --------------------------------------------------------- [ 49.318192] swapper/0/1 just changed the state of lock: [ 49.318624] (watchdog_lock){..-...}, at: [] clocksource_watchdog+0x21/0x450 [ 49.318961] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 49.318961] (cpu_add_remove_lock){+.+.+.} and interrupts could create inverse lock ordering between them. [ 49.318961] [ 49.318961] other info that might help us debug this: [ 49.318961] Possible interrupt unsafe locking scenario: [ 49.318961] [ 49.318961] CPU0 CPU1 [ 49.318961] ---- ---- [ 49.318961] lock(cpu_add_remove_lock); [ 49.318961] local_irq_disable(); [ 49.318961] lock(watchdog_lock); [ 49.318961] lock(cpu_add_remove_lock); [ 49.318961] [ 49.318961] lock(watchdog_lock); [ 49.318961] [ 49.318961] *** DEADLOCK *** [ 49.318961] [ 49.318961] 2 locks held by swapper/0/1: [ 49.318961] #0: (slab_mutex){+.+.+.}, at: [] slab_sysfs_init+0x21/0x1be [ 49.318961] #1: ((&watchdog_timer)){+.-...}, at: [] call_timer_fn+0x0/0x170 [ 49.318961] [ 49.318961] the shortest dependencies between 2nd lock and 1st lock: [ 49.318961] -> (cpu_add_remove_lock){+.+.+.} ops: 16 { [ 49.318961] HARDIRQ-ON-W at: [ 49.318961] [] __lock_acquire+0xa05/0x1880 [ 49.318961] [] lock_acquire+0xd2/0x110 [ 49.318961] [] mutex_lock_nested+0x98/0x790 [ 49.318961] [] register_cpu_notifier+0x20/0x70 [ 49.318961] [] page_writeback_init+0x2e/0x56 [ 49.318961] [] start_kernel+0x9f0/0xa51 [ 49.318961] [] i386_start_kernel+0x19e/0x1af [ 49.318961] SOFTIRQ-ON-W at: [ 49.318961] [] __lock_acquire+0xa53/0x1880 [ 49.318961] [] lock_acquire+0xd2/0x110 [ 49.318961] [] mutex_lock_nested+0x98/0x790 [ 49.318961] [] register_cpu_notifier+0x20/0x70 [ 49.318961] [] page_writeback_init+0x2e/0x56 [ 49.318961] [] start_kernel+0x9f0/0xa51 [ 49.318961] [] i386_start_kernel+0x19e/0x1af [ 49.318961] RECLAIM_FS-ON-W at: [ 49.318961] [] mark_held_locks+0xc0/0x1e0 [ 49.318961] [] lockdep_trace_alloc+0x126/0x230 [ 49.318961] [] kmem_cache_alloc+0x3f/0x260 [ 49.318961] [] __smpboot_create_thread+0x55/0x190 [ 49.318961] [] smpboot_create_threads+0x7b/0xc0 [ 49.318961] [] cpu_up+0x12a/0x520 [ 49.318961] [] smp_init+0x18b/0x1f2 [ 49.318961] [] kernel_init_freeable+0xec/0x3f4 [ 49.318961] [] kernel_init+0x19/0x210 [ 49.318961] [] ret_from_kernel_thread+0x1b/0x28 [ 49.318961] INITIAL USE at: [ 49.318961] [] __lock_acquire+0x47c/0x1880 [ 49.318961] [] lock_acquire+0xd2/0x110 [ 49.318961] [] mutex_lock_nested+0x98/0x790 [ 49.318961] [] register_cpu_notifier+0x20/0x70 [ 49.318961] [] kmem_cache_init+0xfe/0x15d [ 49.318961] [] start_kernel+0x460/0xa51 [ 49.318961] [] i386_start_kernel+0x19e/0x1af [ 49.318961] } [ 49.318961] ... key at: [] cpu_add_remove_lock+0x40/0xf0 [ 49.318961] ... acquired at: [ 49.318961] [] __lock_acquire+0x66b/0x1880 [ 49.318961] [] lock_acquire+0xd2/0x110 [ 49.318961] [] mutex_lock_nested+0x98/0x790 [ 49.318961] [] register_cpu_notifier+0x20/0x70 [ 49.318961] [] clocksource_enqueue_watchdog+0xde/0x230 [ 49.318961] [] __clocksource_register_scale+0xf1/0x140 [ 49.318961] [] tsc_refine_calibration_work+0x1bd/0x2a0 [ 49.318961] [] process_one_work+0x265/0x750 [ 49.318961] [] worker_thread+0x28a/0x6f0 [ 49.318961] [] kthread+0x110/0x150 [ 49.318961] [] ret_from_kernel_thread+0x1b/0x28 [ 49.318961] [ 49.318961] -> (watchdog_lock){..-...} ops: 7 { [ 49.318961] IN-SOFTIRQ-W at: [ 49.318961] [] __lock_acquire+0xbd0/0x1880 [ 49.318961] [] lock_acquire+0xd2/0x110 [ 49.318961] [] _raw_spin_lock+0x67/0x110 [ 49.318961] [] clocksource_watchdog+0x21/0x450 [ 49.318961] [] call_timer_fn+0xa7/0x170 [ 49.318961] [] run_timer_softirq+0x206/0x380 [ 49.318961] [] __do_softirq+0x14c/0x380 [ 49.318961] INITIAL USE at: [ 49.318961] [] __lock_acquire+0x47c/0x1880 [ 49.318961] [] lock_acquire+0xd2/0x110 [ 49.318961] [] _raw_spin_lock_irqsave+0xa5/0x150 [ 49.318961] [] clocksource_enqueue_watchdog+0x22/0x230 [ 49.318961] [] clocksource_register+0x120/0x1f0 [ 49.318961] [] register_refined_jiffies+0xeb/0x110 [ 49.318961] [] setup_arch+0x18cc/0x18e2 [ 49.318961] [] start_kernel+0x145/0xa51 [ 49.318961] [] i386_start_kernel+0x19e/0x1af [ 49.318961] } [ 49.318961] ... key at: [] watchdog_lock+0x10/0x40 [ 49.318961] ... acquired at: [ 49.318961] [] check_usage_forwards+0xb9/0x190 [ 49.318961] [] mark_lock+0x69b/0xc10 [ 49.318961] [] __lock_acquire+0xbd0/0x1880 [ 49.318961] [] lock_acquire+0xd2/0x110 [ 49.318961] [] _raw_spin_lock+0x67/0x110 [ 49.318961] [] clocksource_watchdog+0x21/0x450 [ 49.318961] [] call_timer_fn+0xa7/0x170 [ 49.318961] [] run_timer_softirq+0x206/0x380 [ 49.318961] [] __do_softirq+0x14c/0x380 [ 49.318961] [ 49.318961] [ 49.318961] stack backtrace: [ 49.318961] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.14.0-00262-g3db7490 #69 [ 49.318961] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 49.318961] c3afb3f0 d180dd7c c2892767 d180ddd8 d180ddb8 c10eff09 c2ea90cd c2ea941b [ 49.318961] 00000001 d180ddd8 00000006 c10f01b0 d180ddb8 c10eddd3 00000000 00000000 [ 49.318961] d1860574 d1860000 c10f01b0 d180de04 c10f0269 d1860574 00000001 c2ea941b [ 49.318961] Call Trace: [ 49.318961] [] dump_stack+0xb8/0x108 [ 49.318961] [] print_irq_inversion_bug+0x389/0x4a0 [ 49.318961] [] ? check_usage_backwards+0x190/0x190 [ 49.318961] [] ? find_usage_forwards+0x53/0xd0 [ 49.318961] [] ? check_usage_backwards+0x190/0x190 [ 49.318961] [] check_usage_forwards+0xb9/0x190 [ 49.318961] [] mark_lock+0x69b/0xc10 [ 49.318961] [] ? print_context_stack+0x5d/0x160 [ 49.318961] [] __lock_acquire+0xbd0/0x1880 [ 49.318961] [] lock_acquire+0xd2/0x110 [ 49.318961] [] ? clocksource_watchdog+0x21/0x450 [ 49.318961] [] _raw_spin_lock+0x67/0x110 [ 49.318961] [] ? clocksource_watchdog+0x21/0x450 [ 49.318961] [] ? clocksource_cpu_notify+0xd0/0xd0 [ 49.318961] [] clocksource_watchdog+0x21/0x450 [ 49.318961] [] ? process_timeout+0x30/0x30 [ 49.318961] [] ? clocksource_cpu_notify+0xd0/0xd0 [ 49.318961] [] call_timer_fn+0xa7/0x170 [ 49.318961] [] ? process_timeout+0x30/0x30 [ 49.318961] [] run_timer_softirq+0x206/0x380 [ 49.318961] [] ? clocksource_cpu_notify+0xd0/0xd0 [ 49.318961] [] __do_softirq+0x14c/0x380 [ 49.318961] [] ? _local_bh_enable+0xa0/0xa0 [ 49.318961] [] ? irq_exit+0xcb/0xe0 [ 49.318961] [] ? smp_apic_timer_interrupt+0x6b/0x90 [ 49.318961] [] ? apic_timer_interrupt+0x32/0x38 [ 49.318961] [] ? _raw_spin_unlock_irqrestore+0x74/0xe0 [ 49.318961] [] ? ida_simple_get+0xda/0x1d0 [ 49.318961] [] ? __kernfs_new_node+0xc5/0x150 [ 49.318961] [] ? kernfs_new_node+0x3e/0x90 [ 49.318961] [] ? __kernfs_create_file+0x4b/0x190 [ 49.318961] [] ? sysfs_add_file_mode_ns+0xed/0x2b0 [ 49.318961] [] ? internal_create_group+0x1d5/0x510 [ 49.318961] [] ? sysfs_create_group+0x1a/0x30 [ 49.318961] [] ? sysfs_slab_add+0xbb/0x370 [ 49.318961] [] ? bootstrap+0x149/0x149 [ 49.318961] [] ? slab_sysfs_init+0xc6/0x1be [ 49.318961] [] ? proc_vmalloc_init+0x34/0x46 [ 49.318961] [] ? do_one_initcall+0xf6/0x222 [ 49.318961] [] ? parameq+0x21/0x100 [ 49.318961] [] ? repair_env_string+0x32/0xd3 [ 49.318961] [] ? parse_args+0x2eb/0x5f0 [ 49.318961] [] ? restore_all+0xf/0xf [ 49.318961] [] ? kernel_init_freeable+0x26c/0x3f4 [ 49.318961] [] ? do_early_param+0x131/0x131 [ 49.318961] [] ? kernel_init+0x19/0x210 [ 49.318961] [] ? ret_from_kernel_thread+0x1b/0x28 [ 49.318961] [] ? rest_init+0x170/0x170 [ 49.469746] VFS: Disk quotas dquot_6.5.2 [ 49.474087] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) [ 49.494864] DLM installed /kernel/i386-randconfig-r3-0405/3db74909a7c789e467b126d837f7e0cea4a5c165/dmesg-yocto-f4-38:20140407102309:i386-randconfig-r3-0405:3.14.0-00262-g3db7490:69 /kernel/i386-randconfig-r3-0405/3db74909a7c789e467b126d837f7e0cea4a5c165/dmesg-yocto-inn-30:20140405183504:i386-randconfig-r3-0405:: /kernel/i386-randconfig-r3-0405/3db74909a7c789e467b126d837f7e0cea4a5c165/dmesg-quantal-loslunas-21:20140407102108:i386-randconfig-r3-0405:3.14.0-00262-g3db7490:69 /kernel/i386-randconfig-r3-0405/3db74909a7c789e467b126d837f7e0cea4a5c165/dmesg-yocto-lkp-ib04-73:20140405183459:i386-randconfig-r3-0405:3.14.0-00262-g3db7490:69 /kernel/i386-randconfig-r3-0405/3db74909a7c789e467b126d837f7e0cea4a5c165/dmesg-quantal-inn-13:20140407102022:i386-randconfig-r3-0405:: /kernel/i386-randconfig-r3-0405/3db74909a7c789e467b126d837f7e0cea4a5c165/dmesg-quantal-f4-128:20140407102255:i386-randconfig-r3-0405:3.14.0-00262-g3db7490:69 /kernel/i386-randconfig-r3-0405/3db74909a7c789e467b126d837f7e0cea4a5c165/dmesg-yocto-lkp-ib04-72:20140407102114:i386-randconfig-r3-0405:3.14.0-00262-g3db7490:69 /kernel/i386-randconfig-r3-0405/3db74909a7c789e467b126d837f7e0cea4a5c165/dmesg-yocto-lkp-ib04-74:20140405183502:i386-randconfig-r3-0405:3.14.0-00262-g3db7490:69 /kernel/i386-randconfig-r3-0405/3db74909a7c789e467b126d837f7e0cea4a5c165/dmesg-yocto-lkp-ib04-37:20140407102119:i386-randconfig-r3-0405:3.14.0-00262-g3db7490:69 /kernel/i386-randconfig-r3-0405/3db74909a7c789e467b126d837f7e0cea4a5c165/dmesg-yocto-f3-119:20140407102405:i386-randconfig-r3-0405:3.14.0-00262-g3db7490:69 /kernel/i386-randconfig-r3-0405/3db74909a7c789e467b126d837f7e0cea4a5c165/dmesg-quantal-lkp-ib04-8:20140407102110:i386-randconfig-r3-0405:3.14.0-00262-g3db7490:69 /kernel/i386-randconfig-r3-0405/3db74909a7c789e467b126d837f7e0cea4a5c165/dmesg-quantal-f4-111:20140407102254:i386-randconfig-r3-0405:3.14.0-00262-g3db7490:69 /kernel/i386-randconfig-r3-0405/3db74909a7c789e467b126d837f7e0cea4a5c165/dmesg-yocto-f3-118:20140407102345:i386-randconfig-r3-0405:3.14.0-00262-g3db7490:69 0:13:13 all_good:bad:all_bad boots [detached HEAD 2eb78aa] Revert "clocksource: register cpu notifier to remove timer from dying CPU" 1 file changed, 11 insertions(+), 53 deletions(-) ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linux-devel:devel-roam-i386-201404051825:2eb78aa9cc1621944c4f6b00171387aeff39e295:bisect-linux6 2014-04-07-11:54:46 2eb78aa9cc1621944c4f6b00171387aeff39e295 compiling Queued build task to /kernel-tests/build-queue/i386-randconfig-r3-0405-2eb78aa9cc1621944c4f6b00171387aeff39e295 Check for kernel in /kernel/i386-randconfig-r3-0405/2eb78aa9cc1621944c4f6b00171387aeff39e295 waiting for completion of /kernel-tests/build-queue/i386-randconfig-r3-0405-2eb78aa9cc1621944c4f6b00171387aeff39e295 2014-04-07 11:55:46 Kernel build fail or timeout 2014-04-07-11:55:46 2eb78aa9cc1621944c4f6b00171387aeff39e295 SKIP BROKEN BUILD Check /kernel-tests/build-served/i386-randconfig-r3-0405-2eb78aa9cc1621944c4f6b00171387aeff39e295 ========= upstream ========= Warning: you are leaving 1 commit behind, not connected to any of your branches: 2eb78aa Revert "clocksource: register cpu notifier to remove timer from dying CPU" If you want to keep them by creating a new branch, this may be a good time to do so with: git branch new_branch_name 2eb78aa HEAD is now at 18a1a7a... Merge git://git.kernel.org/pub/scm/linux/kernel/git/cmetcalf/linux-tile ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/linus:master:18a1a7a1d862ae0794a0179473d08a414dd49234:bisect-linux6 2014-04-07-11:55:50 18a1a7a1d862ae0794a0179473d08a414dd49234 compiling Queued build task to /kernel-tests/build-queue/i386-randconfig-r3-0405-18a1a7a1d862ae0794a0179473d08a414dd49234 Check for kernel in /kernel/i386-randconfig-r3-0405/18a1a7a1d862ae0794a0179473d08a414dd49234 waiting for completion of /kernel-tests/build-queue/i386-randconfig-r3-0405-18a1a7a1d862ae0794a0179473d08a414dd49234 waiting for completion of /kernel-tests/build-queue/.i386-randconfig-r3-0405-18a1a7a1d862ae0794a0179473d08a414dd49234 kernel: /kernel/i386-randconfig-r3-0405/18a1a7a1d862ae0794a0179473d08a414dd49234/vmlinuz-3.14.0-10312-g18a1a7a 2014-04-07-12:06:50 detecting boot state 1 4 8 16 19 26 31 37 41 46 47 50 51 55 58 59 60 SUCCESS ========= linux-next ========= Previous HEAD position was 18a1a7a... Merge git://git.kernel.org/pub/scm/linux/kernel/git/cmetcalf/linux-tile HEAD is now at 4b22efd... Add linux-next specific files for 20140403 ls -a /kernel-tests/run-queue/kvm/i386-randconfig-r3-0405/next:master:4b22efdd5595f0acb48f02bf664a451ee98f9a2e:bisect-linux6 2014-04-07-12:15:23 4b22efdd5595f0acb48f02bf664a451ee98f9a2e compiling Queued build task to /kernel-tests/build-queue/i386-randconfig-r3-0405-4b22efdd5595f0acb48f02bf664a451ee98f9a2e Check for kernel in /kernel/i386-randconfig-r3-0405/4b22efdd5595f0acb48f02bf664a451ee98f9a2e waiting for completion of /kernel-tests/build-queue/i386-randconfig-r3-0405-4b22efdd5595f0acb48f02bf664a451ee98f9a2e waiting for completion of /kernel-tests/build-queue/.i386-randconfig-r3-0405-4b22efdd5595f0acb48f02bf664a451ee98f9a2e kernel: /kernel/i386-randconfig-r3-0405/4b22efdd5595f0acb48f02bf664a451ee98f9a2e/vmlinuz-3.14.0-next-20140403 2014-04-07-12:19:23 detecting boot state 4 12 26 42 46 50 52 59 60 SUCCESS