Re: high system cpu load during intense disk i/o

From: Dimitrios Apostolou
Date: Mon Aug 06 2007 - 15:04:24 EST

Andrew Morton wrote:
On Mon, 06 Aug 2007 16:20:30 +0200 Dimitrios Apostolou <jimis@xxxxxxx> wrote:

Andrew Morton wrote:
On Sun, 5 Aug 2007 19:03:12 +0300 Dimitrios Apostolou <jimis@xxxxxxx> wrote:

was my report so complicated?
We're bad.

Seems that your context switch rate when running two instances of
badblocks against two different disks went batshit insane. It doesn't
happen here.

Please capture the `vmstat 1' output while running the problematic
workload.

The oom-killing could have been unrelated to the CPU load problem. iirc
badblocks uses a lot of memory, so it might have been genuine. Keep an eye
on the /proc/meminfo output and send the kernel dmesg output from the
oom-killing event.
Please see the attached files. Unfortunately I don't see any useful info in them:
*_before: before running any badblocks process
*_while: while running badblocks process, but without any cron job having kicked in
*_bad: 5 minutes later that some cron jobs kicked in

About the OOM killer, indeed I believe that it is unrelated. It started killing after about 2 days, that hundreds of processes were stuck as running and taking up memory, so I suppose the 256 MB RAM were truly filled. I just mentioned it because its behaviour is completely non-helpful. It doesn't touch the badblocks process, it rarely touches the stuck as running cron jobs, but it kills other irrelevant processes. If you still want the killing logs, tell me and I'll search for them.

ah. Your context-switch rate during the dual-badblocks run is not high at
all.

I suspect I was fooled by the oprofile output, which showed tremendous
amounts of load in schedule() and switch_to(). The percentages which
opreport shows are the percentage of non-halted CPU time. So if you have a
function in the kernel which is using 1% of the total CPU, and the CPU is
halted for 95% of the time, it appears that the function is taking 20% of
CPU!

The fix for that is to boot with the "idle=poll" boot parameter, to make
the CPU spin when it has nothing else to do.

I'm attaching the new oprofile output. I can't see any difference however. :-s


Dimitris


Mon Aug 6 21:46:59 EEST 2007
+ opcontrol --vmlinux=/usr/src/linux-2.6.22-ARCH/vmlinux
+ opcontrol --start
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Daemon started.
Profiler running.
+ sleep 5
+ opcontrol --shutdown
Stopping profiling.
Killing daemon.
+ echo

+ echo

+ echo

+ opreport
CPU: PIII, speed 798.025 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
samples| %|
------------------
7917 57.2658 vmlinux
2204 15.9421 libc-2.6.so
958 6.9295 libpython2.5.so.1.0
815 5.8951 ide_core
773 5.5913 perl
242 1.7505 ld-2.6.so
216 1.5624 mpop
204 1.4756 bash
121 0.8752 libgnutls.so.13.3.0
64 0.4629 badblocks
CPU_CLK_UNHALT...|
samples| %|
------------------
48 75.0000 badblocks
10 15.6250 anon (tgid:4338 range:0xb7f6c000-0xb7f6d000)
6 9.3750 anon (tgid:4339 range:0xb7f64000-0xb7f65000)
46 0.3327 ISO8859-1.so
37 0.2676 ext3
34 0.2459 imap-login
CPU_CLK_UNHALT...|
samples| %|
------------------
31 91.1765 imap-login
1 2.9412 anon (tgid:3989 range:0xb7f80000-0xb7f81000)
1 2.9412 anon (tgid:3990 range:0xb7f72000-0xb7f73000)
1 2.9412 anon (tgid:4332 range:0xb7fd6000-0xb7fd7000)
33 0.2387 libext2fs.so.2.4
24 0.1736 jbd
24 0.1736 libpthread-2.6.so
21 0.1519 gawk
15 0.1085 oprofile
13 0.0940 libcrypto.so.0.9.8
12 0.0868 ide_disk
11 0.0796 skge
7 0.0506 sshd
6 0.0434 dovecot-auth
CPU_CLK_UNHALT...|
samples| %|
------------------
5 83.3333 dovecot-auth
1 16.6667 anon (tgid:3971 range:0xb7f8c000-0xb7f8d000)
4 0.0289 libnetsnmp.so.15.0.0
4 0.0289 libnetsnmpmibs.so.15.0.0
4 0.0289 imap
4 0.0289 dovecot
3 0.0217 grep
3 0.0217 reiserfs
3 0.0217 locale-archive
1 0.0072 libdl-2.6.so
1 0.0072 init
1 0.0072 screen-4.0.3
+ echo

+ echo

+ echo

+ opreport -l /usr/src/linux-2.6.22-ARCH/vmlinux
CPU: PIII, speed 798.025 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
samples % symbol name
2477 31.2871 __switch_to
1625 20.5255 schedule
246 3.1072 mask_and_ack_8259A
222 2.8041 __blockdev_direct_IO
147 1.8568 follow_page
107 1.3515 put_page
106 1.3389 do_wp_page
91 1.1494 native_load_tls
88 1.1115 __bio_add_page
80 1.0105 delay_tsc
63 0.7958 __handle_mm_fault
50 0.6316 submit_page_section
49 0.6189 get_page_from_freelist
48 0.6063 dio_bio_add_page
45 0.5684 dequeue_task
41 0.5179 get_user_pages
40 0.5052 bio_alloc_bioset
39 0.4926 enable_8259A_irq
38 0.4800 dio_send_cur_page
38 0.4800 page_address
38 0.4800 sysenter_past_esp
37 0.4673 dio_get_page
37 0.4673 find_get_page
36 0.4547 bio_add_page
36 0.4547 kmem_cache_alloc
36 0.4547 unmap_vmas
35 0.4421 kmem_cache_free
34 0.4295 blk_rq_map_sg
33 0.4168 blk_recount_segments
32 0.4042 do_page_fault
31 0.3916 generic_file_direct_write
30 0.3789 __generic_file_aio_write_nolock
30 0.3789 vm_normal_page
28 0.3537 __link_path_walk
28 0.3537 __mutex_lock_slowpath
26 0.3284 irq_entries_start
25 0.3158 filemap_nopage
25 0.3158 vfs_write
23 0.2905 find_vma
22 0.2779 __d_lookup
22 0.2779 dio_bio_complete
22 0.2779 mark_page_accessed
21 0.2653 add_timer_randomness
21 0.2653 max_block
21 0.2653 page_fault
19 0.2400 restore_all
18 0.2274 __add_entropy_words
18 0.2274 preempt_schedule
17 0.2147 __copy_to_user_ll
17 0.2147 __generic_unplug_device
17 0.2147 do_sys_poll
17 0.2147 fget_light
17 0.2147 math_state_restore
16 0.2021 __mod_timer
16 0.2021 generic_file_direct_IO
16 0.2021 preempt_schedule_irq
15 0.1895 block_llseek
15 0.1895 do_sync_write
14 0.1768 current_fs_time
14 0.1768 do_generic_mapping_read
14 0.1768 mempool_free
14 0.1768 native_flush_tlb_single
14 0.1768 radix_tree_lookup
14 0.1768 rw_verify_area
13 0.1642 __alloc_pages
13 0.1642 __make_request
13 0.1642 _spin_lock_irqsave
13 0.1642 cond_resched
12 0.1516 cfq_completed_request
12 0.1516 strnlen_user
11 0.1389 __mutex_unlock_slowpath
11 0.1389 do_lookup
11 0.1389 generic_make_request
11 0.1389 get_request
11 0.1389 restore_nocheck
10 0.1263 blkdev_direct_IO
10 0.1263 load_elf_binary
10 0.1263 unix_poll
9 0.1137 cfq_insert_request
9 0.1137 cfq_set_request
9 0.1137 copy_page_range
9 0.1137 do_mmap_pgoff
9 0.1137 elv_insert
9 0.1137 generic_file_aio_write_nolock
9 0.1137 generic_permission
9 0.1137 recalc_task_prio
8 0.1010 blkdev_get_blocks
8 0.1010 cache_reap
8 0.1010 dio_new_bio
8 0.1010 file_update_time
8 0.1010 sys_llseek
8 0.1010 vsnprintf
7 0.0884 bio_put
7 0.0884 blk_backing_dev_unplug
7 0.0884 cfq_dispatch_requests
7 0.0884 copy_process
7 0.0884 device_not_available
7 0.0884 disk_round_stats
7 0.0884 elv_queue_empty
7 0.0884 free_block
7 0.0884 generic_unplug_device
7 0.0884 hrtimer_interrupt
7 0.0884 kmap_atomic_prot
7 0.0884 mutex_remove_waiter
7 0.0884 need_resched
7 0.0884 page_remove_rmap
7 0.0884 permission
6 0.0758 __blk_put_request
6 0.0758 __copy_from_user_ll
6 0.0758 __copy_user_intel
6 0.0758 __end_that_request_first
6 0.0758 __find_get_block_slow
6 0.0758 bio_init
6 0.0758 blk_remove_plug
6 0.0758 cfq_may_queue
6 0.0758 dio_cleanup
6 0.0758 dio_complete
6 0.0758 dput
6 0.0758 getname
6 0.0758 inotify_inode_queue_event
6 0.0758 kfree
6 0.0758 memcpy
6 0.0758 mempool_alloc
6 0.0758 path_walk
6 0.0758 rb_erase
6 0.0758 read_tsc
6 0.0758 task_rq_lock
6 0.0758 try_to_wake_up
6 0.0758 vfs_llseek
5 0.0632 __const_udelay
5 0.0632 __fput
5 0.0632 blk_plug_device
5 0.0632 cfq_remove_request
5 0.0632 copy_strings
5 0.0632 current_io_context
5 0.0632 debug_mutex_add_waiter
5 0.0632 dnotify_parent
5 0.0632 do_IRQ
5 0.0632 do_filp_open
5 0.0632 do_path_lookup
5 0.0632 drain_array
5 0.0632 elv_rqhash_add
5 0.0632 free_poll_entry
5 0.0632 generic_segment_checks
5 0.0632 handle_level_irq
5 0.0632 idle_cpu
5 0.0632 init_request_from_bio
5 0.0632 io_schedule
5 0.0632 ip_append_data
5 0.0632 note_interrupt
5 0.0632 proc_sys_lookup_table_one
5 0.0632 sched_clock
5 0.0632 sys_write
5 0.0632 up_read
4 0.0505 __dentry_open
4 0.0505 __freed_request
4 0.0505 __pagevec_lru_add_active
4 0.0505 _atomic_dec_and_lock
4 0.0505 bdev_read_only
4 0.0505 cfq_queue_empty
4 0.0505 copy_to_user
4 0.0505 debug_mutex_free_waiter
4 0.0505 do_munmap
4 0.0505 do_sync_read
4 0.0505 do_wait
4 0.0505 dummy_inode_permission
4 0.0505 elv_completed_request
4 0.0505 end_that_request_last
4 0.0505 filemap_write_and_wait
4 0.0505 find_extend_vma
4 0.0505 find_vma_prev
4 0.0505 generic_file_aio_read
4 0.0505 get_empty_filp
4 0.0505 get_request_wait
4 0.0505 hweight32
4 0.0505 internal_add_timer
4 0.0505 irq_exit
4 0.0505 native_read_tsc
4 0.0505 number
4 0.0505 poll_freewait
4 0.0505 proc_lookup
4 0.0505 setup_arg_pages
4 0.0505 should_remove_suid
4 0.0505 sock_poll
4 0.0505 submit_bio
4 0.0505 tick_sched_timer
4 0.0505 unmap_region
4 0.0505 vma_adjust
3 0.0379 __atomic_notifier_call_chain
3 0.0379 __do_page_cache_readahead
3 0.0379 __elv_add_request
3 0.0379 __find_get_block
3 0.0379 __mark_inode_dirty
3 0.0379 __pte_alloc
3 0.0379 anon_vma_prepare
3 0.0379 anon_vma_unlink
3 0.0379 bio_endio
3 0.0379 bit_waitqueue
3 0.0379 call_rcu
3 0.0379 cfq_init_prio_data
3 0.0379 cfq_put_queue
3 0.0379 cfq_service_tree_add
3 0.0379 clear_bdi_congested
3 0.0379 common_interrupt
3 0.0379 credit_entropy_store
3 0.0379 deactivate_task
3 0.0379 debug_mutex_lock_common
3 0.0379 debug_mutex_unlock
3 0.0379 del_timer
3 0.0379 down_read_trylock
3 0.0379 dummy_inode_getattr
3 0.0379 elv_dispatch_sort
3 0.0379 elv_may_queue
3 0.0379 elv_put_request
3 0.0379 elv_set_request
3 0.0379 enqueue_task
3 0.0379 error_code
3 0.0379 flush_old_exec
3 0.0379 free_hot_cold_page
3 0.0379 generic_fillattr
3 0.0379 kmem_cache_zalloc
3 0.0379 link_path_walk
3 0.0379 lock_timer_base
3 0.0379 locks_remove_flock
3 0.0379 may_expand_vm
3 0.0379 native_load_esp0
3 0.0379 open_namei
3 0.0379 path_lookup_open
3 0.0379 pipe_poll
3 0.0379 proc_flush_task
3 0.0379 release_vm86_irqs
3 0.0379 remove_vma
3 0.0379 rq_init
3 0.0379 sock_alloc_send_skb
3 0.0379 strncpy_from_user
3 0.0379 sys_mmap2
3 0.0379 sys_mprotect
3 0.0379 touch_atime
3 0.0379 vfs_read
3 0.0379 vm_acct_memory
3 0.0379 vm_stat_account
3 0.0379 vma_link
2 0.0253 I_BDEV
2 0.0253 __dec_zone_state
2 0.0253 __group_complete_signal
2 0.0253 __inc_zone_state
2 0.0253 __mutex_lock_interruptible_slowpath
2 0.0253 __path_lookup_intent_open
2 0.0253 __pollwait
2 0.0253 __rb_rotate_right
2 0.0253 __vm_enough_memory
2 0.0253 __vma_link_rb
2 0.0253 __wake_up
2 0.0253 __wake_up_bit
2 0.0253 alloc_inode
2 0.0253 anon_vma_ctor
2 0.0253 bio_alloc
2 0.0253 bio_free
2 0.0253 bio_fs_destructor
2 0.0253 bio_get_nr_vecs
2 0.0253 bio_phys_segments
2 0.0253 blk_queue_bounce
2 0.0253 cache_alloc_refill
2 0.0253 check_pgt_cache
2 0.0253 debug_mutex_set_owner
2 0.0253 dio_bio_end_io
2 0.0253 dio_zero_block
2 0.0253 do_notify_parent
2 0.0253 do_notify_resume
2 0.0253 do_sys_open
2 0.0253 down_read
2 0.0253 dst_alloc
2 0.0253 effective_prio
2 0.0253 elv_dequeue_request
2 0.0253 elv_next_request
2 0.0253 file_free_rcu
2 0.0253 file_read_actor
2 0.0253 filp_close
2 0.0253 find_next_bit
2 0.0253 find_next_zero_bit
2 0.0253 find_vma_prepare
2 0.0253 flush_tlb_mm
2 0.0253 flush_tlb_page
2 0.0253 generic_file_mmap
2 0.0253 get_index
2 0.0253 get_unused_fd
2 0.0253 getnstimeofday
2 0.0253 handle_IRQ_event
2 0.0253 icmp_push_reply
2 0.0253 icmp_send
2 0.0253 kill_fasync
2 0.0253 kunmap_atomic
2 0.0253 lru_cache_add_active
2 0.0253 mntput_no_expire
2 0.0253 native_read_cr0
2 0.0253 notifier_call_chain
2 0.0253 ordered_bio_endio
2 0.0253 page_cache_readahead
2 0.0253 percpu_counter_mod
2 0.0253 poll_initwait
2 0.0253 profile_munmap
2 0.0253 put_filp
2 0.0253 put_io_context
2 0.0253 quicklist_trim
2 0.0253 radix_tree_gang_lookup_tag
2 0.0253 rb_insert_color
2 0.0253 remove_suid
2 0.0253 resume_userspace
2 0.0253 sched_balance_self
2 0.0253 seq_printf
2 0.0253 smp_apic_timer_interrupt
2 0.0253 sys_brk
2 0.0253 sys_fstat64
2 0.0253 sys_read
2 0.0253 sys_rt_sigaction
2 0.0253 sys_stat64
2 0.0253 system_call
2 0.0253 vma_merge
2 0.0253 work_notifysig
2 0.0253 zone_watermark_ok
1 0.0126 __activate_task
1 0.0126 __alloc_skb
1 0.0126 __blocking_notifier_call_chain
1 0.0126 __bread
1 0.0126 __delay
1 0.0126 __do_softirq
1 0.0126 __follow_mount
1 0.0126 __iget
1 0.0126 __ip_select_ident
1 0.0126 __put_user_4
1 0.0126 __rb_rotate_left
1 0.0126 __rcu_process_callbacks
1 0.0126 __rmqueue
1 0.0126 __sigqueue_alloc
1 0.0126 __sigqueue_free
1 0.0126 __sk_dst_check
1 0.0126 __tasklet_schedule
1 0.0126 __tcp_ack_snd_check
1 0.0126 __tcp_push_pending_frames
1 0.0126 __user_walk_fd
1 0.0126 __vma_link
1 0.0126 _read_lock_irqsave
1 0.0126 account_system_time
1 0.0126 account_user_time
1 0.0126 add_disk_randomness
1 0.0126 add_wait_queue
1 0.0126 all_vm_events
1 0.0126 alloc_buffer_head
1 0.0126 alloc_pid
1 0.0126 anon_pipe_buf_release
1 0.0126 anon_vma_link
1 0.0126 arch_setup_additional_pages
1 0.0126 bio_hw_segments
1 0.0126 blk_do_ordered
1 0.0126 blk_start_queueing
1 0.0126 blockable_page_cache_readahead
1 0.0126 blocking_notifier_call_chain
1 0.0126 bmap
1 0.0126 can_vma_merge_after
1 0.0126 cfq_add_rq_rb
1 0.0126 cfq_choose_req
1 0.0126 cfq_cic_rb_lookup
1 0.0126 cfq_rb_erase
1 0.0126 check_userspace
1 0.0126 copy_semundo
1 0.0126 core_sys_select
1 0.0126 cp_new_stat64
1 0.0126 csum_partial_copy_generic
1 0.0126 d_alloc
1 0.0126 dec_zone_page_state
1 0.0126 dequeue_signal
1 0.0126 devinet_ioctl
1 0.0126 do_fork
1 0.0126 do_group_exit
1 0.0126 do_sigaction
1 0.0126 drive_stat_acct
1 0.0126 dummy_capable
1 0.0126 dummy_file_alloc_security
1 0.0126 dummy_task_kill
1 0.0126 dummy_vm_enough_memory
1 0.0126 dup_fd
1 0.0126 elv_rb_add
1 0.0126 end_that_request_first
1 0.0126 exit_itimers
1 0.0126 fasync_helper
1 0.0126 fget
1 0.0126 file_kill
1 0.0126 file_move
1 0.0126 file_ra_state_init
1 0.0126 flush_sigqueue
1 0.0126 fn_hash_lookup
1 0.0126 fput
1 0.0126 free_page_and_swap_cache
1 0.0126 free_pgtables
1 0.0126 get_futex_key
1 0.0126 get_io_context
1 0.0126 get_task_mm
1 0.0126 hrtimer_try_to_cancel
1 0.0126 icmp_glue_bits
1 0.0126 ifind_fast
1 0.0126 inotify_dentry_parent_queue_event
1 0.0126 invalidate_inode_buffers
1 0.0126 ip_route_input
1 0.0126 irq_enter
1 0.0126 local_bh_enable_ip
1 0.0126 may_open
1 0.0126 mempool_alloc_slab
1 0.0126 mempool_free_slab
1 0.0126 n_tty_receive_buf
1 0.0126 native_apic_write
1 0.0126 native_set_pte_at
1 0.0126 open_exec
1 0.0126 opost
1 0.0126 page_add_file_rmap
1 0.0126 pipe_release
1 0.0126 pipe_write
1 0.0126 prio_tree_insert
1 0.0126 prio_tree_remove
1 0.0126 proc_sys_lookup_table
1 0.0126 profile_tick
1 0.0126 pty_chars_in_buffer
1 0.0126 put_tty_queue
1 0.0126 rb_first
1 0.0126 recalc_sigpending_tsk
1 0.0126 release_open_intent
1 0.0126 release_task
1 0.0126 restore_sigcontext
1 0.0126 ret_from_exception
1 0.0126 ret_from_intr
1 0.0126 scheduler_tick
1 0.0126 search_binary_handler
1 0.0126 secure_ip_id
1 0.0126 show_stat
1 0.0126 sigprocmask
1 0.0126 snprintf
1 0.0126 sock_fasync
1 0.0126 softlockup_tick
1 0.0126 special_mapping_nopage
1 0.0126 split_vma
1 0.0126 sys_access
1 0.0126 sys_clone
1 0.0126 sys_execve
1 0.0126 sys_futex
1 0.0126 sys_open
1 0.0126 sys_rt_sigprocmask
1 0.0126 syscall_exit
1 0.0126 sysctl_head_next
1 0.0126 task_running_tick
1 0.0126 tcp_poll
1 0.0126 tcp_rcv_space_adjust
1 0.0126 tcp_recvmsg
1 0.0126 tcp_transmit_skb
1 0.0126 test_set_page_writeback
1 0.0126 tick_do_update_jiffies64
1 0.0126 tick_program_event
1 0.0126 timespec_trunc
1 0.0126 try_to_del_timer_sync
1 0.0126 tty_ioctl
1 0.0126 tty_ldisc_ref_wait
1 0.0126 udp_flush_pending_frames
1 0.0126 unuse_table
1 0.0126 update_process_times
1 0.0126 update_wall_time
1 0.0126 vfs_stat_fd
1 0.0126 vma_prio_tree_remove
1 0.0126 wake_up_inode
1 0.0126 wake_up_new_task
1 0.0126 wake_up_process
1 0.0126 write_cache_pages
Mon Aug 6 21:38:04 EEST 2007
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Daemon started.
Profiler running.
Stopping profiling.
Killing daemon.



CPU: PIII, speed 798.025 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
samples| %|
------------------
40131 97.6495 vmlinux
318 0.7738 libc-2.6.so
168 0.4088 oprofiled
CPU_CLK_UNHALT...|
samples| %|
------------------
167 99.4048 oprofiled
1 0.5952 anon (tgid:4159 range:0xb7f55000-0xb7f56000)
162 0.3942 bash
133 0.3236 ld-2.6.so
51 0.1241 oprofile
49 0.1192 ISO8859-1.so
33 0.0803 ext3
20 0.0487 jbd
5 0.0122 locale-archive
4 0.0097 gawk
4 0.0097 imap-login
3 0.0073 grep
3 0.0073 reiserfs
2 0.0049 cat
2 0.0049 libcrypto.so.0.9.8
2 0.0049 dovecot-auth
1 0.0024 ide_core
1 0.0024 ide_disk
1 0.0024 libncurses.so.5.6
1 0.0024 which
1 0.0024 libnetsnmp.so.15.0.0
1 0.0024 libnetsnmpmibs.so.15.0.0
1 0.0024 dovecot
CPU_CLK_UNHALT...|
samples| %|
------------------
1 100.000 anon (tgid:3968 range:0xb7fa7000-0xb7fa8000)



CPU: PIII, speed 798.025 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
samples % symbol name
39019 97.2291 cpu_idle
263 0.6554 poll_idle
131 0.3264 quicklist_trim
92 0.2292 do_wp_page
50 0.1246 check_pgt_cache
44 0.1096 __handle_mm_fault
37 0.0922 get_page_from_freelist
31 0.0772 page_fault
21 0.0523 unmap_vmas
15 0.0374 __copy_to_user_ll
13 0.0324 filemap_nopage
12 0.0299 radix_tree_lookup
11 0.0274 copy_process
10 0.0249 __link_path_walk
10 0.0249 sysenter_past_esp
9 0.0224 do_page_fault
9 0.0224 find_get_page
8 0.0199 kmem_cache_free
6 0.0150 __d_lookup
6 0.0150 __find_get_block
6 0.0150 page_address
5 0.0125 _atomic_dec_and_lock
5 0.0125 do_mmap_pgoff
5 0.0125 free_hot_cold_page
5 0.0125 kmem_cache_alloc
5 0.0125 load_elf_binary
5 0.0125 strnlen_user
4 0.0100 __wake_up_bit
4 0.0100 copy_page_range
4 0.0100 d_alloc
4 0.0100 error_code
4 0.0100 find_vma
4 0.0100 flush_tlb_mm
4 0.0100 inode_init_once
4 0.0100 kmap_atomic_prot
4 0.0100 put_page
4 0.0100 restore_nocheck
4 0.0100 schedule
4 0.0100 vm_normal_page
3 0.0075 __follow_mount
3 0.0075 __mutex_lock_slowpath
3 0.0075 __rmqueue
3 0.0075 anon_vma_unlink
3 0.0075 ktime_get_ts
3 0.0075 mark_page_accessed
3 0.0075 memcpy
3 0.0075 prio_tree_insert
3 0.0075 strncpy_from_user
3 0.0075 sys_close
2 0.0050 __atomic_notifier_call_chain
2 0.0050 __find_get_block_slow
2 0.0050 __lookup_mnt
2 0.0050 __mutex_unlock_slowpath
2 0.0050 __rb_rotate_left
2 0.0050 __rcu_process_callbacks
2 0.0050 __switch_to
2 0.0050 anon_vma_prepare
2 0.0050 cache_alloc_refill
2 0.0050 debug_mutex_add_waiter
2 0.0050 do_path_lookup
2 0.0050 dup_fd
2 0.0050 generic_fillattr
2 0.0050 generic_make_request
2 0.0050 get_task_mm
2 0.0050 get_unused_fd
2 0.0050 getnstimeofday
2 0.0050 hrtimer_interrupt
2 0.0050 hweight32
2 0.0050 inotify_dentry_parent_queue_event
2 0.0050 link_path_walk
2 0.0050 mask_and_ack_8259A
2 0.0050 may_open
2 0.0050 path_walk
2 0.0050 pgd_alloc
2 0.0050 preempt_schedule
2 0.0050 proc_lookup
2 0.0050 tick_nohz_restart_sched_tick
2 0.0050 unlink_file_vma
2 0.0050 vm_stat_account
2 0.0050 vma_merge
2 0.0050 zone_watermark_ok
1 0.0025 __alloc_pages
1 0.0025 __alloc_skb
1 0.0025 __copy_from_user_ll
1 0.0025 __copy_user_intel
1 0.0025 __dentry_open
1 0.0025 __do_softirq
1 0.0025 __end_that_request_first
1 0.0025 __fput
1 0.0025 __free_pages
1 0.0025 __getblk
1 0.0025 __inc_zone_state
1 0.0025 __kmalloc
1 0.0025 __pagevec_lru_add
1 0.0025 __pte_alloc
1 0.0025 __put_task_struct
1 0.0025 __tasklet_schedule
1 0.0025 __vm_enough_memory
1 0.0025 __vma_link_rb
1 0.0025 _read_lock_irqsave
1 0.0025 _spin_lock_irqsave
1 0.0025 account_user_time
1 0.0025 alloc_buffer_head
1 0.0025 alloc_inode
1 0.0025 apic_timer_interrupt
1 0.0025 arch_get_unmapped_area_topdown
1 0.0025 arch_pick_mmap_layout
1 0.0025 attach_pid
1 0.0025 bio_init
1 0.0025 block_read_full_page
1 0.0025 cache_reap
1 0.0025 call_rcu
1 0.0025 can_share_swap_page
1 0.0025 cfq_set_request
1 0.0025 check_userspace
1 0.0025 cleanup_timers
1 0.0025 clockevents_program_event
1 0.0025 clocksource_get_next
1 0.0025 compute_creds
1 0.0025 copy_from_user
1 0.0025 copy_strings
1 0.0025 create_read_pipe
1 0.0025 current_fs_time
1 0.0025 current_io_context
1 0.0025 d_instantiate
1 0.0025 d_splice_alias
1 0.0025 debug_mutex_unlock
1 0.0025 delayed_put_task_struct
1 0.0025 dnotify_flush
1 0.0025 dnotify_parent
1 0.0025 do_exit
1 0.0025 do_fcntl
1 0.0025 do_lookup
1 0.0025 do_munmap
1 0.0025 do_notify_resume
1 0.0025 do_softirq
1 0.0025 do_sys_open
1 0.0025 do_wait
1 0.0025 down_read_trylock
1 0.0025 down_write
1 0.0025 dput
1 0.0025 dummy_inode_getattr
1 0.0025 eligible_child
1 0.0025 elv_dispatch_sort
1 0.0025 end_buffer_write_sync
1 0.0025 enqueue_hrtimer
1 0.0025 exit_mmap
1 0.0025 fget
1 0.0025 fget_light
1 0.0025 file_read_actor
1 0.0025 filp_close
1 0.0025 find_busiest_group
1 0.0025 find_lock_page
1 0.0025 find_next_zero_bit
1 0.0025 find_or_create_page
1 0.0025 find_vma_prev
1 0.0025 flush_old_exec
1 0.0025 flush_sigqueue
1 0.0025 flush_tlb_page
1 0.0025 fput
1 0.0025 free_block
1 0.0025 free_page_and_swap_cache
1 0.0025 free_pgtables
1 0.0025 free_poll_entry
1 0.0025 generic_file_mmap
1 0.0025 generic_segment_checks
1 0.0025 get_empty_filp
1 0.0025 get_next_timer_interrupt
1 0.0025 getname
1 0.0025 inode_change_ok
1 0.0025 internal_add_timer
1 0.0025 invalidate_inode_buffers
1 0.0025 kill_fasync
1 0.0025 kmap
1 0.0025 kmem_cache_zalloc
1 0.0025 ktime_get
1 0.0025 kunmap_atomic
1 0.0025 ll_back_merge_fn
1 0.0025 lookup_hash
1 0.0025 lookup_mnt
1 0.0025 lru_cache_add_active
1 0.0025 mempool_free_slab
1 0.0025 mm_alloc
1 0.0025 mm_release
1 0.0025 mmput
1 0.0025 mntput_no_expire
1 0.0025 native_set_pte_at
1 0.0025 number
1 0.0025 page_add_file_rmap
1 0.0025 page_add_new_anon_rmap
1 0.0025 page_remove_rmap
1 0.0025 page_waitqueue
1 0.0025 path_release
1 0.0025 peer_avl_rebalance
1 0.0025 percpu_counter_mod
1 0.0025 permission
1 0.0025 pipe_write
1 0.0025 prio_tree_replace
1 0.0025 proc_flush_task
1 0.0025 proc_pid_lookup
1 0.0025 radix_tree_tag_set
1 0.0025 rb_insert_color
1 0.0025 remove_vma
1 0.0025 restore_all
1 0.0025 rm_from_queue_full
1 0.0025 rw_verify_area
1 0.0025 sched_balance_self
1 0.0025 set_bh_page
1 0.0025 set_normalized_timespec
1 0.0025 set_task_comm
1 0.0025 setup_sigcontext
1 0.0025 show_stat
1 0.0025 sync_page
1 0.0025 sys_dup2
1 0.0025 sys_faccessat
1 0.0025 sys_ftruncate
1 0.0025 sys_read
1 0.0025 sys_rt_sigprocmask
1 0.0025 sys_select
1 0.0025 task_rq_lock
1 0.0025 task_running_tick
1 0.0025 tcp_poll
1 0.0025 tick_do_update_jiffies64
1 0.0025 tick_nohz_stop_sched_tick
1 0.0025 try_to_wake_up
1 0.0025 unix_poll
1 0.0025 unlock_buffer
1 0.0025 up_read
1 0.0025 update_wall_time
1 0.0025 vfs_getattr
1 0.0025 vfs_permission
1 0.0025 vfs_read
1 0.0025 vm_acct_memory
1 0.0025 vsnprintf
1 0.0025 wake_up_inode
1 0.0025 wake_up_new_task
1 0.0025 work_notifysig
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 2 0 161676 9716 55700 0 0 248 2020 50 131 5 4 84 7
4 2 0 161668 9716 55704 0 0 0 16000 253 504 80 20 0 0
3 2 0 161668 9716 55704 0 0 0 16000 249 522 66 34 0 0
3 2 0 161608 9716 55704 0 0 0 15936 251 508 80 20 0 0
3 2 0 161548 9716 55704 0 0 0 15936 249 523 73 28 0 0
4 2 0 161488 9716 55704 0 0 0 15936 250 517 84 16 0 0
3 2 0 161488 9724 55704 0 0 0 15876 252 469 57 43 0 0
4 2 0 161428 9724 55704 0 0 0 15872 253 504 60 40 0 0
3 2 0 161428 9724 55704 0 0 0 15936 248 500 24 76 0 0
4 2 0 161428 9740 55704 0 0 0 15840 258 475 52 48 0 0