RE: [question] IPC queue filling-up problem?

From: Fortier,Vincent [Montreal]
Date: Wed Sep 05 2007 - 11:49:14 EST

> -----Message d'origine-----
> De : linux-kernel-owner@xxxxxxxxxxxxxxx
> [mailto:linux-kernel-owner@xxxxxxxxxxxxxxx] De la part de Eric Dumazet
>
> On Wed, 5 Sep 2007 09:37:50 -0400
> "Fortier,Vincent [Montreal]" <Vincent.Fortier1@xxxxxxxx> wrote:
>
> >
> > Hi all,
> >
> > We are testing new hardware and planning a switch from our old
redhat
> > 7.3 to a Debian Etch 4.0 for our radar forecast analysis systems.
We
> > found out that our main IPC dispatcher software module would use
100%
> > of a CPU all the time and that the IPC queues would fill up quickly
on a
> > 2.6 kernel. We first tought that it would be a problem of
> > compatibility between a 2.4 vs 2.6 IPC calls vs our radar analysis
> > software but after a lot of work we have been able to test a 2.4
> > kernel on that same hardware and got the exact same problem.
> >
> > So curiously, on our actual systems (see SYSTEM 1 below) our IPC
> > dispatcher module works like a charm and queues gets near 0. On our

> > test system which are way more powerfull systems (see SYSTEM 2) our
> > IPC dispatcher module queue fills up rapidly (depending of the
msgmnb
> > queue size it will wimply take a bit longer to fill).
> >
> > We have tested both our already compiled binaries from rh73 using
gcc
> > 2.9 and a recompiled version of the modules on a debian sarge system

> > and got the exact same problem on either a Debian Sarge 3.1 (running
a
> > 2.4 or 2.6 kernel) and on a Etch 64bit system (using 32bit compat
> > layer) with a 2.6 kernel. In all cases the queues would simply
fill-up.
> >
> > After strac'ing the module I noticed that the time needed to handle
> > the signal & ipc calls are way lower on the new system hence I don't

> > see why the dispatcher queue does fill-up like that?!?!?!
> >
> > Does anyone experienced something similar? Could this be a kernel
> > issue vs material, kernel option? Might this be related to libc?
> >
> > Help / Clues very much appreciated.
>
> Hi Vincent
>
> top shows that something is eating cpu cycles in User mode on
> your new platform, while old platform consumes cycles both in
> User and System land.
>
> This might be related to some programing error, maybe some
> spinlock in user mode or bad multi-threading synchronization,
> or scheduling assumptions, that break because of the quad
> core cpus of your new machine.

Actually, could this be worth trying (adding Ingo in CC):
http://lkml.org/lkml/2007/9/5/75

> So the thread that is supposed to consume IPC messages is not
> scheduled in time, because CPU starves. (beware the four
> cores of each CPU compete for ressources)
>
> You could issue "ps auxm" to check which threads are spining
> in User mode and try to trace them ?

Effectively in this specific test case I had 5 stuck process using each
100% of a CPU... although 3 other cores where still available so there
is (I believe) no reason why it should had starved that much.

Anyhow, that did not happend on all the other testing I made during the
past 3 weeks (except this one).

I restarted the this test (again using a 2.4.35.1 kernel), mde sure no
process where stuck :), and grabbed the ps aux + ipcs -q info (attached)

Again, the ipcs -q shows that the queue is getting full comparing to
SYSTEM 1 which always has a queue of 0.

Note: Also attached a top.txt file showing that the dispatcher uses 100%
of a CPU on SYSTEM 2. This never occurs on SYSTEM 1.

> Eric

PS, thnx for replying.

- vin

> > SYSTEM INFORMATION:
> >
> > SYSTEM 1:
> > ---------
> > HPDL580 G2
> > Quad Intel Xeon 1.90GHz
> > 4G ram
> > DRBD disks on dual-gigabit adapter
> > OS: RedHat 7.3 / kernel: 2.4.33 / libc: 2.2.5 / gcc 2.96
> >
> > SYSTEM 2:
> > ---------
> > Dell PE2950
> > Dual Intel Quad-Core 2.66GHz
> > 16G ram
> > local 300G 15000 RPM SCSI.
> > OS1: Debian Etch 4.0 / kernels 2.6.18 -> 2.6.22 / libc 2.3.6 / gcc
4.1.2
> > OS2: Debian Sarge 3.1 / kernels 2.4.35, 2.6.18 -> 2.6.22 / libc
2.3.2 / gcc 3.3.5

------ Message Queues --------
key msqid owner perms used-bytes messages
0x41018018 0 urp 660 60784 748
0x41018024 32769 urp 660 0 0
0x4101800b 65538 urp 660 0 0
0x41018011 98307 urp 660 0 0
0x4101800e 131076 urp 660 0 0
0x4101803f 163845 urp 660 0 0
0x41018019 196614 urp 660 0 0
0x41018026 229383 urp 660 0 0
0x4101802f 262152 urp 660 0 0
0x4101803a 294921 urp 660 0 0
0x41018036 327690 urp 660 0 0
0x4101802b 360459 urp 660 0 0
0x41018040 393228 urp 660 0 0
0x41018027 425997 urp 660 0 0
0x41018028 458766 urp 660 0 0
0x41018020 491535 urp 660 0 0
0x4101800c 524304 urp 660 0 0
0x4101800d 557073 urp 660 0 0
0x4101803c 589842 urp 660 0 0
0x4101802e 622611 urp 660 0 0
0x41018021 655380 urp 660 0 0
0x41018006 688149 urp 660 0 0
0x4101802a 720918 urp 660 0 0
0x4101800a 753687 urp 660 0 0
0x4101801e 786456 urp 660 0 0
0x41018029 819225 urp 660 0 0
0x41018037 851994 urp 660 0 0
0x41018038 884763 urp 660 0 0
0x41018022 917532 urp 660 0 0

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.4 0.0 1492 500 ? - 10:06 0:04 init [2]
root - 0.4 - - - - S 10:06 0:04 -
root 2 0.0 0.0 0 0 ? - 10:06 0:00 [keventd]
root - 0.0 - - - - S 10:06 0:00 -
root 3 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU0]
root - 0.0 - - - - SN 10:06 0:00 -
root 4 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU1]
root - 0.0 - - - - SN 10:06 0:00 -
root 5 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU2]
root - 0.0 - - - - SN 10:06 0:00 -
root 6 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU3]
root - 0.0 - - - - SN 10:06 0:00 -
root 7 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU4]
root - 0.0 - - - - SN 10:06 0:00 -
root 8 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU5]
root - 0.0 - - - - SN 10:06 0:00 -
root 9 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU6]
root - 0.0 - - - - SN 10:06 0:00 -
root 10 0.0 0.0 0 0 ? - 10:06 0:00 [ksoftirqd_CPU7]
root - 0.0 - - - - SN 10:06 0:00 -
root 11 0.0 0.0 0 0 ? - 10:06 0:00 [kswapd]
root - 0.0 - - - - S 10:06 0:00 -
root 12 0.0 0.0 0 0 ? - 10:06 0:00 [bdflush]
root - 0.0 - - - - S 10:06 0:00 -
root 13 0.0 0.0 0 0 ? - 10:06 0:00 [kupdated]
root - 0.0 - - - - S 10:06 0:00 -
root 26 0.0 0.0 0 0 ? - 10:06 0:00 [scsi_eh_0]
root - 0.0 - - - - S 10:06 0:00 -
root 66 0.0 0.0 0 0 ? - 10:06 0:00 [kjournald]
root - 0.0 - - - - S 10:06 0:00 -
root 85 0.0 0.0 2920 2036 ? - 10:06 0:00 /sbin/devfsd /dev
root - 0.0 - - - - Ss 10:06 0:00 -
root 190 0.0 0.0 0 0 ? - 10:06 0:00 [khubd]
root - 0.0 - - - - S 10:06 0:00 -
root 231 0.0 0.0 0 0 ? - 10:06 0:00 [kjournald]
root - 0.0 - - - - S 10:06 0:00 -
root 232 0.0 0.0 0 0 ? - 10:06 0:00 [kjournald]
root - 0.0 - - - - S 10:06 0:00 -
root 233 0.0 0.0 0 0 ? - 10:06 0:00 [kjournald]
root - 0.0 - - - - S 10:06 0:00 -
root 234 0.0 0.0 0 0 ? - 10:06 0:00 [kjournald]
root - 0.0 - - - - S 10:06 0:00 -
root 236 0.0 0.0 0 0 ? - 10:06 0:00 [xfsbufd]
root - 0.0 - - - - S 10:06 0:00 -
root 237 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/0]
root - 0.0 - - - - S 10:06 0:00 -
root 238 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/1]
root - 0.0 - - - - S 10:06 0:00 -
root 239 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/2]
root - 0.0 - - - - S 10:06 0:00 -
root 240 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/3]
root - 0.0 - - - - S 10:06 0:00 -
root 241 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/4]
root - 0.0 - - - - S 10:06 0:00 -
root 242 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/5]
root - 0.0 - - - - S 10:06 0:00 -
root 243 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/6]
root - 0.0 - - - - S 10:06 0:00 -
root 244 0.0 0.0 0 0 ? - 10:06 0:00 [xfslogd/7]
root - 0.0 - - - - S 10:06 0:00 -
root 245 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/0]
root - 0.0 - - - - S 10:06 0:00 -
root 246 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/1]
root - 0.0 - - - - S 10:06 0:00 -
root 247 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/2]
root - 0.0 - - - - S 10:06 0:00 -
root 248 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/3]
root - 0.0 - - - - S 10:06 0:00 -
root 249 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/4]
root - 0.0 - - - - S 10:06 0:00 -
root 250 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/5]
root - 0.0 - - - - S 10:06 0:00 -
root 251 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/6]
root - 0.0 - - - - S 10:06 0:00 -
root 252 0.0 0.0 0 0 ? - 10:06 0:00 [xfsdatad/7]
root - 0.0 - - - - S 10:06 0:00 -
root 253 0.0 0.0 0 0 ? - 10:06 0:00 [xfssyncd]
root - 0.0 - - - - S 10:06 0:00 -
root 254 0.0 0.0 0 0 ? - 10:06 0:01 [kjournald]
root - 0.0 - - - - S 10:06 0:01 -
daemon 436 0.0 0.0 1608 452 ? - 10:06 0:00 /sbin/portmap
daemon - 0.0 - - - - Ss 10:06 0:00 -
root 520 0.1 0.0 1544 624 ? - 10:07 0:01 /sbin/syslogd
root - 0.1 - - - - Ss 10:07 0:01 -
root 523 0.0 0.0 2176 1236 ? - 10:07 0:00 /sbin/klogd
root - 0.0 - - - - Ss 10:07 0:00 -
clamav 609 0.0 0.0 3032 1228 ? - 10:07 0:00 /usr/bin/freshclam -d --quiet
clamav - 0.0 - - - - Ss 10:07 0:00 -
root 619 0.0 0.0 5784 2664 ? - 10:07 0:00 /usr/sbin/cupsd -F
root - 0.0 - - - - Ss 10:07 0:00 -
message 622 0.0 0.0 2476 976 ? - 10:07 0:00 /usr/bin/dbus-daemon-1 --system
message - 0.0 - - - - Ss 10:07 0:00 -
root 634 0.0 0.0 2504 1072 ? - 10:07 0:00 /usr/sbin/lisa
root - 0.0 - - - - Ss 10:07 0:00 -
root 666 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - Ss 10:07 0:00 -
root 667 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - S 10:07 0:00 -
root 668 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - S 10:07 0:00 -
root 669 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - S 10:07 0:00 -
root 670 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - S 10:07 0:00 -
root 672 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - S 10:07 0:00 -
root 673 0.0 0.0 12356 912 ? - 10:07 0:00 /usr/sbin/nscd
root - 0.0 - - - - S 10:07 0:00 -
root 793 0.0 0.0 2888 1084 ? - 10:07 0:00 /usr/lib/postfix/master
root - 0.0 - - - - Ss 10:07 0:00 -
rwhod 799 0.0 0.0 1524 576 ? - 10:07 0:00 /usr/sbin/rwhod -b
rwhod - 0.0 - - - - Ss 10:07 0:00 -
rwhod 801 0.0 0.0 1524 636 ? - 10:07 0:00 /usr/sbin/rwhod -b
rwhod - 0.0 - - - - S 10:07 0:00 -
postfix 805 0.0 0.0 2964 1072 ? - 10:07 0:00 pickup -l -t fifo -u -c
postfix - 0.0 - - - - S 10:07 0:00 -
postfix 806 0.0 0.0 2996 1100 ? - 10:07 0:00 qmgr -l -t fifo -u -c
postfix - 0.0 - - - - S 10:07 0:00 -
root 808 0.0 0.0 4048 1568 ? - 10:07 0:00 /usr/sbin/sshd
root - 0.0 - - - - Ss 10:07 0:00 -
root 814 0.0 0.0 3048 820 ? - 10:07 0:00 /usr/sbin/vsftpd
root - 0.0 - - - - Ss 10:07 0:00 -
root 823 0.0 0.0 2084 840 ? - 10:07 0:00 /usr/sbin/xinetd -pidfile /var/run/xinetd.pid -stayalive
root - 0.0 - - - - Ss 10:07 0:00 -
root 824 0.0 0.0 1668 728 ? - 10:07 0:00 /sbin/rpc.statd
root - 0.0 - - - - Ss 10:07 0:00 -
daemon 829 0.0 0.0 1480 480 ? - 10:07 0:00 /usr/sbin/atd
daemon - 0.0 - - - - Ss 10:07 0:00 -
root 833 0.0 0.0 1568 692 ? - 10:07 0:00 /usr/sbin/cron
root - 0.0 - - - - Ss 10:07 0:00 -
root 843 0.0 0.0 4572 2108 ? - 10:07 0:00 /usr/sbin/apache
root - 0.0 - - - - S 10:07 0:00 -
www-data 845 0.0 0.0 4760 2200 ? - 10:07 0:00 /usr/sbin/apache
www-data - 0.0 - - - - S 10:07 0:00 -
www-data 846 0.0 0.0 4760 2200 ? - 10:07 0:00 /usr/sbin/apache
www-data - 0.0 - - - - S 10:07 0:00 -
www-data 847 0.0 0.0 4760 2200 ? - 10:07 0:00 /usr/sbin/apache
www-data - 0.0 - - - - S 10:07 0:00 -
www-data 848 0.0 0.0 4760 2200 ? - 10:07 0:00 /usr/sbin/apache
www-data - 0.0 - - - - S 10:07 0:00 -
www-data 849 0.0 0.0 4760 2200 ? - 10:07 0:00 /usr/sbin/apache
www-data - 0.0 - - - - S 10:07 0:00 -
root 860 0.0 0.0 1484 472 vc/1 - 10:07 0:00 /sbin/getty 38400 tty1
root - 0.0 - - - - Ss+ 10:07 0:00 -
root 861 0.0 0.0 1484 472 vc/2 - 10:07 0:00 /sbin/getty 38400 tty2
root - 0.0 - - - - Ss+ 10:07 0:00 -
root 862 0.0 0.0 1484 472 vc/3 - 10:07 0:00 /sbin/getty 38400 tty3
root - 0.0 - - - - Ss+ 10:07 0:00 -
root 863 0.0 0.0 1484 472 vc/4 - 10:07 0:00 /sbin/getty 38400 tty4
root - 0.0 - - - - Ss+ 10:07 0:00 -
root 864 0.0 0.0 1484 472 vc/5 - 10:07 0:00 /sbin/getty 38400 tty5
root - 0.0 - - - - Ss+ 10:07 0:00 -
root 865 0.0 0.0 1484 472 vc/6 - 10:07 0:00 /sbin/getty 38400 tty6
root - 0.0 - - - - Ss+ 10:07 0:00 -
root 872 0.0 0.0 6892 2084 ? - 10:07 0:00 sshd: root@pts/0
root - 0.0 - - - - Ss 10:07 0:00 -
root 874 0.0 0.0 3884 1844 pts/0 - 10:07 0:00 -zsh
root - 0.0 - - - - Ss 10:07 0:00 -
root 885 0.0 0.0 7212 2236 ? - 10:07 0:00 sshd: root@pts/1
root - 0.0 - - - - Ss 10:07 0:00 -
root 890 0.0 0.0 3884 1844 pts/1 - 10:08 0:00 -zsh
root - 0.0 - - - - Ss 10:08 0:00 -
urp 895 0.0 0.0 4352 2384 pts/1 - 10:08 0:00 -su
urp - 0.0 - - - - S 10:08 0:00 -
urp 944 0.0 0.0 3384 1916 pts/0 - 10:09 0:00 -su
urp - 0.0 - - - - S 10:09 0:00 -
urp 22762 90.9 0.0 12292 11304 pts/1 - 10:14 9:24 URPDispatcher
urp - 90.9 - - - - R 10:14 9:24 -
urp 22805 18.6 0.1 57636 31292 pts/1 - 10:14 1:55 bin/URPJobMaker
urp - 18.6 - - - - R 10:14 1:55 -
urp 22822 2.4 0.1 34996 23228 pts/1 - 10:14 0:14 bin/URPCappi
urp - 2.4 - - - - S 10:14 0:14 -
urp 22836 0.3 0.0 15320 12780 pts/1 - 10:14 0:02 bin/URPConvPPI
urp - 0.3 - - - - S 10:14 0:02 -
urp 22850 0.4 0.0 19584 12224 pts/1 - 10:14 0:02 bin/URPClogzPPI
urp - 0.4 - - - - S 10:14 0:02 -
urp 22865 0.6 0.0 20160 13168 pts/1 - 10:14 0:04 bin/URPVrPPI
urp - 0.6 - - - - S 10:14 0:04 -
urp 22880 2.4 0.1 26552 20420 pts/1 - 10:14 0:14 bin/URPEchoTop
urp - 2.4 - - - - S 10:14 0:14 -
urp 22897 0.7 0.0 15920 13140 pts/1 - 10:14 0:04 bin/URPMaxR
urp - 0.7 - - - - S 10:14 0:04 -
urp 22911 0.3 0.0 16512 13456 pts/1 - 10:14 0:02 bin/URPPrecip
urp - 0.3 - - - - S 10:14 0:02 -
urp 22925 1.0 0.1 21932 18908 pts/1 - 10:14 0:06 bin/URPSvrWx
urp - 1.0 - - - - S 10:14 0:06 -
urp 22939 0.0 0.0 10504 9032 pts/1 - 10:14 0:00 bin/URPRFA
urp - 0.0 - - - - S 10:14 0:00 -
urp 22953 2.3 0.1 42032 19324 pts/1 - 10:14 0:13 bin/URPPA
urp - 2.3 - - - - S 10:14 0:13 -
urp 22970 5.0 0.1 28532 21524 pts/1 - 10:14 0:29 bin/URPVVW
urp - 5.0 - - - - S 10:14 0:29 -
urp 23004 0.3 0.0 16704 13020 pts/1 - 10:14 0:01 bin/URPMesoCyclone
urp - 0.3 - - - - S 10:14 0:01 -
urp 23018 0.1 0.0 14668 11208 pts/1 - 10:14 0:00 bin/URPMicroBurst
urp - 0.1 - - - - S 10:14 0:00 -
urp 23032 0.0 0.0 12556 10424 pts/1 - 10:14 0:00 bin/URPGust
urp - 0.0 - - - - S 10:14 0:00 -
urp 23056 1.1 0.0 26996 14180 pts/1 - 10:14 0:06 bin/URPCellID
urp - 1.1 - - - - S 10:14 0:06 -
urp 23073 3.8 0.0 22948 13000 pts/1 - 10:14 0:22 bin/URPCellProps
urp - 3.8 - - - - S 10:14 0:22 -
urp 23089 0.0 0.0 10632 9068 pts/1 - 10:15 0:00 bin/URPTracker
urp - 0.0 - - - - S 10:15 0:00 -
urp 23103 0.3 0.0 17476 11940 pts/1 - 10:15 0:01 bin/URPPPI
urp - 0.3 - - - - S 10:15 0:01 -
urp 23147 1.9 0.1 23568 19508 pts/1 - 10:15 0:11 bin/URPHail
urp - 1.9 - - - - S 10:15 0:11 -
urp 23161 0.0 0.0 10828 9248 pts/1 - 10:15 0:00 bin/URPAssessmwin
urp - 0.0 - - - - S 10:15 0:00 -
urp 23178 1.3 0.0 25124 16312 pts/1 - 10:15 0:08 bin/URPNids
urp - 1.3 - - - - S 10:15 0:08 -
urp 23192 0.0 0.0 12520 9848 pts/1 - 10:15 0:00 bin/URPBWER
urp - 0.0 - - - - S 10:15 0:00 -
urp 23206 0.3 0.0 16660 11600 pts/1 - 10:15 0:02 bin/URPGradient
urp - 0.3 - - - - S 10:15 0:02 -
urp 23220 0.0 0.0 10724 9372 pts/1 - 10:15 0:00 bin/URPMultiRadarMerge
urp - 0.0 - - - - S 10:15 0:00 -
urp 23234 0.0 0.0 10696 9072 pts/1 - 10:15 0:00 bin/URPSAC
urp - 0.0 - - - - S 10:15 0:00 -
urp 23251 0.0 0.0 10500 9004 pts/1 - 10:15 0:00 bin/URPSCIT
urp - 0.0 - - - - S 10:15 0:00 -
urp 23265 0.4 0.0 10360 9188 pts/1 - 10:15 0:02 bin/URPIngester
urp - 0.4 - - - - S 10:15 0:02 -
root 24580 0.0 0.0 7048 2100 ? - 10:17 0:00 sshd: root@pts/2
root - 0.0 - - - - Ss 10:17 0:00 -
root 24601 0.0 0.0 4712 2132 pts/2 - 10:17 0:00 -zsh
root - 0.0 - - - - Ss 10:17 0:00 -
root 24623 0.0 0.0 2572 588 pts/2 - 10:17 0:00 tail -f urplog.log
root - 0.0 - - - - S+ 10:17 0:00 -
urp 26020 0.0 0.0 2024 500 pts/0 - 10:24 0:00 sleep 2
urp - 0.0 - - - - S+ 10:24 0:00 -
urp 26021 0.0 0.0 2292 728 pts/1 - 10:24 0:00 ps auxm
urp - 0.0 - - - - R+ 10:24 0:00 -
top - 10:45:13 up 38 min, 3 users, load average: 1.31, 1.40, 1.30
Tasks: 119 total, 5 running, 114 sleeping, 0 stopped, 0 zombie
Cpu0 : 100.0% user, 0.0% system, 0.0% nice, 0.0% idle
Cpu1 : 1.3% user, 0.0% system, 0.0% nice, 98.7% idle
Cpu2 : 0.7% user, 0.0% system, 0.0% nice, 99.3% idle
Cpu3 : 32.3% user, 0.0% system, 0.0% nice, 67.7% idle
Cpu4 : 0.7% user, 0.3% system, 0.0% nice, 99.0% idle
Cpu5 : 0.0% user, 0.0% system, 0.0% nice, 100.0% idle
Cpu6 : 0.0% user, 0.0% system, 0.0% nice, 100.0% idle
Cpu7 : 0.0% user, 0.0% system, 0.0% nice, 100.0% idle
Mem: 16567608k total, 3797076k used, 12770532k free, 442836k buffers
Swap: 2048276k total, 0k used, 2048276k free, 2690988k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31337 urp 18 0 11308 11m 548 R 99.9 0.1 7:53.39 URPDispatcher
31373 urp 9 0 32296 31m 612 S 31.0 0.2 2:05.00 URPJobMaker
31667 urp 9 0 14132 13m 716 S 1.0 0.1 0:03.81 URPCellID
31838 urp 9 0 11632 11m 704 S 1.0 0.1 0:01.47 URPGradient
885 root 9 0 2236 2236 1732 S 0.3 0.0 0:00.22 sshd
31424 urp 18 0 12228 11m 11m S 0.3 0.1 0:01.32 URPClogzPPI
31444 urp 10 0 12696 12m 700 R 0.3 0.1 0:02.46 URPVrPPI
580 urp 10 0 992 992 760 R 0.3 0.0 0:00.55 top
1205 urp 16 0 13420 13m 11m R 0.3 0.1 0:00.01 URPClogzPPI
1206 urp 18 0 13420 13m 11m R 0.3 0.1 0:00.01 URPClogzPPI
1 root 8 0 500 500 448 S 0.0 0.0 0:04.38 init
2 root 9 0 0 0 0 S 0.0 0.0 0:00.00 keventd
3 root 18 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU0
4 root 18 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU1
5 root 19 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU2
6 root 18 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU3
7 root 18 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU4
8 root 19 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU5
9 root 18 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU6
10 root 18 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd_CPU7
11 root 9 0 0 0 0 S 0.0 0.0 0:00.03 kswapd
12 root 9 0 0 0 0 S 0.0 0.0 0:00.00 bdflush
13 root 9 0 0 0 0 S 0.0 0.0 0:00.11 kupdated
26 root 9 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
66 root 9 0 0 0 0 S 0.0 0.0 0:00.04 kjournald
85 root 9 0 2036 2036 648 S 0.0 0.0 0:00.02 devfsd
190 root 9 0 0 0 0 S 0.0 0.0 0:00.00 khubd
231 root 9 0 0 0 0 S 0.0 0.0 0:00.00 kjournald
232 root 9 0 0 0 0 S 0.0 0.0 0:00.00 kjournald
233 root 9 0 0 0 0 S 0.0 0.0 0:00.01 kjournald
234 root 9 0 0 0 0 S 0.0 0.0 0:00.21 kjournald
236 root 9 0 0 0 0 S 0.0 0.0 0:00.00 xfsbufd
237 root 9 0 0 0 0 S 0.0 0.0 0:00.00 xfslogd/0
238 root 9 0 0 0 0 S 0.0 0.0 0:00.00 xfslogd/1
239 root 9 0 0 0 0 S 0.0 0.0 0:00.00 xfslogd/2
240 root 9 0 0 0 0 S 0.0 0.0 0:00.00 xfslogd/3
241 root 9 0 0 0 0 S 0.0 0.0 0:00.00 xfslogd/4