The document provides an overview of common Linux performance analysis tools including top, mpstat, ps, sar, and others. It discusses how these tools can be used to monitor CPU, memory, disk, network, and process-level performance metrics. Examples are given showing output from top displaying real-time process activity, mpstat showing CPU utilization breakdown, ps displaying currently running processes, and sar historical CPU utilization. The tools can help identify potential performance issues like high CPU usage, memory pressures, and disk or network bottlenecks.

1. Linux Performance Analysis
Will Sterling
Linux & UNIX Consultant
@
PARSEC Group

2. Agenda
Performance Analysis Tools
•
Performance Profiling Tools
•
Identifying Common Performance Issues
•
Linux Performance Analysis

3. Performance Analysis Tools
CPU Memory Process Network
top top ps nettop
mpstat vmstat nmon sar
ps ps Disk ethtool
sar sar nmon nmon
/proc Free Sar
nmon /proc Iostat
nmon vmstat
Linux Performance Analysis

4. Top
top ­ 01:45:25 up 12 min, 1 user, load average: 12.33, 7.84, 3.42
Tasks: 141 total, 14 running, 117 sleeping, 0 stopped, 10 zombie
Cpu(s): 93.4%us, 6.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8191996k total, 267248k used, 7924748k free, 10532k buffers
Swap: 0k total, 0k used, 0k free, 106008k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23202 root 20 0 13060 992 620 R 12.3 0.0 0:00.37 bc
23210 root 20 0 13060 992 620 R 12.3 0.0 0:00.37 bc
23218 root 20 0 13060 992 620 R 9.6 0.0 0:00.29 bc
23226 root 20 0 13060 992 620 R 9.6 0.0 0:00.29 bc
23234 root 20 0 13060 992 620 R 8.0 0.0 0:00.24 bc
23242 root 20 0 13060 944 620 R 6.3 0.0 0:00.19 bc
23250 root 20 0 13060 928 620 R 4.3 0.0 0:00.13 bc
23258 root 20 0 13060 892 620 R 3.3 0.0 0:00.10 bc
2519 root 20 0 104m 1652 1272 S 2.0 0.0 0:05.85 bc.ksh
23266 root 20 0 12928 860 620 R 1.7 0.0 0:00.05 bc
23274 root 20 0 12928 852 620 R 1.3 0.0 0:00.04 bc
1 root 20 0 19400 1508 1192 S 0.0 0.0 0:01.20 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.02 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.07 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
Linux Performance Analysis

5. Top Cont.
top ­ 01:59:43 up 26 min, 1 user, load average: 0.63, 2.83, 4.08
Tasks: 121 total, 3 running, 118 sleeping, 0 stopped, 0 zombie
Cpu(s): 43.3%us, 8.7%sy, 0.0%ni, 48.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8191996k total, 266016k used, 7925980k free, 11944k buffers
Swap: 0k total, 0k used, 0k free, 108088k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24222 root 20 0 103m 928 780 R 98.8 0.0 0:03.16 awk
17758 root 20 0 104m 1572 1264 S 7.9 0.0 0:40.30 bc.ksh
1 root 20 0 19400 1508 1192 S 0.0 0.0 0:01.54 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.83 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.07 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:00.76 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.06 migration/1
9 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
11 root RT 0 0 0 0 S 0.0 0.0 0:00.80 migration/2
12 root RT 0 0 0 0 S 0.0 0.0 0:00.02 migration/2
13 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/2
14 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
15 root RT 0 0 0 0 S 0.0 0.0 0:00.80 migration/3
Linux Performance Analysis

6. mpstat
[root@server­­­62012­04­29­083219­0000 ~]# mpstat ­P ALL 5 2
Linux 2.6.32­220.2.1.el6.x86_64 (server­­­62012­04­29­083219­0000.localdomain)
04/29/2012 _x86_64_ (4 CPU)
02:24:22 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest
%idle
02:24:27 AM all 26.32 0.00 0.05 0.00 0.00 0.00 0.00 0.00
73.63
02:24:27 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
100.00
02:24:27 AM 1 0.00 0.00 0.20 0.00 0.00 0.00 0.00 0.00
99.80
02:24:27 AM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
100.00
02:24:27 AM 3 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00
Linux Performance Analysis

7. mpstat
[root@server­­­62012­04­29­083219­0000 ~]# mpstat ­P ALL 5 2
Linux 2.6.32­220.2.1.el6.x86_64 (server­­­62012­04­29­083219­0000.localdomain)
04/29/2012 _x86_64_ (4 CPU)
02:24:22 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest
%idle
02:24:27 AM all 26.32 0.00 0.05 0.00 0.00 0.00 0.00 0.00
73.63
02:24:27 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
100.00
02:24:27 AM 1 0.00 0.00 0.20 0.00 0.00 0.00 0.00 0.00
99.80
02:24:27 AM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
100.00
02:24:27 AM 3 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00
Linux Performance Analysis

8. ps aux
[root@server­­­62012­04­29­083219­0000 ~]# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 19400 1508 ? Ss 01:33 0:01 /sbin/init
root 2 0.0 0.0 0 0 ? S 01:33 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 01:33 0:01 [migration/0]
root 4 0.0 0.0 0 0 ? S 01:33 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S 01:33 0:00 [migration/0]
root 6 0.0 0.0 0 0 ? S 01:33 0:00 [watchdog/0]
­­snip­­
root 1295 0.0 0.0 97816 3836 ? S 01:34 0:00 sshd: root@pts/
root 1298 0.0 0.0 108356 1908 pts/0 Ss 01:34 0:00 ­bash
root 7420 0.0 0.0 17336 588 ? Ss 02:01 0:00 /usr/sbin/anacr
root 24143 0.0 0.0 110396 1148 pts/0 R+ 02:33 0:00 ps aux
root 24138 88.5 0.0 105996 928 pts/0 R 02:32 0:12 awk BEGIN {for(
Linux Performance Analysis

9. Sar: CPU
[root@server­­­62012­04­29­083219­0000 ~]# sar ­u
Linux 2.6.32­220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM CPU %user %nice %system %iowait %steal %idle
01:50:01 PM all 83.58 0.00 5.92 0.00 0.00 10.50
02:00:01 PM all 30.32 0.00 18.20 0.01 0.00 51.48
02:10:01 PM all 7.97 0.00 15.11 0.02 0.00 76.90
02:20:01 PM all 1.55 0.00 0.02 0.00 0.00 98.42
02:30:01 PM all 1.61 0.00 0.03 0.00 0.00 98.36
02:40:01 PM all 1.83 0.00 0.03 0.00 0.00 98.14
Average: all 21.24 0.00 6.19 0.00 0.00 72.56
Linux Performance Analysis

10. SAR: CPU
[root@server­­­62012­04­29­083219­0000 ~]# sar ­u
Linux 2.6.32­220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM CPU %user %nice %system %iowait %steal %idle
01:50:01 PM all 83.58 0.00 5.92 0.00 0.00 10.50
02:00:01 PM all 30.32 0.00 18.20 0.01 0.00 51.48
02:10:01 PM all 7.97 0.00 15.11 0.02 0.00 76.90
02:20:01 PM all 1.55 0.00 0.02 0.00 0.00 98.42
02:30:01 PM all 1.61 0.00 0.03 0.00 0.00 98.36
02:40:01 PM all 1.83 0.00 0.03 0.00 0.00 98.14
Average: all 21.24 0.00 6.19 0.00 0.00 72.56
Linux Performance Analysis

11. SAR: Memory
[root@server­­­62012­04­29­083219­0000 ~]# sar ­r
Linux 2.6.32­220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit
01:50:01 PM 7930632 261364 3.19 10940 106008 64096 0.78
02:00:01 PM 7926756 265240 3.24 11976 108088 66280 0.81
02:10:01 PM 7926956 265040 3.24 12696 108488 64652 0.79
02:20:01 PM 7926700 265296 3.24 12824 108580 64652 0.79
02:30:01 PM 7925568 266428 3.25 12980 108624 65824 0.80
02:40:01 PM 7926460 265536 3.24 13076 108628 64652 0.79
02:50:01 PM 7925964 266032 3.25 13196 108776 64652 0.79
03:00:01 PM 7925964 266032 3.25 13344 108788 64652 0.79
03:10:01 PM 7925732 266264 3.25 13480 108832 64660 0.79
03:20:01 PM 7913020 278976 3.41 13608 109444 80496 0.98
Average: 7925375 266621 3.25 12812 108426 66462 0.81
Linux Performance Analysis

12. SAR: Disk
[root@server­­­62012­04­29­083219­0000 ~]# sar ­d
Linux 2.6.32­220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM DEV tps rd_sec/s wr_sec/s avgrq­sz avgqu­sz await svctm %util
01:50:01 PM dev252­0 0.47 0.75 12.86 28.91 0.00 1.01 0.94 0.04
01:50:01 PM dev252­16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:00:01 PM dev252­0 0.74 7.79 16.17 32.30 0.00 0.73 0.50 0.04
­­snip­­
[root@server­­­62012­04­29­083219­0000 ~]# ls ­l /dev/vd*
brw­rw­­­­ 1 root disk 252, 0 Apr 29 01:33 /dev/vda
brw­rw­­­­ 1 root disk 252, 16 Apr 29 01:33 /dev/vdb
Linux Performance Analysis

13. Sar: Network
[root@server­­­62012­04­29­083219­0000 ~]# sar ­n DEV
Linux 2.6.32­220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
01:50:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:50:01 PM eth0 1.19 1.02 0.08 0.58 0.00 0.00 0.00
02:00:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:00:01 PM eth0 1.30 0.83 0.10 0.27 0.00 0.00 0.00
02:10:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:10:01 PM eth0 1.47 1.16 0.11 0.18 0.00 0.00 0.00
02:20:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:20:01 PM eth0 0.40 0.20 0.03 0.04 0.00 0.00 0.00
02:30:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:30:01 PM eth0 0.80 0.53 0.07 0.16 0.00 0.00 0.00
02:40:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:40:01 PM eth0 0.64 0.42 0.05 0.27 0.00 0.00 0.00
Linux Performance Analysis

14. /proc: CPU
[root@server­­­62012­04­29­083219­0000 proc]# cat cpuinfo
processor : 0
­­snip­­
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 2
model name : QEMU Virtual CPU version 0.14.0
stepping : 3
cpu MHz : 2666.760
cache size : 4096 KB
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pse36 clflush
mmx fxsr sse sse2 syscall nx lm rep_good unfair_spinlock pni cx16 popcnt
hypervisor lahf_lm
bogomips : 5333.52
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
Linux Performance Analysis

15. /proc: /CPU
[root@server­­­62012­04­29­083219­0000 proc]# cat stat
cpu 295757 0 86819 1581480 160 49 190 0 0
cpu0 70616 0 22135 398368 47 40 55 0 0
cpu1 74539 0 21982 394194 23 0 47 0 0
cpu2 70228 0 21854 399389 42 6 46 0 0
Cpu3 80372 0 20846 389528 46 2 40 0 0
Total cycles spent: USER NICE SYSTEM IDLE IOWAIT
Linux Performance Analysis

16. nmon: CPU
http://nmon.sourceforge.net/pmwiki.php
Linux Performance Analysis

17. Nmon: Memory/Disk/Network
Linux Performance Analysis

18. ksar
Linux Performance Analysis

19. Common Performance Issues
Network Bottle Neck to NAS
•
Slow IO to Virtual Drives
•
Memory Paging
•
CPU
•
Processes Waiting on Upstream Applications
•
Miss-configured Storage
•
Linux Performance Analysis