200.1,2-Capacity Planning

Object 1: Measure and
Troubleshoot Resource Usage

4
System resources
CPU
Memory I/O

5
Linux performance metrics
● Processor metrics
● Memory metrics
● Block device metrics
● Network interface metrics

6
Monitoring tools
●‫نیاز‬ ‫مورد‬ ‫های‬ ‫بسته‬
● sysstat
● iostat, sar, mpstat, ...
● procps
● free, uptime, vmstat, w, ...

10
Process management‫ادامه‬
● Context switching
● Interrupt handling
● Process state (runing,rtopped,intruptible,unintruptible,zombie)
fork()
TASK_STOPPED
TASK_UNINTERRUPTIBLE
TASK_INTERRUPTIBLE
TASK_RUNNING
(ready)
TASK_ZOMBIE
exit()
Processor
TASK_RUNNING

11
Processor metrics
● CPU utilization (Exceed 80%=bottleneck)
● User time, System time,Waiting, Idle time, Nice time
● Load avarage
● Runable processes (Excedd 10 time the amount of prcessors)
● Blocked (Uninterruptible: like process waiting for I/O)
● Context switch, Interrupts(software bottleneck)

13
vmstat
$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs
us sy id wa
1 0 262008 544552 21036 392400 1 4 173 142 136 332
15 5 77 2
$ vmstat
----cpu----
r b swpd free buff cache si so bi bo in cs
us sy id wa
1 0 262008 544552 21036 392400 1 4 173 142 136 332
15 5 77 2
Process (procs) r: The number of processes waiting for runtime
b: The number of processes in uninterruptable
sleep
Memory swpd: The amount of virtual memory used (KB)
free: The amount of idle memory (KB)
buff: The amount of memory used as buffers (KB)
cache: The amount of memory used as cache (KB)
Swap si: Amount of memory swapped from the disk (Kbps)
so: Amount of memory swapped to the disk (Kbps)
IO bi: Blocks sent to a block device (blocks/s)
bo: Blocks received from a block device (blocks/s)
System in: The number of interrupts per second, including the
clock
cs: The number of context switches per second
Process (procs) r: The number of processes waiting for runtime
b: The number of processes in uninterruptable
sleep
Memory swpd: The amount of virtual memory used (KB)
free: The amount of idle memory (KB)
buff: The amount of memory used as buffers (KB)
cache: The amount of memory used as cache (KB)
Swap si: Amount of memory swapped from the disk (Kbps)
so: Amount of memory swapped to the disk (Kbps)
IO bi: Blocks sent to a block device (blocks/s)
bo: Blocks received from a block device (blocks/s)
System in: The number of interrupts per second, including the
clock
cs: The number of context switches per second

14
sar
$ sar -q
02:55:02 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
blocked
03:05:01 PM 0 566 1.83 2.01 2.16
0
$ sar -q
02:55:02 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
blocked
03:05:01 PM 0 566 1.83 2.01 2.16
0
$ sar -P ALL
Average: CPU %user %nice %system %iowait
%steal %idle
Average: all 15.21 0.03 5.20 1.74
0.00 77.81
Average: 0 16.22 0.03 5.25 1.95
0.00 76.54
Average: 1 16.14 0.03 5.38 1.97
0.00 76.48
Average: 2 14.24 0.04 5.02 1.52
0.00 79.18
Average: 3 14.24 0.03 5.16 1.51
0.00 79.06
$ sar -P ALL
Average: CPU %user %nice %system %iowait
%steal %idle
Average: all 15.21 0.03 5.20 1.74
0.00 77.81
Average: 0 16.22 0.03 5.25 1.95
0.00 76.54
Average: 1 16.14 0.03 5.38 1.97
0.00 76.48
Average: 2 14.24 0.04 5.02 1.52
0.00 79.18
Average: 3 14.24 0.03 5.16 1.51
0.00 79.06

15
sar
$ sar -w
Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU)
01:51:22 AM LINUX RESTART
01:55:01 AM proc/s cswch/s
02:05:01 AM 1.14 7087.09
02:15:01 AM 1.19 6955.23
02:25:01 AM 1.12 6855.21
02:35:01 AM 1.21 6311.18
02:45:01 AM 1.20 6006.53
$ sar -w
01:55:01 AM proc/s cswch/s
02:05:01 AM 1.14 7087.09
02:15:01 AM 1.19 6955.23
02:25:01 AM 1.12 6855.21
02:35:01 AM 1.21 6311.18
02:45:01 AM 1.20 6006.53

16
Other command
$ mpstat -P ALL
04:50:28 PM CPU %usr %nice %sys %iowait %irq
%soft %steal %guest %idle
04:50:28 PM all 14.94 0.03 4.62 1.80 0.01 0.47
0.00 0.00 78.13
$ mpstat -P ALL
04:50:28 PM CPU %usr %nice %sys %iowait %irq
%soft %steal %guest %idle
04:50:28 PM all 14.94 0.03 4.62 1.80 0.01 0.47
0.00 0.00 78.13
$ iostat -x
avg-cpu: %user %nice %system %iowait %steal %idle
14.92 0.03 5.10 1.79 0.00 78.15
$ iostat -x
14.92 0.03 5.10 1.79 0.00 78.15

17
Linux memory architecture
● Physical and virtual memory
● Virtual memory manager
● Page frame
● Page frame reclaiming (Swap ,...)
● Buddy system

18
Memory metrics
● Free memory
● Swap usage
● Buffer and caches
● Slabs (kernel usage of memory)
● Active versus inactive memory
● Inactive memory is a likely candidate to be swapped out to disk

19
free, top, vmstat
$ top -b -n 1 | head -5
top - 16:54:00 up 5:39, 3 users, load average: 0.93, 0.65,
0.58
Tasks: 221 total, 1 running, 220 sleeping, 0 stopped, 0
zombie
%Cpu(s): 14.9 us, 4.6 sy, 0.0 ni, 78.2 id, 1.8 wa, 0.0 hi,
0.5 si, 0.0 st
KiB Mem: 2051244 total, 1600512 used, 450732 free, 29236
buffers
KiB Swap: 4954776 total, 254804 used, 4699972 free, 445900
cached
$ top -b -n 1 | head -5
top - 16:54:00 up 5:39, 3 users, load average: 0.93, 0.65,
0.58
Tasks: 221 total, 1 running, 220 sleeping, 0 stopped, 0
zombie
%Cpu(s): 14.9 us, 4.6 sy, 0.0 ni, 78.2 id, 1.8 wa, 0.0 hi,
0.5 si, 0.0 st
KiB Mem: 2051244 total, 1600512 used, 450732 free, 29236
buffers
KiB Swap: 4954776 total, 254804 used, 4699972 free, 445900
cached
$ vmstat -a
----cpu----
r b swpd free inact active si so bi bo in cs
us sy id wa
0 1 254804 453648 613696 844384 1 3 155 129 309 480
15 5 78 2
$ vmstat -a
----cpu----
r b swpd free inact active si so bi bo in cs
us sy id wa
0 1 254804 453648 613696 844384 1 3 155 129 309 480
15 5 78 2

20
sar
$ sar -r
01:55:01 AM kbmemfree kbmemused %memused kbbuffers kbcached
kbcommit %commit kbactive kbinact
02:05:01 AM 289476 1761768 85.89 75248 547476
3017772 43.07 1141260 480772
$ sar -r
01:55:01 AM kbmemfree kbmemused %memused kbbuffers kbcached
kbcommit %commit kbactive kbinact
02:05:01 AM 289476 1761768 85.89 75248 547476
3017772 43.07 1141260 480772

25
Block device metrics
● Iowait
● Average queue length
● Average wait
● Transfers per second
● Blocks read/write per second
● Kilobytes per second read/write

26
iostat
$ iostat -p ALL 1 1$ iostat -p ALL 1 1
$ iostat -x 1 1
8.62 0.03 2.61 2.15 0.00 86.60
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.44 5.36 3.35 4.37 196.43 59.67
66.38 0.25 32.64 12.89 47.76 6.00 4.63
$ iostat -x 1 1
8.62 0.03 2.61 2.15 0.00 86.60
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.44 5.36 3.35 4.37 196.43 59.67
66.38 0.25 32.64 12.89 47.76 6.00 4.63

27
sar
$ sar -b
Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4
CPU)
12:34:50 PM LINUX RESTART
12:35:04 PM tps rtps wtps bread/s bwrtn/s
12:45:01 PM 29.74 21.93 7.81 1389.04 292.90
12:55:01 PM 5.90 1.34 4.56 32.94 97.90
01:05:01 PM 20.23 13.89 6.35 760.67 698.30
01:15:01 PM 83.81 72.50 11.31 1943.13 1598.73
01:25:01 PM 5.55 1.47 4.08 39.11 100.80
$ sar -b
CPU)
12:35:04 PM tps rtps wtps bread/s bwrtn/s
12:45:01 PM 29.74 21.93 7.81 1389.04 292.90
12:55:01 PM 5.90 1.34 4.56 32.94 97.90
01:05:01 PM 20.23 13.89 6.35 760.67 698.30
01:15:01 PM 83.81 72.50 11.31 1943.13 1598.73
01:25:01 PM 5.55 1.47 4.08 39.11 100.80

28
sar
$ sar -d
CPU)
12:35:04 PM DEV tps rd_sec/s wr_sec/s avgrq-sz
avgqu-sz await svctm %util
12:45:01 PM dev8-0 29.74 1389.04 292.90 56.55
0.96 33.07 5.69 16.93
12:55:01 PM dev8-0 5.90 32.94 97.90 22.17
0.07 11.91 5.40 3.19
01:05:01 PM dev8-0 20.23 760.67 698.30 72.10
0.43 21.23 3.50 7.09
$ sar -d
CPU)
12:35:04 PM DEV tps rd_sec/s wr_sec/s avgrq-sz
avgqu-sz await svctm %util
12:45:01 PM dev8-0 29.74 1389.04 292.90 56.55
0.96 33.07 5.69 16.93
12:55:01 PM dev8-0 5.90 32.94 97.90 22.17
0.07 11.91 5.40 3.19
01:05:01 PM dev8-0 20.23 760.67 698.30 72.10
0.43 21.23 3.50 7.09
$ ls -l /dev/sda
brw-rw---T 1 root disk 8, 0 Apr 23 12:34 /dev/sda
$ ls -l /dev/sda
brw-rw---T 1 root disk 8, 0 Apr 23 12:34 /dev/sda

31
Network interface metrics
● Packets received and sent, Bytes received and sent
● Collisions per second
● Packets dropped
● Overruns (ran out of buffer space)
● Errors

32
ifconfig
$ ifconfig eth0
wlan0 Link encap:Ethernet HWaddr 00:08:ca:45:d0:76
inet addr:192.168.13.51 Bcast:192.168.13.255
Mask:255.255.255.0
inet6 addr: fe80::208:caff:fe45:d076/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:272597 errors:0 dropped:0 overruns:0 frame:0
TX packets:259457 errors:0 dropped:0 overruns:0
carrier:0
collisions:0 txqueuelen:1000
RX bytes:350358017 (334.1 MiB) TX bytes:48277958 (46.0
MiB)
$ ifconfig eth0
wlan0 Link encap:Ethernet HWaddr 00:08:ca:45:d0:76
inet addr:192.168.13.51 Bcast:192.168.13.255
Mask:255.255.255.0
inet6 addr: fe80::208:caff:fe45:d076/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:272597 errors:0 dropped:0 overruns:0 frame:0
TX packets:259457 errors:0 dropped:0 overruns:0
carrier:0
collisions:0 txqueuelen:1000
RX bytes:350358017 (334.1 MiB) TX bytes:48277958 (46.0
MiB)

33
sar
$ sar -n DEV
CPU)
12:35:04 PM IFACE rxpck/s txpck/s rxkB/s txkB/s
rxcmp/s txcmp/s rxmcst/s
12:45:01 PM lo 0.07 0.07 0.01 0.01
0.00 0.00 0.00
12:45:01 PM wlan0 12.83 12.76 13.38 2.39
0.00 0.00 0.00
12:45:01 PM kvnet 7.97 8.28 8.10 1.10
0.00 0.00 0.00
12:45:01 PM eth0 0.00 0.00 0.00 0.00
0.00 0.00 0.00
$ sar -n DEV
CPU)
12:35:04 PM IFACE rxpck/s txpck/s rxkB/s txkB/s
rxcmp/s txcmp/s rxmcst/s
12:45:01 PM lo 0.07 0.07 0.01 0.01
0.00 0.00 0.00
12:45:01 PM wlan0 12.83 12.76 13.38 2.39
0.00 0.00 0.00
12:45:01 PM kvnet 7.97 8.28 8.10 1.10
0.00 0.00 0.00
12:45:01 PM eth0 0.00 0.00 0.00 0.00
0.00 0.00 0.00

34
netstat
# netstat -antup
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 192.168.13.51:80 192.168.13.52:53930
ESTABLISHED 27398/apache2
tcp 0 0 192.168.13.51:80 192.168.13.52:53931
tcp 0 0 192.168.13.51:80 192.168.13.52:53930
TIME_WAIT -
tcp 0 0 192.168.13.60:80 192.168.13.52:53931
SYN_RECV 27559/apache2
# netstat -antup
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 192.168.13.51:80 192.168.13.52:53930
tcp 0 0 192.168.13.51:80 192.168.13.52:53931
tcp 0 0 192.168.13.51:80 192.168.13.52:53930
TIME_WAIT -
tcp 0 0 192.168.13.60:80 192.168.13.52:53931
SYN_RECV 27559/apache2
# netstat -s# netstat -s

Object 2: Predict Future
Resource Needs

36
Monitor resource usage
Monitoring
Solution
Collectd
Munin
Nagios
Cacti
Mrtg

37
Collectd
●‫نیاز‬ ‫مورد‬ ‫های‬ ‫بسته‬ ‫نصب‬
●‫ها‬‫مسیر‬
# apt-get install collectd librrds-perl libconfig-general-perl
libhtml-parser-perl libregexp-common-perl libjson-perl
# apt-get install collectd librrds-perl libconfig-general-perl
libhtml-parser-perl libregexp-common-perl libjson-perl
Config file: /etc/collectd/{collectd.conf,collection.conf}
Plugins : /usr/lib/collectd
Data: /var/lib/collectd
Config file: /etc/collectd/{collectd.conf,collection.conf}
Plugins : /usr/lib/collectd
Data: /var/lib/collectd

38
Setting Options
● Global options (Interval, FQDNLookup, …)
● Plugin options (documented on man collectd.conf )“ ”
BaseDir "/var/lib/collectd"
PIDFile "/run/collectd.pid"
Interval 10.0 (if changed delete all rrd files)
LoadPlugin cpu
LoadPlugin df
<LoadPlugin df>
Interval 3600
</LoadPlugin>
BaseDir "/var/lib/collectd"
PIDFile "/run/collectd.pid"
Interval 10.0 (if changed delete all rrd files)
LoadPlugin cpu
LoadPlugin df
<LoadPlugin df>
Interval 3600
</LoadPlugin>

39
Plugins
● Plugins
● Input (read) Plugins (CPU, Memory)
● Output (write) Plugins (RRDtool,CSV)
● Read and Write Plugins (Network)
● Log Plugins (LogFile,Syslog)
# vi /etc/collectd/collectd.conf
...
LoadPlugin apache
LoadPlugin cpu
LoadPlugin df
LoadPlugin network
LoadPlugin interface
# vi /etc/collectd/collectd.conf
...
LoadPlugin apache
LoadPlugin cpu
LoadPlugin df
LoadPlugin network
LoadPlugin interface

41
Creating Graphs
● Debian
● Collection3
# apt-get install apache2-mpm-worker (Wheezy)# apt-get install apache2-mpm-worker (Wheezy)
# cp -r /usr/share/doc/collectd/examples/collection3 /var/www/# cp -r /usr/share/doc/collectd/examples/collection3 /var/www/
# apt-get install apache2 (Jessie)# apt-get install apache2 (Jessie)
# a2enmod cgi
# vi /etc/apache2/sites-enabled/default
<Directory /var/www/collection3>
Options +ExecCgi
AddHandler cgi-script .cgi .pl
</Directory>
# a2enmod cgi
# vi /etc/apache2/sites-enabled/default
<Directory /var/www/collection3>
Options +ExecCgi
AddHandler cgi-script .cgi .pl
</Directory>

200.1,2-Capacity Planning

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to 200.1,2-Capacity Planning

Similar to 200.1,2-Capacity Planning (20)

More from behrad eslamifar

More from behrad eslamifar (15)

200.1,2-Capacity Planning