Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Topic 200: Capacity Planning
Object 1: Measure and
Troubleshoot Resource Usage
4
System resources
CPU
Memory I/O
5
Linux performance metrics
● Processor metrics
● Memory metrics
● Block device metrics
● Network interface metrics
6
Monitoring tools
●‫نیاز‬ ‫مورد‬ ‫های‬ ‫بسته‬
● sysstat
● iostat, sar, mpstat, ...
● procps
● free, uptime, vmstat, w, ...
10
Process management‫ادامه‬
● Context switching
● Interrupt handling
● Process state (runing,rtopped,intruptible,unintrup...
11
Processor metrics
● CPU utilization (Exceed 80%=bottleneck)
● User time, System time,Waiting, Idle time, Nice time
● Lo...
13
vmstat
$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache ...
14
sar
$ sar -q
02:55:02 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
blocked
03:05:01 PM 0 566 1.83 2.01 2.16
0
$ sar -q
...
15
sar
$ sar -w
Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU)
01:51:22 AM LINUX RESTART
01:55:01 AM proc/s cswc...
16
Other command
$ mpstat -P ALL
Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU)
04:50:28 PM CPU %usr %nice %sys ...
17
Linux memory architecture
● Physical and virtual memory
● Virtual memory manager
● Page frame
● Page frame reclaiming (...
18
Memory metrics
● Free memory
● Swap usage
● Buffer and caches
● Slabs (kernel usage of memory)
● Active versus inactive...
19
free, top, vmstat
$ top -b -n 1 | head -5
top - 16:54:00 up 5:39, 3 users, load average: 0.93, 0.65,
0.58
Tasks: 221 to...
20
sar
$ sar -r
Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU)
01:51:22 AM LINUX RESTART
01:55:01 AM kbmemfree k...
25
Block device metrics
● Iowait
● Average queue length
● Average wait
● Transfers per second
● Blocks read/write per seco...
26
iostat
$ iostat -p ALL 1 1$ iostat -p ALL 1 1
$ iostat -x 1 1
Linux 3.2.0-4-amd64 (laptop) 05/29/2014 _x86_64_ (4 CPU)
...
27
sar
$ sar -b
Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4
CPU)
12:34:50 PM LINUX RESTART
12:35:04 PM ...
28
sar
$ sar -d
Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4
CPU)
12:34:50 PM LINUX RESTART
12:35:04 PM ...
31
Network interface metrics
● Packets received and sent, Bytes received and sent
● Collisions per second
● Packets droppe...
32
ifconfig
$ ifconfig eth0
wlan0 Link encap:Ethernet HWaddr 00:08:ca:45:d0:76
inet addr:192.168.13.51 Bcast:192.168.13.25...
33
sar
$ sar -n DEV
Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4
CPU)
12:34:50 PM LINUX RESTART
12:35:04...
34
netstat
# netstat -antup
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreig...
Object 2: Predict Future
Resource Needs
36
Monitor resource usage
Monitoring
Solution
Collectd
Munin
Nagios
Cacti
Mrtg
37
Collectd
●‫نیاز‬ ‫مورد‬ ‫های‬ ‫بسته‬ ‫نصب‬
●‫ها‬‫مسیر‬
# apt-get install collectd librrds-perl libconfig-general-perl
l...
38
Setting Options
● Global options (Interval, FQDNLookup, …)
● Plugin options (documented on man collectd.conf )“ ”
BaseD...
39
Plugins
● Plugins
● Input (read) Plugins (CPU, Memory)
● Output (write) Plugins (RRDtool,CSV)
● Read and Write Plugins ...
41
Creating Graphs
● Debian
● Collection3
# apt-get install apache2-mpm-worker (Wheezy)# apt-get install apache2-mpm-worke...
200.1,2-Capacity Planning
Upcoming SlideShare
Loading in …5
×

200.1,2-Capacity Planning

46 views

Published on

  • Be the first to comment

  • Be the first to like this

200.1,2-Capacity Planning

  1. 1. Topic 200: Capacity Planning
  2. 2. Object 1: Measure and Troubleshoot Resource Usage
  3. 3. 4 System resources CPU Memory I/O
  4. 4. 5 Linux performance metrics ● Processor metrics ● Memory metrics ● Block device metrics ● Network interface metrics
  5. 5. 6 Monitoring tools ●‫نیاز‬ ‫مورد‬ ‫های‬ ‫بسته‬ ● sysstat ● iostat, sar, mpstat, ... ● procps ● free, uptime, vmstat, w, ...
  6. 6. 10 Process management‫ادامه‬ ● Context switching ● Interrupt handling ● Process state (runing,rtopped,intruptible,unintruptible,zombie) fork() TASK_STOPPED TASK_UNINTERRUPTIBLE TASK_INTERRUPTIBLE TASK_RUNNING (ready) TASK_ZOMBIE exit() Processor TASK_RUNNING
  7. 7. 11 Processor metrics ● CPU utilization (Exceed 80%=bottleneck) ● User time, System time,Waiting, Idle time, Nice time ● Load avarage ● Runable processes (Excedd 10 time the amount of prcessors) ● Blocked (Uninterruptible: like process waiting for I/O) ● Context switch, Interrupts(software bottleneck)
  8. 8. 13 vmstat $ vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 262008 544552 21036 392400 1 4 173 142 136 332 15 5 77 2 $ vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 262008 544552 21036 392400 1 4 173 142 136 332 15 5 77 2 Process (procs) r: The number of processes waiting for runtime b: The number of processes in uninterruptable sleep Memory swpd: The amount of virtual memory used (KB) free: The amount of idle memory (KB) buff: The amount of memory used as buffers (KB) cache: The amount of memory used as cache (KB) Swap si: Amount of memory swapped from the disk (Kbps) so: Amount of memory swapped to the disk (Kbps) IO bi: Blocks sent to a block device (blocks/s) bo: Blocks received from a block device (blocks/s) System in: The number of interrupts per second, including the clock cs: The number of context switches per second Process (procs) r: The number of processes waiting for runtime b: The number of processes in uninterruptable sleep Memory swpd: The amount of virtual memory used (KB) free: The amount of idle memory (KB) buff: The amount of memory used as buffers (KB) cache: The amount of memory used as cache (KB) Swap si: Amount of memory swapped from the disk (Kbps) so: Amount of memory swapped to the disk (Kbps) IO bi: Blocks sent to a block device (blocks/s) bo: Blocks received from a block device (blocks/s) System in: The number of interrupts per second, including the clock cs: The number of context switches per second
  9. 9. 14 sar $ sar -q 02:55:02 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked 03:05:01 PM 0 566 1.83 2.01 2.16 0 $ sar -q 02:55:02 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked 03:05:01 PM 0 566 1.83 2.01 2.16 0 $ sar -P ALL Average: CPU %user %nice %system %iowait %steal %idle Average: all 15.21 0.03 5.20 1.74 0.00 77.81 Average: 0 16.22 0.03 5.25 1.95 0.00 76.54 Average: 1 16.14 0.03 5.38 1.97 0.00 76.48 Average: 2 14.24 0.04 5.02 1.52 0.00 79.18 Average: 3 14.24 0.03 5.16 1.51 0.00 79.06 $ sar -P ALL Average: CPU %user %nice %system %iowait %steal %idle Average: all 15.21 0.03 5.20 1.74 0.00 77.81 Average: 0 16.22 0.03 5.25 1.95 0.00 76.54 Average: 1 16.14 0.03 5.38 1.97 0.00 76.48 Average: 2 14.24 0.04 5.02 1.52 0.00 79.18 Average: 3 14.24 0.03 5.16 1.51 0.00 79.06
  10. 10. 15 sar $ sar -w Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) 01:51:22 AM LINUX RESTART 01:55:01 AM proc/s cswch/s 02:05:01 AM 1.14 7087.09 02:15:01 AM 1.19 6955.23 02:25:01 AM 1.12 6855.21 02:35:01 AM 1.21 6311.18 02:45:01 AM 1.20 6006.53 $ sar -w Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) 01:51:22 AM LINUX RESTART 01:55:01 AM proc/s cswch/s 02:05:01 AM 1.14 7087.09 02:15:01 AM 1.19 6955.23 02:25:01 AM 1.12 6855.21 02:35:01 AM 1.21 6311.18 02:45:01 AM 1.20 6006.53
  11. 11. 16 Other command $ mpstat -P ALL Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) 04:50:28 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 04:50:28 PM all 14.94 0.03 4.62 1.80 0.01 0.47 0.00 0.00 78.13 $ mpstat -P ALL Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) 04:50:28 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 04:50:28 PM all 14.94 0.03 4.62 1.80 0.01 0.47 0.00 0.00 78.13 $ iostat -x Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 14.92 0.03 5.10 1.79 0.00 78.15 $ iostat -x Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 14.92 0.03 5.10 1.79 0.00 78.15
  12. 12. 17 Linux memory architecture ● Physical and virtual memory ● Virtual memory manager ● Page frame ● Page frame reclaiming (Swap ,...) ● Buddy system
  13. 13. 18 Memory metrics ● Free memory ● Swap usage ● Buffer and caches ● Slabs (kernel usage of memory) ● Active versus inactive memory ● Inactive memory is a likely candidate to be swapped out to disk
  14. 14. 19 free, top, vmstat $ top -b -n 1 | head -5 top - 16:54:00 up 5:39, 3 users, load average: 0.93, 0.65, 0.58 Tasks: 221 total, 1 running, 220 sleeping, 0 stopped, 0 zombie %Cpu(s): 14.9 us, 4.6 sy, 0.0 ni, 78.2 id, 1.8 wa, 0.0 hi, 0.5 si, 0.0 st KiB Mem: 2051244 total, 1600512 used, 450732 free, 29236 buffers KiB Swap: 4954776 total, 254804 used, 4699972 free, 445900 cached $ top -b -n 1 | head -5 top - 16:54:00 up 5:39, 3 users, load average: 0.93, 0.65, 0.58 Tasks: 221 total, 1 running, 220 sleeping, 0 stopped, 0 zombie %Cpu(s): 14.9 us, 4.6 sy, 0.0 ni, 78.2 id, 1.8 wa, 0.0 hi, 0.5 si, 0.0 st KiB Mem: 2051244 total, 1600512 used, 450732 free, 29236 buffers KiB Swap: 4954776 total, 254804 used, 4699972 free, 445900 cached $ vmstat -a procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free inact active si so bi bo in cs us sy id wa 0 1 254804 453648 613696 844384 1 3 155 129 309 480 15 5 78 2 $ vmstat -a procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free inact active si so bi bo in cs us sy id wa 0 1 254804 453648 613696 844384 1 3 155 129 309 480 15 5 78 2
  15. 15. 20 sar $ sar -r Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) 01:51:22 AM LINUX RESTART 01:55:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact 02:05:01 AM 289476 1761768 85.89 75248 547476 3017772 43.07 1141260 480772 $ sar -r Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) 01:51:22 AM LINUX RESTART 01:55:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact 02:05:01 AM 289476 1761768 85.89 75248 547476 3017772 43.07 1141260 480772
  16. 16. 25 Block device metrics ● Iowait ● Average queue length ● Average wait ● Transfers per second ● Blocks read/write per second ● Kilobytes per second read/write
  17. 17. 26 iostat $ iostat -p ALL 1 1$ iostat -p ALL 1 1 $ iostat -x 1 1 Linux 3.2.0-4-amd64 (laptop) 05/29/2014 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 8.62 0.03 2.61 2.15 0.00 86.60 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 1.44 5.36 3.35 4.37 196.43 59.67 66.38 0.25 32.64 12.89 47.76 6.00 4.63 $ iostat -x 1 1 Linux 3.2.0-4-amd64 (laptop) 05/29/2014 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 8.62 0.03 2.61 2.15 0.00 86.60 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 1.44 5.36 3.35 4.37 196.43 59.67 66.38 0.25 32.64 12.89 47.76 6.00 4.63
  18. 18. 27 sar $ sar -b Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU) 12:34:50 PM LINUX RESTART 12:35:04 PM tps rtps wtps bread/s bwrtn/s 12:45:01 PM 29.74 21.93 7.81 1389.04 292.90 12:55:01 PM 5.90 1.34 4.56 32.94 97.90 01:05:01 PM 20.23 13.89 6.35 760.67 698.30 01:15:01 PM 83.81 72.50 11.31 1943.13 1598.73 01:25:01 PM 5.55 1.47 4.08 39.11 100.80 $ sar -b Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU) 12:34:50 PM LINUX RESTART 12:35:04 PM tps rtps wtps bread/s bwrtn/s 12:45:01 PM 29.74 21.93 7.81 1389.04 292.90 12:55:01 PM 5.90 1.34 4.56 32.94 97.90 01:05:01 PM 20.23 13.89 6.35 760.67 698.30 01:15:01 PM 83.81 72.50 11.31 1943.13 1598.73 01:25:01 PM 5.55 1.47 4.08 39.11 100.80
  19. 19. 28 sar $ sar -d Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU) 12:34:50 PM LINUX RESTART 12:35:04 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 12:45:01 PM dev8-0 29.74 1389.04 292.90 56.55 0.96 33.07 5.69 16.93 12:55:01 PM dev8-0 5.90 32.94 97.90 22.17 0.07 11.91 5.40 3.19 01:05:01 PM dev8-0 20.23 760.67 698.30 72.10 0.43 21.23 3.50 7.09 $ sar -d Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU) 12:34:50 PM LINUX RESTART 12:35:04 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 12:45:01 PM dev8-0 29.74 1389.04 292.90 56.55 0.96 33.07 5.69 16.93 12:55:01 PM dev8-0 5.90 32.94 97.90 22.17 0.07 11.91 5.40 3.19 01:05:01 PM dev8-0 20.23 760.67 698.30 72.10 0.43 21.23 3.50 7.09 $ ls -l /dev/sda brw-rw---T 1 root disk 8, 0 Apr 23 12:34 /dev/sda $ ls -l /dev/sda brw-rw---T 1 root disk 8, 0 Apr 23 12:34 /dev/sda
  20. 20. 31 Network interface metrics ● Packets received and sent, Bytes received and sent ● Collisions per second ● Packets dropped ● Overruns (ran out of buffer space) ● Errors
  21. 21. 32 ifconfig $ ifconfig eth0 wlan0 Link encap:Ethernet HWaddr 00:08:ca:45:d0:76 inet addr:192.168.13.51 Bcast:192.168.13.255 Mask:255.255.255.0 inet6 addr: fe80::208:caff:fe45:d076/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:272597 errors:0 dropped:0 overruns:0 frame:0 TX packets:259457 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:350358017 (334.1 MiB) TX bytes:48277958 (46.0 MiB) $ ifconfig eth0 wlan0 Link encap:Ethernet HWaddr 00:08:ca:45:d0:76 inet addr:192.168.13.51 Bcast:192.168.13.255 Mask:255.255.255.0 inet6 addr: fe80::208:caff:fe45:d076/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:272597 errors:0 dropped:0 overruns:0 frame:0 TX packets:259457 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:350358017 (334.1 MiB) TX bytes:48277958 (46.0 MiB)
  22. 22. 33 sar $ sar -n DEV Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU) 12:34:50 PM LINUX RESTART 12:35:04 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 12:45:01 PM lo 0.07 0.07 0.01 0.01 0.00 0.00 0.00 12:45:01 PM wlan0 12.83 12.76 13.38 2.39 0.00 0.00 0.00 12:45:01 PM kvnet 7.97 8.28 8.10 1.10 0.00 0.00 0.00 12:45:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 $ sar -n DEV Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU) 12:34:50 PM LINUX RESTART 12:35:04 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 12:45:01 PM lo 0.07 0.07 0.01 0.01 0.00 0.00 0.00 12:45:01 PM wlan0 12.83 12.76 13.38 2.39 0.00 0.00 0.00 12:45:01 PM kvnet 7.97 8.28 8.10 1.10 0.00 0.00 0.00 12:45:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  23. 23. 34 netstat # netstat -antup Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 192.168.13.51:80 192.168.13.52:53930 ESTABLISHED 27398/apache2 tcp 0 0 192.168.13.51:80 192.168.13.52:53931 ESTABLISHED 27399/apache2 tcp 0 0 192.168.13.51:80 192.168.13.52:53930 TIME_WAIT - tcp 0 0 192.168.13.60:80 192.168.13.52:53931 SYN_RECV 27559/apache2 # netstat -antup Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 192.168.13.51:80 192.168.13.52:53930 ESTABLISHED 27398/apache2 tcp 0 0 192.168.13.51:80 192.168.13.52:53931 ESTABLISHED 27399/apache2 tcp 0 0 192.168.13.51:80 192.168.13.52:53930 TIME_WAIT - tcp 0 0 192.168.13.60:80 192.168.13.52:53931 SYN_RECV 27559/apache2 # netstat -s# netstat -s
  24. 24. Object 2: Predict Future Resource Needs
  25. 25. 36 Monitor resource usage Monitoring Solution Collectd Munin Nagios Cacti Mrtg
  26. 26. 37 Collectd ●‫نیاز‬ ‫مورد‬ ‫های‬ ‫بسته‬ ‫نصب‬ ●‫ها‬‫مسیر‬ # apt-get install collectd librrds-perl libconfig-general-perl libhtml-parser-perl libregexp-common-perl libjson-perl # apt-get install collectd librrds-perl libconfig-general-perl libhtml-parser-perl libregexp-common-perl libjson-perl Config file: /etc/collectd/{collectd.conf,collection.conf} Plugins : /usr/lib/collectd Data: /var/lib/collectd Config file: /etc/collectd/{collectd.conf,collection.conf} Plugins : /usr/lib/collectd Data: /var/lib/collectd
  27. 27. 38 Setting Options ● Global options (Interval, FQDNLookup, …) ● Plugin options (documented on man collectd.conf )“ ” BaseDir "/var/lib/collectd" PIDFile "/run/collectd.pid" Interval 10.0 (if changed delete all rrd files) LoadPlugin cpu LoadPlugin df <LoadPlugin df> Interval 3600 </LoadPlugin> BaseDir "/var/lib/collectd" PIDFile "/run/collectd.pid" Interval 10.0 (if changed delete all rrd files) LoadPlugin cpu LoadPlugin df <LoadPlugin df> Interval 3600 </LoadPlugin>
  28. 28. 39 Plugins ● Plugins ● Input (read) Plugins (CPU, Memory) ● Output (write) Plugins (RRDtool,CSV) ● Read and Write Plugins (Network) ● Log Plugins (LogFile,Syslog) # vi /etc/collectd/collectd.conf ... LoadPlugin apache LoadPlugin cpu LoadPlugin df LoadPlugin network LoadPlugin interface # vi /etc/collectd/collectd.conf ... LoadPlugin apache LoadPlugin cpu LoadPlugin df LoadPlugin network LoadPlugin interface
  29. 29. 41 Creating Graphs ● Debian ● Collection3 # apt-get install apache2-mpm-worker (Wheezy)# apt-get install apache2-mpm-worker (Wheezy) # cp -r /usr/share/doc/collectd/examples/collection3 /var/www/# cp -r /usr/share/doc/collectd/examples/collection3 /var/www/ # apt-get install apache2 (Jessie)# apt-get install apache2 (Jessie) # a2enmod cgi # vi /etc/apache2/sites-enabled/default <Directory /var/www/collection3> Options +ExecCgi AddHandler cgi-script .cgi .pl </Directory> # a2enmod cgi # vi /etc/apache2/sites-enabled/default <Directory /var/www/collection3> Options +ExecCgi AddHandler cgi-script .cgi .pl </Directory>

×