MySQL Monitoring 101
Upcoming SlideShare
Loading in...5
×
 

MySQL Monitoring 101

on

  • 23,279 views

In this presentation I’ll be discussing the following beginner points to understanding and creating monitoring. ...

In this presentation I’ll be discussing the following beginner points to understanding and creating monitoring.

* Why Monitor?
* What’s the minimum to Monitor?
* How to monitor?
* Monitoring Software Options.
* How to use the most basic of monitoring to help
* The basics of graphing results
* The rule of Everything
* The important on Application metrics and timings

For a very little investment in time, simple monitoring can be in place, and I can guarantee it will be of benefit to any system.

The basis of monitoring are metrics that combined with application measurements can provide trending insights, bottleneck understanding and provide valuable feedback about your growing site.

Statistics

Views

Total Views
23,279
Views on SlideShare
13,533
Embed Views
9,746

Actions

Likes
31
Downloads
719
Comments
0

17 Embeds 9,746

http://ronaldbradford.com 9352
http://learnmysql.blogspot.com 117
http://storify.com 95
http://www.nitinkatkam.com 62
http://www.dbthink.com 50
http://www.slideshare.net 24
http://translate.googleusercontent.com 21
http://feeds.feedburner.com 9
http://learnmysql.blogspot.in 6
http://learnmysql.blogspot.pt 2
http://raulbarraltamayo.wordpress.com 2
http://designcodemobilesecurity.blogspot.com 1
http://www.zhuaxia.com 1
http://74.125.93.132 1
http://learnvbdotnet.blogspot.de 1
https://www.linkedin.com 1
http://learnmysql.blogspot.de 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

MySQL Monitoring 101 MySQL Monitoring 101 Presentation Transcript

  • MySQL Monitoring 101 MySQL Monitoring 101 Simple stuff to save your bacon Ronald Bradford Principal - 42SQL MySQL Users Conference Santa Clara - April 2009 http://ronaldbradford.com Version 1.1 22.Apr.2009
  • MySQL Monitoring 101 What is Your Situation? How do you know your website is down? http://ronaldbradford.com
  • MySQL Monitoring 101 Is this you? How do you know your website is down? ❖ The users will let me know ❖ That's somebody else's problem ❖ Our site is never down ❖ Email/SMS/Pager http://ronaldbradford.com
  • MySQL Monitoring 101 Is this you? You have a performance problem now? ❖ How long has it been happening? ❖ Is it a new problem or a re-occurring problem? ❖ Has it gradually become worse over time? http://ronaldbradford.com
  • MySQL Monitoring 101 Is this you? When did you last recover from a backup? ❖ We need to think about doing backups ❖ We use MySQL replication ❖ That's somebody else's problem ❖ We verify our backups worked (not recovery) http://ronaldbradford.com
  • MySQL Monitoring 101 Goal http://ronaldbradford.com
  • MySQL Monitoring 101 Four Goals to take away Monitoring a MySQL Server(s) ❖ Know what to monitor ❖ Know how you can monitor ❖ Learn practices to diagnose problems ❖ Have a foundation of historical information http://ronaldbradford.com
  • MySQL Monitoring 101 Overview http://ronaldbradford.com
  • MySQL Monitoring 101 Monitoring Overview ❖ Hardware ❖ Software ❖ MySQL http://ronaldbradford.com
  • MySQL Monitoring 101 Hardware http://ronaldbradford.com
  • MySQL Monitoring 101 Know Your Physical Resources ❖ CPU ❖ Memory ❖ Disk ❖ Network http://ronaldbradford.com
  • MySQL Monitoring 101 Monitoring Hardware Resources Bare Essentials vmstat ❖ iostat ❖ ps ❖ netstat ❖ ifconfig ❖ top ❖ http://ronaldbradford.com
  • MySQL Monitoring 101 Monitoring Hardware Resources Very Valuable sar ❖ mpstat ❖ dstat ❖ ethtool ❖ http://ronaldbradford.com
  • MySQL Monitoring 101 Monitoring Hardware Resources Very Valuable ❖ /proc ❖ /cpuinfo ❖ /meminfo ❖ /loadavg http://ronaldbradford.com
  • MySQL Monitoring 101 Identifying Resource Bottlenecks ❖ Which physical resource is being stressed? ❖ Why? http://ronaldbradford.com
  • MySQL Monitoring 101 Identifying Resource Bottlenecks Can you spot Memory swapping, Disk I/O and CPU usage changes? $ vmstat 5 kthr memory page disk faults cpu rbw swap free re mf pi po fr de sr s0 s1 s2 s4 in sy cs us sy id 1 0 0 17114496 2962832 63 211 5180 71 71 0 0 0 0 71 0 4763 4625 3183 72 2 26 1 0 0 17115252 2963184 4 6 439 0 0 0 0 0 0 7 0 3251 4021 2795 79 2 20 1 0 0 17115252 2963144 2 1 0 0 0 0 0 0 0 16 0 3748 4427 3049 89 2 10 1 0 0 17115252 2962912 11 11 1360 0 0 0 0 0 0 47 0 4083 4210 2752 79 2 19 0 0 0 17115248 2962744 4 9 1428 0 0 0 0 0 0 10 0 1072 1015 754 17 1 82 1 0 0 17115248 2962664 4 0 0 0 0 0 0 0 0 42 0 3755 3818 2549 69 2 29 1 0 0 17115240 2962520 2 4 246 0 0 0 0 0 0 3 0 3231 3992 2833 79 1 19 1 0 0 17115228 2962400 3 8 1347 0 0 0 0 0 0 10 0 3706 4339 3063 88 2 10 1 0 0 17115220 2962256 16 11 853 0 0 0 0 0 0 52 0 4275 4201 2672 79 2 19 1 0 0 17115220 2962100 3 5 131 0 0 0 0 0 0 7 0 2742 3275 2385 62 2 36 0 0 0 17115220 2962064 3 0 0 0 0 0 0 0 0 29 0 1887 1247 917 19 1 81 1 0 0 17115216 2961664 7 13 1885 0 0 0 0 0 0 18 0 3597 4674 3270 82 2 17 2 0 0 17114344 2961428 75 281 1953 13 13 0 0 0 0 12 0 3697 4746 3203 87 2 11 1 0 0 17114332 2961396 19 14 211 6 6 0 0 0 0 53 0 4409 4476 2862 77 2 21 1 0 0 17115316 2961544 5 7 426 0 0 0 0 0 0 8 0 2658 3478 2371 64 1 34 0 0 0 17115316 2961500 5 0 0 0 0 0 0 0 0 39 0 2310 1802 1268 26 1 73 1 0 0 17115316 2961428 5 7 823 0 0 0 0 0 0 11 0 3454 4473 3096 82 2 16 1 0 0 17115308 2961224 7 11 920 0 0 0 0 0 0 13 0 3569 4377 3024 86 2 12 1 0 0 17115308 2961016 12 9 870 3 3 0 0 0 0 37 0 4687 5227 3250 82 2 17 http://ronaldbradford.com
  • MySQL Monitoring 101 Customer Examples ❖ Single CPU Kernel on 4 Processor Server $ uptime 10:32:22 up 196 days, 11:46, 2 users, load average: 3.86, 4.08, 4.26 $ uname -a Linux db1 2.4.20-6 #1 Thu Feb 27 10:06:59 EST 2003 i686 i686 i386 GNU/Linux [root@db1 mysql]# cat /proc/cpuinfo processor :0 vendor_id : GenuineIntel cpu family :6 model : 10 model name : Pentium III (Cascades) stepping :1 cpu MHz : 701.636 cache size : 2048 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level :2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse bogomips : 1399.19 http://ronaldbradford.com
  • MySQL Monitoring 101 Customer Examples ❖ Single CPU Kernel - Disabled from boot loader $ more /boot/grub/grub.conf # grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro root=/dev/sda3 # initrd /initrd-version.img #boot=/dev/sda default=0 timeout=10 splashimage=(hd0,0)/grub/splash.xpm.gz root (hd0,0) kernel /vmlinuz-2.4.20-6 ro root=LABEL=/ initrd /initrd-2.4.20-6.img [root@db1 mysql]# ls /boot/vmlinu vmlinux-2.4.20-6 vmlinux-2.4.20-6smp vmlinuz vmlinuz-2.4.20-6 vmlinuz-2.4.20-6smp http://ronaldbradford.com
  • MySQL Monitoring 101 Customer Examples ❖ Excessive Network Load (20GB uncompressed backup) $ ethtool eth0 Settings for eth0: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on Supports Wake-on: d Wake-on: d Link detected: yes http://ronaldbradford.com
  • MySQL Monitoring 101 Customer Examples ❖ Degraded RAID ❖ Who checks RAID status? ❖ How is it checked? http://ronaldbradford.com
  • MySQL Monitoring 101 Additional Resources • Identify Bottlenecks • CPU • Memory http://ronaldbradford.com/blog/identifying-resource-bottlenecks-cpu-2009-03-31/ http://ronaldbradford.com/blog/identifying-resource-bottlenecks-memory-2009-04-02/ • Check back for additional articles http://ronaldbradford.com
  • MySQL Monitoring 101 Software http://ronaldbradford.com
  • MySQL Monitoring 101 Know Your Running Software ❖ Operating System ❖ Database ❖ Other http://ronaldbradford.com
  • MySQL Monitoring 101 Operating System Specifics ❖ Kernel Version ❖ 32/64 bit Kernel ❖ 32bit software running on 64bit kernel ❖ Review all running software ❖ Should it be running? http://ronaldbradford.com
  • MySQL Monitoring 101 MySQL http://ronaldbradford.com
  • MySQL Monitoring 101 MySQL Log Files 1 ❖ Error Log ❖ Could be OS Log ❖ Slow Query Log http://ronaldbradford.com
  • MySQL Monitoring 101 MySQL Process ❖ Is mysqld running? ❖ What is the Memory Usage? http://ronaldbradford.com
  • MySQL Monitoring 101 MySQL Memory Usage 2 ❖ RSS and VSZ $ ps -eopid,fname,rss,vsz,user,command | grep -e quot;RSSquot; -e quot;mysqlquot; PID COMMAND RSS VSZ USER COMMAND 5463 grep 764 5204 ronald grep -e RSS -e mysql 13894 mysqld_s 596 3936 root /bin/sh /usr/bin/mysqld_safe 13933 mysqld 4787812 5127208 mysql /usr/sbin/mysqld --basedir=/usr -- datadir=/vol/mysql/mysqldata --user=mysql --pid-file=/var/run/mysqld/ mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/ mysqld.sock 13934 logger 608 3840 root logger -p daemon.err -t mysqld_safe -i -t mysqld $ ps -eopid,fname,rss,vsz,user,command | grep quot; mysqld quot; | grep -v grep | awk '{print $3,$4}' 4787820 5127208 http://ronaldbradford.com/blog/are-you-monitoring-rss-vsz-2009-03-08/ http://ronaldbradford.com
  • MySQL Monitoring 101 MySQL System Variables 3 ❖ Based on my.cnf configuration ❖ Customizable in running server ❖ SHOW GLOBAL VARIABLES http://dev.mysql.com/doc/refman/5.1/en/show-variables.html http://ronaldbradford.com
  • MySQL Monitoring 101 MySQL Status Variables 4 ❖ Provide valuable clues to the state of your servers ❖ GLOBAL and SESSION scope ❖ SHOW [GLOBAL|SESSION] STATUS http://dev.mysql.com/doc/refman/5.1/en/show-status.html http://ronaldbradford.com
  • MySQL Monitoring 101 MySQL Connections 5 ❖ SHOW FULL PROCESSLIST http://dev.mysql.com/doc/refman/5.1/en/show-status.html http://ronaldbradford.com
  • MySQL Monitoring 101 MySQL Storage Engines 6 ❖ SHOW ENGINE INNODB STATUS ❖ Well incorporated in 5.1 Status Variables http://dev.mysql.com/doc/refman/5.1/en/show-status.html http://ronaldbradford.com
  • MySQL Monitoring 101 Your MySQL Data 7 ❖ Know your data ❖ Disk footprint ❖ Growth http://ronaldbradford.com
  • MySQL Monitoring 101 Instance Summary select table_schema, sum(data_length+index_length)/1024/1024 as total_mb, sum(data_length)/1024/1024 as data_mb, sum(index_length)/1024/1024 as index_mb, count(*) as tables from information_schema.tables group by table_schema order by 2 desc; http://ronaldbradford.com
  • MySQL Monitoring 101 Instance Summary +----------------------------+----------------+----------------+---------------+--------+ | table_schema | total_mb | data_mb | index_mb | tables | +----------------------------+----------------+----------------+---------------+--------+ | xxxxxxxxxxxxxxxx_531 | 25301.01871777 | 19678.43473339 | 5622.58398438 | 189 | | xxxxxxxxxxxxxxxx_528 | 18983.51311207 | 14739.53361988 | 4243.97949219 | 96 | | stats | 14803.18925285 | 5469.20292473 | 9333.98632813 | 1082 | | xxxxxxxxxxxxxxxx_4 | 9631.98090553 | 7600.90180397 | 2031.07910156 | 1119 | | xxxxxxxxxxxxxxxx_501 | 9141.53529739 | 7263.91127396 | 1877.62402344 | 72 | | system_logs | 7973.60180283 | 3237.59008408 | 4736.01171875 | 1553 | | stats_cpu | 4544.43988991 | 3317.35981178 | 1227.08007813 | 1243 | | xxxxxxxxxxxxxxxx_555 | 4024.49787521 | 3436.00080490 | 588.49707031 | 293 | | stats_xxxxxxxxx | 3712.71884155 | 2159.92196655 | 1552.79687500 | 1536 | ... http://ronaldbradford.com
  • MySQL Monitoring 101 Schema Summary select table_name,engine,row_format, table_rows, avg_row_length, (data_length+index_length)/1024/1024 as total_mb, (data_length)/1024/1024 as data_mb, (index_length)/1024/1024 as index_mb from information_schema.tables where table_schema= DATABASE() order by 6 desc; http://ronaldbradford.com
  • MySQL Monitoring 101 Schema Attributes ❖ Collations ❖ Text/Blob ❖ Storage Engines http://ronaldbradford.com
  • MySQL Monitoring 101 Schema Attributes select table_schema,engine,table_collation, count(*) as tables from information_schema.tables where table_schema=DATABASE() group by table_schema,engine,table_collation; select table_schema,table_name,column_name,data_type from information_schema.columns where table_schema= DATABASE() and ( data_type LIKE '%TEXT' OR data_type like '%BLOB'); http://ronaldbradford.com
  • MySQL Monitoring 101 Monitoring http://ronaldbradford.com
  • MySQL Monitoring 101 System Monitoring ❖ dstat $ cat dbc_os.sh #!/bin/sh DATETIME=`date +%y%m%d.%H%M` LOG_DIR=quot;/opt/monitor/log/quot; INTERVAL=5 COUNT=`expr 3600 / $INTERVAL` LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.txt CSV_LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.csv MYSQL_LOG_FILE=${LOG_DIR}os.${DATETIME}.${INTERVAL}.mysql.csv /usr/bin/dstat --time --cpu --mem --disk --net --proc --page --swap --load --nocolor --noheaders --output $ {CSV_LOG_FILE} $INTERVAL $COUNT > $LOG_FILE 2>&1 sed -e quot;1,7dquot; -e quot;s/-/,/quot; -e quot;s/ /,/quot; ${CSV_LOG_FILE} > ${MYSQL_LOG_FILE} exit 0 http://ronaldbradford.com
  • MySQL Monitoring 101 System Monitoring ❖ Example dstat output -----time----- ----total-cpu-usage---- ------memory-usage----- -dsk/total-> date/time |usr sys idl wai hiq siq| used buff cach free| read writ> 19-04 17:00:01| 7 2 91 1 0 0|1151M 475M 1850M 480M| 51k 859k> 19-04 17:00:06| 5 0 95 0 0 0|1168M 475M 1850M 462M|5734B 134k> 19-04 17:00:11| 3 0 96 0 0 0|1175M 475M 1850M 456M| 0 168k> 19-04 17:00:16| 4 0 92 3 0 0|1184M 475M 1850M 447M| 20k 625k> 19-04 17:00:21| 3 0 97 0 0 0|1142M 475M 1850M 488M| 0 307k> 19-04 17:00:26| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 9830B> 19-04 17:00:31| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 2458B> 19-04 17:00:36| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 34k> 19-04 17:00:41| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 4915B> 19-04 17:00:46| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 31k> 19-04 17:00:51| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 42k> 19-04 17:00:56| 0 0 100 0 0 0|1142M 475M 1850M 488M| 0 30k> 19-04 17:01:01| 1 0 99 0 0 0|1151M 475M 1850M 480M| 0 58k> 19-04 17:01:06| 0 0 100 0 0 0|1151M 475M 1850M 480M| 0 9011B> 19-04 17:01:11| 8 0 91 0 0 0|1153M 475M 1850M 478M| 0 3277B> 19-04 17:01:16| 2 0 99 0 0 0|1152M 475M 1850M 479M| 0 37k> 19-04 17:01:21| 0 0 100 0 0 0|1152M 475M 1850M 479M| 0 6554B> http://ronaldbradford.com
  • MySQL Monitoring 101 MySQL Monitoring ❖ hourly.sh ❖ OS ❖ vmstat ❖ ps ❖ uptime ❖ MySQL ❖ Variables ❖ Status ❖ Process List ❖ Innodb Engine Status ❖ Table Status http://ronaldbradford.com
  • MySQL Monitoring 101 MySQL Analysis ❖ statpack - Human Readable STATUS ❖ Database Activity ❖ Statement Activity ❖ Prepared Statements ❖ Admin Commands ❖ Thread Cache ❖ Table Cache ❖ MyISAM Key Cache ❖ .... http://www.markleith.co.uk/?p=21 http://ronaldbradford.com
  • MySQL Monitoring 101 statpack - Read/Write Ratio =================================================================================================== Uptime: 17 days 17 hours 22 mins Snapshot Period 1: 59 minute interval =================================================================================================== Variable Delta/Percentage Per Second Total =================================================================================================== Statement Activity =================================================================================================== SELECT: 13,503,876 3,798.56 4,298,170,239 (94.25%) INSERT: 91,101 25.63 25,327,062 (0.56%) UPDATE: 782,004 219.97 220,640,296 (4.84%) DELETE: 9,674 2.72 2,485,643 (0.05%) REPLACE: 0 0.00 4,980 (0.00%) INSERT ... SELECT: 0 0.00 0 (0.00%) REPLACE ... SELECT: 0 0.00 0 (0.00%) Multi UPDATE: 0 0.00 0 (0.00%) Multi DELETE: 0 0.00 0 (0.00%) COMMIT: 46,422 13.06 13,700,478 (0.30%) ROLLBACK: 0 0.00 13 (0.00%) http://ronaldbradford.com
  • MySQL Monitoring 101 statpack - Unnecessary SQL ================================================================================================= Uptime: 12 hours 17 mins Snapshot Period 1: 1 minute interval ================================================================================================= Variable Delta/Percentage Per Second Total ================================================================================================= Statement Activity ================================================================================================= SELECT: 16,042 267.37 8,177,050 (46.03%) INSERT: 5,838 97.30 1,826,616 (10.28%) UPDATE: 1,109 18.48 738,546 (4.16%) DELETE: 2,018 33.63 1,374,983 (7.74%) REPLACE: 0 0.00 0 (0.00%) INSERT ... SELECT: 0 0.00 27 (0.00%) REPLACE ... SELECT: 0 0.00 0 (0.00%) Multi UPDATE: 0 0.00 0 (0.00%) Multi DELETE: 0 0.00 0 (0.00%) COMMIT: 5,708 95.13 2,161,232 (12.17%) ROLLBACK: 5,746 95.77 3,485,828 (19.62%) http://ronaldbradford.com
  • MySQL Monitoring 101 statpack - Ineffective Query Cache =================================================================================================== Uptime: 12 hours 17 mins Snapshot Period 1: 1 minute interval =================================================================================================== Variable Delta/Percentage Per Second Total =================================================================================================== Query Cache =================================================================================================== QCache Hits / SELECT: 11.11% QCache Hit/Qcache Insert: 31.24% Qcache Hits/Invalidations: 0.00% SELECTs: 16,042 267.37 8,177,050 Query Cache Hits: 1,257 20.95 1,022,301 Query Cache Inserts: 4,181 69.68 2,250,062 Queries Not Cached: 11,864 197.73 5,932,162 Cache Low Memory Prunes: 0 0.00 0 Total Cache Blocks: -421 -7.02 12,051 Queries In Cache: -595 -9.92 4,194 Cache Free Blocks: 770 12.83 3,628 http://ronaldbradford.com
  • MySQL Monitoring 101 Graphing Options ❖ RRDtool ❖ gnuplot ❖ Google Charts API ❖ vmplot.sh ❖ http://www.bigdbahead.com/?p=302 ❖ http://ronaldbradford.com/blog/extending- vmplot-2009-03-31/ http://ronaldbradford.com
  • MySQL Monitoring 101 vmplot.sh output http://ronaldbradford.com
  • MySQL Monitoring 101 Always graph your results 70% 60% 50% 43% 40% 30% http://ronaldbradford.com
  • MySQL Monitoring 101 Products http://ronaldbradford.com
  • MySQL Monitoring 101 MySQL Enterprise Monitor ❖ Commercial Product - $$$ ❖ Developed by Sun/MySQL http://www.mysql.com/products/enterprise/monitor.html http://ronaldbradford.com
  • MySQL Monitoring 101 Nagios ❖ Network and Application Monitoring ❖ Open Source http://www.nagios.org/ http://ronaldbradford.com
  • MySQL Monitoring 101 Cacti ❖ Generic Network Graphing ❖ Uses RRDtool ❖ MySQL specific templates http://www.cacti.net/ http://code.google.com/p/mysql-cacti-templates/ http://ronaldbradford.com
  • MySQL Monitoring 101 Additional ❖ Hyperic ❖ Ganglia ❖ Munin ❖ Big Brother ❖ MONyog ❖ EM7 ❖ Zenoss http://ronaldbradford.com
  • MySQL Monitoring 101 External Services ❖ Pingdom ❖ WebPerform http://ronaldbradford.com
  • MySQL Monitoring 101 What's missing ❖ Some Standards ❖ Active Community ❖ Integration for monitoring all stack products ❖ Application specific metrics ❖ e.g. Number of new customers per hour ❖ e.g. Total Sales for day http://ronaldbradford.com
  • MySQL Monitoring 101 Professional Help is Available ❖ Two decades IT expertise ❖ 10 years in MySQL ❖ System/Data Architecture ❖ Database Performance and Tuning ❖ High Availability and Scalability ❖ Education and Training http://ronaldbradford.com http://ronaldbradford.com