0
Best Practices For Deploying MySQL on Solaris  <ul><li>Ritu Kamboj </li></ul><ul><li>Jenny Chen </li></ul>
Agenda <ul><li>MySQL -Solaris Integration  </li></ul><ul><li>MySQL High Availability Data Service  </li></ul><ul><li>Conso...
<ul><li>MySQL-Solaris Integration </li></ul>
Optimized MySQL on Open Solaris <ul><li>MySQL 5.0.45 (32bit) integrated with Open Solaris build 76 </li></ul><ul><ul><li>S...
Layout of MySQL on Open Solaris <ul><li>MySQL 5.0.45 packages </li></ul><ul><ul><li>Default data directory  </li></ul></ul...
Optimization Of MySQL On Open Solaris <ul><li>Compiled with optimal compiler options using Sun Studio compilers </li></ul>...
Compiler Optimization <ul><li>Enable in-lining </li></ul><ul><ul><li>Change header file univ.i to add Sun Studio </li></ul...
Compiler Optimization (Contd)‏ <ul><li>Feedback optimization (not yet implemented)‏ </li></ul><ul><ul><li>Workload is key ...
Service Management Facility (SMF)‏ <ul><li>Makes Solaris Services Self-Healing </li></ul><ul><li>Services automatically re...
MySQL-Service Management Facility <ul><li>Dynamically configured properties </li></ul><ul><ul><li>mysql/bin </li></ul></ul...
Starting MySQL on Open Solaris <ul><li>Older Open Solaris Build  (Build 76- Build-87)‏ </li></ul><ul><ul><li>32Bit MySQL i...
Optimized MySQL on Solaris 10 <ul><li>CoolStack  </li></ul><ul><ul><li>SAMP stack + more   </li></ul></ul><ul><li>Optimize...
<ul><li>MySQL-High Availability Data Service </li></ul>
Solaris Cluster Overview <ul><li>Provides general purpose HA platform </li></ul>Fifty percent of enterprises that lack a r...
Solaris Cluster : Hardware Components <ul><li>Servers with local storage </li></ul><ul><ul><li>Can have up to 16 nodes  </...
Solaris Cluster Algorithms <ul><li>Cluster membership monitor </li></ul><ul><ul><li>Ensures data integrity </li></ul></ul>...
MySQL High Availability Data Service <ul><li>HA-MySQL is a failover data service </li></ul>Node 1 Node 2 Node 3 Node 4 DB
MySQL High Availability Data Service <ul><li>Supported configurations </li></ul><ul><ul><li>Standalone MySQL server </li><...
HA-MySQL  Service Deployment Node1 Node2:Zone2 Node1:Zone1 Node2 MySQL Stor Host MySQL Stor Host
MySQL-Solaris Cluster Benefits <ul><li>Enhanced end-to-end infrastructure availability </li></ul><ul><li>Continuous MySQL ...
Additional information <ul><li>Step by step deployment guide </li></ul><ul><ul><li>http://docs.sun.com/app/docs/doc/819-30...
<ul><ul><li>Consolidate MySQL installations using Solaris Containers </li></ul></ul>
Solaris Containers <ul><li>Containers : Zones + Resource Management </li></ul><ul><ul><li>Zones: isolated virtual applicat...
MySQL Consolidation Study <ul><li>System Configuration </li></ul><ul><ul><li>Sun Fire X4100 (4 CPU, 8 GB Memory)‏ </li></u...
<ul><li>Dtrace (Jenny Chen)‏ </li></ul>
DTrace <ul><li>Use DTrace with MySQL to drill down Monitoring MySQL Performance  </li></ul><ul><li>Examples & Solutions </...
Why DTrace <ul><li>Solaris 10 Dynamic Tracing Facility to provide comprehensive view of operating system and application b...
DTrace: Monitor I/Os <ul><li>Exam I/O wait time by filename and mysqld(Available at DTraceToolkit)‏ </li></ul><ul><ul><li>...
DTrace: Monitor CPU <ul><li>classic performance problem </li></ul><ul><ul><li>#!/usr/sbin/dtrace -s </li></ul></ul><ul><ul...
<ul><ul><li># ./syscall.d </li></ul></ul><ul><ul><li>EXEC  SYSCALL  COUNT  </li></ul></ul><ul><ul><li>devfsadm  lwp_park  ...
<ul><li>Agrregated user stack backtrace to understand of the nature of pread() in MySQL source code – useful for mysql dev...
Dtrace: Memory Analysis <ul><li>Check which process causes anonymous page in </li></ul><ul><ul><li># dtrace -n anonpgin '{...
Dtrace Probes In MySQL <ul><li>Provide deep view of internal MySQL core server and storage engines' operation & behaviour ...
Query execution Time <ul><li>Enable Slow query log with “–log-slow-queries” requires re-start MySQL server </li></ul><ul><...
Insert DTrace Probes Into MySQL <ul><li>Step 1: Figure out what probes are needed to insert into the source code </li></ul...
<ul><li>Step 3: Define a header file “mysql_dtrace.h” with definitions for probes </li></ul><ul><ul><li>dtrace -h -s mysql...
<ul><li>Step 4: Insert the probes into source code </li></ul><ul><ul><li>#include <mysql_dtrace.h> </li></ul></ul><ul><ul>...
<ul><li>Step 5: Build MySQL with DTrace  </li></ul><ul><li>In the Makefile.in, compile 64-bit MySQL with Dtrace </li></ul>...
<ul><li>Step 6: Use inserted DTrace probes to measure query execution time with other database information </li></ul><ul><...
<ul><li># ./mysqld_qestat.d  </li></ul><ul><ul><li>DATABASE  USER@HOST  ms  RET  QUERY </li></ul></ul><ul><ul><li>sbtest  ...
Index & Table scan cost <ul><li>Identify the places in MySQL source to handle scanning index, and table </li></ul><ul><ul>...
Buffer wait in Innodb <ul><ul><li>Innodb buffer wait is common in I/O-bound MySQL system while reading </li></ul></ul><ul>...
Query cache probes <ul><li>Use DTrace to measure query cache hit and query cache miss to determine how well the query cach...
<ul><ul><li># ./mysqld_qchit.d </li></ul></ul><ul><ul><li>Tracing... Hit Ctrl-C to end. </li></ul></ul><ul><ul><li>^C </li...
DTrace Performance Impact <ul><li>Inserting DTrace Probes into MySQL source code are useful for MySQL DBA,  MySQL & applic...
DTrace Probes In MySQL 6.0 <ul><ul><ul><li>probe insert_row_start(); probe insert_row_end(); probe filesort_start(); probe...
Integrate with Chime Tool <ul><ul><li>Chime is graphical tool for visualizing DTrace aggregations. It </li></ul></ul><ul><...
 
<ul><li>Resources </li></ul><ul><ul><li>>  http:// www.opensolaris.org /os/community/dtrace </li></ul></ul><ul><ul><li>- O...
Upcoming SlideShare
Loading in...5
×

Best Practices For Deploying MySQL on Solaris

3,933

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,933
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
58
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Best Practices For Deploying MySQL on Solaris"

  1. 1. Best Practices For Deploying MySQL on Solaris <ul><li>Ritu Kamboj </li></ul><ul><li>Jenny Chen </li></ul>
  2. 2. Agenda <ul><li>MySQL -Solaris Integration </li></ul><ul><li>MySQL High Availability Data Service </li></ul><ul><li>Consolidate MySQL Deployment Using Solaris Containers </li></ul><ul><li>DTrace (Jenny Chen)‏ </li></ul>
  3. 3. <ul><li>MySQL-Solaris Integration </li></ul>
  4. 4. Optimized MySQL on Open Solaris <ul><li>MySQL 5.0.45 (32bit) integrated with Open Solaris build 76 </li></ul><ul><ul><li>SXDE 01/08 </li></ul></ul><ul><li>MySQL 5.0.45 (64bit) integrated with Open Solaris build 87 </li></ul><ul><ul><li>SXCE </li></ul></ul>
  5. 5. Layout of MySQL on Open Solaris <ul><li>MySQL 5.0.45 packages </li></ul><ul><ul><li>Default data directory </li></ul></ul><ul><ul><ul><li>/var/mysql/5.0/data </li></ul></ul></ul><ul><ul><li>Default configuration directory </li></ul></ul><ul><ul><ul><li>/etc/mysql/5.0 </li></ul></ul></ul><ul><ul><li>Installation directory </li></ul></ul><ul><ul><ul><li>/ usr/mysql/5.0/ </li></ul></ul></ul><ul><li>Latest version accessible from /usr/mysql/bin </li></ul><ul><ul><li>Symbolic link from all directories involved in 5.0.45 release to corresponding /usr/mysql </li></ul></ul>
  6. 6. Optimization Of MySQL On Open Solaris <ul><li>Compiled with optimal compiler options using Sun Studio compilers </li></ul><ul><li>Added SMF support for MySQL </li></ul><ul><ul><li>Easily accommodates varying configurations </li></ul></ul><ul><ul><li>Initializes MySQL database thereby enhancing user experience </li></ul></ul><ul><ul><li>Mysql user can manage MySQL database via SMF </li></ul></ul><ul><li>Support for MySQL cluster engine (ndb)‏ </li></ul>
  7. 7. Compiler Optimization <ul><li>Enable in-lining </li></ul><ul><ul><li>Change header file univ.i to add Sun Studio </li></ul></ul><ul><ul><ul><li>If (!defined(SUNPRO_C)‏ </li></ul></ul></ul><ul><ul><ul><li>#define UNIV_MUST_NOT_INLINE </li></ul></ul></ul><ul><li>Enable pre-fetching </li></ul><ul><ul><li>-xprefetch=auto and -xprefetch_level=3 </li></ul></ul><ul><li>Set optimization level </li></ul><ul><ul><li>- xO4 </li></ul></ul>
  8. 8. Compiler Optimization (Contd)‏ <ul><li>Feedback optimization (not yet implemented)‏ </li></ul><ul><ul><li>Workload is key </li></ul></ul><ul><ul><li>-xprofile -xipo </li></ul></ul><ul><ul><li>Interprocess optimization </li></ul></ul><ul><ul><li>About 10% improvement </li></ul></ul><ul><li>Link with libmtmalloc </li></ul><ul><ul><ul><li>Library for threaded application </li></ul></ul></ul><ul><ul><ul><li>About 8% improvement </li></ul></ul></ul>
  9. 9. Service Management Facility (SMF)‏ <ul><li>Makes Solaris Services Self-Healing </li></ul><ul><li>Services automatically restart in dependency order </li></ul><ul><li>Misbehaving and mis-configured services are easier to debug </li></ul><ul><ul><li>Log files for each service </li></ul></ul><ul><li>Administrators can securely delegate tasks to non-root users </li></ul>
  10. 10. MySQL-Service Management Facility <ul><li>Dynamically configured properties </li></ul><ul><ul><li>mysql/bin </li></ul></ul><ul><ul><ul><li>Installation , default : /usr/mysql/5.0/bin </li></ul></ul></ul><ul><ul><li>mysql/data </li></ul></ul><ul><ul><ul><li>Data Directory , Default: /var/mysql/5.0/data </li></ul></ul></ul><ul><ul><li>mysql/enable_64bit </li></ul></ul><ul><ul><ul><li>Flag to select 32bit or 64bit , Default : false </li></ul></ul></ul><ul><li>First time installations </li></ul><ul><ul><li>Creates system tables (mysql_install_db.sh )‏ </li></ul></ul>
  11. 11. Starting MySQL on Open Solaris <ul><li>Older Open Solaris Build (Build 76- Build-87)‏ </li></ul><ul><ul><li>32Bit MySQL integrated </li></ul></ul><ul><ul><li>Create mysql user </li></ul></ul><ul><ul><li>Enable MySQL SMF service </li></ul></ul><ul><li>Latest Open Solaris Build (Build 87 onwards) ‏ </li></ul><ul><ul><ul><li>32Bit and 64bit MySQL integrated </li></ul></ul></ul><ul><ul><ul><li>Default mode : 32bit service </li></ul></ul></ul><ul><ul><ul><li>For starting 64bit </li></ul></ul></ul><ul><ul><ul><ul><li>Set enable_64bit == true </li></ul></ul></ul></ul>
  12. 12. Optimized MySQL on Solaris 10 <ul><li>CoolStack </li></ul><ul><ul><li>SAMP stack + more </li></ul></ul><ul><li>Optimized MySQL download </li></ul><ul><ul><li>Standalone package </li></ul></ul><ul><ul><li>SAMP stack component </li></ul></ul><ul><li>Available at Sun Download center: </li></ul><ul><ul><li>http://cooltools.sunsource.net/coolstack </li></ul></ul><ul><li>Version MySQL 5.0.45 </li></ul><ul><li>Similar optimizations as in Open Solaris </li></ul>
  13. 13. <ul><li>MySQL-High Availability Data Service </li></ul>
  14. 14. Solaris Cluster Overview <ul><li>Provides general purpose HA platform </li></ul>Fifty percent of enterprises that lack a recovery plan go out of business within one year of a significant disaster Gartner Group Availability is our customers' most critical requirement Sun Cluster VOC Survey
  15. 15. Solaris Cluster : Hardware Components <ul><li>Servers with local storage </li></ul><ul><ul><li>Can have up to 16 nodes </li></ul></ul><ul><li>Shared storage </li></ul><ul><ul><li>Tolerates single-node failures </li></ul></ul><ul><ul><li>Centralizes configuration files </li></ul></ul><ul><li>Cluster interconnect </li></ul><ul><ul><li>At least two redundant networks </li></ul></ul><ul><li>Public network interfaces </li></ul><ul><ul><li>Spreads outbound packets </li></ul></ul>
  16. 16. Solaris Cluster Algorithms <ul><li>Cluster membership monitor </li></ul><ul><ul><li>Ensures data integrity </li></ul></ul><ul><ul><li>Determines cluster membership </li></ul></ul><ul><li>Cluster configuration repository </li></ul><ul><ul><li>Global repository </li></ul></ul><ul><ul><li>Ensures consistent view </li></ul></ul><ul><li>Disk Fencing </li></ul><ul><ul><li>Fences off non-cluster nodes </li></ul></ul><ul><ul><li>Prevents Partition </li></ul></ul><ul><li>Quorum </li></ul><ul><ul><li>Uses a majority voting schema </li></ul></ul>
  17. 17. MySQL High Availability Data Service <ul><li>HA-MySQL is a failover data service </li></ul>Node 1 Node 2 Node 3 Node 4 DB
  18. 18. MySQL High Availability Data Service <ul><li>Supported configurations </li></ul><ul><ul><li>Standalone MySQL server </li></ul></ul><ul><ul><li>MySQL replication server </li></ul></ul><ul><ul><ul><li>Single/Multiple MySQL instances in master configuration </li></ul></ul></ul><ul><ul><ul><li>Single/Multiple MySQL instances in slave configuration </li></ul></ul></ul><ul><li>Solaris containers support </li></ul><ul><ul><li>Global zone </li></ul></ul><ul><ul><li>Non-global failover zone </li></ul></ul><ul><ul><li>Non-global zone </li></ul></ul>
  19. 19. HA-MySQL Service Deployment Node1 Node2:Zone2 Node1:Zone1 Node2 MySQL Stor Host MySQL Stor Host
  20. 20. MySQL-Solaris Cluster Benefits <ul><li>Enhanced end-to-end infrastructure availability </li></ul><ul><li>Continuous MySQL Availability </li></ul><ul><ul><li>Automatic failover if master node fails </li></ul></ul><ul><li>Low cost solution </li></ul><ul><ul><li>Software is free and open sourced </li></ul></ul><ul><li>Efficient Resource Utilization </li></ul><ul><ul><li>Multiple applications can be consolidated </li></ul></ul><ul><li>Ease of operations </li></ul><ul><ul><li>SC enables clustered systems to be managed as if they were on a single system </li></ul></ul>
  21. 21. Additional information <ul><li>Step by step deployment guide </li></ul><ul><ul><li>http://docs.sun.com/app/docs/doc/819-3059 </li></ul></ul><ul><li>Failover study of HA-MySQL </li></ul><ul><ul><li>http://blogs.sun.com/krishs/date/200804 </li></ul></ul><ul><li>Solaris Cluster </li></ul><ul><ul><li>http://www.sun.com/software/solaris/cluster/index.xml </li></ul></ul><ul><ul><li>http://opensolaris.org/os/community/ha-clusters/ </li></ul></ul>
  22. 22. <ul><ul><li>Consolidate MySQL installations using Solaris Containers </li></ul></ul>
  23. 23. Solaris Containers <ul><li>Containers : Zones + Resource Management </li></ul><ul><ul><li>Zones: isolated virtual application environments </li></ul></ul><ul><ul><li>Resource management – resource control (CPU, Memory)‏ </li></ul></ul><ul><li>Achieving Consolidation Goals </li></ul><ul><ul><li>Reduce Hardware </li></ul></ul><ul><ul><ul><li>Combine low utilization systems </li></ul></ul></ul><ul><ul><li>Isolate applications from faults </li></ul></ul><ul><ul><li>Maintain Service Levels </li></ul></ul><ul><ul><ul><li>Fine tune response times </li></ul></ul></ul>
  24. 24. MySQL Consolidation Study <ul><li>System Configuration </li></ul><ul><ul><li>Sun Fire X4100 (4 CPU, 8 GB Memory)‏ </li></ul></ul><ul><li>Local container configuration ( 1 CPU , 1 GB Memory </li></ul><ul><li>Sysbench Read-only Sysbench read-write </li></ul>
  25. 25. <ul><li>Dtrace (Jenny Chen)‏ </li></ul>
  26. 26. DTrace <ul><li>Use DTrace with MySQL to drill down Monitoring MySQL Performance </li></ul><ul><li>Examples & Solutions </li></ul><ul><li>Easy Steps To add DTrace probes into MySQL core server and storage Engines </li></ul><ul><li>Easy Steps to display MySQL DTrace probes into Chime visualization Tool for DTrace </li></ul>
  27. 27. Why DTrace <ul><li>Solaris 10 Dynamic Tracing Facility to provide comprehensive view of operating system and application behaviour </li></ul><ul><li>DTrace to examine particular system areas: disk I/O, CPU, Memory </li></ul><ul><li>Process Tracing and Debugging </li></ul><ul><ul><li>USDT (User-level statically defined tracing) place custom probes in application code </li></ul></ul><ul><ul><li>Add USDT into MySQL source to monitor MySQL and gather the useful data missing by the current MySQL monitor tools: </li></ul></ul>
  28. 28. DTrace: Monitor I/Os <ul><li>Exam I/O wait time by filename and mysqld(Available at DTraceToolkit)‏ </li></ul><ul><ul><li>#!/usr/sbin/dtrace –s </li></ul></ul><ul><ul><li>#pragma D option quiet </li></ul></ul><ul><ul><li>io:::wait-start </li></ul></ul><ul><ul><li>/ execname == “mysqld” / </li></ul></ul><ul><ul><li>{ self->start = timestamp; } </li></ul></ul><ul><ul><li>io:::wait-done </li></ul></ul><ul><ul><li>/ execname == “mysqld” && self->start / </li></ul></ul><ul><ul><li>{ this->elapsed = timestamp - self->start; </li></ul></ul><ul><ul><li>@files[pid,args[1]->dev_pathname, args[2]->fi_pathname] = sum(this->elapsed); </li></ul></ul><ul><ul><li>self->start = 0; </li></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li>profile:::tick-5s </li></ul></ul><ul><ul><li>{ printf(“------------------------------------------------- ”); </li></ul></ul><ul><ul><li>printf(“%6s %8s %20s %50s ”, “PID”, “TIME”, “DEVICE”, “FILE”); </li></ul></ul><ul><ul><li>printa(“%6d %@8d %20s %8s ”, @files); </li></ul></ul><ul><ul><li>printf (“------------------------------------------------ ”); } </li></ul></ul><ul><ul><li>#./mysqliowait.d </li></ul></ul><ul><ul><li>--------------------------------------------------- </li></ul></ul><ul><ul><li>PID TIME DEVICE FILE </li></ul></ul><ul><ul><li>113 234 /dev/dsk/c0t2d0s0 /usr/local/mysql/data/ibdata1 </li></ul></ul>
  29. 29. DTrace: Monitor CPU <ul><li>classic performance problem </li></ul><ul><ul><li>#!/usr/sbin/dtrace -s </li></ul></ul><ul><ul><li>#pragma D option quiet </li></ul></ul><ul><ul><li>syscall:::entry </li></ul></ul><ul><ul><li>{ </li></ul></ul><ul><ul><li> self->ts=vtimestamp; </li></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li>syscall:::return </li></ul></ul><ul><ul><li>/self->ts/ </li></ul></ul><ul><ul><li>{ </li></ul></ul><ul><ul><li> @a[execname, probefunc] = count(); </li></ul></ul><ul><ul><li> @b[execname, probefunc] = sum (vtimestamp - self->ts); </li></ul></ul><ul><ul><li> self->ts=0; </li></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li>END </li></ul></ul><ul><ul><li> { </li></ul></ul><ul><ul><li>printf(&quot;%-16s %-16s %-8s &quot;,&quot;EXEC&quot;,&quot;SYSCALL&quot;,&quot;COUNT&quot;); </li></ul></ul><ul><ul><li>printa(&quot;%-16s %-16s %-@8d &quot;,@a); </li></ul></ul><ul><ul><li>printf(&quot;%-16s %-16s %-8s &quot;,&quot;EXEC&quot;,&quot;SYSCALL&quot;,&quot;TIME&quot;); </li></ul></ul><ul><ul><li>printa(&quot;%-16s %-16s %-@8d &quot;,@b); </li></ul></ul><ul><ul><li>} </li></ul></ul>
  30. 30. <ul><ul><li># ./syscall.d </li></ul></ul><ul><ul><li>EXEC SYSCALL COUNT </li></ul></ul><ul><ul><li>devfsadm lwp_park 1 </li></ul></ul><ul><ul><li>dtrace fstat 1 </li></ul></ul><ul><ul><li>... </li></ul></ul><ul><ul><li>mysqld read 106542 </li></ul></ul><ul><ul><li>mysqld gtime 109613 </li></ul></ul><ul><ul><li>mysqld pread 1181669 </li></ul></ul><ul><ul><li>--------------------------------------------------- </li></ul></ul><ul><ul><li>EXEC SYSCALL TIME </li></ul></ul><ul><ul><li>dtrace lwp_sigmask 218 </li></ul></ul><ul><ul><li>pkill getpid 302 </li></ul></ul><ul><ul><li>... </li></ul></ul><ul><ul><li>mysqld read 259284183 </li></ul></ul><ul><ul><li>mysqld write 267556239 </li></ul></ul><ul><ul><li>mysqld pread 4650457224 </li></ul></ul>
  31. 31. <ul><li>Agrregated user stack backtrace to understand of the nature of pread() in MySQL source code – useful for mysql developers </li></ul><ul><ul><ul><li># dtrace -n 'syscall::pread:entry / execname == &quot;mysqld&quot; / { @[ustack()]=count() } </li></ul></ul></ul><ul><ul><li>dtrace: description 'syscall::pread:entry ' matched 1 probe </li></ul></ul><ul><ul><li>... </li></ul></ul><ul><ul><li>libc.so.1`_pread+0xa </li></ul></ul><ul><ul><li>mysqld`my_pread+0x54 </li></ul></ul><ul><ul><li>mysqld`_mi_read_static_record+0x67 </li></ul></ul><ul><ul><li>mysqld`mi_rnext+0x1fe </li></ul></ul><ul><ul><li>... </li></ul></ul><ul><ul><li>mysqld`handle_one_connection+0x855 </li></ul></ul><ul><ul><li>libc.so.1`_thr_setup+0x67 </li></ul></ul><ul><ul><li>1564811 </li></ul></ul><ul><ul><li>.... </li></ul></ul><ul><ul><li>Replace high cost pread with </li></ul></ul><ul><ul><li>“ --myisam_use_mmap=1”- useful for mysql DBA </li></ul></ul><ul><li>A 94% performance improvement ! </li></ul>
  32. 32. Dtrace: Memory Analysis <ul><li>Check which process causes anonymous page in </li></ul><ul><ul><li># dtrace -n anonpgin '{@[execname] = count()}' </li></ul></ul><ul><ul><li>dtrace: description anonpgin matched 1 probe </li></ul></ul><ul><ul><li>sshd 2 </li></ul></ul><ul><ul><li>vmstat 23 </li></ul></ul><ul><ul><li>mysqld 673 </li></ul></ul><ul><ul><li>Use Dtrace to measure waiting for paging in </li></ul></ul><ul><ul><li># ./ whospaging.d – available at Solaris Internals( http://www.solarisinternals.com/si/dtrace/ )‏ </li></ul></ul><ul><ul><li>Who's on cpu (milliseconds): </li></ul></ul><ul><ul><li>sshd 1 </li></ul></ul><ul><ul><li>vmstat 3 </li></ul></ul><ul><ul><li>mysqld 120 </li></ul></ul><ul><ul><li>sched 43210 </li></ul></ul><ul><ul><li>Who's waiting for pagin (milliseconds): </li></ul></ul><ul><ul><li>mysqld 239082 </li></ul></ul>
  33. 33. Dtrace Probes In MySQL <ul><li>Provide deep view of internal MySQL core server and storage engines' operation & behaviour </li></ul><ul><ul><li>Database information </li></ul></ul><ul><ul><li>Query execution latency </li></ul></ul><ul><ul><li>Index & table scan cost </li></ul></ul><ul><ul><li>Wait events inside MyISAM & Innodb </li></ul></ul><ul><ul><li>Deadlock information </li></ul></ul><ul><ul><li>Query cache hit/miss </li></ul></ul><ul><ul><li>And many more... </li></ul></ul><ul><li>Speed resolution of performance bottlenecks with in database design and MySQL server configurations </li></ul><ul><li>Negligible performance overhead </li></ul><ul><li>Easy steps to create & insert your own Dtrace probes into MySQL </li></ul><ul><li>MySQL DTrace GUI Monitor Tool - Chime </li></ul>
  34. 34. Query execution Time <ul><li>Enable Slow query log with “–log-slow-queries” requires re-start MySQL server </li></ul><ul><li>SQL statements with query execution time longer than “long_query_time” second in the log file </li></ul><ul><li>SQL statements generating most loads on the application may not in slow query log </li></ul><ul><li>Replication query statements are not available in slow query log </li></ul><ul><li>Time spending by the query optimizer to generate query plan is not available in slow query log </li></ul><ul><li>Using DTrace can get mising query execution information online </li></ul>
  35. 35. Insert DTrace Probes Into MySQL <ul><li>Step 1: Figure out what probes are needed to insert into the source code </li></ul><ul><li>Step 2: Define MySQL Provider and probes </li></ul><ul><ul><li># cat mysql_dtrace.d </li></ul></ul><ul><ul><li>provider mysql </li></ul></ul><ul><ul><li>{ </li></ul></ul><ul><ul><li>probe query__execute__start(void *, char *, char *, const char *, char *); </li></ul></ul><ul><ul><li>probe query__execute__finish(void *, char *, char *, const char *, char *,int); </li></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li>Two Probes defined in the mysql provider </li></ul></ul><ul><ul><li>Note to use two underscore(__) translated to hypen automatically </li></ul></ul>
  36. 36. <ul><li>Step 3: Define a header file “mysql_dtrace.h” with definitions for probes </li></ul><ul><ul><li>dtrace -h -s mysql_dtrace.d </li></ul></ul><ul><ul><ul><li>#ifndef _MYSQL_H </li></ul></ul></ul><ul><ul><ul><li>#define _MYSQL_H </li></ul></ul></ul><ul><ul><ul><li>#define DTRACE_QUERY_EXECUTE_START(arg0, arg1, arg2, arg3, arg4) </li></ul></ul></ul><ul><ul><ul><li>__dtrace_mysql___query__execute__start(arg0, arg1, arg2, arg3, arg4)‏ </li></ul></ul></ul><ul><ul><ul><li>#define DTRACE_QUERY_EXECUTE_START_ENABLED() </li></ul></ul></ul><ul><ul><ul><li>__dtraceenabled_mysql___query__execute__start()‏ </li></ul></ul></ul><ul><ul><ul><li>extern void __dtrace_mysql___query__execute__start(void *, char *, char *, char *, char *); </li></ul></ul></ul><ul><ul><ul><li>extern int __dtraceenabled_mysql___query__execute__start(void); </li></ul></ul></ul><ul><ul><ul><li>#endif </li></ul></ul></ul>
  37. 37. <ul><li>Step 4: Insert the probes into source code </li></ul><ul><ul><li>#include <mysql_dtrace.h> </li></ul></ul><ul><ul><li>... </li></ul></ul><ul><ul><li>bool mysql_execute_command(THD *thd)‏ </li></ul></ul><ul><ul><li>{ </li></ul></ul><ul><ul><li>DTRACE_QUERY_EXECUTE_START((void *)thd, thd->db, </li></ul></ul><ul><ul><li>thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip,thd->query); </li></ul></ul><ul><ul><li>... </li></ul></ul><ul><ul><li>DTRACE_QUERY_EXECUTE_FINISH((void*)thd, thd->db, </li></ul></ul><ul><ul><li>thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip, </li></ul></ul><ul><ul><li>thd->query, res ==0 ? 0: -1); </li></ul></ul><ul><ul><li>go to end; </li></ul></ul><ul><ul><li>... </li></ul></ul><ul><ul><li>} </li></ul></ul>
  38. 38. <ul><li>Step 5: Build MySQL with DTrace </li></ul><ul><li>In the Makefile.in, compile 64-bit MySQL with Dtrace </li></ul><ul><ul><li>mysqld_OBJECTS = $(am_mysqld_OBJECTS) mysql_dtrace.$(OBJEXT)‏ </li></ul></ul><ul><ul><li>mysql_dtrace.o:$(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS)‏ </li></ul></ul><ul><ul><li>dtrace -G 64 -s $(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS) </li></ul></ul><ul><li>Inserting DTrace probes comleted , DTrace probes are ready to use! </li></ul>
  39. 39. <ul><li>Step 6: Use inserted DTrace probes to measure query execution time with other database information </li></ul><ul><ul><li>#!/usr/sbin/dtrace -s </li></ul></ul><ul><ul><li>… </li></ul></ul><ul><ul><li>mysql*:::query-execute-start </li></ul></ul><ul><ul><li>{ self->start = timestamp; } </li></ul></ul><ul><ul><li>mysql*:::query-execute-finish </li></ul></ul><ul><ul><li>/self->start/ </li></ul></ul><ul><ul><li>{ this->query = copyinstr(arg4); } </li></ul></ul><ul><ul><li>mysql*:::query-execute-finish </li></ul></ul><ul><ul><li>/self->start/ </li></ul></ul><ul><ul><li>{ this->elapsed = (timestamp - self->start) / 1000000; </li></ul></ul><ul><ul><li>this->who = strjoin(copyinstr(arg2), strjoin(&quot;@&quot;, copyinstr(arg3))); </li></ul></ul><ul><ul><li>printf(&quot; %-16.16s %-18.18s %5d %3d %-32.32s &quot;, arg1 ? copyinstr(arg1)‏ </li></ul></ul><ul><ul><li>: &quot;.&quot;, this->who, this->elapsed, (int)arg5, this->query); self->start = 0; </li></ul></ul><ul><ul><li>} </li></ul></ul>
  40. 40. <ul><li># ./mysqld_qestat.d </li></ul><ul><ul><li>DATABASE USER@HOST ms RET QUERY </li></ul></ul><ul><ul><li>sbtest root@localhost 0 0 show tables </li></ul></ul><ul><ul><li>sbtest root@localhost 0 0 show databases </li></ul></ul><ul><ul><li>sbtest root@localhost 178 0 select * from sbtest </li></ul></ul><ul><li>Use the same steps to insert Dtrace probes to <source_tree>/sql/sql_select.cc at the start and end of choose_plan() function to measure the time spent in query optimization </li></ul><ul><li>optimizer_prune_level=1 reduce query compilation time </li></ul><ul><li>Reduce optimizer_search_depth or optimizer_search_depth=0 </li></ul>
  41. 41. Index & Table scan cost <ul><li>Identify the places in MySQL source to handle scanning index, and table </li></ul><ul><ul><li>Index-scan functions: index_next, index_next_same, index_prev, index_first, index_last </li></ul></ul><ul><ul><li>Table-scan functions: rnd_init, rnd_end, rnd_next, rnd_pos </li></ul></ul><ul><ul><li>Insert DTrace Probes at the start and before return from the functions can measure the time spending on scanning table or index. </li></ul></ul><ul><ul><ul><li>mysql*:::innodb-index-next-start </li></ul></ul></ul><ul><ul><ul><li>{ @indexnext[args[0]] = count(); </li></ul></ul></ul><ul><ul><li> self->inext = timestamp; } </li></ul></ul><ul><ul><ul><li>mysql*:::innodb-index-next-finish </li></ul></ul></ul><ul><ul><ul><li>/self->inext/ </li></ul></ul></ul><ul><ul><ul><li>{ @indexnexttime[args[0]] = sum(timestamp - self->inext); </li></ul></ul></ul><ul><ul><li>self->inext = 0; } </li></ul></ul><ul><li>Expensive index-scan/table-scan report from Dtrace requires to optimize schema accordingly </li></ul>
  42. 42. Buffer wait in Innodb <ul><ul><li>Innodb buffer wait is common in I/O-bound MySQL system while reading </li></ul></ul><ul><ul><ul><li>page synchronous from disk </li></ul></ul></ul><ul><ul><li>Insert Dtrace probes at: innobase/buf/buf0rea.c, </li></ul></ul><ul><ul><li>ulint buf_read_page( ulint space, ulint offset)‏ </li></ul></ul><ul><ul><li>{ ... </li></ul></ul><ul><ul><li>DTRACE_INNODB_BUFFER_WAIT_START(); </li></ul></ul><ul><ul><li>/* We do the i/o in the synchronous aio mode to save thread </li></ul></ul><ul><ul><li>switches: hence TRUE */ </li></ul></ul><ul><ul><li>count2 = buf_read_page_low(&err, TRUE, BUF_READ_ANY_PAGE, space, </li></ul></ul><ul><ul><li>tablespace_version, offset); </li></ul></ul><ul><ul><li>DTRACE_INNODB_BUFFER_WAIT_FINISH(); </li></ul></ul><ul><ul><li>... </li></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li>Increase innodb_buffer_pool size </li></ul></ul><ul><ul><li>Tune SQL to access rows with fewer block reads(i.e. By adding indexes)‏ </li></ul></ul>
  43. 43. Query cache probes <ul><li>Use DTrace to measure query cache hit and query cache miss to determine how well the query cache is performing </li></ul><ul><ul><li>> Insert DTrace Probes at: sql/sql_cache.cc function: send_result_to_client </li></ul></ul><ul><ul><li>> DTrace Test script to report query cache hit and miss counts by query string, and totals </li></ul></ul><ul><ul><li>mysql*:::query-cache-hit, </li></ul></ul><ul><ul><li>mysql*:::query-cache-miss </li></ul></ul><ul><ul><li>{ this->query = copyinstr(arg4); } </li></ul></ul><ul><ul><ul><li>mysql*:::query-cache-hit </li></ul></ul></ul><ul><ul><ul><li>{ @elapsed[this->query, &quot;hit&quot;] = count(); </li></ul></ul></ul><ul><ul><li>hits++; } </li></ul></ul><ul><ul><li>mysql*:::query-cache- miss </li></ul></ul><ul><ul><li>{ @elapsed[this->query, &quot;miss&quot;] = count(); </li></ul></ul><ul><ul><li>misses++; } </li></ul></ul>
  44. 44. <ul><ul><li># ./mysqld_qchit.d </li></ul></ul><ul><ul><li>Tracing... Hit Ctrl-C to end. </li></ul></ul><ul><ul><li>^C </li></ul></ul><ul><ul><li>QUERY RESULT COUNT </li></ul></ul><ul><ul><li>select * from months miss 1 </li></ul></ul><ul><ul><li>select * from months where num > 3 hit 1 </li></ul></ul><ul><ul><li>select * from months where num > 3 and num < 9 miss 1 </li></ul></ul><ul><ul><li>show databases miss 1 </li></ul></ul><ul><ul><li>show tables miss 1 </li></ul></ul><ul><ul><li>select * from months hit 9 </li></ul></ul><ul><ul><li>Hits : 10 </li></ul></ul><ul><ul><li>Misses : 4 </li></ul></ul><ul><ul><li>Hit Rate : 71% </li></ul></ul><ul><li>Tuning “query_cache_size” variable according to the hit/miss rate </li></ul>
  45. 45. DTrace Performance Impact <ul><li>Inserting DTrace Probes into MySQL source code are useful for MySQL DBA, MySQL & application developers. The performance impact of adding DTrace probes is critical for enterprise environment. </li></ul><ul><li>Cost of inserting USDT probe can be basically negligible: </li></ul><ul><ul><li>Each probe inserted into the source code can be enabled by adding the code like: </li></ul></ul><ul><ul><li>if (PROVIDER_PROBE_ENABLED()‏ </li></ul></ul><ul><ul><li>{ </li></ul></ul><ul><ul><li>PROVIDER_PROBE(arg0,...); </li></ul></ul><ul><ul><li>} </li></ul></ul>
  46. 46. DTrace Probes In MySQL 6.0 <ul><ul><ul><li>probe insert_row_start(); probe insert_row_end(); probe filesort_start(); probe filesort_end(); probe delete_start(); probe delete_end(); probe select_start() probe select_end(); probe update_start(); probe update_end(); </li></ul></ul></ul><ul><ul><li>compile with “--enable-dtrace” configure option to use the Dtrace probes </li></ul></ul>
  47. 47. Integrate with Chime Tool <ul><ul><li>Chime is graphical tool for visualizing DTrace aggregations. It </li></ul></ul><ul><ul><li>provides alternative CLI-based tool output with more visually </li></ul></ul><ul><ul><li>apprealing and more useful to display data over time </li></ul></ul><ul><ul><li>Available to download: http://www.opensolaris.org/os/project/dtrace-chime . One-step installation: </li></ul></ul><ul><ul><li>> Run pkgadd -d osol0chime-<arch>-1.4.pkg </li></ul></ul><ul><ul><li>chime requires Solaris Nevada build 35 or later </li></ul></ul><ul><ul><li>> Run local: /opt/OSOL0chime/bin/chime </li></ul></ul><ul><ul><li>> Run remotely: /opt/OSOL0chime/bin/chime <hostname> </li></ul></ul><ul><ul><li>chime provides wizard to automatically generate new display for DTrace script </li></ul></ul>
  48. 49. <ul><li>Resources </li></ul><ul><ul><li>> http:// www.opensolaris.org /os/community/dtrace </li></ul></ul><ul><ul><li>- OpenSolaris Community: Dtrace </li></ul></ul><ul><ul><li>> http://developers.sun.com/solaris/articles/solaris_perftools.html </li></ul></ul><ul><ul><li>- Solaris Performance and Tools </li></ul></ul><ul><ul><li>> http://docs.sun.com/app/docs/doc/817-6223/6mlkidlms?a=view </li></ul></ul><ul><ul><li>- Statically Defined Tracing for User Applications chapter of DTrace </li></ul></ul><ul><ul><li>Manual </li></ul></ul><ul><ul><li>> http://www.brendangregg.com </li></ul></ul><ul><ul><li>- DTrace toolkits </li></ul></ul><ul><ul><li>> http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Databases </li></ul></ul><ul><ul><li>- DTrace Topics Databases </li></ul></ul><ul><ul><li>Acknowledgements </li></ul></ul><ul><ul><li>Brandan Gregg – Sun Microsystems Engineer in Advanced Products Group </li></ul></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×