• Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,759
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
57
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Best Practices For Deploying MySQL on Solaris
    • Ritu Kamboj
    • Jenny Chen
  • 2. Agenda
    • MySQL -Solaris Integration
    • MySQL High Availability Data Service
    • Consolidate MySQL Deployment Using Solaris Containers
    • DTrace (Jenny Chen)‏
  • 3.
    • MySQL-Solaris Integration
  • 4. Optimized MySQL on Open Solaris
    • MySQL 5.0.45 (32bit) integrated with Open Solaris build 76
      • SXDE 01/08
    • MySQL 5.0.45 (64bit) integrated with Open Solaris build 87
      • SXCE
  • 5. Layout of MySQL on Open Solaris
    • MySQL 5.0.45 packages
      • Default data directory
        • /var/mysql/5.0/data
      • Default configuration directory
        • /etc/mysql/5.0
      • Installation directory
        • / usr/mysql/5.0/
    • Latest version accessible from /usr/mysql/bin
      • Symbolic link from all directories involved in 5.0.45 release to corresponding /usr/mysql
  • 6. Optimization Of MySQL On Open Solaris
    • Compiled with optimal compiler options using Sun Studio compilers
    • Added SMF support for MySQL
      • Easily accommodates varying configurations
      • Initializes MySQL database thereby enhancing user experience
      • Mysql user can manage MySQL database via SMF
    • Support for MySQL cluster engine (ndb)‏
  • 7. Compiler Optimization
    • Enable in-lining
      • Change header file univ.i to add Sun Studio
        • If (!defined(SUNPRO_C)‏
        • #define UNIV_MUST_NOT_INLINE
    • Enable pre-fetching
      • -xprefetch=auto and -xprefetch_level=3
    • Set optimization level
      • - xO4
  • 8. Compiler Optimization (Contd)‏
    • Feedback optimization (not yet implemented)‏
      • Workload is key
      • -xprofile -xipo
      • Interprocess optimization
      • About 10% improvement
    • Link with libmtmalloc
        • Library for threaded application
        • About 8% improvement
  • 9. Service Management Facility (SMF)‏
    • Makes Solaris Services Self-Healing
    • Services automatically restart in dependency order
    • Misbehaving and mis-configured services are easier to debug
      • Log files for each service
    • Administrators can securely delegate tasks to non-root users
  • 10. MySQL-Service Management Facility
    • Dynamically configured properties
      • mysql/bin
        • Installation , default : /usr/mysql/5.0/bin
      • mysql/data
        • Data Directory , Default: /var/mysql/5.0/data
      • mysql/enable_64bit
        • Flag to select 32bit or 64bit , Default : false
    • First time installations
      • Creates system tables (mysql_install_db.sh )‏
  • 11. Starting MySQL on Open Solaris
    • Older Open Solaris Build (Build 76- Build-87)‏
      • 32Bit MySQL integrated
      • Create mysql user
      • Enable MySQL SMF service
    • Latest Open Solaris Build (Build 87 onwards) ‏
        • 32Bit and 64bit MySQL integrated
        • Default mode : 32bit service
        • For starting 64bit
          • Set enable_64bit == true
  • 12. Optimized MySQL on Solaris 10
    • CoolStack
      • SAMP stack + more
    • Optimized MySQL download
      • Standalone package
      • SAMP stack component
    • Available at Sun Download center:
      • http://cooltools.sunsource.net/coolstack
    • Version MySQL 5.0.45
    • Similar optimizations as in Open Solaris
  • 13.
    • MySQL-High Availability Data Service
  • 14. Solaris Cluster Overview
    • Provides general purpose HA platform
    Fifty percent of enterprises that lack a recovery plan go out of business within one year of a significant disaster Gartner Group Availability is our customers' most critical requirement Sun Cluster VOC Survey
  • 15. Solaris Cluster : Hardware Components
    • Servers with local storage
      • Can have up to 16 nodes
    • Shared storage
      • Tolerates single-node failures
      • Centralizes configuration files
    • Cluster interconnect
      • At least two redundant networks
    • Public network interfaces
      • Spreads outbound packets
  • 16. Solaris Cluster Algorithms
    • Cluster membership monitor
      • Ensures data integrity
      • Determines cluster membership
    • Cluster configuration repository
      • Global repository
      • Ensures consistent view
    • Disk Fencing
      • Fences off non-cluster nodes
      • Prevents Partition
    • Quorum
      • Uses a majority voting schema
  • 17. MySQL High Availability Data Service
    • HA-MySQL is a failover data service
    Node 1 Node 2 Node 3 Node 4 DB
  • 18. MySQL High Availability Data Service
    • Supported configurations
      • Standalone MySQL server
      • MySQL replication server
        • Single/Multiple MySQL instances in master configuration
        • Single/Multiple MySQL instances in slave configuration
    • Solaris containers support
      • Global zone
      • Non-global failover zone
      • Non-global zone
  • 19. HA-MySQL Service Deployment Node1 Node2:Zone2 Node1:Zone1 Node2 MySQL Stor Host MySQL Stor Host
  • 20. MySQL-Solaris Cluster Benefits
    • Enhanced end-to-end infrastructure availability
    • Continuous MySQL Availability
      • Automatic failover if master node fails
    • Low cost solution
      • Software is free and open sourced
    • Efficient Resource Utilization
      • Multiple applications can be consolidated
    • Ease of operations
      • SC enables clustered systems to be managed as if they were on a single system
  • 21. Additional information
    • Step by step deployment guide
      • http://docs.sun.com/app/docs/doc/819-3059
    • Failover study of HA-MySQL
      • http://blogs.sun.com/krishs/date/200804
    • Solaris Cluster
      • http://www.sun.com/software/solaris/cluster/index.xml
      • http://opensolaris.org/os/community/ha-clusters/
  • 22.
      • Consolidate MySQL installations using Solaris Containers
  • 23. Solaris Containers
    • Containers : Zones + Resource Management
      • Zones: isolated virtual application environments
      • Resource management – resource control (CPU, Memory)‏
    • Achieving Consolidation Goals
      • Reduce Hardware
        • Combine low utilization systems
      • Isolate applications from faults
      • Maintain Service Levels
        • Fine tune response times
  • 24. MySQL Consolidation Study
    • System Configuration
      • Sun Fire X4100 (4 CPU, 8 GB Memory)‏
    • Local container configuration ( 1 CPU , 1 GB Memory
    • Sysbench Read-only Sysbench read-write
  • 25.
    • Dtrace (Jenny Chen)‏
  • 26. DTrace
    • Use DTrace with MySQL to drill down Monitoring MySQL Performance
    • Examples & Solutions
    • Easy Steps To add DTrace probes into MySQL core server and storage Engines
    • Easy Steps to display MySQL DTrace probes into Chime visualization Tool for DTrace
  • 27. Why DTrace
    • Solaris 10 Dynamic Tracing Facility to provide comprehensive view of operating system and application behaviour
    • DTrace to examine particular system areas: disk I/O, CPU, Memory
    • Process Tracing and Debugging
      • USDT (User-level statically defined tracing) place custom probes in application code
      • Add USDT into MySQL source to monitor MySQL and gather the useful data missing by the current MySQL monitor tools:
  • 28. DTrace: Monitor I/Os
    • Exam I/O wait time by filename and mysqld(Available at DTraceToolkit)‏
      • #!/usr/sbin/dtrace –s
      • #pragma D option quiet
      • io:::wait-start
      • / execname == “mysqld” /
      • { self->start = timestamp; }
      • io:::wait-done
      • / execname == “mysqld” && self->start /
      • { this->elapsed = timestamp - self->start;
      • @files[pid,args[1]->dev_pathname, args[2]->fi_pathname] = sum(this->elapsed);
      • self->start = 0;
      • }
      • profile:::tick-5s
      • { printf(“------------------------------------------------- ”);
      • printf(“%6s %8s %20s %50s ”, “PID”, “TIME”, “DEVICE”, “FILE”);
      • printa(“%6d %@8d %20s %8s ”, @files);
      • printf (“------------------------------------------------ ”); }
      • #./mysqliowait.d
      • ---------------------------------------------------
      • PID TIME DEVICE FILE
      • 113 234 /dev/dsk/c0t2d0s0 /usr/local/mysql/data/ibdata1
  • 29. DTrace: Monitor CPU
    • classic performance problem
      • #!/usr/sbin/dtrace -s
      • #pragma D option quiet
      • syscall:::entry
      • {
      • self->ts=vtimestamp;
      • }
      • syscall:::return
      • /self->ts/
      • {
      • @a[execname, probefunc] = count();
      • @b[execname, probefunc] = sum (vtimestamp - self->ts);
      • self->ts=0;
      • }
      • END
      • {
      • printf("%-16s %-16s %-8s ","EXEC","SYSCALL","COUNT");
      • printa("%-16s %-16s %-@8d ",@a);
      • printf("%-16s %-16s %-8s ","EXEC","SYSCALL","TIME");
      • printa("%-16s %-16s %-@8d ",@b);
      • }
  • 30.
      • # ./syscall.d
      • EXEC SYSCALL COUNT
      • devfsadm lwp_park 1
      • dtrace fstat 1
      • ...
      • mysqld read 106542
      • mysqld gtime 109613
      • mysqld pread 1181669
      • ---------------------------------------------------
      • EXEC SYSCALL TIME
      • dtrace lwp_sigmask 218
      • pkill getpid 302
      • ...
      • mysqld read 259284183
      • mysqld write 267556239
      • mysqld pread 4650457224
  • 31.
    • Agrregated user stack backtrace to understand of the nature of pread() in MySQL source code – useful for mysql developers
        • # dtrace -n 'syscall::pread:entry / execname == "mysqld" / { @[ustack()]=count() }
      • dtrace: description 'syscall::pread:entry ' matched 1 probe
      • ...
      • libc.so.1`_pread+0xa
      • mysqld`my_pread+0x54
      • mysqld`_mi_read_static_record+0x67
      • mysqld`mi_rnext+0x1fe
      • ...
      • mysqld`handle_one_connection+0x855
      • libc.so.1`_thr_setup+0x67
      • 1564811
      • ....
      • Replace high cost pread with
      • “ --myisam_use_mmap=1”- useful for mysql DBA
    • A 94% performance improvement !
  • 32. Dtrace: Memory Analysis
    • Check which process causes anonymous page in
      • # dtrace -n anonpgin '{@[execname] = count()}'
      • dtrace: description anonpgin matched 1 probe
      • sshd 2
      • vmstat 23
      • mysqld 673
      • Use Dtrace to measure waiting for paging in
      • # ./ whospaging.d – available at Solaris Internals( http://www.solarisinternals.com/si/dtrace/ )‏
      • Who's on cpu (milliseconds):
      • sshd 1
      • vmstat 3
      • mysqld 120
      • sched 43210
      • Who's waiting for pagin (milliseconds):
      • mysqld 239082
  • 33. Dtrace Probes In MySQL
    • Provide deep view of internal MySQL core server and storage engines' operation & behaviour
      • Database information
      • Query execution latency
      • Index & table scan cost
      • Wait events inside MyISAM & Innodb
      • Deadlock information
      • Query cache hit/miss
      • And many more...
    • Speed resolution of performance bottlenecks with in database design and MySQL server configurations
    • Negligible performance overhead
    • Easy steps to create & insert your own Dtrace probes into MySQL
    • MySQL DTrace GUI Monitor Tool - Chime
  • 34. Query execution Time
    • Enable Slow query log with “–log-slow-queries” requires re-start MySQL server
    • SQL statements with query execution time longer than “long_query_time” second in the log file
    • SQL statements generating most loads on the application may not in slow query log
    • Replication query statements are not available in slow query log
    • Time spending by the query optimizer to generate query plan is not available in slow query log
    • Using DTrace can get mising query execution information online
  • 35. Insert DTrace Probes Into MySQL
    • Step 1: Figure out what probes are needed to insert into the source code
    • Step 2: Define MySQL Provider and probes
      • # cat mysql_dtrace.d
      • provider mysql
      • {
      • probe query__execute__start(void *, char *, char *, const char *, char *);
      • probe query__execute__finish(void *, char *, char *, const char *, char *,int);
      • }
      • Two Probes defined in the mysql provider
      • Note to use two underscore(__) translated to hypen automatically
  • 36.
    • Step 3: Define a header file “mysql_dtrace.h” with definitions for probes
      • dtrace -h -s mysql_dtrace.d
        • #ifndef _MYSQL_H
        • #define _MYSQL_H
        • #define DTRACE_QUERY_EXECUTE_START(arg0, arg1, arg2, arg3, arg4)
        • __dtrace_mysql___query__execute__start(arg0, arg1, arg2, arg3, arg4)‏
        • #define DTRACE_QUERY_EXECUTE_START_ENABLED()
        • __dtraceenabled_mysql___query__execute__start()‏
        • extern void __dtrace_mysql___query__execute__start(void *, char *, char *, char *, char *);
        • extern int __dtraceenabled_mysql___query__execute__start(void);
        • #endif
  • 37.
    • Step 4: Insert the probes into source code
      • #include <mysql_dtrace.h>
      • ...
      • bool mysql_execute_command(THD *thd)‏
      • {
      • DTRACE_QUERY_EXECUTE_START((void *)thd, thd->db,
      • thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip,thd->query);
      • ...
      • DTRACE_QUERY_EXECUTE_FINISH((void*)thd, thd->db,
      • thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip,
      • thd->query, res ==0 ? 0: -1);
      • go to end;
      • ...
      • }
  • 38.
    • Step 5: Build MySQL with DTrace
    • In the Makefile.in, compile 64-bit MySQL with Dtrace
      • mysqld_OBJECTS = $(am_mysqld_OBJECTS) mysql_dtrace.$(OBJEXT)‏
      • mysql_dtrace.o:$(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS)‏
      • dtrace -G 64 -s $(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS)
    • Inserting DTrace probes comleted , DTrace probes are ready to use!
  • 39.
    • Step 6: Use inserted DTrace probes to measure query execution time with other database information
      • #!/usr/sbin/dtrace -s
      • mysql*:::query-execute-start
      • { self->start = timestamp; }
      • mysql*:::query-execute-finish
      • /self->start/
      • { this->query = copyinstr(arg4); }
      • mysql*:::query-execute-finish
      • /self->start/
      • { this->elapsed = (timestamp - self->start) / 1000000;
      • this->who = strjoin(copyinstr(arg2), strjoin(&quot;@&quot;, copyinstr(arg3)));
      • printf(&quot; %-16.16s %-18.18s %5d %3d %-32.32s &quot;, arg1 ? copyinstr(arg1)‏
      • : &quot;.&quot;, this->who, this->elapsed, (int)arg5, this->query); self->start = 0;
      • }
  • 40.
    • # ./mysqld_qestat.d
      • DATABASE USER@HOST ms RET QUERY
      • sbtest root@localhost 0 0 show tables
      • sbtest root@localhost 0 0 show databases
      • sbtest root@localhost 178 0 select * from sbtest
    • Use the same steps to insert Dtrace probes to <source_tree>/sql/sql_select.cc at the start and end of choose_plan() function to measure the time spent in query optimization
    • optimizer_prune_level=1 reduce query compilation time
    • Reduce optimizer_search_depth or optimizer_search_depth=0
  • 41. Index & Table scan cost
    • Identify the places in MySQL source to handle scanning index, and table
      • Index-scan functions: index_next, index_next_same, index_prev, index_first, index_last
      • Table-scan functions: rnd_init, rnd_end, rnd_next, rnd_pos
      • Insert DTrace Probes at the start and before return from the functions can measure the time spending on scanning table or index.
        • mysql*:::innodb-index-next-start
        • { @indexnext[args[0]] = count();
      • self->inext = timestamp; }
        • mysql*:::innodb-index-next-finish
        • /self->inext/
        • { @indexnexttime[args[0]] = sum(timestamp - self->inext);
      • self->inext = 0; }
    • Expensive index-scan/table-scan report from Dtrace requires to optimize schema accordingly
  • 42. Buffer wait in Innodb
      • Innodb buffer wait is common in I/O-bound MySQL system while reading
        • page synchronous from disk
      • Insert Dtrace probes at: innobase/buf/buf0rea.c,
      • ulint buf_read_page( ulint space, ulint offset)‏
      • { ...
      • DTRACE_INNODB_BUFFER_WAIT_START();
      • /* We do the i/o in the synchronous aio mode to save thread
      • switches: hence TRUE */
      • count2 = buf_read_page_low(&err, TRUE, BUF_READ_ANY_PAGE, space,
      • tablespace_version, offset);
      • DTRACE_INNODB_BUFFER_WAIT_FINISH();
      • ...
      • }
      • Increase innodb_buffer_pool size
      • Tune SQL to access rows with fewer block reads(i.e. By adding indexes)‏
  • 43. Query cache probes
    • Use DTrace to measure query cache hit and query cache miss to determine how well the query cache is performing
      • > Insert DTrace Probes at: sql/sql_cache.cc function: send_result_to_client
      • > DTrace Test script to report query cache hit and miss counts by query string, and totals
      • mysql*:::query-cache-hit,
      • mysql*:::query-cache-miss
      • { this->query = copyinstr(arg4); }
        • mysql*:::query-cache-hit
        • { @elapsed[this->query, &quot;hit&quot;] = count();
      • hits++; }
      • mysql*:::query-cache- miss
      • { @elapsed[this->query, &quot;miss&quot;] = count();
      • misses++; }
  • 44.
      • # ./mysqld_qchit.d
      • Tracing... Hit Ctrl-C to end.
      • ^C
      • QUERY RESULT COUNT
      • select * from months miss 1
      • select * from months where num > 3 hit 1
      • select * from months where num > 3 and num < 9 miss 1
      • show databases miss 1
      • show tables miss 1
      • select * from months hit 9
      • Hits : 10
      • Misses : 4
      • Hit Rate : 71%
    • Tuning “query_cache_size” variable according to the hit/miss rate
  • 45. DTrace Performance Impact
    • Inserting DTrace Probes into MySQL source code are useful for MySQL DBA, MySQL & application developers. The performance impact of adding DTrace probes is critical for enterprise environment.
    • Cost of inserting USDT probe can be basically negligible:
      • Each probe inserted into the source code can be enabled by adding the code like:
      • if (PROVIDER_PROBE_ENABLED()‏
      • {
      • PROVIDER_PROBE(arg0,...);
      • }
  • 46. DTrace Probes In MySQL 6.0
        • probe insert_row_start(); probe insert_row_end(); probe filesort_start(); probe filesort_end(); probe delete_start(); probe delete_end(); probe select_start() probe select_end(); probe update_start(); probe update_end();
      • compile with “--enable-dtrace” configure option to use the Dtrace probes
  • 47. Integrate with Chime Tool
      • Chime is graphical tool for visualizing DTrace aggregations. It
      • provides alternative CLI-based tool output with more visually
      • apprealing and more useful to display data over time
      • Available to download: http://www.opensolaris.org/os/project/dtrace-chime . One-step installation:
      • > Run pkgadd -d osol0chime-<arch>-1.4.pkg
      • chime requires Solaris Nevada build 35 or later
      • > Run local: /opt/OSOL0chime/bin/chime
      • > Run remotely: /opt/OSOL0chime/bin/chime <hostname>
      • chime provides wizard to automatically generate new display for DTrace script
  • 48.  
  • 49.
    • Resources
      • > http:// www.opensolaris.org /os/community/dtrace
      • - OpenSolaris Community: Dtrace
      • > http://developers.sun.com/solaris/articles/solaris_perftools.html
      • - Solaris Performance and Tools
      • > http://docs.sun.com/app/docs/doc/817-6223/6mlkidlms?a=view
      • - Statically Defined Tracing for User Applications chapter of DTrace
      • Manual
      • > http://www.brendangregg.com
      • - DTrace toolkits
      • > http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Databases
      • - DTrace Topics Databases
      • Acknowledgements
      • Brandan Gregg – Sun Microsystems Engineer in Advanced Products Group