Increasing MySQL Productivity

Title Increasing MySQL Productivity From design to implementation 2010.06 Ronald Bradford http://ronaldbradford.com Design to Implementation - 2010.06 Improving MySQL Productivity from

Objectives MySQL development tips and tricks Identify poor development practices Recommended best practices Improve knowledge and MySQL skills Optimal deployment & support Productivity ROI • Increase your productivity • Improve development quality Improving MySQL Productivity from Design to Implementation - 2010.06

Session Scope Design Implementation Development Support Security R&D Testing Instrumentation Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review Data Integrity, Storage Engines, Diagnostics, SQL Syntax, Replication, Optimizing SQL, Meta Data, Query Analyzer, Workbench, Data Import/Export, Caching with Memcached, MySQL joins, Database Administration http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review MySQL Idiosyncrasies that BITE What are MySQL Defaults The impact of using defaults Ensuring better compatibility http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review Storage Engines in Review What is the pluggable storage architecture Built In Engines Plug in Engines Engine Characteristics How to create a storage engine http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review InnoDB Usage and Diagnostics MySQL transactional storage engine Clustered and Secondary Indexes Understanding innoDB threads Performance metrics http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review What do you mean "SQL Syntax Error" The ANSI standard Similarities that produce differences What's missing http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review Everything you didn't know about MySQL Replication Replication Basics Asynchronous Topology options Replication usages http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review Importing and Exporting Data with MySQL Loading Data (csv, tsv, ﬁxed, report) Performance beneﬁts Dumping Data http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review Capturing, Analyzing and Optimizing your SQL Various capture methods Bulk and per query analysis Optimization techniques and examples http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review Query Analysis with MySQL Enterprise Monitor GUI SQL Analysis Monitoring Alerts and Advisors http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review Using MySQL Meta Data Effectively the INFORMATION_SCHEMA http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review Navigating MySQL Stored Procedures & Funtions, Views and Triggers The MySQL Syntax Limitations http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review Data Caching with Memcached and MySQL Leveraging Caching Write thru/Write back UDF support http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review Design and Development using MySQL Workbench GUI Schema Development Re-engineering schemas Query browser integration http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review JoinFu - The Art of SQL Join Basics And/Or conditions Hierarchal data Aggregation and ranking http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review The Five Minute DBA Quick DBA tips Conﬁguration Monitoring http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06

Design Improving MySQL Productivity from Design to Implementation - 2010.06

Design Best Practices Data Types Schema Management Sharding 101 Higher Availability Improving MySQL Productivity from Design to Implementation - 2010.06

About MySQL Data Types Numeric Data Types Oracle has 1 MySQL has 9 TINYINT,SMALLINT,MEDIUMINT,INT, BIGINT,FLOAT,DOUBLE,DECIMAL,BIT Improving MySQL Productivity from Design to Implementation - 2010.06

Optimal Auto Increment Primary Key Don't use BIGINT AUTO_INCREMENT Use INT UNSIGNED AUTO_INCREMENT BIGINT is 8 bytes INT is 4 Bytes INT UNSIGNED stores 4.3 billion values Productivity ROI • Can reduce index space by 50+% • Better memory usage, less I/O Improving MySQL Productivity from Design to Implementation - 2010.06

Horror Stories INT(1) is not what it looks like INT(1) This is not 1 byte, it's 4 bytes (1) is only for client display only Client with 10+ ﬂags using INT(1) 40 bytes reduced to 10 bytes 2 bytes using bit operators Improving MySQL Productivity from Design to Implementation - 2010.06

Dates MySQL supports DATE, DATETIME, TIMESTAMP, YEAR TIMESTAMP for Epoch values TIMESTAMP is 4 bytes DATETIME is 8 bytes Supports DEFAULT CURRENT_TIMESTAMP Neither store milliseconds Improving MySQL Productivity from Design to Implementation - 2010.06

The beauty of ENUM Values Check Constraint Ideal for static codes Compact - i.e. 1 byte for 'True', 'False' Human readable values 5.1 Non blocking ALTER Improving MySQL Productivity from Design to Implementation - 2010.06

TRUE/FALSE Datatype Examples CREATE TABLE enums ( flag1 CHAR(1) NOT NULL COMMENT 'T or F, Y or N', flag2 TINYINT NOT NULL COMMENT '0 or 1', flag3 BIT NOT NULL COMMENT 'True or False', flag4 ENUM ('True','False') NOT NULL ); INSERT INTO enums(flag4) VALUES ('True', 'False'); SELECT flag4 FROM enums; Improving MySQL Productivity from Design to Implementation - 2010.06

InnoDB Primary Key Primary key is clustered index 15/16 ﬁll factor for sequential 50% for random Secondary Indexes hold primary key value Best Practice - Keep primary keys short except for high Disk I/O Improving MySQL Productivity from Design to Implementation - 2010.06

Schema Best Practices Have pristine schema Use different schemas Backup copies of tables Temporary objects Improving MySQL Productivity from Design to Implementation - 2010.06

Schema Management Always have current schema.sql Use Patch/Revert SQL for upgrades See http://schemasync.org Productivity ROI • Reproducibility • Upgrade/Downgrade path Improving MySQL Productivity from Design to Implementation - 2010.06

Best Practices Separate Schema changes and application code Productivity ROI • Developers develop • Administrators administer Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Design Sharding 101 Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 A database table consists of Columns Rows Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Columns col1 col2 col3 col4 col5 col6 col7 Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Rows col1 col2 col3 col4 col5 col6 col7 row1 row2 row3 row4 row5 row6 row7 row8 row9 row10 row11 row12 row13 row14 Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 A database consists of Tables Table Rows Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Tables table1 table2 table3 table4 table5 table6 table7 Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Table Rows table1 table2 table3 table4 table5 table6 table7 row1 row2 row3 row4 row5 row6 row7 row8 row9 row10 row11 row12 row13 row14 Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Partitioning is? Grouping like rows in a table together e.g. By Date By Local By parent grouping Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Partition col1 col2 col3 col4 col5 col6 col7 row1 row2 row3 row4 row5 row6 row7 row8 row9 row10 row11 row12 row13 row14 Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Sharding is? Grouping like tables and/or like rows together Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Tables table1 table2 table3 table4 table5 table6 table7 Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Tables + Rows table1 table2 table3 table4 table5 table6 table7 1 1 Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Shards table1 table2 table3 table4 table5 table6 table7 1 1 2 2 3 3 Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Shards table1 table2 table3 table4 table5 table6 table7 1 1 4 4 4 2 2 3 3 Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Shards table1 table2 table3 table4 table5 table6 table7 5 1 1 6 4 4 2 2 7 8 3 3 9 Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 How do you group tables? That’s the secret Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 - First Steps First Steps Consider schema separation Deﬁne a good partition key Balanced solution Support rebalancing Productivity ROI • Write Scalability • Done before application grows in complexity and usage Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 - Next Steps Next Steps Managing Joins Searching Data Unique Constraints Aggregated Reporting Productivity ROI • Write Scalability • Done before application grows in complexity and usage Improving MySQL Productivity from Design to Implementation - 2010.06

Manual Partitioning Ideal for Write Once Data MySQL supports Atomic table rename RENAME TABLE current TO old, new TO current; Improving MySQL Productivity from Design to Implementation - 2010.06

Data Partitioning Feature of MySQL 5.1 RANGE LIST HASH KEY http://dev.mysql.com/doc/refman/5.1/en/partitioning.html Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 Design Supporting Growth Improving MySQL Productivity from Design to Implementation - 2010.06

Supporting Growth Seek Professional Advice Availability Scalability Resilience Improving MySQL Productivity from Design to Implementation - 2010.06

Availability How do you support maintenance? Software upgrades Application releases 24x7 operations Eliminate single points of failure Productivity ROI • Less Downtime • Greater Sales / Better Reputation Improving MySQL Productivity from Design to Implementation - 2010.06

Scalability Read Scalability Replication Write Scalability Sharding Caching Implement from day 1 Use best product for purpose Productivity ROI • Support Demand Improving MySQL Productivity from Design to Implementation - 2010.06

Scalability Implementation Implementation H/W acquisition time Zero new human resource need Zero application changes Productivity ROI • Seamless growth support Improving MySQL Productivity from Design to Implementation - 2010.06

Resilience What is your backup strategy? What is your recovery strategy? How long does it take? Have you actually tested it end to end? Take the survey http://ronaldbradford.com/blog/checked-your-mysql-recovery-process-recently-2010-02-15/ Productivity ROI Improving MySQL Productivity from Design to Implementation - 2010.06 • Company viability

Design MySQL Conﬁguration Improving MySQL Productivity from Design to Implementation - 2010.06

Conﬁguration Best Practices Always use transactions SQL_MODE Data Integrity MySQL Idiosyncrasies that BITE Talk http://ronaldbradford.com/blog/mysql-idiosyncrasies-that-bite-2010-06-28/ Improving MySQL Productivity from Design to Implementation - 2010.06

Design Security Improving MySQL Productivity from Design to Implementation - 2010.06

MySQL User Security Default is woeful Minimum $ mysql_secure_installation Recommended Operating System Permissions & Privileges Improving MySQL Productivity from Design to Implementation - 2010.06

Operating System Security Defaults are not secure Never run as 'root' user Separate Data/Binary Logs/Logs/ Conﬁguration/Backups Individual directory permissions Productivity ROI • Minimize security risk • Better auditability Improving MySQL Productivity from Design to Implementation - 2010.06

MySQL Installation Best Practice Best Practice /mysql /etc/my.cnf /etc /etc/profile.d/mysql.sh /data /etc/init.d/mysqld /binlog /log /mysql-version Improving MySQL Productivity from Design to Implementation - 2010.06

MySQL Installation Security Software installed by root $ chown -R root:root /mysql $ chown -R mysql:mysql /mysql/{data,log,binlog/etc} $ chmod 700 /mysql/{data,binlog} $ chmod 750 /mysql/{etc,log} Improving MySQL Productivity from Design to Implementation - 2010.06

Application User Security Best Practice CREATE USER appuser@localhost IDENTIFIED BY 'sakila'; GRANT SELECT,INSERT,UPDATE,DELETE ON schema.* TO appuser@localhost; Normal Practice CREATE USER superman@'%'; GRANT ALL ON *.* TO superman@'%'; See MySQL Idiosyncrasies that BITE session for more information Improving MySQL Productivity from Design to Implementation - 2010.06

Application User Security Best Practices Application User (Read/Write Access) INSERT, UPDATE, DELETE, SELECT Application Viewer (Read Only Access) SELECT Application DBA (Schema Access Only) CREATE, DROP, CREATE ROUTINE, SELECT, INSERT, UPDATE, DELETE Productivity ROI • Track Data Security • Separation of responsibilities Improving MySQL Productivity from Design to Implementation - 2010.06

Why not use GRANT ALL GRANT ALL ON *.* TO user@’%’ *.* gives you access to all tables in all schemas @’%’ give you access from any external location ALL gives you ALTER, ALTER ROUTINE, CREATE, CREATE ROUTINE, CREATE TEMPORARY TABLES, CREATE USER, CREATE VIEW, DELETE, DROP, EVENT, EXECUTE, FILE, INDEX, INSERT, LOCK TABLES, PROCESS, REFERENCES, RELOAD, REPLICATION CLIENT, REPLICATION SLAVE, SELECT, SHOW DATABASES, SHOW VIEW, SHUTDOWN, SUPER, TRIGGER, UPDATE, USAGE See MySQL Idiosyncrasies that BITE session for more information Improving MySQL Productivity from Design to Implementation - 2010.06

Why SUPER is bad? SUPER Bypasses read_only Bypasses init_connect Can Disable binary logging Change conﬁguration dynamically No reserved connection Improving MySQL Productivity from Design to Implementation - 2010.06

Design Development Security Improving MySQL Productivity from Design to Implementation - 2010.06

Development Best Practices Version control General query log SQL Commenting SQL Formatting Future prooﬁng Embrace CLI Caching Improving MySQL Productivity from Design to Implementation - 2010.06

Version Control Use for all development Especially single user development Use for schema management Building block of automated build/release Productivity ROI • Central repository of code • Track code changes by developer • Create reproducable releases Improving MySQL Productivity from Design to Implementation - 2010.06

General Query Log Turn on for all development environments Logs all SQL Statements Aggregate and Email results to developer See Capturing, Analyzing and Optimizing SQL Presentation Productivity ROI • Developers are seeing SQL in operation • Enables access to SQL to analyze Improving MySQL Productivity from Design to Implementation - 2010.06

Leveraging General Query Log For single user development environment In SQL Session mysql> SELECT 'Function X Start'; In Application Run Function/Process In SQL Session mysql> SELECT 'Function X End'; Improving MySQL Productivity from Design to Implementation - 2010.06

Slow Query Log Turn on for all development environments Logs slow SQL Statements ( > 1 second) Aggregate and Email results to developer See Capturing, Analyzing and Optimizing SQL Presentation Improving MySQL Productivity from Design to Implementation - 2010.06

SQL Commenting Identify queries by code function Identify OLTP / Batch / Cache queries SELECT /* XXX123 */ col1, ... UPDATE /* YYY999 */ ... SELECT /* Batch */... Productivity ROI • DBA/SA can identify code function and purpose quickly Improving MySQL Productivity from Design to Implementation - 2010.06

SQL Commenting (2) 26 Query SELECT /* 5m cache */ ..... 26 Query SELECT /* ViewPost */ t.*, tt.*, tr.object_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON tt.term_id = t.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN ('category', 'post_tag') AND tr.object_id IN (2849, 2842, 2836, 2824, 2812, 2680, 2813, 2800, 2770, 2784) ORDER BY t.name ASC 26 Query SELECT /* batch */ meta_key, meta_value FROM wp_usermeta WHERE user_id = 2 Improving MySQL Productivity from Design to Implementation - 2010.06

SQL Formatting Create single line queries Don't embed newlines Enables per line analysis by CLI tools Productivity ROI • DBA/SA can use simple CLI tools including grep,awk,cut etc for SQL analysis Improving MySQL Productivity from Design to Implementation - 2010.06

SQL Formatting (2) 26 Query SELECT FOUND_ROWS() 26 Query SELECT post_id,start,end,allday,rpt,IF(end>='2010-06-04 00:00:00',1,0) AS active FROM wp_ec3_schedule WHERE post_id IN (2849,2842,2836,2824,2812,2680,2770,2784) ORDER BY start 26 Query SELECT * FROM wp_users WHERE user_login = 'ronald' 26 Query SELECT t.*, tt.*, tr.object_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON tt.term_id = t.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN ('category', 'post_tag') AND tr.object_id IN (2849, 2842, 2836, 2824, 2812, 2680, 2813, 2800, 2770, 2784) ORDER BY t.name ASC 26 Query SELECT meta_key, meta_value FROM wp_usermeta WHERE user_id = 2 Improving MySQL Productivity from Design to Implementation - 2010.06

SQL Analysis Bulk trending analysis Volume of SQL statements Query Execution Plan (QEP) Online v Batch/Cache SQL via commenting Productivity ROI • Identify bottlenecks ASAP without load • Iterative Design feedback Improving MySQL Productivity from Design to Implementation - 2010.06

EXPLAIN Basics EXPLAIN SELECT ... No key used key length Extra - Using Filesort, Using Temporary Improving MySQL Productivity from Design to Implementation - 2010.06

Future prooﬁng your SQL Specify INSERT columns INSERT INTO table(a,b,c) VALUES(...) SELECT * is generally bad What columns are actually used in code TEXT/BLOB can cause extra disk I/O New columns can change performance Productivity ROI • Reduce likelihood of runtime errors when structural changes to objects Improving MySQL Productivity from Design to Implementation - 2010.06

Avoid Fancy Constructs DELAYED IGNORE LOW PRIORITY REPLACE Productivity ROI • Changes ACID and statement precedence • May have additional performance overhead Improving MySQL Productivity from Design to Implementation - 2010.06

Using Deterministic Functions Use '2010-06-21' instead of CURDATE() Same for NOW() Leverage Query Cache (if enabled) Productivity ROI • Leverages database caching when enabled • Allows testing via parameterization Improving MySQL Productivity from Design to Implementation - 2010.06

Common Coding Errors Remove Duplicate Code Productivity ROI • Less code to maintain • Remove chance of human errors Improving MySQL Productivity from Design to Implementation - 2010.06

Common SQL Errors Remove redundant SQL Use general query log You may be surprised! Improving MySQL Productivity from Design to Implementation - 2010.06

Common Coding Errors - Repeating Queries The WRONG way 5 Query SELECT * FROM àrtist` 5 Query SELECT * FROM àrtist` 5 Query SELECT * FROM àrtist` 5 Query SELECT * FROM àrtist` 5 Query SELECT * FROM àrtist` 5 Query SELECT * FROM àrtist` WHERE (ArtistID = 196 ) 5 Query SELECT * FROM àrtist` WHERE (ArtistID = 2188 ) 5 Query SELECT * FROM àrtist` 5 Query SELECT * FROM àrtist` 5 Query SELECT * FROM àrtist` Improving MySQL Productivity from Design to Implementation - 2010.06

Common Coding Errors - Row at a time (RAT) Processing SELECT option_name, option_value FROM wp_options WHERE autoload = 'yes' SELECT option_value FROM wp_options WHERE option_name = 'aiosp_title_format' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_show_only_even' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_num_months' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_day_length' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_hide_event_box' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_advanced' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_navigation' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_disable_popups' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'sidebars_widgets' LIMIT 1 Improving MySQL Productivity from Design to Implementation - 2010.06

RAT v CAT Processing Row (RAT) Processing SELECT * FROM activities_theme WHERE theme_parent_id=0 SELECT * FROM activities_theme WHERE theme_parent_id=1 SELECT * FROM activities_theme WHERE theme_parent_id=2 SELECT * FROM activities_theme WHERE theme_parent_id=11 SELECT * FROM activities_theme WHERE theme_parent_id=16 Chunk (CAT) Processing SELECT * FROM activities_theme WHERE theme_parent_id in (0,1,2,11,16) Improving MySQL Productivity from Design to Implementation - 2010.06

Common Coding Errors - Boundary Conditions The following SQL executed 6,000 times in 5 minute analysis period The WRONG way SELECT pages_id, pages_livestats_code, pages_title, pages_parent, pages_exhibid, pages_theme, pages_accession_num FROM pages WHERE pages_id = 0 0 is an invalid pages_id Improving MySQL Productivity from Design to Implementation - 2010.06

In a highly tuned system the greatest time in a query is network overhead Improving MySQL Productivity from Design to Implementation - 2010.06

Development About Database Connections Improving MySQL Productivity from Design to Implementation - 2010.06

Deﬁning your database connection The WRONG way $ cd public_html $ grep "mysql*_connect" * */* */*/* db-disp.php:$cid = mysql_connect("localhost", "museum", "******") or die ('I cannot connect to the database because: ' . mysql_error()); test.php:$cid = mysql_connect("localhost", "museum", "******"); PMCollection/connection.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass"); PMCollection/connection_live.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass"); PMCollection/connection_local.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass"); PMEcards/connection.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass"); core/connection.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass"); discussion_admin/db_fns.php: $cid = mysql_connect("localhost", "museum", "******"); discussion_admin/header.php:// $cid = mysql_connect("localhost", "museum", "******"); discussion_admin/inc_title.php: //$cid = mysql_connect("localhost", "museum", "******"); discussion_admin/stats.php: //$cid = mysql_connect("localhost", "museum", Improving MySQL Productivity from Design to Implementation - 2010.06

Database connection example The RIGHT way class database { function getConnection($type, $message) { $cp = $this->getConnectionParameter($type); if (empty($cp)) { $message = new message ("fatal", "Unable to determine '$type' ... return; } try { $con = mysqli_connect($cp->host,$cp->user,$cp->passwd,$cp->database); if (!$con) { $message = new message ("fatal", "Unable to obtain a '$type' ... return; } mysqli_query($con, "SET NAMES 'utf8'"); } catch (Exception $e) { $message = new message ("fatal", "Unable to obtain a '$type' ... debug($e->getMessage()); } return $con; } Improving MySQL Productivity from Design to Implementation - 2010.06

Database connection parameters example $ cat config/database.inc.sample <? require_once "classes/core/connectionparameter.inc"; $connections = array(); // New connections "description", "host", "user", "passwd", "database", "settings" $connections["write"] = new connectionparameter( "Write DB Connection", "localhost", "writeuser", "******", "db"); $connections["read"] = new connectionparameter( "Read DB Connection", "localhost", "readuser", "******", "db"); ?> Improving MySQL Productivity from Design to Implementation - 2010.06

Database parameterization and version control Parameters in separate ﬁle conﬁg/database.inc.sample NOTE: .sample in Version Control Does not override deployment Productivity ROI • Automated code deployment to multiple environments Improving MySQL Productivity from Design to Implementation - 2010.06

Horror Stories - Connection Parameters How not to code connection parameters Changing Connector/J Settings (1) Have to reboot server i.e. no way to ﬂush connection pool Improving MySQL Productivity from Design to Implementation - 2010.06

Horror Stories - Connection Parameters How not to code connection parameters Changing Connector/J Settings (2) Rebuild and redeploy jar Improving MySQL Productivity from Design to Implementation - 2010.06

Database Connections Open and close database connections only when necessary Productivity ROI • Reduce unnecessary database load • Increases page serve volume • Increases true DB throughput Improving MySQL Productivity from Design to Implementation - 2010.06

Database Connections Initialization The WRONG way $ cat header.php ... $con = getConnection(); ... if($this_user->user_row["status"]!='ws' && in_array($this_page->getValue(),$page)){ header("Location: /wholesale/permission.php"); exit(); } ... if () { header("Location: abc.php"); exit(); } ... if () { header("Location: xyz.php"); exit(); } ... Improving MySQL Productivity from Design to Implementation - 2010.06

Database Connections Deconstruction The WRONG way $ cat footer.php ... $con->close(); Improving MySQL Productivity from Design to Implementation - 2010.06

Database Connections - Optimal Usage The RIGHT way // do all possible pre-processing first ... // Then get DB connection $con = database->getConnection("read"); // Close your connection ASAP // Always future proof objects method calls if ($con != NULL) { $con->close(); } Improving MySQL Productivity from Design to Implementation - 2010.06

Development Instrumentation Improving MySQL Productivity from Design to Implementation - 2010.06

Application Instrumentation Creating one primary abstract DB call Enable logging of ALL SQL statements Enable embedded HTML output Total Execution Time/Count Individual SQL Execution Time/SQL Productivity ROI • Enable runtime analysis via browser • No additional tools needed to gather Improving MySQL Productivity from Design to Implementation - 2010.06

Application Instrumentation Beneﬁts End to End Timing Component Timing (i.e. a series of SQL) Observe as desired Intelligent Activation e.g. Page Load exceeds x ms Improving MySQL Productivity from Design to Implementation - 2010.06

Development Quick Tips Improving MySQL Productivity from Design to Implementation - 2010.06

Using Text Compression Via Database COMPRESS() UNCOMPRESS() Via Application Spread CPU Load to many servers Database Structure Change TEXT --> BLOB Productivity ROI • Shorter Backup/Recovery times • Greater volume of data for disk usage Improving MySQL Productivity from Design to Implementation - 2010.06

Using Text Compression - Example CREATE TABLE `Mkt` ( `MktID` varchar(10) NOT NULL, `Description` text, PRIMARY KEY (`MktID`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1 select min(length(Description)) as min_len, avg(length(Description)) as avg_len, max(length(Description)) as max_len from Mkt; +---------+---------+---------+ | min_len | avg_len | max_len | +---------+---------+---------+ | 20 |1122.920 | 7999 | +---------+---------+---------+ Improving MySQL Productivity from Design to Implementation - 2010.06

Using Compression Example (2) Best 74% saving SELECT length(Description) as len_before, length(compress(Description)) as len_after FROM Mkt WHERE MktID = '42'; +------------+-----------+ | len_before | len_after | +------------+-----------+ | 7810 | 2025 | +------------+-----------+ ...WHERE MktID LIKE '6_'; +------------+-----------+ | len_before | len_after | No beneﬁt for short +------------+-----------+ data | 11 | 20 | | 53 | 53 | | 113 | 100 | Improving MySQL Productivity from Design to Implementation - 2010.06

Using Text Compression (2) Prime Candidates Large Text Fields Repeating data (e.g. xml elements) Not Indexed Not Searched Improving MySQL Productivity from Design to Implementation - 2010.06

Caching 101 Use caching whenever possible Caching options Query results Objects HTML component content Improving MySQL Productivity from Design to Implementation - 2010.06

Design Development Security Testing Improving MySQL Productivity from Design to Implementation - 2010.06

Testing is not about what works; testing is about breaking your software Improving MySQL Productivity from Design to Implementation - 2010.06

Writing testable code Small code functions Use Refactoring Improving MySQL Productivity from Design to Implementation - 2010.06

Query Analysis Database Size / data volume Version Query QEP Result set size Rows affected/retrieved Table structure Improving MySQL Productivity from Design to Implementation - 2010.06

Function Testability MVC API Improving MySQL Productivity from Design to Implementation - 2010.06

Design Development Security Testing Instrumentation Improving MySQL Productivity from Design to Implementation - 2010.06

If you don't have monitoring in place, make it your top priority Improving MySQL Productivity from Design to Implementation - 2010.06

Instrumentation Objectives System Dashboard Monitoring Alerting Public System Status Improving MySQL Productivity from Design to Implementation - 2010.06

System Dashboard What is the state of NOW! sampling 1,5,10 seconds One page listing most important numbers Red/Yellow/Green Private/Public View Productivity ROI • One starting point for all staff. No manual "let me see" needed. Improving MySQL Productivity from Design to Implementation - 2010.06

Title Screen print Dashboard Example Improving MySQL Productivity from Design to Implementation - 2010.06

Monitoring Records history over time Sampling (e.g. 5 minutes / 1 minute) Graphical Analysis Improving MySQL Productivity from Design to Implementation - 2010.06

Alerting Identify key limits Notiﬁcation rules 24x7 Improving MySQL Productivity from Design to Implementation - 2010.06

Public Status Keep your users informed Planned outage Unplanned outage http://status.twitter.com/ http://status.blogs.smugmug.com/ Host elsewhere Use a Blog Allows for user feedback Improving MySQL Productivity from Design to Implementation - 2010.06

The Rule of Everything Everything Monitor Measure Graph Automate Improving MySQL Productivity from Design to Implementation - 2010.06

Design Development Security Testing Implementation Instrumentation Improving MySQL Productivity from Design to Implementation - 2010.06

Implementation Objectives Instrumentation Implementation No Downtime! Pre-emptive Load Management Higher Availability Automated Deployment Improving MySQL Productivity from Design to Implementation - 2010.06

Development Team: We need a maintenance window for software upgrades and new releases. Improving MySQL Productivity from Design to Implementation - 2010.06

Management Team: No Downtime Improving MySQL Productivity from Design to Implementation - 2010.06

Development Team: But we need this to ﬁx problems and improve performance. Improving MySQL Productivity from Design to Implementation - 2010.06

Management Team: No Downtime Improving MySQL Productivity from Design to Implementation - 2010.06

What is your deﬁnition of no downtime? Improving MySQL Productivity from Design to Implementation - 2010.06

What is your requirements for no downtime? For example: Serve Pages Serve Ads NOTE: No need to add/change content Search data Charge for ad impressions Sell Merchandise Improving MySQL Productivity from Design to Implementation - 2010.06

MySQL Higher Availability - First Step MySQL pair Master / Fail Over Master Only as good as the weakest link Slave SQL_THREAD Slave Disk I/O Productivity ROI • Actively tests primary DR strategy Improving MySQL Productivity from Design to Implementation - 2010.06

Pre-emptive Load Management Improving MySQL Productivity from Design to Implementation - 2010.06

Pre-emptive Load Management (2) What is the Twitter "Failed Whale" Based on page Load time/Volume Reject early http://en.oreilly.com/velocity2009/public/schedule/detail/7479 http://www.slideshare.net/netik/billions-of-hits-scaling-twitter Improving MySQL Productivity from Design to Implementation - 2010.06

Pre-emptive Load Management (3) Combined with "Failed Whale" Darkmode Disable intensive tasks (e.g. Name Search) Improving MySQL Productivity from Design to Implementation - 2010.06

Pre-emptive Load Management (4) Data Messaging Queue Per table/chunk Write Access Read Access No Access Proactively disable/limit application access Improving MySQL Productivity from Design to Implementation - 2010.06

Data Messaging Queue Example Disable Write access to user data Application changes via notification Disable option/Notification message New user registration Edit profile Read Access to login/lookup etc Improving MySQL Productivity from Design to Implementation - 2010.06

Su Design pp or Development t Security Testing Implementation Instrumentation Improving MySQL Productivity from Design to Implementation - 2010.06

Number 1 problem! Don't change a setting without evidence to prove/ disprove the Improving MySQL Productivity from Design to Implementation - 2010.06

Where is the bottleneck? Is the database the problem? Front End Performance Improving MySQL Productivity from Design to Implementation - 2010.06

Perception The website will always be too slow Identify the true components End to End time Database may be only a small portion Improving MySQL Productivity from Design to Implementation - 2010.06

Front End Tips • Know your total website load time http://getfirebug.com/ • How much time is actually database related? • Reduce HTML page size - 15% improvement • Remove full URL’s, inline css styles • Reduce/combine css & js files Improving MySQL Productivity from Design to Implementation - 2010.06

Front End Tips • Split static content to different ServerName • Spread static content over multiple ServerNames (e.g. 3) • Sprites - Combining lightweight images - http:// spriteme.org/ • Cookie-less domain name for static content Improving MySQL Productivity from Design to Implementation - 2010.06

Continual Improvement Lower MTTR (mean time to recovery) Lower MTTD (Mean time to detect) Improving MySQL Productivity from Design to Implementation - 2010.06

Conclusion Improving MySQL Productivity from Design to Implementation - 2010.06

Availability Su g Datatypes Scalability pp n Design di ar Sh Resilience Schema Management or Development t Database Connections Version control Caching SQL Standards Break Security Testing your User Management system NO DOWNTIME Implementation User Experience Automation Instrumentation Dashboard - Monitoring - Alerting Improving MySQL Productivity from Design to Implementation - 2010.06

RonaldBradford.com MySQL4OracleDBA.com Improving MySQL Productivity from Design to Implementation - 2010.06

http://ronaldbradford.com	288
http://translate.googleusercontent.com	3
http://static.slidesharecdn.com	1

Increasing MySQL Productivity

by Ronald Bradford on Jul 01, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 292

Statistics

Increasing MySQL Productivity — Presentation Transcript