Title




    Increasing MySQL
       Productivity
 From design to implementation
                                        ...
Objectives




     MySQL development tips and tricks
     Identify poor development practices
     Recommended best pract...
Session Scope




     Design                                             Implementation
     Development                 ...
ODTUG Presentations Review




     Data Integrity, Storage Engines, Diagnostics,
     SQL Syntax, Replication, Optimizing...
ODTUG Presentations Review




MySQL Idiosyncrasies that BITE
     What are MySQL Defaults
     The impact of using defaul...
ODTUG Presentations Review




Storage Engines in Review
   What is the pluggable storage architecture
   Built In Engines...
ODTUG Presentations Review




InnoDB Usage and Diagnostics
     MySQL transactional storage engine
     Clustered and Sec...
ODTUG Presentations Review




What do you mean "SQL Syntax Error"
     The ANSI standard
     Similarities that produce d...
ODTUG Presentations Review




Everything you didn't know about MySQL Replication
     Replication Basics
          Asynch...
ODTUG Presentations Review




Importing and Exporting Data with MySQL
     Loading Data (csv, tsv, fixed, report)
        ...
ODTUG Presentations Review




Capturing, Analyzing and Optimizing your SQL
     Various capture methods
     Bulk and per...
ODTUG Presentations Review




Query Analysis with MySQL Enterprise Monitor
     GUI SQL Analysis
     Monitoring
     Ale...
ODTUG Presentations Review




Using MySQL Meta Data Effectively
     the INFORMATION_SCHEMA




             http://www.o...
ODTUG Presentations Review




Navigating MySQL Stored Procedures &
Funtions, Views and Triggers
     The MySQL Syntax
   ...
ODTUG Presentations Review




Data Caching with Memcached and MySQL
     Leveraging Caching
     Write thru/Write back
  ...
ODTUG Presentations Review




Design and Development using MySQL Workbench
     GUI Schema Development
     Re-engineerin...
ODTUG Presentations Review




JoinFu - The Art of SQL
     Join Basics
     And/Or conditions
     Hierarchal data
     A...
ODTUG Presentations Review




The Five Minute DBA
     Quick DBA tips
     Configuration
     Monitoring



             h...
Design




Improving MySQL Productivity from Design to Implementation - 2010.06
Design Best Practices




     Data Types
     Schema Management
     Sharding 101
     Higher Availability




Improving ...
About MySQL Data Types




     Numeric Data Types
          Oracle has 1
          MySQL has 9


          TINYINT,SMALLI...
Optimal Auto Increment Primary Key




     Don't use BIGINT AUTO_INCREMENT
     Use INT UNSIGNED AUTO_INCREMENT
     BIGI...
Horror Stories


                                                                       INT(1) is not what it
            ...
Dates




     MySQL supports
          DATE, DATETIME, TIMESTAMP, YEAR
     TIMESTAMP for Epoch values
          TIMESTAM...
The beauty of ENUM




     Values Check Constraint
     Ideal for static codes
     Compact - i.e. 1 byte for 'True', 'Fa...
TRUE/FALSE Datatype Examples




CREATE TABLE enums (
   flag1 CHAR(1) NOT NULL COMMENT 'T or F, Y or N',
   flag2 TINYINT...
InnoDB Primary Key




     Primary key is clustered index
       15/16 fill factor for sequential
       50% for random
  ...
Schema Best Practices




     Have pristine schema
          Use different schemas
               Backup copies of tables...
Schema Management




     Always have current schema.sql
     Use Patch/Revert SQL for upgrades
     See http://schemasyn...
Best Practices




     Separate Schema changes and application
     code




                                            ...
Sharding 101




Design

                 Sharding 101


Improving MySQL Productivity from Design to Implementation - 2010...
Sharding 101




     A database table consists of
         Columns
         Rows




Improving MySQL Productivity from De...
Sharding 101

                                                                Columns

               col1          col2  ...
Sharding 101

                                                           Columns

               col1          col2       ...
Sharding 101

                                                           Columns

               col1          col2       ...
Sharding 101

                                                           Rows

               col1          col2          ...
Sharding 101

                                                           Rows

               col1          col2          ...
Sharding 101

                                                           Rows

               col1          col2          ...
Sharding 101




     A database consists of
         Tables
         Table Rows




Improving MySQL Productivity from Des...
Sharding 101

                                                           Tables

              table1        table2       ...
Sharding 101

                                                           Tables

              table1        table2       ...
Sharding 101

                                                           Tables

              table1        table2       ...
Sharding 101

                                                           Table Rows

              table1        table2   ...
Sharding 101

                                                           Table Rows

              table1        table2   ...
Sharding 101

                                                           Table Rows

              table1        table2   ...
Sharding 101




     Partitioning is?
     Grouping like rows in a table together
     e.g.
               By Date
      ...
Sharding 101

                                                           Partition

               col1          col2     ...
Sharding 101

                                                           Partition

               col1          col2     ...
Sharding 101

                                                                Partition

               col1          col2...
Sharding 101




     Sharding is?
             Grouping like tables and/or like rows
             together




Improving ...
Sharding 101

                                                           Tables

              table1        table2       ...
Sharding 101

                                                           Tables

              table1        table2       ...
Sharding 101

                                                           Tables

              table1        table2       ...
Sharding 101

                                                           Tables + Rows

              table1        table2...
Sharding 101

                                                           Tables + Rows

              table1        table2...
Sharding 101

                                                           Tables + Rows

              table1        table2...
Sharding 101

                                                           Shards

              table1        table2       ...
Sharding 101

                                                           Shards

              table1        table2       ...
Sharding 101

                                                           Shards

              table1        table2       ...
Sharding 101




     How do you group tables?




                   That’s the secret

Improving MySQL Productivity from...
Sharding 101 - First Steps




First Steps
     Consider schema separation
     Define a good partition key
          Balan...
Sharding 101 - Next Steps




Next Steps
     Managing Joins
     Searching Data
     Unique Constraints
     Aggregated R...
Manual Partitioning




     Ideal for Write Once Data
     MySQL supports Atomic table rename


     RENAME TABLE current...
Data Partitioning




     Feature of MySQL 5.1
          RANGE
          LIST
          HASH
          KEY

          htt...
Sharding 101




Design

        Supporting Growth



Improving MySQL Productivity from Design to Implementation - 2010.06
Supporting Growth


                                                                       Seek Professional
             ...
Availability




     How do you support maintenance?
          Software upgrades
          Application releases
     24x7...
Scalability




     Read Scalability
       Replication
     Write Scalability
       Sharding
     Caching
       Implem...
Scalability Implementation




     Implementation
          H/W acquisition time
          Zero new human resource need
 ...
Resilience




     What is your backup strategy?
     What is your recovery strategy?
     How long does it take?
     Ha...
Design

      MySQL Configuration


Improving MySQL Productivity from Design to Implementation - 2010.06
Configuration Best Practices




     Always use transactions
     SQL_MODE
          Data Integrity


          MySQL Idio...
Design

                                                              Security


Improving MySQL Productivity from Design ...
MySQL User Security




     Default is woeful
     Minimum
          $ mysql_secure_installation
     Recommended
       ...
Operating System Security




     Defaults are not secure
     Never run as 'root' user
     Separate Data/Binary Logs/Lo...
MySQL Installation Best Practice




     Best Practice


/mysql                                                     /etc/...
MySQL Installation Security




     Software installed by root
$   chown      -R root:root /mysql
$   chown      -R mysql...
Application User Security




Best Practice
CREATE USER appuser@localhost IDENTIFIED BY 'sakila';
GRANT SELECT,INSERT,UPDA...
Application User Security Best Practices




     Application User (Read/Write Access)
       INSERT, UPDATE, DELETE, SELE...
Why not use GRANT ALL




     GRANT ALL ON *.* TO user@’%’
          *.* gives you access to all tables in all schemas
  ...
Why SUPER is bad?




     SUPER
          Bypasses read_only
          Bypasses init_connect
          Can Disable binary...
Design
  Development
         Security


Improving MySQL Productivity from Design to Implementation - 2010.06
Development Best Practices




     Version control
     General query log
     SQL Commenting
     SQL Formatting
     Fu...
Version Control




     Use for all development
          Especially single user development
     Use for schema manageme...
General Query Log




     Turn on for all development environments
     Logs all SQL Statements
     Aggregate and Email ...
Leveraging General Query Log




     For single user development environment
          In SQL Session
     mysql> SELECT ...
Slow Query Log




     Turn on for all development environments
     Logs slow SQL Statements ( > 1 second)
     Aggregat...
SQL Commenting




     Identify queries by code function
     Identify OLTP / Batch / Cache queries


     SELECT /* XXX1...
SQL Commenting (2)




   26 Query SELECT /* 5m cache */ .....
   26 Query SELECT /* ViewPost */ t.*, tt.*, tr.object_id F...
SQL Formatting




     Create single line queries
          Don't embed newlines
     Enables per line analysis by CLI to...
SQL Formatting (2)




   26 Query SELECT FOUND_ROWS()
   26 Query SELECT post_id,start,end,allday,rpt,IF(end>='2010-06-04...
SQL Analysis




     Bulk trending analysis
          Volume of SQL statements
     Query Execution Plan (QEP)
          ...
EXPLAIN Basics




     EXPLAIN SELECT ...


     No key used
     key length
     Extra - Using Filesort, Using Temporary...
Future proofing your SQL




     Specify INSERT columns
          INSERT INTO table(a,b,c) VALUES(...)

     SELECT * is g...
Avoid Fancy Constructs




     DELAYED
     IGNORE
     LOW PRIORITY
     REPLACE


                                     ...
Using Deterministic Functions




     Use '2010-06-21' instead of CURDATE()
     Same for NOW()


     Leverage Query Cac...
Common Coding Errors




     Remove Duplicate Code




                                                                  ...
Common SQL Errors




     Remove redundant SQL
          Use general query log
          You may be surprised!




Improv...
Common Coding Errors - Repeating Queries


                                                                       The WRON...
Common Coding Errors - Row at a time (RAT) Processing




SELECT   option_name,    option_value FROM wp_options WHERE auto...
RAT v CAT Processing




Row (RAT) Processing
SELECT       *   FROM     activities_theme               WHERE         theme...
Common Coding Errors - Boundary Conditions




     The following SQL executed 6,000 times in
     5 minute analysis perio...
In a highly tuned system
  the greatest time in a
    query is network
        overhead

Improving MySQL Productivity from...
Development
                  About Database
                   Connections

Improving MySQL Productivity from Design to I...
Defining your database connection


                                                                       The WRONG way

 ...
Database connection example


                                                                       The RIGHT way

class ...
Database connection parameters example




$ cat config/database.inc.sample

<?
   require_once "classes/core/connectionpa...
Database parameterization and version control




     Parameters in separate file
          config/database.inc.sample


  ...
Horror Stories - Connection Parameters


                                                                         How not ...
Horror Stories - Connection Parameters


                                                                         How not ...
Database Connections




     Open and close database connections only
     when necessary




                           ...
Database Connections Initialization


                                                                       The WRONG way...
Database Connections Deconstruction


                                                                       The WRONG way...
Database Connections - Optimal Usage


                                                                       The RIGHT wa...
Development

                 Instrumentation


Improving MySQL Productivity from Design to Implementation - 2010.06
Application Instrumentation




     Creating one primary abstract DB call
     Enable logging of ALL SQL statements
     ...
Application Toolset




Performance Analysis – Step by Step
                                                              ...
Performance Analysis – Step by Step
                                                                          Cold Fusion
...
Application Instrumentation Benefits




     End to End Timing
     Component Timing
          (i.e. a series of SQL)
    ...
Development

                              Quick Tips


Improving MySQL Productivity from Design to Implementation - 2010....
Using Text Compression




     Via Database
          COMPRESS() UNCOMPRESS()
     Via Application
          Spread CPU L...
Using Text Compression - Example




CREATE TABLE `Mkt` (
  `MktID` varchar(10) NOT NULL,
  `Description` text,
  PRIMARY ...
Using Compression Example (2)


                                                                         Best 74% saving

...
Using Text Compression (2)




     Prime Candidates
          Large Text Fields
          Repeating data (e.g. xml elemen...
Caching 101




     Use caching whenever possible
     Caching options
          Query results
          Objects
        ...
Design
    Development
           Security
Testing




  Improving MySQL Productivity from Design to Implementation - 2010...
Testing is not about
     what works; testing is
     about breaking your
            software

Improving MySQL Productivi...
Writing testable code




     Small code functions
     Use Refactoring




Improving MySQL Productivity from Design to I...
Query Analysis




     Database
      Size / data volume
      Version
     Query
      QEP
      Result set size
      R...
Function Testability




     MVC
     API




Improving MySQL Productivity from Design to Implementation - 2010.06
Design
    Development
           Security
Testing




                   Instrumentation
  Improving MySQL Productivity f...
If you don't have
        monitoring in
        place, make it
      your top priority
Improving MySQL Productivity from D...
Instrumentation Objectives




     System Dashboard
     Monitoring
     Alerting
     Public System Status




Improving...
System Dashboard




     What is the state of NOW!
          sampling 1,5,10 seconds
     One page listing most important...
Title




     Screen print




                                                                       Dashboard
         ...
Monitoring




     Records history over time
     Sampling (e.g. 5 minutes / 1 minute)
     Graphical Analysis




Improv...
Alerting




     Identify key limits
     Notification rules
     24x7




Improving MySQL Productivity from Design to Imp...
Public Status




     Keep your users informed
          Planned outage
          Unplanned outage
                      ...
The Rule of Everything




     Everything
          Monitor
          Measure
          Graph
          Automate



Impro...
Design
    Development
           Security
Testing



    Implementation
    Instrumentation
  Improving MySQL Productivit...
Implementation Objectives




     Instrumentation Implementation
     No Downtime!
     Pre-emptive Load Management
     ...
Development Team:

     We need a maintenance
      window for software
       upgrades and new
           releases.
Impro...
Management Team:

                        No Downtime




Improving MySQL Productivity from Design to Implementation - 201...
Development Team:

       But we need this to fix
       problems and improve
           performance.

Improving MySQL Prod...
Management Team:

                        No Downtime




Improving MySQL Productivity from Design to Implementation - 201...
What is your
          definition of no
            downtime?

Improving MySQL Productivity from Design to Implementation -...
What is your requirements for no downtime?




     For example:
       Serve Pages
       Serve Ads

          NOTE:
    ...
MySQL Higher Availability - First Step




     MySQL pair
          Master / Fail Over Master


     Only as good as the ...
Pre-emptive Load Management




Improving MySQL Productivity from Design to Implementation - 2010.06
Pre-emptive Load Management (2)




     What is the Twitter "Failed Whale"
          Based on page Load time/Volume
     ...
Pre-emptive Load Management (3)




     Combined with "Failed Whale"
     Darkmode
          Disable intensive tasks (e.g...
Pre-emptive Load Management (4)




     Data Messaging Queue
     Per table/chunk
          Write Access
          Read A...
Data Messaging Queue Example




     Disable Write access to user data
     Application changes via notification
         ...
Su
  Design                                                                      pp
                 or
    Development t
...
Number 1 problem!




  Don't change a
 setting without
evidence to prove/
   disprove the
Improving MySQL Productivity fr...
Where is the bottleneck?




     Is the database the problem?
     Front End Performance




Improving MySQL Productivity...
Perception




     The website will always be too slow
     Identify the true components
     End to End time
          D...
Front End Tips




• Know your total website load time
     http://getfirebug.com/

     • How much time is actually datab...
Front End Tips




• Split static content to different ServerName
• Spread static content over multiple
     ServerNames (...
Continual Improvement




     Lower MTTR (mean time to recovery)
     Lower MTTD (Mean time to detect)




Improving MySQ...
Conclusion


Improving MySQL Productivity from Design to Implementation - 2010.06
Availability                                      Su
         g          Datatypes Scalability
                           ...
RonaldBradford.com

  MySQL4OracleDBA.com


Improving MySQL Productivity from Design to Implementation - 2010.06
Upcoming SlideShare
Loading in...5
×

Increasing MySQL Productivity

6,660

Published on

What sets apart the every day web developer from the knowledgeable and highly desired developer that can provide a better return on investment (ROI) for their business or organization?

This session will provide MySQL development tips, tricks and best practices that elevate your knowledge and use of MySQL from the perspective of a MySQL Expert. In this master class we will touch on many aspects that make an impact for a successful application and minimize additional work or re-work from commonly observed poor development practices. From architectural design, coding, development practices, testability, debugging, security, application instrumentation, optimal deployment and monitoring we will touch much of the software development lifecycle.

This class is all about improving your knowledge and maximizing the strengths of the MySQL product for increasing productivity and increasing ROI the MySQL way.

Published in: Technology
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,660
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
439
Comments
0
Likes
10
Embeds 0
No embeds

No notes for slide

Increasing MySQL Productivity

  1. 1. Title Increasing MySQL Productivity From design to implementation 2010.06 Ronald Bradford http://ronaldbradford.com Design to Implementation - 2010.06 Improving MySQL Productivity from
  2. 2. Objectives MySQL development tips and tricks Identify poor development practices Recommended best practices Improve knowledge and MySQL skills Optimal deployment & support Productivity ROI • Increase your productivity • Improve development quality Improving MySQL Productivity from Design to Implementation - 2010.06
  3. 3. Session Scope Design Implementation Development Support Security R&D Testing Instrumentation Improving MySQL Productivity from Design to Implementation - 2010.06
  4. 4. ODTUG Presentations Review Data Integrity, Storage Engines, Diagnostics, SQL Syntax, Replication, Optimizing SQL, Meta Data, Query Analyzer, Workbench, Data Import/Export, Caching with Memcached, MySQL joins, Database Administration http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  5. 5. ODTUG Presentations Review MySQL Idiosyncrasies that BITE What are MySQL Defaults The impact of using defaults Ensuring better compatibility http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  6. 6. ODTUG Presentations Review Storage Engines in Review What is the pluggable storage architecture Built In Engines Plug in Engines Engine Characteristics How to create a storage engine http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  7. 7. ODTUG Presentations Review InnoDB Usage and Diagnostics MySQL transactional storage engine Clustered and Secondary Indexes Understanding innoDB threads Performance metrics http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  8. 8. ODTUG Presentations Review What do you mean "SQL Syntax Error" The ANSI standard Similarities that produce differences What's missing http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  9. 9. ODTUG Presentations Review Everything you didn't know about MySQL Replication Replication Basics Asynchronous Topology options Replication usages http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  10. 10. ODTUG Presentations Review Importing and Exporting Data with MySQL Loading Data (csv, tsv, fixed, report) Performance benefits Dumping Data http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  11. 11. ODTUG Presentations Review Capturing, Analyzing and Optimizing your SQL Various capture methods Bulk and per query analysis Optimization techniques and examples http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  12. 12. ODTUG Presentations Review Query Analysis with MySQL Enterprise Monitor GUI SQL Analysis Monitoring Alerts and Advisors http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  13. 13. ODTUG Presentations Review Using MySQL Meta Data Effectively the INFORMATION_SCHEMA http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  14. 14. ODTUG Presentations Review Navigating MySQL Stored Procedures & Funtions, Views and Triggers The MySQL Syntax Limitations http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  15. 15. ODTUG Presentations Review Data Caching with Memcached and MySQL Leveraging Caching Write thru/Write back UDF support http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  16. 16. ODTUG Presentations Review Design and Development using MySQL Workbench GUI Schema Development Re-engineering schemas Query browser integration http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  17. 17. ODTUG Presentations Review JoinFu - The Art of SQL Join Basics And/Or conditions Hierarchal data Aggregation and ranking http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  18. 18. ODTUG Presentations Review The Five Minute DBA Quick DBA tips Configuration Monitoring http://www.odtugkaleidoscope.com/MySQL.html Improving MySQL Productivity from Design to Implementation - 2010.06
  19. 19. Design Improving MySQL Productivity from Design to Implementation - 2010.06
  20. 20. Design Best Practices Data Types Schema Management Sharding 101 Higher Availability Improving MySQL Productivity from Design to Implementation - 2010.06
  21. 21. About MySQL Data Types Numeric Data Types Oracle has 1 MySQL has 9 TINYINT,SMALLINT,MEDIUMINT,INT, BIGINT,FLOAT,DOUBLE,DECIMAL,BIT Improving MySQL Productivity from Design to Implementation - 2010.06
  22. 22. Optimal Auto Increment Primary Key Don't use BIGINT AUTO_INCREMENT Use INT UNSIGNED AUTO_INCREMENT BIGINT is 8 bytes INT is 4 Bytes INT UNSIGNED stores 4.3 billion values Productivity ROI • Can reduce index space by 50+% • Better memory usage, less I/O Improving MySQL Productivity from Design to Implementation - 2010.06
  23. 23. Horror Stories INT(1) is not what it looks like INT(1) This is not 1 byte, it's 4 bytes (1) is only for client display only Client with 10+ flags using INT(1) 40 bytes reduced to 10 bytes 2 bytes using bit operators Improving MySQL Productivity from Design to Implementation - 2010.06
  24. 24. Dates MySQL supports DATE, DATETIME, TIMESTAMP, YEAR TIMESTAMP for Epoch values TIMESTAMP is 4 bytes DATETIME is 8 bytes Supports DEFAULT CURRENT_TIMESTAMP Neither store milliseconds Improving MySQL Productivity from Design to Implementation - 2010.06
  25. 25. The beauty of ENUM Values Check Constraint Ideal for static codes Compact - i.e. 1 byte for 'True', 'False' Human readable values 5.1 Non blocking ALTER Improving MySQL Productivity from Design to Implementation - 2010.06
  26. 26. TRUE/FALSE Datatype Examples CREATE TABLE enums ( flag1 CHAR(1) NOT NULL COMMENT 'T or F, Y or N', flag2 TINYINT NOT NULL COMMENT '0 or 1', flag3 BIT NOT NULL COMMENT 'True or False', flag4 ENUM ('True','False') NOT NULL ); INSERT INTO enums(flag4) VALUES ('True', 'False'); SELECT flag4 FROM enums; Improving MySQL Productivity from Design to Implementation - 2010.06
  27. 27. InnoDB Primary Key Primary key is clustered index 15/16 fill factor for sequential 50% for random Secondary Indexes hold primary key value Best Practice - Keep primary keys short except for high Disk I/O Improving MySQL Productivity from Design to Implementation - 2010.06
  28. 28. Schema Best Practices Have pristine schema Use different schemas Backup copies of tables Temporary objects Improving MySQL Productivity from Design to Implementation - 2010.06
  29. 29. Schema Management Always have current schema.sql Use Patch/Revert SQL for upgrades See http://schemasync.org Productivity ROI • Reproducibility • Upgrade/Downgrade path Improving MySQL Productivity from Design to Implementation - 2010.06
  30. 30. Best Practices Separate Schema changes and application code Productivity ROI • Developers develop • Administrators administer Improving MySQL Productivity from Design to Implementation - 2010.06
  31. 31. Sharding 101 Design Sharding 101 Improving MySQL Productivity from Design to Implementation - 2010.06
  32. 32. Sharding 101 A database table consists of Columns Rows Improving MySQL Productivity from Design to Implementation - 2010.06
  33. 33. Sharding 101 Columns col1 col2 col3 col4 col5 col6 col7 Improving MySQL Productivity from Design to Implementation - 2010.06
  34. 34. Sharding 101 Columns col1 col2 col3 col4 col5 col6 col7 Improving MySQL Productivity from Design to Implementation - 2010.06
  35. 35. Sharding 101 Columns col1 col2 col3 col4 col5 col6 col7 Improving MySQL Productivity from Design to Implementation - 2010.06
  36. 36. Sharding 101 Rows col1 col2 col3 col4 col5 col6 col7 row1 row2 row3 row4 row5 row6 row7 row8 row9 row10 row11 row12 row13 row14 Improving MySQL Productivity from Design to Implementation - 2010.06
  37. 37. Sharding 101 Rows col1 col2 col3 col4 col5 col6 col7 row1 row2 row3 row4 row5 row6 row7 row8 row9 row10 row11 row12 row13 row14 Improving MySQL Productivity from Design to Implementation - 2010.06
  38. 38. Sharding 101 Rows col1 col2 col3 col4 col5 col6 col7 row1 row2 row3 row4 row5 row6 row7 row8 row9 row10 row11 row12 row13 row14 Improving MySQL Productivity from Design to Implementation - 2010.06
  39. 39. Sharding 101 A database consists of Tables Table Rows Improving MySQL Productivity from Design to Implementation - 2010.06
  40. 40. Sharding 101 Tables table1 table2 table3 table4 table5 table6 table7 Improving MySQL Productivity from Design to Implementation - 2010.06
  41. 41. Sharding 101 Tables table1 table2 table3 table4 table5 table6 table7 Improving MySQL Productivity from Design to Implementation - 2010.06
  42. 42. Sharding 101 Tables table1 table2 table3 table4 table5 table6 table7 Improving MySQL Productivity from Design to Implementation - 2010.06
  43. 43. Sharding 101 Table Rows table1 table2 table3 table4 table5 table6 table7 row1 row2 row3 row4 row5 row6 row7 row8 row9 row10 row11 row12 row13 row14 Improving MySQL Productivity from Design to Implementation - 2010.06
  44. 44. Sharding 101 Table Rows table1 table2 table3 table4 table5 table6 table7 row1 row2 row3 row4 row5 row6 row7 row8 row9 row10 row11 row12 row13 row14 Improving MySQL Productivity from Design to Implementation - 2010.06
  45. 45. Sharding 101 Table Rows table1 table2 table3 table4 table5 table6 table7 row1 row2 row3 row4 row5 row6 row7 row8 row9 row10 row11 row12 row13 row14 Improving MySQL Productivity from Design to Implementation - 2010.06
  46. 46. Sharding 101 Partitioning is? Grouping like rows in a table together e.g. By Date By Local By parent grouping Improving MySQL Productivity from Design to Implementation - 2010.06
  47. 47. Sharding 101 Partition col1 col2 col3 col4 col5 col6 col7 row1 row2 row3 row4 row5 row6 row7 row8 row9 row10 row11 row12 row13 row14 Improving MySQL Productivity from Design to Implementation - 2010.06
  48. 48. Sharding 101 Partition col1 col2 col3 col4 col5 col6 col7 row1 row2 row3 row4 row5 row6 row7 row8 row9 row10 row11 row12 row13 row14 Improving MySQL Productivity from Design to Implementation - 2010.06
  49. 49. Sharding 101 Partition col1 col2 col3 col4 col5 col6 col7 row1 row2 row3 row4 row5 row6 row7 row8 row9 row10 row11 row12 row13 row14 Improving MySQL Productivity from Design to Implementation - 2010.06
  50. 50. Sharding 101 Sharding is? Grouping like tables and/or like rows together Improving MySQL Productivity from Design to Implementation - 2010.06
  51. 51. Sharding 101 Tables table1 table2 table3 table4 table5 table6 table7 Improving MySQL Productivity from Design to Implementation - 2010.06
  52. 52. Sharding 101 Tables table1 table2 table3 table4 table5 table6 table7 Improving MySQL Productivity from Design to Implementation - 2010.06
  53. 53. Sharding 101 Tables table1 table2 table3 table4 table5 table6 table7 Improving MySQL Productivity from Design to Implementation - 2010.06
  54. 54. Sharding 101 Tables + Rows table1 table2 table3 table4 table5 table6 table7 1 1 Improving MySQL Productivity from Design to Implementation - 2010.06
  55. 55. Sharding 101 Tables + Rows table1 table2 table3 table4 table5 table6 table7 2 2 Improving MySQL Productivity from Design to Implementation - 2010.06
  56. 56. Sharding 101 Tables + Rows table1 table2 table3 table4 table5 table6 table7 3 3 Improving MySQL Productivity from Design to Implementation - 2010.06
  57. 57. Sharding 101 Shards table1 table2 table3 table4 table5 table6 table7 1 1 2 2 3 3 Improving MySQL Productivity from Design to Implementation - 2010.06
  58. 58. Sharding 101 Shards table1 table2 table3 table4 table5 table6 table7 1 1 4 4 4 2 2 3 3 Improving MySQL Productivity from Design to Implementation - 2010.06
  59. 59. Sharding 101 Shards table1 table2 table3 table4 table5 table6 table7 5 1 1 6 4 4 2 2 7 8 3 3 9 Improving MySQL Productivity from Design to Implementation - 2010.06
  60. 60. Sharding 101 How do you group tables? That’s the secret Improving MySQL Productivity from Design to Implementation - 2010.06
  61. 61. Sharding 101 - First Steps First Steps Consider schema separation Define a good partition key Balanced solution Support rebalancing Productivity ROI • Write Scalability • Done before application grows in complexity and usage Improving MySQL Productivity from Design to Implementation - 2010.06
  62. 62. Sharding 101 - Next Steps Next Steps Managing Joins Searching Data Unique Constraints Aggregated Reporting Productivity ROI • Write Scalability • Done before application grows in complexity and usage Improving MySQL Productivity from Design to Implementation - 2010.06
  63. 63. Manual Partitioning Ideal for Write Once Data MySQL supports Atomic table rename RENAME TABLE current TO old, new TO current; Improving MySQL Productivity from Design to Implementation - 2010.06
  64. 64. Data Partitioning Feature of MySQL 5.1 RANGE LIST HASH KEY http://dev.mysql.com/doc/refman/5.1/en/partitioning.html Improving MySQL Productivity from Design to Implementation - 2010.06
  65. 65. Sharding 101 Design Supporting Growth Improving MySQL Productivity from Design to Implementation - 2010.06
  66. 66. Supporting Growth Seek Professional Advice Availability Scalability Resilience Improving MySQL Productivity from Design to Implementation - 2010.06
  67. 67. Availability How do you support maintenance? Software upgrades Application releases 24x7 operations Eliminate single points of failure Productivity ROI • Less Downtime • Greater Sales / Better Reputation Improving MySQL Productivity from Design to Implementation - 2010.06
  68. 68. Scalability Read Scalability Replication Write Scalability Sharding Caching Implement from day 1 Use best product for purpose Productivity ROI • Support Demand Improving MySQL Productivity from Design to Implementation - 2010.06
  69. 69. Scalability Implementation Implementation H/W acquisition time Zero new human resource need Zero application changes Productivity ROI • Seamless growth support Improving MySQL Productivity from Design to Implementation - 2010.06
  70. 70. Resilience What is your backup strategy? What is your recovery strategy? How long does it take? Have you actually tested it end to end? Take the survey http://ronaldbradford.com/blog/checked-your-mysql-recovery-process-recently-2010-02-15/ Productivity ROI Improving MySQL Productivity from Design to Implementation - 2010.06 • Company viability
  71. 71. Design MySQL Configuration Improving MySQL Productivity from Design to Implementation - 2010.06
  72. 72. Configuration Best Practices Always use transactions SQL_MODE Data Integrity MySQL Idiosyncrasies that BITE Talk http://ronaldbradford.com/blog/mysql-idiosyncrasies-that-bite-2010-06-28/ Improving MySQL Productivity from Design to Implementation - 2010.06
  73. 73. Design Security Improving MySQL Productivity from Design to Implementation - 2010.06
  74. 74. MySQL User Security Default is woeful Minimum $ mysql_secure_installation Recommended Operating System Permissions & Privileges Improving MySQL Productivity from Design to Implementation - 2010.06
  75. 75. Operating System Security Defaults are not secure Never run as 'root' user Separate Data/Binary Logs/Logs/ Configuration/Backups Individual directory permissions Productivity ROI • Minimize security risk • Better auditability Improving MySQL Productivity from Design to Implementation - 2010.06
  76. 76. MySQL Installation Best Practice Best Practice /mysql /etc/my.cnf /etc /etc/profile.d/mysql.sh /data /etc/init.d/mysqld /binlog /log /mysql-version Improving MySQL Productivity from Design to Implementation - 2010.06
  77. 77. MySQL Installation Security Software installed by root $ chown -R root:root /mysql $ chown -R mysql:mysql /mysql/{data,log,binlog/etc} $ chmod 700 /mysql/{data,binlog} $ chmod 750 /mysql/{etc,log} Improving MySQL Productivity from Design to Implementation - 2010.06
  78. 78. Application User Security Best Practice CREATE USER appuser@localhost IDENTIFIED BY 'sakila'; GRANT SELECT,INSERT,UPDATE,DELETE ON schema.* TO appuser@localhost; Normal Practice CREATE USER superman@'%'; GRANT ALL ON *.* TO superman@'%'; See MySQL Idiosyncrasies that BITE session for more information Improving MySQL Productivity from Design to Implementation - 2010.06
  79. 79. Application User Security Best Practices Application User (Read/Write Access) INSERT, UPDATE, DELETE, SELECT Application Viewer (Read Only Access) SELECT Application DBA (Schema Access Only) CREATE, DROP, CREATE ROUTINE, SELECT, INSERT, UPDATE, DELETE Productivity ROI • Track Data Security • Separation of responsibilities Improving MySQL Productivity from Design to Implementation - 2010.06
  80. 80. Why not use GRANT ALL GRANT ALL ON *.* TO user@’%’ *.* gives you access to all tables in all schemas @’%’ give you access from any external location ALL gives you ALTER, ALTER ROUTINE, CREATE, CREATE ROUTINE, CREATE TEMPORARY TABLES, CREATE USER, CREATE VIEW, DELETE, DROP, EVENT, EXECUTE, FILE, INDEX, INSERT, LOCK TABLES, PROCESS, REFERENCES, RELOAD, REPLICATION CLIENT, REPLICATION SLAVE, SELECT, SHOW DATABASES, SHOW VIEW, SHUTDOWN, SUPER, TRIGGER, UPDATE, USAGE See MySQL Idiosyncrasies that BITE session for more information Improving MySQL Productivity from Design to Implementation - 2010.06
  81. 81. Why SUPER is bad? SUPER Bypasses read_only Bypasses init_connect Can Disable binary logging Change configuration dynamically No reserved connection Improving MySQL Productivity from Design to Implementation - 2010.06
  82. 82. Design Development Security Improving MySQL Productivity from Design to Implementation - 2010.06
  83. 83. Development Best Practices Version control General query log SQL Commenting SQL Formatting Future proofing Embrace CLI Caching Improving MySQL Productivity from Design to Implementation - 2010.06
  84. 84. Version Control Use for all development Especially single user development Use for schema management Building block of automated build/release Productivity ROI • Central repository of code • Track code changes by developer • Create reproducable releases Improving MySQL Productivity from Design to Implementation - 2010.06
  85. 85. General Query Log Turn on for all development environments Logs all SQL Statements Aggregate and Email results to developer See Capturing, Analyzing and Optimizing SQL Presentation Productivity ROI • Developers are seeing SQL in operation • Enables access to SQL to analyze Improving MySQL Productivity from Design to Implementation - 2010.06
  86. 86. Leveraging General Query Log For single user development environment In SQL Session mysql> SELECT 'Function X Start'; In Application Run Function/Process In SQL Session mysql> SELECT 'Function X End'; Improving MySQL Productivity from Design to Implementation - 2010.06
  87. 87. Slow Query Log Turn on for all development environments Logs slow SQL Statements ( > 1 second) Aggregate and Email results to developer See Capturing, Analyzing and Optimizing SQL Presentation Improving MySQL Productivity from Design to Implementation - 2010.06
  88. 88. SQL Commenting Identify queries by code function Identify OLTP / Batch / Cache queries SELECT /* XXX123 */ col1, ... UPDATE /* YYY999 */ ... SELECT /* Batch */... Productivity ROI • DBA/SA can identify code function and purpose quickly Improving MySQL Productivity from Design to Implementation - 2010.06
  89. 89. SQL Commenting (2) 26 Query SELECT /* 5m cache */ ..... 26 Query SELECT /* ViewPost */ t.*, tt.*, tr.object_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON tt.term_id = t.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN ('category', 'post_tag') AND tr.object_id IN (2849, 2842, 2836, 2824, 2812, 2680, 2813, 2800, 2770, 2784) ORDER BY t.name ASC 26 Query SELECT /* batch */ meta_key, meta_value FROM wp_usermeta WHERE user_id = 2 Improving MySQL Productivity from Design to Implementation - 2010.06
  90. 90. SQL Formatting Create single line queries Don't embed newlines Enables per line analysis by CLI tools Productivity ROI • DBA/SA can use simple CLI tools including grep,awk,cut etc for SQL analysis Improving MySQL Productivity from Design to Implementation - 2010.06
  91. 91. SQL Formatting (2) 26 Query SELECT FOUND_ROWS() 26 Query SELECT post_id,start,end,allday,rpt,IF(end>='2010-06-04 00:00:00',1,0) AS active FROM wp_ec3_schedule WHERE post_id IN (2849,2842,2836,2824,2812,2680,2770,2784) ORDER BY start 26 Query SELECT * FROM wp_users WHERE user_login = 'ronald' 26 Query SELECT t.*, tt.*, tr.object_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON tt.term_id = t.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN ('category', 'post_tag') AND tr.object_id IN (2849, 2842, 2836, 2824, 2812, 2680, 2813, 2800, 2770, 2784) ORDER BY t.name ASC 26 Query SELECT meta_key, meta_value FROM wp_usermeta WHERE user_id = 2 Improving MySQL Productivity from Design to Implementation - 2010.06
  92. 92. SQL Analysis Bulk trending analysis Volume of SQL statements Query Execution Plan (QEP) Online v Batch/Cache SQL via commenting Productivity ROI • Identify bottlenecks ASAP without load • Iterative Design feedback Improving MySQL Productivity from Design to Implementation - 2010.06
  93. 93. EXPLAIN Basics EXPLAIN SELECT ... No key used key length Extra - Using Filesort, Using Temporary Improving MySQL Productivity from Design to Implementation - 2010.06
  94. 94. Future proofing your SQL Specify INSERT columns INSERT INTO table(a,b,c) VALUES(...) SELECT * is generally bad What columns are actually used in code TEXT/BLOB can cause extra disk I/O New columns can change performance Productivity ROI • Reduce likelihood of runtime errors when structural changes to objects Improving MySQL Productivity from Design to Implementation - 2010.06
  95. 95. Avoid Fancy Constructs DELAYED IGNORE LOW PRIORITY REPLACE Productivity ROI • Changes ACID and statement precedence • May have additional performance overhead Improving MySQL Productivity from Design to Implementation - 2010.06
  96. 96. Using Deterministic Functions Use '2010-06-21' instead of CURDATE() Same for NOW() Leverage Query Cache (if enabled) Productivity ROI • Leverages database caching when enabled • Allows testing via parameterization Improving MySQL Productivity from Design to Implementation - 2010.06
  97. 97. Common Coding Errors Remove Duplicate Code Productivity ROI • Less code to maintain • Remove chance of human errors Improving MySQL Productivity from Design to Implementation - 2010.06
  98. 98. Common SQL Errors Remove redundant SQL Use general query log You may be surprised! Improving MySQL Productivity from Design to Implementation - 2010.06
  99. 99. Common Coding Errors - Repeating Queries The WRONG way 5 Query SELECT * FROM `artist` 5 Query SELECT * FROM `artist` 5 Query SELECT * FROM `artist` 5 Query SELECT * FROM `artist` 5 Query SELECT * FROM `artist` 5 Query SELECT * FROM `artist` WHERE (ArtistID = 196 ) 5 Query SELECT * FROM `artist` WHERE (ArtistID = 2188 ) 5 Query SELECT * FROM `artist` 5 Query SELECT * FROM `artist` 5 Query SELECT * FROM `artist` Improving MySQL Productivity from Design to Implementation - 2010.06
  100. 100. Common Coding Errors - Row at a time (RAT) Processing SELECT option_name, option_value FROM wp_options WHERE autoload = 'yes' SELECT option_value FROM wp_options WHERE option_name = 'aiosp_title_format' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_show_only_even' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_num_months' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_day_length' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_hide_event_box' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_advanced' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_navigation' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'ec3_disable_popups' LIMIT 1 SELECT option_value FROM wp_options WHERE option_name = 'sidebars_widgets' LIMIT 1 Improving MySQL Productivity from Design to Implementation - 2010.06
  101. 101. RAT v CAT Processing Row (RAT) Processing SELECT * FROM activities_theme WHERE theme_parent_id=0 SELECT * FROM activities_theme WHERE theme_parent_id=1 SELECT * FROM activities_theme WHERE theme_parent_id=2 SELECT * FROM activities_theme WHERE theme_parent_id=11 SELECT * FROM activities_theme WHERE theme_parent_id=16 Chunk (CAT) Processing SELECT * FROM activities_theme WHERE theme_parent_id in (0,1,2,11,16) Improving MySQL Productivity from Design to Implementation - 2010.06
  102. 102. Common Coding Errors - Boundary Conditions The following SQL executed 6,000 times in 5 minute analysis period The WRONG way SELECT pages_id, pages_livestats_code, pages_title, pages_parent, pages_exhibid, pages_theme, pages_accession_num FROM pages WHERE pages_id = 0 0 is an invalid pages_id Improving MySQL Productivity from Design to Implementation - 2010.06
  103. 103. In a highly tuned system the greatest time in a query is network overhead Improving MySQL Productivity from Design to Implementation - 2010.06
  104. 104. Development About Database Connections Improving MySQL Productivity from Design to Implementation - 2010.06
  105. 105. Defining your database connection The WRONG way $ cd public_html $ grep "mysql*_connect" * */* */*/* db-disp.php:$cid = mysql_connect("localhost", "museum", "******") or die ('I cannot connect to the database because: ' . mysql_error()); test.php:$cid = mysql_connect("localhost", "museum", "******"); PMCollection/connection.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass"); PMCollection/connection_live.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass"); PMCollection/connection_local.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass"); PMEcards/connection.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass"); core/connection.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass"); discussion_admin/db_fns.php: $cid = mysql_connect("localhost", "museum", "******"); discussion_admin/header.php:// $cid = mysql_connect("localhost", "museum", "******"); discussion_admin/inc_title.php: //$cid = mysql_connect("localhost", "museum", "******"); discussion_admin/stats.php: //$cid = mysql_connect("localhost", "museum", Improving MySQL Productivity from Design to Implementation - 2010.06
  106. 106. Database connection example The RIGHT way class database { function getConnection($type, $message) { $cp = $this->getConnectionParameter($type); if (empty($cp)) { $message = new message ("fatal", "Unable to determine '$type' ... return; } try { $con = mysqli_connect($cp->host,$cp->user,$cp->passwd,$cp->database); if (!$con) { $message = new message ("fatal", "Unable to obtain a '$type' ... return; } mysqli_query($con, "SET NAMES 'utf8'"); } catch (Exception $e) { $message = new message ("fatal", "Unable to obtain a '$type' ... debug($e->getMessage()); } return $con; } Improving MySQL Productivity from Design to Implementation - 2010.06
  107. 107. Database connection parameters example $ cat config/database.inc.sample <? require_once "classes/core/connectionparameter.inc"; $connections = array(); // New connections "description", "host", "user", "passwd", "database", "settings" $connections["write"] = new connectionparameter( "Write DB Connection", "localhost", "writeuser", "******", "db"); $connections["read"] = new connectionparameter( "Read DB Connection", "localhost", "readuser", "******", "db"); ?> Improving MySQL Productivity from Design to Implementation - 2010.06
  108. 108. Database parameterization and version control Parameters in separate file config/database.inc.sample NOTE: .sample in Version Control Does not override deployment Productivity ROI • Automated code deployment to multiple environments Improving MySQL Productivity from Design to Implementation - 2010.06
  109. 109. Horror Stories - Connection Parameters How not to code connection parameters Changing Connector/J Settings (1) Have to reboot server i.e. no way to flush connection pool Improving MySQL Productivity from Design to Implementation - 2010.06
  110. 110. Horror Stories - Connection Parameters How not to code connection parameters Changing Connector/J Settings (2) Rebuild and redeploy jar Improving MySQL Productivity from Design to Implementation - 2010.06
  111. 111. Database Connections Open and close database connections only when necessary Productivity ROI • Reduce unnecessary database load • Increases page serve volume • Increases true DB throughput Improving MySQL Productivity from Design to Implementation - 2010.06
  112. 112. Database Connections Initialization The WRONG way $ cat header.php ... $con = getConnection(); ... if($this_user->user_row["status"]!='ws' && in_array($this_page->getValue(),$page)){ header("Location: /wholesale/permission.php"); exit(); } ... if () { header("Location: abc.php"); exit(); } ... if () { header("Location: xyz.php"); exit(); } ... Improving MySQL Productivity from Design to Implementation - 2010.06
  113. 113. Database Connections Deconstruction The WRONG way $ cat footer.php ... $con->close(); Improving MySQL Productivity from Design to Implementation - 2010.06
  114. 114. Database Connections - Optimal Usage The RIGHT way // do all possible pre-processing first ... // Then get DB connection $con = database->getConnection("read"); // Close your connection ASAP // Always future proof objects method calls if ($con != NULL) { $con->close(); } Improving MySQL Productivity from Design to Implementation - 2010.06
  115. 115. Development Instrumentation Improving MySQL Productivity from Design to Implementation - 2010.06
  116. 116. Application Instrumentation Creating one primary abstract DB call Enable logging of ALL SQL statements Enable embedded HTML output Total Execution Time/Count Individual SQL Execution Time/SQL Productivity ROI • Enable runtime analysis via browser • No additional tools needed to gather Improving MySQL Productivity from Design to Implementation - 2010.06
  117. 117. Application Toolset Performance Analysis – Step by Step Joomla CMS Copyright 2007 MySQL Inc MySQL - The Best Online Database for modern applications 117
  118. 118. Performance Analysis – Step by Step Cold Fusion Copyright 2007 MySQL Inc MySQL - The Best Online Database for modern applications 118
  119. 119. Application Instrumentation Benefits End to End Timing Component Timing (i.e. a series of SQL) Observe as desired Intelligent Activation e.g. Page Load exceeds x ms Improving MySQL Productivity from Design to Implementation - 2010.06
  120. 120. Development Quick Tips Improving MySQL Productivity from Design to Implementation - 2010.06
  121. 121. Using Text Compression Via Database COMPRESS() UNCOMPRESS() Via Application Spread CPU Load to many servers Database Structure Change TEXT --> BLOB Productivity ROI • Shorter Backup/Recovery times • Greater volume of data for disk usage Improving MySQL Productivity from Design to Implementation - 2010.06
  122. 122. Using Text Compression - Example CREATE TABLE `Mkt` ( `MktID` varchar(10) NOT NULL, `Description` text, PRIMARY KEY (`MktID`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1 select min(length(Description)) as min_len, avg(length(Description)) as avg_len, max(length(Description)) as max_len from Mkt; +---------+---------+---------+ | min_len | avg_len | max_len | +---------+---------+---------+ | 20 |1122.920 | 7999 | +---------+---------+---------+ Improving MySQL Productivity from Design to Implementation - 2010.06
  123. 123. Using Compression Example (2) Best 74% saving SELECT length(Description) as len_before, length(compress(Description)) as len_after FROM Mkt WHERE MktID = '42'; +------------+-----------+ | len_before | len_after | +------------+-----------+ | 7810 | 2025 | +------------+-----------+ ...WHERE MktID LIKE '6_'; +------------+-----------+ | len_before | len_after | No benefit for short +------------+-----------+ data | 11 | 20 | | 53 | 53 | | 113 | 100 | Improving MySQL Productivity from Design to Implementation - 2010.06
  124. 124. Using Text Compression (2) Prime Candidates Large Text Fields Repeating data (e.g. xml elements) Not Indexed Not Searched Improving MySQL Productivity from Design to Implementation - 2010.06
  125. 125. Caching 101 Use caching whenever possible Caching options Query results Objects HTML component content Improving MySQL Productivity from Design to Implementation - 2010.06
  126. 126. Design Development Security Testing Improving MySQL Productivity from Design to Implementation - 2010.06
  127. 127. Testing is not about what works; testing is about breaking your software Improving MySQL Productivity from Design to Implementation - 2010.06
  128. 128. Writing testable code Small code functions Use Refactoring Improving MySQL Productivity from Design to Implementation - 2010.06
  129. 129. Query Analysis Database Size / data volume Version Query QEP Result set size Rows affected/retrieved Table structure Improving MySQL Productivity from Design to Implementation - 2010.06
  130. 130. Function Testability MVC API Improving MySQL Productivity from Design to Implementation - 2010.06
  131. 131. Design Development Security Testing Instrumentation Improving MySQL Productivity from Design to Implementation - 2010.06
  132. 132. If you don't have monitoring in place, make it your top priority Improving MySQL Productivity from Design to Implementation - 2010.06
  133. 133. Instrumentation Objectives System Dashboard Monitoring Alerting Public System Status Improving MySQL Productivity from Design to Implementation - 2010.06
  134. 134. System Dashboard What is the state of NOW! sampling 1,5,10 seconds One page listing most important numbers Red/Yellow/Green Private/Public View Productivity ROI • One starting point for all staff. No manual "let me see" needed. Improving MySQL Productivity from Design to Implementation - 2010.06
  135. 135. Title Screen print Dashboard Example Improving MySQL Productivity from Design to Implementation - 2010.06
  136. 136. Monitoring Records history over time Sampling (e.g. 5 minutes / 1 minute) Graphical Analysis Improving MySQL Productivity from Design to Implementation - 2010.06
  137. 137. Alerting Identify key limits Notification rules 24x7 Improving MySQL Productivity from Design to Implementation - 2010.06
  138. 138. Public Status Keep your users informed Planned outage Unplanned outage http://status.twitter.com/ http://status.blogs.smugmug.com/ Host elsewhere Use a Blog Allows for user feedback Improving MySQL Productivity from Design to Implementation - 2010.06
  139. 139. The Rule of Everything Everything Monitor Measure Graph Automate Improving MySQL Productivity from Design to Implementation - 2010.06
  140. 140. Design Development Security Testing Implementation Instrumentation Improving MySQL Productivity from Design to Implementation - 2010.06
  141. 141. Implementation Objectives Instrumentation Implementation No Downtime! Pre-emptive Load Management Higher Availability Automated Deployment Improving MySQL Productivity from Design to Implementation - 2010.06
  142. 142. Development Team: We need a maintenance window for software upgrades and new releases. Improving MySQL Productivity from Design to Implementation - 2010.06
  143. 143. Management Team: No Downtime Improving MySQL Productivity from Design to Implementation - 2010.06
  144. 144. Development Team: But we need this to fix problems and improve performance. Improving MySQL Productivity from Design to Implementation - 2010.06
  145. 145. Management Team: No Downtime Improving MySQL Productivity from Design to Implementation - 2010.06
  146. 146. What is your definition of no downtime? Improving MySQL Productivity from Design to Implementation - 2010.06
  147. 147. What is your requirements for no downtime? For example: Serve Pages Serve Ads NOTE: No need to add/change content Search data Charge for ad impressions Sell Merchandise Improving MySQL Productivity from Design to Implementation - 2010.06
  148. 148. MySQL Higher Availability - First Step MySQL pair Master / Fail Over Master Only as good as the weakest link Slave SQL_THREAD Slave Disk I/O Productivity ROI • Actively tests primary DR strategy Improving MySQL Productivity from Design to Implementation - 2010.06
  149. 149. Pre-emptive Load Management Improving MySQL Productivity from Design to Implementation - 2010.06
  150. 150. Pre-emptive Load Management (2) What is the Twitter "Failed Whale" Based on page Load time/Volume Reject early http://en.oreilly.com/velocity2009/public/schedule/detail/7479 http://www.slideshare.net/netik/billions-of-hits-scaling-twitter Improving MySQL Productivity from Design to Implementation - 2010.06
  151. 151. Pre-emptive Load Management (3) Combined with "Failed Whale" Darkmode Disable intensive tasks (e.g. Name Search) Improving MySQL Productivity from Design to Implementation - 2010.06
  152. 152. Pre-emptive Load Management (4) Data Messaging Queue Per table/chunk Write Access Read Access No Access Proactively disable/limit application access Improving MySQL Productivity from Design to Implementation - 2010.06
  153. 153. Data Messaging Queue Example Disable Write access to user data Application changes via notification Disable option/Notification message New user registration Edit profile Read Access to login/lookup etc Improving MySQL Productivity from Design to Implementation - 2010.06
  154. 154. Su Design pp or Development t Security Testing Implementation Instrumentation Improving MySQL Productivity from Design to Implementation - 2010.06
  155. 155. Number 1 problem! Don't change a setting without evidence to prove/ disprove the Improving MySQL Productivity from Design to Implementation - 2010.06
  156. 156. Where is the bottleneck? Is the database the problem? Front End Performance Improving MySQL Productivity from Design to Implementation - 2010.06
  157. 157. Perception The website will always be too slow Identify the true components End to End time Database may be only a small portion Improving MySQL Productivity from Design to Implementation - 2010.06
  158. 158. Front End Tips • Know your total website load time http://getfirebug.com/ • How much time is actually database related? • Reduce HTML page size - 15% improvement • Remove full URL’s, inline css styles • Reduce/combine css & js files Improving MySQL Productivity from Design to Implementation - 2010.06
  159. 159. Front End Tips • Split static content to different ServerName • Spread static content over multiple ServerNames (e.g. 3) • Sprites - Combining lightweight images - http:// spriteme.org/ • Cookie-less domain name for static content Improving MySQL Productivity from Design to Implementation - 2010.06
  160. 160. Continual Improvement Lower MTTR (mean time to recovery) Lower MTTD (Mean time to detect) Improving MySQL Productivity from Design to Implementation - 2010.06
  161. 161. Conclusion Improving MySQL Productivity from Design to Implementation - 2010.06
  162. 162. Availability Su g Datatypes Scalability pp n Design di ar Sh Resilience Schema Management or Development t Database Connections Version control Caching SQL Standards Break Security Testing your User Management system NO DOWNTIME Implementation User Experience Automation Instrumentation Dashboard - Monitoring - Alerting Improving MySQL Productivity from Design to Implementation - 2010.06
  163. 163. RonaldBradford.com MySQL4OracleDBA.com Improving MySQL Productivity from Design to Implementation - 2010.06
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×