MySQL crash course by moshe kaplan

1,557 views
1,483 views

Published on

A short course I had few weeks ago that I wanted to share with you. All the MySQL issues from basics to experts: tuning, ERD, DDL, DML, Backup, Security

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,557
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
0
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • The Matrix, The Architect Speach http://www.youtube.com/watch?v=mod6xr-lyBc A better w/ titles: http://www.youtube.com/watch?v=K3OXs_5AD2A Second, Kong foo: http://www.youtube.com/watch?v=j82GKTgVDkw Twitter Source: http://www.slideshare.net/Blaine/scaling-twitter
  • The world is changing more and more fast You have to minimize NRE You must support unexpected demand You must provide top service (people now leaves cell operator after single incident, rather then 5, 10 years ago Firms a vulnerable – Citi worth 20Bill $ instead of 200 Bill a year ago Break and Mortar bookstores 15 years ago and Amazon…. Will it happen again to banks, insurance, real estate agencies… IS YOUR MARKET THE NEXT FOR PENETRATION – Finance? Real Estate?How to win a rival that is not existing yet? http://www.johnmwillis.com/ibm/cloud-computing-and-the-enterprise/ -
  • Sharded database – OLTP Little/No reporting OLAP – must be implemented for reporting Loads data from sharded DBs Custom mechanism Any commercial
  • http://www.prelovac.com/vladimir/wordpress-optimization-guide
  • Start with nothing: storage, FW, LB, Server and grow… Can buy servers for more than hour
  • MySQL crash course by moshe kaplan

    1. 1. The VP R&D Open Seminar MySQL Crash Course From basics to experts mokplan@gmail.com http://top-performance.blogspot.com
    2. 2. The VP R&D Open SeminarWHY WE CHOSE MYSQL? http://top-performance.blogspot.com
    3. 3. Who is Using MySQL?3 http://top-performance.blogspot.com
    4. 4. Who is Behind MySQLhttp://top-performance.blogspot.com
    5. 5. What MySQL is Made of?5 http://top-performance.blogspot.com http://dev.mysql.com/tech-resources/articles/mysql_5.0_psea1.html
    6. 6. MySQL Limitations 50-150M Records/Table 50-5000 SELECT Statements/Second6 http://top-performance.blogspot.com
    7. 7. Why Do I Care? From 0 to 100 (US mass adaptation) Phone: 100 yrs Radio: 40 yrs TV: 30 yrs Mobile: 20 yrs Internet: 10 yrs Facebook: 2 yrs7 http://top-performance.blogspot.com
    8. 8. 100K New Users/Week8 http://top-performance.blogspot.com
    9. 9. The VP R&D Open SeminarWHAT MYSQL IS MADE OF? http://top-performance.blogspot.com
    10. 10. MySQL StructureDaemon/var/lib/mysql ibdata1 ib_logfile0, Ib_logfile1 /mysql /database /performance_schema/etc/my.cnf http://top-performance.blogspot.com
    11. 11. What to Installwget mysql_packageyum –y localinstall mysql_packageThe Packages: Devel+Client+Server Relevant platform (x86/x64) Relevant operating system http://top-performance.blogspot.com
    12. 12. MySQL Installationhttp://top-performance.blogspot.com
    13. 13. The VP R&D Open SeminarDATABASE LOGICAL DESIGN(OR ERD) http://top-performance.blogspot.com
    14. 14. Instances, Schemas and inside information http://top-performance.blogspot.com
    15. 15. 1:ManyschoolId: schoolId:Primary Key Foreign Keyhttp://top-performance.blogspot.com
    16. 16. Many:Many?: ?:Primary Key Foreign Key http://top-performance.blogspot.com
    17. 17. Many:Many in PracticetestId: testId, studentId:Primary Key studentId: Primary Key Foreign Key http://top-performance.blogspot.com
    18. 18. Design a Student Information System ERD http://top-performance.blogspot.com
    19. 19. The VP R&D Open SeminarFROM ERD TO A DATABASE http://top-performance.blogspot.com
    20. 20. Connect to the Databasehttp://top-performance.blogspot.com
    21. 21. Create a Schema/Databasehttp://top-performance.blogspot.com
    22. 22. Create a Tablehttp://top-performance.blogspot.com
    23. 23. Implement the a Student Information System ERD http://top-performance.blogspot.com
    24. 24. The VP R&D Open SeminarSQL: DDL http://top-performance.blogspot.com
    25. 25. Databases ManipulationSHOW DATABASES;CREATE DATABASE mydb;DROP DATABASE mydb;USE mydb; http://top-performance.blogspot.com
    26. 26. Tables ManipulationSHOW TABLES;CREATE TABLE users (id int,INDEX users_id (id));DROP TABLE users;ALTER TABLE users ADD INDEX IX_id (id);ALTER TABLE users DROP INDEX IX_id; http://top-performance.blogspot.com
    27. 27. The VP R&D Open SeminarSQL: DML http://top-performance.blogspot.com
    28. 28. SELECTSELECT table.field, table.*FROM tableWHERE table.field = 123ORDER BY field1, field2 DESC http://top-performance.blogspot.com
    29. 29. SELECT + JoinSELECT c.*, u.*FROM classes c INNER JOIN users u ON c.teacherId = u.id http://top-performance.blogspot.com
    30. 30. LEFT JOINhttp://top-performance.blogspot.com
    31. 31. GROUP BYSELECT gender, count(id)FROM usersWHERE gender = ‘M’GROUP BY genderHAVING count(id) > 10 http://top-performance.blogspot.com
    32. 32. UNIONSELECT idFROM usersWHERE gender = ‘M’UNION ALLSELECT idFROM usersWHERE gender = ‘F’ORDER BY id; http://top-performance.blogspot.com
    33. 33. UPDATEUPDATE usersSET gender = ‘M’WHERE gender IS NULL; http://top-performance.blogspot.com
    34. 34. INSERTINSERT INTO users (gender)VALUES (‘M’);INSERT INTO users (gender)SELECT genderFROM usersWHERE gender = ‘M’; http://top-performance.blogspot.com
    35. 35. DELETEDELETEFROM usersWHERE gender = ‘M’; http://top-performance.blogspot.com
    36. 36. The VP R&D Open SeminarMySQL Authentication and AuthorizationSECURITY http://top-performance.blogspot.com
    37. 37. Root UserFirst Time mysqladmin -u root password NEWPASSWORDNth Time mysqladmin -u root -poldpassword password newpassHow to enter Mysql –uuser_name –ppassword –D database_name -h host_name/ip -A http://top-performance.blogspot.com
    38. 38. Users Tablemysql> SELECT * FROM mysql.user limit 1G*************************** 1. row ***************************Host: localhostUser: rootPassword: *4FB5C598BC3FF56E262F863A715F8CB2976A9C69Select_priv: YInsert_priv: YUpdate_priv: YDelete_priv: YCreate_priv: YDrop_priv: Y….1 row in set (0.00 sec) http://top-performance.blogspot.com
    39. 39. Create a Users and Provide Permissions-- Create a user w/ passwordCREATE USER readonly@% IDENTIFIED BY‘sakexkas;-- Grant PermissionsGRANT select ON my_db.* to readonly@%;GRANT ALL PRIVILEGES ON *.* to‘master_user’@’%’ WITH GRANT OPTION;FLUSH PRIVILEGES;-- Beware of too complex passwords http://top-performance.blogspot.com
    40. 40. The VP R&D Open SeminarMYSQL AND LINUX ISSUES http://top-performance.blogspot.com
    41. 41. Built in Linux Security MeasuresIP TABLES sudo /etc/init.d/iptables stop sudo yum –y remove iptables Configure the relevant rulesSELinux sudo echo 0 > /selinux/enforce vi /etc/selinux/config: SELINUX=enforcing  SELINUX=disabledFiles Limit (Table Partitioning) /etc/security/limits.conf mysql soft nofile 8192 mysql hard nofile 8192 my.cnf: open_files_limit = 8192 http://top-performance.blogspot.com
    42. 42. Disk Resizinghttp://top-performance.blogspot.com/2012/02/do-you-need-more-disk-space-for-mysql.html http://top-performance.blogspot.com
    43. 43. Create a Master/Slave configuration http://top-performance.blogspot.com
    44. 44. The VP R&D Open SeminarHIGH AVAILABILITYARCHITECTURES http://top-performance.blogspot.com
    45. 45. Master/Slavehttp://top-performance.blogspot.com
    46. 46. Master/Master ?http://top-performance.blogspot.com
    47. 47. Show Status (1)mysql> show slave statusG*************************** 1. row ***************************Slave_IO_State: Waiting for master to send eventMaster_Host: 10.12.12.167Master_User: replMaster_Port: 3306Connect_Retry: 60Master_Log_File: master.000094Read_Master_Log_Pos: 167250867Relay_Log_File: mysqld-relay-bin.000137Relay_Log_Pos: 167251010Relay_Master_Log_File: master.000094Slave_IO_Running: YesSlave_SQL_Running: YesReplicate_Do_DB:Replicate_Ignore_DB:Replicate_Do_Table:Replicate_Ignore_Table:Replicate_Wild_Do_Table:Replicate_Wild_Ignore_Table: mysql.%,information_schema.% http://top-performance.blogspot.com
    48. 48. Show Status (2)Last_Errno: 0Last_Error:Skip_Counter: 0Exec_Master_Log_Pos: 167250867Relay_Log_Space: 167251310Until_Condition: NoneUntil_Log_File:Until_Log_Pos: 0Master_SSL_Allowed: NoMaster_SSL_CA_File:Master_SSL_CA_Path:Master_SSL_Cert:Master_SSL_Cipher:Master_SSL_Key:Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: NoLast_IO_Errno: 0Last_IO_Error:Last_SQL_Errno: 0Last_SQL_Error:Replicate_Ignore_Server_Ids:Master_Server_Id: 11 row in set (0.02 sec) http://top-performance.blogspot.com
    49. 49. SkipDON’T DO THAT unless you have to!The Process: STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; START SLAVE; SHOW SLAVE STATUSG http://top-performance.blogspot.com
    50. 50. How to Implement?Stop a Master and record positionCopy a Master (instance/disk) to a SlaveStart the MasterChange slave server-idSET MASTER HOST …START SLAVEVerifyOr…http://top-performance.blogspot.co.il/2012/03/how-to-setup-mysql-replication-in-11.html http://top-performance.blogspot.com
    51. 51. Create a Master/Slave configuration http://top-performance.blogspot.com
    52. 52. The VP R&D Open SeminarSCALE-ABLE ARCHITECTURES http://top-performance.blogspot.com
    53. 53. Do You Remember?53 http://top-performance.blogspot.com
    54. 54. Strategy A: Sharding54 http://top-performance.blogspot.com
    55. 55. Strategy B: In Memory Databases55 http://top-performance.blogspot.com
    56. 56. 700 Inserts/Sec In Memory Engine 3000 Inserts/Sec Amazon AWS Standard InnoDB Engine Large Instance 700 Inserts/Sec56 http://top-performance.blogspot.com
    57. 57. IMDB - Implementation57 http://top-performance.blogspot.com
    58. 58. Lessons Cache: MySQL IMDB HA: MySQL Replication Persistence: Periodic 5min Batch Scalability: Sharding 4 Bill Req/Day using 25 machines (2000/s)58 http://top-performance.blogspot.com
    59. 59. Strategy C: MapReduce59 http://top-performance.blogspot.com
    60. 60. Strategy D: NoSQL insert get multiget remove truncate <Key, Value> http://wiki.apache.org/cassandra/API60 http://top-performance.blogspot.com
    61. 61. When Should I Choose NoSQL? Eventually Consistent Document Store Key Value http://guyharrison.squarespace.com/blog/tag/nosql61 http://top-performance.blogspot.com
    62. 62. Sharding Again62 http://top-performance.blogspot.com
    63. 63. Vertical Sharding63 http://top-performance.blogspot.com
    64. 64. Horizontal Sharding Static Hashing Complex growth Mod 10 = 0 Mod 10 = 1 Simple Mod 10 = 2 Mod 10 = 3 Mod 10 = 4 Mod 10 = 5 Mod 10 = 6 Mod 10 = 7 Mod 10 = 8 Mod 10 = 964 http://top-performance.blogspot.com
    65. 65. Horizontal Sharding Key locations are defined in a directory Simple growth Directory is SPOF65 http://top-performance.blogspot.com
    66. 66. Horizontal Sharding Static Hashing with Directory Mapping Simple Growth Small Directory still SPOF Mod 1000 = 466 http://top-performance.blogspot.com
    67. 67. Horizontal Sharding Each key signed by DB# generated on creation Simple growth New key generation is SPOF67 http://top-performance.blogspot.com
    68. 68. Limitations: Reporting68 http://top-performance.blogspot.com
    69. 69. Best Practices $connection = new_db_connection("customer://1234"); $statement = $connection->prepare( $sql_statement, $params ); $result = $statement->execute();69 http://top-performance.blogspot.com
    70. 70. The VP R&D Open SeminarBACKUP AND RESTORE http://top-performance.blogspot.com
    71. 71. OptionsmysqldumpPhysical disk copy3rd Party solution: XtraDB, amanda http://top-performance.blogspot.com
    72. 72. Backup Your Serverhttp://top-performance.blogspot.com
    73. 73. The VP R&D Open SeminarHANDLING PROBLEMS http://top-performance.blogspot.com
    74. 74. Error Log Get the Location:[/]$ sudo more /etc/my.cnf | grep -i errorlog-error=/var/lib/mysql/log.err Look for issues[/]$ sudo tail -n 20 /var/lib/mysql/log.err121025 13:24:23 InnoDB: Initializing buffer pool, size = 128.0M121025 13:24:23 InnoDB: Completed initialization of buffer pool121025 13:24:24 InnoDB: highest supported file format is Barracuda.121025 13:24:25 InnoDB: Waiting for the background threads to start http://top-performance.blogspot.com
    75. 75. The VP R&D Open SeminarDEVOPS MYSQL http://top-performance.blogspot.com
    76. 76. -emysql-uuser-ppassword-Ddatabase-hserver-e”SELECT * FROM table;” http://top-performance.blogspot.com
    77. 77. Import Datamysql-uuser-ppassword-Ddatabase-hserver< /dir/to/filename.sql http://top-performance.blogspot.com
    78. 78. Export Datamysqldump-uuser-ppassword--databases database_name--single-transaction --master-data=1 >/path/to/backup/file.sql http://top-performance.blogspot.com
    79. 79. The VP R&D Open SeminarMYSQL TUNING http://top-performance.blogspot.com
    80. 80. my.cnfhttp://top-performance.blogspot.com
    81. 81. mysqltunerhttp://top-performance.blogspot.com
    82. 82. Slow Querylog-slow-queries = /var/log/mysql/mysql-slow.loglong_query_time = 1log-queries-not-using-indexes http://top-performance.blogspot.com
    83. 83. explainmysql> explain SELECT COUNT(*), post_type FROM wp_posts GROUPBY post_type;+--+-------------+--------+-----+-------------+-----------+-------+----+----+-----------+|id| select_type |table | type|possible_keys|key |key_len|ref |rows| Extra+--+-------------+--------+-----+-------------+-----------+-------+----+----+-----------+|1 | SIMPLE |wp_posts|index|NULL |type_status| 140 |NULL|693 | Using index+--+-------------+--------+-----+-------------+-----------+-------+----+----+-----------+1 row in set (0.02 sec) http://top-performance.blogspot.com
    84. 84. Profilinghttp://top-performance.blogspot.co.il/2011/07/mysql-statement-profiling.html http://top-performance.blogspot.com
    85. 85. RAM Usage Optimization Preferred: SizeOf(database) < SizeOf(RAM) innodb_buffer_pool_size 50-70% of your RAM85 http://top-performance.blogspot.com
    86. 86. Do We Utilize the Cache? SHOW STATUS LIKE Qcache%; Qcache_free_blocks 718 Qcache_free_memory 13004008 Qcache_hits 780759 Qcache_inserts 56292 Qcache_lowmem_prunes 0 Qcache_not_cached 3711 Qcache_queries_in_cache 1715 Qcache_total_blocks 434486 http://top-performance.blogspot.com
    87. 87. Cache Queries query_cache_type = 1 query_cache_limit = 1M query_cache_size = 16M87 http://top-performance.blogspot.com
    88. 88. Buffer Optimization key_reads/key_read_requests < 0.01 If not Increase Key Buffer88 http://top-performance.blogspot.com
    89. 89. CPU Optimization thread_concurrency = 2 X #(CPU Cores)89 http://top-performance.blogspot.com
    90. 90. Tune Your Serverhttp://top-performance.blogspot.com
    91. 91. Summaryhttp://top-performance.blogspot.com
    92. 92. What happens when mysql fails to start…Create my.cnf touch /etc/my.cnfCreate the error log vi /etc/my.cnf [mysqld] log-error=/var/lib/mysql/log.errIf the my.cnf is already configured more /etc/my.cnf | grep –i errorRestart mysqlFind the error tail –n 50 /var/lib/mysql/log.errIn this case we kill all the mysql proceses ps –aux | grep –i mysql sudo kill process_id http://top-performance.blogspot.com
    93. 93. The Bottom Line: Grow ∞ Thank you! and Keep Performing! Moshe Kaplan93 http://top-performance.blogspot.com

    ×