Sometimesreplicationis enough!     1                             © Ralf Schwoebelnext: Content   puzzler@tradebit.com / tw...
Content•   Intro•   A real life setup•   Replication with MySQL 5.5•   Load balancing tricks•   Sphinx, we love it!•   The...
Intro: Tradebit and Ralf  Ralf (puzzler) – born 1973:  • BBS and Game Dev on Commodore Amiga  • Started online shops in ‘9...
What we will talk about...                              Source: http://www.getelastic.com/site-speed-infographic/         ...
First – What do you REALLY need?                      5                                © Ralf Schwoebel             next: ...
The bottleneck is your setup!                    6                                 © Ralf Schwoebel           next: The wh...
Real Life Setup: Tradebit.com 2012   USA         Latin A.     Europe           Asia          AustraliaContent Distribution...
Setup: Does it make sense?                         Oh, yeah – it does!               8                               © Ral...
Great things about ... squidSquid & Snort Layer:• Takes the load from the webservers (and DB servers)• Makes you IPv6 read...
Great things about ... CachingAPC:              One of 6                               10                               © ...
Great things about ... The RestMemcache:     Holds most requested pages as a wholeNFS:          Darn stable, stone old, no...
Some bad things...Things you probably do not want to do in this setup:• directory browsing via NFS for anything speedy• ha...
Short Live Demohttp://www.tradebit.com/filedetail.php/178346105-new-york-empire-state-of-mind-                            ...
MySQL ... Replication basicsReplication means:write to a MASTERMySQL replicatesto a SLAVE                                 ...
MySQL ... Replication basicsReplication is ONLY SOMETIMES enough:• Consumer websites with heavy traffic reading  (4 Mio UV...
MySQL ... and friendsMySQL Setup for millions of products:• Is good, but you need to know what you are doing!• Search: for...
MySQL ... and friends                Main product table:                • 37 million products                • Around 12 m...
MySQL ... replicationOn a side note: InnoDB and some config basics!                   innodb_file_per_table               ...
MySQL ... replicationweb writes – master 1                                             shell writes – master 2            ...
MySQL ... replicationInserts, updates, etc. - September, 17th on tradebit:•   920 new products•   1,930 new orders•   15,5...
MySQL ... replicationSo, 3.5 Mio reads per day, 100k writing… fine, but:  • Why 8 servers?    2 x 4 combination gives us f...
MySQL ... replication                  22                                      © Ralf Schwoebel     next: SQL principles o...
MySQL ... replicationBe careful, what you update:     UPDATE mytable SET mycol=2 WHERE my2ndcol LIKE ‘%.jpg’is EVIL and co...
MySQL ... replicationstatic function dbUpdateRowsById($strTable, $strSetCols, $strIdName, $strSelectIdSql) {        $nMax ...
MySQL ... replicationNever stop to optimize…                           25                         © Ralf Schwoebel        ...
MySQL ... Replication MonitoringSimple checking:• Count the crucial entries via script• grep the important infos[root@w1 ~...
MySQL ... Replication MonitoringOr use the percona tools…http://www.percona.com/doc/percona-toolkit/2.1/pt-heartbeat.html ...
MySQL ... When *it hits the fan...Replication gets out of sync, sooner or later:                                   28     ...
MySQL ... Replication FixingPoor admin solution…[root@DS7-SP5 ~]# more /cloud/MySQLbackup.sh#!/bin/bashcd /cloud/baks/mysq...
MySQL ... Replication FixingPoor admin solution… play it back instop slave;mysql DBname <table.sqlmysql> CHANGE MASTER TO ...
MySQL ... Replication FixingPro solution: Percona tools rule, thanks buddies!pt-table-sync --print u=tradebit,h=dm2,D=trad...
MySQL ... Replication SummaryConclusion:• Simplicity – do not let people talk you into complicated setups• Speed – rather ...
ThanksI am available for questions:Ralf SchwoebeleMail:      puzzler@tradebit.comTwitter:    trabitXING:       https://www...
Upcoming SlideShare
Loading in …5
×

MySQL Install for Replication - Real Life Tutorial

3,308 views

Published on

This presentation from PerconaLive New York 2012 gives you a detailed description on a live MySQL replication integration on http://www.tradebit.com/ - it should help you to increase your page loading speed and gives away some cool geek tricks to take load from your webservers.

Mail to puzzler(at)tradebit(dot)com for infos, if you need them!

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
3,308
On SlideShare
0
From Embeds
0
Number of Embeds
195
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

MySQL Install for Replication - Real Life Tutorial

  1. 1. Sometimesreplicationis enough! 1 © Ralf Schwoebelnext: Content puzzler@tradebit.com / twitter: trabit
  2. 2. Content• Intro• A real life setup• Replication with MySQL 5.5• Load balancing tricks• Sphinx, we love it!• The crazy daily• Q&A 2 © Ralf Schwoebel next: Intro puzzler@tradebit.com / twitter: trabit
  3. 3. Intro: Tradebit and Ralf Ralf (puzzler) – born 1973: • BBS and Game Dev on Commodore Amiga • Started online shops in ‘95 - Perl on SuSE • Codes in PHP, RealBasic, Scripts in bash • Went into SEO in 2002 Tradebit.com – born 2004: • First a simple download shop • Became a marketplace • Grew from single server to mini-cloud • Turns around $5 Mio. per year 3 © Ralf Schwoebel next: Simplicity puzzler@tradebit.com / twitter: trabit
  4. 4. What we will talk about... Source: http://www.getelastic.com/site-speed-infographic/ 4 © Ralf Schwoebel next: More Revenue puzzler@tradebit.com / twitter: trabit
  5. 5. First – What do you REALLY need? 5 © Ralf Schwoebel next: The Bottleneck puzzler@tradebit.com / twitter: trabit
  6. 6. The bottleneck is your setup! 6 © Ralf Schwoebel next: The whole setup puzzler@tradebit.com / twitter: trabit
  7. 7. Real Life Setup: Tradebit.com 2012 USA Latin A. Europe Asia AustraliaContent Distribution Network: cdn.tradebit.org (Softlayer) pa pb pc pd pe pf pgSquid Layer, FTP Frontend, SNORT & Guardian (iptables) w1 w2 w3 w4 w5 w6PHP Layer, Apache 2, Local Memcache, apcdm1 dm2 ds1 ds2 ds3 ds4 ds5 ds6Database Layer: 2 Masters, 6 Slaves s1 s2 s3 s4 cloudfuse sp1 sp2Storage Layer: NFS Exports, 60 Terabytes MD3k, 500 TB cloud Search Layer: Sphinx 7 © Ralf Schwoebel next: Does it make sense? puzzler@tradebit.com / twitter: trabit
  8. 8. Setup: Does it make sense? Oh, yeah – it does! 8 © Ralf Schwoebel next: squid puzzler@tradebit.com / twitter: trabit
  9. 9. Great things about ... squidSquid & Snort Layer:• Takes the load from the webservers (and DB servers)• Makes you IPv6 ready in less than 20 minutes• Keeps the basic hack attacks under control• Less vulnerable on hardware failure (just with Round Robin) 9 © Ralf Schwoebel next: Caching in general puzzler@tradebit.com / twitter: trabit
  10. 10. Great things about ... CachingAPC: One of 6 10 © Ralf Schwoebel next: Other tools puzzler@tradebit.com / twitter: trabit
  11. 11. Great things about ... The RestMemcache: Holds most requested pages as a wholeNFS: Darn stable, stone old, no surprisesSphinx: Stable, fast and easy to manageHosted CDN: Optional quick solution for your static files 11 © Ralf Schwoebel next: Bad stuff puzzler@tradebit.com / twitter: trabit
  12. 12. Some bad things...Things you probably do not want to do in this setup:• directory browsing via NFS for anything speedy• have 3rd party software open holes (aka Wordpress)• having the SNORT behave oddly (like blocking Google)• loosing orders (who wants that in any setup?) 12 © Ralf Schwoebel next: Live Demo puzzler@tradebit.com / twitter: trabit
  13. 13. Short Live Demohttp://www.tradebit.com/filedetail.php/178346105-new-york-empire-state-of-mind- 13 © Ralf Schwoebel next: Replication basics puzzler@tradebit.com / twitter: trabit
  14. 14. MySQL ... Replication basicsReplication means:write to a MASTERMySQL replicatesto a SLAVE 14 © Ralf Schwoebel next: When is replication ok? puzzler@tradebit.com / twitter: trabit
  15. 15. MySQL ... Replication basicsReplication is ONLY SOMETIMES enough:• Consumer websites with heavy traffic reading (4 Mio UV/day is OK, I vouch for that...)• Large datasets for archiving• NON critical requests: like statistics or backup strategiesFor A LOT of write events throughout the day (think constantly10 million or more a day): you would need a different approach! 15 © Ralf Schwoebel next: Replication and friends puzzler@tradebit.com / twitter: trabit
  16. 16. MySQL ... and friendsMySQL Setup for millions of products:• Is good, but you need to know what you are doing!• Search: forget “match against” – quickly!• Browsing: any filtering on big tables is too slow• 2 rules (I learned from Percona): • Use numbers/boolean where possible • Avoid “like ‘%xyz%’” like the plague! 16 © Ralf Schwoebel next: DB sizes puzzler@tradebit.com / twitter: trabit
  17. 17. MySQL ... and friends Main product table: • 37 million products • Around 12 million in 2 or more languages 17 © Ralf Schwoebel next: Config tricks puzzler@tradebit.com / twitter: trabit
  18. 18. MySQL ... replicationOn a side note: InnoDB and some config basics! innodb_file_per_table replicate-ignore-db=mysql replicate-ignore-db=test replicate-wild-ignore-table=tradebitTmp.% slave-skip-errors=all 18 © Ralf Schwoebel next: Replication servers puzzler@tradebit.com / twitter: trabit
  19. 19. MySQL ... replicationweb writes – master 1 shell writes – master 2 Dual Quad Core Dual Quad Core 32 Gig RAM, RAID5 32 Gig RAM, RAID5 Dual Quad Core Dual Quad Core Dual Quad Core Dual Quad Core 32 GB, RAID0 32 GB, RAID0 32 GB, RAID0 32 GB, RAID0 Dual Quad Core DQuad, 24 GB, Dual Quad Core DQuad, 24 GB, 32 GB, RAID0 RAID0: sphinx 32 GB, RAID0 RAID0: sphinx Global: config.php Config::$Dbwrite[0]=“dm1”; Config::$Dbwrite[1]=“dm2”; Config::$Dbread[0..5] = “ds1...6”; 19 © Ralf Schwoebel next: Numbers on tb puzzler@tradebit.com / twitter: trabit
  20. 20. MySQL ... replicationInserts, updates, etc. - September, 17th on tradebit:• 920 new products• 1,930 new orders• 15,500 debug entries in the DB• Around 35,000 additional modification operations (affiliates)• Around 130,000 unique visits Roughly 100,000 inserts/update operations Roughly 3.5 Mio. read operations (hello Googlebot, Bingbot, etc.) [root@DM1 ~]# uptime …., load average: 0.02, 0.05, 0.02 20 © Ralf Schwoebel next: Why so much? puzzler@tradebit.com / twitter: trabit
  21. 21. MySQL ... replicationSo, 3.5 Mio reads per day, 100k writing… fine, but: • Why 8 servers? 2 x 4 combination gives us failover! The site can fully run (slower) with 1 strain… • Why no cluster? Replication is MUCH easier to setup than a cluster and recovery is easier! 21 © Ralf Schwoebel next: When to write? puzzler@tradebit.com / twitter: trabit
  22. 22. MySQL ... replication 22 © Ralf Schwoebel next: SQL principles on updates puzzler@tradebit.com / twitter: trabit
  23. 23. MySQL ... replicationBe careful, what you update: UPDATE mytable SET mycol=2 WHERE my2ndcol LIKE ‘%.jpg’is EVIL and could easily kill your replication setup on large tables! 23 © Ralf Schwoebel next: PHP code for updates puzzler@tradebit.com / twitter: trabit
  24. 24. MySQL ... replicationstatic function dbUpdateRowsById($strTable, $strSetCols, $strIdName, $strSelectIdSql) { $nMax = 1000; $nAff=0; $result=dbQuery($strSelectIdSql); if ($result) { $strIds = ; $i=0; while (($row=mysql_fetch_array($result))) { $strIds .= ($strIds?,:).$row[0]; $i++; if ($i>$nMax) { dbQuery("update $strTable set $strSetCols where $strIdName in ($strIds)"); if (mysql_errno()) return false; $nAff += mysql_affected_rows(); $i=0; $strIds=; } } dbFreeResult($result); if ($strIds) { dbQuery("update $strTable set $strSetCols where $strIdName in ($strIds)"); $nAff += mysql_affected_rows(); } } return $nAff; } 24 © Ralf Schwoebel next: Debug line puzzler@tradebit.com / twitter: trabit
  25. 25. MySQL ... replicationNever stop to optimize… 25 © Ralf Schwoebel next: Monitor puzzler@tradebit.com / twitter: trabit
  26. 26. MySQL ... Replication MonitoringSimple checking:• Count the crucial entries via script• grep the important infos[root@w1 ~]# cat /usr/bin/wdbsynccheck#!/bin/bashecho "___________________________________________________________ DM1 Master: |"ssh dm1 mysql -e "show master statusG" |egrep "(File|Position)"ssh dm1 mysql tradebit -e "select count(TB_INDEX) AS SALESCOUNT from TBSALES;"‘echo "___________________________________________________________ DM2 Slave: |"ssh dm2 mysql -e "show slave statusG" |egrep "(Master_Log|Seconds_Behind_Master|Slave_IO_Running|Slave_SQL_Running)“echo "___________________________________________________________ DS1 Slave: |"ssh ds1 mysql -e "show slave statusG" |egrep "(Master_Log|Seconds_Behind_Master|Slave_IO_Running|Slave_SQL_Running)" 26 © Ralf Schwoebel next: monitor with PT puzzler@tradebit.com / twitter: trabit
  27. 27. MySQL ... Replication MonitoringOr use the percona tools…http://www.percona.com/doc/percona-toolkit/2.1/pt-heartbeat.html 27 © Ralf Schwoebel next: It will fail puzzler@tradebit.com / twitter: trabit
  28. 28. MySQL ... When *it hits the fan...Replication gets out of sync, sooner or later: 28 © Ralf Schwoebel next: Poor man fixing puzzler@tradebit.com / twitter: trabit
  29. 29. MySQL ... Replication FixingPoor admin solution…[root@DS7-SP5 ~]# more /cloud/MySQLbackup.sh#!/bin/bashcd /cloud/baks/mysql MYtbDB -e "show slave statusG" >masterstatus_tradebit.txtfor i in `mysql MYtbDB -e "show tables" | grep -v Tables_` do mysqldump --add-drop-table --complete-insert --insert-ignore MYtbDB $i >tb_$i.sqldone=> Dumps around 80 Gig of text in roughly 15 minutes 29 © Ralf Schwoebel next: Poor man fixing 2 puzzler@tradebit.com / twitter: trabit
  30. 30. MySQL ... Replication FixingPoor admin solution… play it back instop slave;mysql DBname <table.sqlmysql> CHANGE MASTER TO -> MASTER_HOST=’dm1, -> MASTER_USER=‚*, -> MASTER_PASSWORD=‘*, -> MASTER_LOG_FILE= mylog.000416 , // Master_Log_File -> MASTER_LOG_POS=12345; // Read_Master_Log_Pos 30 © Ralf Schwoebel next: PT fixing puzzler@tradebit.com / twitter: trabit
  31. 31. MySQL ... Replication FixingPro solution: Percona tools rule, thanks buddies!pt-table-sync --print u=tradebit,h=dm2,D=tradebit,t=$t $i --password * --where $wh >"$d/$i-$t“with example values:pt-table-sync --print u=tradebit,h=dm2,D=tradebit,t=TBSALES ds5 --password * --where “TB_INDEX>100” >fix.sql(note: --sync-to-master) 31 © Ralf Schwoebel next: Summary puzzler@tradebit.com / twitter: trabit
  32. 32. MySQL ... Replication SummaryConclusion:• Simplicity – do not let people talk you into complicated setups• Speed – rather kill features than risk revenue• Relativity – your config and code must co-exist optimized• Use Tools – think outside the box to relieve stressYes, replication CAN be enough. 32 © Ralf Schwoebel next: Downloads puzzler@tradebit.com / twitter: trabit
  33. 33. ThanksI am available for questions:Ralf SchwoebeleMail: puzzler@tradebit.comTwitter: trabitXING: https://www.xing.com/profile/Ralf_SchwoebelDownload: http://www.slideshare.net/extremelongusername 33 © Ralf Schwoebel puzzler@tradebit.com / twitter: trabit

×