Scaling MySQL – Sharding Made Easy
2Agenda• Scalability Issues• MySQL 5.6• Why Do-It-Yourself (DIY) Sharding Sucks• ScaleBase Data Distribution:– Successful ...
3Doron Levari, Founder & CTODoron Levari,Founder & CTOA technologist and long-time veteran of thedatabase industry. Priort...
4What We DoSimply and cost-effectively scaleMySQL to support an infinitenumber of users, transactions and datawith NO disr...
Scalability Issues and MySQL 5.6
6MySQL Scalability Challenges• Too many transactions• Too many users• Too much data• Too many writes• Capacity• Throughput...
7Improvements in MySQL 5.6 – Single BoxPartitioning Improvements– Explicit Partition Selection:SELECT * FROM employeesPART...
8What are my Options1. More/Bigger Hardware?– Temporary fix…you will need new hardware again– More memory…helps mostly wit...
9Scale Out your Existing MySQL• Keep your MySQL - keep your InnoDB• Ecosystem compatibility, preserve skills• 100% applica...
Why Do It Yourself Sharding “Sucks”
11What is Sharding?Wikipedia - Shard (database architecture) http://en.wikipedia.org/wiki/Shard_(database_architecture)A d...
12DIY Sharding ChallengesApplications must be modified to support multiple shards
13• Maintaining DB ops and IPs in the app• Non-optimized sharding strategies– No good way to maintain global tablesreplica...
ScaleBase Data Distribution Overview
15Data Distribution: Application ExperienceWithout ScaleBase: App must be customized to support shardsWith ScaleBase: App ...
16Manual Sharding versus ScaleBaseSharding Limitations:• Major app rewrite, maintaining code• Maintaining DB ops & IPs in ...
17Typical ScaleBase Data Traffic Manager DeploymentApplicationServersBIManagementDatabase A Replica ADatabase B Replica BD...
ScaleBase Data Distribution – In Detail
19ScaleBase Enables MySQL Scale Out without Re-writing Apps• Data distribution and scale-out is part of the databasearchit...
20ScaleBase Enable Scale Out on AWS and PrivateClouds• A virtualized DB environment makes it easy to change realinfrastruc...
21ScaleBase Supports Scale Out on Single & MultipleMachinesAdvantages of severalshards on one machine:– Several smaller My...
22ScaleBase Enables Splitting Shards• ScaleBase also redistributes data across the array to eliminate hotspots, splitting ...
23ScaleBase Re-balances Shards• Special analysis and alerts about approaching limits• ScaleBase dynamically redistributes ...
24ScaleBase Provides Optimal Data Distribution PoliciesA good data distribution policy ensures that a specifictransaction ...
25ScaleBase Eliminates Data SilosWhen a query needs datafrom several databases,ScaleBase:– Runs the query in parallelon al...
26ScaleBase Provides a Fault Tolerant ArchitectureApplicationServersBIManagementDatabase A Replica ADatabase B Replica BDa...
Summary
28ScaleBase Delivers ScalabilityScale toUnlimitedThroughputNo SpecializedHardwareNoRe-architectureNo ApplicationRewrites
30Detailed Scale Out Case StudiesLarge Chip Co• Scalability• Multiple Apps• Multiple growingusers• Availability• MySQL DBS...
31ScaleBase DeploymentEnvironments– Public Cloud– AWS, Rackspace, any– Private cloud– Hosted / on-premiseDatabases Support...
32SummaryScaleBase provides cost-effective Scale-Out solutions• Scale to an infinite number of users, data and transaction...
33Questions (please enter directly into the GTW side panel)paul.campaniello@scalebase.comdoron.levari@scalebase.comwww.Sca...
34Thank You
ScaleBase Webinar: Scaling MySQL - Sharding Made Easy!
Upcoming SlideShare
Loading in...5
×

ScaleBase Webinar: Scaling MySQL - Sharding Made Easy!

724

Published on

Home-grown sharding is hard - REALLY HARD! ScaleBase scales-out MySQL, delivering all the benefits of MySQL sharding, with NONE of the sharding headaches. This webinar explains: MySQL scale-out without embedding code and re-writing apps, Successful sharding on Amazon and private clouds, Single vs. multiple shards per server, Eliminating data silos, Creating a redundant, fault tolerant architecture with no single-point-of-failure, Re-balancing and splitting shards

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
724
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
35
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

ScaleBase Webinar: Scaling MySQL - Sharding Made Easy!

  1. 1. Scaling MySQL – Sharding Made Easy
  2. 2. 2Agenda• Scalability Issues• MySQL 5.6• Why Do-It-Yourself (DIY) Sharding Sucks• ScaleBase Data Distribution:– Successful sharding on Amazon and private clouds– Single vs. multiple shards per server– Eliminating data silos– Creating a redundant, fault-tolerant architecture– Re-balancing and splitting shards• Q & A
  3. 3. 3Doron Levari, Founder & CTODoron Levari,Founder & CTOA technologist and long-time veteran of thedatabase industry. Priorto foundingScaleBase, Doron wasCEO to Aluna.
  4. 4. 4What We DoSimply and cost-effectively scaleMySQL to support an infinitenumber of users, transactions and datawith NO disruption to the existing infrastructure
  5. 5. Scalability Issues and MySQL 5.6
  6. 6. 6MySQL Scalability Challenges• Too many transactions• Too many users• Too much data• Too many writes• Capacity• Throughput• Performance inconsistencies
  7. 7. 7Improvements in MySQL 5.6 – Single BoxPartitioning Improvements– Explicit Partition Selection:SELECT * FROM employeesPARTITION (p0, p2);– Import / Export for Partitioned Tables:Bring a new data set into a partitionedtable, or export a partition to manage itas a regular table ALTER TABLE eEXCHANGE PARTITION p0 WITHTABLE e2;http://dev.mysql.co/tech-resources/articles/whats-new-in-mysql-5.6.htmlReplication Improvements– Optimizations to Row-BasedReplication– Multi-Threaded Slaves– Improvements to Data Integrity– Crash-Safe Slaves– Replication ChecksumsSCALABILITY issues remain due to the limitations of a single box:To ensure ACID, you still face limitations with:- Memory management - Thread management- Semaphores - Locking- Recovery tasksNo new functionality for sharing workloads across multiple boxes
  8. 8. 8What are my Options1. More/Bigger Hardware?– Temporary fix…you will need new hardware again– More memory…helps mostly with “reads,” but not with “writes”– Every write operation is at least 4 write operations in database, plusmultiple activities in the database engine memory2. Application re-architecture?– Steer workload away from the database– Example: introduce a caching layer– Force application re-writes; new test & QA cycles3. Do it Yourself Sharding?4. Migrate to new database architecture– Other RDBMS/NewSQL / NoSQL?– Force application re-writes; new test & QA cycles– ACID/Durability Issues
  9. 9. 9Scale Out your Existing MySQL• Keep your MySQL - keep your InnoDB• Ecosystem compatibility, preserve skills• 100% application compatibility• Smoother migration, no down-time, no forklift• Your data is safe• No “in-memory” magic• No “in-memory” size limitDon’t throw out the baby with the bath water!
  10. 10. Why Do It Yourself Sharding “Sucks”
  11. 11. 11What is Sharding?Wikipedia - Shard (database architecture) http://en.wikipedia.org/wiki/Shard_(database_architecture)A database shard is a horizontal partition in adatabase or search engine. Each individual partitionis referred to as a shard.Horizontal partitioning is a database designprinciple whereby rows of a database table are heldseparately, rather than being split into columns.Each partition forms part of a shard, which may inturn be located on a separate database server orphysical location.
  12. 12. 12DIY Sharding ChallengesApplications must be modified to support multiple shards
  13. 13. 13• Maintaining DB ops and IPs in the app• Non-optimized sharding strategies– No good way to maintain global tablesreplicated across all database• Sacrifices development agility,additional administrative complexity• Results in database silos• Database ecosystem breaks becausethe application “conceals” shardingstrategies internally• Risks for data inconsistency• Adding and removing databasesis not supported…overprovisioning…• Jeopardizes high availability, backups & disaster recovery• Demands custom application code that can fail ACID complianceDIY Sharding ChallengesChallenges exist becauseapplication code changes arerequired to support multipledatabase instances.
  14. 14. ScaleBase Data Distribution Overview
  15. 15. 15Data Distribution: Application ExperienceWithout ScaleBase: App must be customized to support shardsWith ScaleBase: App sees ONE database……and doesn’t require any customizationScaleBase acts as a proxy between the app and thedatabase, virtualizing the database environment
  16. 16. 16Manual Sharding versus ScaleBaseSharding Limitations:• Major app rewrite, maintaining code• Maintaining DB ops & IPs in the app• Administration/3rd party tools are broken• DB silos/Database ecosystem is blind– Application “hides” sharding strategies• Non-optimized data distribution policy– No good way to maintain globaltables, replicated across all database• Sacrifices development agility• Adding/removing DBs is not supported• Risks for data inconsistency• Demands custom application code thatcan fail ACID compliance• Jeopardizes highavailability, backups, and disasterrecoveryScaleBase Benefits:• No hard-coding application re-writes• Unlimited scalability• Improve performance• Real time elasticity• ACID compliance• Verified data consistency• Real time monitoring, traffic analysis• Carefully analyze distribution policy• Enable system upgrades and updates• Simplified, centralized admin– Adding users– Changing schemas– Maintenance scripts– Management queries
  17. 17. 17Typical ScaleBase Data Traffic Manager DeploymentApplicationServersBIManagementDatabase A Replica ADatabase B Replica BDatabase C Replica CDatabase D Replica DUnlimited ScaleScaleBaseArchitectureis Fault Tolerant
  18. 18. ScaleBase Data Distribution – In Detail
  19. 19. 19ScaleBase Enables MySQL Scale Out without Re-writing Apps• Data distribution and scale-out is part of the databasearchitecture, not the application• One IP to connect to, and “see a unified database”– The application– Entire ecosystem (ETL, mysqldump, PHPMyAdmin)– No special sharding wizard developer– No app re-design, re-dev, re-QA, re-test, re-deploy– No hard-coded variables lost in the code– No special documentation
  20. 20. 20ScaleBase Enable Scale Out on AWS and PrivateClouds• A virtualized DB environment makes it easy to change realinfrastructure, because it’s decoupled from the application• No cloud makes your database elastic• ScaleBase enables elasticity of MySQL in the cloud (EC2, RDS, etc.)Scale-up hitsAWS’s tieredconfigurationlimits fastScale-out isunlimited andgives cloudflexibility
  21. 21. 21ScaleBase Supports Scale Out on Single & MultipleMachinesAdvantages of severalshards on one machine:– Several smaller MySQLinstances better utilizecores, memory– When data grows, eachinstance can later onmigrate to a biggermachine of its ownAdvantages of several shardson multiple machines– Leverage commodity hardware– When reaches machine limits -ScaleBase enables online dataredistribution (resharding) andshard-split
  22. 22. 22ScaleBase Enables Splitting Shards• ScaleBase also redistributes data across the array to eliminate hotspots, splitting the hot spot into two databases
  23. 23. 23ScaleBase Re-balances Shards• Special analysis and alerts about approaching limits• ScaleBase dynamically redistributes data (resharding) - moving thedata across the array from the over-utilized to the under-utilized
  24. 24. 24ScaleBase Provides Optimal Data Distribution PoliciesA good data distribution policy ensures that a specifictransaction is directed to a specific database1,000 transactions250transactions250transactions250transactions250transactions1,000 transactions
  25. 25. 25ScaleBase Eliminates Data SilosWhen a query needs datafrom several databases,ScaleBase:– Runs the query in parallelon all databases– Aggregates results into onemeaningful result-set to be returned to the client – the sameresult-set that would have been returned from a single DB!– Including cross-db GROUP BY, ORDER BY, aggregate functions– Including cross-db JOIN operations– Enables 2-phase commit for transactions spanning multipledatabases
  26. 26. 26ScaleBase Provides a Fault Tolerant ArchitectureApplicationServersBIManagementDatabase A Replica ADatabase B Replica BDatabase C Replica CDatabase D Replica DFully RedundantResilience to failuresScheduledmaintenance withoutdowntime
  27. 27. Summary
  28. 28. 28ScaleBase Delivers ScalabilityScale toUnlimitedThroughputNo SpecializedHardwareNoRe-architectureNo ApplicationRewrites
  29. 29. 30Detailed Scale Out Case StudiesLarge Chip Co• Scalability• Multiple Apps• Multiple growingusers• Availability• MySQL DBSolar Edge• Next GenMonitoring App• Massive Scale• Monitors realtime data fromthousands ofdistributedsystemsMozilla• New Product/Next Gen App/AppStore• Scalability• Geo-clusteringAppDynamics• Next gen APMcompany• Scalability for theNetfliximplementation
  30. 30. 31ScaleBase DeploymentEnvironments– Public Cloud– AWS, Rackspace, any– Private cloud– Hosted / on-premiseDatabases Supported– MySQL 5.1, 5.5, 5.6 (undercertification)– AWS RDS MySQL 5.1, 5.5– Maria DB 10.0 (undercertification)Path to Scale-Out:1. Data DistributionPolicy Analysis2. Functional Test3. Load Test4. Production Migration(safe, online)
  31. 31. 32SummaryScaleBase provides cost-effective Scale-Out solutions• Scale to an infinite number of users, data and transactions• Improve performance• No application rewrites• Real-time elasticity• ACID Compliant• Expert analysis and simple deployment• Leverage existing MySQL ecosystem/skills• Improve database visibility with real-time monitoring• Simplified, centralized administration
  32. 32. 33Questions (please enter directly into the GTW side panel)paul.campaniello@scalebase.comdoron.levari@scalebase.comwww.ScaleBase.com617.630.2800Additional Resourceshttp://www.scalebase.com/blog/http://www.scalebase.com/resources/@scalebase
  33. 33. 34Thank You
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×