Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
A Global In-memory Data System for MySQLDaniel Austin, PayPal Technical StaffPercona Live!London, Oct. 25, 2011   v1.1
AGENDA               Intro: Globalizing NDB              Proposed Architecture                    What We Learned         ...
Our Mission “UserBase: Develop a globally distributed DB For User-related data” •   Must Not Fail (99.999%) •   Must Not L...
THE FUNDAMENTAL PROBLEM INDISTRIBUTED DATA SYSTEMS“How Do We Manage ReliableDistribution of Data Across GeographicalDistan...
To SQL or Not to SQL,’Tis the Query • Modern NoSQL Systems as solutions   – Hive, Cassandra, Mongo, 120+ more   – Mostly d...
SYSTEM AVAILABILITY DEFINED• Availability of the entire  system:            n   mAsys = 1 – P(1-Pri)j           i=1 j=1   ...
What about “High Performance”?  • Maximum lightspeed distance on Earth’s    Surface: ~67 ms  • Target: data available worl...
Intro: Globalizing NDB       Proposed Architecture                What We Learned                         Q&AGlobal In-mem...
WHY NDB?            Pro                       Con•   True HA by design     •   Some semantic     – Fast recovery          ...
How NDB Works in One SlideGraphics courtesy dev.mysql.com                                  Confidential and Proprietary
CIRCULAR REPLICATION/FAILOVERGraphics courtesy O’Reilly OnLamp.com                                        Confidential and...
AWS Meets NDB• Why AWS?  – Cheap and easy infrastructure-in-a-box     (Or so we thought! Ha!)• Services Used:  – EC2 (Cent...
ARCHITECTURAL TILES                                  AWS Availability ZonesTiling Rules                                  A...
Architecture StackScale by Tiling                  A   B         A   B           A      B                   A   B         ...
Other Technologies Considered • Paxos   – Elegant-but-complex consensus-based     messaging protocol   – Used in Google Me...
Intro: Globalizing NDB          Proposed Architecture             What We Learned                         Q&AGlobal In-mem...
SYSTEM READ/WRITE PERFORMANCE (!) What we tested: • 32 & 256 byte char fields                                             ...
Data Models and Query Optimization for NDB  • Network Latency is an obvious issue  • Data model requires all segments pres...
Commit Ordering • Why does commit ordering matter? • Write operators are non-commutative    [W(d,t1),W(d,t2)] != 0 unless ...
Dark Side of AWS • Deploying NDB at scale on AWS is hard   – Dynamic IPs (use hostfile)   – DNS issues   – Security groups...
NOTES ON GLOBAL LOAD BALANCING• Don‟t try this at home   – Rent or buy a      solution   – BGP-based solutions      are be...
Hard Lessons, Shared • Be Careful…    – With “Eventual Consistency”-related concepts    – ACID, CAP are not really as well...
Summing Up on v0.7• It works!• Very fast, very reliable• Very complicated!• AWS poses challenges that private data  center...
Future Directions • Alternate solution using Pacemaker,   Heartbeat   – From Yves Trudeau @ Percona   – Uses InnoDB, not N...
“In the long run, we are all deadeventually consistent.”Maynard Keynes on NoSQL DatabasesTwitter: @daniel_b_austinEmai: da...
Upcoming SlideShare
Loading in …5
×

A Global In-memory Data System for MySQL

2,841 views

Published on

This is my presentation from Percona Live! London last year. I describe my Big Data system built on MySQL/NDB and show that we can preserve the relational model for most Big Data purposes in the real world.

Published in: Technology
  • Be the first to comment

A Global In-memory Data System for MySQL

  1. 1. A Global In-memory Data System for MySQLDaniel Austin, PayPal Technical StaffPercona Live!London, Oct. 25, 2011 v1.1
  2. 2. AGENDA Intro: Globalizing NDB Proposed Architecture What We Learned Q&A Global In-memory MySQL Confidential and Proprietary 2
  3. 3. Our Mission “UserBase: Develop a globally distributed DB For User-related data” • Must Not Fail (99.999%) • Must Not Lose Data. Period. • Must Support Transactions • Must Be FAST • Must Support (some) SQL • May Be Buzzword-compliant (RFC 2119) Confidential and Proprietary
  4. 4. THE FUNDAMENTAL PROBLEM INDISTRIBUTED DATA SYSTEMS“How Do We Manage ReliableDistribution of Data Across GeographicalDistances?” Confidential and Proprietary
  5. 5. To SQL or Not to SQL,’Tis the Query • Modern NoSQL Systems as solutions – Hive, Cassandra, Mongo, 120+ more – Mostly designed for „Big Data‟ problems – Low levels of maturity • Trade-offs: – Consistency vs. Availability, Fault Tolerance (dreaded CAP theorem) – Relational Model & X-actions dropped Confidential and Proprietary
  6. 6. SYSTEM AVAILABILITY DEFINED• Availability of the entire system: n mAsys = 1 – P(1-Pri)j i=1 j=1 V I P• Number of Parallel Components Needed to Parallel Achieve Availability Amin: SerialNmin =[ln(1-Amin)/ln(1-r)] Confidential and Proprietary
  7. 7. What about “High Performance”? • Maximum lightspeed distance on Earth’s Surface: ~67 ms • Target: data available worldwide in < 1000 ms Sound Easy? Think Again! Confidential and Proprietary
  8. 8. Intro: Globalizing NDB Proposed Architecture What We Learned Q&AGlobal In-memory MySQL Confidential and Proprietary 8
  9. 9. WHY NDB? Pro Con• True HA by design • Some semantic – Fast recovery limitations on fields• Supports (some) X- • Size constraints (2 actions TB?)• Relational Model – Hardware limits• In-memory also architecture = high • Higher cost/byte performance • Requires reasonable• Disk storage for data partitioning non-indexed data • Higher complexity (since 5.1)• APIs, APIs, APIs Confidential and Proprietary
  10. 10. How NDB Works in One SlideGraphics courtesy dev.mysql.com Confidential and Proprietary
  11. 11. CIRCULAR REPLICATION/FAILOVERGraphics courtesy O’Reilly OnLamp.com Confidential and Proprietary
  12. 12. AWS Meets NDB• Why AWS? – Cheap and easy infrastructure-in-a-box (Or so we thought! Ha!)• Services Used: – EC2 (Centos 5.3, small instances for mgm & query nodes, XL for data – Elastic IPs/ELB – EBS Volumes – S3 – Cloudwatch Confidential and Proprietary
  13. 13. ARCHITECTURAL TILES AWS Availability ZonesTiling Rules A B• Never separate NDB & SQL• Ndb:2-SQL:1-MGM:1• Scale by adding more tiles• Failover 1st to nearest AZ• Then to nearest DC• At least 1 replica/AZ C ELB• Don‟t share nodes• Mgmt nodes are redundantLimitations Unused (not present in all locations)• AWS is network-bound @ 250 MBPS – ouch!• Need specific ACL across AZ boundaries• AZs not uniform!• No GSLB NDB MGM SQL• Dynamic IPs• ELB sticky sessions !reliable Confidential and Proprietary
  14. 14. Architecture StackScale by Tiling A B A B A B A B A B A B A B 5 AWS Data Centers: US-E, US-W, TK, EU, AS Confidential and Proprietary
  15. 15. Other Technologies Considered • Paxos – Elegant-but-complex consensus-based messaging protocol – Used in Google Megastore, Bing metadata • Java Query Caching – Queries as serialized objects – Not yet working • Multiple Ring Architectures – Even more complicated = no way Confidential and Proprietary
  16. 16. Intro: Globalizing NDB Proposed Architecture What We Learned Q&AGlobal In-memory MySQL Confidential and Proprietary 16
  17. 17. SYSTEM READ/WRITE PERFORMANCE (!) What we tested: • 32 & 256 byte char fields In-region replication tests • Reads, writes, query speed vs. volume • Data replication speeds Results: • Global replication < 350 ms • 256 byte read < 10ms worldwide 06/19/2011 06/20/2011 06/21/2011 06/22/2011 06/23/2011 Confidential and Proprietary
  18. 18. Data Models and Query Optimization for NDB • Network Latency is an obvious issue • Data model requires all segments present in each geo-region • Parameterized (Linked) Joins – SPJ technique from Clustra (see Clement Frazer‟s blog for details) Confidential and Proprietary
  19. 19. Commit Ordering • Why does commit ordering matter? • Write operators are non-commutative [W(d,t1),W(d,t2)] != 0 unless t1=t2 – Can lead to inconsistency – Can lead to timestamp corruption – Forcing sequential writes defeats Amdahl‟s rul • Can show up in GSLB scenarios Confidential and Proprietary
  20. 20. Dark Side of AWS • Deploying NDB at scale on AWS is hard – Dynamic IPs (use hostfile) – DNS issues – Security groups (ec2-authorize) – Inconsistent EC2 deployments • Are availability zones independent network segments? – No GSLB (!) (rent or buy) Be Prepared to struggle a bit! Confidential and Proprietary
  21. 21. NOTES ON GLOBAL LOAD BALANCING• Don‟t try this at home – Rent or buy a solution – BGP-based solutions are best• Deployment and config for AWS are tough• We tried both Zeus and Dyn – Liked Dyn better, $$ & ease of use• Absolutely crucial part of infrastructure! Graphics courtesy http://www.oes.co.th Confidential and Proprietary
  22. 22. Hard Lessons, Shared • Be Careful… – With “Eventual Consistency”-related concepts – ACID, CAP are not really as well-defined as we‟d like considering how often we invoke them • NDB is a good solution – Real HA, real SQL – Notable limitations around fields, datatypes – Successfully competes with NoSQL systems for most use cases – better in many cases • NoSQL Systems – All have relatively low levels of maturity – More suitable for simple key-value models – Better for very high volumes Confidential and Proprietary
  23. 23. Summing Up on v0.7• It works!• Very fast, very reliable• Very complicated!• AWS poses challenges that private data centers may not experience• You can achieve high performance and availability without giving up relational models and read consistency! Confidential and Proprietary
  24. 24. Future Directions • Alternate solution using Pacemaker, Heartbeat – From Yves Trudeau @ Percona – Uses InnoDB, not NDB • Implement Memcached plugin – To test NoSQL functionality, APIs • Add simple connection-based persistence to preserve connections during failover • Better data node distribution • Better testing & monitoring Confidential and Proprietary
  25. 25. “In the long run, we are all deadeventually consistent.”Maynard Keynes on NoSQL DatabasesTwitter: @daniel_b_austinEmai: daaustin@paypal.comWith apologies and thanks to the real DB experts, Andrew Goodman, YvesTrudeau, Clement Frazer, Daniel Abadi, and everyone else!

×