Building RightScale's Globally Distributed Datastore - RightScale Compute 2013
 

Like this? Share it with your network

Share

Building RightScale's Globally Distributed Datastore - RightScale Compute 2013

on

  • 424 views

Speaker: Josep Blanquer - Chief Architect, RightScale ...

Speaker: Josep Blanquer - Chief Architect, RightScale

Is your database holding back your application? Find out how RightScale uses NoSQL databases such as Cassandra to provide a scalable, distributed, and highly available service around the world that is designed to recover from failures of a whole cloud region.

Statistics

Views

Total Views
424
Views on SlideShare
424
Embed Views
0

Actions

Likes
0
Downloads
7
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Building RightScale's Globally Distributed Datastore - RightScale Compute 2013 Presentation Transcript

  • 1. april25-26sanfranciscocloud success starts hereBuilding RightScale’s GloballyDistributed DatastoreJosep M. Blanquer, Chief Architect
  • 2. #2#2#RightscaleComputeIn this talk…• Intro• Data Taxonomy• Data Storage Design• Scale, HA and DR considerations• Conclusion
  • 3. #3#3#RightscaleComputeIntro: Expectations and scopeWhat this is and what is not• IS a talk about:• how RightScale has designed and implemented its backing datastores• …for a few of the most representative internal systems• …with the rationale behind it• Is NOT a talk about• RightScale’s overall architecture• Nodes or hosts, it’s about Systems• RightScale’s data modeling
  • 4. #4#4#RightscaleComputeIntro: Tools and Technologies• RightScale uses a mix of RDBMS and NoSQL technologies:• MySQL , Cassandra and S3 (for backups and archiving)• Transactionality:• MySQL: strong ACID properties• Cassandra: no Atomicity, eventually Consistent, some Isolation, Durable• Availability:• MySQL: async replication. Master-SlaveN or Master-Master• Cassandra: Distributed, master-less, highly-replicated (multi-DC)• Queryability:• MySQL: Extremely flexible at adding indexes and changing data model• Cassandra: More difficult to change the querying patterns
  • 5. #5#5#RightscaleComputeTaxonomy of RightScale’s DataRepresentative systemswith different data semantics:Global Objects Marketplace AssetsDashboard Objects Audits Tags Recent EventsCloud Polling DataRouting DataMonitoring/Syslog
  • 6. #6#6#RightscaleComputeTaxonomy of RightScale’s DataRepresentative systemswith different data semantics:Global Objects Marketplace AssetsDashboard Objects Audits Tags Recent EventsCloud Polling DataRouting DataMonitoring/SyslogCommon across accounts: Users Plans Settings MultiCloud Marketplace: Published Assets Sharing Groups …
  • 7. #7#7#RightscaleComputeTaxonomy of RightScale’s DataRepresentative systemswith different data semantics:Global Objects Marketplace AssetsDashboard Objects Audits Tags Recent EventsCloud Polling DataRouting DataMonitoring/SyslogPrivate to each account: Deployments Imported assets Alert Specifications Server Inputs Audit Tags User Events …
  • 8. #8#8#RightscaleComputeTaxonomy of RightScale’s DataRepresentative systemswith different data semantics:Global Objects Marketplace AssetsDashboard Objects Audits Tags Recent EventsCloud Polling DataRouting DataMonitoring/SyslogPrivate to each account: Cloud resource states (cache) Cloud credentials
  • 9. #9#9#RightscaleComputeTaxonomy of RightScale’s DataRepresentative systemswith different data semantics:Global Objects Marketplace AssetsDashboard Objects Audits Tags Recent EventsCloud Polling DataRouting DataMonitoring/SyslogPrivate to each account: Instance agents location Core agents location Agent action registry …
  • 10. #10#10#RightscaleComputeTaxonomy of RightScale’s DataRepresentative systemswith different data semantics:Global Objects Marketplace AssetsDashboard Objects Audits Tags Recent EventsCloud Polling DataRouting DataMonitoring/SyslogPrivate to each account: Collected metric data Collected syslog data …
  • 11. #11#11#RightscaleComputeTaxonomy of RightScale’s DataUsersInstancesGlobal Objects Marketplace AssetsDashboard Objects Audits Tags Recent EventsCloud Polling DataRouting DataMonitoring/SyslogWho uses the data?• Users through the Dash/API• Instances from the CloudData close to the UsersData close to the CloudData Placement
  • 12. #12#12#RightscaleComputeTaxonomy of RightScale’s DataX-acctAccountGlobal Objects Marketplace AssetsDashboard Objects Audits Tags Recent EventsCloud Polling DataRouting DataMonitoring/SyslogWhich data do we need?• Data for all accounts• Data for a single accountData shared between accountsData required within scopeof a single accountData scope and containment
  • 13. #13#13Talk with the Experts.UsersTaxonomy of RightScale’s DataInstancesX-acctAccountGlobal Objects Marketplace AssetsDashboard Objects Audits Tags Recent EventsCloud Polling DataRouting DataMonitoring/SyslogWho uses the data? Proximity to User vs. CloudWhich data do we need? Scope of data availableClose to cloud resourcesAccount-shardable* dataClose to userAccount-shardable dataClose to userGlobally accessible data
  • 14. #14#14#RightscaleComputeUsersInstancesAccountX-Account
  • 15. #15#15#RightscaleComputeUsersInstancesglobalX-AccountCustom replicationWhy custom? More control• Multiple sources• Individual columns• Apply transformations• Smart re-sync featuresGlobal: MySQL• ACID semantics• Master-Slave replication
  • 16. #16#16#RightscaleComputeUsersInstancesAccountglobal dashS3eventstagsauditX-AccountDashboard: MySQL• ACID semantics• Master-SlaveN replication• Slave reads• Rows tagged by accountOther systems: Cassandra• Simpler Key-Value access• Great scalability• Great replica control• High write availability• Time-to-live expiration as cache• Rows tagged by accountData archive: S3• Low read rate• Globally accessible
  • 17. #17#17#RightscaleComputeUsersInstancesAccountglobal dashS3eventstagsauditX-AccountdasheventstagsauditSo we can horizontally scale ourdashboard by partitioning objectsbased on account groups:Clusters
  • 18. #18#18#RightscaleComputeUsersAccountCluster1dashS3eventstagsauditClusterNdashS3eventstagsauditAccount Set 1 Account Set 2RightScale AccountsCluster3dashS3eventstagsaudit…Features:• 1 cluster: N accounts• 1 account: 1 home• Migratable accountsBenefits:• Great horizontal growth• Better failure isolation• Independent scale• Load rebalancing• Versionable code• Differentiated service
  • 19. #19#19#RightscaleComputedasheventstagsauditUsersInstancesAccountglobal dashS3eventstagsauditroutingpollingmonitorX-Account
  • 20. #20#20#RightscaleComputeroutingpollingmonitorroutingpollingmonitorUsersInstancesAccountglobal dashS3eventstagsauditX-AccountAnd partition our cloud objects based on the cloudthe instances of an account run on:Islands
  • 21. #21#21#RightscaleComputeAccountInstancesServices co-locatedwith resourcesServices co-locatedwith resourcesServices co-locatedwith resourcesroutingpollingmonitorIsland1Island2IslandNroutingpollingmonitorroutingpollingmonitorCloud 1 Cloud 2 Cloud N
  • 22. #22#22#RightscaleComputeAccountInstancesFeatures:• 1 instance: 1 home island• 1 Island can serve N clouds• Core Agents: global dataBenefits:• Close to cloud resources• Good failure isolation• As good as cloud • Good scale: global replicasacross cassandra DCsroutingpollingmonitorIsland1Island2IslandNroutingpollingmonitorroutingpollingmonitorroutingpollingmonitorroutingpollingmonitorroutingpollingmonitorIsland1Island2IslandNPolling Clouds: MySQL• Master-Slave replication• Can port to NoSQL easily• Mostly a resource cache• But cloud partitionableMonitoring: Custom• Replicated files• Backup to S3• Archive to S3Routing: Cassandra• Simpler Key-Value access• Very high availability• Great scalability• Great replica control• Plus cross DC replication*
  • 23. #23#23#RightscaleComputeUsersAccountCluster1dashS3eventstagsauditClusterNdashS3eventstagsauditCluster3dashS3eventstagsaudit…routingpollingmonitorroutingpollingmonitorroutingpollingmonitorIsland1Island2IslandNInstancesDifferent GeographiesDifferent CloudsWhat if the cloudwhere the clusteris deployed on…Fails?What if the cloudwhere the islandis deployed on…Fails?
  • 24. #24#24#RightscaleComputeUsersAccountCluster1dashS3eventstagsauditClusterNdashS3eventstagsauditCluster3dashS3eventstagsaudit…routingpollingmonitorroutingpollingmonitorroutingpollingmonitorIsland1Island2IslandNInstancesSister ClustersFull replicaFeatures:• Each master has an extra remote slave• Each cluster in a pair is a DC replica of the other’slocalringAt Disaster Recovery time:• Apps are told to start serving an extra shard• No need to provision more infrastructure to recover(try to avoid since everybody is on the same boat)• New resources can be allocated over time to helpoffload existing ones
  • 25. #25#25#RightscaleComputeConclusions• Shown that RightScale uses multiple database technologies:• RDBMS – MySQL for the ACID semantics and ‘queryability’• Using a Master to N-Slaves for RO scale, and quick failure recovery• And ReadOnly Provisioning – To increase RO availability and scale remote systems• NoSQL: Cassandra for Availability and Scalability• for higher Read/Write availability within a cluster• For fully replicated regions across the globe (for Read/Write!)• Shown how RightScale uses them in different techniques• It partitions resource data into Islands based on cloud proximity• Can achieve in-cloud polling,and keep monitoring/syslog data storage next to instances• Can provide routing availability, colocated with instances for any world region• It partitions core data into Clusters based on account groups• To scale the core horizontally, and independently and achieve account isolation/differentiation• Enhances fault isolation: Assigning accounts to Clusters deployed away their cloud resources• It maintains cluster pairs (sister sites)• To recover from full cloud region failures• It doesn’t require massive amounts of new resources to recover
  • 26. april25-26sanfranciscocloud success starts hereQuestions?