How RightScale Architects            Its Databases(for Worldwide Scale, HA and DR Scenarios)January 30, 2013              ...
2#Your Panel TodayPresenting• Rafael H. Saavedra, VP Engineering, RightScale• Josep Blanquer, Chief Architect, RightScaleQ...
3#       Menu        Intro  Data TaxonomyData Storage Design   Scale, HA and DR    Conclusion                      #rights...
4#Intro: Expectations and scope                  What this is and what is not• IS a talk about:    • how RightScale has de...
5#Intro: Tools and Technologies• RightScale uses a mix of RDBMS and NoSQL technologies:   • MySQL , Cassandra and S3 (for ...
6#Glossary: Examples we will use    Marketplace Assets                             Configuration data objects that are    ...
7#Taxonomy of RightScale’s Data    Representative systems with different data semantics:      Global Objects          Mar...
8#Taxonomy of RightScale’s Data    Representative systems with different data semantics:      Global Objects          Mar...
9#Taxonomy of RightScale’s Data    Representative systems with different data semantics:      Global Objects          Mar...
10#Taxonomy of RightScale’s Data    Representative systems with different data semantics:      Global Objects          Ma...
11#Taxonomy of RightScale’s Data    Representative systems with different data semantics:      Global Objects          Ma...
12#Taxonomy of RightScale’s Data    Representative systems with different data semantics:      Global Objects          Ma...
13#Taxonomy of RightScale’s Data                                      Which data do we need?            Global Objects  X-...
14#Taxonomy of RightScale’s Data                                     Who uses the data?            Global Objects         ...
15#Taxonomy of RightScale’s Data                      Who uses the data? Proximity to User vs. Cloud                      ...
16#            X-Account   AccountUsersInstances                                  #rightscale
17#            X-Account                                                   Why custom? More control                       ...
18#            X-Account              Account                                                                        Data ...
19#            X-Account          Account                                                  S3                             ...
20#                           Account                                              S3                                     ...
21#            X-Account              Account                                                       S3                    ...
22#            X-Account            Account                                                       S3                      ...
23#                                        Account                              polling                            polling...
24#                                    Account                                                           S3               ...
25#                                    Account                                                    Sister Clusters         ...
26#Conclusions• Shown that RightScale uses multiple database technologies   • RDBMS – MySQL for the ACID semantics and ‘qu...
27#Next Steps                                       Contact RightScale                                                    ...
Upcoming SlideShare
Loading in …5
×

RightScale Webinar: How RightScale Architects Its Databases (for Worldwide Scale, HA and DR Scenarios)

919 views

Published on

Is your database holding back your application? Find out how we at RightScale use SQL and NoSQL databases such as MySQL and Cassandra to provide a scalable, distributed, and highly available service around the world, that is designed to recover from failures of a whole cloud region.

In this webinar, we will:

- Share the data taxonomy for specific RightScale systems
- Give you insights on how to think through your own data taxonomy
- Go deep into RightScale's distributed database architecture

Join RightScale's VP of Engineering and Chief Architect and learn directly from the team who architected RightScale's databases for scale, HA and DR.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
919
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
35
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Telcos built point-2-point networks for their customers
  • RightScale Webinar: How RightScale Architects Its Databases (for Worldwide Scale, HA and DR Scenarios)

    1. 1. How RightScale Architects Its Databases(for Worldwide Scale, HA and DR Scenarios)January 30, 2013 Watch the recording of this webinar #rightscale
    2. 2. 2#Your Panel TodayPresenting• Rafael H. Saavedra, VP Engineering, RightScale• Josep Blanquer, Chief Architect, RightScaleQ&A• Jared Marcell, Account Manager, RightScale• David Manriquez, Account Manager, RightScalePlease use the “Questions” window to ask questions any time! #rightscale
    3. 3. 3# Menu Intro Data TaxonomyData Storage Design Scale, HA and DR Conclusion #rightscale
    4. 4. 4#Intro: Expectations and scope What this is and what is not• IS a talk about: • how RightScale has designed and implemented its backing datastores • …for a few of the most representative internal systems • …with the rationale behind it• Is NOT a talk about • RightScale’s overall architecture • Nodes or hosts, it’s about Systems • RightScale’s data modeling Note: Most of the design is implemented and in production but some of the most advanced things that are still in beta, or are still being worked on #rightscale
    5. 5. 5#Intro: Tools and Technologies• RightScale uses a mix of RDBMS and NoSQL technologies: • MySQL , Cassandra and S3 (for backups and archiving)• Transactionality: • MySQL: strong ACID properties • Cassandra: no Atomicity, eventually Consistent, some Isolation, Durable• Availability: • MySQL: async replication. Master-SlaveN or Master-Master • Cassandra: Distributed, master-less, highly-replicated (multi-DC)• Sharding: • MySQL: no explicit inter-node tools. (Sharding done by application) • Cassandra: partitions data internally across nodes. #rightscale
    6. 6. 6#Glossary: Examples we will use Marketplace Assets Configuration data objects that are RightScripts user-generated, private or shared ServerTemplates Resource data that drives automation and Tags reporting Data used to communicate recent events and Events news feeds to users Data that records actions and states of external Cloud Polling and Gateway API-linked services Data used to locate and transport messages Routing across instances and/or our services Infrastructure monitoring data recorded and Monitoring presented on behalf of users #rightscale
    7. 7. 7#Taxonomy of RightScale’s Data Representative systems with different data semantics: Global Objects  Marketplace Assets Dashboard Objects  Audits  Tags  Recent Events Cloud Polling Data Routing Data Monitoring/Syslog #rightscale
    8. 8. 8#Taxonomy of RightScale’s Data Representative systems with different data semantics: Global Objects  Marketplace Assets Common across accounts:  Users Dashboard Objects  Account Plans  Audits  Settings  MultiCloud Marketplace:  Tags  Published Assets  Recent Events  Sharing Groups  … Cloud Polling Data Routing Data Monitoring/Syslog #rightscale
    9. 9. 9#Taxonomy of RightScale’s Data Representative systems with different data semantics: Global Objects  Marketplace Assets Dashboard Objects  Audits Private to each account:  Tags  Deployments  Imported assets  Recent Events  Alert Specifications  Server Inputs Cloud Polling Data  Audit Routing Data  Tags  User Events Monitoring/Syslog  … #rightscale
    10. 10. 10#Taxonomy of RightScale’s Data Representative systems with different data semantics: Global Objects  Marketplace Assets Dashboard Objects  Audits Private to each account:  Tags  Cloud resource states (cache)  Cloud credentials  Recent Events Cloud Polling Data Routing Data Monitoring/Syslog #rightscale
    11. 11. 11#Taxonomy of RightScale’s Data Representative systems with different data semantics: Global Objects  Marketplace Assets Dashboard Objects  Audits Private to each account:  Tags  Instance agents location  Recent Events  Core agents location  Agent action registry Cloud Polling Data  … Routing Data Monitoring/Syslog #rightscale
    12. 12. 12#Taxonomy of RightScale’s Data Representative systems with different data semantics: Global Objects  Marketplace Assets Dashboard Objects  Audits  Tags  Recent Events Private to each account: Cloud Polling Data  Collected metric data  Collected syslog data Routing Data  … Monitoring/Syslog #rightscale
    13. 13. 13#Taxonomy of RightScale’s Data Which data do we need? Global Objects X-acct • Data for all accounts  Marketplace Assets • Data for a single account Dashboard Objects Data shared between accounts  Audits  Tags Data scope and containment Account  Recent Events Cloud Polling Data Data required within scope of a single account Routing Data Monitoring/Syslog #rightscale
    14. 14. 14#Taxonomy of RightScale’s Data Who uses the data? Global Objects • Users through the Dash/API  Marketplace Assets • Instances from the CloudUsers Dashboard Objects  Audits Data close to the Users  Tags  Recent Events Data Placement Cloud Polling DataInstances Routing Data Data close to the Cloud Monitoring/Syslog #rightscale
    15. 15. 15#Taxonomy of RightScale’s Data Who uses the data? Proximity to User vs. Cloud Which data do we need? Scope of data available Global Objects X-acct Close to user  Marketplace Assets Globally accessible dataUsers Dashboard Objects  Audits Close to user  Tags Account-shardable data Account  Recent Events Cloud Polling DataInstances Close to cloud resources Routing Data Account-shardable* data Monitoring/Syslog #rightscale
    16. 16. 16# X-Account AccountUsersInstances #rightscale
    17. 17. 17# X-Account Why custom? More control • Multiple sourcesUsers global • Individual columns Custom replication • Apply transformations • Smart re-sync features Global: MySQL • ACID semantics • Master-Slave replicationInstances #rightscale
    18. 18. 18# X-Account Account Data archive: S3 S3 • Low read rate tags • Globally accessibleUsers global dash audit Other systems: Cassandra events • Simpler Key-Value access • Great scalability • Great replica control • High write availability Dashboard: MySQL • Time-to-live expiration as cache • ACID semantics • Rows tagged by account • Master-SlaveN replication • Slave readsInstances • Rows tagged by account #rightscale
    19. 19. 19# X-Account Account S3 tags tagsUsers global dash audit dash audit events events So we can horizontally scale our dashboard by partitioning objects based on account groups: ClustersInstances #rightscale
    20. 20. 20# Account S3 S3 S3 tags tags tags Cluster 1 Cluster 3 Cluster dash audit dash audit … dash audit N event event event s s s Features:Users • 1 cluster: N accounts • 1 account: 1 home RightScale Accounts • Migratable accounts Benefits: • Great horizontal growth Account Set 2 • Better failure isolation Account Set 1 • Independent scale • Load rebalancing • Versionable code • Differentiated service #rightscale
    21. 21. 21# X-Account Account S3 tags tagsUsers global dash audit dash audit events events pollingInstances monitor routing #rightscale
    22. 22. 22# X-Account Account S3 tags dash And partition our cloud objects based on the cloudUsers global audit the instances of an account run on: events Islands polling pollingInstances monitor monitor routing routing #rightscale
    23. 23. 23# Account polling polling polling polling Polling Clouds: MySQL Monitoring: Custom • Master-Slave replication • Replicated files Island N Island 1 Island 2 • Can port to NoSQL easily monitor monitor monitor • Backup to S3 monitor • Mostly a resource cache • Archive to S3 • But cloud partitionable routing routing routing routingInstances Features: • 1 instance: 1 home island • 1 Island can serve N clouds • Core Agents: global data Routing: Cassandra • Simpler Key-Value access Benefits: • Very high availability • Close to cloud resources Services co-located • Services co-located Great scalability • Good failure isolation Services co-located with resources • with resources Great replica control • As good resources with as cloud  • Plus cross DC replication* • Good scale: global replicas across Cassandra DCs Cloud 1 Cloud 2 Cloud N #rightscale
    24. 24. 24# Account S3 S3 S3 tags tags tags Cluster 1 Cluster 3Users Cluster dash audit dash audit … dash audit N event event event s s s Differentif the cloud What Geographies where the cluster is deployed on… polling Fails? polling pollingInstances Island N Island 1 Island 2 monitor monitor monitor routing routing routing Different Clouds #rightscale
    25. 25. 25# Account Sister Clusters S3 S3 S3 tags tags tags Cluster 1 Cluster 3Users Cluster audit Full replica dash dash audit … dash audit N event event event s s s Features: • Each master has an extra remote slave • Each cluster in a pair is a DC replica of the other’s polling polling pollingInstances localring Island N Island 1 Island 2 monitor At Disaster Recovery time: monitor monitor • Apps are told to start serving an extra shard routing • No need to provision more infrastructure to recover routing routing (try to avoid since everybody is on the same boat) • New resources can be allocated over time to help offload existing ones #rightscale
    26. 26. 26#Conclusions• Shown that RightScale uses multiple database technologies • RDBMS – MySQL for the ACID semantics and ‘queryability’ • Using a Master to N-Slaves for RO scale, and quick failure recovery • And ReadOnly Provisioning – To increase RO availability and scale remote systems • NoSQL: Cassandra for Availability and Scalability • for higher Read/Write availability within a cluster • For fully replicated regions across the globe (for Read/Write!)• Shown how RightScale uses them in different techniques • It partitions resource data into Islands based on cloud proximity • Can achieve in-cloud polling,and keep monitoring/syslog data storage next to instances • Can provide routing availability, colocated with instances for any world region • It partitions core data into Clusters based on account groups • To scale the core horizontally, and independently and achieve account isolation/differentiation • Enhances fault isolation: Assigning accounts to Clusters deployed away their cloud resources • It maintains cluster pairs (sister sites) • To recover from full cloud region failures • It doesn’t require massive amounts of new resources to recover #rightscale
    27. 27. 27#Next Steps Contact RightScale (866) 720-02081. Learn: Building Scalable Applications sales@rightscale.com in the Cloud Whitepaper www.rightscale.com www.rightscale.com/whitepapers2. Analyze: Deployment review of your environment The next big RightScale Community Event! www.rightscale.com/contact April 25-26 in San Francisco www.RightScaleCompute.com3. Try: Free Edition •Attend technical breakout sessions www.rightscale.com/free •Get RightScale training •Talk with RightScale customers •Ask questions at the Expert Bar #rightscale

    ×