Continuent Tungsten - Scalable Saa S Data Management


Published on

The key needs of SaaS vendors include:
i) managing multi-tenant architectures with shared DBMS, ii) maintaining customer SLAs for uptime and performance and iii) optimized, efficient operations.

The key benefits Continuent Tungsten offers SaaS vendors are:
i) high availability and protection from data loss, ii) simple, efficient cluster management and iii) enable complex database topologies.

Tungsten offers high-availability, database cluster management and management of complex topologies for multi-tenant architectures.

Tungsten high availability and data protection features include maintaining live copies with data consistency checking and tightly coupled backup/restore integration with cluster management tools.

Tungsten cluster management allows SaaS vendors to migrate customers and perform system upgrades without downtime, thus enabling these maintenance operations during normal business hours.

Tungsten also enables complex replication topologies, including data filtering and data archiving strategies, maintaining extra data copies for data-marts, routing different customers to different DBMS copies, and providing cross-site multi-master replication.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Continuent Tungsten - Scalable Saa S Data Management

  1. 1. Scalable SaaS Data Management with Tungsten Robert Hodges CTO, Continuent, Inc.
  2. 2. Agenda <ul><li>Introductions </li></ul><ul><li>What’s So Hard about SaaS Data Management? </li></ul><ul><li>Introducing Tungsten Clustering </li></ul><ul><li>Implementing SaaS Data Management with Tungsten </li></ul><ul><li>Commercial Support for SaaS from Continuent </li></ul><ul><li>Definition: </li></ul><ul><li>SaaS = “Software as a Service” </li></ul>
  3. 3. About Continuent <ul><li>Our Business: Full service database management for SaaS applications </li></ul><ul><li>Our Value: </li></ul><ul><ul><li>Ensure data are available when and where you need them </li></ul></ul><ul><ul><li>TCO less than 20% of comparable solutions </li></ul></ul><ul><li>Our Technical Expertise </li></ul><ul><ul><li>Database replication </li></ul></ul><ul><ul><li>Cluster management </li></ul></ul><ul><ul><li>Application connectivity </li></ul></ul><ul><ul><li>SaaS applications </li></ul></ul>
  4. 4. What’s So Hard about SaaS Data Management?
  5. 5. The SaaS Business Problem <ul><li>Meet customer SLAs </li></ul><ul><ul><li>Keep applications available </li></ul></ul><ul><ul><li>Don’t lose or confuse customer data </li></ul></ul><ul><ul><li>Meet response time requirements </li></ul></ul><ul><li>Enable growth </li></ul><ul><ul><li>More customers </li></ul></ul><ul><ul><li>More volume from existing customers </li></ul></ul><ul><ul><li>New service offerings </li></ul></ul><ul><li>Raise bottom-line profitability </li></ul><ul><ul><li>Keep costs of operation as low as possible </li></ul></ul><ul><ul><li>Enhance add-on revenue </li></ul></ul>
  6. 6. Successful SaaS Systems Are Complex <ul><li>SaaS databases quickly evolve into “big league” topologies </li></ul>Tenant Data Tenant Data Shared Data Tenant Apps Tenant Apps Shared Data Tenant Analytics Tenant Analytics Tenant and Shared Data
  7. 7. Core Technical Issue: Multi-Tenancy <ul><li>Defining a flexible multi-tenant data architecture </li></ul><ul><li>Implementing easy-to-manage clusters </li></ul><ul><li>Scaling performance on shared databases </li></ul><ul><li>Protecting tenant data from loss or corruption </li></ul><ul><li>Handling failures efficiently </li></ul><ul><li>Upgrades and maintenance windows </li></ul><ul><li>And do it on economical open source databases! </li></ul>
  8. 8. Introducing Tungsten Clustering
  9. 9. What Is Tungsten? <ul><li>Tungsten implements master/slave clusters to: </li></ul><ul><ul><li>Protect data </li></ul></ul><ul><ul><li>Maintain high availability </li></ul></ul><ul><ul><li>Improve resource utilization </li></ul></ul><ul><ul><li>Raise performance </li></ul></ul><ul><li>Install and set up in a few minutes </li></ul><ul><li>Integrated backup/restore and data integrity checks </li></ul><ul><li>Rule-driven management with automated failover </li></ul><ul><li>Simple procedures for updates and maintenance </li></ul><ul><li>SQL scaling for multi-tenant systems </li></ul>
  10. 10. What’s Inside Tungsten? <ul><li>Replication - Making copies </li></ul><ul><ul><li>Tungsten Replicator -- Database-neutral, platform independent master/slave replication </li></ul></ul><ul><li>Connectivity -- Finding databases </li></ul><ul><ul><li>Tungsten Connector -- Fast MySQL/PostgreSQL client to JDBC proxying </li></ul></ul><ul><ul><li>Tungsten SQL Router --JDBC wrapper for high-performance and transparent failover, load-balancing, and partitioning (no proxy required) </li></ul></ul><ul><li>Management -- Administering the database </li></ul><ul><ul><li>Tungsten Manager -- Distributed administration with autonomic, rule-based configuration and no single point of failure </li></ul></ul><ul><ul><li>Tungsten Monitor -- Track resource status and </li></ul></ul>
  11. 11. Tungsten Clustering In Action Master DB Slave DB Master Host Slave Host Replicator Manager/Monitor Replicator Manager/Monitor Application Server SQL Router/Connector Application Server SQL Router/Connector Management Client Management Client
  12. 12. Distributed Rule-Based Management Broadcast commands and monitoring data Business Rules Manager #1 (Coordinator) Manager #3 Manager #2 Admin Client #1 Admin Client #3 Group Communications Admin Client #2 Local DBMS Resources Local DBMS Resources Local DBMS Resources
  13. 13. Implementing SaaS Data Management with Tungsten
  14. 14. Best-Practice Multi-Tenant Architecture <ul><li>Separate global and tenant data </li></ul><ul><ul><li>Global data are same for all customers </li></ul></ul><ul><ul><li>Tenant data belong to individual customer </li></ul></ul><ul><li>Establish shards to hold tenants </li></ul><ul><ul><li>Bucket tenant data into databases, schemas, or groups of tables </li></ul></ul><ul><li>Define data services </li></ul><ul><ul><li>Global data service plus a set of tenant services is typical </li></ul></ul><ul><li>Master data in a single location for each service </li></ul><ul><ul><li>Services have one master and 1+N slaves for HA/scaling </li></ul></ul><ul><li>Add data services as you grow </li></ul><ul><li>TIP: Use database as basic sharding unit </li></ul>
  15. 15. Typical SaaS Architecture globalds1 tenantds1 tenantds4 tenantds4 tenantds2 Global tenant definitions and billing data Per tenant data (usually many tenants per service) globalds1 Master/slave pair for each service Implementation tenantds2 tenantds4 tenantds3 tenantds1
  16. 16. Installing a Tungsten Cluster <ul><li>Four easy steps: </li></ul><ul><li>Set up host prerequisites (MySQL, Ruby, Java) </li></ul><ul><li>Download release from and un-tar into local directory </li></ul><ul><li>Run ‘configure’ script </li></ul><ul><ul><li>Ensure MySQL is correctly configured </li></ul></ul><ul><ul><li>Configure cluster networking </li></ul></ul><ul><ul><li>Configure replication and choose backup method </li></ul></ul><ul><ul><li>Configure connectivity </li></ul></ul><ul><li>Perform customizations (e.g., Master VIP setup) </li></ul><ul><li>Configure starts services automatically </li></ul>
  17. 17. Advanced But Simple Management Tools <ul><li>‘ cctrl’ is the Tungsten management client </li></ul><ul><li>List data sources in the cluster </li></ul><ul><ul><li>[LOGICAL] /cluster/demo/> ls </li></ul></ul><ul><li>Check cluster liveness </li></ul><ul><ul><li>[LOGICAL] /cluster/demo/> cluster heartbeat </li></ul></ul><ul><li>Ensure that replication is synchronized </li></ul><ul><ul><li>[LOGICAL] /cluster/demo/> cluster flush </li></ul></ul><ul><li>Start and stop operating system services </li></ul><ul><ul><li>[LOGICAL] /cluster/demo/> service */mysql restart </li></ul></ul><ul><li>Get help </li></ul><ul><ul><li>[LOGICAL] /cluster/demo/> help </li></ul></ul>
  18. 18. Multiple Routes to Databases Java App Server Tungsten SQL Router PHP Application Tungsten Connector libmysqlclient.a Tungsten Cluster MySQL JDBC Driver Admin & Monitoring Admin & Monitoring Virtual IP Address
  19. 19. Connecting Applications to Databases <ul><li>Tungsten SQL Router embeds in Java applications </li></ul><ul><ul><li>Requires Java library substitution </li></ul></ul><ul><ul><li>Very high performance, seamless planned failover </li></ul></ul><ul><li>Tungsten Connector proxies any MySQL application </li></ul><ul><ul><li>Also very high performance with seamless planned failover </li></ul></ul><ul><ul><li>No library changes </li></ul></ul><ul><li>Master Virtual IP maintained by Tungsten </li></ul><ul><ul><li>Failover is not seamless (broken connections) </li></ul></ul><ul><ul><li>Completely transparent to applications </li></ul></ul><ul><li>Direct connection to database(s) </li></ul><ul><ul><li>Use whatever you want </li></ul></ul><ul><ul><li>You are responsible for locating master/slaves yourself </li></ul></ul>
  20. 20. Scaling Reads on Slaves <ul><li>Tungsten “session-consistency” distributes tenant reads to slaves if caught up with last tenant write </li></ul><ul><li>Works with tenants sharded by database or with independent database connections </li></ul>Master DB Slave DB SQL Router Record tenant write position Read if slave caught up with last tenant write
  21. 21. A Short Digression on Backups MySQL Server DBMS Data Files Active Host Applications Backup HOT/COLD: mysqldump Backup HOT: LVM snapshot HOT: InnoDB Hot Backup COLD: File system copy
  22. 22. Setting Up Backups <ul><li>Tungsten provides multiple backup methods </li></ul><ul><li>Backups can run hot while DBMS is live or cold </li></ul><ul><li>Logical backups use native DBMS utilities </li></ul><ul><ul><li>mysqldump, pg_dump, etc. </li></ul></ul><ul><ul><li>Can run hot on slaves </li></ul></ul><ul><ul><li>Slow to dump/load and in the case of MySQL may have bugs </li></ul></ul><ul><li>LVM (Logical Volume Manager) backups take a snapshot of the file system </li></ul><ul><ul><li>Very fast; run hot on slaves </li></ul></ul><ul><ul><li>Tend to be large </li></ul></ul><ul><ul><li>Snapshot maintenance affects write performance </li></ul></ul><ul><li>Script backups integrate arbitrary backup programs </li></ul><ul><li>TIP: Configure backups and test them regularly </li></ul>
  23. 23. Running Backups With Tungsten <ul><li>Hot backup using one-line command on slave </li></ul><ul><ul><li>[LOGICAL] > datasource centos5a backup lvm </li></ul></ul><ul><li>Cold backups run on off-line master or slave </li></ul><ul><ul><li>[LOGICAL] > datasource centos5b offline </li></ul></ul><ul><ul><li>[LOGICAL] > replicator centos5b offline </li></ul></ul><ul><ul><li>[LOGICAL] > datasource centos5b backup </li></ul></ul><ul><ul><li>[LOGICAL] > replicator centos5b online </li></ul></ul><ul><li>Restore with one-line command </li></ul><ul><ul><li>[LOGICAL] > datasource centos5a restore </li></ul></ul><ul><li>TIP: Keep data services small enough to backup and restore easily! </li></ul>
  24. 24. Verifying Tenant Data Copies <ul><li>Tungsten incremental checks verify data without stopping replication </li></ul><ul><ul><li>Implemented as replication events inside Tungsten Replicator </li></ul></ul><ul><ul><li>Compute MD5 on all or part of tables </li></ul></ul><ul><ul><li>Can do single tables or all tables in a database </li></ul></ul><ul><li>Run checks from manager in combination with flush command </li></ul><ul><ul><li>[LOGICAL] /cluster/demo/> cluster check tenant1.* </li></ul></ul><ul><ul><li>[LOGICAL] /cluster/demo/> cluster flush </li></ul></ul><ul><li>Slaves can either fail or warn of failures </li></ul><ul><li>TIP: Install consistency checks as a cron job </li></ul><ul><li>TIP: Verify backups with restore + consistency check </li></ul>
  25. 25. Handling Database Failures <ul><li>Automatic policy mode performs master failover automatically whenever: </li></ul><ul><ul><li>A database crashes </li></ul></ul><ul><ul><li>A replicator fails or crashes </li></ul></ul><ul><ul><li>A server reboots or drops off the network </li></ul></ul><ul><li>Manual policy mode lets you failover manually </li></ul><ul><ul><li>[LOGICAL] /cluster/demo/> failover </li></ul></ul><ul><li>Failover procedure promotes most advanced slave </li></ul><ul><ul><li>And points other slaves to the slave </li></ul></ul><ul><li>Recover command brings back failed master </li></ul><ul><ul><li>[LOGICAL] /cluster/demo/> datasource centos5a recover </li></ul></ul><ul><li>Slaves recover without admin intervention in automatic mode </li></ul>
  26. 26. Performing Planned Maintenance <ul><li>Tungsten supports “daylight maintenance” </li></ul><ul><li>To perform maintenance on a slave, just take it offline </li></ul><ul><ul><li>[LOGICAL] /cluster/demo/> set policy maintenance </li></ul></ul><ul><ul><li>[LOGICAL] /cluster/demo/> datasource centos5b offline </li></ul></ul><ul><ul><li>[LOGICAL] /cluster/demo/> replicator centos5b offline </li></ul></ul><ul><li>To perform maintenance on a master, first do a switch to promote a new master </li></ul><ul><ul><li>[LOGICAL] /cluster/demo/> switch </li></ul></ul><ul><li>Then perform maintenance using the slave procedure </li></ul><ul><li>Switch the master back when you are done </li></ul>
  27. 27. Upgrading Applications and Databases <ul><li>If you have a maintenance window, just upgrade the master and let replication take care of it </li></ul><ul><ul><li>Stop old applications and replace code </li></ul></ul><ul><ul><li>Run upgrade script against master </li></ul></ul><ul><ul><li>Flush and wait for slave to catch up (if necessary) </li></ul></ul><ul><ul><li>Restart new applications </li></ul></ul><ul><li>If you don’t have a maintenance window, upgrade a slave and then switch </li></ul><ul><ul><li>Take slave offline so that apps point to master only </li></ul></ul><ul><ul><li>Upgrade the slave </li></ul></ul><ul><ul><li>Turn on replication to slave so that it catches up </li></ul></ul><ul><ul><li>Switch the master to the upgraded slave; take old master offline </li></ul></ul><ul><ul><li>Restart applications with new code </li></ul></ul><ul><ul><li>Upgrade old master and bring back online </li></ul></ul><ul><li>TIP: Test all upgrades in a pre-prod environment </li></ul>
  28. 28. SaaS Problems and Tungsten Solutions MySQL 5.0/5.1 Open source databases Cctrl switch operation and offline/online commands System maintenance Upgrades via replication; slave upgrade Application upgrades Automated and manual failover; live replicas; recovery procedures Handling failures Backups; live replicas; consistency checks Protecting tenant data from loss or corruption Tenant-based load balancing Performance scaling Simple installation; cctrl distributed commands Implementing easy-to-manage clusters “ Best-practices” database design Multi-tenant architecture Tungsten Features(s) Problem
  29. 29. Solution Roadmap for SaaS Applications <ul><li>Continuent is dedicated to making MySQL the foundation for SaaS businesses </li></ul><ul><li>We will talk about these in our February 11 webinar “Secrets of Successful SaaS Data Architecture” </li></ul>Q2-3 Q2 Tenant management using shards Q2-3 Q2 Cross-site cluster management Q2-3 Q2 Multi-master replication Parallel replication SaaS Feature Q2-3 Q1 Prod Beta
  30. 30. Commercial Support for SaaS from Continuent
  31. 31. Tungsten Products <ul><li>Tungsten Enterprise offers subscription-licensed supported clustering for SaaS applications </li></ul>X Built-in Jump-Start consulting X Guaranteed bug fixes X 8x5, 24x7 support X Custom backup/restore X Virtual IP support X Automatic failover X X Manual failover X X SQL routing/proxying X X Advanced management tools Advanced MySQL replication Feature X X Enterprise Community
  32. 32. Continuent Services <ul><li>Practical consulting by industry experts with deep MySQL and SaaS industry background </li></ul><ul><li>Structured Jump-Starts </li></ul><ul><ul><li>Hardware/OS setup, education, installation, production roll-out </li></ul></ul><ul><li>Solution architecture workshops </li></ul><ul><ul><li>Collaborative engagements to specify SaaS data management requirements, survey problems, and lay out options for solutions </li></ul></ul><ul><li>Proof-of-Concept implementations </li></ul><ul><ul><li>Fast design and implementation of specific cluster configurations </li></ul></ul>
  33. 33. Tungsten Design Partner Program <ul><li>Continuent is developing advanced SaaS features in collaboration with leading SaaS vendors </li></ul><ul><li>We offer expertise not only in database clustering but operation of SaaS applications </li></ul><ul><li>Partner with Continuent to “push the envelope” on scalable data management using open source databases </li></ul><ul><li>Please contact us today to find out more about joining our efforts </li></ul>
  34. 34. Conclusion <ul><li>SaaS applications have a complex database problem that requires “big league” architecture </li></ul><ul><li>Continuent offers a complete solution for SaaS data management to raise overall profitability: </li></ul><ul><ul><li>Raise SLAs by improving availability and performance </li></ul></ul><ul><ul><li>Improve management efficiency </li></ul></ul><ul><ul><li>Grow customers and product offerings through scalable database architecture </li></ul></ul><ul><ul><li>Increase SaaS business profitability </li></ul></ul><ul><li>Contact us to try Tungsten now! </li></ul>
  35. 35. Contact Information EMEA and APAC Lars Sonckin kaari 16 02600 Espoo, Finland Tel +358 50 517 9059 Fax +358 9 863 0060 Continuent Web Site: HQ and Americas 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009 e -mail: [email_address]
  36. 36. Backup Slides
  37. 37. How Do I Manage SaaS Databases? <ul><li>Tungsten clusters consist of “data sources” </li></ul><ul><li>You can view data source state using the ‘cctrl’ command </li></ul><ul><ul><li>$ tungsten-manager/bin/cctrl </li></ul></ul><ul><ul><li>[LOGICAL] /cluster/demo/> ls </li></ul></ul><ul><ul><li>COORDINATOR[centos5a:MANUAL] </li></ul></ul><ul><ul><li>ROUTERS(centos5a:10999:ONLINE) </li></ul></ul><ul><ul><li>DATASOURCES: </li></ul></ul><ul><ul><li>centos5a(slave:ONLINE, progress=10, latency=0.443)REPLICATOR(role=slave, master=centos5b, state=ONLINE) </li></ul></ul><ul><ul><li>centos5b(master:ONLINE, progress=10)REPLICATOR(role=master, state=ONLINE) </li></ul></ul><ul><li>Use ‘datasource’ and ‘replicator’ commands to manage database state </li></ul>
  38. 38. How Do I Recover Failed Databases? <ul><li>Recover procedures depend on what actually happened to your database </li></ul><ul><li>To fix a failed slave database, just restart the database </li></ul><ul><ul><li>In automatic mode, rules take care of the rest </li></ul></ul><ul><li>To fix a failed master, reprovision it as a slave, then add it back to the cluster </li></ul><ul><ul><li>[LOGICAL] /cluster/demo/> set policy maintenance </li></ul></ul><ul><ul><li>[LOGICAL] /cluster/demo/> datasource centos5a slave </li></ul></ul><ul><ul><li>[LOGICAL] /cluster/demo/> replicator centos5b slave </li></ul></ul><ul><ul><li>[LOGICAL] /cluster/demo/> datasource centos5a welcome </li></ul></ul><ul><ul><li>[LOGICAL] /cluster/demo/> set policy automatic </li></ul></ul>