Marketing Automation at Scale: How Marketo Solved Key Data Management Challenges with Continuent Tungsten


Published on

Marketo provides the leading cloud-based marketing software platform for companies of all sizes to build and sustain engaging customer relationships. Marketo's SaaS platform runs on MySQL and has faced data management challenges common to all 24x7 SaaS businesses:

- Keeping data available regardless of DBMS failures or planned maintenance
- Utilizing hardware optimized for multi-terabyte MySQL servers
- Keeping replicas caught up and ready for instant failover despite high transaction loads

In this webinar, Nick Bonfiglio, VP of Operations at Marketo, describes how Marketo manages thousands of customers and processes a billion marketing analytics transactions a day using Continuent Tungsten and MySQL atop an innovative hardware architecture. He explains how Tungsten parallel replication paved the way to rapid growth by solving Marketo's biggest MySQL challenge: keeping DBMS replicas up to date despite massive transaction loads.

Published in: Technology

Marketing Automation at Scale: How Marketo Solved Key Data Management Challenges with Continuent Tungsten

  1. 1. ©Continuent 2013 Marketing Automation at Scale: How Marketo Solved Key Data Management Challenges with Continuent Tungsten Nick Bonfiglio – VP Operations, Marketo Robert Hodges – CEO, Continuent
  2. 2. About Us • Nick Bonfiglio - Marketo VP of Operations / Responsible for Cloud Services and Security at Marketo / Extensive background in scaling SaaS products and services • Robert Hodges - Continuent CEO / Tungsten Replicator designer / DBMS internals and applications for three decades / Author of Scale-Out Blog (http://scale-out-
  3. 3. Marketo is the global leader in Revenue Performance Management. Marketo’s powerful, yet easy-to-use marketing automation and sales effectiveness solutions, transform how marketing and sales teams of all sizes work — and work together — to drive dramatically increased revenue performance and fuel business growth. The company’s proven technology, comprehensive services, and expert guidance are helping corporations around the world to turn marketing from a cost center to a business-building revenue driver. About Marketo
  4. 4. / The leading provider of data replication and clustering for open source databases / Our Product: Continuent Tungsten • Clustering – Commercial-grade HA, performance scaling, and data management for MySQL • Replication – Flexible, high-performance data movement About Continuent
  5. 5. 4 Customers from hot startups to Fortune 500 companies
  6. 6. ©Continuent 2013 Marketo’s Data Management Challenge Nick Bonfiglio
  7. 7. Marketo Architecture • Marketo uses a mixed stack architecture to support its customers’ subscriptions, leveraging the best technologies for the solution. • Marketo is a platform consisting of a collection of systems that provide user interface, landing pages/forms, fast secondary search (solr), big data (hadoop) and archiving (cassandra). • Our systems employ several custom queue- based background processing, which are key to a successful marketing automation platform. And, custom email assembly, scripting and tracking facilities.
  8. 8. Systems Architecture Marketo SaaS Platform File Services Database Services Batch Processes User Interface Munchkin (tracking) APILanding Pages/Forms Email Services
  9. 9. Infrastructure Architecture
  10. 10. Marketo DB Design Goals • Deliver increased availability by providing redundancy and real-time resiliency at all layers. • Deliver an architecture that allows us to upgrade software and routine database schema changes without incurring downtime to customers. • Improve overall performance, scalability and management, including backups and disaster recovery of large 7TB+ DB servers.
  11. 11. Marketo DB Architecture DB on FlexVol DB on FlexVol DB on FlexVol Replicator ReplicatorReplicator MasterSlave Slave Marketo Web Cluster Marketo Queue Cluster MySQL Client LibraryMySQL Client Library Manager ManagerManager Pod Data DBMS Vendor Client Library DBMS Vendor Client LibraryTungsten Connector Tungsten Connector Monitoringandcontrol Monitoringandcontrol
  12. 12. Marketo DB Hardware Architecture • Leveraging NetApp FlexVols for snapshot, near-line and DR replication • Symmetric DB servers so failover yields full performance • Aggregates are 300GB drives for increased spindles • Each DB Server ~6-8T • Leverage flexcache for certain volumes
  13. 13. Performance Metrics • Over 600 Million MySQL Transactions per day • Over 600TB of data • Processing 25 Million Campaigns per day • Sending over 500M Emails per month • Serving 15M Landing Page Visits per day • A few proof points: – Several failovers which would have caused an outage – Parallel replication allows is to keep lag to <300sec – Several roll rotations for maintenance (schema changes, dropping large dbs, db pruning, etc.)
  14. 14. ©Continuent 2013 So… How Does Tungsten Do That? Robert Hodges
  15. 15. Challenges to Hosting Critical Data on MySQL / Get back online quickly after DBMS failures / Perform DBMS maintenance and SQL upgrades without stopping applications / Load balance SQL across replicas to use hardware efficiently / Ensure slaves keep up with high master transaction loads / Address all of this without migrating or rewriting applications
  16. 16. Tungsten Database-as-a-Service for MySQL ReplicatorReplicator • Cluster constructed from off-the-shelf MySQL servers • Automatic failover • Online maintenance • Query Load balancing for speed and h/w utilization • No migration of data or application changes Manager db2 Manager Replicator db1 Manager db3 Slave SlaveMaster Application Stack Tungsten Connector Application Stack Tungsten Connector ReplicatorReplicator
  17. 17. Transparent connectivity enables illusion of single DBMS to applications • Locate appropriate server for transactions • Select most up-to-date server for reads • Shift connections from offline nodes • Hide offline databases • Supports SSL Send writes to master Load balance reads over slaves Application Stack Application StackApplication Stack Tungsten Connector db2 db1 db3
  18. 18. ©Continuent 2013 Demo: Introducing a Tungsten Data Service
  19. 19. What Causes Slave Lag? Application Application Application Master InnoDB Buffer Pool NetApp Storage Slave InnoDB Buffer Pool NetApp Storage Typical SaaS Application: 1.) Data size >> buffer pool 2.) Persistent stored on disk
  20. 20. SaaS Applications Can Replicate Tenants in Parallel! • Sharded replication divides updates into independent streams (“shards”) tenant_1 tenant_2 tenant_3 tenant_4 Tenants independent from each other Tenant database updates applied in parallel tenant_1 tenant_2 tenant_3 tenant_4
  21. 21. High-Speed Parallel Replication Lets Slaves Keep Pace with Busy Masters • Shard transactions by schema on master • Transfer log serially • Apply shard updates over “channels” • 5x or more speed boost in best cases • Works on all MySQL versions from 5.0 to 5.6 Replicator db1 Master db2 Slave Replicator Application Stack Application StackApplication Stack
  22. 22. Problem: Uneven SaaS Tenant Size
  23. 23. Problem: “Hot” Tables within Single Schemas that Block Other Tenants
  24. 24. Tungsten’s Innovative Caching Minimizes Effects of Lumpy SaaS Workload Slave InnoDB Buffer Pool NetApp Storage Tungsten Replication Service NetApp Storage Fetch from Master Write to Log Apply to Slave Linux Page Cache Read log from page cache (No reads from storage) Write log to page cache Fsync to storage Shards apply Independently!
  25. 25. Innovative Features of Tungsten Parallel Replication Support • Tungsten leans on OS page cache to keep long updates from blocking progress • Use any number of apply channels • Support for clean failover and backup • Completely crash-safe for InnoDB • Configuration checks to prevent accidental slave corruption • Sharding algorithm is configurable!
  26. 26. ©Continuent 2013 Demo: Maintenance on MySQL master under load
  27. 27. What Else Can Tungsten Do? • Cross-site data services – Primary/disaster recovery sites – Multi-master replication between sites • Fast data warehouse/NoSQL loading from MySQL/Oracle • Oracle-to-Oracle replication
  28. 28. What’s Next? Push-Button Cloud Operation Configure DBMS type and resource levels for servers Define backup and recovery policies Confirm configuration and launch Choose Service Type Manage Deployment 1 2 3 4
  29. 29. Making it Better – Visual Management
  30. 30. Making it Better – Visual Management
  31. 31. Failure detected and failover completed within 15 seconds Making it Better – Visual Management
  32. 32. Failed db instance is isolated from the service Making it Better – Visual Management
  33. 33. ©Continuent 2013 Conclusion and Q&A
  34. 34. In Summary… 33 • Marketo performs 600M transactions daily on MySQL running over NetApp • Tungsten delivers fast failover, rolling maintenance, and parallel replication to make architecture work • Marketo and Continuent together have enabled an architecture that supports economical scaling to 1000s of customers
  35. 35. Contact Us! Continuent, Inc. 560 S. Winchester Blvd Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: Marketo, Inc. 901 Mariners Island Blvd Suite 200 San Mateo, CA 94404 Tel +1 (877) 260-MKTO (6586) Fax +1 (650) 376-2331 e-mail: