Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How Totango uses Apache Spark

Overview presentation on how totango uses apache spark in our data architecture. Presented at the 'Big-Data Israel Meetup'. Nov. 2015

  • Login to see the comments

How Totango uses Apache Spark

  1. 1. Enterprise Grade Spark Processing at Totango 2015-11-10 Oren Raboy, VP Eng. @ Totango
  2. 2. AGENDA (PART 1) • Background about Totango and our data architecture • Spark in the Totango Architecture • Quality: Testing Spark code in production
  3. 3. About us: Founded: 2010 Offices: TLV, SF Team: ~60 Customers: ~200
  4. 4. We help online business make their customers more successful through use of data Totango:
  5. 5. ~ 500M accounts ~ $5B revenue under management ~ 100M events per day Our Customers: The Worlds Leading Cloud Services
  6. 6. ANALYTICS • Usage Metrics • Trends over time • Trends across customers • Health score
  7. 7. • Alerts • Triggered Workflows • Email Campaigns AUTOMATION
  8. 8. Totango Data Architecture Collection Real-time processing Batch processing Pixel 3rd Party (SFDC) CSV Serving Layer • ‘Lambda Architecture’ • Hosted on AWS • AWS and Open-source technologies • Java with a dash of Python
  9. 9. Totango Data Architecture Pixel 3rd Party (SFDC) CSV • Hosted on AWS • ‘Lambda Architecture’ • AWS and Open-source technologies • Java with a dash of Python Kinesis Kinesis S3 ELB
  10. 10. Totango Data Architecture Pixel 3rd Party (SFDC) CSV • Hosted on AWS • ‘Lambda Architecture’ • AWS and Open-source technologies • Java with a dash of Python Kinesis Kinesis S3 ELB
  11. 11. Batch Processing • Executed once a day (midnight at customer’s local-time) • Each task calculates a set of account-metrics (e.g. Health, Change) • One Spark cluster runs all tasks for all customers • Pipeline executed by Pipeline Runner, using Spotify Luigi calc some metrics calc other metrics more merge results Some dependent computation Merge results Into final document Raw Events Account Documents
  12. 12. Environment • Multi tenant: Shared infrastructure for all Totango customers (Services) • Daily, hourly and on-demand schedule • Standalone Spark cluster on AWS EC2 instances • Input and Output on S3. Final results also indexed on Elasticsearch Service A calc some metrics calc other metrics more merge results Some dependent computation Merge results Into final document Raw Events Account Documents Service A calc some metrics calc other metrics more merge results Some dependent computation Merge results Into final document Raw Events Account Documents Service XYZ calc some metrics calc other metrics more merge results Some dependent computation Merge results Into final document Raw Events Account Documents
  13. 13. Challenge: Quality
  14. 14. Requirements from infrastructure: • Reliability: Calculate metrics accurately at all times • Velocity: Frequent release of new data processing code Challenge: High quality and highly automated regression testing calc some metrics calc some metric more merge results Some dependent computation Merge results Into final document Raw Events Account Documents NEW VERSION How do we make sure the new version didn’t break anything?
  15. 15. calc some metrics merge results Some dependent computation Merge results Into final document Raw Events Account Documents NEW VERSION SHADOW OLD VERSION compare csv Testing In Production: How • Before deployment, run release-candidate ‘side by side’ older version. • New version runs in Shadow mode and does not propagate results • Compare old and new version results. Output unexpected diffs • Deploy to production only if no diffs across all customer data sets
  16. 16. 1. 2. 3. 4. 5. Unit testing Test environment: Integration testing Side by side testing in production of new code New code rolled-out, old version side-by-side as backup Rollout complete! Deployment Flow • We know the new version works correctly • We do not need to think of all the corner test-cases • We do not need to write lots of regression tests
  17. 17. QUESTIONS? • labs.totango.com <-- engineering team blog • oren@totango.com <-- me! • Yes, we are hiring!

×