Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scale up - How to build adaptive data systems in the age of virality

273 views

Published on

In this talk we share details about glomex's award-winning data management infrastructure. They’ll show you how a serverless approach can scale automatically to the demands of a highly unpredictable industry as video clips go viral arbitrarily. What is the best architecture for real time data processing? How does a batch-driven BI workflow fit in? What are the key benefits of going to the Cloud? Which AWS services should you use?

Published in: Internet
  • Be the first to comment

  • Be the first to like this

Scale up - How to build adaptive data systems in the age of virality

  1. 1. 1glomex – A company of ProSiebenSat.1 Media SE glomex THE GLOBAL MEDIA EXCHANGE Scale up - How to build adaptive data systems in the age of virality Winner “Best Data Management and Infrastructure” Gartner Data & Analytics Summit 2017, London
  2. 2. 2glomex – A company of ProSiebenSat.1 Media SE About us Head of Site Reliability Engineering @jobrandstetter • 1 year with glomex (founded in May ‘16) • Deeply involved in building Data Platform • Software Developer, MongoDB Master, Metrics Nerd Vice President of Engineering @michael_muckel • Founding member of the glomex Data Platform Team • Responsible for glomex Engineering • Software Architecture, Analytics, Machine Learning
  3. 3. 3glomex – A company of ProSiebenSat.1 Media SE No.1 TV sales house in GSA ~3bn video views per year 50+ online & mobile platforms 20 years of technical excellence Where We Come From No.1 TV broadcaster in Germany glomex – a ProSiebenSat.1 company
  4. 4. 4glomex – A company of ProSiebenSat.1 Media SE glomex is … the platform as a service provider for online video management, serving 3bn. video views per year today the platform owner of a global marketplace for online video distribution the pioneer in video delivery management for publishers and content owners around the globe Who we are
  5. 5. 5glomex – A company of ProSiebenSat.1 Media SE Video Value Service Media Exchange Service Media Delivery Service glomex Content providers Publishers The global B2B content marketplace TV broadcasters and web-only content producers
  6. 6. 6glomex – A company of ProSiebenSat.1 Media SE Video Value Service Media Delivery Service Media Exchange Service Data Platform Real-Time-Monitoring Analytics Machine Learning Data Platform
  7. 7. 7glomex – A company of ProSiebenSat.1 Media SE Serverless Computing
  8. 8. 8glomex – A company of ProSiebenSat.1 Media SE Serverless Basics Evolution of Computing Weeks Minutes Seconds On-premise Virtual Machines Containers Amazon EC2 Amazon ECS
  9. 9. 9glomex – A company of ProSiebenSat.1 Media SE AWS Lambda Notification Amazon S3 AWS Lambda processes the object Amazon S3 New object uploaded Amazon DynamoDB
  10. 10. 10glomex – A company of ProSiebenSat.1 Media SE AWS Lambda AWS Lambda Amazon API GatewayAmazon Kinesis Serverless – Ingest and Serving
  11. 11. 11glomex – A company of ProSiebenSat.1 Media SE • Read data from Kinesis Firehose / S3 • Server downtime / scheduler • Load to ElasticSearch • Clean ElasticSearch and Redshift • Advanced Redshift monitoring • EBS Snapshots Be Serverless. Everywhere.
  12. 12. 12glomex – A company of ProSiebenSat.1 Media SE AWS Lambda Execution
  13. 13. 13glomex – A company of ProSiebenSat.1 Media SE Some Facts 50 GB 50 Million Per day click-stream data IN Click-stream records processed per day ~100 ms Data freshness to S3 25 GB 1 Billion Per day as zipped CDN log-files CDN record processed per day < 1 min Data freshness to API
  14. 14. 14glomex – A company of ProSiebenSat.1 Media SE • AWS gives you primitives • AWS tools totally unopinionated • In beginning of 2016 hardly any good tooling available • Lots of problems with reliable Lambda deployments (binary packages) • Lack of common workflow for various deployment scenarios • Better Developer Experience Development Workflow
  15. 15. 15glomex – A company of ProSiebenSat.1 Media SE glomex Cloud Deployment Tools Agile Cloud Deployment 雲 kumo ラムダ ramuda 幽玄 yugen 展開 tenkai • Used by other teams • Slack and monitoring integration • Simplify build automation • Codify proven practices • Enable self-service • Automate deployments • 6000 deployments in 02/2017 • 1100 deployments on prod in 02/2017 • 300 errors (5%)
  16. 16. 16glomex – A company of ProSiebenSat.1 Media SE Use Case - Monitoring Video Streaming Experience
  17. 17. 17glomex – A company of ProSiebenSat.1 Media SE Focus on Metrics from the User‘s Perspective From Server-Uptime To (anonymized) Real-User Monitoring Monitoring Video Streaming Experience
  18. 18. 18glomex – A company of ProSiebenSat.1 Media SE Scalable, Stream-based Ingest Pipeline
  19. 19. 19glomex – A company of ProSiebenSat.1 Media SE Traffic Patterns
  20. 20. 20glomex – A company of ProSiebenSat.1 Media SE Use Case 2 - Serverless Recommender System Publisher’s URL Serverless Recommender Generated PlaylistPublisher Website  Adaptive Page Crawler  Entity Extraction  Topic Modeling  Search Metadata Index
  21. 21. 21glomex – A company of ProSiebenSat.1 Media SE Infrastructure as Code Agile Cloud Deployment Structure Speed SecurityHealth
  22. 22. 22glomex – A company of ProSiebenSat.1 Media SE Quick Lessons Learned • Focus on feature development and robust pipelines not on infrastructure management • AWS managed services provide a robust way to run complex big data infrastructures • Cross functional teams help velocity and quality • Use a micro service architecture
  23. 23. 23glomex – A company of ProSiebenSat.1 Media SE Thanks for Listening! Visit us at – explore.glomex.com We‘ve recently opened a London office and we‘re hiring: • Content & Publishers Sales Managers • Media Sales Leaders • Account Managers

×