Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Druid meetup @ Netflix (11/14/2018 )

133 views

Published on

https://www.meetup.com/druidio/events/254994344/

https://www.youtube.com/watch?v=Qvhqe4yUKpw&feature=youtu.be

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Druid meetup @ Netflix (11/14/2018 )

  1. 1. Gian Merlino, Matt Herman, Vivek Pasari, Samarth Jain Druid Meetup 11/14/18 - Los Gatos, CA @
  2. 2. Agenda ● Druid Deployment @ Netflix ● Scaling & Sketch Strings ● Druid Roadmap ● Q&A
  3. 3. Druid Deployment & Use Cases @Netflix
  4. 4. Overview ● Druid in Netflix D/W ● Data ingestion ● Deploying Druid @ Netflix ● Use Cases
  5. 5. Netflix Data Warehouse Pipeline
  6. 6. Druid Ingestion ● Batch ingestion ● Druid hadoop indexer ● Input - Hive Text/Parquet tables ● S3 deep storage
  7. 7. Druid Ingestion
  8. 8. Druid Ingestion import BigDataApi as bda tbl = bda.Table("hive/table_name") # build spec spec = bda.druid.DruidSpec.from_table(tbl) .spec(ingestion_spec.json) # create a job from the spec job = bda.genie.DruidIndexerJob(spec) .cluster(kg.druid.clusters.DRUID_CLUSTER_NAME) # submit the job… job.execute()
  9. 9. Druid Cluster @ Netflix ● r 4.16 x large instance type ● 0.12.2 version ● ~100s nodes
  10. 10. Multitenancy ● Single Tier ● Router ○ Ad hoc ○ Experimental - broker downtime acceptable. Used for query fine tuning etc. ○ Reporting - pre-defined queries /dashboards
  11. 11. Autoscale ● Favor segments in memory ● Autoscale up - cluster disk utilization beyond 80% ● Handle large data ingestion without having to worry about cluster tripping over
  12. 12. Deployment Pipeline ● Spinnaker (https://www.spinnaker.io/) ● Clusters upgraded using red black ○ Jenkins jobs - druid tar ball and debian package ○ Deploy components with new code line ○ Wait for segments to load ○ Switch dns records ○ Scale down old cluster ● Rollback ○ Switch dns back to old cluster
  13. 13. Deployment Pipeline
  14. 14. Use Cases ● Dashboard backend ● Sub second query times ○ User interactive slice and dice ○ Longer data retention vs Redshift ○ More dimensions vs Redshift ● Custom UI
  15. 15. AWS Capacity Planning
  16. 16. AWS Capacity Planning
  17. 17. AWS Capacity Planning
  18. 18. Other use cases ● Payments analysis ● Algorithms comparison ● Security ● Quality of Experience (QoE)
  19. 19. Future work ● Real time ingestion ○ Tranquility or Kafka indexing ● Open source T-Digest based Histogram module ● Investigate tiering ● Change auto-scaling policy considering EBS
  20. 20. Scaling & Sketch Strings How Netflix Processes 160B Daily Customer Actions to Monitor Client Performance
  21. 21. #netflixeverywhere
  22. 22. “With this launch, consumers around the world will be able to enjoy TV shows and movies simultaneously -- no more waiting. With the help of the Internet, we are putting power in consumers’ hands to watch whenever, wherever and on whatever device.” “With this launch, consumers around the world will be able to enjoy TV shows and movies simultaneously -- no more waiting. With the help of the Internet, we are putting power in consumers’ hands to watch whenever, wherever and on whatever device.” “With this launch, consumers around the world will be able to enjoy TV shows and movies simultaneously -- no more waiting. With the help of the Internet, we are putting power in consumers’ hands to watch whenever, wherever and on whatever device.” “With this launch, consumers around the world will be able to enjoy TV shows and movies simultaneously -- no more waiting. With the help of the Internet, we are putting power in consumers’ hands to watch whenever, wherever and on whatever device.” “With this launch, consumers around the world will be able to enjoy TV shows and movies simultaneously -- no more waiting. With the help of the Internet, we are putting power in consumers’ hands to watch whenever, wherever and on whatever device.”
  23. 23. ● 160 Billion client side data points daily ● 135+ million members ● 190 countries ● 300 million devices ● 4 major UI platforms TVUI, Web, iOS, Android Measure Everything Consistently
  24. 24. Client Performance
  25. 25. ● Metrics ○ App launch times ○ Play delay ○ Details page time ● 29 dimensions ○ Geo ○ Network ○ Device ○ AB test cell Client Performance Metrics
  26. 26. Architecture
  27. 27. Show Me The Data
  28. 28. Summarize Instead
  29. 29. Anscombe’s Quartet
  30. 30. Saved by Sketch Strings
  31. 31. Box & Whisker Plots Median application load times are similar Country B has a larger IQR and long tail
  32. 32. Cumulative Distribution Functions
  33. 33. Recap ● Ingesting consistent and highly dimensional data ● Analyzing data via custom web visualizations ● Summarizing responsibly via sketch strings ● Druid helps us provide the best customer experience
  34. 34. Druid Roadmap
  35. 35. roadmap and community update Gian Merlino gian@imply.io
  36. 36. Who am I? Gian Merlino Committer & PMC member on Cofounder at >10 years working on scalable systems 40
  37. 37. Druid 0.13.0 …and beyond!!
  38. 38. Druid 0.13.0 400 new features and bug fixes from 81 contributors! 42
  39. 39. Druid 0.13.0 Our first Apache release! (After years as an independent project.) 43
  40. 40. Druid 0.13.0 ● Native parallel batch indexing (phase 1) ● Automatic compaction (phase 1) ● Ingestion statistics and errors via API ● SQL system tables: segments, tasks, servers ● SQL standard-compliant null handling option ● Additional aggregators (stringFirst/stringLast, new HllSketch) ● Support for multiple grouping specs in groupBy query ● Backpressure, compact result formats for large result sets 44
  41. 41. …and beyond!! ● Native parallel batch indexing (phase 2) ● Automatic compaction (phase 2) ● Smaller, faster compression (FastPFOR, etc) ● Faster quantiles: Fixed-bin histograms, moments sketches ● Dynamic prioritization ● Simpler, self-configuring deployment ● … your item here!! 45
  42. 42. Try this at home 46
  43. 43. Download Druid community site (current): http://druid.io/ Druid community site (new): https://druid.apache.org/ Imply distribution: https://imply.io/get-started 47
  44. 44. Contribute 48 https://github.com/apache/druid
  45. 45. Stay in touch 49 @druidio http://druid.io/community
  46. 46. Q&A

×