Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Enterprise Grade Streaming under 2ms on Hadoop

300 views

Published on

Enterprise Grade Streaming under 2ms on Hadoop

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Enterprise Grade Streaming under 2ms on Hadoop

  1. 1. Enterprise Grade Streaming Under 2ms On Hadoop @vijaysbhat
  2. 2. 2
  3. 3. 3 VS.
  4. 4. 4
  5. 5. 5
  6. 6. 6
  7. 7. 7 X (predictor) Spend amount, geo Y (response) Simple Velocity Advanced
  8. 8. 8
  9. 9. 9
  10. 10. 10
  11. 11. 11 Hard Metrics Goal Latency < 40ms Ideally < 16ms Throughput Goal of 2000 events / second Durability No loss, every message gets exactly one response Availability 99.5% uptime (downtime of 1.83 days / year); Ideally 99.999% uptime (downtime of 5.26 minutes / year) Scalability Can add resources, still meet latency requirements Integration Transparently connected to existing systems – Hardware, Messaging, HDFS Soft Metrics Goal Open Source All components licensed as open source Extensibility Rules can be updated, model is regularly refreshed
  12. 12. 12
  13. 13. 13 Onyx
  14. 14. 14 Enterprise Readiness RoadmapPerformance Community
  15. 15. 15
  16. 16. 16
  17. 17. 17
  18. 18. 18
  19. 19. 19
  20. 20. 20
  21. 21. 21
  22. 22. 22
  23. 23. 23
  24. 24. 24 Failure Handling
  25. 25. 25
  26. 26. 26 • Avg. 0.25ms, @70k records/sec, w/ 600GB RAM Thread Local on ~54M events Percentiles (in ms) Throughput Count Avg (ms) 90% 95% 99% 99.9% 4 9’s 5 9’s 6 9’s 70k/sec 54,126,1 22 0.19 1 1 1 2 2 5 6 Performance
  27. 27. 27 Durability • Two physically independent pipelines on the same cluster processing identical data • For the same tuple, we find the best-case time between two pipelines – 39 records out of 5.2M exceeded 16ms – 173 out of 5.2M exceeded 16ms in one pipeline but succeeded in the other • 99.99925% success rate – “Five Nines” • Average Latency of 0.0981ms
  28. 28. 28 @vijaysbhat linkedin.com/in/vijaysbhat

×