Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Streaming, Fast and Slow_Caito Scherr

71 views

Published on

What if you were given 2 weeks to prepare for running your first marathon, and had to be able to keep up with the fastest runner? This is the story of a small team that jumped head first into building a top tier stream processing service on a tight timeline, and how they prepared for being able to keep up with the fastest components.

This talk will be of interest to those who want to build larger, more complex stream processing applications (and quickly), but will also benefit anyone who is looking to adopt a new streaming technology (or use it in a way that is new to their company), at any scale. Other takeaways include pragmatic steps to leverage unique features of various stream processing frameworks to more effectively solve common, real world data streaming problems.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Streaming, Fast and Slow_Caito Scherr

  1. 1. STREAMING, FAST AND SLOW June 2020 Caito Scherr Caito_200_OK
  2. 2. • Caito Scherr • Portland, OR, USA • Software Engineer • Apache Flink Who am I? Caito_200_OK
  3. 3. Agenda Background Integration Operations Analytics Caito_200_OK
  4. 4. Agenda Caito_200_OK Background Integration Operations Analytics
  5. 5. Agenda Background Integration Operations Analytics Caito_200_OK
  6. 6. Agenda Background Integration Operations Analytics Caito_200_OK
  7. 7. BACKGROUND Caito_200_OK
  8. 8. Background Caito_200_OK Product A usage Product B usage Product C usage Subscription Info Account Info Data Store, UI Data pipeline
  9. 9. Background Caito_200_OK
  10. 10. STREAMING, FAST AND SLOW Caito_200_OK
  11. 11. Speed Caito_200_OK
  12. 12. … Like training for your first marathon Caito_200_OK
  13. 13. Caito_200_OK … Like training for a new marathon
  14. 14. Caito_200_OK … Like training for a new marathon
  15. 15. Caito_200_OK … Like training for a new marathon
  16. 16. Caito_200_OK … Like training for a new marathon Integration
  17. 17. Caito_200_OK … Like training for a new marathon Integration Operations
  18. 18. Caito_200_OK … Like training for a new marathon Integration Operations Analytics
  19. 19. INTEGRATION Caito_200_OK
  20. 20. Ecosystem Some of the best features of stream processing for your team can also be the biggest drawbacks for teams that interact with you. Caito_200_OK
  21. 21. Ecosystem Start by assuming each team may be treating their data differently from the next. Caito_200_OK
  22. 22. Ecosystem Optimize for: understanding your application’s impact, and allowing that data to drive the architecture and design process. Caito_200_OK
  23. 23. Streaming Community Caito_200_OK
  24. 24. Streaming Community Caito_200_OK Documentation that is interactive, A safe space for others to experiment, And, take a map!
  25. 25. Streaming Community Caito_200_OK
  26. 26. Streaming Community Caito_200_OK Don’t underestimate the power of a good map
  27. 27. OPERATIONS Caito_200_OK
  28. 28. Resources Can your resources handle this new challenge? Caito_200_OK
  29. 29. ANALYTICS Caito_200_OK
  30. 30. Monitoring Caito_200_OK
  31. 31. Alerting Caito_200_OK
  32. 32. Alerting Caito_200_OK
  33. 33. Alerting Caito_200_OK
  34. 34. Conclusion Caito_200_OK
  35. 35. • Pulsar Summit • Jun, Rosalie & the conference event staff! • Maureen Dugan Thank you! • Ruby Andrews • Eric Shamow • Adam Klein • Matthew Scharr • Max Aguirre Caito_200_OK
  36. 36. ❖ Squirrel photo: https://www.blipfoto.com/entry/1844394 ❖ Track logo: https://www.wikidata.org/wiki/Q1004435 ❖ shoes: https://www.ems.com/hoka-one-one-mens- speedgoat-2-trail-running-shoes/2043264.html ❖ All other photos of Caito: Hood to Coast team (Van 2) ❖ All other non-Caito photos: Caito ❖ Blog: caito-200-ok.com/blog/streaming-fast-and-slow ❖ “Community” photo models: Zack Hobson, Cory Johannsen Image Credits Caito_200_OK
  37. 37. ❖ https://kafkaesque.io/performance-comparison-between-apache- pulsar-and-kafka-latency/ ❖ http://pulsar.apache.org/docs/en/concepts-messaging/ Credits Caito_200_OK

×