Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Sweet Streams Are Made of These: Data Driven Development for Stream Processing

The strength of a powerful stream processing engine is in how fast, and how much data it can process. This naturally adds complexity to existing integration points and can lead to development overhead.

Luckily, there is a set of data-driven development principles that are built to alleviate precisely these challenges. This talk will go over what these are and how to apply them at various points throughout the development process, using real world successes (and failures!) as examples.

Although the examples are for highly complex systems, this talk will be beginner friendly and applicable to non-streaming use cases.

  • Be the first to comment

  • Be the first to like this

Sweet Streams Are Made of These: Data Driven Development for Stream Processing

  1. 1. © 2020 Ververica 1
  2. 2. © 2020 Ververica 2 ● Caito Scherr Introduction
  3. 3. © 2020 Ververica 3 ● Caito Scherr ● Developer Advocate Introduction
  4. 4. © 2020 Ververica 4 ● Caito Scherr ● Developer Advocate ● Ververica, GmbH Introduction
  5. 5. © 2020 Ververica 5 ● Caito Scherr ● Developer Advocate ● Ververica, GmbH ● Portland, OR, USA Introduction
  6. 6. © 2020 Ververica 6 Introduction Introduction
  7. 7. © 2020 Ververica 7 ● The Challenge ● “Metrics-Driven Metrics” ● Metrics as a Shared Language Agenda
  8. 8. © 2020 Ververica 8 ● The Challenge ● “Metrics-Driven Metrics” ● Metrics as a Shared Language Agenda
  9. 9. © 2020 Ververica 9 ● The Challenge ● “Metrics-Driven Metrics” ● Metrics as a Shared Language Agenda
  10. 10. © 2020 Ververica
  11. 11. © 2020 Ververica 11 Before We Start...
  12. 12. © 2020 Ververica 12
  13. 13. © 2020 Ververica 13 The Challenge
  14. 14. © 2020 Ververica 14 The Challenge
  15. 15. © 2020 Ververica Metrics-Driven Metrics
  16. 16. © 2020 Ververica 16 Metrics-Driven Metrics “You can measure almost anything, but you cannot pay attention to everything”
  17. 17. © 2020 Ververica 17 Metrics-Driven Metrics “Any situation where people create their own dashboards without structure, quickly starts to look like the cockpit of a 747”
  18. 18. © 2020 Ververica 18 Metrics-Driven Metrics
  19. 19. © 2020 Ververica 19 Metrics-Driven Metrics
  20. 20. © 2020 Ververica 20 Metrics-Driven Metrics The dashboard should be: • Meaningful • Iterative • Accessible
  21. 21. © 2020 Ververica 21 Metrics-Driven Metrics The dashboard should reflect your: • Roadmap • Highest Risk • Most uncertain metrics
  22. 22. © 2020 Ververica 22 Metrics-Driven Metrics
  23. 23. © 2020 Ververica 23 Metrics-Driven Metrics
  24. 24. © 2020 Ververica 24 Metrics-Driven Metrics
  25. 25. © 2020 Ververica 25 Metrics-Driven Metrics
  26. 26. © 2020 Ververica 26 Metrics-Driven Metrics
  27. 27. © 2020 Ververica 27 Metrics-Driven Metrics Prometheus DataDog
  28. 28. © 2020 Ververica 28 Metrics-Driven Metrics
  29. 29. © 2020 Ververica 29 Metrics-Driven Metrics
  30. 30. © 2020 Ververica 30 Metrics-Driven Metrics Highest Risk
  31. 31. © 2020 Ververica 31 Metrics-Driven Metrics
  32. 32. © 2020 Ververica 32 Metrics-Driven Metrics Most Uncertain
  33. 33. © 2020 Ververica 33 Metrics-Driven Metrics
  34. 34. © 2020 Ververica 34 Metrics-Driven Metrics Current Roadmap Priority
  35. 35. © 2020 Ververica 35 Metrics-Driven Metrics Priority #2
  36. 36. © 2020 Ververica 36 Metrics-Driven Metrics Priority #2 Current Roadmap Priority Most Uncertain Highest Risk
  37. 37. © 2020 Ververica 37
  38. 38. © 2020 Ververica 38 Shared Language “Even the best metrics driven development fails easily, when it’s only built for yourself”
  39. 39. © 2020 Ververica 39 Shared Language Implementation: • Identify your impacted groups • Identify the most effective tools for each group • Enable automation
  40. 40. © 2020 Ververica 40 Shared Language
  41. 41. © 2020 Ververica 41 Shared Language
  42. 42. © 2020 Ververica 42 Shared Language
  43. 43. © 2020 Ververica 43 Shared Language
  44. 44. © 2020 Ververica 44 Shared Language
  45. 45. © 2020 Ververica 45 Shared Language
  46. 46. © 2020 Ververica 46 Shared Language
  47. 47. © 2020 Ververica 47 Shared Language
  48. 48. © 2020 Ververica 48 Shared Language
  49. 49. © 2020 Ververica 49 Shared Language
  50. 50. © 2020 Ververica 50 Shared Language
  51. 51. © 2020 Ververica 51 Shared Language
  52. 52. © 2020 Ververica 52 Shared Language
  53. 53. © 2020 Ververica 53
  54. 54. © 2020 Ververica 54 Conclusion For systems with complex integration points: • Metrics-Driven Metrics: streamline technical complexity • Metrics as a Shared Language: streamline interpersonal complications
  55. 55. © 2020 Ververica 55 Twitter Caito_200_OK Content https://medium.com/@caito http://caito-200-ok.com/ Email Caito@ververica.com Introduction
  56. 56. © 2020 Ververica ● YOW! Conference staff!! ● “Shared Language” photo models: ○ Zack Hobson, CTO ○ Cory Johannsen, Senior Software Engineer 56 Thank You! @Caito_200_OK Quoted Contributors & Reviewers: ● Mandy Riso - DevOps Engineer ● Eric Shamow - Engineering Manager ● Mike Hix - Senior Software Engineer ● Ben Ford - Product Manager ● Noçnica Fee - Developer Advocate ● Lucy Wyman - Senior Software Engineer ● Logan Ballard - Site Reliability Engineer
  57. 57. © 2020 Ververica Resources ● Prometheus push gateway - when to use it/docuemntation: https://prometheus.io/docs/practices/pushing/ ● Prometheus push gateway on GitHub: https://github.com/prometheus/pushgateway ● Prometheus Flink example: https://github.com/mbode/flink-prometheus-example ● Flink monitoring: https://flink.apache.org/news/2019/02/25/monitoring-best-practices.html 57 @Caito_200_OK
  58. 58. © 2020 Ververica Credits - Images ● Flink logos: https://wints.github.io/flink-web//community.html ● Data-Driven/Aware/Informed Design: https://uxdesign.cc/becoming-a-data-aware-designer-1d7614ebc3ed ● Stream processing diagram: https://www.ververica.com/what-is-stream-processing ● 747 Airplane cockpit: https://www.reddit.com/r/pics/comments/5vv8qt/the_pilots_seat_and_cockpit_of_a_boein g_747/ ● DataDog Flink dashboard: https://www.datadoghq.com/blog/monitor-apache-flink-with-datadog/ ● All other photos & Images: Caito Scherr 58 @Caito_200_OK
  59. 59. © 2020 Ververica Credits ● Basic Data-Driven Development principles https://www.portable.com.au/reports/principles-of-data-driven-design ● Data-driven design: https://www.springboard.com/blog/data-driven-design/ ● Becoming A Data-Aware Designer, + definition image - Illustration from “Designing with Data” by King, Churchill, & Tan https://uxdesign.cc/becoming-a-data-aware-designer-1d7614ebc3ed ● https://digitalprinciples.org/wp-content/uploads/PDD_Principle-BeDataDriven_v2.p df ● Harvard Business Review - Data Driven Culture: https://hbr.org/2020/02/10-steps-to-creating-a-data-driven-culture 59 @Caito_200_OK

    Be the first to comment

The strength of a powerful stream processing engine is in how fast, and how much data it can process. This naturally adds complexity to existing integration points and can lead to development overhead. Luckily, there is a set of data-driven development principles that are built to alleviate precisely these challenges. This talk will go over what these are and how to apply them at various points throughout the development process, using real world successes (and failures!) as examples. Although the examples are for highly complex systems, this talk will be beginner friendly and applicable to non-streaming use cases.

Views

Total views

79

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

0

Shares

0

Comments

0

Likes

0

×