Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Real-world Streaming Architectures

869 views

Published on

Presentation by Tim Berglund

Published in: Software
  • Be the first to comment

Real-world Streaming Architectures

  1. 1. 1 @tlberglund Four Streaming Data Architectures
  2. 2. 2 The Old Story
  3. 3. 3
  4. 4. 4
  5. 5. 5
  6. 6. 6
  7. 7. 7 The Streaming Story
  8. 8. 8Maria-José Viñas, NASA Earth Science News Team.
  9. 9. 9Image credit: NASA/JPL/UCSD/JSC
  10. 10. 10
  11. 11. 11 Kafka
  12. 12. 12
  13. 13. 13
  14. 14. 14
  15. 15. 15 Examples
  16. 16. 16 Yelp https://engineeringblog.yelp.com/2016/07/billions-of-messages-a-day-yelps-real-time-data-pipeline.html Justin C., Software Engineer (May, 2016)
  17. 17. 17 • 150 production services • Metcalf’s Law • REST scales poorly • N+1 problem and bulk data services A Streaming Backbone
  18. 18. 18 💩 💩 💩 💩 💩 💩 💩
  19. 19. 19
  20. 20. 20 • Stream-table duality • But some services still want databases • Individual services use MySQL • Millions of rows across services Streaming Databases
  21. 21. 21 • In 2010, commitment to MapReduce was strong • That fun was soon over • Want to capitalize on schema translation tools • Want near-real-time results • Storm didn’t support enough Python • Wrote their own! Computing Streams
  22. 22. 22 PaaStorm Instance
  23. 23. 23 Yelp Conclusions • Streaming pipeline ensures evolvability • Stream/Table Duality • Schema matters • Yelp often ahead of the game!
  24. 24. 24 LINE Corp https://engineering.linecorp.com/en/blog/detail/80
  25. 25. 25 Goals • Unified data delivery pipeline between systems • Replace a legacy, bespoke system for processing background tasks • Had to pick a framework
  26. 26. 26 Option: Apache Samza™
  27. 27. 27 Option: Apache Kafka™ Streams
  28. 28. 28 KStreamBuilder builder = new KStreamBuilder(); // Read the input Kafka topic into a KStream instance. KStream<byte[], String> textLines = builder.stream("TextLinesTopic"); // Convert to upper case (:: is Java 8 syntax) KStream<byte[], String> uppercasedWithMapValues = textLines.mapValues(String::toUpperCase); // Write the results to a new Kafka topic called "UppercasedTextLinesTopic". uppercasedWithMapValues.to("UppercasedTextLinesTopic"); KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration); streams.start(); Streaming API for Kafka
  29. 29. 29 Streaming API for Kafka Stateful Transformations
  30. 30. 30 Loopback Replicator • One big input topic of all TalkOperations (1M messages/sec) • Some services want a limited view of these • Simple filtering
  31. 31. 31 Decaton Source: LINE Corp
  32. 32. 32 LINE Conclusions • Classical Kafka Streams implementation • Better to “buy” than build
  33. 33. 33 Altspace VR
  34. 34. 34
  35. 35. 35
  36. 36. 36
  37. 37. 37
  38. 38. 38
  39. 39. 39 Altspace Architecture • Photon game engine • Real-time play • Capture for playback • An entirely classical stream processing system
  40. 40. 40 “Mirror User A to room R2” Source: Altspace
  41. 41. 41 6 months later: “Capture User A” Source: Altspace
  42. 42. 42 Playbacks • Source capture data from Kafka or S3 • “Produce” it in a timed fashion
  43. 43. 43 “Playback capture to room R2” Source: Altspace
  44. 44. 44
  45. 45. 45 The Dream Stream
  46. 46. 46
  47. 47. 47
  48. 48. 48
  49. 49. 49
  50. 50. 50
  51. 51. 51
  52. 52. 52
  53. 53. 53
  54. 54. 54
  55. 55. 55
  56. 56. 56
  57. 57. 57
  58. 58. 58
  59. 59. 59
  60. 60. 60
  61. 61. 61
  62. 62. 62
  63. 63. 63
  64. 64. 64
  65. 65. 65
  66. 66. 66
  67. 67. 67
  68. 68. 68
  69. 69. 69 Get Started with Apache Kafka Today! THE place to start with Apache Kafka! Thoroughly tested and quality assured More extensible developer experience Easy upgrade path to Confluent Enterprise https://www.confluent.io/download/
  70. 70. 70 Confluent Community Join us in Slack https://slackpass.io/confluentcommunity Find your local Meetup Group https://www.confluent.io/apache-kafka-meetups/
  71. 71. 71 Thank you @tlberglund

×