Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Kostas Tzoumas
@kostas_tzoumas
Hadoop Summit San Jose
June 6, 2016
Streaming in the Wild with
Apache FlinkTM
2
Streaming technology is enabling the
obvious: continuous processing on data that
is continuously produced
Hint: you are ...
Why embrace streaming?
 Monitor your business and react in real time
 Implement robust continuous applications
 Adopt a...
React in real time
4
Streaming versus real-time
 Streaming != Real-time
 E.g., streaming that is not real time:
continuous applications with ...
How real-time is Flink?
6
Yahoo! benchmark* data Artisans benchmarks**
* https://yahooeng.tumblr.com/post/135321837876/ben...
When and why does this matter?
 Immediate reaction to life
• E.g., generate alerts on
anomaly/pattern/special event
 Avo...
Bouygues Telecom – LUX
8
One of the largest telcos in
France. System (among
others) used for real time
diagnostics and ala...
Robust continuous
applications
9
Continuous application
 A production data application that needs to
be live 24/7 feeding other systems (perhaps
customer-...
Continuous apps with “batch”
11
file 1
file 2
Job 1
Job 2
time
file 3 Job 3
Scheduler
Serve&store
Continuous apps with “lambda”
12
file 1
file 2
Job 1
Job 2
Scheduler
Streaming job
Serve&
store
Problems with batch and λ
 Way too many moving parts (and code dup)
 Implicit treatment of time
 Out of order event han...
Continuous apps with streaming
14
Streaming job
Serve&
store
Extending the Yahoo! benchmark
 Work of Jamie Grier, inspired by a real continuous
application at Twitter
15
http://data-...
What is the use case?
 Counting!
• Tweet impressions or ad views
 Most analytics is continuous counting and
aggregations...
Requirements
 Performance: millions of events/sec, millions of
keys
 Correctness: counts correlated with timestamps
 Co...
Before Flink
 Performance: 1000s of cores needed to sustain
workload
 Correctness: time handled in application code (or
...
After Flink
 Performance: 10s of cores needed to sustain
workload
 Correctness: time handled by framework
 Consistency:...
Results (yet to be beaten!)
 Same program as Yahoo! benchmark
 30x over Storm, plus consistent results
20
Manageability
 Flink savepoints (Flink 1.0): consistent
snapshots of stateful applications
• Planned downtime for code up...
Decentralized architecture
22
Streaming and microservices
23
App App
App
local statelocal state
Archive
A decentralized architecture favors
a streaming-...
Zalando
24
Slides at http://www.slideshare.net/ZalandoTech/flink-in-zalandos-world-of-microservices-62376341
Zalando
25
Transitioning from monolithic
architecture to microservices
New BI stack
26
Flink @ Zalando (present & future)
 Business process monitoring
• Check if Zalando platform works
• Order & delivery velo...
Consolidate analytics
28
Stream Processing as a Service
 How do we make stream processing more
accessible to the data analyst?
 More familiar int...
King.com
30
King.com - RBEA
 RBEA – a platform
designed to make
stream processing
available inside
King.com
 Data scientists submit
...
Netflix
 Netflix plans to offer
Stream Processing as a
Service internally in the
company
 Currently testing Flink
and Ap...
Closing
33
Disclaimer
 A lot of this presentation is based on the work of
very talented engineers building data products
with Flink
...
More Flink tales at Hadoop Summit
35
Xiaowei Jiang
Blink−Improved Runtime for Flink and its
Application in Alibaba Search
...
Flink Forward 2016, Berlin
Submission deadline: June 30, 2016 (watch website)
Early bird deadline: July 15, 2016
www.flink...
We are hiring!
data-artisans.com/careers
Upcoming SlideShare
Loading in …5
×

Streaming in the Wild with Apache Flink

1,219 views

Published on

This talk is an application-driven walkthrough to modern stream processing, exemplified by Apache Flink, and how this enables new applications and makes old applications easier and more efficient. In this talk, we will walk through several real-world stream processing application scenarios of Apache Flink, highlighting unique features in Flink that make these applications possible. In particular, we will see (1) how support for handling out of order streams enables real-time monitoring of cloud infrastructure, (2) how the ability handle high-volume data streams with low latency SLAs enables real-time alerts in network equipment, (3) how the combination of high throughput and the ability to handle batch as a special case of streaming enables an architecture where the same exact program is used for real-time and historical data processing, and (4) how stateful stream processing can enable an architecture that eliminates the need for an external database store, leading to more than 100x performance speedup, among many other benefits.

Published in: Technology
  • Be the first to comment

Streaming in the Wild with Apache Flink

  1. 1. Kostas Tzoumas @kostas_tzoumas Hadoop Summit San Jose June 6, 2016 Streaming in the Wild with Apache FlinkTM
  2. 2. 2 Streaming technology is enabling the obvious: continuous processing on data that is continuously produced Hint: you are already doing streaming
  3. 3. Why embrace streaming?  Monitor your business and react in real time  Implement robust continuous applications  Adopt a decentralized architecture  Consolidate analytics infrastructure 3
  4. 4. React in real time 4
  5. 5. Streaming versus real-time  Streaming != Real-time  E.g., streaming that is not real time: continuous applications with large windows  E.g., real-time that is not streaming: very fast data warehousing queries  However: streaming applications can be fast 5 Streaming Real time
  6. 6. How real-time is Flink? 6 Yahoo! benchmark* data Artisans benchmarks** * https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-computation-engines-at ** http://data-artisans.com/extending-the-yahoo-streaming-benchmark/ and http://data-artisans.com/high-throughput- low-latency-and-exactly-once-stream-processing-with-apache-flink/
  7. 7. When and why does this matter?  Immediate reaction to life • E.g., generate alerts on anomaly/pattern/special event  Avoid unnecessary tradeoffs • Even if application is not latency-critical • With Flink you do not pay a price for latency! 7
  8. 8. Bouygues Telecom – LUX 8 One of the largest telcos in France. System (among others) used for real time diagnostics and alarming. Read more: http://data- artisans.com/flink-at- bouygues-html/
  9. 9. Robust continuous applications 9
  10. 10. Continuous application  A production data application that needs to be live 24/7 feeding other systems (perhaps customer-facing)  Need to be efficient, consistent, correct, and manageable  Stream processing is a great way to implement continuous applications robustly 10
  11. 11. Continuous apps with “batch” 11 file 1 file 2 Job 1 Job 2 time file 3 Job 3 Scheduler Serve&store
  12. 12. Continuous apps with “lambda” 12 file 1 file 2 Job 1 Job 2 Scheduler Streaming job Serve& store
  13. 13. Problems with batch and λ  Way too many moving parts (and code dup)  Implicit treatment of time  Out of order event handling  Implicit batch boundaries 13
  14. 14. Continuous apps with streaming 14 Streaming job Serve& store
  15. 15. Extending the Yahoo! benchmark  Work of Jamie Grier, inspired by a real continuous application at Twitter 15 http://data-artisans.com/extending-the-yahoo-streaming-benchmark/
  16. 16. What is the use case?  Counting! • Tweet impressions or ad views  Most analytics is continuous counting and aggregations grouped by dimensions • E.g., anomaly detection 16
  17. 17. Requirements  Performance: millions of events/sec, millions of keys  Correctness: counts correlated with timestamps  Consistency: counts should be correct under failures  Manageability: ability to pause & restart, reprocess, change code, etc 17
  18. 18. Before Flink  Performance: 1000s of cores needed to sustain workload  Correctness: time handled in application code (or not)  Consistency: approximate results during the day, exact results once a day (lambda)  Manageability: acceptable 18
  19. 19. After Flink  Performance: 10s of cores needed to sustain workload  Correctness: time handled by framework  Consistency: correct results on demand  Manageability: acceptable 19
  20. 20. Results (yet to be beaten!)  Same program as Yahoo! benchmark  30x over Storm, plus consistent results 20
  21. 21. Manageability  Flink savepoints (Flink 1.0): consistent snapshots of stateful applications • Planned downtime for code upgrades, maintenance, migration, debugging, etc  Monitoring (Flink 1.1)  Dynamic scaling (Flink 1.2+) 21
  22. 22. Decentralized architecture 22
  23. 23. Streaming and microservices 23 App App App local statelocal state Archive A decentralized architecture favors a streaming-based data infrastructure with local application state
  24. 24. Zalando 24 Slides at http://www.slideshare.net/ZalandoTech/flink-in-zalandos-world-of-microservices-62376341
  25. 25. Zalando 25 Transitioning from monolithic architecture to microservices
  26. 26. New BI stack 26
  27. 27. Flink @ Zalando (present & future)  Business process monitoring • Check if Zalando platform works • Order & delivery velocities • SLAs of related events  Continuous ETL • Transformation, combination, pre-aggregation • Data cleansing and validation  Complex Event Processing  Sales monitoring 27
  28. 28. Consolidate analytics 28
  29. 29. Stream Processing as a Service  How do we make stream processing more accessible to the data analyst?  More familiar interfaces • Flink 1.1 includes the first version of SQL for static data sets and data streams  Easier deployment 29
  30. 30. King.com 30
  31. 31. King.com - RBEA  RBEA – a platform designed to make stream processing available inside King.com  Data scientists submit scripts in Groovy  Flink backend executes these scripts 31 https://techblog.king.com/rbea-scalable-real-time-analytics-king/
  32. 32. Netflix  Netflix plans to offer Stream Processing as a Service internally in the company  Currently testing Flink and Apache Beam 32 http://www.slideshare.net/mdaxini/netflix-keystone-streaming-data-pipeline-scale-in-the-clouddbtb2016-62076009
  33. 33. Closing 33
  34. 34. Disclaimer  A lot of this presentation is based on the work of very talented engineers building data products with Flink  Special thanks to: • Amine Abdessemed (Bouygues Telecom) • Mihail Vieru, Javier Lopez (Zalando) • Gyula Fora, Mattias Andersson (King.com) • Monal Daxini (Netflix) 34
  35. 35. More Flink tales at Hadoop Summit 35 Xiaowei Jiang Blink−Improved Runtime for Flink and its Application in Alibaba Search Wednesday, June 29, 2016, 2:10PM - 2:50PM 210C Stephan Ewen Turning the Stream Processor into a Database: Building Online Applications on Streams Thursday, June 30, 2016, 12:20PM - 1:00PM 212
  36. 36. Flink Forward 2016, Berlin Submission deadline: June 30, 2016 (watch website) Early bird deadline: July 15, 2016 www.flink-forward.org
  37. 37. We are hiring! data-artisans.com/careers

×