Robust Stream Processing with Apache Flink

•

5 likes•2,320 views

The document discusses Apache Flink, an open source platform for distributed stream and batch data processing. It describes how Flink allows for stateful stream processing, windowed computations over streams, and robust handling of time, failures, planned downtimes, and reprocessing. The presentation provides an overview of these concepts and includes a code example and details on Flink's distributed and parallel deployment architecture.

Data & Analytics

Robust
Stream Processing
with
Apache Flink
Jamie Grier
@jamiegrier
jamie@data-artisans.com

Who am I?
• Director of Applications Engineering at data
Artisans
• Previously working on streaming computation at
Twitter, Gnip and Boulder Imaging
• Involved in various kinds of stream processing for
about a decade
• High-speed video, social media streaming, general
frameworks for stream processing

Overview
• What is Apache Flink?
• What is Stateful Stream Processing?
• Windowed computation over streams
• Robust Time Handling (Event Time vs Processing Time)
• Robust Failure Handling
• Robust Planned Downtime Handling
• Robust Reprocessing

What is
Apache Flink?
Apache Flink is an open source platform for distributed
stream and batch data processing.

Stream Processing
Your
Code
Data Stream Data Stream

Stateful
Stream Processing
Your
Code
Data Stream Data Stream
State

More Complex
Example
Kafka
Files
Rabb
itMQ
Filter
Map
Join /
Sum
Influx
DB
C*

Distributed and Parallel
Deployment
Kafka
Files
Rabb
it
MQ
Filter
Parse
Join /
Sum
Influx
DB
C*

Robust Stream Processing
with Apache Flink

Windowing in Processing
Time
0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9
Processing Time
Event Time

Windowing in Event
Time
0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9
Event Time

We’re Hiring!
http://data-artisans.com/careers

Flink Forward 2016, Berlin
Submission deadline: June 30, 2016
Early bird deadline: July 15, 2016
www.flink-forward.org

What's hot

Big Data WarsawMaximilian Michels

Streaming in the Wild with Apache FlinkKostas Tzoumas

Ted Dunning-Faster and Furiouser- Flink DriftFlink Forward

Flink 1.0-slidesJamie Grier

data Artisans Product AnnouncementFlink Forward

Robust Stream Processing With Apache FlinkJamie Grier

Stream Processing with Apache FlinkC4Media

Flink. Pure StreamingIndizen Technologies

Extending the Yahoo Streaming Benchmark + MapR BenchmarksJamie Grier

The Evolution of (Open Source) Data ProcessingAljoscha Krettek

Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache ZeppelinFlink Forward

Introduction to Stream Processing with Apache Flink (2019-11-02 Bengaluru Mee...Timo Walther

Apache Flink 101 - the rise of stream processing and beyondBowen Li

A look at Flink 1.2Stefan Richter

Aljoscha Krettek - Apache Flink for IoT: How Event-Time Processing Enables Ea...Ververica

Abstractions for managed stream processing platform (Arya Ketan - Flipkart)KafkaZone

Bay Area Apache Flink Meetup Community Update August 2015Henry Saputra

The Past, Present, and Future of Apache Flink®Aljoscha Krettek

Building a real time Tweet map with Flink in six weeksMatthias Kricke

Apache Flink and what it is used forAljoscha Krettek

What's hot (20)

Big Data Warsaw

Streaming in the Wild with Apache Flink

Ted Dunning-Faster and Furiouser- Flink Drift

Flink 1.0-slides

data Artisans Product Announcement

Robust Stream Processing With Apache Flink

Stream Processing with Apache Flink

Flink. Pure Streaming

Extending the Yahoo Streaming Benchmark + MapR Benchmarks

The Evolution of (Open Source) Data Processing

Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache Zeppelin

Introduction to Stream Processing with Apache Flink (2019-11-02 Bengaluru Mee...

Apache Flink 101 - the rise of stream processing and beyond

A look at Flink 1.2

Aljoscha Krettek - Apache Flink for IoT: How Event-Time Processing Enables Ea...

Abstractions for managed stream processing platform (Arya Ketan - Flipkart)

Bay Area Apache Flink Meetup Community Update August 2015

The Past, Present, and Future of Apache Flink®

Building a real time Tweet map with Flink in six weeks

Apache Flink and what it is used for

Similar to Robust Stream Processing with Apache Flink

Robust Stream Processing with Apache FlinkJamie Grier

Robust stream processing with Apache FlinkAljoscha Krettek

Trend Micro Big Data Platform and Apache BigtopEvans Ye

Big Data Analytics Platforms by KTH and RISE SICSBig Data Value Association

Flink in actionArtem Semenenko

Near real-time anomaly detection at Lyftmarkgrover

A Practical Guide to Selecting a Stream Processing Technology confluent

Cloud lunch and learn real-time streaming in azureTimothy Spann

Data Stream Processing with Apache FlinkFabian Hueske

Scylla Summit 2022: Learning Rust the Hard Way for a Production Kafka+ScyllaD...ScyllaDB

Streaming in the Wild with Apache FlinkDataWorks Summit/Hadoop Summit

Stinger.Next by Alan Gates of HortonworksData Con LA

Hail hydrate! from stream to lake using open sourceTimothy Spann

Berlin Apache Flink Meetup May 2015, Community UpdateRobert Metzger

The Purpose of Cloud-technology for Video ProductionMaarten Verwaest

Flink forward-2017-netflix keystones-paasMonal Daxini

Reintroducing the Stream Processor: A universal tool for continuous data anal...Paris Carbone

Getting started with apache flink streaming apiPreetdeep Kumar

An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.Data Con LA

Stream Processing with Apache Flink (Flink.tw Meetup 2016/07/19)Apache Flink Taiwan User Group

Similar to Robust Stream Processing with Apache Flink (20)

Robust Stream Processing with Apache Flink

Robust stream processing with Apache Flink

Trend Micro Big Data Platform and Apache Bigtop

Big Data Analytics Platforms by KTH and RISE SICS

Flink in action

Near real-time anomaly detection at Lyft

A Practical Guide to Selecting a Stream Processing Technology

Cloud lunch and learn real-time streaming in azure

Data Stream Processing with Apache Flink

Scylla Summit 2022: Learning Rust the Hard Way for a Production Kafka+ScyllaD...

Streaming in the Wild with Apache Flink

Stinger.Next by Alan Gates of Hortonworks

Hail hydrate! from stream to lake using open source

Berlin Apache Flink Meetup May 2015, Community Update

The Purpose of Cloud-technology for Video Production

Flink forward-2017-netflix keystones-paas

Reintroducing the Stream Processor: A universal tool for continuous data anal...

Getting started with apache flink streaming api

An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.

Stream Processing with Apache Flink (Flink.tw Meetup 2016/07/19)

Recently uploaded

Delhi 99530 vip 56974 Genuine Escort Service Call Girls in Kishangarh9953056974 Low Rate Call Girls In Saket, Delhi NCR

EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM TRACKING WITH GOOGLE ANALYTICS.pptxthyngster

Predicting Employee Churn: A Data-Driven Approach Project PresentationBoston Institute of Analytics

Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...Jack DiGiovanna

Log Analysis using OSSEC sasoasasasas.pptxJohnnyPlasten

定制英国白金汉大学毕业证（UCB毕业证书）成绩单原版一比一ffjhghh

Delhi Call Girls CP 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Callshivangimorya083

Kantar AI Summit- Under Embargo till Wednesday, 24th April 2024, 4 PM, IST.pdfSocial Samosa

Ukraine War presentation: KNOW THE BASICSAishani27

High Class Call Girls Noida Sector 39 Aarushi 🔝8264348440🔝 Independent Escort...soniya singh

FESE Capital Markets Fact Sheet 2024 Q1.pdfMarinCaroMartnezBerg

VIP Call Girls Service Miyapur Hyderabad Call +91-8250192130Suhani Kapoor

VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...Suhani Kapoor

Full night 🥵 Call Girls Delhi New Friends Colony {9711199171} Sanya Reddy ✌️o...shivangimorya083

Beautiful Sapna Vip Call Girls Hauz Khas 9711199012 Call /Whatsappssapnasaifi408

(PARI) Call Girls Wanowrie ( 7001035870 ) HI-Fi Pune Escorts Serviceranjana rawat

Unveiling Insights: The Role of a Data AnalystSamantha Rae Coolbeth

VIP Call Girls Service Charbagh { Lucknow Call Girls Service 9548273370 } Boo...Call Girls In Delhi Whatsup 9873940964 Enjoy Unlimited Pleasure

From idea to production in a day – Leveraging Azure ML and Streamlit to build...Florian Roscheck

Brighton SEO | April 2024 | Data StorytellingNeil Barnes

Recently uploaded (20)

Delhi 99530 vip 56974 Genuine Escort Service Call Girls in Kishangarh

EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM TRACKING WITH GOOGLE ANALYTICS.pptx

Predicting Employee Churn: A Data-Driven Approach Project Presentation

Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...

Log Analysis using OSSEC sasoasasasas.pptx

定制英国白金汉大学毕业证（UCB毕业证书）成绩单原版一比一

Delhi Call Girls CP 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call

Kantar AI Summit- Under Embargo till Wednesday, 24th April 2024, 4 PM, IST.pdf

Ukraine War presentation: KNOW THE BASICS

High Class Call Girls Noida Sector 39 Aarushi 🔝8264348440🔝 Independent Escort...

FESE Capital Markets Fact Sheet 2024 Q1.pdf

VIP Call Girls Service Miyapur Hyderabad Call +91-8250192130

VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...

Full night 🥵 Call Girls Delhi New Friends Colony {9711199171} Sanya Reddy ✌️o...

Beautiful Sapna Vip Call Girls Hauz Khas 9711199012 Call /Whatsapps

(PARI) Call Girls Wanowrie ( 7001035870 ) HI-Fi Pune Escorts Service

Unveiling Insights: The Role of a Data Analyst

VIP Call Girls Service Charbagh { Lucknow Call Girls Service 9548273370 } Boo...

From idea to production in a day – Leveraging Azure ML and Streamlit to build...

Brighton SEO | April 2024 | Data Storytelling

Robust Stream Processing with Apache Flink

1. Robust Stream Processing with Apache Flink Jamie Grier @jamiegrier jamie@data-artisans.com

2. Who am I? • Director of Applications Engineering at data Artisans • Previously working on streaming computation at Twitter, Gnip and Boulder Imaging • Involved in various kinds of stream processing for about a decade • High-speed video, social media streaming, general frameworks for stream processing

3. Overview • What is Apache Flink? • What is Stateful Stream Processing? • Windowed computation over streams • Robust Time Handling (Event Time vs Processing Time) • Robust Failure Handling • Robust Planned Downtime Handling • Robust Reprocessing

4. What is Apache Flink? Apache Flink is an open source platform for distributed stream and batch data processing.

5. What is Apache Flink?

6. Stream Processing Your Code Data Stream Data Stream

7. Stateful Stream Processing Your Code Data Stream Data Stream State

8. More Complex Example Kafka Files Rabb itMQ Filter Map Join / Sum Influx DB C*

9. Distributed and Parallel Deployment Kafka Files Rabb it MQ Filter Parse Join / Sum Influx DB C*

10. Robust Stream Processing with Apache Flink

11. Code Example!

12. Windowing

13. Processing Time vs Event Time

14. Windowing in Processing Time 0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9 Processing Time Event Time

15. Windowing in Event Time 0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9 Event Time