SlideShare a Scribd company logo
1 of 15
1
Aljoscha Krettek
@aljoscha
December 8, 2016
Google Singapore
Apache Flink™
A Stream Processing Engine
2
Original creators of Apache
Flink®
Providers of the
dA Platform, a supported
Flink distribution
What is Apache Flink?
3
Distributed Deployment
Stateful Stream Processing
DataStream API, Windows
Table API
Building Block
Core API
Declarative DSL
Basis
Stream Processing
4
Data Stream Data Stream
Your
code
Stateful Stream Processing
5
State
Data Stream Data Stream
Your
code
Stateful Steam Processing
6
Source
Filter /
Transform
State
read/write
Sink
Stateful Steam Processing
7
Scalable embedded state
Access at memory speed &
scales with parallel operators
Stateful Steam Processing
8
Re-load state
Reset positions
in input streams
Rolling back computation
Re-processing
Stateful Steam Processing
9
Restore to different
programs
Bugfixes, Upgrades, A/B testing, etc
Beam/Flink Interactions
10
Dataflow/
MillWheel influenced
runs on
Code Example!
11
Based on http://data-artisans.com/robust-stream-processing-flink-walkthrough/
1
Thank you!
@aljoscha
@ApacheFlink
@dataArtisans
http://flink.apache.org
Appendix
13
We are hiring!
data-artisans.com/careers
15
Flink Forward 2017
San Francisco Berlin
• April 10-11, 2017
• The first Flink Forward event
outside of Berlin
• Register at sf.flink-forward.org/
• September 11-13
• Over 350 attendees at the
last event
• Registration opening soon!

More Related Content

What's hot

What's hot (20)

Rails 3.1 sneak peak
Rails 3.1 sneak peakRails 3.1 sneak peak
Rails 3.1 sneak peak
 
Coprocessors - Uses, Abuses, Solutions - presented at HBaseCon East 2016
Coprocessors - Uses, Abuses, Solutions - presented at HBaseCon East 2016Coprocessors - Uses, Abuses, Solutions - presented at HBaseCon East 2016
Coprocessors - Uses, Abuses, Solutions - presented at HBaseCon East 2016
 
SPARQList
SPARQListSPARQList
SPARQList
 
Apache Phoenix Query Server
Apache Phoenix Query ServerApache Phoenix Query Server
Apache Phoenix Query Server
 
Running Apache Zeppelin production
Running Apache Zeppelin productionRunning Apache Zeppelin production
Running Apache Zeppelin production
 
De-Mystifying the Apache Phoenix QueryServer
De-Mystifying the Apache Phoenix QueryServerDe-Mystifying the Apache Phoenix QueryServer
De-Mystifying the Apache Phoenix QueryServer
 
(ATS6-PLAT04) Query service
(ATS6-PLAT04) Query service (ATS6-PLAT04) Query service
(ATS6-PLAT04) Query service
 
Building Your First Apache Apex (Next Gen Big Data/Hadoop) Application
Building Your First Apache Apex (Next Gen Big Data/Hadoop) ApplicationBuilding Your First Apache Apex (Next Gen Big Data/Hadoop) Application
Building Your First Apache Apex (Next Gen Big Data/Hadoop) Application
 
Request-Response Cycle of Ruby on Rails App
Request-Response Cycle of Ruby on Rails AppRequest-Response Cycle of Ruby on Rails App
Request-Response Cycle of Ruby on Rails App
 
Installation of sap HANA DB system on SUSE Linux
Installation of sap HANA DB system on SUSE LinuxInstallation of sap HANA DB system on SUSE Linux
Installation of sap HANA DB system on SUSE Linux
 
apidays LIVE India - REST the Events - REST APIs for Event-Driven Architectur...
apidays LIVE India - REST the Events - REST APIs for Event-Driven Architectur...apidays LIVE India - REST the Events - REST APIs for Event-Driven Architectur...
apidays LIVE India - REST the Events - REST APIs for Event-Driven Architectur...
 
(ATS6-DEV06) Using Packages for Protocol, Component, and Application Delivery
(ATS6-DEV06) Using Packages for Protocol, Component, and Application Delivery(ATS6-DEV06) Using Packages for Protocol, Component, and Application Delivery
(ATS6-DEV06) Using Packages for Protocol, Component, and Application Delivery
 
(ATS6-PLAT07) Managing AEP in an enterprise environment
(ATS6-PLAT07) Managing AEP in an enterprise environment(ATS6-PLAT07) Managing AEP in an enterprise environment
(ATS6-PLAT07) Managing AEP in an enterprise environment
 
(ATS6-GS02) Integrating Contur and HEOS
(ATS6-GS02) Integrating Contur and HEOS(ATS6-GS02) Integrating Contur and HEOS
(ATS6-GS02) Integrating Contur and HEOS
 
Plugging holes in your SharePoint 2010 disaster recovery strategy
Plugging holes in your SharePoint 2010 disaster recovery strategyPlugging holes in your SharePoint 2010 disaster recovery strategy
Plugging holes in your SharePoint 2010 disaster recovery strategy
 
(ATS6-PLAT02) Accelrys Catalog and Protocol Validation
(ATS6-PLAT02) Accelrys Catalog and Protocol Validation(ATS6-PLAT02) Accelrys Catalog and Protocol Validation
(ATS6-PLAT02) Accelrys Catalog and Protocol Validation
 
Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForwa...
Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForwa...Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForwa...
Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForwa...
 
Tale of two streaming frameworks (Karthik D - Walmart)
Tale of two streaming frameworks (Karthik D - Walmart)Tale of two streaming frameworks (Karthik D - Walmart)
Tale of two streaming frameworks (Karthik D - Walmart)
 
Flink history, roadmap and vision
Flink history, roadmap and visionFlink history, roadmap and vision
Flink history, roadmap and vision
 
C++ shared libraries and loading
C++ shared libraries and loadingC++ shared libraries and loading
C++ shared libraries and loading
 

Viewers also liked

Projet_Krutarth Desai_A20387996
Projet_Krutarth Desai_A20387996Projet_Krutarth Desai_A20387996
Projet_Krutarth Desai_A20387996
Krutarth Desai
 
Resume for Priscilla Green
Resume for Priscilla GreenResume for Priscilla Green
Resume for Priscilla Green
Priscilla Green
 
PartnershipBrochure_forprint
PartnershipBrochure_forprintPartnershipBrochure_forprint
PartnershipBrochure_forprint
Stacia Wompierski
 
Ken Hughes Resume
Ken Hughes ResumeKen Hughes Resume
Ken Hughes Resume
KEN HUGHES
 
luisfernandez controles formulario vf
luisfernandez controles formulario vfluisfernandez controles formulario vf
luisfernandez controles formulario vf
lafp36
 
PRESENTACIÓN FINAL DE LA CARTILLA
PRESENTACIÓN FINAL DE LA CARTILLAPRESENTACIÓN FINAL DE LA CARTILLA
PRESENTACIÓN FINAL DE LA CARTILLA
Alejita Sanchez
 
Curso : Robótica 6º - prof. Ronald Estela
Curso : Robótica 6º - prof. Ronald EstelaCurso : Robótica 6º - prof. Ronald Estela
Curso : Robótica 6º - prof. Ronald Estela
ronaldestelafencyt
 
BE Project Fortnight Reports on IVRS
BE Project Fortnight Reports on IVRSBE Project Fortnight Reports on IVRS
BE Project Fortnight Reports on IVRS
Abhishek Nadkarni
 

Viewers also liked (18)

Pergamonski muzej u berlinu
Pergamonski muzej u berlinuPergamonski muzej u berlinu
Pergamonski muzej u berlinu
 
Projet_Krutarth Desai_A20387996
Projet_Krutarth Desai_A20387996Projet_Krutarth Desai_A20387996
Projet_Krutarth Desai_A20387996
 
Synergy
SynergySynergy
Synergy
 
Phân tích tình hình tài chính tại công ty cổ phần my way hospitality
Phân tích tình hình tài chính tại công ty cổ phần my way hospitalityPhân tích tình hình tài chính tại công ty cổ phần my way hospitality
Phân tích tình hình tài chính tại công ty cổ phần my way hospitality
 
Rapport de stage Agence de développement du Val-de-Marne
Rapport de stage Agence de développement du Val-de-MarneRapport de stage Agence de développement du Val-de-Marne
Rapport de stage Agence de développement du Val-de-Marne
 
Ada #2
Ada #2Ada #2
Ada #2
 
Tallerpractico10 subgrupo 4 grupo 2 experiencia significativa sobre el átomo
Tallerpractico10 subgrupo 4  grupo 2 experiencia significativa sobre el átomoTallerpractico10 subgrupo 4  grupo 2 experiencia significativa sobre el átomo
Tallerpractico10 subgrupo 4 grupo 2 experiencia significativa sobre el átomo
 
Resume for Priscilla Green
Resume for Priscilla GreenResume for Priscilla Green
Resume for Priscilla Green
 
PartnershipBrochure_forprint
PartnershipBrochure_forprintPartnershipBrochure_forprint
PartnershipBrochure_forprint
 
Reflection
ReflectionReflection
Reflection
 
Ken Hughes Resume
Ken Hughes ResumeKen Hughes Resume
Ken Hughes Resume
 
luisfernandez controles formulario vf
luisfernandez controles formulario vfluisfernandez controles formulario vf
luisfernandez controles formulario vf
 
Ada #3 b3
Ada #3 b3Ada #3 b3
Ada #3 b3
 
Aaj ka trend 09.12.16
Aaj ka trend 09.12.16Aaj ka trend 09.12.16
Aaj ka trend 09.12.16
 
PRESENTACIÓN FINAL DE LA CARTILLA
PRESENTACIÓN FINAL DE LA CARTILLAPRESENTACIÓN FINAL DE LA CARTILLA
PRESENTACIÓN FINAL DE LA CARTILLA
 
Chap011 2
Chap011 2Chap011 2
Chap011 2
 
Curso : Robótica 6º - prof. Ronald Estela
Curso : Robótica 6º - prof. Ronald EstelaCurso : Robótica 6º - prof. Ronald Estela
Curso : Robótica 6º - prof. Ronald Estela
 
BE Project Fortnight Reports on IVRS
BE Project Fortnight Reports on IVRSBE Project Fortnight Reports on IVRS
BE Project Fortnight Reports on IVRS
 

Similar to Apache Flink - A Stream Processing Engine

Similar to Apache Flink - A Stream Processing Engine (20)

Building Scalable Data Pipelines - 2016 DataPalooza Seattle
Building Scalable Data Pipelines - 2016 DataPalooza SeattleBuilding Scalable Data Pipelines - 2016 DataPalooza Seattle
Building Scalable Data Pipelines - 2016 DataPalooza Seattle
 
Apache flink
Apache flinkApache flink
Apache flink
 
Apache flink
Apache flinkApache flink
Apache flink
 
Apache flink
Apache flinkApache flink
Apache flink
 
Apache Flink Online Training
Apache Flink Online TrainingApache Flink Online Training
Apache Flink Online Training
 
Apache Flink Training: System Overview
Apache Flink Training: System OverviewApache Flink Training: System Overview
Apache Flink Training: System Overview
 
A Data Streaming Architecture with Apache Flink (berlin Buzzwords 2016)
A Data Streaming Architecture with Apache Flink (berlin Buzzwords 2016)A Data Streaming Architecture with Apache Flink (berlin Buzzwords 2016)
A Data Streaming Architecture with Apache Flink (berlin Buzzwords 2016)
 
Building scalable data with kafka and spark
Building scalable data with kafka and sparkBuilding scalable data with kafka and spark
Building scalable data with kafka and spark
 
E2E Data Pipeline - Apache Spark/Airflow/Livy
E2E Data Pipeline - Apache Spark/Airflow/LivyE2E Data Pipeline - Apache Spark/Airflow/Livy
E2E Data Pipeline - Apache Spark/Airflow/Livy
 
Apache Flink: Past, Present and Future
Apache Flink: Past, Present and FutureApache Flink: Past, Present and Future
Apache Flink: Past, Present and Future
 
Real time cloud native open source streaming of any data to apache solr
Real time cloud native open source streaming of any data to apache solrReal time cloud native open source streaming of any data to apache solr
Real time cloud native open source streaming of any data to apache solr
 
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
 
Flink in action
Flink in actionFlink in action
Flink in action
 
Apache Flink Berlin Meetup May 2016
Apache Flink Berlin Meetup May 2016Apache Flink Berlin Meetup May 2016
Apache Flink Berlin Meetup May 2016
 
Robust stream processing with Apache Flink
Robust stream processing with Apache FlinkRobust stream processing with Apache Flink
Robust stream processing with Apache Flink
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 
Unified Batch and Real-Time Stream Processing Using Apache Flink
Unified Batch and Real-Time Stream Processing Using Apache FlinkUnified Batch and Real-Time Stream Processing Using Apache Flink
Unified Batch and Real-Time Stream Processing Using Apache Flink
 
GOTO Night Amsterdam - Stream processing with Apache Flink
GOTO Night Amsterdam - Stream processing with Apache FlinkGOTO Night Amsterdam - Stream processing with Apache Flink
GOTO Night Amsterdam - Stream processing with Apache Flink
 
SnapLogic- iPaaS (Elastic Integration Cloud and Data Integration)
SnapLogic- iPaaS (Elastic Integration Cloud and Data Integration) SnapLogic- iPaaS (Elastic Integration Cloud and Data Integration)
SnapLogic- iPaaS (Elastic Integration Cloud and Data Integration)
 
Present and future of unified, portable, and efficient data processing with A...
Present and future of unified, portable, and efficient data processing with A...Present and future of unified, portable, and efficient data processing with A...
Present and future of unified, portable, and efficient data processing with A...
 

More from Aljoscha Krettek

Unified stateful big data processing in Apache Beam (incubating)
Unified stateful big data processing in Apache Beam (incubating)Unified stateful big data processing in Apache Beam (incubating)
Unified stateful big data processing in Apache Beam (incubating)
Aljoscha Krettek
 

More from Aljoscha Krettek (15)

Apache Flink(tm) - A Next-Generation Stream Processor
Apache Flink(tm) - A Next-Generation Stream ProcessorApache Flink(tm) - A Next-Generation Stream Processor
Apache Flink(tm) - A Next-Generation Stream Processor
 
Talk Python To Me: Stream Processing in your favourite Language with Beam on ...
Talk Python To Me: Stream Processing in your favourite Language with Beam on ...Talk Python To Me: Stream Processing in your favourite Language with Beam on ...
Talk Python To Me: Stream Processing in your favourite Language with Beam on ...
 
The Evolution of (Open Source) Data Processing
The Evolution of (Open Source) Data ProcessingThe Evolution of (Open Source) Data Processing
The Evolution of (Open Source) Data Processing
 
Apache Flink and what it is used for
Apache Flink and what it is used forApache Flink and what it is used for
Apache Flink and what it is used for
 
The Past, Present, and Future of Apache Flink®
The Past, Present, and Future of Apache Flink®The Past, Present, and Future of Apache Flink®
The Past, Present, and Future of Apache Flink®
 
(Past), Present, and Future of Apache Flink
(Past), Present, and Future of Apache Flink(Past), Present, and Future of Apache Flink
(Past), Present, and Future of Apache Flink
 
Python Streaming Pipelines with Beam on Flink
Python Streaming Pipelines with Beam on FlinkPython Streaming Pipelines with Beam on Flink
Python Streaming Pipelines with Beam on Flink
 
The Past, Present, and Future of Apache Flink
The Past, Present, and Future of Apache FlinkThe Past, Present, and Future of Apache Flink
The Past, Present, and Future of Apache Flink
 
Unified stateful big data processing in Apache Beam (incubating)
Unified stateful big data processing in Apache Beam (incubating)Unified stateful big data processing in Apache Beam (incubating)
Unified stateful big data processing in Apache Beam (incubating)
 
Stream processing for the practitioner: Blueprints for common stream processi...
Stream processing for the practitioner: Blueprints for common stream processi...Stream processing for the practitioner: Blueprints for common stream processi...
Stream processing for the practitioner: Blueprints for common stream processi...
 
Advanced Flink Training - Design patterns for streaming applications
Advanced Flink Training - Design patterns for streaming applicationsAdvanced Flink Training - Design patterns for streaming applications
Advanced Flink Training - Design patterns for streaming applications
 
Adventures in Timespace - How Apache Flink Handles Time and Windows
Adventures in Timespace - How Apache Flink Handles Time and WindowsAdventures in Timespace - How Apache Flink Handles Time and Windows
Adventures in Timespace - How Apache Flink Handles Time and Windows
 
Flink 0.10 - Upcoming Features
Flink 0.10 - Upcoming FeaturesFlink 0.10 - Upcoming Features
Flink 0.10 - Upcoming Features
 
Data Analysis with Apache Flink (Hadoop Summit, 2015)
Data Analysis with Apache Flink (Hadoop Summit, 2015)Data Analysis with Apache Flink (Hadoop Summit, 2015)
Data Analysis with Apache Flink (Hadoop Summit, 2015)
 
Apache Flink Hands-On
Apache Flink Hands-OnApache Flink Hands-On
Apache Flink Hands-On
 

Recently uploaded

%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
masabamasaba
 

Recently uploaded (20)

%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
 
WSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go Platformless
 
tonesoftg
tonesoftgtonesoftg
tonesoftg
 
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
 
WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
 
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
 
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
 
VTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learnVTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learn
 
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
 
Announcing Codolex 2.0 from GDK Software
Announcing Codolex 2.0 from GDK SoftwareAnnouncing Codolex 2.0 from GDK Software
Announcing Codolex 2.0 from GDK Software
 
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
 
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park %in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
 
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
 
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
 
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
 
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
Direct Style Effect Systems -The Print[A] Example- A Comprehension AidDirect Style Effect Systems -The Print[A] Example- A Comprehension Aid
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
 
Artyushina_Guest lecture_YorkU CS May 2024.pptx
Artyushina_Guest lecture_YorkU CS May 2024.pptxArtyushina_Guest lecture_YorkU CS May 2024.pptx
Artyushina_Guest lecture_YorkU CS May 2024.pptx
 
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
 
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdfMicrosoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
 

Apache Flink - A Stream Processing Engine