Flink Forward, profile picture
Uploaded byFlink Forward
1,287 views

Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink

The document discusses challenges and implementations related to windowless stream processing using Flink, particularly in the context of scientific data sharing through ResearchGate. Key topics include the need for fault-tolerant processing, real-time updates, and complexities in joining distributed data sources. It outlines various data models and prototypes for managing and joining accounts and publications in a streaming environment, emphasizing the issues of data consistency and accuracy.

Data & Analytics◦
Related topics:
Apache Flink•

Embed presentation

Joining Infinity — Windowless Stream Processing with Flink Sanjar Akhmedov, Software Engineer, ResearchGate
It started when two researchers discovered first- hand that collaborating with a friend or colleague on the other side of the world was no easy task. There are many variations of passages of Lorem Ipsum ResearchGate is a social network for scientists.
Connect the world of science. Make research open to all.
Structured system There are many variations of passages of Lorem Ipsum We have, and are continuing to change how scientific knowledge is shared and discovered.
10,000,000 Members 102,000,000 Publications 30,000, Visitor
Feature: Research Timeline
Feature: Research Timeline
Diverse data sources Proxy Frontend Services memcache MongoDB Solr PostgreSQL Infinispan HBaseMongoDB Solr
Big data pipeline Change data capture Import Hadoop cluster Export
Data Model Account Publication Claim 1 * Author Authorship 1*
Hypothetical SQL Publication Authorship 1* CREATE TABLE publications ( id SERIAL PRIMARY KEY, author_ids INTEGER[] ); Account Claim 1 * Author CREATE TABLE accounts ( id SERIAL PRIMARY KEY, claimed_author_ids INTEGER[] ); CREATE MATERIALIZED VIEW account_publications REFRESH FAST ON COMMIT AS SELECT accounts.id AS account_id, publications.id AS publication_id FROM accounts JOIN publications ON ANY (accounts.claimed_author_ids) = ANY (publications.author_ids);
• Data sources are distributed across different DBs • Dataset doesn’t fit in memory on a single machine • Join process must be fault tolerant • Deploy changes fast • Up-to-date join result in near real-time • Join result must be accurate Challenges
Change data capture (CDC) User Microservice DB Request Write Cache Sync Solr/ES Sync HBase/HDFS Sync
Change data capture (CDC) User Microservice DB Request Write Log K2 1 K1 4 Extract
Change data capture (CDC) User Microservice DB Request Write Log K2 1 K1 4 K1 Ø Extract
Change data capture (CDC) User Microservice DB Request Write Log K2 1 K1 4 K1 Ø KN 42 … Extract
Change data capture (CDC) User Microservice DB Cache Request Write Log K2 1 K1 4 K1 Ø KN 42 … Extract Sync
Change data capture (CDC) User Microservice DB Cache Request Write Log K2 1 K1 4 K1 Ø KN 42 … Extract Sync HBase/HDFSSolr/ES
Join two CDC streams into one NoSQL1 SQL Kafka Kafka Flink Streaming Join Kafka NoSQL2
Flink job topology Accounts Stream Join(CoFlatMap) Account Publications Publications Stream … Author 2 Author 1 Author N
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
Example dataflow Account Publications Accounts Alice 2 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Author N …
Example dataflow Account Publications Accounts Alice 2 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Author N …
Example dataflow Account Publications Accounts Alice 2 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Alice Author N …
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Alice Author N …
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Alice Author N … (Bob, 1)
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Bob Author 2 Alice Author N … (Bob, 1)
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Author 2 Alice Author N …
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Author 2 Alice Author N …
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Example dataflow Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
• ✔ Data sources are distributed across different DBs • ✔ Dataset doesn’t fit in memory on a single machine • ✔ Join process must be fault tolerant • ✔ Deploy changes fast • ✔ Up-to-date join result in near real-time • ? Join result must be accurate Challenges
Paper1 gets deleted Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Paper1 gets deleted Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Paper1 gets deleted Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N … ?
Paper1 gets deleted Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N … Need previous value
Paper1 gets deleted Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Diff with Previous State Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Paper1 gets deleted Account Publications K1 (Bob, Paper1) K1 Ø Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Diff with Previous State Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Paper1 gets deleted Account Publications K1 (Bob, Paper1) K1 Ø Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Diff with Previous State Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N … Need K1 here, e.g. K1 = 𝒇(Bob, Paper1)
Paper1 gets updated Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Paper1 gets updated Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Paper1 gets updated Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
Paper1 gets updated Account Publications K1 (Bob, Paper1) (Alice, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N … (Alice, Paper1)
Paper1 gets updated Account Publications K1 (Bob, Paper1) ?? (Alice, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N … 2. (Alice, 1)
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1)
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1) (Alice, Paper1)
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø K2 (Alice, Paper1) K2 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1) (Alice, Paper1)
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø K3 (Alice, Paper1) K2 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1) (Alice, Paper1) Pick correct natural IDs e.g. K3 = 𝒇(Alice, Author1, Paper1)
• Keep previous element state to update previous join result • Stream elements are not domain entities but commands such as delete or upsert • Joined stream must have natural IDs to propagate deletes and updates How to solve deletes and updates
Generic join graph Account Publications Accounts Stream Publications Stream Diff Alice Bob … Diff Paper1 PaperN … Join Author1 AuthorM …
Generic join graph Operate on commands Account Publications Accounts Stream Publications Stream Diff Alice Bob … Diff Paper1 PaperN … Join Author1 AuthorM …
Memory requirements Account Publications Accounts Stream Publications Stream Diff Alice Bob … Diff Paper1 PaperN … Join Author1 AuthorM … Full copy of Accounts stream Full copy of Publications stream Full copy of Accounts stream on left side Full copy of Publications stream on right side
Network load Account Publications Accounts Stream Publications Stream Diff Alice Bob … Diff Paper1 PaperN … Join Author1 AuthorM … Reshuffle Reshuffle Network Network
• In addition to handling Kafka traffic we need to reshuffle all data twice over the network • We need to keep two full copies of each joined stream in memory Resource considerations
Questions We are hiring - www.researchgate.net/careers
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Mauris pharetra interdum felis, sit amet aliquet mauris. Proin non fermentum sem. Vivamus a ligula vel arcu convallis porttitor.
Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink
Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink
Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink
Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink
Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink
Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink
Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink

More Related Content

PDF
Storing State Forever: Why It Can Be Good For Your Analytics
byYaroslav Tkachenko
 
PDF
Introducing the Apache Flink Kubernetes Operator
byFlink Forward
 
PDF
Kafka 101 and Developer Best Practices
byconfluent
 
PPTX
Using Queryable State for Fun and Profit
byFlink Forward
 
PDF
ksqlDB: A Stream-Relational Database System
byconfluent
 
PDF
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
byKai Wähner
 
PPTX
Apache Kafka at LinkedIn
byDiscover Pinterest
 
PPTX
Rds data lake @ Robinhood
byBalajiVaradarajan13
 
Storing State Forever: Why It Can Be Good For Your Analytics
byYaroslav Tkachenko
 
Introducing the Apache Flink Kubernetes Operator
byFlink Forward
 
Kafka 101 and Developer Best Practices
byconfluent
 
Using Queryable State for Fun and Profit
byFlink Forward
 
ksqlDB: A Stream-Relational Database System
byconfluent
 
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
byKai Wähner
 
Apache Kafka at LinkedIn
byDiscover Pinterest
 
Rds data lake @ Robinhood
byBalajiVaradarajan13
 

What's hot

PPTX
Autoscaling Flink with Reactive Mode
byFlink Forward
 
PDF
Building a fully managed stream processing platform on Flink at scale for Lin...
byFlink Forward
 
PDF
Intro to Telegraf
byInfluxData
 
PDF
ksqlDB - Stream Processing simplified!
byGuido Schmutz
 
PDF
Introduction to Apache Airflow
bymutt_data
 
PPTX
Tuning Apache Kafka Connectors for Flink.pptx
byFlink Forward
 
PDF
KFServing and Feast
byAnimesh Singh
 
PDF
2022-06-23 Apache Arrow and DataFusion_ Changing the Game for implementing Da...
byAndrew Lamb
 
PPTX
Evening out the uneven: dealing with skew in Flink
byFlink Forward
 
PDF
Automating Network Infrastructure : Ansible
byBangladesh Network Operators Group
 
PPTX
Using the New Apache Flink Kubernetes Operator in a Production Deployment
byFlink Forward
 
PDF
Using Spark Streaming and NiFi for the Next Generation of ETL in the Enterprise
byDataWorks Summit
 
PPTX
Apache Kafka Best Practices
byDataWorks Summit/Hadoop Summit
 
PDF
Airflow Intro-1.pdf
byBagustTriCahyo1
 
PPTX
Introduction to Kafka Connectors (Knolx).pptx
byKnoldus Inc.
 
PDF
Intro to Airflow: Goodbye Cron, Welcome scheduled workflow management
byBurasakorn Sabyeying
 
PDF
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
byGetInData
 
PPTX
Practical learnings from running thousands of Flink jobs
byFlink Forward
 
PDF
Prometheus - basics
byJuraj Hantak
 
PPTX
Introduction to Kafka Cruise Control
byJiangjie Qin
 
Autoscaling Flink with Reactive Mode
byFlink Forward
 
Building a fully managed stream processing platform on Flink at scale for Lin...
byFlink Forward
 
Intro to Telegraf
byInfluxData
 
ksqlDB - Stream Processing simplified!
byGuido Schmutz
 
Introduction to Apache Airflow
bymutt_data
 
Tuning Apache Kafka Connectors for Flink.pptx
byFlink Forward
 
KFServing and Feast
byAnimesh Singh
 
2022-06-23 Apache Arrow and DataFusion_ Changing the Game for implementing Da...
byAndrew Lamb
 
Evening out the uneven: dealing with skew in Flink
byFlink Forward
 
Automating Network Infrastructure : Ansible
byBangladesh Network Operators Group
 
Using the New Apache Flink Kubernetes Operator in a Production Deployment
byFlink Forward
 
Using Spark Streaming and NiFi for the Next Generation of ETL in the Enterprise
byDataWorks Summit
 
Apache Kafka Best Practices
byDataWorks Summit/Hadoop Summit
 
Airflow Intro-1.pdf
byBagustTriCahyo1
 
Introduction to Kafka Connectors (Knolx).pptx
byKnoldus Inc.
 
Intro to Airflow: Goodbye Cron, Welcome scheduled workflow management
byBurasakorn Sabyeying
 
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
byGetInData
 
Practical learnings from running thousands of Flink jobs
byFlink Forward
 
Prometheus - basics
byJuraj Hantak
 
Introduction to Kafka Cruise Control
byJiangjie Qin
 

Viewers also liked

PPTX
Ted Dunning-Faster and Furiouser- Flink Drift
byFlink Forward
 
PDF
Julian Hyde - Streaming SQL
byFlink Forward
 
PDF
Zoltán Zvara - Advanced visualization of Flink and Spark jobs

byFlink Forward
 
PPTX
Aljoscha Krettek - The Future of Apache Flink
byFlink Forward
 
PPTX
S. Bartoli & F. Pompermaier – A Semantic Big Data Companion
byFlink Forward
 
PDF
Jamie Grier - Robust Stream Processing with Apache Flink
byFlink Forward
 
PPTX
Kostas Tzoumas_Stephan Ewen - Keynote -The maturing data streaming ecosystem ...
byFlink Forward
 
PPTX
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
byFlink Forward
 
PPTX
Flink Case Study: OKKAM
byFlink Forward
 
PDF
Márton Balassi Streaming ML with Flink-
byFlink Forward
 
PDF
Flink Case Study: Amadeus
byFlink Forward
 
PDF
Dongwon Kim – A Comparative Performance Evaluation of Flink
byFlink Forward
 
PDF
Francesco Versaci - Flink in genomics - efficient and scalable processing of ...
byFlink Forward
 
PDF
Trevor Grant - Apache Zeppelin - A friendlier way to Flink
byFlink Forward
 
PDF
Alexander Kolb - Flinkspector – Taming the squirrel
byFlink Forward
 
PDF
Automatic Detection of Web Trackers by Vasia Kalavri
byFlink Forward
 
PDF
Maxim Fateev - Beyond the Watermark- On-Demand Backfilling in Flink
byFlink Forward
 
PDF
Ana M Martinez - AMIDST Toolbox- Scalable probabilistic machine learning with...
byFlink Forward
 
PPTX
Eron Wright - Introducing Flink on Mesos
byFlink Forward
 
PPTX
Jim Dowling - Multi-tenant Flink-as-a-Service on YARN
byFlink Forward
 
Ted Dunning-Faster and Furiouser- Flink Drift
byFlink Forward
 
Julian Hyde - Streaming SQL
byFlink Forward
 
Zoltán Zvara - Advanced visualization of Flink and Spark jobs

byFlink Forward
 
Aljoscha Krettek - The Future of Apache Flink
byFlink Forward
 
S. Bartoli & F. Pompermaier – A Semantic Big Data Companion
byFlink Forward
 
Jamie Grier - Robust Stream Processing with Apache Flink
byFlink Forward
 
Kostas Tzoumas_Stephan Ewen - Keynote -The maturing data streaming ecosystem ...
byFlink Forward
 
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
byFlink Forward
 
Flink Case Study: OKKAM
byFlink Forward
 
Márton Balassi Streaming ML with Flink-
byFlink Forward
 
Flink Case Study: Amadeus
byFlink Forward
 
Dongwon Kim – A Comparative Performance Evaluation of Flink
byFlink Forward
 
Francesco Versaci - Flink in genomics - efficient and scalable processing of ...
byFlink Forward
 
Trevor Grant - Apache Zeppelin - A friendlier way to Flink
byFlink Forward
 
Alexander Kolb - Flinkspector – Taming the squirrel
byFlink Forward
 
Automatic Detection of Web Trackers by Vasia Kalavri
byFlink Forward
 
Maxim Fateev - Beyond the Watermark- On-Demand Backfilling in Flink
byFlink Forward
 
Ana M Martinez - AMIDST Toolbox- Scalable probabilistic machine learning with...
byFlink Forward
 
Eron Wright - Introducing Flink on Mesos
byFlink Forward
 
Jim Dowling - Multi-tenant Flink-as-a-Service on YARN
byFlink Forward
 

Similar to Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink

PDF
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022
byHostedbyConfluent
 
PPTX
Apache Flink Training: DataStream API Part 2 Advanced
byFlink Forward
 
PPTX
Flink Forward Berlin 2017: Patrick Gunia - Migration of a realtime stats prod...
byFlink Forward
 
PDF
Open west 2015 talk ben coverston
bybcoverston
 
PPTX
Flink Forward San Francisco 2019: Build a Table-centric Apache Flink Ecosyste...
byFlink Forward
 
PPTX
Flink Forward San Francisco 2019: Build a Table-centric Apache Flink Ecosyste...
byFlink Forward
 
PDF
Apache Flink Stream Processing
bySuneel Marthi
 
PDF
CDC Stream Processing with Apache Flink
byTimo Walther
 
PDF
Tapping into Scientific Data with Hadoop and Flink
byMichael Häusler
 
PDF
Apache Flink - a Gentle Start
byLiangjun Jiang
 
PPTX
Stream processing - Apache flink
byRenato Guimaraes
 
PPTX
Advanced
bymxmxm
 
PDF
Flink Forward Berlin 2018: Xingcan Cui - "Stream Join in Flink: from Discrete...
byFlink Forward
 
PPTX
January 2016 Flink Community Update & Roadmap 2016
byRobert Metzger
 
PDF
Streaming Dataflow with Apache Flink
byhuguk
 
PPTX
Flink Forward SF 2017: David Hardwick, Sean Hester & David Brelloch - Dynami...
byFlink Forward
 
PDF
Introduction to Apache Flink - Fast and reliable big data processing
byTill Rohrmann
 
PPTX
Real Time Visibility with Flink
byRafi Aroch
 
PDF
Changelog Stream Processing with Apache Flink
byFlink Forward
 
PDF
Deeply Declarative Data Pipelines
byHostedbyConfluent
 
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022
byHostedbyConfluent
 
Apache Flink Training: DataStream API Part 2 Advanced
byFlink Forward
 
Flink Forward Berlin 2017: Patrick Gunia - Migration of a realtime stats prod...
byFlink Forward
 
Open west 2015 talk ben coverston
bybcoverston
 
Flink Forward San Francisco 2019: Build a Table-centric Apache Flink Ecosyste...
byFlink Forward
 
Flink Forward San Francisco 2019: Build a Table-centric Apache Flink Ecosyste...
byFlink Forward
 
Apache Flink Stream Processing
bySuneel Marthi
 
CDC Stream Processing with Apache Flink
byTimo Walther
 
Tapping into Scientific Data with Hadoop and Flink
byMichael Häusler
 
Apache Flink - a Gentle Start
byLiangjun Jiang
 
Stream processing - Apache flink
byRenato Guimaraes
 
Advanced
bymxmxm
 
Flink Forward Berlin 2018: Xingcan Cui - "Stream Join in Flink: from Discrete...
byFlink Forward
 
January 2016 Flink Community Update & Roadmap 2016
byRobert Metzger
 
Streaming Dataflow with Apache Flink
byhuguk
 
Flink Forward SF 2017: David Hardwick, Sean Hester & David Brelloch - Dynami...
byFlink Forward
 
Introduction to Apache Flink - Fast and reliable big data processing
byTill Rohrmann
 
Real Time Visibility with Flink
byRafi Aroch
 
Changelog Stream Processing with Apache Flink
byFlink Forward
 
Deeply Declarative Data Pipelines
byHostedbyConfluent
 

More from Flink Forward

PPTX
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
byFlink Forward
 
PDF
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
byFlink Forward
 
PDF
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
byFlink Forward
 
PPTX
One sink to rule them all: Introducing the new Async Sink
byFlink Forward
 
PDF
Flink powered stream processing platform at Pinterest
byFlink Forward
 
PPTX
Apache Flink in the Cloud-Native Era
byFlink Forward
 
PPTX
Where is my bottleneck? Performance troubleshooting in Flink
byFlink Forward
 
PPTX
The Current State of Table API in 2022
byFlink Forward
 
PDF
Flink SQL on Pulsar made easy
byFlink Forward
 
PPTX
Dynamic Rule-based Real-time Market Data Alerts
byFlink Forward
 
PPTX
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
byFlink Forward
 
PPTX
Processing Semantically-Ordered Streams in Financial Services
byFlink Forward
 
PDF
Tame the small files problem and optimize data layout for streaming ingestion...
byFlink Forward
 
PDF
Batch Processing at Scale with Flink & Iceberg
byFlink Forward
 
PPTX
Welcome to the Flink Community!
byFlink Forward
 
PPTX
Extending Flink SQL for stream processing use cases
byFlink Forward
 
PPTX
The top 3 challenges running multi-tenant Flink at scale
byFlink Forward
 
PPTX
Large Scale Real Time Fraudulent Web Behavior Detection
byFlink Forward
 
PPTX
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
byFlink Forward
 
PPTX
Building Reliable Lakehouses with Apache Flink and Delta Lake
byFlink Forward
 
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
byFlink Forward
 
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
byFlink Forward
 
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
byFlink Forward
 
One sink to rule them all: Introducing the new Async Sink
byFlink Forward
 
Flink powered stream processing platform at Pinterest
byFlink Forward
 
Apache Flink in the Cloud-Native Era
byFlink Forward
 
Where is my bottleneck? Performance troubleshooting in Flink
byFlink Forward
 
The Current State of Table API in 2022
byFlink Forward
 
Flink SQL on Pulsar made easy
byFlink Forward
 
Dynamic Rule-based Real-time Market Data Alerts
byFlink Forward
 
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
byFlink Forward
 
Processing Semantically-Ordered Streams in Financial Services
byFlink Forward
 
Tame the small files problem and optimize data layout for streaming ingestion...
byFlink Forward
 
Batch Processing at Scale with Flink & Iceberg
byFlink Forward
 
Welcome to the Flink Community!
byFlink Forward
 
Extending Flink SQL for stream processing use cases
byFlink Forward
 
The top 3 challenges running multi-tenant Flink at scale
byFlink Forward
 
Large Scale Real Time Fraudulent Web Behavior Detection
byFlink Forward
 
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
byFlink Forward
 
Building Reliable Lakehouses with Apache Flink and Delta Lake
byFlink Forward
 

Recently uploaded

DOCX
What Are Buy Verified Apple Pay Accounts and Why Do You Need Them_.docx
byhttps://pvaallshop.com/product/buy-verified-binance-accounts/
 
PDF
chapter one introduction to Probability theory.pdf
bySeid42
 
PDF
Unlocking the Future of Logistics & Supply Chain
byKunal Suri
 
PPTX
Supervised Machine Learning - Orange Data Mining
byandreimcpe123
 
PPTX
Top Institute for Data Analyst Course in Navi Mumbai
byPatil sagar
 
PDF
US Digital Fan Engagement Index 2026: Benchmarking Online Demand for Sports T...
byHyperset Group Ltd
 
PDF
Columbus Data and Analytics Wednesday - People Analytics 101: Making Sense of...
byTim Wilson
 
PPTX
End to End Supply Chain Management- Analytics Case Study.
byNandhuNandha1
 
PDF
Khan Traders Fish Meal – MSDS Material Safety Data Sheet
byKhan Traders Karachi
 
PDF
LECTURE - Overcoming the AI Failure Rate - AIG, DQM.pdf
bySami Laine
 
PPTX
AI IN CYBERSECURITY Best Practices for IT
byNIKHILGR3
 
PDF
WORLD-LITERATURE-2ND-SEMESTER2025-2026.pdf
byCHERIEMAYCAMO
 
PDF
Trustworthy AI : Governance of AI through Ethics
byKarim Baïna
 
PDF
Understanding Data Analytics: Concepts, Types, and Use Cases
byThe IoT Academy
 
PPTX
Organizational Structure and Design Topic 4
bydavenieralayaay
 
PDF
Analysis of Union Budget 2026-27: Major Announcements and their Impact
byDeeshiPavecha
 
PPT
Microsoft_SQL_ServerConsolidationTWP.ppt
byssuser8c3398
 
What Are Buy Verified Apple Pay Accounts and Why Do You Need Them_.docx
byhttps://pvaallshop.com/product/buy-verified-binance-accounts/
 
chapter one introduction to Probability theory.pdf
bySeid42
 
Unlocking the Future of Logistics & Supply Chain
byKunal Suri
 
Supervised Machine Learning - Orange Data Mining
byandreimcpe123
 
Top Institute for Data Analyst Course in Navi Mumbai
byPatil sagar
 
US Digital Fan Engagement Index 2026: Benchmarking Online Demand for Sports T...
byHyperset Group Ltd
 
Columbus Data and Analytics Wednesday - People Analytics 101: Making Sense of...
byTim Wilson
 
End to End Supply Chain Management- Analytics Case Study.
byNandhuNandha1
 
Khan Traders Fish Meal – MSDS Material Safety Data Sheet
byKhan Traders Karachi
 
LECTURE - Overcoming the AI Failure Rate - AIG, DQM.pdf
bySami Laine
 
AI IN CYBERSECURITY Best Practices for IT
byNIKHILGR3
 
WORLD-LITERATURE-2ND-SEMESTER2025-2026.pdf
byCHERIEMAYCAMO
 
Trustworthy AI : Governance of AI through Ethics
byKarim Baïna
 
Understanding Data Analytics: Concepts, Types, and Use Cases
byThe IoT Academy
 
Organizational Structure and Design Topic 4
bydavenieralayaay
 
Analysis of Union Budget 2026-27: Major Announcements and their Impact
byDeeshiPavecha
 
Microsoft_SQL_ServerConsolidationTWP.ppt
byssuser8c3398
 

Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink

  • 1.
    Joining Infinity —Windowless Stream Processing with Flink Sanjar Akhmedov, Software Engineer, ResearchGate
  • 2.
    It started whentwo researchers discovered first- hand that collaborating with a friend or colleague on the other side of the world was no easy task. There are many variations of passages of Lorem Ipsum ResearchGate is a social network for scientists.
  • 3.
    Connect the worldof science. Make research open to all.
  • 4.
    Structured system There aremany variations of passages of Lorem Ipsum We have, and are continuing to change how scientific knowledge is shared and discovered.
  • 5.
  • 6.
  • 7.
  • 8.
    Diverse data sources Proxy Frontend Services memcacheMongoDB Solr PostgreSQL Infinispan HBaseMongoDB Solr
  • 9.
  • 10.
  • 11.
    Hypothetical SQL Publication Authorship 1* CREATE TABLEpublications ( id SERIAL PRIMARY KEY, author_ids INTEGER[] ); Account Claim 1 * Author CREATE TABLE accounts ( id SERIAL PRIMARY KEY, claimed_author_ids INTEGER[] ); CREATE MATERIALIZED VIEW account_publications REFRESH FAST ON COMMIT AS SELECT accounts.id AS account_id, publications.id AS publication_id FROM accounts JOIN publications ON ANY (accounts.claimed_author_ids) = ANY (publications.author_ids);
  • 12.
    • Data sourcesare distributed across different DBs • Dataset doesn’t fit in memory on a single machine • Join process must be fault tolerant • Deploy changes fast • Up-to-date join result in near real-time • Join result must be accurate Challenges
  • 13.
    Change data capture(CDC) User Microservice DB Request Write Cache Sync Solr/ES Sync HBase/HDFS Sync
  • 14.
    Change data capture(CDC) User Microservice DB Request Write Log K2 1 K1 4 Extract
  • 15.
    Change data capture(CDC) User Microservice DB Request Write Log K2 1 K1 4 K1 Ø Extract
  • 16.
    Change data capture(CDC) User Microservice DB Request Write Log K2 1 K1 4 K1 Ø KN 42 … Extract
  • 17.
    Change data capture(CDC) User Microservice DB Cache Request Write Log K2 1 K1 4 K1 Ø KN 42 … Extract Sync
  • 18.
    Change data capture(CDC) User Microservice DB Cache Request Write Log K2 1 K1 4 K1 Ø KN 42 … Extract Sync HBase/HDFSSolr/ES
  • 19.
    Join two CDCstreams into one NoSQL1 SQL Kafka Kafka Flink Streaming Join Kafka NoSQL2
  • 20.
  • 21.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 22.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 23.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 24.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 25.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 26.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 27.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 28.
    Example dataflow Account Publications Accounts Alice2 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Author N …
  • 29.
    Example dataflow Account Publications Accounts Alice2 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Author N …
  • 30.
    Example dataflow Account Publications Accounts Alice2 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Alice Author N …
  • 31.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Alice Author N …
  • 32.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Alice Author N … (Bob, 1)
  • 33.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Bob Author 2 Alice Author N … (Bob, 1)
  • 34.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Author 2 Alice Author N …
  • 35.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Author 2 Alice Author N …
  • 36.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 37.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 38.
    Example dataflow Account Publications K1(Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 39.
    • ✔ Datasources are distributed across different DBs • ✔ Dataset doesn’t fit in memory on a single machine • ✔ Join process must be fault tolerant • ✔ Deploy changes fast • ✔ Up-to-date join result in near real-time • ? Join result must be accurate Challenges
  • 40.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 41.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 42.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N … ?
  • 43.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N … Need previous value
  • 44.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Diff with Previous State Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 45.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) K1 Ø Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Diff with Previous State Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 46.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) K1 Ø Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Diff with Previous State Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N … Need K1 here, e.g. K1 = 𝒇(Bob, Paper1)
  • 47.
    Paper1 gets updated AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 48.
    Paper1 gets updated AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 49.
    Paper1 gets updated AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
  • 50.
    Paper1 gets updated AccountPublications K1 (Bob, Paper1) (Alice, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N … (Alice, Paper1)
  • 51.
    Paper1 gets updated AccountPublications K1 (Bob, Paper1) ?? (Alice, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
  • 52.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
  • 53.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
  • 54.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
  • 55.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N …
  • 56.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N …
  • 57.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N …
  • 58.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N … 2. (Alice, 1)
  • 59.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1)
  • 60.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1) (Alice, Paper1)
  • 61.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø K2 (Alice, Paper1) K2 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1) (Alice, Paper1)
  • 62.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø K3 (Alice, Paper1) K2 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1) (Alice, Paper1) Pick correct natural IDs e.g. K3 = 𝒇(Alice, Author1, Paper1)
  • 63.
    • Keep previouselement state to update previous join result • Stream elements are not domain entities but commands such as delete or upsert • Joined stream must have natural IDs to propagate deletes and updates How to solve deletes and updates
  • 64.
  • 65.
    Generic join graph Operateon commands Account Publications Accounts Stream Publications Stream Diff Alice Bob … Diff Paper1 PaperN … Join Author1 AuthorM …
  • 66.
    Memory requirements Account Publications Accounts Stream Publications Stream Diff Alice Bob … Diff Paper1 PaperN … Join Author1 AuthorM … Full copyof Accounts stream Full copy of Publications stream Full copy of Accounts stream on left side Full copy of Publications stream on right side
  • 67.
  • 68.
    • In additionto handling Kafka traffic we need to reshuffle all data twice over the network • We need to keep two full copies of each joined stream in memory Resource considerations
  • 69.
    Questions We are hiring- www.researchgate.net/careers
  • 72.
    Lorem ipsum dolorsit amet, consectetur adipiscing elit. Mauris pharetra interdum felis, sit amet aliquet mauris. Proin non fermentum sem. Vivamus a ligula vel arcu convallis porttitor.