SlideShare a Scribd company logo
© TNG Technology Consulting 2011 - 2017
Queryable State 
How to Build a Billing System Without a Database
 
,
 
Maximilian Bode Konstantin Knauf
© TNG Technology Consulting 2011 - 2017
Software Engineers with
TNG Technology Consulting
Focus on
Distributed Systems
Real-Time Processing
Flink in Production for 1.5 y
© TNG Technology Consulting 2011 - 2017
Agenda
1. Intro to Queryable State
2. Use Case: A Queryable Billing System
3. Live Demo of Our Prototype
© TNG Technology Consulting 2011 - 2017
Intro to Queryable State
© TNG Technology Consulting 2011 - 2017
Evolution of Streaming Architecture
© TNG Technology Consulting 2011 - 2017
λ
Stream Processor
Data
Correct Results
Previews
Query Service
Speed Layer
Cache
Batch Processor KV Store
Serving Layer
Batch Layer
© TNG Technology Consulting 2011 - 2017
κ
Stream ProcessorData Query Service
KV Store
© TNG Technology Consulting 2011 - 2017
Queryable State
Stream ProcessorData Query Service
© TNG Technology Consulting 2011 - 2017
Implementation in Flink
© TNG Technology Consulting 2011 - 2017
QueryableStateClient
KvStateLocationLookupService
LeaderRetrievalService
JobManager
LeaderElectionService
KvStateClient
TaskManager
KvStateServer
KvStateRegistry
KvStateLocationRegistry
KvStateRegistryListener
Flink Cluster Client
© TNG Technology Consulting 2011 - 2017
Use Case: “Queryable Billing”
© TNG Technology Consulting 2011 - 2017
Requirements
© TNG Technology Consulting 2011 - 2017
© TNG Technology Consulting 2011 - 2017
Robustness
ScalabilityAvailability
Correctness
© TNG Technology Consulting 2011 - 2017
Architecture
© TNG Technology Consulting 2011 - 2017
Distributed
Filesystem
Invoicing
System
© TNG Technology Consulting 2011 - 2017
Events from Kafka
Final Invoices
TimeWindow with
FoldFunction
BucketingSink
Previews
TimeWindow with
FoldFunction
CountTrigger.of(1)
Queryable State in
WindowFunction
© TNG Technology Consulting 2011 - 2017
Demo
© TNG Technology Consulting 2011 - 2017
Requirements
Invoices
Live Updates
Correctness
Availability
(Scalability)
Robustness
Late Events
TaskManager
JobManager
Sink
© TNG Technology Consulting 2011 - 2017
Invoice Generation
© TNG Technology Consulting 2011 - 2017
Requirement Demo
Requirement Demo
Invoices 
Live Updates
Correctness ()
Availability
Robustness Late Events 
TaskManager
JobManager
Sink
© TNG Technology Consulting 2011 - 2017
Live Updates
© TNG Technology Consulting 2011 - 2017
Requirement Demo
Requirement DemoInvoices 
Live Updates 
Correctness ()
Availability
Robustness Late Events 
TaskManager
JobManager
Sink
© TNG Technology Consulting 2011 - 2017
TaskManager Failure
© TNG Technology Consulting 2011 - 2017
Requirement Demo
Requirement DemoInvoices 
Live Updates 
Correctness ()
Availability ()
Robustness Late Events 
TaskManager 
JobManager
Sink
© TNG Technology Consulting 2011 - 2017
JobManager Failure
© TNG Technology Consulting 2011 - 2017
Requirement Demo
Requirement DemoInvoices 
Live Updates 
Correctness ()
Availability ()
Robustness Late Events 
TaskManager 
JobManager 
Sink
© TNG Technology Consulting 2011 - 2017
Failure of Downstream Systems
© TNG Technology Consulting 2011 - 2017
Requirement Demo
Requirement DemoInvoices 
Live Updates 
Correctness 
Availability (())
Robustness Late Events 
TaskManager 
JobManager 
Sink 
© TNG Technology Consulting 2011 - 2017
Current Limitations
Native Support for Queryable State in Windows
Improvements of Client API
State Size
Availability in Case of Job Failures
© TNG Technology Consulting 2011 - 2017
Questions?
 
 
 
maximilian.bode@tng.tech konstantin.knauf@tng.tech
lnked.in/maxbode @snntrable

More Related Content

What's hot

AVEVA World Conference NA - Greg Lawes, Point3D
AVEVA World Conference NA - Greg Lawes, Point3DAVEVA World Conference NA - Greg Lawes, Point3D
AVEVA World Conference NA - Greg Lawes, Point3D
AVEVA-Americas
 
CamundaCon 2018: Cawemo: Collaborating on Workflow Automation (Camunda)
CamundaCon 2018: Cawemo: Collaborating on Workflow Automation (Camunda)CamundaCon 2018: Cawemo: Collaborating on Workflow Automation (Camunda)
CamundaCon 2018: Cawemo: Collaborating on Workflow Automation (Camunda)
camunda services GmbH
 
Ivan Katunov. Comaqa Spring 2018. Test Design and Automation for Rest API.
Ivan Katunov. Comaqa Spring 2018. Test Design and Automation for Rest API.Ivan Katunov. Comaqa Spring 2018. Test Design and Automation for Rest API.
Ivan Katunov. Comaqa Spring 2018. Test Design and Automation for Rest API.
COMAQA.BY
 
CamundaCon 2018: The Camunda Stack: Status Quo and Further Roadmap (Camunda)
CamundaCon 2018: The Camunda Stack: Status Quo and Further Roadmap (Camunda)CamundaCon 2018: The Camunda Stack: Status Quo and Further Roadmap (Camunda)
CamundaCon 2018: The Camunda Stack: Status Quo and Further Roadmap (Camunda)
camunda services GmbH
 
Bhargava Banda
Bhargava BandaBhargava Banda
Bhargava Banda
Bhargava Rao
 
Monitor and Measure to Success: Pivotal & New Relic
 Monitor and Measure to Success: Pivotal & New Relic Monitor and Measure to Success: Pivotal & New Relic
Monitor and Measure to Success: Pivotal & New Relic
VMware Tanzu
 
Drivelt
DriveltDrivelt
Continuous Requirements Engineering - from requirements engineering to requir...
Continuous Requirements Engineering - from requirements engineering to requir...Continuous Requirements Engineering - from requirements engineering to requir...
Continuous Requirements Engineering - from requirements engineering to requir...
Stowarzyszenie Jakości Systemów Informatycznych (SJSI)
 
Ac2017 1. bit & bites
Ac2017   1. bit & bitesAc2017   1. bit & bites
Ac2017 1. bit & bites
Nesma
 
Project Experience4
Project Experience4Project Experience4
Project Experience4
ajith k
 
Engineering Services Capabilities
Engineering Services CapabilitiesEngineering Services Capabilities
Engineering Services Capabilities
senthilkumaran
 
Geode Performance Architecture for the Agile Enterprise Using Cloud Native API's
Geode Performance Architecture for the Agile Enterprise Using Cloud Native API'sGeode Performance Architecture for the Agile Enterprise Using Cloud Native API's
Geode Performance Architecture for the Agile Enterprise Using Cloud Native API's
VMware Tanzu
 
GISinc Cost Map for Water Utilities ArcGIS | Opelika Case Study
GISinc Cost Map for Water Utilities ArcGIS | Opelika Case StudyGISinc Cost Map for Water Utilities ArcGIS | Opelika Case Study
GISinc Cost Map for Water Utilities ArcGIS | Opelika Case Study
Gordian
 
Changepond Engineering Services Experiences
Changepond Engineering Services ExperiencesChangepond Engineering Services Experiences
Changepond Engineering Services Experiences
senthilkumaran
 
Don’t Let Technology Slow Down Your Digital Transformation
Don’t Let Technology Slow Down Your Digital TransformationDon’t Let Technology Slow Down Your Digital Transformation
Don’t Let Technology Slow Down Your Digital Transformation
DevOps.com
 
Greg Ward Resume
Greg Ward ResumeGreg Ward Resume
Greg Ward Resume
Greg Ward
 
B2b Project Kick Off 012208
B2b Project Kick Off 012208B2b Project Kick Off 012208
B2b Project Kick Off 012208
rapplebee
 
CMS Capabilities
CMS CapabilitiesCMS Capabilities
CMS Capabilities
Martin Smith
 
Project Controls Expo - 31st Oct 2012 - Accurate Management Reports on 1me, e...
Project Controls Expo - 31st Oct 2012 - Accurate Management Reports on 1me, e...Project Controls Expo - 31st Oct 2012 - Accurate Management Reports on 1me, e...
Project Controls Expo - 31st Oct 2012 - Accurate Management Reports on 1me, e...
Project Controls Expo
 
Testing SAP PI/PO systems Full version
Testing SAP PI/PO systems Full versionTesting SAP PI/PO systems Full version
Testing SAP PI/PO systems Full version
Daniel Graversen
 

What's hot (20)

AVEVA World Conference NA - Greg Lawes, Point3D
AVEVA World Conference NA - Greg Lawes, Point3DAVEVA World Conference NA - Greg Lawes, Point3D
AVEVA World Conference NA - Greg Lawes, Point3D
 
CamundaCon 2018: Cawemo: Collaborating on Workflow Automation (Camunda)
CamundaCon 2018: Cawemo: Collaborating on Workflow Automation (Camunda)CamundaCon 2018: Cawemo: Collaborating on Workflow Automation (Camunda)
CamundaCon 2018: Cawemo: Collaborating on Workflow Automation (Camunda)
 
Ivan Katunov. Comaqa Spring 2018. Test Design and Automation for Rest API.
Ivan Katunov. Comaqa Spring 2018. Test Design and Automation for Rest API.Ivan Katunov. Comaqa Spring 2018. Test Design and Automation for Rest API.
Ivan Katunov. Comaqa Spring 2018. Test Design and Automation for Rest API.
 
CamundaCon 2018: The Camunda Stack: Status Quo and Further Roadmap (Camunda)
CamundaCon 2018: The Camunda Stack: Status Quo and Further Roadmap (Camunda)CamundaCon 2018: The Camunda Stack: Status Quo and Further Roadmap (Camunda)
CamundaCon 2018: The Camunda Stack: Status Quo and Further Roadmap (Camunda)
 
Bhargava Banda
Bhargava BandaBhargava Banda
Bhargava Banda
 
Monitor and Measure to Success: Pivotal & New Relic
 Monitor and Measure to Success: Pivotal & New Relic Monitor and Measure to Success: Pivotal & New Relic
Monitor and Measure to Success: Pivotal & New Relic
 
Drivelt
DriveltDrivelt
Drivelt
 
Continuous Requirements Engineering - from requirements engineering to requir...
Continuous Requirements Engineering - from requirements engineering to requir...Continuous Requirements Engineering - from requirements engineering to requir...
Continuous Requirements Engineering - from requirements engineering to requir...
 
Ac2017 1. bit & bites
Ac2017   1. bit & bitesAc2017   1. bit & bites
Ac2017 1. bit & bites
 
Project Experience4
Project Experience4Project Experience4
Project Experience4
 
Engineering Services Capabilities
Engineering Services CapabilitiesEngineering Services Capabilities
Engineering Services Capabilities
 
Geode Performance Architecture for the Agile Enterprise Using Cloud Native API's
Geode Performance Architecture for the Agile Enterprise Using Cloud Native API'sGeode Performance Architecture for the Agile Enterprise Using Cloud Native API's
Geode Performance Architecture for the Agile Enterprise Using Cloud Native API's
 
GISinc Cost Map for Water Utilities ArcGIS | Opelika Case Study
GISinc Cost Map for Water Utilities ArcGIS | Opelika Case StudyGISinc Cost Map for Water Utilities ArcGIS | Opelika Case Study
GISinc Cost Map for Water Utilities ArcGIS | Opelika Case Study
 
Changepond Engineering Services Experiences
Changepond Engineering Services ExperiencesChangepond Engineering Services Experiences
Changepond Engineering Services Experiences
 
Don’t Let Technology Slow Down Your Digital Transformation
Don’t Let Technology Slow Down Your Digital TransformationDon’t Let Technology Slow Down Your Digital Transformation
Don’t Let Technology Slow Down Your Digital Transformation
 
Greg Ward Resume
Greg Ward ResumeGreg Ward Resume
Greg Ward Resume
 
B2b Project Kick Off 012208
B2b Project Kick Off 012208B2b Project Kick Off 012208
B2b Project Kick Off 012208
 
CMS Capabilities
CMS CapabilitiesCMS Capabilities
CMS Capabilities
 
Project Controls Expo - 31st Oct 2012 - Accurate Management Reports on 1me, e...
Project Controls Expo - 31st Oct 2012 - Accurate Management Reports on 1me, e...Project Controls Expo - 31st Oct 2012 - Accurate Management Reports on 1me, e...
Project Controls Expo - 31st Oct 2012 - Accurate Management Reports on 1me, e...
 
Testing SAP PI/PO systems Full version
Testing SAP PI/PO systems Full versionTesting SAP PI/PO systems Full version
Testing SAP PI/PO systems Full version
 

Similar to Flink Forward SF 2017: Konstantin Knauf & Maximilian Bode - Queryable State or How to Build a Billing System Without a Database

How to Stop Over Paying Your VMware Taxes
How to Stop Over Paying Your VMware TaxesHow to Stop Over Paying Your VMware Taxes
How to Stop Over Paying Your VMware Taxes
Veritas Technologies LLC
 
Deepak_Waghulde_CV
Deepak_Waghulde_CVDeepak_Waghulde_CV
Deepak_Waghulde_CV
Deepak Waghulde
 
Avoiding disaster recovery disasters
Avoiding disaster recovery disastersAvoiding disaster recovery disasters
Avoiding disaster recovery disasters
Alexandra Matthiesen
 
Avoiding disaster recovery disasters
Avoiding disaster recovery disastersAvoiding disaster recovery disasters
Avoiding disaster recovery disasters
Veritas Technologies LLC
 
Service Virtualization: What, Who, When, and How
Service Virtualization: What, Who, When, and HowService Virtualization: What, Who, When, and How
Service Virtualization: What, Who, When, and How
TechWell
 
TedCallowayRes 2014 BA w HC
TedCallowayRes 2014 BA w HCTedCallowayRes 2014 BA w HC
TedCallowayRes 2014 BA w HC
condor_tmc
 
Latest Resume latest uploaded wala
Latest Resume latest uploaded walaLatest Resume latest uploaded wala
Latest Resume latest uploaded wala
Amit Mishra
 
Adrian lancaster cv 2013 v1.1
Adrian lancaster cv 2013 v1.1Adrian lancaster cv 2013 v1.1
Adrian lancaster cv 2013 v1.1
lancasad
 
RamananS_Resume
RamananS_ResumeRamananS_Resume
RamananS_Resume
Ramanan Sivanarayanan
 
201302 Application Modernization kalman tiboldi
201302 Application Modernization   kalman tiboldi201302 Application Modernization   kalman tiboldi
201302 Application Modernization kalman tiboldi
Francisco Calzado
 
Torben Verdich Resume TS-SCI
Torben Verdich Resume TS-SCITorben Verdich Resume TS-SCI
Torben Verdich Resume TS-SCI
Torben Verdich, MCITP
 
Tejaswi Desai Resume ASP Dot Net WPF WCF MVC LINQ Agile
Tejaswi Desai Resume ASP Dot Net WPF WCF MVC LINQ AgileTejaswi Desai Resume ASP Dot Net WPF WCF MVC LINQ Agile
Tejaswi Desai Resume ASP Dot Net WPF WCF MVC LINQ Agile
Tejaswi Desai
 
Ranjana biswas resume
Ranjana biswas resumeRanjana biswas resume
Ranjana biswas resume
Ranjana Biswas
 
Re-Architect Your Legacy Environment To Enable An Agile, Future-Ready Enterprise
Re-Architect Your Legacy Environment To Enable An Agile, Future-Ready EnterpriseRe-Architect Your Legacy Environment To Enable An Agile, Future-Ready Enterprise
Re-Architect Your Legacy Environment To Enable An Agile, Future-Ready Enterprise
Dell World
 
Resume_Kaushik_Mitra_05November2016
Resume_Kaushik_Mitra_05November2016Resume_Kaushik_Mitra_05November2016
Resume_Kaushik_Mitra_05November2016
Kaushik Mitra
 
TUCON 2011- Identifying and resolving Middleware Issues 4x faster at 1/5th th...
TUCON 2011- Identifying and resolving Middleware Issues 4x faster at 1/5th th...TUCON 2011- Identifying and resolving Middleware Issues 4x faster at 1/5th th...
TUCON 2011- Identifying and resolving Middleware Issues 4x faster at 1/5th th...
SL Corporation
 
Prajwal Resume
Prajwal ResumePrajwal Resume
Prajwal Resume
Prajwal Gangadhar
 
Gtpl
GtplGtpl
Gtpl
mnsh_gtpl
 
Data-Driven Software Engineering for Agile Teams
Data-Driven Software Engineering for Agile TeamsData-Driven Software Engineering for Agile Teams
Data-Driven Software Engineering for Agile Teams
TechWell
 
Juniper Networks®, Tech Mahindra & CA Discuss New Perspectives & Partnership...
Juniper Networks®, Tech Mahindra & CA  Discuss New Perspectives & Partnership...Juniper Networks®, Tech Mahindra & CA  Discuss New Perspectives & Partnership...
Juniper Networks®, Tech Mahindra & CA Discuss New Perspectives & Partnership...
CA Technologies
 

Similar to Flink Forward SF 2017: Konstantin Knauf & Maximilian Bode - Queryable State or How to Build a Billing System Without a Database (20)

How to Stop Over Paying Your VMware Taxes
How to Stop Over Paying Your VMware TaxesHow to Stop Over Paying Your VMware Taxes
How to Stop Over Paying Your VMware Taxes
 
Deepak_Waghulde_CV
Deepak_Waghulde_CVDeepak_Waghulde_CV
Deepak_Waghulde_CV
 
Avoiding disaster recovery disasters
Avoiding disaster recovery disastersAvoiding disaster recovery disasters
Avoiding disaster recovery disasters
 
Avoiding disaster recovery disasters
Avoiding disaster recovery disastersAvoiding disaster recovery disasters
Avoiding disaster recovery disasters
 
Service Virtualization: What, Who, When, and How
Service Virtualization: What, Who, When, and HowService Virtualization: What, Who, When, and How
Service Virtualization: What, Who, When, and How
 
TedCallowayRes 2014 BA w HC
TedCallowayRes 2014 BA w HCTedCallowayRes 2014 BA w HC
TedCallowayRes 2014 BA w HC
 
Latest Resume latest uploaded wala
Latest Resume latest uploaded walaLatest Resume latest uploaded wala
Latest Resume latest uploaded wala
 
Adrian lancaster cv 2013 v1.1
Adrian lancaster cv 2013 v1.1Adrian lancaster cv 2013 v1.1
Adrian lancaster cv 2013 v1.1
 
RamananS_Resume
RamananS_ResumeRamananS_Resume
RamananS_Resume
 
201302 Application Modernization kalman tiboldi
201302 Application Modernization   kalman tiboldi201302 Application Modernization   kalman tiboldi
201302 Application Modernization kalman tiboldi
 
Torben Verdich Resume TS-SCI
Torben Verdich Resume TS-SCITorben Verdich Resume TS-SCI
Torben Verdich Resume TS-SCI
 
Tejaswi Desai Resume ASP Dot Net WPF WCF MVC LINQ Agile
Tejaswi Desai Resume ASP Dot Net WPF WCF MVC LINQ AgileTejaswi Desai Resume ASP Dot Net WPF WCF MVC LINQ Agile
Tejaswi Desai Resume ASP Dot Net WPF WCF MVC LINQ Agile
 
Ranjana biswas resume
Ranjana biswas resumeRanjana biswas resume
Ranjana biswas resume
 
Re-Architect Your Legacy Environment To Enable An Agile, Future-Ready Enterprise
Re-Architect Your Legacy Environment To Enable An Agile, Future-Ready EnterpriseRe-Architect Your Legacy Environment To Enable An Agile, Future-Ready Enterprise
Re-Architect Your Legacy Environment To Enable An Agile, Future-Ready Enterprise
 
Resume_Kaushik_Mitra_05November2016
Resume_Kaushik_Mitra_05November2016Resume_Kaushik_Mitra_05November2016
Resume_Kaushik_Mitra_05November2016
 
TUCON 2011- Identifying and resolving Middleware Issues 4x faster at 1/5th th...
TUCON 2011- Identifying and resolving Middleware Issues 4x faster at 1/5th th...TUCON 2011- Identifying and resolving Middleware Issues 4x faster at 1/5th th...
TUCON 2011- Identifying and resolving Middleware Issues 4x faster at 1/5th th...
 
Prajwal Resume
Prajwal ResumePrajwal Resume
Prajwal Resume
 
Gtpl
GtplGtpl
Gtpl
 
Data-Driven Software Engineering for Agile Teams
Data-Driven Software Engineering for Agile TeamsData-Driven Software Engineering for Agile Teams
Data-Driven Software Engineering for Agile Teams
 
Juniper Networks®, Tech Mahindra & CA Discuss New Perspectives & Partnership...
Juniper Networks®, Tech Mahindra & CA  Discuss New Perspectives & Partnership...Juniper Networks®, Tech Mahindra & CA  Discuss New Perspectives & Partnership...
Juniper Networks®, Tech Mahindra & CA Discuss New Perspectives & Partnership...
 

More from Flink Forward

Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...
Flink Forward
 
Evening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkEvening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in Flink
Flink Forward
 
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
Flink Forward
 
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Flink Forward
 
Introducing the Apache Flink Kubernetes Operator
Introducing the Apache Flink Kubernetes OperatorIntroducing the Apache Flink Kubernetes Operator
Introducing the Apache Flink Kubernetes Operator
Flink Forward
 
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeAutoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive Mode
Flink Forward
 
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Flink Forward
 
One sink to rule them all: Introducing the new Async Sink
One sink to rule them all: Introducing the new Async SinkOne sink to rule them all: Introducing the new Async Sink
One sink to rule them all: Introducing the new Async Sink
Flink Forward
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptx
Flink Forward
 
Flink powered stream processing platform at Pinterest
Flink powered stream processing platform at PinterestFlink powered stream processing platform at Pinterest
Flink powered stream processing platform at Pinterest
Flink Forward
 
Apache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native EraApache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native Era
Flink Forward
 
Where is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in FlinkWhere is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in Flink
Flink Forward
 
Using the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production DeploymentUsing the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production Deployment
Flink Forward
 
The Current State of Table API in 2022
The Current State of Table API in 2022The Current State of Table API in 2022
The Current State of Table API in 2022
Flink Forward
 
Flink SQL on Pulsar made easy
Flink SQL on Pulsar made easyFlink SQL on Pulsar made easy
Flink SQL on Pulsar made easy
Flink Forward
 
Dynamic Rule-based Real-time Market Data Alerts
Dynamic Rule-based Real-time Market Data AlertsDynamic Rule-based Real-time Market Data Alerts
Dynamic Rule-based Real-time Market Data Alerts
Flink Forward
 
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Exactly-Once Financial Data Processing at Scale with Flink and PinotExactly-Once Financial Data Processing at Scale with Flink and Pinot
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Flink Forward
 
Processing Semantically-Ordered Streams in Financial Services
Processing Semantically-Ordered Streams in Financial ServicesProcessing Semantically-Ordered Streams in Financial Services
Processing Semantically-Ordered Streams in Financial Services
Flink Forward
 
Tame the small files problem and optimize data layout for streaming ingestion...
Tame the small files problem and optimize data layout for streaming ingestion...Tame the small files problem and optimize data layout for streaming ingestion...
Tame the small files problem and optimize data layout for streaming ingestion...
Flink Forward
 
Batch Processing at Scale with Flink & Iceberg
Batch Processing at Scale with Flink & IcebergBatch Processing at Scale with Flink & Iceberg
Batch Processing at Scale with Flink & Iceberg
Flink Forward
 

More from Flink Forward (20)

Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...
 
Evening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkEvening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in Flink
 
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
 
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
 
Introducing the Apache Flink Kubernetes Operator
Introducing the Apache Flink Kubernetes OperatorIntroducing the Apache Flink Kubernetes Operator
Introducing the Apache Flink Kubernetes Operator
 
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeAutoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive Mode
 
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
 
One sink to rule them all: Introducing the new Async Sink
One sink to rule them all: Introducing the new Async SinkOne sink to rule them all: Introducing the new Async Sink
One sink to rule them all: Introducing the new Async Sink
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptx
 
Flink powered stream processing platform at Pinterest
Flink powered stream processing platform at PinterestFlink powered stream processing platform at Pinterest
Flink powered stream processing platform at Pinterest
 
Apache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native EraApache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native Era
 
Where is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in FlinkWhere is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in Flink
 
Using the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production DeploymentUsing the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production Deployment
 
The Current State of Table API in 2022
The Current State of Table API in 2022The Current State of Table API in 2022
The Current State of Table API in 2022
 
Flink SQL on Pulsar made easy
Flink SQL on Pulsar made easyFlink SQL on Pulsar made easy
Flink SQL on Pulsar made easy
 
Dynamic Rule-based Real-time Market Data Alerts
Dynamic Rule-based Real-time Market Data AlertsDynamic Rule-based Real-time Market Data Alerts
Dynamic Rule-based Real-time Market Data Alerts
 
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Exactly-Once Financial Data Processing at Scale with Flink and PinotExactly-Once Financial Data Processing at Scale with Flink and Pinot
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
 
Processing Semantically-Ordered Streams in Financial Services
Processing Semantically-Ordered Streams in Financial ServicesProcessing Semantically-Ordered Streams in Financial Services
Processing Semantically-Ordered Streams in Financial Services
 
Tame the small files problem and optimize data layout for streaming ingestion...
Tame the small files problem and optimize data layout for streaming ingestion...Tame the small files problem and optimize data layout for streaming ingestion...
Tame the small files problem and optimize data layout for streaming ingestion...
 
Batch Processing at Scale with Flink & Iceberg
Batch Processing at Scale with Flink & IcebergBatch Processing at Scale with Flink & Iceberg
Batch Processing at Scale with Flink & Iceberg
 

Recently uploaded

06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Timothy Spann
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
manishkhaire30
 
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
v7oacc3l
 
Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...
Bill641377
 
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
zsjl4mimo
 
Global Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headedGlobal Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headed
vikram sood
 
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
Timothy Spann
 
Intelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicineIntelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicine
AndrzejJarynowski
 
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
nyfuhyz
 
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
nuttdpt
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Kiwi Creative
 
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
74nqk8xf
 
The Ipsos - AI - Monitor 2024 Report.pdf
The  Ipsos - AI - Monitor 2024 Report.pdfThe  Ipsos - AI - Monitor 2024 Report.pdf
The Ipsos - AI - Monitor 2024 Report.pdf
Social Samosa
 
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
Social Samosa
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
aqzctr7x
 
Influence of Marketing Strategy and Market Competition on Business Plan
Influence of Marketing Strategy and Market Competition on Business PlanInfluence of Marketing Strategy and Market Competition on Business Plan
Influence of Marketing Strategy and Market Competition on Business Plan
jerlynmaetalle
 
Udemy_2024_Global_Learning_Skills_Trends_Report (1).pdf
Udemy_2024_Global_Learning_Skills_Trends_Report (1).pdfUdemy_2024_Global_Learning_Skills_Trends_Report (1).pdf
Udemy_2024_Global_Learning_Skills_Trends_Report (1).pdf
Fernanda Palhano
 
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
g4dpvqap0
 
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
g4dpvqap0
 
Analysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performanceAnalysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performance
roli9797
 

Recently uploaded (20)

06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
 
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
 
Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...
 
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
 
Global Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headedGlobal Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headed
 
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
 
Intelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicineIntelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicine
 
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
 
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
 
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
 
The Ipsos - AI - Monitor 2024 Report.pdf
The  Ipsos - AI - Monitor 2024 Report.pdfThe  Ipsos - AI - Monitor 2024 Report.pdf
The Ipsos - AI - Monitor 2024 Report.pdf
 
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
 
Influence of Marketing Strategy and Market Competition on Business Plan
Influence of Marketing Strategy and Market Competition on Business PlanInfluence of Marketing Strategy and Market Competition on Business Plan
Influence of Marketing Strategy and Market Competition on Business Plan
 
Udemy_2024_Global_Learning_Skills_Trends_Report (1).pdf
Udemy_2024_Global_Learning_Skills_Trends_Report (1).pdfUdemy_2024_Global_Learning_Skills_Trends_Report (1).pdf
Udemy_2024_Global_Learning_Skills_Trends_Report (1).pdf
 
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
 
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
 
Analysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performanceAnalysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performance
 

Flink Forward SF 2017: Konstantin Knauf & Maximilian Bode - Queryable State or How to Build a Billing System Without a Database

  • 1. © TNG Technology Consulting 2011 - 2017 Queryable State  How to Build a Billing System Without a Database   ,   Maximilian Bode Konstantin Knauf
  • 2. © TNG Technology Consulting 2011 - 2017 Software Engineers with TNG Technology Consulting Focus on Distributed Systems Real-Time Processing Flink in Production for 1.5 y
  • 3. © TNG Technology Consulting 2011 - 2017 Agenda 1. Intro to Queryable State 2. Use Case: A Queryable Billing System 3. Live Demo of Our Prototype
  • 4. © TNG Technology Consulting 2011 - 2017 Intro to Queryable State
  • 5. © TNG Technology Consulting 2011 - 2017 Evolution of Streaming Architecture
  • 6. © TNG Technology Consulting 2011 - 2017 λ Stream Processor Data Correct Results Previews Query Service Speed Layer Cache Batch Processor KV Store Serving Layer Batch Layer
  • 7. © TNG Technology Consulting 2011 - 2017 κ Stream ProcessorData Query Service KV Store
  • 8. © TNG Technology Consulting 2011 - 2017 Queryable State Stream ProcessorData Query Service
  • 9. © TNG Technology Consulting 2011 - 2017 Implementation in Flink
  • 10. © TNG Technology Consulting 2011 - 2017 QueryableStateClient KvStateLocationLookupService LeaderRetrievalService JobManager LeaderElectionService KvStateClient TaskManager KvStateServer KvStateRegistry KvStateLocationRegistry KvStateRegistryListener Flink Cluster Client
  • 11. © TNG Technology Consulting 2011 - 2017 Use Case: “Queryable Billing”
  • 12. © TNG Technology Consulting 2011 - 2017 Requirements
  • 13. © TNG Technology Consulting 2011 - 2017
  • 14. © TNG Technology Consulting 2011 - 2017 Robustness ScalabilityAvailability Correctness
  • 15. © TNG Technology Consulting 2011 - 2017 Architecture
  • 16. © TNG Technology Consulting 2011 - 2017 Distributed Filesystem Invoicing System
  • 17. © TNG Technology Consulting 2011 - 2017 Events from Kafka Final Invoices TimeWindow with FoldFunction BucketingSink Previews TimeWindow with FoldFunction CountTrigger.of(1) Queryable State in WindowFunction
  • 18. © TNG Technology Consulting 2011 - 2017 Demo
  • 19. © TNG Technology Consulting 2011 - 2017 Requirements Invoices Live Updates Correctness Availability (Scalability) Robustness Late Events TaskManager JobManager Sink
  • 20. © TNG Technology Consulting 2011 - 2017 Invoice Generation
  • 21. © TNG Technology Consulting 2011 - 2017 Requirement Demo Requirement Demo Invoices  Live Updates Correctness () Availability Robustness Late Events  TaskManager JobManager Sink
  • 22. © TNG Technology Consulting 2011 - 2017 Live Updates
  • 23. © TNG Technology Consulting 2011 - 2017 Requirement Demo Requirement DemoInvoices  Live Updates  Correctness () Availability Robustness Late Events  TaskManager JobManager Sink
  • 24. © TNG Technology Consulting 2011 - 2017 TaskManager Failure
  • 25. © TNG Technology Consulting 2011 - 2017 Requirement Demo Requirement DemoInvoices  Live Updates  Correctness () Availability () Robustness Late Events  TaskManager  JobManager Sink
  • 26. © TNG Technology Consulting 2011 - 2017 JobManager Failure
  • 27. © TNG Technology Consulting 2011 - 2017 Requirement Demo Requirement DemoInvoices  Live Updates  Correctness () Availability () Robustness Late Events  TaskManager  JobManager  Sink
  • 28. © TNG Technology Consulting 2011 - 2017 Failure of Downstream Systems
  • 29. © TNG Technology Consulting 2011 - 2017 Requirement Demo Requirement DemoInvoices  Live Updates  Correctness  Availability (()) Robustness Late Events  TaskManager  JobManager  Sink 
  • 30. © TNG Technology Consulting 2011 - 2017 Current Limitations Native Support for Queryable State in Windows Improvements of Client API State Size Availability in Case of Job Failures
  • 31. © TNG Technology Consulting 2011 - 2017 Questions?       maximilian.bode@tng.tech konstantin.knauf@tng.tech lnked.in/maxbode @snntrable