SlideShare a Scribd company logo
1 of 88
SPRINGONE2GX
WASHINGTON, DC
Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Who Needs Batch?
By Michael Minella and Gunnar Hillert
@michaelminella, @ghillert
Michael Minella
Twitter: @michaelminella
Podcast: http://javaOffHeap.com 
or @OffHeap
Website: http://spring.io
Gunnar Hillert
Twitter: @ghillert
Website: http://blog.hillert.com
Conference: http://devnexus.com
https://github.com/ghillert/s1-2gx2015-who-needs-batch
W E H AV E A N 
ANNOUNCEMENT
WE HAVE A 
DISEASE
OBJECTIVIUS SHINIUM
SYNDROMUS
Shiny Object Syndrom
SHINY OBJECT
SYNDROME
IT MAKES US
INNOVATE
ROAD TO 
RECOVERY
COOL KIDS ARE DOING
STREAM PROCESSING
Stream Analytics
Cloud Dataflow
LET’S TAKE A
CLOSER LOOK
BATCH AND STREAM
DEFINITIONS
LOOK AT COMMON
USE CASES
CONCLUSIONS
DEFINITION:
STREAM PROCESSING
A system for processing an unbounded 
set of data in an asynchronous manor.
DEFINITION:
BATCH PROCESSING
DEPENDS ON 
WHO YOU ASK
“Batch … applications run … as special 
cases of stream processing applications”
- Apache Flink Documentation
Batch processing is the processing of
a bounded data set without interruption
or interaction
KEY
DIFFERENCES
1
BOUNDED VS
UNBOUNDED
2
SYNCHRONOUS VS
ASYNCHRONOUS
3
STATEFUL VS
STATELESS
STATE OF
STREAMING
Stream Analytics
Cloud Dataflow
Stream Analytics
Cloud Dataflow
STATE OF
STREAMING
JSR-352
WHY THE 
END OF BATCH?
LATENCY
WHAT DOES THE 
END OF BATCH MEAN?
LIMIT 
LATENCY
“ENTERPRISE GRADE”
JSR-352
LOOK AT COMMON
USE CASES
E.T.L.
DATABASE TO HDFS
DEMO
STREAMING CAN BE USEFUL
IN INGESTION USE CASES
TWITTER TO HDFS
DATA SCIENCE
BATCH TRAINING
PREDICTIVE MODELS
CONNECTED
CAR
transformerhttp filter hdfs
type-transformer
python
Gemfire
REST
Hadoop
Gemfire
STREAMING 
APROXIMIATIONS EXIST
≈
BREADTH AND ACCURACY
DON’T MATCH BATCH
≠
WORKFLOW
ORCHESTRATION
forEachDir
start
getDirectoryInfo
join
dirSubProcss dirSubProcss…
Condition 1
start
getDirAgeAndNumberOfFiles
Ingest
Condition 2
sendReminderEmail
ArchiveEnd
DEMO
HOW IS THIS DONE
VIA STREAMING?
NON-INTERACTIVE
PROCESSING
SAME AS ETL
PROCESSING
PORT SCANNER
DEMO
STREAMING 
DOESN’T MAKE SENSE
STREAMING 
DOES MAKE SENSE
WHERE LATENCY
IS A PRIORITY
DATA LOSS
IS ACCEPTABLE
UNBOUNDED
DATASET
BUT DOES NOT
IN OTHER USE CASES
HIGHLY
COMPLEX
ERROR HANDLING
NOT AS ROBUST
DATA GUARANTEES
NOT AS ROBUST
WHERE BATCH 
IS BETTER
FINITE
DATASET
ROBUST 
ERROR HANDLING
RESOURCES
ARE A CONCERN
NOT EITHER OR
COMBINING THE TWO 
PROVIDES THE MOST POWER
E.T.L.v2
mailhttp mysql2hdfs
Database to HDFS v2
DEMO
BATCH HAS LASTED
BECAUSE IT MAKSE SENSE
SPRING XD/DATA FLOW
BEST OF BOTH WORLDS
ENTERPRISE GRADE
STREAM PROCESSING
BEST BATCH OPTION
ON THE JVM
Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 88
Check out Spring Cloud Data Flow on https://spring.io
Apache Spark for Big Data Processing – Salon N-P Up next!
High Performance Stream Processing – Salon N-P 8:30AM Tomorrow
Spring Integration Extensions Ecosystem – Here! 12:45 Tomorrow
Learn More. Stay Connected.
@springcentral Spring.io/video

More Related Content

Viewers also liked

4 de noviembre
4 de noviembre4 de noviembre
4 de noviembre
linmji
 
Protocolo de toxoplasmose
Protocolo de toxoplasmoseProtocolo de toxoplasmose
Protocolo de toxoplasmose
tvf
 
Toxoplasmose congênita
Toxoplasmose congênita Toxoplasmose congênita
Toxoplasmose congênita
Amanda Thomé
 
Amigdalas e vegetações adenóides
Amigdalas e vegetações adenóidesAmigdalas e vegetações adenóides
Amigdalas e vegetações adenóides
Amanda Thomé
 
Annual Employee Performance Eval.
Annual Employee Performance Eval.Annual Employee Performance Eval.
Annual Employee Performance Eval.
April Clements
 

Viewers also liked (11)

141208 AER Public Forum - TransGrid Revenue Proposal
141208 AER Public Forum - TransGrid Revenue Proposal141208 AER Public Forum - TransGrid Revenue Proposal
141208 AER Public Forum - TransGrid Revenue Proposal
 
Simbolos patrios
Simbolos patriosSimbolos patrios
Simbolos patrios
 
TransGrid's Renewable Energy Hub
TransGrid's Renewable Energy Hub TransGrid's Renewable Energy Hub
TransGrid's Renewable Energy Hub
 
4 de noviembre
4 de noviembre4 de noviembre
4 de noviembre
 
Electricity Storage Forum 2016
Electricity Storage Forum 2016Electricity Storage Forum 2016
Electricity Storage Forum 2016
 
NSW City Rail Model Model
NSW City Rail Model ModelNSW City Rail Model Model
NSW City Rail Model Model
 
Protocolo de toxoplasmose
Protocolo de toxoplasmoseProtocolo de toxoplasmose
Protocolo de toxoplasmose
 
Toxoplasmose congênita
Toxoplasmose congênita Toxoplasmose congênita
Toxoplasmose congênita
 
2015 TAPR Forum Presentation
2015 TAPR Forum Presentation2015 TAPR Forum Presentation
2015 TAPR Forum Presentation
 
Amigdalas e vegetações adenóides
Amigdalas e vegetações adenóidesAmigdalas e vegetações adenóides
Amigdalas e vegetações adenóides
 
Annual Employee Performance Eval.
Annual Employee Performance Eval.Annual Employee Performance Eval.
Annual Employee Performance Eval.
 

Similar to s2gx2015 who needs batch

Similar to s2gx2015 who needs batch (20)

Traversing hyper driven developpement to do great technical choices and make ...
Traversing hyper driven developpement to do great technical choices and make ...Traversing hyper driven developpement to do great technical choices and make ...
Traversing hyper driven developpement to do great technical choices and make ...
 
Applied Data Science: Building a Beer Recommender | Data Science MD - Oct 2014
Applied Data Science: Building a Beer Recommender | Data Science MD - Oct 2014Applied Data Science: Building a Beer Recommender | Data Science MD - Oct 2014
Applied Data Science: Building a Beer Recommender | Data Science MD - Oct 2014
 
DevOps Roadtrip NYC
DevOps Roadtrip NYC DevOps Roadtrip NYC
DevOps Roadtrip NYC
 
Cloud computing for libraries an introduction
Cloud computing for libraries an introductionCloud computing for libraries an introduction
Cloud computing for libraries an introduction
 
Enhance system transparency and truthfulness with request tracing
Enhance system transparency and truthfulness with request tracingEnhance system transparency and truthfulness with request tracing
Enhance system transparency and truthfulness with request tracing
 
Securing a Great Developer Experience - DevOps Indonesia Meetup by Stefan Str...
Securing a Great Developer Experience - DevOps Indonesia Meetup by Stefan Str...Securing a Great Developer Experience - DevOps Indonesia Meetup by Stefan Str...
Securing a Great Developer Experience - DevOps Indonesia Meetup by Stefan Str...
 
Briefing room 20160913-ep0016-sap-anomalies-or-alerts-streaming-analytics-to-...
Briefing room 20160913-ep0016-sap-anomalies-or-alerts-streaming-analytics-to-...Briefing room 20160913-ep0016-sap-anomalies-or-alerts-streaming-analytics-to-...
Briefing room 20160913-ep0016-sap-anomalies-or-alerts-streaming-analytics-to-...
 
DevSecCon London 2017: Their-problems-are-your-problems-devseccon by Tim Kadlec
DevSecCon London 2017: Their-problems-are-your-problems-devseccon by Tim KadlecDevSecCon London 2017: Their-problems-are-your-problems-devseccon by Tim Kadlec
DevSecCon London 2017: Their-problems-are-your-problems-devseccon by Tim Kadlec
 
Seminario OpenStack - Universita' Tor Vergata - Cattedra Cloud Computing
Seminario OpenStack - Universita'  Tor Vergata - Cattedra Cloud Computing Seminario OpenStack - Universita'  Tor Vergata - Cattedra Cloud Computing
Seminario OpenStack - Universita' Tor Vergata - Cattedra Cloud Computing
 
Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)
 
Blue team reboot - HackFest
Blue team reboot - HackFest Blue team reboot - HackFest
Blue team reboot - HackFest
 
The Evolution of Blockchain and What it Means For Your Marketing Strategy
The Evolution of Blockchain and What it Means For Your Marketing StrategyThe Evolution of Blockchain and What it Means For Your Marketing Strategy
The Evolution of Blockchain and What it Means For Your Marketing Strategy
 
Kranky geek15 - securing IoT with webrtc
Kranky geek15 - securing IoT with webrtcKranky geek15 - securing IoT with webrtc
Kranky geek15 - securing IoT with webrtc
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)
 
Twitter Storm: Ereignisverarbeitung in Echtzeit
Twitter Storm: Ereignisverarbeitung in EchtzeitTwitter Storm: Ereignisverarbeitung in Echtzeit
Twitter Storm: Ereignisverarbeitung in Echtzeit
 
WJAX 2019 - Taking Distributed Tracing to the next level
WJAX 2019 - Taking Distributed Tracing to the next levelWJAX 2019 - Taking Distributed Tracing to the next level
WJAX 2019 - Taking Distributed Tracing to the next level
 
2019 Performance Monitoring and Management Trends and Insights
2019 Performance Monitoring and Management Trends and Insights2019 Performance Monitoring and Management Trends and Insights
2019 Performance Monitoring and Management Trends and Insights
 
DevSecOps and the New Path Forward
DevSecOps and the New Path ForwardDevSecOps and the New Path Forward
DevSecOps and the New Path Forward
 
Lock That Shit Down! Auth Security Patterns for Apps, APIs, and Infra - Joker...
Lock That Shit Down! Auth Security Patterns for Apps, APIs, and Infra - Joker...Lock That Shit Down! Auth Security Patterns for Apps, APIs, and Infra - Joker...
Lock That Shit Down! Auth Security Patterns for Apps, APIs, and Infra - Joker...
 
TheWriteId > components
TheWriteId > componentsTheWriteId > components
TheWriteId > components
 

More from Gunnar Hillert

More from Gunnar Hillert (16)

High Precision GPS Positioning for Spring Developers
High Precision GPS Positioning for Spring DevelopersHigh Precision GPS Positioning for Spring Developers
High Precision GPS Positioning for Spring Developers
 
Migrating to Angular 5 for Spring Developers
Migrating to Angular 5 for Spring DevelopersMigrating to Angular 5 for Spring Developers
Migrating to Angular 5 for Spring Developers
 
The Spring Update
The Spring UpdateThe Spring Update
The Spring Update
 
Ajug - The Spring Update
Ajug - The Spring UpdateAjug - The Spring Update
Ajug - The Spring Update
 
Spring Batch Performance Tuning
Spring Batch Performance TuningSpring Batch Performance Tuning
Spring Batch Performance Tuning
 
Creating Modular Test-Driven SPAs with Spring and AngularJS
Creating Modular Test-Driven SPAs with Spring and AngularJSCreating Modular Test-Driven SPAs with Spring and AngularJS
Creating Modular Test-Driven SPAs with Spring and AngularJS
 
Modular Test-driven SPAs with Spring and AngularJS
Modular Test-driven SPAs with Spring and AngularJSModular Test-driven SPAs with Spring and AngularJS
Modular Test-driven SPAs with Spring and AngularJS
 
Atlanta JUG - Integrating Spring Batch and Spring Integration
Atlanta JUG - Integrating Spring Batch and Spring IntegrationAtlanta JUG - Integrating Spring Batch and Spring Integration
Atlanta JUG - Integrating Spring Batch and Spring Integration
 
DevNexus 2013 - Introduction to WebSockets
DevNexus 2013 - Introduction to WebSocketsDevNexus 2013 - Introduction to WebSockets
DevNexus 2013 - Introduction to WebSockets
 
Introduction to WebSockets
Introduction to WebSocketsIntroduction to WebSockets
Introduction to WebSockets
 
S2GX 2012 - Spring Projects Infrastructure
S2GX 2012 - Spring Projects InfrastructureS2GX 2012 - Spring Projects Infrastructure
S2GX 2012 - Spring Projects Infrastructure
 
S2GX 2012 - What's New in Spring Integration
S2GX 2012 - What's New in Spring IntegrationS2GX 2012 - What's New in Spring Integration
S2GX 2012 - What's New in Spring Integration
 
S2GX 2012 - Introduction to Spring Integration and Spring Batch
S2GX 2012 - Introduction to Spring Integration and Spring BatchS2GX 2012 - Introduction to Spring Integration and Spring Batch
S2GX 2012 - Introduction to Spring Integration and Spring Batch
 
Spring Projects Infrastructure
Spring Projects InfrastructureSpring Projects Infrastructure
Spring Projects Infrastructure
 
Cloud Foundry for Spring Developers
Cloud Foundry for Spring DevelopersCloud Foundry for Spring Developers
Cloud Foundry for Spring Developers
 
jRecruiter - The AJUG Job Posting Service
jRecruiter - The AJUG Job Posting ServicejRecruiter - The AJUG Job Posting Service
jRecruiter - The AJUG Job Posting Service
 

Recently uploaded

Recently uploaded (20)

IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & Ireland
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptxBT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 

s2gx2015 who needs batch