SlideShare a Scribd company logo
1 of 96
Kai Wähner
Technology Evangelist
kontakt@kai-waehner.de
LinkedIn
@KaiWaehner
www.kai-waehner.de
O’Reilly Software Architecture Conference 2016 (London, UK)
How to apply big data analytics and machine learning
to real-time processing of microservice events
© Copyright 2000-2016 TIBCO Software Inc.
Digital Transformation - Physical and Digital Worlds are Merging
© Copyright 2000-2016 TIBCO Software Inc.
Apply Big Data Analytics to Real Time Processing
© Copyright 2000-2016 TIBCO Software Inc.
Analyze and Act on Critical Business Moments
© Copyright 2000-2016 TIBCO Software Inc.
Key Take-Aways
Ø Insights are hidden in Historical Data on Big Data Platforms
Ø Machine Learning and Big Data Analytics find these Insights by building Analytics Models
Ø Event Processing uses these Models (without Redevelopment) to take Action in Real Time
© Copyright 2000-2016 TIBCO Software Inc.
Agenda
1) Machine Learning and Big Data Analytics
2) Building an Analytic Model
3) Real Time Processing
4) Live Demo
5) Intelligent Microservices
© Copyright 2000-2016 TIBCO Software Inc.
Agenda
1) Machine Learning and Big Data Analytics
2) Building an Analytic Model
3) Real Time Processing
4) Live Demo
5) Intelligent Microservices
Machine Learning
…. allows computers to find hidden insights without being
explicitly programmed where to look.
Real World Examples of Machine Learning
Spam Detection
Search Results +
Product Recommendation
Picture Detection
(Friends, Locations, Products)
Machine Learning is already present in daily life…
Now, every enterprise is beginning to leverage it!
The Next Disruption:
Google Beats Go Champion
© Copyright 2000-2016 TIBCO Software Inc.
Example: Decision Tree – Titanic Survival Rate
family size
Wikipedia
Decision Tree – Product Pass / Fail by Equipment Sensor Readings
Bad Product
Good Product
Step 8 Temperature
< 122 C >= 122 C
Step 2 Recipe
A B
Step 11 Pressure
TV Color Display Problem
Decision Tree – Training and Test Data Sets
© Copyright 2000-2016 TIBCO Software Inc.
Ensemble Tree Algorithms
• Random Forest, Gradient Boosting Machine (GBM)
• Method – Average many simple trees
• Sample the data: fit a simple tree
• Re-sample the data; up-weighting the observations that weren’t fitted well in
previous model
• Continue adding trees until fit is good
• Save all the trees and average them
• Better fit + prediction than single trees
© Copyright 2000-2016 TIBCO Software Inc.
Closed Loop for Big Data Analytics
© Copyright 2000-2016 TIBCO Software Inc.
Analytics Maturity Model
Immediate
Long-Term	
Competitive	AdvantageValue to the Organization
A good Big Data Analytics platform can provide value to the organization
across the full spectrum of use cases
Self-service	
Dashboards
Event	Processing	Advanced	Analytics
Measure Diagnose Predict Optimize Alert Automate
Analytics Maturity
Visual	Analytics Event	Processing	
Analytics
© Copyright 2000-2016 TIBCO Software Inc.
Analytics Maturity Model
Immediate
Long-Term	
Competitive	AdvantageValue to the Organization
Visual	Analytics Event	Processing	Advanced	Analytics
Measure Diagnose Predict Optimize Alert Automate
Analytics Maturity
A good Big Data Analytics platform can provide value to the organization
across the full spectrum of use cases
Analytics
© Copyright 2000-2016 TIBCO Software Inc.
Analytics Maturity Model
Immediate
Long-Term	
Competitive	AdvantageValue to the Organization
Self-service	
Dashboards
Event	Processing	Advanced	Analytics
Measure Diagnose Predict Optimize Alert Automate
Analytics Maturity
A good Big Data Analytics platform can provide value to the organization
across the full spectrum of use cases
Visual	Analytics Event	Processing	
Analytics
© Copyright 2000-2016 TIBCO Software Inc.
The first task in a new analytics projects
is to define a Business Case!
© Copyright 2000-2016 TIBCO Software Inc.
From a Business Case to Proactive Actions
Model
Present	
Data Wrangling Signals Dashboards
SAP
Historian
Production
Well
Filter
Enrich
Merge
Shape
Explore
Clean
Assemble	DataBusiness	Case
Increase	
Productivity
Grow	
Revenue
Completions
Visualize GeoLocation
Production
Value	
Theses
Reduce	Risk
G&G
Equipment
Decision,	Action	
Prediction Action
Develop	Model
Pressure
Temperature
Production
Interrupt
Drill	Bit	
Movement
Equipment
Failure
© Copyright 2000-2016 TIBCO Software Inc.
Agenda
1) Machine Learning and Big Data Analytics
2) Building an Analytic Model
3) Real Time Processing
4) Live Demo
5) Intelligent Microservices
© Copyright 2000-2016 TIBCO Software Inc.
Analytical Pipeline
© Copyright 2000-2016 TIBCO Software Inc.
Analytics Maturity Model
Immediate
Long-Term	
Competitive	AdvantageValue to the Organization
A good Big Data Analytics platform can provide value to the organization
across the full spectrum of use cases
Self-service	
Dashboards
Event	Processing	Advanced	Analytics
Measure Diagnose Predict Optimize Alert Automate
Analytics Maturity
Visual	Analytics Event	Processing	
Analytics
What is Predictive Analytics?
© Copyright 2000-2016 TIBCO Software Inc.
Analytical Pipeline
© Copyright 2000-2016 TIBCO Software Inc.
Variety of Data in Enterprises
Custom	GUI-driven	
data	access	via	SDK
Siebel
eBusiness
Local	data	sources
AccessExcel STDF
Drag-and-drop
MySQL
SQL	Server
Oracle
Information	Services
(join,	transform,	reusable,	
parameterized,	dynamic	query	
for	in-memory	use)
Databases
JDBC/ODBC
Hadoop
SFDC
PostgreSQL
Teradata
Netezza
Etc.XML
RDBMS
Flat
Files
Spread-
sheets
Web
Services
Oracle
E-Business
RDBMS
RDBMS
RDBMS
SAP BWSAP R/3 D
A
T
A
F
A
B
R
I
C
Salesforce
ODBC
OLE	DB
SqlClient
Direct	
connection
Oracle
TeradataAsterMS	SSAS
Teradata
Direct	Query
(dynamically	query	and	retrieve	data	for	
visualization	and	analysis)
Databases
MySQL
Etc.
OBIEE
Netezza
Hadoop
© Copyright 2000-2016 TIBCO Software Inc.
Data Acquisition
“Smart Recommendation Engine”
© Copyright 2000-2016 TIBCO Software Inc.
Analytical Pipeline
© Copyright 2000-2016 TIBCO Software Inc.
Data Munging / Wrangling / Mash-up
cust_id dept sku dollar gift date
1 104 C 12003 2.40 FALSE 2016-10-17
2 105 A 12005 62.85 FALSE 2016-10-17
3 102 C 12007 69.23 TRUE 2016-10-17
4 104 B 12004 9.33 FALSE 2016-10-18
5 105 C 12010 14.16 TRUE 2016-10-18
6 101 B 12003 90.43 FALSE 2016-10-19
7 103 C 12005 90.97 FALSE 2016-10-19
n … … … … … …
cust_id A B C total # orders first_dat
e
last_dat
e
1 100 21.76 23.67 0.00 45.43 2 2016-10-
19
2016-10-
20
2 101 0.01 74.65 0.00 74.66 3 2016-10-
19
2016-10-
20
3 102 0.00 60.92 50.29 111.21 6 2016-10-
17
2016-10-
20
4 103 0.00 0.00 52.30 52.30 2 2016-10-
19
2016-10-
20© Copyright 2000-2016 TIBCO Software Inc.
Data Munging - Transformations
© Copyright 2000-2016 TIBCO Software Inc.
Analytical Pipeline
© Copyright 2000-2016 TIBCO Software Inc.
Exploratory Data Analysis
Exploratory Data Analysis (EDA) is
an approach/philosophy for data
analysis that employs a variety of
techniques (mostly graphical)
1. to maximize insight into a data set
2. uncover underlying structure
3. extract important variables
4. detect outliers and anomalies
5. test underlying assumptions
6. develop parsimonious models
7. determine optimal factor settings
© Copyright 2000-2016 TIBCO Software Inc.
Exploratory Data Analysis
“The greatest value of a picture
is when it forces us to notice
what we never expected to see”
John W. Tukey, 1977
© Copyright 2000-2016 TIBCO Software Inc.
Exploratory Data Analysis
Visual Analytics - Interactive Brush-Linked
© Copyright 2000-2016 TIBCO Software Inc.
… and “Inline Data Wrangling” à Ad-hoc data preparation instead of just ETL
© Copyright 2000-2016 TIBCO Software Inc.
Analytics Maturity Model
Immediate
Long-Term	
Competitive	AdvantageValue to the Organization
Visual	Analytics Event	Processing	Advanced	Analytics
Measure Diagnose Predict Optimize Alert Automate
Analytics Maturity
A good Big Data Analytics platform can provide value to the organization
across the full spectrum of use cases
Analytics
What is Predictive Analytics?
© Copyright 2000-2016 TIBCO Software Inc.
Analytical Pipeline
© Copyright 2000-2016 TIBCO Software Inc.
Which picture represents a model?
A model is a simplification of the truth that helps you with decision making.
© Copyright 2000-2016 TIBCO Software Inc.
Model Building
© Copyright 2000-2016 TIBCO Software Inc.
Model Building
Employees who write longer emails earn higher salaries!
© Copyright 2000-2016 TIBCO Software Inc.
Model Building
© Copyright 2000-2016 TIBCO Software Inc.
Model Improvement
Managers
Staff
© Copyright 2000-2016 TIBCO Software Inc.
Model Improvement
© Copyright 2000-2016 TIBCO Software Inc.
Analytical Pipeline
© Copyright 2000-2016 TIBCO Software Inc.
Model Validation
How is the IQ of a kid related to the IQ of his / her mum?
© Copyright 2000-2016 TIBCO Software Inc.
Frameworks and Tooling
© Copyright 2000-2016 TIBCO Software Inc.
“…as a next-generation data discovery capability that automatically finds and explains
insights from advanced analytics to business users or citizen data scientists”
Smart Data Discovery (for the Business User)
Leverage Machine Learning
without the help of a Data Scientist
Advanced Analytics and Big Data Tools (for Data Scientists)
Many more ….
R Language
• Built for data scientists
• Very active community
© Copyright 2000-2016 TIBCO Software Inc.
R with Revolution Analytics (now Microsoft)
© Copyright 2000-2016 TIBCO Software Inc.
Open Source GPL License
(including its restrictions) http://www.revolutionanalytics.com/webinars/introducing-revolution-r-open-enhanced-open-source-r-distribution-
revolution-analytics
TIBCO has rewritten R as a Commercial Compute Engine
• Latest statistics scripting engine: S a S-PLUS® a R a TERR
• Runs R code including CRAN packages
Engine internals rebuilt from scratch at low-level
• Redesigned data objects, memory management
• High performance + Big Data
TERR is licensed from TIBCO
• TERR Installs (free) with Spotfire Analyst / Desktop + other TIBCO products
• Spotfire Server can manage all TERR / R scripts, artifacts for reuse
• Standalone Developer Edition
• Supported by TIBCO
• No GPL license issues
© Copyright 2000-2016 TIBCO Software Inc.
TERR - TIBCO’s Enterprise Runtime for R
Which R to use?
© Copyright 2000-2016 TIBCO Software Inc.
http://www.forbes.com/sites/danwoods/2016/01/27/microsofts-revolution-analytics-acquisition-is-the-wrong-
way-to-embrace-r/
© Copyright 2000-2016 TIBCO Software Inc.
Apache Spark
General	Data-processing	Framework
à However,	focus is	especially	on	Analytics (at least	these days)
Apache Spark MLlib
© Copyright 2000-2016 TIBCO Software Inc.
Spark ML is Spark’s machine
learning library.
Its goal is to make practical
machine learning scalable and easy.
It consists of common learning
algorithms and utilities, including
classification, regression, clustering
and collaborative filtering.
General	Data-processing	Framework
à However,	focus is	especially	on	Analytics	(at	least	these	days)
x
© Copyright 2000-2016 TIBCO Software Inc.
H2O.ai
An Extensible Open Source Platform for
Analytics
• Best of Breed Open Source Technology
• Easy-to-use Web UI and Familiar Interfaces
• Data Agnostic Support for all Common
Database and File Types
• Massively Scalable Big Data Analysis
• Real-time Data Scoring (“Nanofast Scoring
Engine”)
http://www.h2o.ai/
TIBCO Spotfire with R / TERR Integration
© Copyright 2000-2016 TIBCO Software Inc.
Let the business user leverage Analytic Models (created by the Data Scientist) to find insights!
Example: Customer Churn with Random Forest Algorithm
• ‘refresh model’ button lives a ‘random forest algorithm’
• requires no a priori assumptions at all, it just always works
• The business user doesn’t need to know what random forest is to be empowered by it
Select variables
for the model
TIBCO Spotfire with H2O Integration
© Copyright 2000-2016 TIBCO Software Inc.
Example: Predictive Analytics for Manufacturing (“scrap parts as early as possible”)
TIBCO Spotfire with H2O Integration
© Copyright 2000-2016 TIBCO Software Inc.
Example: Predictive Analytics for Manufacturing (“scrap parts as early as possible”)
© Copyright 2000-2016 TIBCO Software Inc.
SaaS Machine Learning
• Managed SaaS service for building ML models and generating predictions
• Integrated into the corresponding cloud ecosystem
• Easy to use, but limited feature set and potential latency issues if combined
with external data or applications
http://docs.aws.amazon.com/machine-learning/latest/dg/tutoria
© Copyright 2000-2016 TIBCO Software Inc.
PMML (Predictive Model Markup Language )
• XML-based de facto standard to represent predictive analytic models
• Developed by the Data Mining Group (DMG)
• Easily share models between PMML compliant applications
(e.g. between model creation and deployment for operations)
© Copyright 2000-2016 TIBCO Software Inc.
Agenda
1) Machine Learning and Big Data Analytics
2) Building an Analytic Model
3) Real Time Processing
4) Live Demo
5) Intelligent Microservices
© Copyright 2000-2016 TIBCO Software Inc.
Analytics Maturity Model
Immediate
Long-Term	
Competitive	AdvantageValue to the Organization
Self-service	
Dashboards
Event	Processing	Advanced	Analytics
Measure Diagnose Predict Optimize Alert Automate
Analytics Maturity
A good Big Data Analytics platform can provide value to the organization
across the full spectrum of use cases
Visual	Analytics Event	Processing	
Analytics
What is Prescriptive Analytics?
© Copyright 2000-2016 TIBCO Software Inc.
Real Time Streaming Analytics
time
1 2 3 4 5 6 7 8 9
Event	Streams
• Continuous	Queries
• Sliding	Windows
• Filter
• Aggregation
• Correlation
• …
© Copyright 2000-2016 TIBCO Software Inc.
Operational Intelligence and Human Interaction
Actions by Operations
Human	decisions	in	real	time	informed	
by	up	to	date	information
65
Automated	action	based	on	models	of	history	
combined	with	live	context	and	business	rules
Machine-to-Machine Automation
© Copyright 2000-2016 TIBCO Software Inc.
Alternatives for Streaming Analytics (no	complete	list!)
Azure	Microsoft
Stream	Analytics
CLOSED	SOURCEOPEN	SOURCE
FRAMEWORK
PRODUCT
© Copyright 2000-2016 TIBCO Software Inc.
What Kind of Streaming Analytics do you need?
Visual IDE (Dev, Test, Debug)
Simulation (Feed Testing, Test Generation)
Live UI (monitoring, proactive interaction)
Maturity (24/7 support, consulting)
Integration (out-of-the-box: ESB, MDM, etc.)
Library (Java, .NET, Python)
Query Language (often similar to SQL)
Scalability (horizontal and vertical, fail over)
Connectivity (technologies, markets, products)
Operators (Filter, Sort, Aggregate)
Time
to
Market
Streaming
Frameworks
Streaming
Products
Slow Fast
Streaming
Concepts
© Copyright 2000-2016 TIBCO Software Inc.
Comparison of Stream Processing Frameworks and Products
Slide Deck from JavaOne 2015:
http://www.kai-waehner.de/blog/2015/10/25/
comparison-of-stream-processing-frameworks-and-products/ Updated slide deck coming
in November 2016
(Big Data Spain, Madrid)
© Copyright 2000-2016 TIBCO Software Inc.
Visual Coding for Streaming Analytics
• Streaming	Operators
• Connectivity
• Visual	Development
• Testing	&	Simulation
• Mature	Tooling	/	Support
• Middleware	Integration
© Copyright 2000-2016 TIBCO Software Inc.
Live Visual Analytics UI
Dynamic	aggregation	
Live	visualization
Ad-hoc	continuous	query
Alerts
Action
© Copyright 2000-2016 TIBCO Software Inc.
How to
apply analytic models
to real time processing
without redevelopment?
Stream
Processing
H20.ai
Open
Source
R
TERR
Spark
ML
MATLAB
SAS
PMML
© Copyright 2000-2016 TIBCO Software Inc.
TIBCO StreamBase Connector for R and TERR
© Copyright 2000-2016 TIBCO Software Inc.
TIBCO StreamBase Connector for H2O.ai
© Copyright 2000-2016 TIBCO Software Inc.
TIBCO StreamBase Connector for PMML
© Copyright 2000-2016 TIBCO Software Inc.
Real World Streaming Application for Customer Churn
© Copyright 2000-2016 TIBCO Software Inc.
Closed Loop à Automatically Re-Compute (and Improve) the Analytic Model
Compute
your
performance
metric Spot not
good enough
performance
Re-compute
model
© Copyright 2000-2016 TIBCO Software Inc.
Agenda
1) Machine Learning and Big Data Analytics
2) Building an Analytic Model
3) Real Time Processing
4) Live Demo
5) Intelligent Microservices
• Reactive – Run to failure
• Preventive – Scheduled service (reliability)
• Condition-based – Monitor condition (sensors)
• Predictive – Predict failures
• Proactive – Deploy automatic actions
Evolution of Equipment Maintenance Strategies
Scenario: Predictive Scrapping of Parts in an Assembly Line
Goal: Scrap parts as early as possible automatically to reduce costs in a manufacturing process.
Question: When to scrap a part in Station 1 instead of doing re-work or sending it to Station 2?
Station 1 Station 2
Cost Before
9€
7€ 13€
Total Cost
29€
(or more)
Scrap? Scrap?
Fast Data Architecture for Predictive Maintenance
Operational	Analytics
Operations
Live	UI
CSV Batch
JSON Real Time
XML Real Time
Streaming	AnalyticsAction
Aggregate
Rules
Analytics
Correlate
Live	Datamart
Continuous	query	
processing
Alerts
Manual	action,	
escalation
HISTORICAL	ANALYSIS Data	
Scientists
Flume
HDFS
Spotfire
R	/	TERR
HDFS
Hadoop (Cloudera)
StreamBase
TIBCO Fast Data Platform
H2O
Oracle	RDBMS
Avro Parquet … PMML
Internal	Data
TIBCO Spotfire with H2O Integration
Data Discovery / Data Mining (“Are parts that repeat a station more likely scrap parts?”)
TIBCO Live Datamart
Operational Intelligence (“Monitor the manufacturing process and change rules in real time!”)
Live Dartmart Desktop Client
TIBCO Live Datamart
Operational Intelligence (“Monitor the manufacturing process and change rules in real time!”)
Live Dartmart Web API
TIBCO Spotfire + StreamBase + H2O.ai + Live Datamart
Live DemoLive Demo
© Copyright 2000-2016 TIBCO Software Inc.
TIBCO Accelerator for Apache Spark
1. Fast Data Preparation for IoT
Dozens of enterprise and IoT data preparation adapters:
MQTT, Databases; inbound creation of HDFS, Parquet, Hbase,
Avro…
2. Spotfire Model Discovery Template
Use Spotfire to explore Spark data lake, create predictive
model, train in H20, and deploy to Streaming Analytics.
3. Operationalize Predictive Models
Zookeeper deployment to StreamBase nodes living in Spark
cluster via H20, PMML, TERR models
4. Streaming Analytics for Automation
Automate action based on predictive models – make offers to
customers, stop fraudulent transactions, alert.
5. Monitor & Retrain Model
Monitor behavior of model, retrain when necessary.
6. Drag & Drop for Business Solution Developers
Code-free development environment for work with H20, HDFS,
Avro, TERR
The TIBCO Accelerator for Spark is a TIBCO
engineered, light-weight open-source fast-
start for systems to stream data into Spark,
discover patterns in Spark with Spotfire, and
operationalize the insights on Big Data.
FUNCTIONAL COMPONENTS
© Copyright 2000-2016 TIBCO Software Inc.
Agenda
1) Machine Learning and Big Data Analytics
2) Building an Analytic Model
3) Real Time Processing
4) Live Demo
5) Intelligent Microservices
© Copyright 2000-2016 TIBCO Software Inc.
Evolving Demands from the Business
AGILITY &
SPEED
REDUCED
CYCLE
TIMES
WEB
SCALE
LOWER
COST
FAIL FAST
© Copyright 2000-2016 TIBCO Software Inc.
Development of
Intelligent Microservices
© Copyright 2000-2016 TIBCO Software Inc.
12 Factor Apps for Cloud Native Microservices
Codebase
One codebase
tracked in
revision control,
many deploys.
Dependencies
Explicitly declare
and isolate
dependencies.
Config
Store config in
the environment.
Backing
Services
Treat backing
services as
attached
resources.
Build, Release,
Run
Strictly separate
build and run
stages.
Processes
Execute the app
as one or more
stateless
processes.
Port Binding
Export services
via port binding.
Concurrency
Scale out via the
process model.
Disposability
Maximize
robustness with
fast startup and
graceful
shutdown.
Dev / Prod
Parity
Keep dev,
staging, and
prod as similar as
possible.
Logs
Treat logs as
event streams.
Admin
Processes
Run
admin/mgmt
tasks as one-off
processes.
https://12factor.net/
© Copyright 2000-2016 TIBCO Software Inc.
Why Containers?
http://www.slideshare.net/andersjanmyr/docker-the-future-of-devops
Containers enable:
• Lightweight deployment
• Automation
• Better resource utilization
• Scaling up and down quickly
• Platform agnostic deployment
• Innovation and Fail Fast Concepts
• Standardization ? Ø The Open Container Initiative (OCI)
Ø Docker Fork Discussions (!!!)
© Copyright 2000-2016 TIBCO Software Inc.
DevOps Elements – Culture and Technology!
Process
Tools
Automation
Culture
Continuous Integration/
Continuous Development
APIs
MicroservicesFrequent releases
Collaboration
© Copyright 2000-2016 TIBCO Software Inc.
Develop fast. Fail fast. Change fast.
Visual Analytics + Visual Coding + DevOps
= Agile Intelligent Microservices
© Copyright 2000-2016 TIBCO Software Inc.
Application of Analytic Models
to other Microservices
© Copyright 2000-2016 TIBCO Software Inc.
Real Time Streaming Analytics
time
1 2 3 4 5 6 7 8 9
Event	Streams
Apply your intelligent (micro)service to any event.
Microservice event. Application event. Legacy event. IoT event. You name it.
© Copyright 2000-2016 TIBCO Software Inc.
Key Take-Aways
Ø Insights are hidden in Historical Data on Big Data Platforms
Ø Machine Learning and Big Data Analytics find these Insights by building Analytics Models
Ø Event Processing uses these Models (without Redevelopment) to take Action in Real Time
Questions? Please contact me!
Kai Wähner
Technology Evangelist at TIBCO
kontakt@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
LinkedIn

More Related Content

What's hot

Advanced data science algorithms applied to scalable stream processing by Dav...
Advanced data science algorithms applied to scalable stream processing by Dav...Advanced data science algorithms applied to scalable stream processing by Dav...
Advanced data science algorithms applied to scalable stream processing by Dav...Big Data Spain
 
VP of WW Partners by Alan Chhabra
VP of WW Partners by Alan ChhabraVP of WW Partners by Alan Chhabra
VP of WW Partners by Alan ChhabraBig Data Spain
 
Enabling the Bank of the Future by Ignacio Bernal
Enabling the Bank of the Future by Ignacio BernalEnabling the Bank of the Future by Ignacio Bernal
Enabling the Bank of the Future by Ignacio BernalBig Data Spain
 
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...Shirshanka Das
 
Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...
Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...
Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...Impetus Technologies
 
Jan van der Vegt. Challenges faced with machine learning in practice
Jan van der Vegt. Challenges faced with machine learning in practiceJan van der Vegt. Challenges faced with machine learning in practice
Jan van der Vegt. Challenges faced with machine learning in practiceLviv Startup Club
 
Turning an idea into a Data-Driven Production System: An Energy Load Forecas...
 Turning an idea into a Data-Driven Production System: An Energy Load Forecas... Turning an idea into a Data-Driven Production System: An Energy Load Forecas...
Turning an idea into a Data-Driven Production System: An Energy Load Forecas...Big Data Spain
 
Deep Learning Image Processing Applications in the Enterprise
Deep Learning Image Processing Applications in the EnterpriseDeep Learning Image Processing Applications in the Enterprise
Deep Learning Image Processing Applications in the EnterpriseGanesan Narayanasamy
 
Lambda Architecture 2.0 Convergence between Real-Time Analytics, Context-awar...
Lambda Architecture 2.0 Convergence between Real-Time Analytics, Context-awar...Lambda Architecture 2.0 Convergence between Real-Time Analytics, Context-awar...
Lambda Architecture 2.0 Convergence between Real-Time Analytics, Context-awar...Sabri Skhiri
 
Real Time Analytics: Algorithms and Systems
Real Time Analytics: Algorithms and SystemsReal Time Analytics: Algorithms and Systems
Real Time Analytics: Algorithms and SystemsArun Kejariwal
 
Reactive Java Robotics & IoT with Spring Reactor
Reactive Java Robotics & IoT with Spring ReactorReactive Java Robotics & IoT with Spring Reactor
Reactive Java Robotics & IoT with Spring ReactorTrayan Iliev
 
ParStream - Big Data for Business Users
ParStream - Big Data for Business UsersParStream - Big Data for Business Users
ParStream - Big Data for Business UsersParStream Inc.
 
Vertex AI: Pipelines for your MLOps workflows
Vertex AI: Pipelines for your MLOps workflowsVertex AI: Pipelines for your MLOps workflows
Vertex AI: Pipelines for your MLOps workflowsMárton Kodok
 
ML Model Deployment and Scoring on the Edge with Automatic ML & DF
ML Model Deployment and Scoring on the Edge with Automatic ML & DFML Model Deployment and Scoring on the Edge with Automatic ML & DF
ML Model Deployment and Scoring on the Edge with Automatic ML & DFSri Ambati
 
Lessons from building a stream-first metadata platform | Shirshanka Das, Stealth
Lessons from building a stream-first metadata platform | Shirshanka Das, StealthLessons from building a stream-first metadata platform | Shirshanka Das, Stealth
Lessons from building a stream-first metadata platform | Shirshanka Das, StealthHostedbyConfluent
 
Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...
Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...
Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...Impetus Technologies
 
Oracle Stream Explorer - Simplifying Event/Stream Processing
Oracle Stream Explorer - Simplifying Event/Stream ProcessingOracle Stream Explorer - Simplifying Event/Stream Processing
Oracle Stream Explorer - Simplifying Event/Stream ProcessingGuido Schmutz
 
Real-Time AI: Designing for Low Latency and High Throughput - Dr. Sergei Izra...
Real-Time AI: Designing for Low Latency and High Throughput - Dr. Sergei Izra...Real-Time AI: Designing for Low Latency and High Throughput - Dr. Sergei Izra...
Real-Time AI: Designing for Low Latency and High Throughput - Dr. Sergei Izra...Sri Ambati
 
Dive into H2O: NYC
Dive into H2O: NYCDive into H2O: NYC
Dive into H2O: NYCSri Ambati
 
QCon 2018 | Gimel | PayPal's Analytic Platform
QCon 2018 | Gimel | PayPal's Analytic PlatformQCon 2018 | Gimel | PayPal's Analytic Platform
QCon 2018 | Gimel | PayPal's Analytic PlatformDeepak Chandramouli
 

What's hot (20)

Advanced data science algorithms applied to scalable stream processing by Dav...
Advanced data science algorithms applied to scalable stream processing by Dav...Advanced data science algorithms applied to scalable stream processing by Dav...
Advanced data science algorithms applied to scalable stream processing by Dav...
 
VP of WW Partners by Alan Chhabra
VP of WW Partners by Alan ChhabraVP of WW Partners by Alan Chhabra
VP of WW Partners by Alan Chhabra
 
Enabling the Bank of the Future by Ignacio Bernal
Enabling the Bank of the Future by Ignacio BernalEnabling the Bank of the Future by Ignacio Bernal
Enabling the Bank of the Future by Ignacio Bernal
 
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
 
Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...
Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...
Real-time Streaming Analytics: Business Value, Use Cases and Architectural Co...
 
Jan van der Vegt. Challenges faced with machine learning in practice
Jan van der Vegt. Challenges faced with machine learning in practiceJan van der Vegt. Challenges faced with machine learning in practice
Jan van der Vegt. Challenges faced with machine learning in practice
 
Turning an idea into a Data-Driven Production System: An Energy Load Forecas...
 Turning an idea into a Data-Driven Production System: An Energy Load Forecas... Turning an idea into a Data-Driven Production System: An Energy Load Forecas...
Turning an idea into a Data-Driven Production System: An Energy Load Forecas...
 
Deep Learning Image Processing Applications in the Enterprise
Deep Learning Image Processing Applications in the EnterpriseDeep Learning Image Processing Applications in the Enterprise
Deep Learning Image Processing Applications in the Enterprise
 
Lambda Architecture 2.0 Convergence between Real-Time Analytics, Context-awar...
Lambda Architecture 2.0 Convergence between Real-Time Analytics, Context-awar...Lambda Architecture 2.0 Convergence between Real-Time Analytics, Context-awar...
Lambda Architecture 2.0 Convergence between Real-Time Analytics, Context-awar...
 
Real Time Analytics: Algorithms and Systems
Real Time Analytics: Algorithms and SystemsReal Time Analytics: Algorithms and Systems
Real Time Analytics: Algorithms and Systems
 
Reactive Java Robotics & IoT with Spring Reactor
Reactive Java Robotics & IoT with Spring ReactorReactive Java Robotics & IoT with Spring Reactor
Reactive Java Robotics & IoT with Spring Reactor
 
ParStream - Big Data for Business Users
ParStream - Big Data for Business UsersParStream - Big Data for Business Users
ParStream - Big Data for Business Users
 
Vertex AI: Pipelines for your MLOps workflows
Vertex AI: Pipelines for your MLOps workflowsVertex AI: Pipelines for your MLOps workflows
Vertex AI: Pipelines for your MLOps workflows
 
ML Model Deployment and Scoring on the Edge with Automatic ML & DF
ML Model Deployment and Scoring on the Edge with Automatic ML & DFML Model Deployment and Scoring on the Edge with Automatic ML & DF
ML Model Deployment and Scoring on the Edge with Automatic ML & DF
 
Lessons from building a stream-first metadata platform | Shirshanka Das, Stealth
Lessons from building a stream-first metadata platform | Shirshanka Das, StealthLessons from building a stream-first metadata platform | Shirshanka Das, Stealth
Lessons from building a stream-first metadata platform | Shirshanka Das, Stealth
 
Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...
Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...
Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus...
 
Oracle Stream Explorer - Simplifying Event/Stream Processing
Oracle Stream Explorer - Simplifying Event/Stream ProcessingOracle Stream Explorer - Simplifying Event/Stream Processing
Oracle Stream Explorer - Simplifying Event/Stream Processing
 
Real-Time AI: Designing for Low Latency and High Throughput - Dr. Sergei Izra...
Real-Time AI: Designing for Low Latency and High Throughput - Dr. Sergei Izra...Real-Time AI: Designing for Low Latency and High Throughput - Dr. Sergei Izra...
Real-Time AI: Designing for Low Latency and High Throughput - Dr. Sergei Izra...
 
Dive into H2O: NYC
Dive into H2O: NYCDive into H2O: NYC
Dive into H2O: NYC
 
QCon 2018 | Gimel | PayPal's Analytic Platform
QCon 2018 | Gimel | PayPal's Analytic PlatformQCon 2018 | Gimel | PayPal's Analytic Platform
QCon 2018 | Gimel | PayPal's Analytic Platform
 

Similar to Apply Machine Learning to Microservices

Findability Day 2016 - Big data analytics and machine learning
Findability Day 2016 - Big data analytics and machine learningFindability Day 2016 - Big data analytics and machine learning
Findability Day 2016 - Big data analytics and machine learningFindwise
 
How to Apply Big Data Analytics and Machine Learning to Real Time Processing ...
How to Apply Big Data Analytics and Machine Learning to Real Time Processing ...How to Apply Big Data Analytics and Machine Learning to Real Time Processing ...
How to Apply Big Data Analytics and Machine Learning to Real Time Processing ...Codemotion
 
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...Codemotion Tel Aviv
 
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...Codemotion
 
Big Data LDN 2017: How Big Data Insights Become Easily Accessible With Workfl...
Big Data LDN 2017: How Big Data Insights Become Easily Accessible With Workfl...Big Data LDN 2017: How Big Data Insights Become Easily Accessible With Workfl...
Big Data LDN 2017: How Big Data Insights Become Easily Accessible With Workfl...Matt Stubbs
 
Machine Learning Applied to Real Time Scoring in Manufacturing and Energy Uti...
Machine Learning Applied to Real Time Scoring in Manufacturing and Energy Uti...Machine Learning Applied to Real Time Scoring in Manufacturing and Energy Uti...
Machine Learning Applied to Real Time Scoring in Manufacturing and Energy Uti...Kai Wähner
 
AI in Software for Augmenting Intelligence Across the Enterprise
AI in Software for Augmenting Intelligence Across the EnterpriseAI in Software for Augmenting Intelligence Across the Enterprise
AI in Software for Augmenting Intelligence Across the EnterpriseThe Hive
 
The TIBCO OEM Partnership
The TIBCO OEM PartnershipThe TIBCO OEM Partnership
The TIBCO OEM PartnershipMinerva Dutta
 
TIBCO presentation at the Chief Analytics Officer Forum East Coast 2016 (#CAO...
TIBCO presentation at the Chief Analytics Officer Forum East Coast 2016 (#CAO...TIBCO presentation at the Chief Analytics Officer Forum East Coast 2016 (#CAO...
TIBCO presentation at the Chief Analytics Officer Forum East Coast 2016 (#CAO...Chief Analytics Officer Forum
 
TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming An...
TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming An...TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming An...
TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming An...Nelson Petracek
 
Bitrock manufacturing
Bitrock manufacturing Bitrock manufacturing
Bitrock manufacturing cosma_r
 
Introduction to jaspersoft7 customer webinar
Introduction to jaspersoft7 customer webinarIntroduction to jaspersoft7 customer webinar
Introduction to jaspersoft7 customer webinarTIBCO Jaspersoft
 
Make your application stand out with bi that blends in
Make your application stand out with bi that blends inMake your application stand out with bi that blends in
Make your application stand out with bi that blends inTIBCO Jaspersoft
 
Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...
Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...
Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...Big Data Spain
 
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Ser...
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Ser...Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Ser...
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Ser...Kai Wähner
 
Sensor Data Management & Analytics: Advanced Process Control
Sensor Data Management & Analytics: Advanced Process ControlSensor Data Management & Analytics: Advanced Process Control
Sensor Data Management & Analytics: Advanced Process ControlTIBCO_Software
 
Intelligent Business Process Management Suites (iBPMS) - The Next-Generation ...
Intelligent Business Process Management Suites (iBPMS) - The Next-Generation ...Intelligent Business Process Management Suites (iBPMS) - The Next-Generation ...
Intelligent Business Process Management Suites (iBPMS) - The Next-Generation ...Kai Wähner
 
JASPERSOFT LIVE DEMO - NAM
JASPERSOFT LIVE DEMO - NAMJASPERSOFT LIVE DEMO - NAM
JASPERSOFT LIVE DEMO - NAMTIBCO Jaspersoft
 
Tibco Augmented Intelligence - Analytics, IoT, Big Data, Streaming 20161025
Tibco Augmented Intelligence - Analytics, IoT, Big Data, Streaming 20161025Tibco Augmented Intelligence - Analytics, IoT, Big Data, Streaming 20161025
Tibco Augmented Intelligence - Analytics, IoT, Big Data, Streaming 20161025Nicola Sandoli
 
Role of Data in Digital Transformation
Role of Data in Digital TransformationRole of Data in Digital Transformation
Role of Data in Digital TransformationVMware Tanzu
 

Similar to Apply Machine Learning to Microservices (20)

Findability Day 2016 - Big data analytics and machine learning
Findability Day 2016 - Big data analytics and machine learningFindability Day 2016 - Big data analytics and machine learning
Findability Day 2016 - Big data analytics and machine learning
 
How to Apply Big Data Analytics and Machine Learning to Real Time Processing ...
How to Apply Big Data Analytics and Machine Learning to Real Time Processing ...How to Apply Big Data Analytics and Machine Learning to Real Time Processing ...
How to Apply Big Data Analytics and Machine Learning to Real Time Processing ...
 
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...
 
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...
How to Leverage Machine Learning (R, Hadoop, Spark, H2O) for Real Time Proces...
 
Big Data LDN 2017: How Big Data Insights Become Easily Accessible With Workfl...
Big Data LDN 2017: How Big Data Insights Become Easily Accessible With Workfl...Big Data LDN 2017: How Big Data Insights Become Easily Accessible With Workfl...
Big Data LDN 2017: How Big Data Insights Become Easily Accessible With Workfl...
 
Machine Learning Applied to Real Time Scoring in Manufacturing and Energy Uti...
Machine Learning Applied to Real Time Scoring in Manufacturing and Energy Uti...Machine Learning Applied to Real Time Scoring in Manufacturing and Energy Uti...
Machine Learning Applied to Real Time Scoring in Manufacturing and Energy Uti...
 
AI in Software for Augmenting Intelligence Across the Enterprise
AI in Software for Augmenting Intelligence Across the EnterpriseAI in Software for Augmenting Intelligence Across the Enterprise
AI in Software for Augmenting Intelligence Across the Enterprise
 
The TIBCO OEM Partnership
The TIBCO OEM PartnershipThe TIBCO OEM Partnership
The TIBCO OEM Partnership
 
TIBCO presentation at the Chief Analytics Officer Forum East Coast 2016 (#CAO...
TIBCO presentation at the Chief Analytics Officer Forum East Coast 2016 (#CAO...TIBCO presentation at the Chief Analytics Officer Forum East Coast 2016 (#CAO...
TIBCO presentation at the Chief Analytics Officer Forum East Coast 2016 (#CAO...
 
TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming An...
TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming An...TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming An...
TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming An...
 
Bitrock manufacturing
Bitrock manufacturing Bitrock manufacturing
Bitrock manufacturing
 
Introduction to jaspersoft7 customer webinar
Introduction to jaspersoft7 customer webinarIntroduction to jaspersoft7 customer webinar
Introduction to jaspersoft7 customer webinar
 
Make your application stand out with bi that blends in
Make your application stand out with bi that blends inMake your application stand out with bi that blends in
Make your application stand out with bi that blends in
 
Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...
Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...
Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...
 
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Ser...
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Ser...Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Ser...
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Ser...
 
Sensor Data Management & Analytics: Advanced Process Control
Sensor Data Management & Analytics: Advanced Process ControlSensor Data Management & Analytics: Advanced Process Control
Sensor Data Management & Analytics: Advanced Process Control
 
Intelligent Business Process Management Suites (iBPMS) - The Next-Generation ...
Intelligent Business Process Management Suites (iBPMS) - The Next-Generation ...Intelligent Business Process Management Suites (iBPMS) - The Next-Generation ...
Intelligent Business Process Management Suites (iBPMS) - The Next-Generation ...
 
JASPERSOFT LIVE DEMO - NAM
JASPERSOFT LIVE DEMO - NAMJASPERSOFT LIVE DEMO - NAM
JASPERSOFT LIVE DEMO - NAM
 
Tibco Augmented Intelligence - Analytics, IoT, Big Data, Streaming 20161025
Tibco Augmented Intelligence - Analytics, IoT, Big Data, Streaming 20161025Tibco Augmented Intelligence - Analytics, IoT, Big Data, Streaming 20161025
Tibco Augmented Intelligence - Analytics, IoT, Big Data, Streaming 20161025
 
Role of Data in Digital Transformation
Role of Data in Digital TransformationRole of Data in Digital Transformation
Role of Data in Digital Transformation
 

More from Kai Wähner

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Kai Wähner
 
When NOT to use Apache Kafka?
When NOT to use Apache Kafka?When NOT to use Apache Kafka?
When NOT to use Apache Kafka?Kai Wähner
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKai Wähner
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaKai Wähner
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareKai Wähner
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Kai Wähner
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureKai Wähner
 
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Kai Wähner
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail IndustryKai Wähner
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKai Wähner
 
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Kai Wähner
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingKai Wähner
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKai Wähner
 
The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesKai Wähner
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Kai Wähner
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Kai Wähner
 

More from Kai Wähner (20)

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
 
When NOT to use Apache Kafka?
When NOT to use Apache Kafka?When NOT to use Apache Kafka?
When NOT to use Apache Kafka?
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
 
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid Cloud
 
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and Manufacturing
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology Comparison
 
The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
 

Recently uploaded

The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFMichael Gough
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxfnnc6jmgwh
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsYoss Cohen
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...BookNet Canada
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 

Recently uploaded (20)

The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDF
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platforms
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 

Apply Machine Learning to Microservices

  • 1. Kai Wähner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.kai-waehner.de O’Reilly Software Architecture Conference 2016 (London, UK) How to apply big data analytics and machine learning to real-time processing of microservice events
  • 2. © Copyright 2000-2016 TIBCO Software Inc. Digital Transformation - Physical and Digital Worlds are Merging
  • 3. © Copyright 2000-2016 TIBCO Software Inc. Apply Big Data Analytics to Real Time Processing
  • 4. © Copyright 2000-2016 TIBCO Software Inc. Analyze and Act on Critical Business Moments
  • 5. © Copyright 2000-2016 TIBCO Software Inc. Key Take-Aways Ø Insights are hidden in Historical Data on Big Data Platforms Ø Machine Learning and Big Data Analytics find these Insights by building Analytics Models Ø Event Processing uses these Models (without Redevelopment) to take Action in Real Time
  • 6. © Copyright 2000-2016 TIBCO Software Inc. Agenda 1) Machine Learning and Big Data Analytics 2) Building an Analytic Model 3) Real Time Processing 4) Live Demo 5) Intelligent Microservices
  • 7. © Copyright 2000-2016 TIBCO Software Inc. Agenda 1) Machine Learning and Big Data Analytics 2) Building an Analytic Model 3) Real Time Processing 4) Live Demo 5) Intelligent Microservices
  • 8. Machine Learning …. allows computers to find hidden insights without being explicitly programmed where to look.
  • 9. Real World Examples of Machine Learning Spam Detection Search Results + Product Recommendation Picture Detection (Friends, Locations, Products) Machine Learning is already present in daily life… Now, every enterprise is beginning to leverage it! The Next Disruption: Google Beats Go Champion
  • 10. © Copyright 2000-2016 TIBCO Software Inc. Example: Decision Tree – Titanic Survival Rate family size Wikipedia
  • 11. Decision Tree – Product Pass / Fail by Equipment Sensor Readings Bad Product Good Product Step 8 Temperature < 122 C >= 122 C Step 2 Recipe A B Step 11 Pressure TV Color Display Problem
  • 12. Decision Tree – Training and Test Data Sets
  • 13. © Copyright 2000-2016 TIBCO Software Inc. Ensemble Tree Algorithms • Random Forest, Gradient Boosting Machine (GBM) • Method – Average many simple trees • Sample the data: fit a simple tree • Re-sample the data; up-weighting the observations that weren’t fitted well in previous model • Continue adding trees until fit is good • Save all the trees and average them • Better fit + prediction than single trees
  • 14. © Copyright 2000-2016 TIBCO Software Inc. Closed Loop for Big Data Analytics
  • 15. © Copyright 2000-2016 TIBCO Software Inc. Analytics Maturity Model Immediate Long-Term Competitive AdvantageValue to the Organization A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases Self-service Dashboards Event Processing Advanced Analytics Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Visual Analytics Event Processing Analytics
  • 16. © Copyright 2000-2016 TIBCO Software Inc. Analytics Maturity Model Immediate Long-Term Competitive AdvantageValue to the Organization Visual Analytics Event Processing Advanced Analytics Measure Diagnose Predict Optimize Alert Automate Analytics Maturity A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases Analytics
  • 17. © Copyright 2000-2016 TIBCO Software Inc. Analytics Maturity Model Immediate Long-Term Competitive AdvantageValue to the Organization Self-service Dashboards Event Processing Advanced Analytics Measure Diagnose Predict Optimize Alert Automate Analytics Maturity A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases Visual Analytics Event Processing Analytics
  • 18. © Copyright 2000-2016 TIBCO Software Inc. The first task in a new analytics projects is to define a Business Case!
  • 19. © Copyright 2000-2016 TIBCO Software Inc. From a Business Case to Proactive Actions Model Present Data Wrangling Signals Dashboards SAP Historian Production Well Filter Enrich Merge Shape Explore Clean Assemble DataBusiness Case Increase Productivity Grow Revenue Completions Visualize GeoLocation Production Value Theses Reduce Risk G&G Equipment Decision, Action Prediction Action Develop Model Pressure Temperature Production Interrupt Drill Bit Movement Equipment Failure
  • 20. © Copyright 2000-2016 TIBCO Software Inc. Agenda 1) Machine Learning and Big Data Analytics 2) Building an Analytic Model 3) Real Time Processing 4) Live Demo 5) Intelligent Microservices
  • 21. © Copyright 2000-2016 TIBCO Software Inc. Analytical Pipeline
  • 22. © Copyright 2000-2016 TIBCO Software Inc. Analytics Maturity Model Immediate Long-Term Competitive AdvantageValue to the Organization A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases Self-service Dashboards Event Processing Advanced Analytics Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Visual Analytics Event Processing Analytics
  • 23. What is Predictive Analytics?
  • 24. © Copyright 2000-2016 TIBCO Software Inc. Analytical Pipeline
  • 25. © Copyright 2000-2016 TIBCO Software Inc. Variety of Data in Enterprises Custom GUI-driven data access via SDK Siebel eBusiness Local data sources AccessExcel STDF Drag-and-drop MySQL SQL Server Oracle Information Services (join, transform, reusable, parameterized, dynamic query for in-memory use) Databases JDBC/ODBC Hadoop SFDC PostgreSQL Teradata Netezza Etc.XML RDBMS Flat Files Spread- sheets Web Services Oracle E-Business RDBMS RDBMS RDBMS SAP BWSAP R/3 D A T A F A B R I C Salesforce ODBC OLE DB SqlClient Direct connection Oracle TeradataAsterMS SSAS Teradata Direct Query (dynamically query and retrieve data for visualization and analysis) Databases MySQL Etc. OBIEE Netezza Hadoop
  • 26. © Copyright 2000-2016 TIBCO Software Inc. Data Acquisition “Smart Recommendation Engine”
  • 27. © Copyright 2000-2016 TIBCO Software Inc. Analytical Pipeline
  • 28. © Copyright 2000-2016 TIBCO Software Inc. Data Munging / Wrangling / Mash-up
  • 29. cust_id dept sku dollar gift date 1 104 C 12003 2.40 FALSE 2016-10-17 2 105 A 12005 62.85 FALSE 2016-10-17 3 102 C 12007 69.23 TRUE 2016-10-17 4 104 B 12004 9.33 FALSE 2016-10-18 5 105 C 12010 14.16 TRUE 2016-10-18 6 101 B 12003 90.43 FALSE 2016-10-19 7 103 C 12005 90.97 FALSE 2016-10-19 n … … … … … … cust_id A B C total # orders first_dat e last_dat e 1 100 21.76 23.67 0.00 45.43 2 2016-10- 19 2016-10- 20 2 101 0.01 74.65 0.00 74.66 3 2016-10- 19 2016-10- 20 3 102 0.00 60.92 50.29 111.21 6 2016-10- 17 2016-10- 20 4 103 0.00 0.00 52.30 52.30 2 2016-10- 19 2016-10- 20© Copyright 2000-2016 TIBCO Software Inc. Data Munging - Transformations
  • 30. © Copyright 2000-2016 TIBCO Software Inc. Analytical Pipeline
  • 31. © Copyright 2000-2016 TIBCO Software Inc. Exploratory Data Analysis
  • 32. Exploratory Data Analysis (EDA) is an approach/philosophy for data analysis that employs a variety of techniques (mostly graphical) 1. to maximize insight into a data set 2. uncover underlying structure 3. extract important variables 4. detect outliers and anomalies 5. test underlying assumptions 6. develop parsimonious models 7. determine optimal factor settings © Copyright 2000-2016 TIBCO Software Inc. Exploratory Data Analysis
  • 33. “The greatest value of a picture is when it forces us to notice what we never expected to see” John W. Tukey, 1977 © Copyright 2000-2016 TIBCO Software Inc. Exploratory Data Analysis
  • 34. Visual Analytics - Interactive Brush-Linked © Copyright 2000-2016 TIBCO Software Inc. … and “Inline Data Wrangling” à Ad-hoc data preparation instead of just ETL
  • 35. © Copyright 2000-2016 TIBCO Software Inc. Analytics Maturity Model Immediate Long-Term Competitive AdvantageValue to the Organization Visual Analytics Event Processing Advanced Analytics Measure Diagnose Predict Optimize Alert Automate Analytics Maturity A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases Analytics
  • 36. What is Predictive Analytics?
  • 37. © Copyright 2000-2016 TIBCO Software Inc. Analytical Pipeline
  • 38. © Copyright 2000-2016 TIBCO Software Inc. Which picture represents a model? A model is a simplification of the truth that helps you with decision making.
  • 39. © Copyright 2000-2016 TIBCO Software Inc. Model Building
  • 40. © Copyright 2000-2016 TIBCO Software Inc. Model Building
  • 41. Employees who write longer emails earn higher salaries! © Copyright 2000-2016 TIBCO Software Inc. Model Building
  • 42. © Copyright 2000-2016 TIBCO Software Inc. Model Improvement
  • 43. Managers Staff © Copyright 2000-2016 TIBCO Software Inc. Model Improvement
  • 44. © Copyright 2000-2016 TIBCO Software Inc. Analytical Pipeline
  • 45. © Copyright 2000-2016 TIBCO Software Inc. Model Validation How is the IQ of a kid related to the IQ of his / her mum?
  • 46. © Copyright 2000-2016 TIBCO Software Inc. Frameworks and Tooling
  • 47. © Copyright 2000-2016 TIBCO Software Inc. “…as a next-generation data discovery capability that automatically finds and explains insights from advanced analytics to business users or citizen data scientists” Smart Data Discovery (for the Business User) Leverage Machine Learning without the help of a Data Scientist
  • 48. Advanced Analytics and Big Data Tools (for Data Scientists) Many more ….
  • 49. R Language • Built for data scientists • Very active community © Copyright 2000-2016 TIBCO Software Inc.
  • 50. R with Revolution Analytics (now Microsoft) © Copyright 2000-2016 TIBCO Software Inc. Open Source GPL License (including its restrictions) http://www.revolutionanalytics.com/webinars/introducing-revolution-r-open-enhanced-open-source-r-distribution- revolution-analytics
  • 51. TIBCO has rewritten R as a Commercial Compute Engine • Latest statistics scripting engine: S a S-PLUS® a R a TERR • Runs R code including CRAN packages Engine internals rebuilt from scratch at low-level • Redesigned data objects, memory management • High performance + Big Data TERR is licensed from TIBCO • TERR Installs (free) with Spotfire Analyst / Desktop + other TIBCO products • Spotfire Server can manage all TERR / R scripts, artifacts for reuse • Standalone Developer Edition • Supported by TIBCO • No GPL license issues © Copyright 2000-2016 TIBCO Software Inc. TERR - TIBCO’s Enterprise Runtime for R
  • 52. Which R to use? © Copyright 2000-2016 TIBCO Software Inc. http://www.forbes.com/sites/danwoods/2016/01/27/microsofts-revolution-analytics-acquisition-is-the-wrong- way-to-embrace-r/
  • 53. © Copyright 2000-2016 TIBCO Software Inc. Apache Spark General Data-processing Framework à However, focus is especially on Analytics (at least these days)
  • 54. Apache Spark MLlib © Copyright 2000-2016 TIBCO Software Inc. Spark ML is Spark’s machine learning library. Its goal is to make practical machine learning scalable and easy. It consists of common learning algorithms and utilities, including classification, regression, clustering and collaborative filtering. General Data-processing Framework à However, focus is especially on Analytics (at least these days) x
  • 55. © Copyright 2000-2016 TIBCO Software Inc. H2O.ai An Extensible Open Source Platform for Analytics • Best of Breed Open Source Technology • Easy-to-use Web UI and Familiar Interfaces • Data Agnostic Support for all Common Database and File Types • Massively Scalable Big Data Analysis • Real-time Data Scoring (“Nanofast Scoring Engine”) http://www.h2o.ai/
  • 56. TIBCO Spotfire with R / TERR Integration © Copyright 2000-2016 TIBCO Software Inc. Let the business user leverage Analytic Models (created by the Data Scientist) to find insights! Example: Customer Churn with Random Forest Algorithm • ‘refresh model’ button lives a ‘random forest algorithm’ • requires no a priori assumptions at all, it just always works • The business user doesn’t need to know what random forest is to be empowered by it Select variables for the model
  • 57. TIBCO Spotfire with H2O Integration © Copyright 2000-2016 TIBCO Software Inc. Example: Predictive Analytics for Manufacturing (“scrap parts as early as possible”)
  • 58. TIBCO Spotfire with H2O Integration © Copyright 2000-2016 TIBCO Software Inc. Example: Predictive Analytics for Manufacturing (“scrap parts as early as possible”)
  • 59. © Copyright 2000-2016 TIBCO Software Inc. SaaS Machine Learning • Managed SaaS service for building ML models and generating predictions • Integrated into the corresponding cloud ecosystem • Easy to use, but limited feature set and potential latency issues if combined with external data or applications http://docs.aws.amazon.com/machine-learning/latest/dg/tutoria
  • 60. © Copyright 2000-2016 TIBCO Software Inc. PMML (Predictive Model Markup Language ) • XML-based de facto standard to represent predictive analytic models • Developed by the Data Mining Group (DMG) • Easily share models between PMML compliant applications (e.g. between model creation and deployment for operations)
  • 61. © Copyright 2000-2016 TIBCO Software Inc. Agenda 1) Machine Learning and Big Data Analytics 2) Building an Analytic Model 3) Real Time Processing 4) Live Demo 5) Intelligent Microservices
  • 62. © Copyright 2000-2016 TIBCO Software Inc. Analytics Maturity Model Immediate Long-Term Competitive AdvantageValue to the Organization Self-service Dashboards Event Processing Advanced Analytics Measure Diagnose Predict Optimize Alert Automate Analytics Maturity A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases Visual Analytics Event Processing Analytics
  • 63. What is Prescriptive Analytics?
  • 64. © Copyright 2000-2016 TIBCO Software Inc. Real Time Streaming Analytics time 1 2 3 4 5 6 7 8 9 Event Streams • Continuous Queries • Sliding Windows • Filter • Aggregation • Correlation • …
  • 65. © Copyright 2000-2016 TIBCO Software Inc. Operational Intelligence and Human Interaction Actions by Operations Human decisions in real time informed by up to date information 65 Automated action based on models of history combined with live context and business rules Machine-to-Machine Automation
  • 66. © Copyright 2000-2016 TIBCO Software Inc. Alternatives for Streaming Analytics (no complete list!) Azure Microsoft Stream Analytics CLOSED SOURCEOPEN SOURCE FRAMEWORK PRODUCT
  • 67. © Copyright 2000-2016 TIBCO Software Inc. What Kind of Streaming Analytics do you need? Visual IDE (Dev, Test, Debug) Simulation (Feed Testing, Test Generation) Live UI (monitoring, proactive interaction) Maturity (24/7 support, consulting) Integration (out-of-the-box: ESB, MDM, etc.) Library (Java, .NET, Python) Query Language (often similar to SQL) Scalability (horizontal and vertical, fail over) Connectivity (technologies, markets, products) Operators (Filter, Sort, Aggregate) Time to Market Streaming Frameworks Streaming Products Slow Fast Streaming Concepts
  • 68. © Copyright 2000-2016 TIBCO Software Inc. Comparison of Stream Processing Frameworks and Products Slide Deck from JavaOne 2015: http://www.kai-waehner.de/blog/2015/10/25/ comparison-of-stream-processing-frameworks-and-products/ Updated slide deck coming in November 2016 (Big Data Spain, Madrid)
  • 69. © Copyright 2000-2016 TIBCO Software Inc. Visual Coding for Streaming Analytics • Streaming Operators • Connectivity • Visual Development • Testing & Simulation • Mature Tooling / Support • Middleware Integration
  • 70. © Copyright 2000-2016 TIBCO Software Inc. Live Visual Analytics UI Dynamic aggregation Live visualization Ad-hoc continuous query Alerts Action
  • 71. © Copyright 2000-2016 TIBCO Software Inc. How to apply analytic models to real time processing without redevelopment? Stream Processing H20.ai Open Source R TERR Spark ML MATLAB SAS PMML
  • 72. © Copyright 2000-2016 TIBCO Software Inc. TIBCO StreamBase Connector for R and TERR
  • 73. © Copyright 2000-2016 TIBCO Software Inc. TIBCO StreamBase Connector for H2O.ai
  • 74. © Copyright 2000-2016 TIBCO Software Inc. TIBCO StreamBase Connector for PMML
  • 75. © Copyright 2000-2016 TIBCO Software Inc. Real World Streaming Application for Customer Churn
  • 76. © Copyright 2000-2016 TIBCO Software Inc. Closed Loop à Automatically Re-Compute (and Improve) the Analytic Model Compute your performance metric Spot not good enough performance Re-compute model
  • 77. © Copyright 2000-2016 TIBCO Software Inc. Agenda 1) Machine Learning and Big Data Analytics 2) Building an Analytic Model 3) Real Time Processing 4) Live Demo 5) Intelligent Microservices
  • 78. • Reactive – Run to failure • Preventive – Scheduled service (reliability) • Condition-based – Monitor condition (sensors) • Predictive – Predict failures • Proactive – Deploy automatic actions Evolution of Equipment Maintenance Strategies
  • 79. Scenario: Predictive Scrapping of Parts in an Assembly Line Goal: Scrap parts as early as possible automatically to reduce costs in a manufacturing process. Question: When to scrap a part in Station 1 instead of doing re-work or sending it to Station 2? Station 1 Station 2 Cost Before 9€ 7€ 13€ Total Cost 29€ (or more) Scrap? Scrap?
  • 80. Fast Data Architecture for Predictive Maintenance Operational Analytics Operations Live UI CSV Batch JSON Real Time XML Real Time Streaming AnalyticsAction Aggregate Rules Analytics Correlate Live Datamart Continuous query processing Alerts Manual action, escalation HISTORICAL ANALYSIS Data Scientists Flume HDFS Spotfire R / TERR HDFS Hadoop (Cloudera) StreamBase TIBCO Fast Data Platform H2O Oracle RDBMS Avro Parquet … PMML Internal Data
  • 81. TIBCO Spotfire with H2O Integration Data Discovery / Data Mining (“Are parts that repeat a station more likely scrap parts?”)
  • 82. TIBCO Live Datamart Operational Intelligence (“Monitor the manufacturing process and change rules in real time!”) Live Dartmart Desktop Client
  • 83. TIBCO Live Datamart Operational Intelligence (“Monitor the manufacturing process and change rules in real time!”) Live Dartmart Web API
  • 84. TIBCO Spotfire + StreamBase + H2O.ai + Live Datamart Live DemoLive Demo
  • 85. © Copyright 2000-2016 TIBCO Software Inc. TIBCO Accelerator for Apache Spark 1. Fast Data Preparation for IoT Dozens of enterprise and IoT data preparation adapters: MQTT, Databases; inbound creation of HDFS, Parquet, Hbase, Avro… 2. Spotfire Model Discovery Template Use Spotfire to explore Spark data lake, create predictive model, train in H20, and deploy to Streaming Analytics. 3. Operationalize Predictive Models Zookeeper deployment to StreamBase nodes living in Spark cluster via H20, PMML, TERR models 4. Streaming Analytics for Automation Automate action based on predictive models – make offers to customers, stop fraudulent transactions, alert. 5. Monitor & Retrain Model Monitor behavior of model, retrain when necessary. 6. Drag & Drop for Business Solution Developers Code-free development environment for work with H20, HDFS, Avro, TERR The TIBCO Accelerator for Spark is a TIBCO engineered, light-weight open-source fast- start for systems to stream data into Spark, discover patterns in Spark with Spotfire, and operationalize the insights on Big Data. FUNCTIONAL COMPONENTS
  • 86. © Copyright 2000-2016 TIBCO Software Inc. Agenda 1) Machine Learning and Big Data Analytics 2) Building an Analytic Model 3) Real Time Processing 4) Live Demo 5) Intelligent Microservices
  • 87. © Copyright 2000-2016 TIBCO Software Inc. Evolving Demands from the Business AGILITY & SPEED REDUCED CYCLE TIMES WEB SCALE LOWER COST FAIL FAST
  • 88. © Copyright 2000-2016 TIBCO Software Inc. Development of Intelligent Microservices
  • 89. © Copyright 2000-2016 TIBCO Software Inc. 12 Factor Apps for Cloud Native Microservices Codebase One codebase tracked in revision control, many deploys. Dependencies Explicitly declare and isolate dependencies. Config Store config in the environment. Backing Services Treat backing services as attached resources. Build, Release, Run Strictly separate build and run stages. Processes Execute the app as one or more stateless processes. Port Binding Export services via port binding. Concurrency Scale out via the process model. Disposability Maximize robustness with fast startup and graceful shutdown. Dev / Prod Parity Keep dev, staging, and prod as similar as possible. Logs Treat logs as event streams. Admin Processes Run admin/mgmt tasks as one-off processes. https://12factor.net/
  • 90. © Copyright 2000-2016 TIBCO Software Inc. Why Containers? http://www.slideshare.net/andersjanmyr/docker-the-future-of-devops Containers enable: • Lightweight deployment • Automation • Better resource utilization • Scaling up and down quickly • Platform agnostic deployment • Innovation and Fail Fast Concepts • Standardization ? Ø The Open Container Initiative (OCI) Ø Docker Fork Discussions (!!!)
  • 91. © Copyright 2000-2016 TIBCO Software Inc. DevOps Elements – Culture and Technology! Process Tools Automation Culture Continuous Integration/ Continuous Development APIs MicroservicesFrequent releases Collaboration
  • 92. © Copyright 2000-2016 TIBCO Software Inc. Develop fast. Fail fast. Change fast. Visual Analytics + Visual Coding + DevOps = Agile Intelligent Microservices
  • 93. © Copyright 2000-2016 TIBCO Software Inc. Application of Analytic Models to other Microservices
  • 94. © Copyright 2000-2016 TIBCO Software Inc. Real Time Streaming Analytics time 1 2 3 4 5 6 7 8 9 Event Streams Apply your intelligent (micro)service to any event. Microservice event. Application event. Legacy event. IoT event. You name it.
  • 95. © Copyright 2000-2016 TIBCO Software Inc. Key Take-Aways Ø Insights are hidden in Historical Data on Big Data Platforms Ø Machine Learning and Big Data Analytics find these Insights by building Analytics Models Ø Event Processing uses these Models (without Redevelopment) to take Action in Real Time
  • 96. Questions? Please contact me! Kai Wähner Technology Evangelist at TIBCO kontakt@kai-waehner.de @KaiWaehner www.kai-waehner.de LinkedIn