The document discusses applying machine learning models to real-time data streams. It covers building analytic models from historical data using tools like R, TensorFlow and Hadoop. It then discusses applying these pre-built models to real-time streaming data using frameworks like Apache Spark, TIBCO StreamBase and H2O to power applications like predictive maintenance and manufacturing analytics. The key takeaway is that machine learning on historical data finds insights, while stream processing applies these models in real-time to drive closed-loop analytics and enable real-time action.

1. Kai Wähner
Technology Evangelist
kontakt@kai-waehner.de
LinkedIn
@KaiWaehner
www.kai-waehner.de
Codemotion Tel Aviv, Israel (March 2017)
Advanced Analytics and Machine Learning
with R, Spark, TensorFlow and H2O for Real Time Processing

2. © Copyright 2000-2017 TIBCO Software Inc.
Agenda
1) Machine Learning and Big Data Analytics
2) Building an Analytic Model
3) Applying an Analytic Model in Real Time

3. © Copyright 2000-2017 TIBCO Software Inc.
Agenda
1) Machine Learning and Big Data Analytics
2) Building an Analytic Model
3) Applying an Analytic Model in Real Time

4. Machine Learning
…. allows computers to find hidden insights without being
explicitly programmed where to look.

5. Real World Examples of Machine Learning
Spam Detection
Search Results +
Product Recommendation
Picture Detection
(Friends, Locations, Products)
Your Company
The Next Disruption:
Google Beats Go Champion

6. © Copyright 2000-2017 TIBCO Software Inc.
From Insight to Action - Closed Loop for Big Data Analytics

7. © Copyright 2000-2017 TIBCO Software Inc.
Agenda
1) Machine Learning and Big Data Analytics
2) Building an Analytic Model
3) Applying an Analytic Model in Real Time

8. © Copyright 2000-2017 TIBCO Software Inc.
Analytical Pipeline
1. Data Access
2. Data Preparation
3. Exploratory Data Analysis
4. Model Building
5. Model Validation
6. Model Execution
7. Deployment

9. © Copyright 2000-2017 TIBCO Software Inc.
Variety of Data in Enterprises
Custom GUI-driven
data access via SDK
Siebel
eBusiness
Local data sources
AccessExcel STDF
Drag-and-drop
MySQL
SQL Server
Oracle
Information Services
(join, transform, reusable,
parameterized, dynamic query
for in-memory use)
Databases
JDBC/ODBC
Hadoop
SFDC
PostgreSQL
Teradata
Netezza
Etc.XML
RDBMS
Flat
Files
Spread-
sheets
Web
Services
Oracle
E-Business
RDBMS
RDBMS
RDBMS
SAP BWSAP R/3 D
A
T
A
F
A
B
R
I
C
Salesforce
ODBC
OLE DB
SqlClient
Direct
connection
Oracle
TeradataAsterMS SSAS
Teradata
Direct Query
(dynamically query and retrieve data for
visualization and analysis)
Databases
MySQL
Etc.
OBIEE
Netezza
Hadoop

10. © Copyright 2000-2017 TIBCO Software Inc.
Data Preparation
http://www.slideshare.net/odsc/feature-engineering
Data Preparation

11. Visual Analytics - Interactive Brush-Linked
© Copyright 2000-2017 TIBCO Software Inc.

12. © Copyright 2000-2017 TIBCO Software Inc.
Model Building
A model is a simplification of the truth
that helps you with decision making.

13. © Copyright 2000-2017 TIBCO Software Inc.
Cross-Validation Procedure
https://genome.tugraz.at/proclassify/help/pages/XV.html

14. © Copyright 2000-2017 TIBCO Software Inc.
Execution via Code / Scripting

15. Execution within the Visual Analytics Tooling
© Copyright 2000-2017 TIBCO Software Inc.
Customer Churn with Random Forest Algorithm:
Select variables
for the model

16. © Copyright 2000-2017 TIBCO Software Inc.
Frameworks and Tooling

17. Advanced Analytics and Big Data Tools for Data Scientists
Many more ….
Portable Format
for Analytics (PFA)

18. Development of Analytic Models
with R, TensorFlow, Apache Spark, RapidMiner, TIBCO Spotfire
Live DemoLive Demo

19. © Copyright 2000-2017 TIBCO Software Inc.
Agenda
1) Machine Learning and Big Data Analytics
2) Building an Analytic Model
3) Applying an Analytic Model in Real Time

20. © Copyright 2000-2017 TIBCO Software Inc.
Analytical Pipeline
1. Data Access
2. Data Preparation
3. Exploratory Data Analysis
4. Model Building
5. Model Validation
6. Model Execution
7. Deployment

21. © Copyright 2000-2017 TIBCO Software Inc.
Streaming Analytics - Processing Pipeline
APIs
Adapters /
Channels
Integration
Messaging
Stream Ingest
Transformation
Aggregation
Enrichment
Filtering
Stream
Preprocessing
Process
Management
Analytics
(Real Time)
Applications
& APIs
Analytics /
DW Reporting
Stream
Outcomes
• Contextual Rules
• Windowing
• Patterns
• Analytics
• Deep ML
• …
Stream Analytics &
Processing
Index / SearchNormalization
Applying an Analytic Model
is just a piece of the puzzle!

22. © Copyright 2000-2017 TIBCO Software Inc.
Frameworks and Products
(no complete list!)
OPEN SOURCE CLOSED SOURCE
PRODUCT
FRAMEWORK
Azure Microsoft
Stream Analytics

23. © Copyright 2000-2017 TIBCO Software Inc.
How to
apply analytic models
to real time processing
without redevelopment?
Stream
Processing
H20.ai
Open
Source
R
TERR
Spark
ML
MATLAB
SAS
PMML

24. Apache Spark ML and Spark Streaming with PMML Models
https://github.com/jpmml/jpmml-spark
© Copyright 2000-2017 TIBCO Software Inc.
Apply model to new events
Build
model

25. © Copyright 2000-2017 TIBCO Software Inc.
TIBCO StreamBase Connector for R and TERR

26. © Copyright 2000-2017 TIBCO Software Inc.
TIBCO StreamBase Connector for H2O.ai

27. © Copyright 2000-2017 TIBCO Software Inc.
TIBCO StreamBase Connector for PMML

28. Scenario: Predictive Scrapping of Parts in an Assembly Line
Station 1 Station 2
Cost Before
9€
7€ 13€
Total Cost
29€
(or more)
Scrap? Scrap?
Example: Predictive Analytics for Manufacturing (“scrap parts as early as possible”)

29. Fast Data Architecture for Predictive Maintenance
Operational Analytics
Operations
Live UI
CSV Batch
JSON Real Time
XML Real Time
Streaming AnalyticsAction
Aggregate
Rules
Analytics
Correlate
Live Datamart
Continuous query
processing
Alerts
Manual action,
escalation
HISTORICAL ANALYSIS Data
Scientists
Flume
HDFS
Spotfire
R / TERR
HDFS
Hadoop (Cloudera)
StreamBase
TIBCO Fast Data Platform
H2O
Oracle RDBMS
Avro Parquet … PMML
Internal Data

30. TIBCO Spotfire with H2O Integration
© Copyright 2000-2017 TIBCO Software Inc.
Example: Predictive Analytics for Manufacturing (“scrap parts as early as possible”)

31. TIBCO StreamBase / Live Datamart + H2O.ai
Live DemoLive Demo

32. © Copyright 2000-2017 TIBCO Software Inc.
From Insight to Action - Closed Loop for Big Data Analytics
Insight Action
MONITOR
PREDICT
ACT
DECIDE
MODEL
ACCESS
ANALYZE
WRANGLE

33. © Copyright 2000-2017 TIBCO Software Inc.
Key Take-Aways
Ø Insights are hidden in Historical Data on Big Data Platforms
Ø Machine Learning and Big Data Analytics find these Insights by building Analytics Models
Ø Event Processing uses these Models (without Redevelopment) to take Action in Real Time

34. Questions? Please contact me!
Kai Wähner
Technology Evangelist
kontakt@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
LinkedIn