TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming Analytics

Presented by: Nelson Petracek, Office of the CTO | Strategic Enablement Group
Q4, 2016
Innovation Workshop Series
Reducing Decision Latency with Streaming Analytics

Messaging:
Information processing and analytics cannot be focused only on “store-first” or batch-based approaches.
To provide maximum business value, information must also be analyzed closer to the source, and at the
speed in which it is being created. Streaming analytics utilizes various techniques for intelligently
processing data as it arrives at the edge or within the data center, with the purpose of proactively
identifying threats or opportunities for your business.

Please join us at the next TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming
Analytics to understand how to deliver timely, relevant, and contextual information in real-time to the
systems and people that need it.

What You Will Learn:
•  What is Streaming Analytics, and how does it help deliver business value?
•  What are the key components of a Streaming Analytics processing pipeline?
•  Where does Streaming Analytics fit within an enterprise architecture?
•  What are some common tools and frameworks used to provide stream processing capabilities?
•  How to build a streaming analytics pipeline & real-time visualization, including a hands-on session.
© Copyright 2000-2016 TIBCO Software Inc.
Reducing Decision Latency with Streaming Analytics

BATCH ANALYSIS REAL-TIME ANALYSIS
PROCESS ZONE STRATEGIC TACTICAL OPERATIONS EXECUTION
TIME INCREMENT
IMPORTANCE OF
ALERTS
Quarters/
Years
Months/
Quarters
Hours/
Days/Weeks
Seconds/
Minutes
Not Important Important Necessary
The Evolution of Analytics

STREAMING ANALYTICS
VISUAL ANALYTICS
ADVANCED ANALYTICS
•  Gain insights at the speed of thought
•  Prepare data visually
•  Access connected visualizations –
out of the box
•  Build beautiful dashboards in minutes
•  Collaborate with a click
•  Put your insights on a map
(GeoAnalytics)
DATA DISCOVERY DASHBOARDS
•  Simplify statistical modeling
•  Prescribe your next actions
PREDICTIVE PRESCRIPTIVE
•  Gain continuous awareness
•  Automate actions and alerts
•  Enable human interaction
with live data
REAL TIME ACTIONABLE
The Evolution of Analytics

Insight Platform
Insights Actions
MONITOR
PREDICT
ACT
DECIDE
MODEL
ORGANIZE
ANALYZE
WRANGLE

Traditional Data Processing
•  Data is collected from a variety of
sources, and placed in a persistent
store.
–  Relational database.
–  NoSQL store.
–  Hadoop environment.
•  Analytical processes are executed
against the stored data to detect
opportunities or threats.
•  Actions are identified, delivered,
and executed across various
business channels.
Store
Analyze
Act

Traditional Data Processing: Challenges
Store
Analyze
Act
•  Introduces too much “decision
latency” into the business.
•  Responses are delivered “after-the-
fact”.
•  Maximum value of the identified
situation is lost.
–  Cross-sell / up-sell opportunities are
lost, impending equipment failure is
missed, business processes are slow to
respond and lack timely context.
•  Decisions are made on old and stale
data.

Event Value Decreases Over Time
Value
Time
•  Events are often most
valuable “close to” the
point of collection.
•  As time passes, events tend
to lose their value.
•  The ability to proactively
identify “threats” or
“opportunities” will typically
decrease.
•  Real-time capability is
needed to maximize event
value.

The New Era: Streaming Analytics
•  Events are analyzed and processed in
real-time as they arrive.
•  Decisions are timely, contextual, and
based on fresh data.
•  Decision latency is eliminated, resulting
in:
ü  Superior Customer Experience
ü  Operational Excellence
ü  Instant Awareness and Timely Decisions
Act &
Monitor
Analyze
Store

Streaming Analytics: Real-Time & Actionable
When the temp reading of any sensor
trends toward 150ºC in any 15 minute
window, then alert and create a case
“
”

Market Trends
Big Data
Focus on volume,
velocity, and
variety.
Incorporate real-
time processing
with traditional
data lakes to
improve response
times.
Edge
Computing
Enterprise systems
must be built to
handle data flows
in the data center
& at the edge.
Shift towards
processing data
closer to the
source.
IoT
The vast collection
of devices,
protocols,
standards, and
data must be tied
to new and
existing IT systems
in an efficient and
scalable fashion.
Cloud &
ePaaS
Solutions must be
built to take
advantage of
cloud, ePaaS, and
DevOps
technologies to
reduce TCO and
improve time to
market.
Citizen
Developers
Real-time data
flows need to be
created not only
by IT, but also by
business solution
developers across
various business
units. Self-service
is key.

Evolution of Streaming Analytics
Complex Event
Processing
~ 2000 - 2010
FSI, Transportation
and Logistics,
Retail, Airlines,
Telco,
Manufacturing
Event
Processing
~ 2011 - 2014
Big Data as DW,
open source as
toolkit, cloud.
Clickstream,
Social.
Stream
Analytics
~ 2016+
Gartner term used
to encapsulate EP
& SA.
Streaming
Analytics
~ 2015+
Big Data & open
source as platform,
IoT, Self-Service,
Sensor, Telemetry,
ML & AI, revisit old
use cases.

Initial Focus from Consumer Companies
Clickstream
Sensors
Usage Data
Logs
•  Stream processing is quickly
becoming a mandatory
component of a data processing
architecture.
•  Numerous frameworks have emerged
over the past couple of years – largely
driven by consumer-oriented
companies.
•  Focus is shifting to the integration
of stream processing with
predictive analytics, machine
learning, and rules at the edge
and in the core.

What’s Missing for the Enterprise?
EntityState&
Relationships
In Memory, IMDG/DB, SSD, …
Data Stream
Contextual, Reactive State (Rules & ML) •  Stream processing is not just
about applying mathematical
operations on time-ordered,
unbounded data.
•  Business runs on rules and
algorithms applied to context.
•  Rules are not always
sequential, and may not be
ordered.
•  Often more important to
detect when things “do
not” happen.
•  Manage “reactive state” in
addition to streams.

Use Case Example: Manufacturing Optimization
•  Problem
•  Producing defective,
unreliable or poor
performing product.
•  Shorter lifecycles à faster
new product ramps.
•  Value
•  Increase Yield / % Good
product
•  Reduce Defects & Rework
•  Improve Reliability
•  Use Cases
•  Quality Monitors and
Dashboards
•  Root Cause Analyses
Root Cause:
Machine
Effects
Monitor Product
Quality KPIs and flag
outliers.

Use Case Example: Retail / Real-Time Inventory
•  Problem:
•  Lack of visibility into inventory levels across all
channels.
•  Difficulty allowing shoppers to buy anywhere, fulfill
from anywhere, and return anywhere.
•  Difficulty identifying the best fulfillment locations.
•  Value:
•  Increased customer satisfaction.
•  Reduction in excess inventory.
•  Improved inventory visibility across all channels.
•  Use Cases:
•  “Order Online, Pick-up In Store”
•  Beacons / Contextual Offers
•  Endless Aisle
•  Personalized Shopping Experience

Use Case Example: Distribution Optimization

Streaming Analytics Processing Pipeline

Batch
APIs
Adapters /
Channels
Integration
Messaging
Stream Ingest
Transformation
Aggregation
Enrichment
Filtering
Stream
Preprocessing
Process
Management
Analytics
(Real Time)
Applications
& APIs
Analytics / DW
Reporting
Stream
Outcomes
•  Transform
•  Deep ML
•  Analytics
•  Data Lake
•  …
Stream Analytics &
Processing
Real-Time
Index / SearchNormalization
•  RT Analytics
•  Contextual
Rules
•  Windowing
•  Patterns
•  …

Streaming Analytics: What Is A “Stream”?
Clickstream
Sensors
Usage Data
Logs
•  Consists of pieces of data
typically generated due to a
change of state.
•  One or more identifiers
•  Timestamp & payload
•  Immutable
•  Typically unbounded; there is no
end to the data.
•  Batch dataset: “bounded”.
•  Can be raw or derived.
•  Web logs, mobile usage, sensor
data, clickstream data, etc.
See also: http://research.google.com/pubs/pub43864.html

Streaming Analytics: Ingest
APIs
Adapters /
Channels
Integration
Messaging
Stream Ingest
•  Stream data may come from a number sources,
either at the edge, in the data center, or via the
cloud.
•  Need to handle a variety of data formats and protocols, all at global
scale.
•  Pay attention to “event time” vs. “processing
time” !!
•  Event Time: Time the event was created.
•  Processing Time: Time the event was received or processed.
•  Event time is typically more relevant, and will lead
to more predictable results.
•  Eliminate time skew associated with clock synchronization, system
outages, processing latency, network issues, etc.

Streaming Analytics: Preprocessing
Transformation
Aggregation
Enrichment
Filtering
Stream
Preprocessing
Normalization •  Stream data often needs to be manipulated before it is
processed by downstream components.
•  Normalization
•  Transformation
•  May filter unwanted events close to the source to
eliminate “noise”.
•  Events may also be enriched with additional context to
provide additional data for further processing.
•  Customer details, equipment details, location information, etc.
•  Data may be stored in a high-speed cache or other store for rapid
access.

Streaming Analytics: Processing
Batch
•  Transform
•  Deep ML
•  Analytics
•  Data Lake
•  …
Stream Analytics &
Processing
Real-Time
•  RT Analytics
•  Contextual
Rules
•  Windowing
•  Patterns
•  …
•  Streams may be immediately pushed to a data lake.
•  May be raw or preprocessed.
•  Used for subsequent analysis as part of an immutable data layer.
•  Typically processed in batch in this part of the architecture.
•  In parallel, streams may be processed in real-time
against a number of constructs.
•  Real-time analytics.
•  Graph analysis / Geo Analysis
•  Rules.
•  Results from the real-time processing may be fed into
the batch component.
•  The results of batch processing may also be pushed into the real-
time layer.

Streaming Analytics: “Windows”
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102

Streaming Analytics: “Windows”
select count(*) from /NetworkPing
{policy: maintain last 2 minutes} dosattack
group by 1 having count(*) > 120;

Streaming Analytics: Declarative or Procedural
Sensor
Weather
Write to
DB
Shut Down
SplitUnion
Clean
Clean
Window
Normal
Event
Log
Take
Action

Streaming Analytics: Declarative or Procedural
Data Streams
e.g. Flight Status,
Check-ins,
etc.)
EntityState&
Relationships
In Memory, IMDG/DB, SSD, …
Rulesets
Apply Flight Status Rules
Against Passenger
Manifest,
Determine Notifications,
Update Passenger and
Flight State
Correlate
Inbound
Streams, Load
Associated
Business Entities
(Passenger,
Flight, Loyalty,
etc.
Write Changed
Business Entities
to Backend
State Store
Deliver
Outcomes &
Responses
Change in Passenger
State Automatically
Triggers Passenger
Loyalty Rules, Response
Determined
…

Streaming Analytics: Logic Examples
Simple and Complex Events
•  If F(E) then A(E)
•  If F(E1,E2) then A(E1,E2)
Spatial
•  If x outbreak notifications are “close to” each
other, then …
Business Entities (Concepts)
•  If Changed(Customer) then A(E)
•  If Changed(Customer) & Changed(Product)
Event Routing
•  Route on event contents, historical data, etc.
Events and Business Entities
•  If NewOrder(Customer) & Changed(Product)
…
Event Correlation (Stateful)
•  Correlate on events, business entities, time.
Temporal / Missing Events
•  If E1 followed by E2 within 20 secs then …
•  If No Status Update in 30 mins then…
Rules with Basic Math
•  If slope of pressure > 5 PSI, then investigate
equipment
Aggregation
•  If Volume within last hour > Avg Volume
then ...
Rules on Top of Analytical Models
•  Predictive Model produces prediction
•  Rule applied to prediction
•  If Prob(Fraud) > 0.8, then investigate claim

Streaming Analytics: Machine Learning
Predictive
Attempt to predict
what is “going” to
happen, based on
what “has”
happened in the
past.
Prescriptive
Once the
predictive model is
found, prescribe
your next best or
specific action to
take.
Supervised
Infer a prediction
from a model
produced via set
of training data,
where the inputs
and outputs are
representative of
the real-world.
Unsupervised
Draw inferences
based on input
data with no pre-
determined
responses. Find
hidden patterns or
groupings.

Streaming Analytics: Graph Analysis
Graph databases transform a complex web of dynamic data into meaningful
(and understandable) relationships.
•  Stream Analytics Contextual Data
•  Efficient Entity Link Analysis Triggered by Event Arrival
•  Leverage Connected Data during Stream Processing
•  Eliminate complex joins and self-joins in a relational model.
Flexible Schema
Assumes objects and nodes are linked
by relationships; designed to
constantly evolve, without impacting
performance of existing queries and
app functionality.
Consistent Performance
Index-free adjacency negates
requirement for index lookups –
enabling query performance to
remain relatively consistent, even as
datasets grow.
Increased Value
Enables quick extraction of new insight
from large and complex databases.
Helps uncover unknown interactions
and relationships. Provides valuable
insight into semantic context.

Streaming Analytics: Outcomes
Process
Management
Analytics
(Real Time)
Applications
& APIs
Analytics / DW
Reporting
Stream
Outcomes
Index / Search
•  Stream outcomes or results may be delivered to many
locations either inside or outside the firewall.
•  Data Lake or other analytical store.
•  Search index.
•  Downstream applications.
•  Outcomes may also be used to trigger business
processes or cases for threat or opportunity resolution.
•  Truck Roll to resolve a maintenance issue.
•  Up-sell / Cross-sell, SLA Violation,…
•  Real-time outcomes may also be continuously delivered
to end users based on registered interests.
•  Live datamarts.
•  Operational intelligence.

Streaming Analytics within an Enterprise Architecture

Streaming Analytics: Lambda Architecture Approach
Batch ViewsBatch Storage Batch Layer
Delta Views
Stream
Processing
Speed Layer
Data Stream
Merged Views Serving Layer

Streaming Analytics: Lambda Considerations
•  May result in the need to build and maintain two separate systems.
•  Batch + “Speed” (real-time).
•  May not be possible to merge these systems, due to substantial differences in
the calculations and algorithms run against the data.
•  Frameworks are emerging to address this requirement.
•  However, the systems often have limited or evolving batch or stream processing
features.
•  Speed layer may be just an estimate of results.
•  Correct results are generated when the batch is re-run.
•  Assumes that all data has been received at that time.
•  Speed layer results are eventually replaced with the results of the
batch layer.

Streaming Analytics: Edge Processing
•  Execute event processing logic
at multiple levels within edge
processing “pods”.
•  Integration, rules, analytics...
•  Raw and derived events bubble
up to the enterprise domain.
•  Enterprise domain may push
new knowledge to the lower
levels.
•  View processing results at all
levels in real-time.
•  Execute logic at the edge even
when disconnected.
Level 1
EPP
“Pod”
•  Messaging
•  Event Processing
•  RT Analytics
Context
Level 2
EPP
“Pod”
•  Messaging
•  Event Processing
•  RT Analytics
Context
…
Enterprise Domain

Microservices and Stream Processing
•  Stream processing can play various roles with regards to microservices:
•  Receiver or producer of microservice events (typically async / message-centric services).
•  Event driven choreography / rule evaluation.
•  Event Sourcing / CQRS patterns.
•  Event Sourcing*
•  Persist the events that lead up to a particular state, not the state itself.
•  Easy to recreate a state at any point in time.
•  Other services (e.g. Big Data, Streaming Analytics) can subscribe to event store changes.
•  CQRS*
•  Command Query Responsibility Segregation
•  Services that handle updates are different from those that handle queries.
•  Utilize a “view store” to handle queries.
* http://martinfowler.com/

LoadBalancer(ELB)
EP Microservice
(ECS)
Inventory Service
(BusinessEvents)
Validation
Rules
Choreography
EP Microservice
(ECS)
Inventory Service
(BusinessEvents)
Validation
Rules
Choreography
Clients
Event Store
View Store Aggregate Store
state changes
change events
view queries
Streaming Analytics …
Big Data Store…

https://speakerdeck.com/bobbycalderwood/commander-decoupled-immutable-rest-apis-with-kafka-streams

Device Domain Enterprise SystemsApplication / CEP Zone
Data Zone
Comm Domain
External
Integration
Stream & Event ProcessingMessaging /
Appliances
Analytics
Real-Time
Dashboard
Internal Integration
Routing
DWH
Big Data ETL
Real Time Actions
and ResponsesSMS
Other
Systems SMTP
Real-
TimePublish
Subscribe
Micro
Services
APIs
In-Memory Distributed State Store
Microservices / APIs
Live Datamart
Predictive
Analytics
TERR / PMML
Analytics
On
Hadoop / Spark
QoS Monitoring & Alerting
De-dupe
Buffering
Sequence
Off-route Speed
Stationary
Dwell-time
Clustering
Alerting
Streaming Analytics
Event Driven Rules
Stateful Rule Execution
R Analytics (Enterprise)
BPM & Case
Management
Edge Computing
Routing
Messaging
Microservices / APIs
Event Processing
Analytics
Streaming Analytics: Reference Architecture

Batch
APIs
Adapters /
Channels
Integration
Messaging
Stream Ingest
Transformation
Aggregation
Enrichment
Filtering
Stream
Preprocessing
Process
Management
Analytics
(Real Time)
Applications
Analytics / DW
Reporting
Stream
Outcomes
•  Transform
•  Deep ML
•  Analytics
•  Data Lake
•  …
Stream Analytics &
Processing
Real-Time
Index / SearchNormalization
•  RT Analytics
•  Event Rules
•  Windowing
•  Patterns
•  …
StreamBase
BusinessEvents
Live Datamart
Flogo

1ST EVER
STEPBACK DEBUGGER
FOR FLOWS
TIBCO FLOGO
World’s Lightest Cloud-Native Integration
Designed for Microservice and IoT Driven Integration Use Cases
FLOW-BASED UX FOR
MILLENNIAL INTEGRATORS
GO-BASED ENGINE
100X LIGHTER THAN JAVA
OPEN FOR
COLLABORATION

TIBCO StreamBase
Build Streaming Analytics Apps for Preprocessing & Scoring
Developer Highlights
•  Eclipse-based IDE
•  Visual programming language and debugging
•  Integrated predictive models via TERR
•  Data connectivity with numerous integration points
Runtime Highlights
•  Immense throughput at extremely low latencies
•  Increased scalability without programming errors
•  Execute predictive and prescriptive models in real-time
•  Big Data preparation and preprocessing
•  Simulation / playback capabilities
StreamBase Studio UI

TIBCO BusinessEvents
Build Event-Driven Apps for Contextual Control
Developer Highlights
•  Integrated development environment
•  Graphic editors and model-driven environment
•  Non-linear programming model in the form of declarative rules
Business User Highlights
•  Define and implement rules and logic in a web browser.
•  Decision tables and rule templates
•  Adjust system behavior with no downtime
Runtime Highlights
•  Multi-protocol channel support
•  Event-driven rule evaluation and execution
•  Stateful for reasoning across time and space
•  Multiple deployment topology options
•  Horizontal scalability / Memory management strategies
•  Missing Event Detection
•  Data grid securityBusinessEvents WebStudio UI

TIBCO Live Datamart®
Provide Insight and Instant Command and Control
Business User Highlights
•  On-the-fly actions on detected opportunities and threats
•  Live interaction with stream data: ad-hoc queries, alerts
•  Live drill down to linked visualizations
•  Fully customizable by the end user
TIBCO LiveView Server Highlights
•  Ultra-fast, continuous querying
•  Push based notifications and alerting
•  Color-keyed prioritization based on rules
•  Connectivity to numerous endpoints
•  Mashup of historical and real-time data
Live Desktop UI

LiveView Web: Active Visualization & Alerting
Zero install, streaming ready.
WebSockets-Powered HTML5
Highcharts-powered visualizations display
your data as it changes.
Real-time Visualizations
Fine-tune your experience by authoring or
downloading Javascript extensions from
the community exchange.
JavaScript Plug-ins

TIBCO Spotfire: Visual & In-Depth Analytics
VISUAL ANALYTICS
•  Gain insights at the speed of thought.
•  Prepare data visually.
•  Access connected visualizations –
out of the box.
•  Build beautiful dashboards in minutes.
•  Collaborate with a click.
•  Put your insights on a map (GeoAnalytics).
DATA DISCOVERY DASHBOARDS
IN-DEPTH ANALYTICS
•  Simplify statistical modeling.
•  R, TERR, SAS, PMML
•  Prescribe the right actions to transform
your business, build loyalty, increase
revenues.
•  Forecast your future accurately.
PREDICTIVE PRESCRIPTIVE

More Information
TIBCO Blog and Community:
http://www.tibco.com/blog/
https://community.tibco.com/
TIBCO BusinessEvents
http://www.tibco.com/products/event-processing/complex-event-processing/businessevents
TIBCO StreamBase
http://www.tibco.com/products/event-processing/complex-event-processing/streambase-complex-event-
processing
TIBCO Live Datamart:
http://www.tibco.com/products/event-processing/complex-event-processing/streambase-liveview

Summary
•  Streaming Analytics is a core component
of the TIBCO product suite, and a “must-
have” for solving today’s issues.
•  By moving towards a real-time, event-
enabled architecture, business can
optimize their real-time analytics
solutions, and realize the benefits of an
intelligent event network.
•  TIBCO and the Insight Platform allows
organizations to rapidly develop event-
driven systems that enable the next
generation of analytics solutions.
Insights Actions

TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming Analytics

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming Analytics

Similar to TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming Analytics (20)

Recently uploaded

Recently uploaded (20)

TIBCO Innovation Workshop Series: Reducing Decision Latency with Streaming Analytics