Continuous Intelligence: Staying Ahead with Streaming Analytics

Welcome

Host:
Eric Kavanagh
eric.kavanagh@bloorgroup.com

Twitter Tag: #briefr The Briefing Room

Mission

!   Reveal the essential characteristics of enterprise software,
good and bad

!   Provide a forum for detailed analysis of today s innovative
technologies

!   Give vendors a chance to explain their product to savvy
analysts

!   Allow audience members to pose serious questions... and get
answers!


MARCH: Operational Intelligence

April: INTELLIGENCE
May: INTEGRATION
June: DATABASE


Operational Intelligence

REAL-TIME…

Processing Monitoring Alerts/triggers/actions


Analyst: Mark Madsen

Mark Madsen is president
of Third Nature, Inc.


SQLstream

! SQLstream is an enterprise software company focused on
making businesses responsive to real-time big data assets

!   Its platform provides a relational stream for analyzing large
volumes of service, sensor, and machine and log file data

!   SQL queries in SQLstream generate results continuously as
data becomes available


Damian Black
Damian Black is the founder and CEO of
SQLstream, a pioneer in Streaming Big Data.
Damian has worked for almost two decades
in Silicon Valley, with senior roles in a variety
of companies including Hewlett-Packard,
Neustar, Xacct Technologies and Followap.
He has spoken at many conferences, and was
on GigaOM’s first Big Data panel in 2008.
Damian graduated from Manchester
University and was one of the first research
scientists to join HPLabs Europe. He was
selected for the International Management
Challenge in conjunction with the Financial
Times and Ashridge business school while at
Hewlett-Packard. Damian is the author of
eleven granted patents with five more
pending.


BIG DATA ON TAP™
Continuous Intelligence:
Staying Ahead with Streaming Log File Analytics
M a r c h 2 0 1 3

D a m i a n B l a c k , C E O , S Q L s t r e a m

Copyright © SQLstream Inc.

M a c h i n e - G e n e ra t e d B i g D a t a E x p l o s i o n
High volume, high velocity, structured and unstructured data from software platforms, applications and systems

Machine-generated data will increase
to 42% of all data
by 2020, up from GPS
11% in 2005. Telematics
“The Digital Universe in 2020”
IDC
IP Networks, Video
Servers, Social Media, Security
Servers, Applications, Storage Networks

Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com | 10

O P E R AT I O N A L I N T E L L I G E N C E
Bridging the Chasm Between Analytics and Operations

VELOCITY

UNSTRUCTURED DATA
VOLUME
VARIETY
VISUAL
VALUE
STRUCTURED DATA
TRANSACTIONS

➔  Predictive analytics
➔  Automated actions
Business Intelligence
➔  Ops optimization
➔  Post-hoc analysis
➔  Tactical execution
Business Applications ➔  Data warehousing
➔  Transactions ➔  Strategic direction
➔  Everyday business

Real-time, continuous Historical, periodic Real-time, continuous

M AC H I N E D AT A T O O P E R AT I O N A L I N T E L L I G E N C E

PROACTIVE

REACTIVE

REAL-TIME WEB SERVER LOG MONITORING
M o z i l l a ( G o o g l e : “ Yo u t u b e M o z i l l a G l o w ” )

Real-time monitoring across all download web Web Server Log Files (Remote)

servers across the world simultaneously.

Streaming collection, real-time
analysis and continuous integration
Collect by location

Remote agents transform log ﬁles into real-time
streams

Hadoop HBase

Analyze
Real-time analysis & aggregation by location

Share
Continuous ETL into Hadoop Hbase

Internet ‘Glow’ app for real-time visualization



parse parse
parse parse off
Filter Merge
parse parse
Logs Parse Bad recs

Streaming Analyze Filter out Add
Analytics Errors Bots Location

Streaming
Visualization
HBase
Historical
Charts



Mozilla Firefox 4 – Real-time Download Monitor

Continuous processing of download requests

Real-time integration with Hadoop and HBase


M AC H I N E D AT A
Where is the intelligence?

Transaction TRANS,2013-02-17-15:30:22,3458783,2347897953,128.56.0.253,STATUS:-15, DE69975, 4157588342
Log Details

Web Server [Sun Feb 17 15:30:49 2013] [notice] srv-sfo-08 caught SIGTERM, shutting down
Logs [Sun Feb 17 15:30:49 2013] [notice] Apache/2.2.21 -- resuming normal operations

TERMINATE,ctl09gsx,01299796304,GMT-08:00,02-17-13,15:21:00,9,387,64ms,02-17-13,15:30:55,0005,
CDR Records IP-TO-IP,4157588342,8775715775,1,0,4157588342,RD_AXY_NN0_001,SFR01AAG34,40.50.245.60,
234.234.60.75,65678,411,399,SIP,SANFRANCISCO,0x4B1698,0x0005E,0x49768,4157588342,0198873465

<id>1597831220</id><deviceid>0198873465</deviceid><lat>lat=47.643957</lat><lon>lon=
-122.3269</lon><time>2013-02-17T15:37:26Z</time><bearing>223.4535</bearing>

Smartphone <id>1597865781</id><deviceid>0198873465</deviceid><lat>lat=47.645982</
GPS Updates lat><lon>lon=-122.327500</lon><time>2013-02-17T15:37:26Z</time><bearing>200.6138</bearing>
<id>1597940125</id><deviceid>0198873465</deviceid><lat>lat=47.647381</
lat><lon>lon=-122.326501</lon><time>2013-02-17T15:37:26Z</time><bearing>87.4357</bearing>

{"created_at:Thu Feb 17 15:30:55 +0000 2013,id:304612775055998976,id_str:
Twitter 304612775055998976,text:@MyServiceProvider today sucks, keeps dropped!,source:u006ca
href=http:www.url.com rel=nofollow,followers_count:147,friends_count:10142, location: San Francisco,
time_zone: Pacific, geo_enabled:true, location:u00dcT: -6.1987552,106.8661953, screen_name:APerson


M AC H I N E D AT A

Timestamp
Log Details

Timestamp

Timestamp
CDR Records IP-TO-IP,4157588342,8775715775,1,0,4157588342,RD_AXY_NN0_001,SFR01AAG34,40.50.245.60,


Timestamp
Timestamp


M AC H I N E D AT A

Timestamp Customer Fail Code Mobile #
Log Details

Timestamp Server

Timestamp
Mobile #
CDR Records Term Reason Device ID
IP-TO-IP,4157588342,8775715775,1,0,4157588342,RD_AXY_NN0_001,SFR01AAG34,40.50.245.60,

Device ID Location

Timestamp
Timestamp
Service Provider
Location


O P E R AT I O N A L S T R E A M I N G B I G D AT A – PA I N P O I N T S

DATA EXPLOSION

Too costly to analyse voluminous real-time data

BUSINESS AGILITY

Too slow to respond to new requirements

COMPLEXITY

Too difﬁcult to build & maintain real-time apps


O P E R AT I O N A L S T R E A M I N G B I G D AT A – PA I N P O I N T S

DATA EXPLOSION

Too costly to analyse voluminous real-time data

SQLstream slashes TCO for real-time analysis.

BUSINESS AGILITY

Too slow to respond to new requirements

SQLstream allows you to add new apps easily.

COMPLEXITY

Too difﬁcult to build & maintain real-time apps

SQLstream eliminates your development risk.


C O N T I N U O U S O P E R AT I O N A L I N T E L L I G E N C E

M2M

Automotive

Telecom

Banking
Logs
Manufacturing

Oil & Gas RFIDs
GPS
Logistics
Servers
Smart
grid Networks
Retail
Real-time
alerts, action
Sensors Social and
media visualization
Telematics

Data
centers
Internet


C O N T I N U O U S O P E R AT I O N A L I N T E L L I G E N C E

•  Collect, transform and deliver: ETL++
M2M •  Analyze unstructured data & enhance
•  Predictive analytics & actions
Automotive

Telecom

Banking
Logs
Manufacturing

Oil & Gas RFIDs
GPS
Logistics
Servers
Smart
grid Networks
Retail
Real-time
alerts, action
Sensors
Enhance Store and
Social
media with detail and visualization
Telematics aggregate
historical
Data
information data
centers
Internet


M O V I N G F R O M H I G H L AT E N C Y
TO REAL-TIME RESPONSIVENESS

Traditional approach leads to high latency
COLLECT

CLEANSE

ENRICH

ANALYZE

SHARE

HIGH LATENCY


M O V I N G F R O M H I G H L AT E N C Y
TO REAL-TIME RESPONSIVENESS

Traditional approach leads to high latency
COLLECT

SQLstream streaming approach:
»  Continuous Parallel Dataﬂow Execution

CLEANSE

»  Generate real-time answers immediately

»  Deliver and share the results immediately

ENRICH

ANALYZE

SHARE

LOW LATENCY


S Q L S T R E A M D AT A F L O W T E C H N O L O G Y
P I P E L I N I N G A N D S U P E R S C A L A R PA R A L L E L P R O C E S S I N G

Query
=

Processor

Fine-grained parallelism: simple, massively scalable, super fast.


S H A R E S T R E A M I N G B I G D AT A

Use SQLstream and ISO/ANSI standard SQL
»  Proven performance, optimization and scalability

»  Rapid app development with familiar language

»  Leverage existing SQL skills & investment

Streaming SQL Views

CREATE VIEW compliant_orders AS
SELECT STREAM *
FROM orders OVER sla
GENERATES THE STREAM
JOIN shipments
ON orders.id = shipments.orderid
OF NEW YORK ORDERS
WHERE city = 'New York' SHIPPING WITHIN A
WINDOW sla AS SERVICE LEVEL OF 1hr

(RANGE INTERVAL '1' HOUR PRECEDING)


A S T R E A M I N G S Q L Q U E RY
CLOUD INFRASTRUCTURE MONITORING WITH BOLLINGER BANDS

SELECT STREAM ROWTIME, url, numErrorsLastMinute
FROM ( SELECT STREAM ROWTIME, url, numErrorsLastMinute,
AVG(numErrorsLastMinute) OVER lastMinute AS avgErrorsPerMinute,
STDDEV(numErrorsLastMinute) OVER lastMinute AS stdDevErrorsPerMinute
FROM ServiceRequestsPerMinute
WINDOW lastMinute AS (PARTITION BY url RANGE INTERVAL ‘1’ MINUTE PRECEDING) ) AS S
WHERE S.numErrorsLastMinute > S.avgErrorsPerMinute + 2 * S.stdDevErrorsPerMinute;

BUSINESS NEED:

Detect run-away applications
before resource consumption
becomes an issue.


T H E R E A L - T I M E DATA M A N AG E M E N T H E A DAC H E

Finance

Supply Chain

CRM

Operations

Business Intelligence:

&

&

&

&

Hadoop HBase &

Accounting

ERP

Billing

Management

Data Warehouses

TIME, MONEY, COMPLEXITY


T H E R E A L - T I M E DATA M A N AG E M E N T S O LU T I O N

Finance

Supply Chain

CRM

Operations

Business Intelligence:

&

&

&

&

Hadoop HBase &

Accounting

ERP

Billing

Management

Data Warehouses

STREAMING
STEAMING EVENT STREAMING

CONTINUOUS

ANALYTICS AND
CORRELATION

ALERTS & ALARMS

ETL

AGGREGATION


S Q L S T R E A M S T A N D A R D I N T E G R AT I O N A D A P T E R S

Core Database Adapter

DATA B A S E S

Table Reader

Table Update

Table Lookup (any JDBC)

B I G D A T A

Hadoop

BigQuery

+ HDFS

+ HBase

MACHINE DATA

Log Files

XML Parse

Sockets

JDBC

+ Remote Agent

+ XPath

+ TCP

+ JMS

XML

+ FileWriter

+ UDP

+ log4j

+ FileReader

Middleware

Web Feeds

GATE

Email

+ Twitter

+ RSS

+ ATOM etc

T

Semantic
Streaming

STORM


S T R E A M I N G V I S U A L I Z AT I O N


R E A L - T I M E O P E R AT I O N A L I N T E L L I G E N C E
M A R K E T C O M PA R I S O N

ENTERPRISE

OPERATIONAL INTELLIGENCE

OPERATIONAL INTELLIGENCE
REQUIREMENT

WITH OTHERS

WITH SQLSTREAM

Time Series Analytics

Simplistic answers without time series.

Comprehensive times series support.

Complex Analysis

Simple pattern matching and statistics.

Elegantly solves hardest problems.

Join Correlate

Does not combine or join streams.

Joins data streams in real-time.

Enrich Integrate

Does not enrich or integrate data.

Gives rich answers in real-time.

Big Data Scalability

No parallel processing; limited scalability.

Massively parallel, auto-optimizing.

Painless TCO

Very expensive, proprietary, with only Low TCO, ANSI/ISO standard
basic visualization.

queries, rich real-time visualization.


S Q L S T R E A M : B I G DATA O N TA P ™ , d e l i ve r e d

Slashing TCO for real-time analysis

DATA EXPLOSION

•  Scales easily without transaction bottlenecks.

Adding new apps easily

BUSINESS AGILITY

•  Shares dynamic results and data across the organization.

Eliminating the development risk

COMPLEXITY

•  Fine-grained parallel processing: simple, scalable and fast.


O P E R AT I O N A L I N T E L L I G E N G E - B E YO N D I T

ENVIRONMENTAL TRANSPORTATION NETWORKS

Environmental Monitoring Location-based services Machine-to-Machine

Smart Grid Cars as Sensors Logistics


About
the
Presenter

Mark
Madsen
is
president
of
Third

Nature,
a
technology
research
and

consul8ng
ﬁrm
focused
on
business

intelligence,
data
integra8on
and
data

management.
Mark
is
an
award-‐winning

author,
architect
and
CTO
whose
work

has
been
featured
in
numerous
industry

publica8ons.
Over
the
past
ten
years

Mark
received
awards
for
his
work
from

the
American
Produc8vity

Quality

Center,
TDWI,
and
the
Smithsonian

Ins8tute.
He
is
an
interna8onal
speaker,

a
contributor
at
Forbes
Online
and

Informa8on
Management.
For
more

informa8on
or
to
contact
Mark,
follow

@markmadsen
on
TwiMer
or
visit

hMp://ThirdNature.net

Con.nuous

Intelligence:
Staying

Ahead
with

Streaming
Analy.cs

March,
12
2013

Mark
Madsen

www.ThirdNature.net

@markmadsen

The “E” in EDW
was a lie…

Transac.ons
vs.
Events

Transac8ons:

▪  Each
one
is
valuable

▪  The
elements
of
a
transac8on
can
be
aggregated
easily

▪  A
set
of
transac8ons
does
not
usually
have
important
ordering

or
dependency

Events:

▪  A
single
event
oUen
has
no
value,
e.g.
what
is
the
value
of
one

click
or
one
temperature
reading
in
a
series?

▪  Some
events
are
extremely
valuable,
but
this
is
only

detectable
within
the
context
of
other
events.

▪  Elements
of
events
are
oUen
not
easily
aggregated

▪  A
set
of
events
usually
has
a
natural
order
and
dependencies

General
model
for
organiza.onal
use
of
data

Collect Act on the process
new data Usually days/longer timeframe

Analyze Analyze
Monitor Decide Act
Exceptions Causes

No problem No idea Do nothing

Act within the process
Usually real-time to daily

You
need
to
be
able
to
support
both
paths

Analytics and BI
Collect
new data Act on the process

Analyze Analyze
Monitor Decide Act
Exceptions Causes

Act within the process
Streaming technologies

Diﬀerent
Usage
Model
Than
Conven.onal
BI

A)  Monitoring
and
detec8on
is
not
repor8ng
and

dashboards.
Self-‐service
BI
doesn’t
do
it

B)  Lots
of
data,
decreasing
in
value
as
the
events

recede
in
8me

C)  Analy8cs
oUen
required
to
surface
meaningful

events,
which
requires
collec8on
and
processing

of
(B)
to
process
in
real
8me
to
deliver
(A).

D)  Actua8on:
machine
managed,
human
mediated

The
future
is
not
data
to
eyeballs,
its
machines
to
machines

Measurement
started
with
the
convenient
data

The
convenient
data
is

transac8onal
data.

▪  Goes
in
the
DW
and
is
used,
even

if
it
isn’t
the
right
measurement.

The
inconvenient
data
is

observa8onal
data.

▪  It’s
not
neat,
clean,
or
designed

into
most
systems
of
opera8on.

We
need
to
build
infrastructure

that
manages
and
enables
use
of

data
at
rest
and
data
in
mo8on.

Bridge
the
data
warehouse
to
other
uses:
SOA,
not
SQL

New
technologies
are
needed
to
extend
current
capability.

http://flickr.com/photos/higaara/228673603/

Ques.ons

1.  Queues
and
streams
process
messages
and

objects.
How
is
that
made
SQL
compa8ble?

2.  Why
SQL
when
the
standard
is
missing

temporal
constructs
for
this?

3.  How
do
you
use
a
single
SQL
statement
across

mul8ple
streams
(i.e.,
scale
out
the
query)?

4.  How
much
work
is
human-‐monitored,
vs.

human
no8ﬁed,
vs.
machine
actuated?
How

big
is
this
problem,
really?

Ques.ons

5.  What
about
playback?
How
do
you
replay

history
to
trace
an
event?

6.  What
tooling
is
required?
Is
it
possible
to
add

stream
monitoring
and
use
exis8ng
BI
tools,
or

do
we
need
new
end
user
tools?

7.  Linking
the
in-‐mo8on
to
the
sta8onary,
what

are
the
mechanisms?

About
Third
Nature

Third Nature is a research and consulting firm focused on new and
emerging technology and practices in analytics, business intelligence, and
performance management. If your question is related to data, analytics,
information strategy and technology infrastructure then you‘re at the right
place.
Our goal is to help companies take advantage of information-driven
management practices and applications. We offer education, consulting
and research services to support business and IT organizations as well as
technology vendors.
We fill the gap between what the industry analyst firms cover and what IT
needs. We specialize in product and technology analysis, so we look at
emerging technologies and markets, evaluating technology and hw it is
applied rather than vendor market positions.

Upcoming Topics

April: INTELLIGENCE
May: INTEGRATION
June: DATABASE

www.insideanalysis.com


Thank You
for Your
Attention

Certain images and/or photos in this presentation are the copyrighted property of 123RF Limited, their Contributors or Licensed Partners and are being
used with permission under license. These images and/or photos may not be copied or downloaded without permission from 123RF Limited.


Continuous Intelligence: Staying Ahead with Streaming Analytics

More Related Content

Similar to Continuous Intelligence: Staying Ahead with Streaming Analytics

More from Inside Analysis

Recently uploaded

Continuous Intelligence: Staying Ahead with Streaming Analytics