In today’s connected world organizations have access to an enormous amount of data. We often don’t know what they mean or how we can use them, in terms of hindsight, oversight, insight and foresight, to gain competitive advantage in the market. Use cases ranging from simple system monitoring to complex fraud analysis demands this. The WSO2 Data Analytics platform lets you collect data, allows you to explore it through batch, real-time, interactive and predictive processing technologies and allows you to communicate your results. In this talk, we will discuss the WSO2 Data Analytics platform and how it brings together all analytics technologies into a single platform and user experience.

1. An
Introduc+on
to
the
WSO2
Analy+cs
Pla7orm
Sriskandarajah
Suhothayan
(Suho)
Technical
Lead
WSO2

2. Why
Analy)cs?

3. Why
Analy)cs?
Reduce
cost
and
increase
revenue
Improve
customer
sa)sfac)on
Improve
process
efficiency
Track
progress
over
+me
Act
immediately
Predict
future
opportuni+es
and
threats
...

4. WSO2
Data
Analy)cs
PlaAorm

5. Collect
Data

6. Collec)ng
Data
*
Supports
custom
event
receivers
via
its
pluggable
architecture!

7. Store
Data
Data
Abstrac)on
Layer
enabling
pluggable
data
connectors
● Cassandra
● RDBMS
● HBase
● …
With
a
common
API

8. Analysing

9. Interac)ve
Analy)cs
What
has
happened
&
Why
?
● On
demand
analysis
of
data
○ Full
text
search
support
● Selec+ve
deep
analysis
○ Drill
down
search
● Distributed
data
indexing.
● Near
real
)me
data
indexing
and
retrieval
message: “Error” AND
date: [2015-11-01 TO
2015-11-03]

10. Batch
Analy)cs
Summarize
&
Tack
progress
over
)me
…
● Powered
by
Apache
Spark
○ Up
to
30x
higher
performance
than
Hadoop
● Op+mized
in-­‐memory
processing
● Spark
SQL
SQL-­‐like
query
language
● Schedulable
Scripts
○ Simple
Configs
&
Cron
Expressions

11. Real)me
Analy)cs
Detect
and
act
immediately
…
● Streaming
Complex
Event
Processing
○ Uses
WSO2
Siddhi
engine
○ Process
more
than
100k
Events/Sec
● SiddhiQL
SQL-­‐like
Query
Language
● Extension
Support
(Geo,
NLP,
Math,
String,
Regex,
…
)
● Distributed
Processing
with
Apache
Storm
Siddhi

12. Predic)ve
Analy)cs
Predict
future
&
act
upon
it
!
● Explore
and
build
models
○ Via
Spark
MlLib
&
R
(from
WSO2
ML
1.1)
○ Export
as
PMML
(from
WSO2
ML
1.1)
● Run
models
with
CEP,
DAS
and
ESB.
● Run
R
Scripts,
Regression
and
Anomaly
Detec+on
on
real+me
● Deep
learning
&
NLP
(In
roadmap)

13. Machine
Learner

14. Communicate

15. Visualizing
Results
Overall
idea
of
what's
going
on
…
● Dashboard
using
○ Google
Gadget
○ HTML5
+
javascripts
● Support
custom
gadgets
and
dashboards
● Gadget
genera)on
with
○ D3
and
Vega

16. Gadget
Genera)on
Tool
Customize
with
few
clicks
…
● Support
real)me
and
stored
data
visualiza)on
How ?
Start with data in tabular format
Select chart type (Bar, Line, Map, ...)
Map columns to dimensions (X Y, color, point size, … )
Also do drill downs

17. Communica)ng
Results
*
Supports
custom
event
publishers
via
its
pluggable
architecture!

18. Beyond
Boundaries
● Expose
analy+cs
results
as
API
○ Mobile
Apps,
Third
Party
● Provides
○ Security,
Billing,
○ Thro`ling,
Quotas
&
SLA
● How
?
○ Write
data
to
database
from
DAS
○ Build
Services
via
WSO2
Data
Services
Server
or
use
Analy)cs
REST
API
○ Expose
them
as
APIs
via
WSO2
API
Manager

19. Developer
Tools
&
Deployment

20. Developer
Tools
● Event
Flow

21. Developer
Tools
...
● Event
Flow
● Event
Tracing

22. Developer
Tools
...
● Event
Flow
● Event
Tracing
● Event
Sta+s+cs

23. Developer
Tools
...
● Event
Flow
● Event
Tracing
● Event
Sta+s+cs
● Siddhi
Try
It

24. Developer
Tools
...
● Event
Flow
● Event
Tracing
● Event
Sta+s+cs
● Siddhi
Try
It
● Event
Simulator

25. Developer
Tools
...
● Event
Flow
● Event
Tracing
● Event
Sta+s+cs
● Siddhi
Try
It
● Event
Simulator
● Spark
Console

26. Developer
Tools
...
● Event
Flow
● Event
Tracing
● Event
Sta+s+cs
● Siddhi
Try
It
● Event
Simulator
● Spark
Console
● Data
Explorer

27. Developer
Tools
...
● Event
Flow
● Event
Tracing
● Event
Sta+s+cs
● Siddhi
Try
It
● Event
Simulator
● Spark
Console
● Data
Explorer
● Analy+cs
Tooling

28. ● One
Analy)cs
Archive
format
for
all
analy+cs
○ Based
to
Carbon
Archive
● Minimum
HA
Deployment
(2
Nodes)
● Fully
Distributed
Deployment
(8+
Nodes)
● Real)me
Distributed
Deployment
○ Siddhi
runs
on
top
of
Apache
Storm
Deployment

29. Solu)ons

30. Solu)ons
● Fraud
Detec)on
○ Alerts
for
known
and
unknown
frauds
○ Deep
Search
Analy+cs
○ h`ps://goo.gl/TWV5C1
● IoT
Analy)cs
(WIP)
○ With
configurable
Aler+ng
&
Monitoring
capabili+es.
● Log
Analy)cs
(WIP)
● Analy)cs
for
Products
(WIP)
○ Each
WSO2
Product
will
have
analy+cs
of
its
own
kind.

31. Custom
Solu)ons
UI
to
configure
predefined
analysis
with
Execu+on
Manager.
Good
for
Business
Users

32. Use
Cases
&
Case
studies

33. Use
Cases
● Smart
Parking
solu+ons
● Health
&
Financial
Monitoring
● Smart
City
project,
Vehicle
tracking,
Building
monitoring
● Railway
monitoring
● Thro`ling
and
Anomaly
Detec+on
● API
Analy+cs
● Connected
Car

34. Case
Studies
● Real+me
Soccer
Analy+cs
(DEBS
2013)
○ h`ps://goo.gl/c2gPrQ
● TFL
Traffic
Analysis
○ Using
Transport
for
London
open
data
feeds.
○ h`p://goo.gl/04tX6k,
h`p://goo.gl/9xNiCm
● Smart
Meters
(DEBS
2014)
○ Using
smart
home
electricity
data
:
2000
sensors,
40
houses,
4
Billion
events
○ Processes
400K
events/sec,
one
of
the
four
finalists
at
DEBS
2014

35. How
to
get
started
?

36. How
to
get
started
?
●
(WSO2
DAS)
3.0
○ Combines
all
types
of
analy+cs.
●
(WSO2
CEP)
4.0
○ For
who
need
to
analyze
real)me
event
streaming.
●
(WSO2
ML)
1.0
○ For
building
Predic)ve
Models
hAp://wso2.com/analyIcs

37. WSO2
DAS
vs
WSO2
BAM
What's
New
?
● Interac+ve
&
predic+ve
analy+cs
● Data
Abstrac+on
Layer
suppor+ng
○ RDBMS,
Cassandra,
HBase
● Improved
performance
with
Spark
&
Siddhi
● Be`er
Dashboard
● Distributed
Real+me
Processing
● Simple
deployment
model
● ...

38. “The price of light is less than the cost of darkness”
~ Arthur C. Nielsen, Market Researcher & Founder of ACNielsen

39. Thank
You