Data to Drive Decision-Making - CaliStream Meetup

Take Control ofYour Data: CaliStream.com © CaliStream.com 2015
Data to Drive Decision-Making
2015-03-03
Jerome Boulon
1

Quick History
1999 2008 2009 2010 2012
Yahoo!: Chukwa
Hadoop Monitoring Solution
Netflix: Honu
Data Collection Pipeline
CaliStream: Founder
Honu: Data As a Service
Monitoring Solution
for cable modems/TV
network
Ontology Search
Acquired by Microsoft

Agenda
3
• Agenda
• Decision-Making, the process
• Netflix Recommendation
• Big Data @ Riot Games
• Data Pipeline
• Conclusion

Decision-Making, The Process
• Explicit
• Theory Driven
• Data-driven
• Measurable Outcomes
• Iterative
Prove a hypothesis right (or wrong)
Want result AND explanation

Iterative
Hypothesis
Gather DataAnalyze Data

Offline
Testing
Online
A/BTesting
Roll-out
To Prod
Offline/Online Testing
Fail
Success
Success
Iterations
Iterations

Netflix Recommendation

Presentation 1
1/3
0/3
0+/3
1+/3 Presentation 2
1/3
1/3
0+/3
2+/3
Netflix Recommendation

Offline
Testing
Online
A/BTesting
Roll-out
To Prod
The Data
Fail
Success
Success
Search
Time
Rating
Impression
Demographic
Social
User Behavior
Geo Information

The Gaming Space

Use Case: Honu @
• Error code
• AVG Ping latency
• AVG Queue
Wait time
• ClientVersion
• Operating System
• Hardware Profile
• Etc

The Data Pipeline
12

Data Pipeline
• Direct correlation to the success of your Big Data
project
• Structured and Multi-Structured Events
• Schema evolution
• Collecting massive amount of data live
• 60% of BI project resources is consume here!
• Most “underestimated” and “unsexy” but MOST
important phase

S3
Apps
Sensor
Data
Click
Stream
Location
…
• Built for Netflix scale
• 100+ Billions events/Day
• Automatic Discovery,
Load-balancing, Fail-over
• Schema-less
• …
HONU

S3
Apps
Sensor
Data
Click
Stream
Location
…
• Team
• Event format
• Protocol
• Discovery
• Load balancing
• Fail-Over
• Kinesis/Kafka/Scrib
e …
• …
Collect
Collect
Collect
Collectors

S3
Apps
Sensor
Data
Click
Stream
Location
…
Collect
Collect
Collect
Collectors
• HadoopTeam
• Hadoop knowledge
• Hive
• Schema evolution
• Upgrades
• Files optimization
• …

S3
Apps
Sensor
Data
Click
Stream
Location
…
Time to market
> 18 Months !
Collect
Collect
Collect
Collectors

CaliStreamTake Control ofYour Data

S3
Apps
Sensor
Data
Click
Stream
Location
…
HONU
CaliStream:
Honu as a Service
Time to market (Hours)

Our Solution: CaliStream
CaliStream provides
a SaaS data processing pipeline
to easily stream large volume of events
from your applications directly to Hive/Hadoop
in a robust, scalable and cost effective way
without any prior Hadoop Knowledge

CaliStream, Native Hive integration
Big
Data
Sensor
Data
Social
Click
Stream
Location
logs
Sensor
Data
Click
Stream
Location logsSocial
… …
CaliStream

CaliStream
Everything in CaliStream
is represented as a Hive table
so you can easily analyze your data
Select […]
from[…]
where[…]
group by […] ;

CaliStream
HiveTable
Java API

Generic REST API + JSON
CaliStream
HiveTable

HTTPRestProxy
Collectors
BI Analytics
Redshift
CaliStream
EchoService
AppsClusters

CaliStream.comTake Control ofYour Data
Jerome Boulon
jboulon@caliStream.com
CaliStreamTake Control ofYour Data

Backup Slides
27

BeaconServers
Collectors
BI Analytics
Redshift
AppsClusters
*
Ownership
CaliStream
* Run on your account
Connectors
Your
Company
Your
Company
Your
S3 Bucket

IoT Use Cases

IoT Use Case 1
30

Use Case: Global Boats tracking system
• Boats information
• Telemetry data
• etc

IoT Use Case 2
32

Use case: Small company in
Montreal
• Business model: Sensor data acquisition for cities
worldwide (Montreal,Toronto, Paris, …)
Database
Montreal
Database
City 2
Database
City n

Problems
• Data silos: each city is self-contained, managed
with his own deployment
• Strict data access and security rules that limit
business opportunities
• Deriving value from larger data set is a tedious
and manual task
• Multiple copies of the same data
• Etc

Customer Initial Architecture
35
Database
Sensors
Sensor Listener Service

CaliStream Integration (< 1 Week)
36
Sensors
Sensor Listener Service
CaliStream
Database
Table-1 Table-2 Table-n
S3
Data
Warehous
e
ive
CaliStream Client SDK
CaliStream Events
Unified
S3 Data warehouse
Big Data:
TheValue

Data to Drive Decision-Making - CaliStream Meetup

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Data to Drive Decision-Making - CaliStream Meetup

Similar to Data to Drive Decision-Making - CaliStream Meetup (20)

Recently uploaded

Recently uploaded (20)

Data to Drive Decision-Making - CaliStream Meetup