SnappyData is a new open source project started by Pivotal GemFire founders to build a unified cluster capable of OLTP, OLAP, and streaming analytics using Spark. SnappyData fuses an elastic, highly available in-memory store for OLTP with Spark's memory manager and query engine to provide a single system for mixed workloads with fast ingestion, high concurrency and the ability to work with live, mutable data.

1. SnappyData
Building Continuous Applications Driven By Real Time Insights
Version 0.1 | © SnappyData Inc 2017
www.snappydata.io
Sudhir Menon , Founder, COO

2.
Who Are We?
2
• New Spark-­‐based open source project started by Pivotal GemFire
founders + engineers
• Decades of in-­‐memory data management experience
• Focus on real-­‐Ome, operaOonal analyOcs: Spark based OLTP+OLAP
database
Spinout
SnappyData
Funded by
Pivotal, GE, GTD
Capital

3. www.snappydata.io
• Applications that are intelligent, proactive, learn from past interactions, and are
context aware in their decision making
• Fast and reliable ingestion capabilities
• Support high memory density
• Utilize memory to reduce response time
• Support high concurrency
• Work on live data
• Support data mutability
What Is A Continuous Application?

4. www.snappydata.io
• Market Surveillance Systems (Trading exchanges, Market makers)
• Real Time Scoring Systems (Product recommendations, real time offers)
• Telco Analytics (Location based services, Predictive analytics)
• Sensor Analytics (Real time alerting for parking management, lighting etc.)
• Ad analytics + Ad placement systems
• Credit Card Fraud
• Detecting and Stopping Malware
Lets Discuss Some Use Cases

5. www.snappydata.io
• Elapsed time from event occurrence to event
analytics matters
• Latency in using information for learning matters
• Concurrency matters
• Recovery time matters
• User-kernel crossings matter
In short, liveness of data matters when it comes to
making decisions based on current information
Time Value of Information – Why does it matter

6.
Mixed Workloads Are Everywhere
6
Stream
Processing
Transac@on
(point lookups, small
updates)
Interac@ve
Analy@cs
Analytics on
mutating data
Correlating and
joining streams with
large histories
Maintaining state or
counters while
ingesting streams

7.
Mixed Workloads in Industrial IOT
7
IOT
Devices
Anomaly detecOon –
score against models
-­‐ Map sensors to tags
-­‐ Monitor temperature
-­‐ Send alerts
Correlate current
temperature trend with
history….
Interact using
dynamic queries….
Event Stream

8.
How Mixed Workloads Are Supported Today
8
Query
New
Data
Batch layer
Master
Datasheet
2
Serving layer
Batch view
3
Batch view
Speed layer
4
Real-­‐@me View Real-­‐@me View
1
Query
5

9.
Lambda Architecture is Complex
9
KAFKA
STORM
CASSANDRA
.....
SOURCE APPS
• Complexity: learn and master mulOple
products, data models, disparate APIs,
configs
• Slower
• Wasted resources

10. 10
Can
We
Simplify
&
Op0mize?

11.
11
How about a single clustered DB that can manage
stream state, transactional data & run OLAP queries?
Stream processing
Scalable writes, point reads, OLAP queries
Apps
Framework for Stream Processing, etc
RDB
MPP DB
HDFS
Tables
Txn

12. 12
© Snappydata INC 2017
Our
Solu@on
SnappyData
A Single Unified Cluster:
OLTP + OLAP + Streaming for real-time analytics

13.
Our Solution
13
Deep Scale,
High Volume
MPP DB
Real-­‐@me design
Low latency, HA,
concurrency
Batch design, high
throughput
Rapidly Maturing Matured over 13 years
Single Unified HA Cluster
OLTP + OLAP + Streaming for real-time analytics

14.
A Remarkable Marriage
14
Deep Scale,
High Volume
MPP DB
A lineage-­‐based
system designed for
high-­‐throughput
A (consensus-­‐driven)
replica@on-­‐based
system designed for
low-­‐latency
GEMFIRETwo drastically different
breeds of distributed
systems . . .

15.
15
Deep Fusion
w/ Spark Extreme
Speed
Synopsis
Data
Engine
Deep Fusion with Spark
Elas0c,
highly
available
in-­‐memory
store
for
OLTP
fused
with
Spark’s
memory
manager
and
the
Catalyst/Tungsten
engine.
The
store
itself
is
exposed
as
na0ve
Spark
data
frames.
Extreme Speed thru CPU code gen, vectoriza@on
Extend
Spark’s
Tungsten
engine
with
bePer
code
genera0on,
coloca0on
schemes,
..
Use
Sta0s0cal
techniques
to
reduce
data
by
100-­‐1000x
Answer
queries
in
frac0on
of
0me
and
resources
Synopses Data Engine
What is unique

16.
We transform Spark from this…
16
Deep Scale,
High Volume
MPP DB
USER 1 / APP 1
SPARK
MASTER
Spark Execu@on (Worker)
Framework for
streaming SQL,
ML…
Immutable
CACHE
USER 2 / APP 2
SPARK
MASTER
Spark Execu@on (Worker)
Framework for
streaming SQL,
ML…
Immutable
CACHE
HDFS
SQL
NoSQL
• Cannot update
• Repeated for each User/
APP
Boaleneck

17.
… Into “an always-on hybrid database !
17
Deep Scale,
High Volume
MPP DB
HDFS
SQL
NoSQL
HISTORY
Spark Execu@on (Worker) JVM
- Long running
Framework for
streaming SQL,
ML…
Spark
Driver
IN-­‐Memory
ROW + COLUMN
Start with
Indexing
Store
- Mutable,
- TransactionalSPARK
Cluster
JDBC
ODBC
Spark Job
Shared Nothing
Persistence

18.
Architecture
18
Cluster Manager
& Scheduler
Snappy Data Server (Spark Executor + Store)
Parser
OLAP
TXN
Synopsis Data Engine
Distributed Membership
Service
H
A
Stream Processing
Data Frame
RDD
Low
Latency
High
Latency
HYBRID Store
ProbabilisOc Rows Columns
Index
Query
OpOmizer
Add / Remove
Server
Tables ODBC/JDBC

19.
Unified API
19
• ML, graph, batch & streaming, SQL (selects)
Spark’s DataFrame API allows for:
• Mutability semanOcs (DML & transacOons)
• Indexing
• SQL-­‐based streaming
SnappyData adds full SQL support and extends DataFrame and DataSource APIs for:

20.
High-Level Accuracy Guarantees
20
1 0 1 1 0 0
2 1 2 0 0 1
2 0 0 0 1 1
0 1 0 2 0 2
Quality cer,fied
Approx Answers
Query Engine
HAC
Bias Es,mate
Variance Es,mate
STREAMS
Aging
SNAPPY STORE
Stra,fied Samples Stra,fied Samples
Interac,ve Query
Con,nuous Query
Pipelined
bootstrapped
operator
Row store Memory Column Store Disk

21.
Cloud Ready
21

22. Dealing with Credit Card Fraud
SnappyData Cluster
Credit Card
transacOon
stream
User History
PredicOon
Model
Streaming ApplicaOon
……….
Black
Listed
Cards
Data
Lake
No0fica0on
to
owner
No0fica0on
to
merchant

23. SnappyData Cluster
Customers
Approaching
Limit
Plan
Info
CDR Stream
Schedule callback
through call center
Streaming ApplicaOon Immediate SMS
to customer
Data
Lake
Preventing Bill Shock, Real Time Upgrades
The
system
detects
approaching
usage
limits,
no0fies
users
and
gives
them
a
chance
to
buy
a
one
0me
upgrade
or
a
new
plan,
increasing
loyalty
&
revenue

24. www.snappydata.io
Stream IngesOon
Reference
Data
• Stream analyOcs
• Insider detecOon
• Apply Rules
• Detect Market
ManipulaOon
Alert & NoOfy Downstream
Systems
Trigger InvesOgaOons
Spark
Streaming
SQL
Querying
Con0nuous
Queries
Par00oned
Stream
Inges0on
Summaries
&
Alerts
Messaging
Machine
Learning
Market Surveillance For Market Makers

25. Connected Car Real Time Data Flow
SnappyData Cluster
Ka]a
Receiver
Vehicle Time
Series Data
Vehicle
History
Driver
History
Streaming ApplicaOon
HDFS,
HBase
Raw
Data
Store
Custom
Summary
Dashboard
No0fica0on
to
owner
……….
System
KPIs
Asset
Metadata

26. Offline
Analysis
REAL TIME MATCHING ENGINE
MATCHING
ENGINE
Customer
History
NoOficaOon
Sub-­‐system
!
Historical Customer
Profiles
User by
Geo locaOon
PERSONALIZED
CAMPAIGNS TO
USERS
Ingest Stream
REAL
TIME
OFFERS
from
Merchants
Real Time Marketing Campaigns
A
stream
matching
engine
that
uses
customer
history,
their
current
loca0on
and
relevant
offers
to
Effec0vely
target
users
creates
differen0a0on
&
generates
revenue

27. www.snappydata.io
000’s data points/sec
Emergency Shutdown
Tuning & Optimization,
Monitor & Control
Continuous Real-time
Analysis
Maintenance
Billing
Sensor Analytics

28. Message
Bus
Stream IngesOon
Reference
Data
ETL
• OLAP
and
Low
Latency
Querying
in
SQL
• Machine
Learning
in
Spark
RFQs/Trades/Quotes streams
Analytic Dashboards
SnappyData
RFQ Analytics

29.
Ad Analytics
29
1.5-­‐2x faster ingestion, faster trx
7-­‐142× faster analytics (at 300M records)

30.
Data Synopsis Engine
30

31.
TPCH
31
Avg Latency
SnappyData
MemSQL
Spark
5.7s
100 GB
12.0s
66.9s

32. THANK YOU !
Try it out: hlp://snappydata.io/download