Warp10: collect, store and manipulate sensor data - BreizhCamp - 2016 03-24

@FinistSeb @LostInBrittany#BzhCmp #Warp10
Warp 10:
Collect, store and manipulate sensor data
Horacio GonzalezSébastien Lambour

Horacio Gonzalez
@LostInBrittany
Cityzen Data
Spaniard lost in Brittany,
developer, dreamer and all-
around geek

Sébastien Lambour
@FinistSeb
Cityzen Data
Runner, 2 Kids,
Geek, Handyman,
Polyglot JVM Developer

Warp 10

Introduction
Geo-Time SeriesTM
Image: Spacetime distorsions

Time Series
Image: Mike Bostock

Time series storage and analysis
Image: Hamza Fessi and ABC Bourse
Not suited for your vanilla SQL RDBMS
One simple example: moving averages...

Geo-Time SeriesTM
Image: AIS Vessel Tracking

Geo-Time Series

Geo-Time Series and the IoT
Image: LinkedIn

IoT means talking thing
How fast are they talking?

Internet of very introverted Things
Long range transmissions

Internet of introverted Things
Personal Area Network

Internet of shy Things
Local Area Network
Cellular Networks

Lots of shy thing generate a huge lot of data
Image: Universal Studios

Internet of chatty Things
10 000 Hz
670 000 sensors
20 000 metrics
per second

Internet of garrulous Things
Image: Google
Millions of
metrics per second

Warp 10 : A software platform for IoT
Warp 10 is a software platform that
● Ingests and stores data
● Manipulates and analyzes data
● Is dedicated to data from sensors, meters, IoT and any real or
virtual probe

Warp 10 General Synoptic
Stockage
Architecture
Language,
Functions,
Algorithms
Application
access
VizualisationReal
Time

#collect
How do you get these metrics?
Image: Games Radar

Using our own Sensision agent

Using our own Sensision agent
With queue forwarder

Using plugins for other collecting systems

Or simply pushing data directly

Choosing an input format

XML? JSON?
139 bytes 108 bytes

Warp 10 GTS Input Format
57 bytes
But size isn't the most important reason
parsing time is way more important
XML or even JSON parsing is slow and costly
Warp 10 GTS input format isn't

timestamp (us by default)
latitude:longitude (WGS84)
elevation (millimeters)
classname*
labels (key=value)
value* (long, double, boolean or string)
* mandatory fields
Warp 10 GTS Input Format

#store
From tiny to huge
Image: Games Radar

Warp 10 on Raspberry Pi B+
1 000 datapoints per second

Warp 10 on Raspberry Pi 2 B

Warp 10 on a modern server

Warp 10 on a cluster
3 millions of datapoints per second
(our current record on input traffic)

#analyse
From tiny to huge
Image: Amazon

Many time-series solutions
TSAR

But they are only stores...
Fetching data is only the tip of the iceberg

Analysing the data
High level analysis must be done elsewhere

Algorithms are resource hungry

Your computer is not a datacenter

Manipulating GTS
To be scalable, analysis must be done in
Warp 10 platform, not in user's computer

A true GTS analysis toolbox
○ Hundreds of functions
○ Manipulation frameworks
○ Analysis workflow
Manipulating GTS

GTS manipulation
Why not a simple REST API?
● One endpoint by function?
● How to chain a workflow analysis?
REST API not suitable for
complex manipulations

GTS manipulation
Why not a SQL dialect?
● How do you do a simple moving average in SQL?
● How do you geo-time fencing in SQL?
SQL is not adapted to (G)TS analysis!

GTS manipulation language
Our solution: a GTS manipulation language
WarpScript

A stack based language

WarpScript
Non-compiled Optimized functions,
fast execution

Basic operations

Five frameworks
●
●
●
●
●

More than 500 functions

Time series functions

Geo-Time Series functions
Geo mapping (WKT)
Horizontal & vertical speed
Horizontal & vertical distance
Haversine
...

Quantum IDE

Enough teasing...

Fuel prices data
16 297 448 metrics
11 379 fuel stations
42 885 Geo Time Series

Basic analysis
Average diesel fuel prices in France
since 2007
Image: LEGO Ideas

First Fetch Data (SQL vs WarpScript )

FETCH gives us a GTS list

Timestamp
(microseconds since epoch)

Location
(latitude, longitude)

Value

Calculate the average
Using Groovy:

Calculate the average with WarpScript
1- Calculate the mean price by station

BUCKETIZE framework
Put the data of a GTS into regularly spaced buckets

2- Reduce to get the global average

REDUCE framework
Apply a function on a set of GTS tick by tick

Too verbose? Write it differently

Even more concise

Basic analysis
Mean of the last available
diesel fuel prices in France
Image: LEGO Ideas

Fetching Data (SQL vs WarpScript )

Mean of those last prices
align ticks with BUCKETIZE framework
compute the average with REDUCE

Geo-time analysis
Find the cheapest fuel station near here
48.115434, -1.636877

WKT: Well-known text geometry

…
WKT in WarpScript

Geo-filtering points of GTS

Geo-filtering points of GTS
MAPPER framework
Apply a function on values of a GTS
that fall into a sliding window

The stations near my position

There can only be one

And this is only the surface
Possibilities are endless

Think differently
Geo-Time Series are everywhere

Warp 10 platform and tools

Everything is on GitHub
https://github.com/cityzendata/

Thank you !

Warp10: collect, store and manipulate sensor data - BreizhCamp - 2016 03-24

More Related Content

Similar to Warp10: collect, store and manipulate sensor data - BreizhCamp - 2016 03-24

More from Horacio Gonzalez

Recently uploaded

Warp10: collect, store and manipulate sensor data - BreizhCamp - 2016 03-24