Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03

Warp 10
Mathias Herberts - @herberts
@warp10io – @CityzenData

Our singular history
First focus on Smart Fabric Then all verticals with sensors

Privacy / Integrity / Authorization

Warp 10 manages Geo Time Series®
+

1411725015746208/48:-4.558/23000 water.temp{sensorid=42,sensornet=X} 16.9
TIMESTAMP [LOCATION] [ELEVATION] CLASS LABELS VALUE
Timestamp confgurable from ms to ns precision
Location stored with 32 bits precision for latitude and longitude
Elevation is a 64 bits LONG, assumed to be expressed in mm
Class can be any valid UTF-8 sequence (URL % Encoded) – type of sensor
Labels are immutable K/V pairs of UTF-8 sequences (URL % encoded) – id in class
Attributes are mutable K/V pairs which can be associated with a Geo Time Series (GTS)
Values can be LONG, DOUBLE, BOOLEAN or UTF-8 STRING

Pushing data is a HTTP request away!
POST /api/v0/update HTTP/1.1
Host: warp.cityzendata.net
Transfer-Encoding: chunked
Content-Type: text/plain ← we also support application/gzip with a gzipped body
X-Warp10-Token: MySecretToken
X-Warp10-Now: 1442558287045000
Connection: close
1442558287045000// linux.proc.vmstat.pgmajfault{cell=prod,dc=rbx4,hname=prod-6,rack=55c14} 15429
1442558287045000// linux.proc.vmstat.pgrefill_dma{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0
1442558287045000// linux.proc.vmstat.pgrefill_dma32{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 992
1442558287045000// linux.proc.vmstat.pgrefill_normal{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 1321759
1442558287045000// linux.proc.vmstat.pgrefill_movable{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0
=1442558287046000// 0
=1442558287047000// 0
=1442558287048000// 0
// foo{label=value} 'String%20value'
=T+1000// bar{label=value} T
=T-1000// foobar{label=value} 1234.5

Or pushed on a WebSocket
UPDATE MySecretToken
1442558287045000// linux.proc.vmstat.pgrefill_dma{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0
1442558287045000// linux.proc.vmstat.pgrefill_dma32{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 992
1442558287045000// linux.proc.vmstat.pgrefill_normal{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 1321759
1442558287045000// linux.proc.vmstat.pgrefill_movable{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0
=1442558287046000// 0
=1442558287047000// 0
=1442558287048000// 0
// foo{label=value} 'String%20value'
=T+1000// bar{label=value} T
=T-1000// foobar{label=value} 1234.5

A stack based language designed for working with Geo Time Series®

A feature rich language
600+functions
5high level frameworks
BUCKETIZE MAP REDUCE APPLY FILTER
Loops, Conditionals, Server Side Macros, Java UDFs

Support for alternate scripting languages
JSR 223

No SDK, simply use HTTP and JSON libs
POST /dist/api/v0/exec HTTP/1.1
Host: warp.cityzendata.net
Content-Length: 358
content-type: application/x-www-form-urlencoded; charset=UTF-8
Referer: https://api0.cityzendata.net/widgets/static/
'lMYI.....Kpnl9Lk' 'token' STORE
NOW 'now' STORE
'PT2h' DURATION 'duration' STORE
// Fetch raw data
[ $token '~.*ingress.*requests' { 'cell' 'prod' } $now $duration ] FETCH
// Compute QPS
[ SWAP mapper.rate 1 0 0 ] MAP
'Hello'
NOW

Build complex systems
WEBCALL allows to issue HTTP calls from within WarpScript
UPDATE enables storing new data from within WarpScript
META can set attributes for specifc Geo Time Series®
Runner can schedule WarpScript programs periodically
Right within Warp 10

Quantum, a WarpScript execution widget

Quantum-viz graphing widget
<!doctype html>
<html>
<head>
<meta name="viewport" content="width=device-width, minimum-scale=1.0, initial-scale=1.0, user-scalable=yes">
<script src="https://api0.cityzendata.net/widgets/quantumviz/webcomponentsjs/webcomponents.js"></script>
<link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/polymer/polymer.html">
<link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/warp10-quantumviz/warp10-quantumviz.html">
<body>
<warp10-quantumviz
width="500" height="400"
show-axis="true" tooltip="true" line-width="2" reload="0"
host="https://warp.cityzendata.net/dist" >
NEWGTS
1 720
<% DUP 'i' STORE 10000000 * NaN NaN NaN $i TORADIANS COS ADDVALUE %>
FOR
[ SWAP ] 'gts' STORE
[ { 'color' '#00d4ff' 'key' 'Sine' } ] 'params' STORE
{ 'interpolate' 'linear' } 'globalParams' STORE
{ 'gts' $gts 'params' $params 'globalParams' $globalParams }
</warp10-quantumviz>
</body>
</html>

WarpScript <3 Processing
800 'width' STORE 800 'height' STORE
$width $height '2D' PGraphics
400.0 'maxspeed' STORE 40000.0 'maxalt' STORE
3.0 2.0 2.0 @orbit/heatmap/kernel/triangular 'kernel' STORE
@orbit/heatmap/palette/classic 'palette' STORE
'MULTIPLY' PblendMode 'CENTER' PimageMode
'TOKEN''token' STORE
[ $token '~(ALT|CAS)' {} NOW -2000000 ] FETCH
DUP 0 GET LASTTICK 'now' STORE
[ SWAP bucketizer.last $now STU 0 ] BUCKETIZE
// Create heatmap
<%
7 GET
LIST-> DROP 'CAS' STORE 'ALT' STORE
<% $CAS ISNULL NOT $ALT ISNULL NOT && %>
<% $kernel $CAS $maxspeed / $width * $ALT $maxalt / 1.0 SWAP -
$height * Pimage %>
IFT
0 NaN NaN NaN NULL
%> MACROREDUCER 'GRAPHER' STORE
[ SWAP [] $GRAPHER ] REDUCE DROP
// Colorize
Ppixels
<% DROP Palpha $palette SWAP GET %> LMAP
PupdatePixels
Pencode Pdecode
$width $height '2D' PGraphics
// Do the grid
PnoFill 0 0 $width 1 - $height 1 - Prect
2.0 PstrokeWeight 200.0 Pcolor Pstroke
250.0 $maxspeed / $width * DUP 0 SWAP $height Pline
0 10000 $maxalt / 1.0 SWAP - $height * DUP $width SWAP Pline
SWAP 0 0 Pimage
Pencode

Quantum-image image display widget
<!doctype html>
<html>
<head>
<link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
<link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css">
<script src="//cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
<script src="//cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
<script src="https://api0.cityzendata.net/widgets/quantumviz/webcomponentsjs/webcomponents.js"></script>
<link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/polymer/polymer.html">
<link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/warp10-quantumviz/quantumviz-warpscript-image.html">
<body>
<warp10-img
width="300" height="300" reload="0"
host="https://warp.cityzendata.net/dist">
200 'width' CSTORE 200 'height' CSTORE
$width $height '2D' Pgraphics
Ppixels
<% DROP DROP RAND 0xFFFFFFFF * TOLONG %> LMAP
PupdatePixels Pencode
</warp10-img>
</body>
</html>

Plasma and Mobius
Stream raw data (Plasma) or WarpScript results (Mobius)

Use WarpScript from other tools

Provide, replace or complement these
Time series databases
Data historians
IoT analytics platforms
OpenTSDB / KairosDB / InfluxDB / Prometheus
OSISoft Pi / Proprietary solutions /
Sensor data processing solution
Monitoring / alerting platform
Time series manipulation library

www.warp10.io – github.com/cityzendata

Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03

Similar to Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03 (20)

More from Mathias Herberts

More from Mathias Herberts (8)

Recently uploaded

Recently uploaded (20)

Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03