Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Warp 10
Mathias Herberts - @herberts
@warp10io – @CityzenData
Introduction
Our singular history
First focus on Smart Fabric Then all verticals with sensors
Architecture
Security
Privacy / Integrity / Authorization
Ingesting Data
Warp 10 manages Geo Time Series®
+
1411725015746208/48:-4.558/23000 water.temp{sensorid=42,sensornet=X} 16.9
TIMESTAMP [LOCATION] [ELEVATION] CLASS LABELS VA...
Pushing data is a HTTP request away!
POST /api/v0/update HTTP/1.1
Host: warp.cityzendata.net
Transfer-Encoding: chunked
Co...
Or pushed on a WebSocket
UPDATE MySecretToken
1442558287045000// linux.proc.vmstat.pgrefill_dma{cell=prod,dc=rbx4,hname=pr...
Performance is great!
Manipulating Data
A stack based language designed for working with Geo Time Series®
A feature rich language
600+functions
5high level frameworks
BUCKETIZE MAP REDUCE APPLY FILTER
Loops, Conditionals, Server...
Support for alternate scripting languages
JSR 223
No SDK, simply use HTTP and JSON libs
POST /dist/api/v0/exec HTTP/1.1
Host: warp.cityzendata.net
Content-Length: 358
conte...
Build complex systems
WEBCALL allows to issue HTTP calls from within WarpScript
UPDATE enables storing new data from withi...
Visualizing Data
Quantum, a WarpScript execution widget
Quantum-viz graphing widget
<!doctype html>
<html>
<head>
<meta name="viewport" content="width=device-width, minimum-scale...
Integration
WarpScript <3 Processing
800 'width' STORE 800 'height' STORE
$width $height '2D' PGraphics
400.0 'maxspeed' STORE 40000.0...
Quantum-image image display widget
<!doctype html>
<html>
<head>
<link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/bo...
Use any JavaScript library
Real-Time and Batch
Plasma and Mobius
Stream raw data (Plasma) or WarpScript results (Mobius)
Use WarpScript from other tools
Use Cases
Provide, replace or complement these
Time series databases
Data historians
IoT analytics platforms
OpenTSDB / KairosDB / I...
www.warp10.io – github.com/cityzendata
contact @ cityzendata . com
Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03
Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03
Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03
Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03
Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03
Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03
Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03
Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03
Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03
Upcoming SlideShare
Loading in …5
×

Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03

2,113 views

Published on

Warp 10 Platform presentation done at a Criteo Beer & Tech event on 2016-02-03

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03

  1. 1. Warp 10 Mathias Herberts - @herberts @warp10io – @CityzenData
  2. 2. Introduction
  3. 3. Our singular history First focus on Smart Fabric Then all verticals with sensors
  4. 4. Architecture
  5. 5. Security
  6. 6. Privacy / Integrity / Authorization
  7. 7. Ingesting Data
  8. 8. Warp 10 manages Geo Time Series® +
  9. 9. 1411725015746208/48:-4.558/23000 water.temp{sensorid=42,sensornet=X} 16.9 TIMESTAMP [LOCATION] [ELEVATION] CLASS LABELS VALUE Timestamp confgurable from ms to ns precision Location stored with 32 bits precision for latitude and longitude Elevation is a 64 bits LONG, assumed to be expressed in mm Class can be any valid UTF-8 sequence (URL % Encoded) – type of sensor Labels are immutable K/V pairs of UTF-8 sequences (URL % encoded) – id in class Attributes are mutable K/V pairs which can be associated with a Geo Time Series (GTS) Values can be LONG, DOUBLE, BOOLEAN or UTF-8 STRING
  10. 10. Pushing data is a HTTP request away! POST /api/v0/update HTTP/1.1 Host: warp.cityzendata.net Transfer-Encoding: chunked Content-Type: text/plain ← we also support application/gzip with a gzipped body X-Warp10-Token: MySecretToken X-Warp10-Now: 1442558287045000 Connection: close 1442558287045000// linux.proc.vmstat.pgmajfault{cell=prod,dc=rbx4,hname=prod-6,rack=55c14} 15429 1442558287045000// linux.proc.vmstat.pgrefill_dma{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0 1442558287045000// linux.proc.vmstat.pgrefill_dma32{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 992 1442558287045000// linux.proc.vmstat.pgrefill_normal{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 1321759 1442558287045000// linux.proc.vmstat.pgrefill_movable{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0 =1442558287046000// 0 =1442558287047000// 0 =1442558287048000// 0 // foo{label=value} 'String%20value' =T+1000// bar{label=value} T =T-1000// foobar{label=value} 1234.5
  11. 11. Or pushed on a WebSocket UPDATE MySecretToken 1442558287045000// linux.proc.vmstat.pgrefill_dma{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0 1442558287045000// linux.proc.vmstat.pgrefill_dma32{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 992 1442558287045000// linux.proc.vmstat.pgrefill_normal{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 1321759 1442558287045000// linux.proc.vmstat.pgrefill_movable{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0 =1442558287046000// 0 =1442558287047000// 0 =1442558287048000// 0 // foo{label=value} 'String%20value' =T+1000// bar{label=value} T =T-1000// foobar{label=value} 1234.5
  12. 12. Performance is great!
  13. 13. Manipulating Data
  14. 14. A stack based language designed for working with Geo Time Series®
  15. 15. A feature rich language 600+functions 5high level frameworks BUCKETIZE MAP REDUCE APPLY FILTER Loops, Conditionals, Server Side Macros, Java UDFs
  16. 16. Support for alternate scripting languages JSR 223
  17. 17. No SDK, simply use HTTP and JSON libs POST /dist/api/v0/exec HTTP/1.1 Host: warp.cityzendata.net Content-Length: 358 content-type: application/x-www-form-urlencoded; charset=UTF-8 Referer: https://api0.cityzendata.net/widgets/static/ 'lMYI.....Kpnl9Lk' 'token' STORE NOW 'now' STORE 'PT2h' DURATION 'duration' STORE // Fetch raw data [ $token '~.*ingress.*requests' { 'cell' 'prod' } $now $duration ] FETCH // Compute QPS [ SWAP mapper.rate 1 0 0 ] MAP 'Hello' NOW
  18. 18. Build complex systems WEBCALL allows to issue HTTP calls from within WarpScript UPDATE enables storing new data from within WarpScript META can set attributes for specifc Geo Time Series® Runner can schedule WarpScript programs periodically Right within Warp 10
  19. 19. Visualizing Data
  20. 20. Quantum, a WarpScript execution widget
  21. 21. Quantum-viz graphing widget <!doctype html> <html> <head> <meta name="viewport" content="width=device-width, minimum-scale=1.0, initial-scale=1.0, user-scalable=yes"> <script src="https://api0.cityzendata.net/widgets/quantumviz/webcomponentsjs/webcomponents.js"></script> <link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/polymer/polymer.html"> <link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/warp10-quantumviz/warp10-quantumviz.html"> <body> <warp10-quantumviz width="500" height="400" show-axis="true" tooltip="true" line-width="2" reload="0" host="https://warp.cityzendata.net/dist" > NEWGTS 1 720 <% DUP 'i' STORE 10000000 * NaN NaN NaN $i TORADIANS COS ADDVALUE %> FOR [ SWAP ] 'gts' STORE [ { 'color' '#00d4ff' 'key' 'Sine' } ] 'params' STORE { 'interpolate' 'linear' } 'globalParams' STORE { 'gts' $gts 'params' $params 'globalParams' $globalParams } </warp10-quantumviz> </body> </html>
  22. 22. Integration
  23. 23. WarpScript <3 Processing 800 'width' STORE 800 'height' STORE $width $height '2D' PGraphics 400.0 'maxspeed' STORE 40000.0 'maxalt' STORE 3.0 2.0 2.0 @orbit/heatmap/kernel/triangular 'kernel' STORE @orbit/heatmap/palette/classic 'palette' STORE 'MULTIPLY' PblendMode 'CENTER' PimageMode 'TOKEN''token' STORE [ $token '~(ALT|CAS)' {} NOW -2000000 ] FETCH DUP 0 GET LASTTICK 'now' STORE [ SWAP bucketizer.last $now STU 0 ] BUCKETIZE // Create heatmap <% 7 GET LIST-> DROP 'CAS' STORE 'ALT' STORE <% $CAS ISNULL NOT $ALT ISNULL NOT && %> <% $kernel $CAS $maxspeed / $width * $ALT $maxalt / 1.0 SWAP - $height * Pimage %> IFT 0 NaN NaN NaN NULL %> MACROREDUCER 'GRAPHER' STORE [ SWAP [] $GRAPHER ] REDUCE DROP // Colorize Ppixels <% DROP Palpha $palette SWAP GET %> LMAP PupdatePixels Pencode Pdecode $width $height '2D' PGraphics // Do the grid PnoFill 0 0 $width 1 - $height 1 - Prect 2.0 PstrokeWeight 200.0 Pcolor Pstroke 250.0 $maxspeed / $width * DUP 0 SWAP $height Pline 0 10000 $maxalt / 1.0 SWAP - $height * DUP $width SWAP Pline SWAP 0 0 Pimage Pencode
  24. 24. Quantum-image image display widget <!doctype html> <html> <head> <link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css"> <link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css"> <script src="//cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script> <script src="//cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script> <script src="https://api0.cityzendata.net/widgets/quantumviz/webcomponentsjs/webcomponents.js"></script> <link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/polymer/polymer.html"> <link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/warp10-quantumviz/quantumviz-warpscript-image.html"> <body> <warp10-img width="300" height="300" reload="0" host="https://warp.cityzendata.net/dist"> 200 'width' CSTORE 200 'height' CSTORE $width $height '2D' Pgraphics Ppixels <% DROP DROP RAND 0xFFFFFFFF * TOLONG %> LMAP PupdatePixels Pencode </warp10-img> </body> </html>
  25. 25. Use any JavaScript library
  26. 26. Real-Time and Batch
  27. 27. Plasma and Mobius Stream raw data (Plasma) or WarpScript results (Mobius)
  28. 28. Use WarpScript from other tools
  29. 29. Use Cases
  30. 30. Provide, replace or complement these Time series databases Data historians IoT analytics platforms OpenTSDB / KairosDB / InfluxDB / Prometheus OSISoft Pi / Proprietary solutions / Sensor data processing solution Monitoring / alerting platform Time series manipulation library
  31. 31. www.warp10.io – github.com/cityzendata
  32. 32. contact @ cityzendata . com

×