This document summarizes the key capabilities of Warp 10, a time series data ingestion, processing, and visualization platform: 1. Warp 10 can ingest high volumes of time series data from sensors and other IoT devices via HTTP, WebSockets, and many collection tools in a performant manner. 2. It provides a feature-rich scripting language called WarpScript that allows users to manipulate, analyze, and transform ingested time series data using over 690 functions and frameworks. 3. Warp 10 includes tools to visualize time series data in real-time through widgets that can display charts, images, and more generated from WarpScript. Dynamic tile widgets also enable building configurable

1. This is not
your father's
Monitoring.
@herberts
@warp10io @CityzenData

5. 1994
PERL – SNMP – Limited to … 10 routers

14. 2010

20. Rev 1840 | Blame | Compare with Previous | Last modification | View Log | Download
d .nls
loaded: 0b;
nls: `nls 2:(`q_nls;6);
defaultOpts: (`maxfev;`nprint) ! (2000; 0);
loaded:1b;
d .
nlsfit:{[df;model;options]
df: () xkey df;
options: .nls.defaultOpts,options;
options: key[options] ! `float$ value[options];
model[`xvars]: (),model`xvars;
df: @[df; model[`xvars],model`yvar; {`float$x}];
Y: (0N;1) # ?[df;();();model`yvar];
fn: raze string model`function;
X: flip value ?[df;();(); model[`xvars]!model`xvars];
Th: (0N;1) # model`initial;
W: (count[Y];1) # `float$1.0;
if[`weights in key model; W:(0N;1) # `float$ ?[df;();();model`weights]];
call: .[ .nls.nls ; (Y;fn;X;Th;W;options); {'x}];
call: `info`coef`resid`dof!call;
:call;
};

25. Warp 10

26. Introduction

27. Our singular history
First focus on Smart Fabric Then all verticals with sensors
Including monitoring

28. 3 pillars

29. Warp 10 manages Geo Time Series®
+

30. Privacy / Integrity / Authorization

31. A stack based language designed for working with Geo Time Series®

33. Ingesting Data

34. 1411725015746208/48:-4.558/23000 water.temp{sensorid=42,sensornet=X} 16.9
TIMESTAMP [LOCATION] [ELEVATION] CLASS LABELS VALUE
Timestamp confgurable from ms to ns precision
Location stored with 32 bits precision for latitude and longitude
Elevation is a 64 bits LONG, assumed to be expressed in mm
Class can be any valid UTF-8 sequence (URL % Encoded) – type of sensor
Labels are immutable K/V pairs of UTF-8 sequences (URL % encoded) – id in class
Attributes are mutable K/V pairs which can be associated with a Geo Time Series (GTS)
Values can be LONG, DOUBLE, BOOLEAN or UTF-8 STRING

35. Pushing data is a HTTP request away!
POST /api/v0/update HTTP/1.1
Host: warp.cityzendata.net
Transfer-Encoding: chunked
Content-Type: text/plain ← we also support application/gzip with a gzipped body
X-Warp10-Token: MySecretToken
X-Warp10-Now: 1442558287045000
Connection: close
1442558287045000// linux.proc.vmstat.pgmajfault{cell=prod,dc=rbx4,hname=prod-6,rack=55c14} 15429
1442558287045000// linux.proc.vmstat.pgrefill_dma{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0
1442558287045000// linux.proc.vmstat.pgrefill_dma32{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 992
1442558287045000// linux.proc.vmstat.pgrefill_normal{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 1321759
1442558287045000// linux.proc.vmstat.pgrefill_movable{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0
=1442558287046000// 0
=1442558287047000// 0
=1442558287048000// 0
// foo{label=value} 'String%20value'
=T+1000// bar{label=value} T
=T-1000// foobar{label=value} 1234.5

36. Or pushed on a WebSocket
UPDATE MySecretToken
1442558287045000// linux.proc.vmstat.pgrefill_dma{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0
1442558287045000// linux.proc.vmstat.pgrefill_dma32{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 992
1442558287045000// linux.proc.vmstat.pgrefill_normal{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 1321759
1442558287045000// linux.proc.vmstat.pgrefill_movable{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0
=1442558287046000// 0
=1442558287047000// 0
=1442558287048000// 0
// foo{label=value} 'String%20value'
=T+1000// bar{label=value} T
=T-1000// foobar{label=value} 1234.5

37. Performance is great!

38. Collect via many available tools

39. Not all time series are metrics!

40. Manipulating Data

41. A feature rich language
690+functions
5high level frameworks
BUCKETIZE MAP REDUCE APPLY FILTER
Loops, Conditionals, Server Side Macros, Java UDFs

42. Support for alternate scripting languages
JSR 223

43. No SDK, simply use HTTP and JSON libs
POST /dist/api/v0/exec HTTP/1.1
Host: warp.cityzendata.net
Content-Length: 358
content-type: application/x-www-form-urlencoded; charset=UTF-8
Referer: https://api0.cityzendata.net/widgets/static/
'lMYI.....Kpnl9Lk' 'token' STORE
NOW 'now' STORE
'PT2h' DURATION 'duration' STORE
// Fetch raw data
[ $token '~.*ingress.*requests' { 'cell' 'prod' } $now $duration ] FETCH
// Compute QPS
[ SWAP mapper.rate 1 0 0 ] MAP
'Hello'
NOW

44. Build complex systems
WEBCALL allows to issue HTTP calls from within WarpScript
UPDATE enables storing new data from within WarpScript
META can set attributes for specifc Geo Time Series®
Runner can schedule WarpScript programs periodically
Right within Warp 10
UDFs to augment WarpScript with custom functions

45. WarpScript is independent of storage
Any time series database is just one UDF away

46. WarpScript works in Zeppelin

47. Visualizing Data

48. Quantum, a WarpScript execution widget

49. Quantum-viz graphing widget
<!doctype html>
<html>
<head>
<meta name="viewport" content="width=device-width, minimum-scale=1.0, initial-scale=1.0, user-scalable=yes">
<script src="https://api0.cityzendata.net/widgets/quantumviz/webcomponentsjs/webcomponents.js"></script>
<link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/polymer/polymer.html">
<link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/warp10-quantumviz/warp10-quantumviz.html">
<body>
<warp10-quantumviz
width="500" height="400"
show-axis="true" tooltip="true" line-width="2" reload="0"
host="https://warp.cityzendata.net/dist" >
NEWGTS
1 720
<% DUP 'i' STORE 10000000 * NaN NaN NaN $i TORADIANS COS ADDVALUE %>
FOR
[ SWAP ] 'gts' STORE
[ { 'color' '#00d4ff' 'key' 'Sine' } ] 'params' STORE
{ 'interpolate' 'linear' } 'globalParams' STORE
{ 'gts' $gts 'params' $params 'globalParams' $globalParams }
</warp10-quantumviz>
</body>
</html>

50. Dynamic tile widgets
<!doctype html>
<html>
...
<body>
<warp10-dashboard-tile-container>
<warp10-dashboard-element reload="120" tile="normal" warpscript-base-url=”...”>
@warpscript-macro1
</warp10-dashboard-element>
<warp10-dashboard-element reload="120" tile="normal" warpscript-base-url=”...”>
@warpscript-macro2
</warp10-dashboard-element>
<warp10-dashboard-element reload="120" tile="normal" warpscript-base-url=”...”>
@warpscript-macro3
</warp10-dashboard-element>
</body>
</html>
WarpScript code returns JSON content which configures the appearance
{
'type' 'chart'
'params' {
'label' $label
'tile' $tile
'fgColor' $fgColor
'hexFgColor' $hexFgColor
'bgColor' $bgColor
'data' [ $currentGTS ]
}
}
{
'type' 'gauge'
'params' {
'label'
'Alert: Number of datapoints added too high in last minute'
'tile' 'large'
'fgColor' $fgColor 'bgColor' $bgColor
'value' $value
'max' 4000 'min' 0
}
}

52. WarpScript <3 Processing
800 'width' STORE 800 'height' STORE
$width $height '2D' PGraphics
400.0 'maxspeed' STORE 40000.0 'maxalt' STORE
3.0 2.0 2.0 @orbit/heatmap/kernel/triangular 'kernel' STORE
@orbit/heatmap/palette/classic 'palette' STORE
'MULTIPLY' PblendMode 'CENTER' PimageMode
'TOKEN''token' STORE
[ $token '~(ALT|CAS)' {} NOW -2000000 ] FETCH
DUP 0 GET LASTTICK 'now' STORE
[ SWAP bucketizer.last $now STU 0 ] BUCKETIZE
// Create heatmap
<%
7 GET
LIST-> DROP 'CAS' STORE 'ALT' STORE
<% $CAS ISNULL NOT $ALT ISNULL NOT && %>
<% $kernel $CAS $maxspeed / $width * $ALT $maxalt / 1.0 SWAP -
$height * Pimage %>
IFT
0 NaN NaN NaN NULL
%> MACROREDUCER 'GRAPHER' STORE
[ SWAP [] $GRAPHER ] REDUCE DROP
// Colorize
Ppixels
<% DROP Palpha $palette SWAP GET %> LMAP
PupdatePixels
Pencode Pdecode
$width $height '2D' PGraphics
// Do the grid
PnoFill 0 0 $width 1 - $height 1 - Prect
2.0 PstrokeWeight 200.0 Pcolor Pstroke
250.0 $maxspeed / $width * DUP 0 SWAP $height Pline
0 10000 $maxalt / 1.0 SWAP - $height * DUP $width SWAP Pline
SWAP 0 0 Pimage
Pencode

54. Quantum-image image display widget
<!doctype html>
<html>
<head>
<link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
<link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css">
<script src="//cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
<script src="//cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script>
<script src="https://api0.cityzendata.net/widgets/quantumviz/webcomponentsjs/webcomponents.js"></script>
<link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/polymer/polymer.html">
<link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/warp10-quantumviz/quantumviz-warpscript-image.html">
<body>
<warp10-img
width="300" height="300" reload="0"
host="https://warp.cityzendata.net/dist">
200 'width' CSTORE 200 'height' CSTORE
$width $height '2D' Pgraphics
Ppixels
<% DROP DROP RAND 0xFFFFFFFF * TOLONG %> LMAP
PupdatePixels Pencode
</warp10-img>
</body>
</html>

55. Use any JavaScript library

56. Real-Time and Batch

57. Plasma and Mobius
Stream raw data (Plasma) or WarpScript results (Mobius)

58. Use WarpScript from other tools

59. www.warp10.io – github.com/cityzendata

60. contact @ cityzendata . com