Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Time Series Data
Tick Stack
Hello!
I am Gianluca Arbezzano
Software Engineer at CurrencyFair
@gianarb on Twitter and GitHub
2
3
Open Source Maintainer and Contributors
penny.gianarb.it vim-php.org
4
scaledocker.com
The book that drives you in production with
docker
“
“Try again. Fail again.
Fail better.”
cit. Samuel Beckett
5
Trust your system
To be familiar with your applications
you need to know what they are doing.
6
1
Widespread
Monitoring Tools
tail -f /var/log/app.log
7
8
9
2016/04/15 15:42:46 [warn] 2330#0: *167 using uninitialized
variable,
client: 10.0.1.1, server: localhost.dev, request: ...
2
Expensive to store
10
2
Difficult to index
11
Difficult not impossible
They do an amazing work. And there are other tools!!
12
They are awesome for some use cases
▪ Extract informations
▪ They can be “human readable”
▪ … and others, and others
13
Keep your life amazing
Reduce your time series
to a timestamp
and a value (int or float)
14
This is cheap and useful
We are here to speak about
Time Series
[
{
"name": "log_lines",
"columns": ["time", "line"],
"point": [1400425947368, "her...
EASY! EASY! EASY!
{
"name": "cpu_percent_use",
"columns": ["value"],
"point": 40
}
16
“
“Any intelligent fool can make things
bigger, more complex, and more violent.
It takes a touch of genius — and a lot of
...
Time is a perfect sharding key
It means that Time Series scale really well
18
We have a set of tools to use
19
20
InfluxDB
● Optimized to store time series data
● Open Source and easy to install (go
binary)
● Big community and ecosys...
21
Easy
Install and start a go binary
wget https://dl.influxdata.com/influxdb/releases/influxdb_1.0.0_amd64.deb
sudo dpkg ...
22
Easy
HTTP API on port 8086
Support for UDP connections
Admin Panel on port 8083
23
Easy
SELECT value
FROM cpu_load_short
WHERE region='us-west'
24
T-Shirts time!
25
26
27
[key] [fields] [timestamp]
temperature,machine=unit internal=3,external=10 1434055562000000035
Inline Protocol thinked ...
28
CorleyBenchmarksInfluxDBAdapterEvent
Method Name Iterations Average Time Ops/second
------------------------ ----------...
29
CREATE CONTINUOUS QUERY minnie ON world
BEGIN SELECT min(mouse) INTO min_mouse
FROM zoo GROUP BY time(30m) END
Continuo...
Telegraf
https://github.com/influxdata/telegraf
Collector to grab and send
data from different sources to
InfluxDB and oth...
31
Telegraf Plugins
Kapacitor
https://github.com/influxdata/kapacitor
Trigger notifications and make
action in case of specific
behaviors
32
f...
33
Kapacitor high CPU alert
stream
|from()
.measurement('cpu_usage_idle')
.groupBy('host')
|window()
.period(1m)
.every(1m...
34
Demo
https://github.com/gianarb/tick-php
35
When you start to work with "micro"services
understand the topology of your connections is
really important
Time series...
36
Use a dashboard to put together
different metrics and create not
obvious relations
37
Why InfluxDB and not something else?
https://www.influxdata.com/influxdb-is-27x-faster-vs-mongodb-for-time-series-workl...
38
That’s it!
A series of great tools to monitor your
applications and your infrastructure
39
A monitoring system isn’t for all
Thanks!
See you around
twitter.com/gianarb
40
Upcoming SlideShare
Loading in …5
×

Time Series Database and Tick Stack

1,867 views

Published on

Design a monitoring system with InfluxDB and the Tick stack.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Time Series Database and Tick Stack

  1. 1. Time Series Data Tick Stack
  2. 2. Hello! I am Gianluca Arbezzano Software Engineer at CurrencyFair @gianarb on Twitter and GitHub 2
  3. 3. 3 Open Source Maintainer and Contributors penny.gianarb.it vim-php.org
  4. 4. 4 scaledocker.com The book that drives you in production with docker
  5. 5. “ “Try again. Fail again. Fail better.” cit. Samuel Beckett 5
  6. 6. Trust your system To be familiar with your applications you need to know what they are doing. 6
  7. 7. 1 Widespread Monitoring Tools tail -f /var/log/app.log 7
  8. 8. 8
  9. 9. 9 2016/04/15 15:42:46 [warn] 2330#0: *167 using uninitialized variable, client: 10.0.1.1, server: localhost.dev, request: "POST /auth HTTP/1.1", host: "localhost" 2016/04/15 15:44:44 [error] 2330#0: *171 FastCGI sent in stderr: " PHP message: PHP Fatal error: Uncaught exception 'RuntimeException' with message 'All broken)[500]' in /var/www/my/project.php:237 Stack trace: #0 /var/www/index.php:45 ObjectService->flush() #1 [internal function] ->save()
  10. 10. 2 Expensive to store 10
  11. 11. 2 Difficult to index 11
  12. 12. Difficult not impossible They do an amazing work. And there are other tools!! 12
  13. 13. They are awesome for some use cases ▪ Extract informations ▪ They can be “human readable” ▪ … and others, and others 13
  14. 14. Keep your life amazing Reduce your time series to a timestamp and a value (int or float) 14 This is cheap and useful
  15. 15. We are here to speak about Time Series [ { "name": "log_lines", "columns": ["time", "line"], "point": [1400425947368, "here's some useful log info"] } ] 15
  16. 16. EASY! EASY! EASY! { "name": "cpu_percent_use", "columns": ["value"], "point": 40 } 16
  17. 17. “ “Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius — and a lot of courage to move in the opposite direction.” Cit. Ernest F. Shumacher 17
  18. 18. Time is a perfect sharding key It means that Time Series scale really well 18
  19. 19. We have a set of tools to use 19
  20. 20. 20 InfluxDB ● Optimized to store time series data ● Open Source and easy to install (go binary) ● Big community and ecosystem to manage alert and collect metrics
  21. 21. 21 Easy Install and start a go binary wget https://dl.influxdata.com/influxdb/releases/influxdb_1.0.0_amd64.deb sudo dpkg -i influxdb_1.0.0_amd64.deb Influxd -config /usr/local/etc/influxdb.conf
  22. 22. 22 Easy HTTP API on port 8086 Support for UDP connections Admin Panel on port 8083
  23. 23. 23 Easy SELECT value FROM cpu_load_short WHERE region='us-west'
  24. 24. 24 T-Shirts time!
  25. 25. 25
  26. 26. 26
  27. 27. 27 [key] [fields] [timestamp] temperature,machine=unit internal=3,external=10 1434055562000000035 Inline Protocol thinked to be smart and slim
  28. 28. 28 CorleyBenchmarksInfluxDBAdapterEvent Method Name Iterations Average Time Ops/second ------------------------ ------------ -------------- ------------- sendDataUsingHttpAdapter: [1,000 ] [0.0026700308323] [374.52751] sendDataUsingUdpAdapter : [1,000 ] [0.0000436344147] [22,917.69026] UDP vs TCP protocol
  29. 29. 29 CREATE CONTINUOUS QUERY minnie ON world BEGIN SELECT min(mouse) INTO min_mouse FROM zoo GROUP BY time(30m) END Continuous Query
  30. 30. Telegraf https://github.com/influxdata/telegraf Collector to grab and send data from different sources to InfluxDB and other databases 30 Based on Input and out Plugin System
  31. 31. 31 Telegraf Plugins
  32. 32. Kapacitor https://github.com/influxdata/kapacitor Trigger notifications and make action in case of specific behaviors 32 framework for processing, monitoring, and alerting on time series data
  33. 33. 33 Kapacitor high CPU alert stream |from() .measurement('cpu_usage_idle') .groupBy('host') |window() .period(1m) .every(1m) |mean('value') |eval(lambda: 100.0 - "mean").as('used') |alert() .message('{{ .Level}}: {{ .Name }}/{{ index .Tags "host" }} has high cpu: {{ index .Fields "used" }}') .warn(lambda: "used" > 70.0) .crit(lambda: "used" > 85.0) // Send alert to hander of choice. // Slack .slack() .channel('#alerts') // PagerDuty .pagerDuty()
  34. 34. 34 Demo https://github.com/gianarb/tick-php
  35. 35. 35 When you start to work with "micro"services understand the topology of your connections is really important Time series can help you
  36. 36. 36 Use a dashboard to put together different metrics and create not obvious relations
  37. 37. 37 Why InfluxDB and not something else? https://www.influxdata.com/influxdb-is-27x-faster-vs-mongodb-for-time-series-workloads/ 27x greater write throughput 84x less disk space
  38. 38. 38 That’s it! A series of great tools to monitor your applications and your infrastructure
  39. 39. 39 A monitoring system isn’t for all
  40. 40. Thanks! See you around twitter.com/gianarb 40

×