This document discusses using Elasticsearch as a time series database. It covers why Elasticsearch was chosen over other options for storing metrics from the open source performance monitoring tool Stagemonitor. The document discusses Elasticsearch's ability to scale, its functions and visualization support in Kibana. It also covers how Stagemonitor's data is modeled in Elasticsearch, including the use of tags, and how index management is handled through a hot/cold node architecture and tools like Curator.

1. Elasticsearch as a time series
database
Felix Barnsteiner | iSYS Software

2. Felix Barnsteiner
● Author of the blog “Elasticsearch
as a Time Series Data Store”
● Project Lead stagemonitor
○ Open Source performance
monitoring
● iSYS Software GmbH
○ custom software
○ jstage.de
■ E-Commerce Platform, Consulting
and Services
○ We’re hiring!

3. Agenda
● Time Series What?
● Why Elasticsearch?
● Data Model
● Elasticsearch Mapping Tuning
● Index Management

5. Time Series what?
Time
t

6. Time Series what?
Series
t

7. Time Series what?
Series
t

8. Time Series what?
t
CPUUtilisation%

9. Time Series what?
t
Revenue/minute

10. Why Elasticsearch?
Decision Process

11. In search for a new TSDB
● New database for stagemonitor
● Replacing Graphite
○ Installation can be tricky
○ Scaling issues
○ Memory hungry
○ No support for special chars
■ GET /index.html
■ GET_|index:html
● Good things about Graphite
○ Grafana
○ Functions
○ Community/Tools/Collectors

12. Stagemonitor
● Open Source Java Performance Monitoring
○ Mainly Web Applications
● Monitoring of clustered environments
● For Dev, QA and Ops

13. In Browser Widget
●

14. Request Analysis
● Stores Request Traces directly to
Elasticsearch
○ Response Time
○ Status Code
○ Browser/Device/OS/User Agent Type
○ Exceptions
● No Logstash/Log parsing needed
● Kibana dashboard

15. Metrics
● Response Times
● Real User Monitoring
● JVM
● Operating System
● EhCache
● Logging
● JDBC

16. TSDB Requirements
● Easy to install
● Scalability
● Variety of functions
○ derivative
○ max/min/std
○ percentiles
○ ...
● Great visualization support
● Maturity

17. Many Options
● InfluxDB
● OpenTSDB
● Elasticsearch
● KairosDB
● Prometheus
● Druid

18. Don’t Match Requirements
● OpenTSDB
○ Not easy to install
○ Hadoop and Hbase
● KairosDB
○ Not easy to install
○ Cassandra + KairosDB + Config
● Prometheus
○ No Clustering
● Druid
○ No visualization tool

19. Easy to install
● InfluxDB
○ rpm, deb, homebrew
● Elasticsearch
○ rpm, deb, download & run
○ Runs anywhere
○ Stagemonitor already requires Elasticsearch

20. Scaleability
● InfluxDB
○ CERN Presentation*
● Elasticsearch
* http://cds.cern.ch/record/2011172/files/LHCb-TALK-2015-060.pdf
*

21. Maturity
● InfluxDB
○ Many core changes
○ Clustering in alpha state (max 3 nodes)
○ New Storage Engine in beta
● Elasticsearch
○ Yes!

22. Functions
● InfluxDB
○ docs.influxdata.com/influxdb/v0.
9/query_language/functions/
● Elasticsearch
○ www.elastic.
co/guide/en/elasticsearch/reference/current/se
arch-aggregations.html
○ Hold-Winters: predictions based on seasonal
patterns

23. Visualization Support
● InfluxDB
○ Grafana
○ Chronograf
● Elasticsearch
○ Kibana
○ Grafana
○ Timelion

24. Elasticsearch Benefits
● Large community
● Proven to scale
● Redundancy - replicas
● Backups - snapshot and restore
● Easy to install and manage
● Already needed for stagemonitor
○ Only one instead of two data bases to install
● Marvel uses ES as a TSDB
● CERN says

25. Stagemonitor Outputs
● Elasticsearch
○ Default
○ Preconfigured Dashboards
● InfluxDB
● Graphite

26. Data Model

27. How do Timers work?
● Stagemonitor uses Dropwizard Metrics
● Not one report per request
○ Not scaleable
○ Sampling
● Response times are aggregated in memory
● Stores 1000 representative response times
● Computes metrics like percentiles, min/max,
stdev, throughput/rate
● Reports each 60 seconds (configurable)

28. metics_store_benchmark.my_hostname.local.timer_1.count 10
metics_store_benchmark.my_hostname.local.timer_1.mean 123
metics_store_benchmark.my_hostname.local.timer_1.min 12
metics_store_benchmark.my_hostname.local.timer_1.max 1234
metics_store_benchmark.my_hostname.local.timer_1.p50 124
…
Doesn’t really fit into Elasticsearch’s data model
How do I filter by a certain host? Regex??
Classic Data Model - Timer
Application Host Instance Name Value

29. Regex??

30. Data Model 2.0
● Tags and values
● Self descriptive
● Add tags
(dimensions)
afterwards
○ Without changing
identity
○ Backwards
compatible

31. Data Model - Timer
● Metric Timer name
○ http_time
○ jdbc_time

32. Data Model - Timer
● Metric Timer name
○ http_time
○ jdbc_time

33. Data Model - Timer
● Global tags
○ Application
○ Host
○ Instance
○ Region
○ Datacenter
○ Availability Zone
○ ...

34. Data Model - Timer
● Metrics
○ mean/min/max
○ percentiles
○ throughput, count

35. Elasticsearch Mapping Tuning

36. Mapping Template

37. Mapping
● _all: contains all values of all other fields
● _source: contains original JSON
● Saves disk space
● BUT…
● JSON is not visible
● Data can only be retrieved via aggregations

38. Mapping
● Don’t analyze tags (like application, host, ...)
○ No full text search/stemming needed
○ Only filter by exact values
● doc_values: true
○ Reduces heap usage (and OOMEs) at the cost of
disk space
○ Needed for aggregations (un-inverted index)
○ Default in 2.0

39. Mapping
● index: no for metric values
○ Not searchable by value
○ Saves disk space
● Using integers and floats instead of longs and
doubles
○ Saves disk space
○ OK for our use case
○ May not be OK for yours

40. Benchmark
● ~23 M datapoints
● A week’s worth of stagemonitor metric data
● when reporting every minute
-> ~500MB
InfluxDB: ~360MB (0.9.4.1)

41. Index Management

42. One Index Per Day
● Logstash like index format
● stagemonitor-metrics-2016.01.18
● Only relevant indices have to be queried
● Easier/more efficient to delete
● Mapping can be changed every day
○ Number of shards

43. Hot/Cold Architecture
● Recent data is queried more often
● Beefy nodes hold recent data (hot)
○ More CPU, RAM
○ SSDs
● Cheaper nodes hold older data (cold)
○ Less CPU, RAM
○ HDDs

44. Hot/Cold Architecture
Beefy Node(s) Cheap Node(s)
elasticsearch.yml
node.box_type: hot
elasticsearch.yml
node.box_type: cold
Index
Today
Index
Yesterday
Index 2
days ago
... ... ...

45. Shard allocation filtering
● New Indices are allocated to hot nodes
{
"template": "stagemonitor-metrics-*",
"settings": {
"index": {
"routing.allocation.require.box_type": "hot"
}
}
…
}

46. Shard allocation filtering
● As indices grow old, they are moved to cold
nodes
PUT /stagemonitor-requests-*,
-stagemonitor-requests-2016.01.18,
-stagemonitor-requests-2016.01.17,
-stagemonitor-requests-2016.01.16/_settings
{
index.routing.allocation.require.box_type=cold
}

47. Hot/Cold Architecture
Beefy Node(s) Cheap Node(s)
elasticsearch.yml
node.box_type: hot
elasticsearch.yml
node.box_type: cold
New
Index
Old
Index
Old
Index

48. Force Merge (optimize)
● Improves resource use
● Improves cluster recovery speed and restarts
of node
● Requires a lot of CPU and disk resources
while running
● Should be performed on cold nodes
○ They usually aren’t doing much
○ Hot nodes usually are busy due to
indexing/querying

49. Deleting indices
● Limit storage requirements
● Only keep metrics for one year

50. Lifecycle
New Old
Very
Old
Cheap Node
Optimize
Beefy Node Delete
Archive

51. Tools
● Curator
○ Command line tool by Elastic
○ Execute 3 jobs as cron
● Stagemonitor
○ Automatically handled
○ How many days on hot nodes
○ Configurable delete delay

52. Questions?
Blog: elastic.co/blog/elasticsearch-as-a-time-
series-data-store
Stagemonitor: stagemonitor.org
Twitter: @felix_b, @stagemonitor
@felix_b