Introduction to InfluxDB, an Open Source Distributed Time Series Database by Paul Dix

22,551 views
21,964 views

Published on

In this presentation, Paul introduces InfluxDB, a distributed time series database that he open sourced based on the backend infrastructure at Errplane. He talks about why you'd want a database specifically for time series and he covers the API and some of the key features of InfluxDB, including:

• Stores metrics (like Graphite) and events (like page views, exceptions, deploys)
• No external dependencies (self contained binary)
• Fast. Handles many thousands of writes per second on a single node
• HTTP API for reading and writing data
• SQL-like query language
• Distributed to scale out to many machines
• Built in aggregate and statistics functions
• Built in downsampling

Published in: Technology
2 Comments
56 Likes
Statistics
Notes
  • http://stackoverflow.com/questions/25540722/how-to-make-a-choice-between-opentsdb-and-influxdb-or-other-tsds
    anyone may give some advice?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Video: https://www.youtube.com/watch?v=sRi64imN7xg
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
22,551
On SlideShare
0
From Embeds
0
Number of Embeds
2,926
Actions
Shares
0
Downloads
278
Comments
2
Likes
56
Embeds 0
No embeds

No notes for slide

Introduction to InfluxDB, an Open Source Distributed Time Series Database by Paul Dix

  1. 1. Introducing InfluxDB, an open source distributed time series database Paul Dix @pauldix paul@errplane.com
  2. 2. About me ● Co-founder, CEO of Errplane (YC W13) ● Organizer of NYC Machine Learning ● Author of “Service Oriented Design with Ruby & Rails”
  3. 3. Series editor for Addison Wesley’s “Data & Analytics”
  4. 4. What is a time series?
  5. 5. Metrics
  6. 6. Events ● ● ● ● ● ● ● Measurements Exceptions Page Views User actions Commits Deploys Things happening in time...
  7. 7. Analytics operations, developers, users, business
  8. 8. Things you want to ask questions about, visualize, or summarize over time.
  9. 9. Actually a summarization
  10. 10. Also a summarization
  11. 11. What about... “...order by some_time_col”
  12. 12. Why a database for time series?
  13. 13. Billions of data points. Scale horizontally.
  14. 14. HTTP native. API to build on.
  15. 15. Built in tools for downsampling and summarizing
  16. 16. Automatically clear out old data if we want
  17. 17. Process or monitor data as it comes in, like Storm
  18. 18. Visualize and Summarize ● ● ● ● ● ● ● ● Graphs & dashboards Last 10 minutes Last 4 hours Last 24 hours Past week Past month YTD All Time
  19. 19. Data Collection ● Statsd - https://github.com/etsy/statsd/ ● CollectD - http://collectd.org/ ● Heka - https://github.com/mozillaservices/heka ● l2met - https://github. com/ryandotsmith/l2met ● Libraries ● Framework integrations ● Cloud integrations (AWS, OpenStack) ● Third-party integrations
  20. 20. Existing Tools ● ● ● ● ● RRDTool (metrics) Graphite (metrics) OpenTSDB (metrics + events) Kairos (metrics + events) and others...
  21. 21. Something missing...
  22. 22. InfluxDB: harness lightning, get 1.21 gigawatts.
  23. 23. InfluxDB ● ● ● ● ● Written in Go Uses LevelDB for storage (may change) Self contained binary No external dependencies Distributed (in December)
  24. 24. HTTP Native ● Read/write data via HTTP ● Manage via HTTP ● Security model to allow access directly from browser
  25. 25. How data is organized ● Databases (like in MySQL, Postgres, etc) ● Time series (kind of like tables) ● Points or events (kind of like rows)
  26. 26. Security ● Cluster admins ● Database admins ● Database users ○ read permissions ■ only certain series ■ only queries with a column having a specific value (e.g. customer_id=32) ○ write permissions ■ only certain series ■ only with columns having a specific value
  27. 27. InfluDB Setup ● http://play.influxdb.org ● OSX ○ brew update && brew install influxdb ● http://influxdb.org/download ● Ubuntu ○ sudo dpkg -i influxdb_latest_amd64.deb ● RedHat ○ sudo rpm -ivh influxdb-latest-1.i686.rpm
  28. 28. Examples, but sadly no R :(
  29. 29. HTTP API docs at http://influxdb.org/docs/api/http
  30. 30. https://github.com /influxdb/influxdb-r fork, write sweet code, submit PR, be loved and adored FOREVER
  31. 31. Create a database curl -X POST 'http://localhost:8086/db?u=root&p=root' -d '{"name":"mydb", "replicationFactor": 3}'
  32. 32. Add a user curl -X POST 'http://.../db/mydb/users?u=root&p=root' -d '{"name":"paul", "password": "foo", "admin": true}'
  33. 33. Write points curl -X POST 'http://localhost:8086db/mydb/series?u=paul&p=pass' -d '[{"name":"foo", "columns":["val"], "points": [[3]]}]'
  34. 34. Querying curl 'http://...:8086/db/mydb/series?u=paul&p=pass&q=...'
  35. 35. SQL(ish) Query Language select * from user_events where time > now() - 4h
  36. 36. JSON data returned [{ "name": "foo", "columns": [ "time", "sequence_number", "val1", "val2" ], "points": [ [1384295094, 3, "paul", 23], [1384295094, 2, "john", 92], [1384295094, 1, "todd", 61] ] }, {...}]
  37. 37. select count(state) from user_events group by time(5m), state where time > now() - 7d
  38. 38. select percentile(value, 90) from response_times group by time(30s) where time > now() - 1h
  39. 39. Continuous Queries (downsampling) select percentile(value, 90) from response_times group by time(5m) into response_times.percentiles.90
  40. 40. Continuous queries for real-time processing & monitoring
  41. 41. Regexes select * from events where email =~ /.*gmail.com/
  42. 42. select percentile(value, 99) from /stats.*/ into :series_name.percentiles.99
  43. 43. select count(value) from seriesA merge seriesB
  44. 44. Querying ● Functions ○ count, min, max, mean, distinct, median, mode, percentiles, derivative, stddev ● Where clauses ● Group by clauses (time and other columns) ● Periodically delete old raw data
  45. 45. Built in UI
  46. 46. CLI
  47. 47. Libraries ● ● ● ● ● ● ● Ruby Frontend JS Node Python PHP Go (soon) Java (soon)
  48. 48. Ideas to come... ● Custom functions ○ Embedded LUA, YARN like interface, or both? ● Custom real-time queries ○ define custom logic and InfluxDB will feed it data ● Queries triggering web hooks ○ pair with custom functions for monitoring/anomaly detection
  49. 49. Project Status ● Based on work at https://errplane.com ○ 2 billion points per month ● ● ● ● ● http://influxdb.org Code available at https://github.com/influxdb API finalized in the next month Clustered version in December Production ready by end of year
  50. 50. We’re available for consulting/help
  51. 51. We need your help ● ● ● ● ● ● API, what else would you like to see? Client libraries Visualization tools Data collection integrations Comments/feedback on the mailing list http://influxdb.org/overview/
  52. 52. Share the love ● Star or watch the project on http://github. com/influxdb/influxdb ● Tweet, blog, shout, whisper ● Participate in discussions on mailing list
  53. 53. Come to the hackfest ● Monday, December 2nd at Pivotal ● http://meetup.com/nyc-influxdb-user-group
  54. 54. OSS lives and dies by adoption/popularity
  55. 55. MongoDB has 4,406 stars
  56. 56. MongoDB valued at $1.2B
  57. 57. Each star worth $272,355.00
  58. 58. Help InfluxDB get to 10k stars! go forth and build!
  59. 59. Thanks! @pauldix paul@errplane.com

×