Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Flux and InfluxDB 2.0 by Paul Dix

410 views

Published on

In this InfluxDays NYC 2019 talk, InfluxData Founder & CTO Paul Dix will outline his vision around the platform and its new data scripting and query language Flux, and he will give the latest updates on InfluxDB time series database. This talk will walk through the vision and architecture with demonstrations of working prototypes of the projects.

Published in: Technology
  • Login to see the comments

  • Be the first to like this

Flux and InfluxDB 2.0 by Paul Dix

  1. 1. Flux and InfluxDB 2.0 Paul Dix @pauldix paul@influxdata.com
  2. 2. • Data-scripting language • Functional • MIT Licensed • Language, VM, engine, planner, optimizer
  3. 3. Language + Query Engine
  4. 4. 2.0
  5. 5. Biggest Change Since 0.9
  6. 6. Clean Migration Path
  7. 7. Compatibility Layer
  8. 8. • MIT Licensed • Multi-tenanted • Telegraf, InfluxDB, Chronograf, Kapacitor rolled into 1 • OSS single server • Cloud usage based pricing • Dedicated Cloud • Enterprise on-premise
  9. 9. Consistent Documented API Collection, Write/Query, Streaming & Batch Processing, Dashboards
  10. 10. Officially Supported Client Libraries Go, Node.js, Ruby, Python, PHP, Java, C#, C, Kotlin
  11. 11. Visualization Libraries
  12. 12. Multi-tenant roles • Operator • Organization Administrator • User
  13. 13. Data Model • Organizations • Buckets (retention) • Time series data • Tasks • Runs • Logs • Dashboards • Users • Tokens • Authorizations • Protos (templates) • Scrapers • Telegrafs • Labels
  14. 14. Ways to run Flux - (interpreter, InfluxDB 1.7 & 2.0)
  15. 15. Flux Basics
  16. 16. // get all data from the telegraf db from(bucket:”telegraf/autogen”) // filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
  17. 17. // get all data from the telegraf db from(bucket:”telegraf/autogen”) // filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Comments
  18. 18. // get all data from the telegraf db from(bucket:”telegraf/autogen”) // filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Named Arguments
  19. 19. // get all data from the telegraf db from(bucket:”telegraf/autogen”) // filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") String Literals
  20. 20. // get all data from the telegraf db from(bucket:”telegraf/autogen”) // filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Buckets, not DBs
  21. 21. // get all data from the telegraf db from(bucket:”telegraf/autogen”) // filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Duration Literal
  22. 22. // get all data from the telegraf db from(bucket:”telegraf/autogen”) // filter that by the last hour |> range(start:2018-11-07T00:00:00Z) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Time Literal
  23. 23. // get all data from the telegraf db from(bucket:”telegraf/autogen”) // filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Pipe forward operator
  24. 24. // get all data from the telegraf db from(bucket:”telegraf/autogen”) // filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Anonymous Function
  25. 25. // get all data from the telegraf db from(bucket:”telegraf/autogen”) // filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => (r._measurement == "cpu" or r._measurement == “cpu") and r.host == “serverA") Predicate Function
  26. 26. // variables some_int = 23
  27. 27. // variables some_int = 23 some_float = 23.2
  28. 28. // variables some_int = 23 some_float = 23.2 some_string = “cpu"
  29. 29. // variables some_int = 23 some_float = 23.2 some_string = “cpu" some_duration = 1h
  30. 30. // variables some_int = 23 some_float = 23.2 some_string = “cpu" some_duration = 1h some_time = 2018-10-10T19:00:00
  31. 31. // variables some_int = 23 some_float = 23.2 some_string = “cpu" some_duration = 1h some_time = 2018-10-10T19:00:00 some_array = [1, 6, 20, 22]
  32. 32. // variables some_int = 23 some_float = 23.2 some_string = “cpu" some_duration = 1h some_time = 2018-10-10T19:00:00 some_array = [1, 6, 20, 22] some_object = {foo: "hello" bar: 22}
  33. 33. // defining a pipe forwardable function square = (tables=<-) => tables |> map(fn: (r) => {r with _value: r._value * r._value})
  34. 34. // defining a pipe forwardable function square = (tables=<-) => tables |> map(fn: (r) => {r with _value: r._value * r._value}) This is potentially new
  35. 35. // defining a pipe forwardable function square = (tables=<-) => tables |> map(fn: (r) => {r with _value: r._value * r._value}) from(bucket:"foo") |> range(start: -1h) |> filter(fn: (r) => r._measurement == "samples") |> square() |> filter(fn: (r) => r._value > 23.2)
  36. 36. Data Sources (inputs)
  37. 37. Data Sinks (outputs)
  38. 38. Tasks
  39. 39. option task = { name: "email alert digest", cron: "0 5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message)
  40. 40. option task = { name: "email alert digest", cron: "0 5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message) tasks
  41. 41. option task = { name: "email alert digest", cron: "0 5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message) cron scheduling
  42. 42. option task = { name: "email alert digest", cron: "0 5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message) packages & imports
  43. 43. option task = { name: "email alert digest", cron: "0 5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message) map
  44. 44. option task = { name: "email alert digest", cron: "0 5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message) String interpolation
  45. 45. option task = { name: "email alert digest", cron: "0 5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message) Ship data elsewhere
  46. 46. option task = { name: "email alert digest", cron: "0 5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message) Store secrets in a store like Vault
  47. 47. Open Questions
  48. 48. User Packages & Dependencies
  49. 49. // in a file called package.flux package "paul" option version = "0.1.1" // define square here or…
  50. 50. // in a file called package.flux package "paul" option version = "0.1.1" // import the other package files // they must have package "paul" declaration at the top // only package.flux has the version import “packages” packages.load(files: ["square.flux", "utils.flux"])
  51. 51. // in a file called package.flux package "paul" option version = "0.1.1" // import the other package files // they must have package "paul" declaration at the top // only package.flux has the version import “packages” packages.load(files: ["square.flux", "utils.flux"]) // or this packages.load(glob: "*.flux")
  52. 52. import "myorg/paul" // latest, will load package.flux data |> paul.square()
  53. 53. import "myorg/paul", "0.1.0" // specific version // 1. look in $fluxhome/myorg/paul/package.flux // 2. look in $fluxhome/myorg/paul/0.1.0/package.flux // 3. look in cloud2.influxdata.com/api/v1/packages/myorg/paul data |> paul.square()
  54. 54. import "myorg/paul", ">=0.1.0" // at least this version data |> paul.square()
  55. 55. Error Handling?
  56. 56. import "slack" // what if this returns an error? ret = slack.send(room: "foo", message: "testing this", token: "...")
  57. 57. Option Types?
  58. 58. match ret { // on match ret gets mapped as the new type Error => { // do something ret }, Else => { // do something with ret } }
  59. 59. Loops?
  60. 60. records = [ {name: "foo", value: 23}, {name: "bar", value: 23}, {name: "asdf", value: 56} ]
  61. 61. records = [ {name: "foo", value: 23}, {name: "bar", value: 23}, {name: "asdf", value: 56} ] // simple loop over each records |> map(fn: (r) => {name: r.name, value: r.value + 1})
  62. 62. records = [ {name: "foo", value: 23}, {name: "bar", value: 23}, {name: "asdf", value: 56} ] // simple loop over each records |> map(fn: (r) => {name: r.name, value: r.value + 1}) // compute the sum sum = records |> reduce( fn: (r, accumulator) => r.value + accumulator, i: 0 )
  63. 63. records = [ {name: "foo", value: 23}, {name: "bar", value: 23}, {name: "asdf", value: 56} ] // simple loop over each records |> map(fn: (r) => {name: r.name, value: r.value + 1}) // compute the sum sum = records |> reduce( fn: (r, accumulator) => r.value + accumulator, i: 0 ) // get matching records foos = records |> filter(fn: (r) => r.name == "foo")
  64. 64. while(fn: () => { // do stuff })
  65. 65. while(fn: () => { // do stuff }) while = (fn) => if fn() while(fn)
  66. 66. // or loop some number of times loop(fn: (i) => { // do stuff here }, times: 10)
  67. 67. // or loop some number of times loop(fn: (i) => { // do stuff here }, times: 10) loop = (fn, times) => loopUntil(fn, 0, times)
  68. 68. // or loop some number of times loop(fn: (i) => { // do stuff here }, times: 10) loop = (fn, times) => loopUntil(fn, 0, times) loopUntil = (fn, iteration, times) => if iteration < times { fn(iteration) loopUntil(fn, iteration + 1, times) }
  69. 69. Syntactic Sugar
  70. 70. // <stream object>[<predicate>,<time>:<time>,<list of strings>]
  71. 71. // <stream object>[<predicate>,<time>:<time>,<list of strings>] // and here's an example from(bucket:"foo")[_measurement == "cpu" and _field == "usage_user", 2018-11-07:2018-11-08, ["_measurement", "_time", "_value", “_field”]]
  72. 72. // <stream object>[<predicate>,<time>:<time>,<list of strings>] // and here's an example from(bucket:"foo")[_measurement == "cpu" and _field == "usage_user", 2018-11-07:2018-11-08, ["_measurement", "_time", "_value", “_field”]] from(bucket:"foo") |> filter(fn: (row) => row._measurement == "cpu" and row._field == "usage_user") |> range(start: 2018-11-07, stop: 2018-11-08) |> keep(columns: ["_measurement", "_time", "_value", “_field"])
  73. 73. from(bucket:"foo")[_measurement == "cpu"] // notice the trailing commas can be left off from(bucket: "foo") |> filter(fn: (row) => row._measurement == "cpu") |> last()
  74. 74. from(bucket:"foo")["some tag" == "asdf",,] from(bucket: "foo") |> filter(fn: (row) => row["some tag"] == "asdf") |> last()
  75. 75. from(bucket:"foo")[foo=="bar",-1h] from(bucket: "foo") |> filter(fn: (row) => row.foo == "bar") |> range(start: -1h)
  76. 76. bucket = "foo" start = -3 from(bucket: bucket) |> range(start: start, end: -1) // shortcut if the variable name is the same as the argument from(bucket) |> range(start, end: -1)
  77. 77. Flux Office Hours Tomorrow
  78. 78. InfluxDB 2.0 Status
  79. 79. Thank you Paul Dix paul@influxdata.com @pauldix

×