Successfully reported this slideshow.

Scott Anderson [InfluxData] | Map & Reduce – The Powerhouses of Custom Flux Functions | InfluxDays Virtual Experience London 2020

0

Share

1 of 94
1 of 94

More Related Content

More from InfluxData

Related Books

Free with a 14 day trial from Scribd

See all

Scott Anderson [InfluxData] | Map & Reduce – The Powerhouses of Custom Flux Functions | InfluxDays Virtual Experience London 2020

  1. 1. Scott Anderson Technical Writer @ InfluxData Map & Reduce The Powerhouses of Custom Flux Functions
  2. 2. © 2020 InfluxData. All rights reserved. 2 Disclaimer This presentation is not about Hadoop MapReduce-like functionality in Flux. This is about building custom functions with map() and reduce().
  3. 3. © 2020 InfluxData. All rights reserved. 3 Introduction to Flux www.influxdata.com/resources/introduction-to- flux-and-functional-data-scripting/
  4. 4. A quick background Flux
  5. 5. © 2020 InfluxData. All rights reserved. 5 “ Flux is a functional data scripting language that lets you query, process, write, and alert on data in a single syntax. ”
  6. 6. Design Goals ● Turing-complete ● Usable ● Readable ● Flexible ● Extensible ● Testable ● Contributable ● Shareable
  7. 7. Design Goals ● Turing-complete ● Usable ● Readable ● Flexible ● Extensible ● Testable ● Contributable ● Shareable
  8. 8. Design Goals ● Turing-complete ● Usable ● Readable ● Flexible ● Extensible ● Testable ● Contributable ● Shareable
  9. 9. © 2020 InfluxData. All rights reserved. 9 Extensibility ● InfluxQL was and is hard to extend. ● TICKscript was and is easier to extend, but depends on external languages for true customization.
  10. 10. © 2020 InfluxData. All rights reserved. 10 Extensibility ● Early strategy for Flux has been to create “primitive” functions that can be used to create other functions. ● Flux recently introduced architecture and guidelines for community contributions.
  11. 11. If it doesn’t exist, create it Custom Functions
  12. 12. © 2020 InfluxData. All rights reserved. 12 Custom Function Definition funcName = (parameters) => funcOperations Function square = (x) => x * x Transformation somethingCool = (tables=<-) => tables |> ...
  13. 13. © 2020 InfluxData. All rights reserved. 13 Functions & Transformations Function bool(v: 1) // Returns true Transformation toBool = (tables=<-) => tables |> map(fn: (r) => ({ r with _value: bool(v: r._value) }))
  14. 14. © 2020 InfluxData. All rights reserved. 14 Functions & Transformations from(bucket: "myData") |> range(start: -1h) |> filter(fn: (r) => r._field == "enabled") |> toBool()
  15. 15. © 2020 InfluxData. All rights reserved. 15 map() & reduce()
  16. 16. Remap rows fer’days map()
  17. 17. © 2020 InfluxData. All rights reserved. 17 _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempC 21.29 2020-06-01T00:12:01Z TM02001 tempC 21.36 2020-06-01T00:12:02Z TM02001 tempC 21.34
  18. 18. © 2020 InfluxData. All rights reserved. 18 _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempC 21.29 { _time: 2020-06-01T00:12:00Z, sensorID: "TM02001", _field: "tempC", _value: 21.29 }
  19. 19. © 2020 InfluxData. All rights reserved. 19 map( fn: ) (r) => ({ ... }) )
  20. 20. © 2020 InfluxData. All rights reserved. 20 (r) => ({ ... }) r = { _time: 2020-06-01T00:12:00Z, sensorID: "TM02001", _field: "tempC", _value: 21.29 }
  21. 21. © 2020 InfluxData. All rights reserved. 21 r = { _time: 2020-06-01T00:12:00Z, sensorID: "TM02001", _field: "tempC", _value: 21.29 } (r) => ({ _value: r._value })
  22. 22. © 2020 InfluxData. All rights reserved. 22 r = { _time: 2020-06-01T00:12:00Z, sensorID: "TM02001", _field: "tempC", _value: 21.29 } map(fn: (r) => ({ _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 }))
  23. 23. © 2020 InfluxData. All rights reserved. 23 { _field: "tempF", _value: 70.322 } map(fn: (r) => ({ _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 })) _field _value tempF 70.322
  24. 24. © 2020 InfluxData. All rights reserved. 24 { _time: 2020-06-01T00:12:00Z, sensorID: "TM02001", _field: "tempF", _value: 70.322 } map(fn: (r) => ({ r with _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 }))
  25. 25. © 2020 InfluxData. All rights reserved. 25 map(fn: (r) => ({ r with _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 })) _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempF 70.322
  26. 26. © 2020 InfluxData. All rights reserved. 26 |> map(fn: (r) => ({ r with _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 })) _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempC 21.29 2020-06-01T00:12:01Z TM02001 tempC 21.36 2020-06-01T00:12:02Z TM02001 tempC 21.34 _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempF 70.322 2020-06-01T00:12:01Z TM02001 tempF 70.448 2020-06-01T00:12:02Z TM02001 tempF 70.412
  27. 27. © 2020 InfluxData. All rights reserved. 27 toF Create a custom transformation toF = (tables=<-)toF = (tables=<-) => tables toF = (tables=<-) => tables |> map( fn: (r) => ({ r with _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 }) )
  28. 28. © 2020 InfluxData. All rights reserved. 28 |> map(fn: (r) => ({ r with _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 })) _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempC 21.29 2020-06-01T00:12:01Z TM02001 tempC 21.36 2020-06-01T00:12:02Z TM02001 tempC 21.34 _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempF 70.322 2020-06-01T00:12:01Z TM02001 tempF 70.448 2020-06-01T00:12:02Z TM02001 tempF 70.412 |> toF()
  29. 29. © 2020 InfluxData. All rights reserved. 29 Temperature Range Temperature Text below 0°C freezing 0°C - 15.5°C cold 15.5°C - 26.5°C mild 26.5°C - 37.5°C hot 37.5°C or higher very hot Add text based on temperature range?
  30. 30. © 2020 InfluxData. All rights reserved. 30 (r) => ({ column: if r._value == "foo" then "bar" else "baz" }) Use conditional logic!
  31. 31. © 2020 InfluxData. All rights reserved. 31 tempToText = (tables=<-) => tables |> map(fn: (r) => ({ r with }) ) tempToText = (tables=<-) => tables |> map(fn: (r) => ({ r with temp_text: }) ) tempToText = (tables=<-) => tables |> map(fn: (r) => ({ r with temp_text: if r._value <= 0.0 then "freezing" }) ) tempToText = (tables=<-) => tables |> map(fn: (r) => ({ r with temp_text: if r._value <= 0.0 then "freezing" else if r._value > 0.0 and r._value <= 15.5 then "cold" }) ) tempToText = (tables=<-) => tables |> map(fn: (r) => ({ r with temp_text: if r._value <= 0.0 then "freezing" else if r._value > 0.0 and r._value <= 15.5 then "cold" else if r._value > 15.5 and r._value <= 26.5 then "mild" else if r._value > 26.5 and r._value <= 37.5 then "hot" else if r._value > 37.5 then "very hot" }) ) tempToText = (tables=<-) => tables |> map(fn: (r) => ({ r with temp_text: if r._value <= 0.0 then "freezing" else if r._value > 0.0 and r._value <= 15.5 then "cold" else if r._value > 15.5 and r._value <= 26.5 then "mild" else if r._value > 26.5 and r._value <= 37.5 then "hot" else if r._value > 37.5 then "very hot" else "unknown" }) )
  32. 32. © 2020 InfluxData. All rights reserved. 32 |> tempToText() _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempC -1.2 2020-06-01T00:12:01Z TM02001 tempC 22.1 2020-06-01T00:12:02Z TM02001 tempC 31.8 _time sensorID _field _value temp_text 2020-06-01T00:12:00Z TM02001 tempC -1.2 freezing 2020-06-01T00:12:01Z TM02001 tempC 22.1 mild 2020-06-01T00:12:02Z TM02001 tempC 31.8 hot
  33. 33. Example custom functions using map()
  34. 34. © 2020 InfluxData. All rights reserved. 34 Basic mathematical operations multiplyByX = (tables=<-, x) => tables |> map(fn: (r) => ({ r with _value: r._value * x })) square = (tables=<-) => tables |> map(fn: (r) => ({ r with _value: r._value * r._value })) quarter = (tables=<-) => tables |> map(fn: (r) => ({ r with _value: r._value / 4.0 }))
  35. 35. © 2020 InfluxData. All rights reserved. 35 Other mathematical operations import "math" sin = (tables=<-) => tables |> map(fn: (r) => ({ r with _value: math.sin(x: r._value) })) circumference = (tables=<-) => tables |> map(fn: (r) => ({ r with circumference: 2.0 * math.pi * r._value })) circularArea = (tables=<-) => tables |> map(fn: (r) => ({ r with area: math.pi * r._value ^ 2.0 }))
  36. 36. © 2020 InfluxData. All rights reserved. 36 String manipulationimport "strings" rmHostPrefix = (tables=<-) => tables |> map(fn: (r) => ({ r with host: strings.trimPrefix(v: r.host, prefix: "host_") })) replaceSpaces = (tables=<-, char="_") => tables |> map(fn: (r) => ({ r with _value: strings.replaceAll(v: r._value, t: " ", u: char) })) humanReadableMessage = (tables=<-) => tables |> map(fn: (r) => ({ r with humanReadable: "The current value is ${string(v: r._value)}." }))
  37. 37. © 2020 InfluxData. All rights reserved. 37 Common gotchas ● The with operator is your friend. ● Mathematical operands must be of the same data type. ● map() does not add new columns to group keys. ● map() only operates on a single row at a time. ○ Can’t use values from previous or subsequent rows. ○ If your operation requires values from separate fields, use pivot() or join() to align field values in rows.
  38. 38. Reduce all the things!! reduce()
  39. 39. © 2020 InfluxData. All rights reserved. 39 |> reduce(...)
  40. 40. © 2020 InfluxData. All rights reserved. 40 _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempC 21.29 2020-06-01T00:12:01Z TM02001 tempC 21.36 2020-06-01T00:12:02Z TM02001 tempC 21.34 sensorID _field _value TM02001 tempC 21.33
  41. 41. © 2020 InfluxData. All rights reserved. 41 reduce( fn: identity: )
  42. 42. © 2020 InfluxData. All rights reserved. 42 reduce( fn: (r, accumulator) => ({ ... }), identity: )
  43. 43. © 2020 InfluxData. All rights reserved. 43 reduce( fn: (r, accumulator) => ({ ... }), identity: { ... } )
  44. 44. © 2020 InfluxData. All rights reserved. 44 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0001, _field: "example", _value: 1.3} group_key = [ _field ] fn: (r, accumulator) => ({ sum: r._value + accumulator.sum }) accumulator = {sum: 0.0} identity: {sum: 0.0}
  45. 45. © 2020 InfluxData. All rights reserved. 45 { sum: r._value + accumulator.sum } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0001, _field: "example", _value: 1.3} group_key = [ _field ] accumulator = {sum: 0.0} identity: {sum: 0.0} { sum: 1.3 + 0.0 }{ sum: 1.3 } Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  46. 46. © 2020 InfluxData. All rights reserved. 46 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0002, _field: "example", _value: 12.2} group_key = [ _field ] accumulator = {sum: 1.3} identity: {sum: 0.0} { sum: 12.2 + 1.3 }{ sum: 13.5 } Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  47. 47. © 2020 InfluxData. All rights reserved. 47 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0003, _field: "example", _value: 4.9} group_key = [ _field ] fn: (r, accumulator) => ({ sum: r._value + accumulator.sum }) accumulator = {sum: 13.5} identity: {sum: 0.0} { sum: 18.4 } Returns:
  48. 48. © 2020 InfluxData. All rights reserved. 48 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0003, _field: "example", _value: 3.2} group_key = [ _field ] accumulator = {sum: 18.4} identity: {sum: 0.0} { sum: 21.6 } Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  49. 49. © 2020 InfluxData. All rights reserved. 49 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0003, _field: "example", _value: 15.8} group_key = [ _field ] accumulator = {sum: 21.6} identity: {sum: 0.0} { sum: 37.4 } Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  50. 50. © 2020 InfluxData. All rights reserved. 50 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 group_key = [ _field ] Final accumulator: { sum: 37.4 } identity: {sum: 0.0} fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  51. 51. © 2020 InfluxData. All rights reserved. 51 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 group_key = [ _field ] ● Final accumulator ● Columns in the group key ● Drops all other columns identity: {sum: 0.0} Output table fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  52. 52. © 2020 InfluxData. All rights reserved. 52 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 group_key = [ _field ] identity: {sum: 0.0} _field sum example 37.4 fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  53. 53. © 2020 InfluxData. All rights reserved. 53 _start _stop _time _measurement tag _field _value 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-01T00:00:00Z foo bar example 1.3 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-02T00:00:00Z foo bar example 12.2 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-03T00:00:00Z foo bar example 4.9 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-04T00:00:00Z foo bar example 3.2 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-05T00:00:00Z foo bar example 15.8 group_key = [ _start,_stop,_measurement,tag,_field ] identity: {sum: 0.0} fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  54. 54. © 2020 InfluxData. All rights reserved. 54 _start _stop _measurement tag _field sum 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z foo bar example 37.4 group_key = [ _start,_stop,_measurement,tag,_field ] identity: {_value: 0.0} fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  55. 55. © 2020 InfluxData. All rights reserved. 55 _start _stop _time _measurement tag _field _value 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-01T00:00:00Z foo bar example 1.3 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-02T00:00:00Z foo bar example 12.2 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-03T00:00:00Z foo bar example 4.9 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-04T00:00:00Z foo bar example 3.2 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-05T00:00:00Z foo bar example 15.8 group_key = [ _start,_stop,_measurement,tag,_field ] _start _stop _time _measurement tag _field _value 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-01T00:00:00Z foo baz example 8.1 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-02T00:00:00Z foo baz example 12.4 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-03T00:00:00Z foo baz example 13.7 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-04T00:00:00Z foo baz example 4.8 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-05T00:00:00Z foo baz example 6.5
  56. 56. © 2020 InfluxData. All rights reserved. 56 _start _stop _measurement tag _field sum 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z foo bar example 37.4 group_key = [ _start,_stop,_measurement,tag,_field ] _start _stop _measurement tag _field sum 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z foo baz example 45.5
  57. 57. © 2020 InfluxData. All rights reserved. 57 average
  58. 58. © 2020 InfluxData. All rights reserved. 58 average = (tables=<-) => tables |> reduce( )
  59. 59. © 2020 InfluxData. All rights reserved. 59 average = (tables=<-) => tables |> reduce( fn: (r, accumulator) => ({ }), )
  60. 60. © 2020 InfluxData. All rights reserved. 60 average = (tables=<-) => tables |> reduce( fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, }), )
  61. 61. © 2020 InfluxData. All rights reserved. 61 average = (tables=<-) => tables |> reduce( fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, }), )
  62. 62. © 2020 InfluxData. All rights reserved. 62 average = (tables=<-) => tables |> reduce( fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }), )
  63. 63. © 2020 InfluxData. All rights reserved. 63 average = (tables=<-) => tables |> reduce( fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }), identity: { sum: 0.0, count: 1.0, avg: 0.0 } )
  64. 64. © 2020 InfluxData. All rights reserved. 64 { sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0001, _field: "example", _value: 1.3} group_key = [ _field ] accumulator = {sum: 0.0, count: 1.0, avg: 0.0} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  65. 65. © 2020 InfluxData. All rights reserved. 65 { sum: 1.3 + 0.0, count: 1.0 + 1.0, avg: (1.3 + 0.0) / 1.0 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0001, _field: "example", _value: 1.3} group_key = [ _field ] accumulator = {sum: 0.0, count: 1.0, avg: 0.0} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  66. 66. © 2020 InfluxData. All rights reserved. 66 { sum: 1.3, count: 2.0, avg: 1.3 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0001, _field: "example", _value: 1.3} group_key = [ _field ] accumulator = {sum: 0.0, count: 1.0, avg: 0.0} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  67. 67. © 2020 InfluxData. All rights reserved. 67 { sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0002, _field: "example", _value: 12.2} group_key = [ _field ] accumulator = {sum: 1.3, count: 2.0, avg: 1.3} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  68. 68. © 2020 InfluxData. All rights reserved. 68 { sum: 12.2 + 1.3, count: 2.0 + 1.0, avg: (12.2 + 1.3) / 2.0 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0002, _field: "example", _value: 12.2} group_key = [ _field ] accumulator = {sum: 1.3, count: 2.0, avg: 1.3} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  69. 69. © 2020 InfluxData. All rights reserved. 69 { sum: 14.5, count: 3.0, avg: 6.75 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0002, _field: "example", _value: 12.2} group_key = [ _field ] accumulator = {sum: 1.3, count: 2.0, avg: 1.3} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  70. 70. © 2020 InfluxData. All rights reserved. 70 { sum: 18.4, count: 4.0, avg: 6.13 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0003, _field: "example", _value: 4.9} group_key = [ _field ] accumulator = {sum: 14.5, count: 3.0, avg: 6.75} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  71. 71. © 2020 InfluxData. All rights reserved. 71 { sum: 21.6, count: 5.0, avg: 5.4 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0004, _field: "example", _value: 3.2} group_key = [ _field ] accumulator = {sum: 18.4, count: 4.0, avg: 6.47} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  72. 72. © 2020 InfluxData. All rights reserved. 72 { sum: 37.4, count: 6.0, avg: 7.48 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0005, _field: "example", _value: 15.8} group_key = [ _field ] accumulator = {sum: 21.6, count: 5.0, avg: 5.4} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  73. 73. © 2020 InfluxData. All rights reserved. 73 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 group_key = [ _field ] fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 } Final accumulator: { sum: 37.4, count: 6.0, avg: 7.48 }
  74. 74. © 2020 InfluxData. All rights reserved. 74 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 group_key = [ _field ] fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 } _field sum count avg example 37.4 6.0 7.48
  75. 75. © 2020 InfluxData. All rights reserved. 75 average = (tables=<-) => tables |> reduce( fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }), identity: { sum: 0.0, count: 1.0, avg: 0.0 } ) |> drop(columns: ["sum","count"]) |> rename(columns: {avg: "_value"})
  76. 76. © 2020 InfluxData. All rights reserved. 76 _time tag _field _value 0001 foo example 98.3 0002 foo example 89.1 0003 foo example 72.9 0004 foo example 100.6 0005 foo example 210.4 _time tag _field _value 0001 bar example 56.1 0002 bar example 49.3 0003 bar example 67.6 0004 bar example 69.6 0005 bar example 131.4 tag _field _value foo example 114.26 tag _field _value bar example 74.8 |> average()
  77. 77. Example custom functions using reduce()
  78. 78. © 2020 InfluxData. All rights reserved. 78 minMaxMean = (tables=<-) => tables |> reduce( identity: {count: 0, sum: 0.0, min: 0.0, max: 0.0, mean:0.0}, fn: (r, accumulator) => ({ count: accumulator.count + 1, sum: r._value + accumulator.sum, min: if accumulator.count == 0 then r._value else if r._value < accumulator.min then r._value else accumulator.min, max: if accumulator.count == 0 then r._value else if r._value > accumulator.max then r._value else accumulator.max, mean: if accumulator.count == 0 then r._value else (r._value + accumulator.sum) / float(v: accumulator.count + 1) }) ) |> drop(columns: ["count", "sum"])
  79. 79. © 2020 InfluxData. All rights reserved. 79 candlestick = (tables=<-) => tables |> reduce( identity: { index: 0, first: 0.0, last: 0.0, max: 0.0, min: 0.0 }, fn: (r, accumulator) => ({ index: accumulator.index + 1, first: if accumulator.index == 0 then r._value else accumulator.first, last: r._value, max: if accumulator.index == 0 then r._value else if r._value > accumulator.max then r._value else accumulator.max, min: if accumulator.index == 0 then r._value else if r._value < accumulator.min then r._value else accumulator.min }) ) |> drop(columns: ["index"])
  80. 80. © 2020 InfluxData. All rights reserved. 80 import "strings" commaSeparatedList = (tables=<-) => tables |> reduce( identity: {values: ""}, fn: (r, accumulator) => ({ values: accumulator.values + "${string(v: r._value)}," }) ) |> map(fn: (r) => ({ r with values: strings.trimRight(v: r.values, cutset: ",") }))
  81. 81. © 2020 InfluxData. All rights reserved. 81 Common gotchas ● reduce() does not support the with operator. ● Columns not in the group key or explicitly mapped in the reduce operation are dropped. ● Mathematical operands must be of the same data type. ● reduce() is “destructive” and will only output a single row per table.
  82. 82. Contribute custom functions to Flux Contribute & Share
  83. 83. © 2020 InfluxData. All rights reserved. 83 Contribute to Flux 1. Fork and clone the Flux repo – github.com/influxdata/flux 2. Navigate to flux/stdlib/contrib and create a new directory with your GitHub username. flux/stdlib/contrib/sanderson 1. Create directories for each sub-package. flux/stdlib/contrib/sanderson/finance 4. Add a package definition file. flux/stdlib/contrib/sanderson/finance/finance.flux
  84. 84. © 2020 InfluxData. All rights reserved. 84 package finance
  85. 85. © 2020 InfluxData. All rights reserved. 85 package finance import "math" import "regexp"
  86. 86. © 2020 InfluxData. All rights reserved. 86 package finance import "math" import "regexp" option currency = "$"
  87. 87. © 2020 InfluxData. All rights reserved. 87 package finance import "math" import "regexp" option currency = "$" nyse = ["A","AA","AACG","AAL","AAMC","AAME" ... ]
  88. 88. © 2020 InfluxData. All rights reserved. 88 package finance import "math" import "regexp" option currency = "$" nyse = ["A","AA","AACG","AAL","AAMC","AAME" ... ] // Comments that explain what functions do and how they work candlestick = (tables=<-) => tables |> ... // Calculate profit and loss profitAndLoss = (tables=<-) => tables |> ... // Calculate the days until retirement retireIn = (tables=<-, goal) => tables |> ...
  89. 89. © 2020 InfluxData. All rights reserved. 89 5. Add tests for each function. flux/stdlib/contrib/sanderson/finance/retireIn_test.f lux Contribute to Flux
  90. 90. © 2020 InfluxData. All rights reserved. 90 package retireIn_test import "testing" import "contrib/sanderson/finance" inData= "...some annotated CSV..." outData = "...some more annotated CSV..." t_retireIn = (table=<-) => table |> range(start: 2020-06-01T00:00:00Z, stop: 2020-06-05T00:00:00Z) |> finance.retireIn(goal: 5000000) test _retireIn = () => ({ input: testing.loadStorage(csv: inData), want: testing.loadMem(csv: outData), fn: t_retireIn })
  91. 91. © 2020 InfluxData. All rights reserved. 91 6. Generate the standard library. make or go generate ./stdlib 6. Push your changes to GitHub. 7. Submit a pull request on the Flux repo. 8. Use your contributed package. Contribute to Flux 5. Add tests for each function. flux/stdlib/contrib/sanderson/finance/retireIn_test.f lux
  92. 92. © 2020 InfluxData. All rights reserved. 92 import "contrib/sanderson/finance" from(bucket: "financials") |> range(start: -2y) |> filter(fn: (r) => r._measurement =~ /assets|liabilities|stocks/) |> finance.retireIn(goal: 5000000)
  93. 93. © 2020 InfluxData. All rights reserved. 93 6. Generate the standard library. make or go generate ./stdlib 6. Push your changes to GitHub. 7. Submit a pull request on the Flux repo. 8. Use your contributed package. 9. Share your contribution. Contribute to Flux 5. Add tests for each function. flux/stdlib/contrib/sanderson/finance/retireIn_test.f lux
  94. 94. © 2020 InfluxData. All rights reserved. 94 Thank you! Email: scott@influxdata.com Github: @sanderson InfluxData Community: @scott InfluxDB Community Slack: @Scott Anderson

×