Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scott Anderson [InfluxData] | Map & Reduce – The Powerhouses of Custom Flux Functions | InfluxDays Virtual Experience London 2020

30 views

Published on

Use the Flux `map` and `reduce` functions to create powerful custom Flux functions. Aggregate, calculate, extrapolate and transform your data to gain valuable insights specific to your use case.

Published in: Technology
  • DOWNLOAD THE BOOK INTO AVAILABLE FORMAT (New Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THE can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THE is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBOOK .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookBOOK, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, EBOOK, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THE Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THE the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THE Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Scott Anderson [InfluxData] | Map & Reduce – The Powerhouses of Custom Flux Functions | InfluxDays Virtual Experience London 2020

  1. 1. Scott Anderson Technical Writer @ InfluxData Map & Reduce The Powerhouses of Custom Flux Functions
  2. 2. © 2020 InfluxData. All rights reserved. 2 Disclaimer This presentation is not about Hadoop MapReduce-like functionality in Flux. This is about building custom functions with map() and reduce().
  3. 3. © 2020 InfluxData. All rights reserved. 3 Introduction to Flux vimeo.com/377375687
  4. 4. A quick background Flux
  5. 5. © 2020 InfluxData. All rights reserved. 5 “ Flux is a functional data scripting language that lets you query, process, write, and alert on data in a single syntax. ”
  6. 6. Design Goals ● Turing-complete ● Usable ● Readable ● Flexible ● Extensible ● Testable ● Contributable ● Shareable
  7. 7. Design Goals ● Turing-complete ● Usable ● Readable ● Flexible ● Extensible ● Testable ● Contributable ● Shareable
  8. 8. Design Goals ● Turing-complete ● Usable ● Readable ● Flexible ● Extensible ● Testable ● Contributable ● Shareable
  9. 9. © 2020 InfluxData. All rights reserved. 9 Extensibility ● InfluxQL was and is hard to extend. ● TICKscript was and is easier to extend, but depends on external languages for true customization.
  10. 10. © 2020 InfluxData. All rights reserved. 10 Extensibility ● Early strategy for Flux has been to create “primitive” functions that can be used to create other functions. ● Flux recently introduced architecture and guidelines for community contributions.
  11. 11. If it doesn’t exist, create it Custom Functions
  12. 12. © 2020 InfluxData. All rights reserved. 12 Custom Function Definition funcName = (parameters) => funcOperations Function square = (x) => x * x Transformation somethingCool = (tables=<-) => tables |> ...
  13. 13. © 2020 InfluxData. All rights reserved. 13 Functions & Transformations Function bool(v: 1) // Returns true Transformation toBool = (tables=<-) => tables |> map(fn: (r) => ({ r with _value: bool(v: r._value) }))
  14. 14. © 2020 InfluxData. All rights reserved. 14 Functions & Transformations from(bucket: "myData") |> range(start: -1h) |> filter(fn: (r) => r._field == "enabled") |> toBool()
  15. 15. © 2020 InfluxData. All rights reserved. 15 map() & reduce()
  16. 16. Remap rows fer’days map()
  17. 17. © 2020 InfluxData. All rights reserved. 17 _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempC 21.29 2020-06-01T00:12:01Z TM02001 tempC 21.36 2020-06-01T00:12:02Z TM02001 tempC 21.34
  18. 18. © 2020 InfluxData. All rights reserved. 18 _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempC 21.29 { _time: 2020-06-01T00:12:00Z, sensorID: "TM02001", _field: "tempC", _value: 21.29 }
  19. 19. © 2020 InfluxData. All rights reserved. 19 map( fn: ) (r) => ({ ... }) )
  20. 20. © 2020 InfluxData. All rights reserved. 20 (r) => ({ ... }) r = { _time: 2020-06-01T00:12:00Z, sensorID: "TM02001", _field: "tempC", _value: 21.29 }
  21. 21. © 2020 InfluxData. All rights reserved. 21 r = { _time: 2020-06-01T00:12:00Z, sensorID: "TM02001", _field: "tempC", _value: 21.29 } (r) => ({ _value: r._value })
  22. 22. © 2020 InfluxData. All rights reserved. 22 r = { _time: 2020-06-01T00:12:00Z, sensorID: "TM02001", _field: "tempC", _value: 21.29 } map(fn: (r) => ({ _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 }))
  23. 23. © 2020 InfluxData. All rights reserved. 23 { _field: "tempF", _value: 70.322 } map(fn: (r) => ({ _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 })) _field _value tempF 70.322
  24. 24. © 2020 InfluxData. All rights reserved. 24 { _time: 2020-06-01T00:12:00Z, sensorID: "TM02001", _field: "tempF", _value: 70.322 } map(fn: (r) => ({ r with _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 }))
  25. 25. © 2020 InfluxData. All rights reserved. 25 map(fn: (r) => ({ r with _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 })) _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempF 70.322
  26. 26. © 2020 InfluxData. All rights reserved. 26 |> map(fn: (r) => ({ r with _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 })) _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempC 21.29 2020-06-01T00:12:01Z TM02001 tempC 21.36 2020-06-01T00:12:02Z TM02001 tempC 21.34 _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempF 70.322 2020-06-01T00:12:01Z TM02001 tempF 70.448 2020-06-01T00:12:02Z TM02001 tempF 70.412
  27. 27. © 2020 InfluxData. All rights reserved. 27 toF Create a custom transformation toF = (tables=<-)toF = (tables=<-) => tables toF = (tables=<-) => tables |> map( fn: (r) => ({ r with _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 }) )
  28. 28. © 2020 InfluxData. All rights reserved. 28 |> map(fn: (r) => ({ r with _field: "tempF", _value: (r._value * 9.0 / 5.0) + 32.0 })) _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempC 21.29 2020-06-01T00:12:01Z TM02001 tempC 21.36 2020-06-01T00:12:02Z TM02001 tempC 21.34 _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempF 70.322 2020-06-01T00:12:01Z TM02001 tempF 70.448 2020-06-01T00:12:02Z TM02001 tempF 70.412 |> toF()
  29. 29. © 2020 InfluxData. All rights reserved. 29 Temperature Range Temperature Text below 0°C freezing 0°C - 15.5°C cold 15.5°C - 26.5°C mild 26.5°C - 37.5°C hot 37.5°C or higher very hot Add text based on temperature range?
  30. 30. © 2020 InfluxData. All rights reserved. 30 (r) => ({ column: if r._value == "foo" then "bar" else "baz" }) Use conditional logic!
  31. 31. © 2020 InfluxData. All rights reserved. 31 tempToText = (tables=<-) => tables |> map(fn: (r) => ({ r with }) ) tempToText = (tables=<-) => tables |> map(fn: (r) => ({ r with temp_text: }) ) tempToText = (tables=<-) => tables |> map(fn: (r) => ({ r with temp_text: if r._value <= 0.0 then "freezing" }) ) tempToText = (tables=<-) => tables |> map(fn: (r) => ({ r with temp_text: if r._value <= 0.0 then "freezing" else if r._value > 0.0 and r._value <= 15.5 then "cold" }) ) tempToText = (tables=<-) => tables |> map(fn: (r) => ({ r with temp_text: if r._value <= 0.0 then "freezing" else if r._value > 0.0 and r._value <= 15.5 then "cold" else if r._value > 15.5 and r._value <= 26.5 then "mild" else if r._value > 26.5 and r._value <= 37.5 then "hot" else if r._value > 37.5 then "very hot" }) ) tempToText = (tables=<-) => tables |> map(fn: (r) => ({ r with temp_text: if r._value <= 0.0 then "freezing" else if r._value > 0.0 and r._value <= 15.5 then "cold" else if r._value > 15.5 and r._value <= 26.5 then "mild" else if r._value > 26.5 and r._value <= 37.5 then "hot" else if r._value > 37.5 then "very hot" else "unknown" }) )
  32. 32. © 2020 InfluxData. All rights reserved. 32 |> tempToText() _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempC -1.2 2020-06-01T00:12:01Z TM02001 tempC 22.1 2020-06-01T00:12:02Z TM02001 tempC 31.8 _time sensorID _field _value temp_text 2020-06-01T00:12:00Z TM02001 tempC -1.2 freezing 2020-06-01T00:12:01Z TM02001 tempC 22.1 mild 2020-06-01T00:12:02Z TM02001 tempC 31.8 hot
  33. 33. Example custom functions using map()
  34. 34. © 2020 InfluxData. All rights reserved. 34 Basic mathematical operations multiplyByX = (tables=<-, x) => tables |> map(fn: (r) => ({ r with _value: r._value * x })) square = (tables=<-) => tables |> map(fn: (r) => ({ r with _value: r._value * r._value })) quarter = (tables=<-) => tables |> map(fn: (r) => ({ r with _value: r._value / 4.0 }))
  35. 35. © 2020 InfluxData. All rights reserved. 35 Other mathematical operations import "math" sin = (tables=<-) => tables |> map(fn: (r) => ({ r with _value: math.sin(x: r._value) })) circumference = (tables=<-) => tables |> map(fn: (r) => ({ r with circumference: 2.0 * math.pi * r._value })) circularArea = (tables=<-) => tables |> map(fn: (r) => ({ r with area: math.pi * r._value ^ 2.0 }))
  36. 36. © 2020 InfluxData. All rights reserved. 36 String manipulationimport "strings" rmHostPrefix = (tables=<-) => tables |> map(fn: (r) => ({ r with host: strings.trimPrefix(v: r.host, prefix: "host_") })) replaceSpaces = (tables=<-, char="_") => tables |> map(fn: (r) => ({ r with _value: strings.replaceAll(v: r._value, t: " ", u: char) })) humanReadableMessage = (tables=<-) => tables |> map(fn: (r) => ({ r with humanReadable: "The current value is ${string(v: r._value)}." }))
  37. 37. © 2020 InfluxData. All rights reserved. 37 Common gotchas ● The with operator is your friend. ● Mathematical operands must be of the same data type. ● map() does not add new columns to group keys. ● map() only operates on a single row at a time. ○ Can’t use values from previous or subsequent rows. ○ If your operation requires values from separate fields, use pivot() or join() to align field values in rows.
  38. 38. Reduce all the things!! reduce()
  39. 39. © 2020 InfluxData. All rights reserved. 39 |> reduce(...)
  40. 40. © 2020 InfluxData. All rights reserved. 40 _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 tempC 21.29 2020-06-01T00:12:01Z TM02001 tempC 21.36 2020-06-01T00:12:02Z TM02001 tempC 21.34 sensorID _field _value TM02001 tempC 21.33
  41. 41. © 2020 InfluxData. All rights reserved. 41 reduce( fn: identity: )
  42. 42. © 2020 InfluxData. All rights reserved. 42 reduce( fn: (r, accumulator) => ({ ... }), identity: )
  43. 43. © 2020 InfluxData. All rights reserved. 43 reduce( fn: (r, accumulator) => ({ ... }), identity: { ... } )
  44. 44. © 2020 InfluxData. All rights reserved. 44 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0001, _field: "example", _value: 1.3} group_key = [ _field ] fn: (r, accumulator) => ({ sum: r._value + accumulator.sum }) accumulator = {sum: 0.0} identity: {sum: 0.0}
  45. 45. © 2020 InfluxData. All rights reserved. 45 { sum: r._value + accumulator.sum } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0001, _field: "example", _value: 1.3} group_key = [ _field ] accumulator = {sum: 0.0} identity: {sum: 0.0} { sum: 1.3 + 0.0 }{ sum: 1.3 } Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  46. 46. © 2020 InfluxData. All rights reserved. 46 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0002, _field: "example", _value: 12.2} group_key = [ _field ] accumulator = {sum: 1.3} identity: {sum: 0.0} { sum: 12.2 + 1.3 }{ sum: 13.5 } Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  47. 47. © 2020 InfluxData. All rights reserved. 47 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0003, _field: "example", _value: 4.9} group_key = [ _field ] fn: (r, accumulator) => ({ sum: r._value + accumulator.sum }) accumulator = {sum: 13.5} identity: {sum: 0.0} { sum: 18.4 } Returns:
  48. 48. © 2020 InfluxData. All rights reserved. 48 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0003, _field: "example", _value: 3.2} group_key = [ _field ] accumulator = {sum: 18.4} identity: {sum: 0.0} { sum: 21.6 } Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  49. 49. © 2020 InfluxData. All rights reserved. 49 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0003, _field: "example", _value: 15.8} group_key = [ _field ] accumulator = {sum: 21.6} identity: {sum: 0.0} { sum: 37.4 } Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  50. 50. © 2020 InfluxData. All rights reserved. 50 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 group_key = [ _field ] Final accumulator: { sum: 37.4 } identity: {sum: 0.0} fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  51. 51. © 2020 InfluxData. All rights reserved. 51 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 group_key = [ _field ] ● Final accumulator ● Columns in the group key ● Drops all other columns identity: {sum: 0.0} Output table fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  52. 52. © 2020 InfluxData. All rights reserved. 52 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 group_key = [ _field ] identity: {sum: 0.0} _field sum example 37.4 fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  53. 53. © 2020 InfluxData. All rights reserved. 53 _start _stop _time _measurement tag _field _value 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-01T00:00:00Z foo bar example 1.3 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-02T00:00:00Z foo bar example 12.2 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-03T00:00:00Z foo bar example 4.9 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-04T00:00:00Z foo bar example 3.2 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-05T00:00:00Z foo bar example 15.8 group_key = [ _start,_stop,_measurement,tag,_field ] identity: {sum: 0.0} fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  54. 54. © 2020 InfluxData. All rights reserved. 54 _start _stop _measurement tag _field sum 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z foo bar example 37.4 group_key = [ _start,_stop,_measurement,tag,_field ] identity: {_value: 0.0} fn: (r, accumulator) => ({ sum: r._value + accumulator.sum })
  55. 55. © 2020 InfluxData. All rights reserved. 55 _start _stop _time _measurement tag _field _value 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-01T00:00:00Z foo bar example 1.3 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-02T00:00:00Z foo bar example 12.2 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-03T00:00:00Z foo bar example 4.9 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-04T00:00:00Z foo bar example 3.2 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-05T00:00:00Z foo bar example 15.8 group_key = [ _start,_stop,_measurement,tag,_field ] _start _stop _time _measurement tag _field _value 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-01T00:00:00Z foo baz example 8.1 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-02T00:00:00Z foo baz example 12.4 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-03T00:00:00Z foo baz example 13.7 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-04T00:00:00Z foo baz example 4.8 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z 2020-06-05T00:00:00Z foo baz example 6.5
  56. 56. © 2020 InfluxData. All rights reserved. 56 _start _stop _measurement tag _field sum 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z foo bar example 37.4 group_key = [ _start,_stop,_measurement,tag,_field ] _start _stop _measurement tag _field sum 2020-06-01T00:00:00Z 2020-06-05T23:59:59Z foo baz example 45.5
  57. 57. © 2020 InfluxData. All rights reserved. 57 average
  58. 58. © 2020 InfluxData. All rights reserved. 58 average = (tables=<-) => tables |> reduce( )
  59. 59. © 2020 InfluxData. All rights reserved. 59 average = (tables=<-) => tables |> reduce( fn: (r, accumulator) => ({ }), )
  60. 60. © 2020 InfluxData. All rights reserved. 60 average = (tables=<-) => tables |> reduce( fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, }), )
  61. 61. © 2020 InfluxData. All rights reserved. 61 average = (tables=<-) => tables |> reduce( fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, }), )
  62. 62. © 2020 InfluxData. All rights reserved. 62 average = (tables=<-) => tables |> reduce( fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }), )
  63. 63. © 2020 InfluxData. All rights reserved. 63 average = (tables=<-) => tables |> reduce( fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }), identity: { sum: 0.0, count: 1.0, avg: 0.0 } )
  64. 64. © 2020 InfluxData. All rights reserved. 64 { sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0001, _field: "example", _value: 1.3} group_key = [ _field ] accumulator = {sum: 0.0, count: 1.0, avg: 0.0} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  65. 65. © 2020 InfluxData. All rights reserved. 65 { sum: 1.3 + 0.0, count: 1.0 + 1.0, avg: (1.3 + 0.0) / 1.0 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0001, _field: "example", _value: 1.3} group_key = [ _field ] accumulator = {sum: 0.0, count: 1.0, avg: 0.0} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  66. 66. © 2020 InfluxData. All rights reserved. 66 { sum: 1.3, count: 2.0, avg: 1.3 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0001, _field: "example", _value: 1.3} group_key = [ _field ] accumulator = {sum: 0.0, count: 1.0, avg: 0.0} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  67. 67. © 2020 InfluxData. All rights reserved. 67 { sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0002, _field: "example", _value: 12.2} group_key = [ _field ] accumulator = {sum: 1.3, count: 2.0, avg: 1.3} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  68. 68. © 2020 InfluxData. All rights reserved. 68 { sum: 12.2 + 1.3, count: 2.0 + 1.0, avg: (12.2 + 1.3) / 2.0 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0002, _field: "example", _value: 12.2} group_key = [ _field ] accumulator = {sum: 1.3, count: 2.0, avg: 1.3} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  69. 69. © 2020 InfluxData. All rights reserved. 69 { sum: 14.5, count: 3.0, avg: 6.75 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0002, _field: "example", _value: 12.2} group_key = [ _field ] accumulator = {sum: 1.3, count: 2.0, avg: 1.3} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  70. 70. © 2020 InfluxData. All rights reserved. 70 { sum: 18.4, count: 4.0, avg: 6.13 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0003, _field: "example", _value: 4.9} group_key = [ _field ] accumulator = {sum: 14.5, count: 3.0, avg: 6.75} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  71. 71. © 2020 InfluxData. All rights reserved. 71 { sum: 21.6, count: 5.0, avg: 5.4 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0004, _field: "example", _value: 3.2} group_key = [ _field ] accumulator = {sum: 18.4, count: 4.0, avg: 6.47} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  72. 72. © 2020 InfluxData. All rights reserved. 72 { sum: 37.4, count: 6.0, avg: 7.48 } _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 r = {_time: 0005, _field: "example", _value: 15.8} group_key = [ _field ] accumulator = {sum: 21.6, count: 5.0, avg: 5.4} Returns: fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 }
  73. 73. © 2020 InfluxData. All rights reserved. 73 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 group_key = [ _field ] fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 } Final accumulator: { sum: 37.4, count: 6.0, avg: 7.48 }
  74. 74. © 2020 InfluxData. All rights reserved. 74 _time _field _value 0001 example 1.3 0002 example 12.2 0003 example 4.9 0004 example 3.2 0005 example 15.8 group_key = [ _field ] fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }) identity: { sum: 0.0, count: 1.0, avg: 0.0 } _field sum count avg example 37.4 6.0 7.48
  75. 75. © 2020 InfluxData. All rights reserved. 75 average = (tables=<-) => tables |> reduce( fn: (r, accumulator) => ({ sum: r._value + accumulator.sum, count: accumulator.count + 1.0, avg: (r._value + accumulator.sum) / accumulator.count }), identity: { sum: 0.0, count: 1.0, avg: 0.0 } ) |> drop(columns: ["sum","count"]) |> rename(columns: {avg: "_value"})
  76. 76. © 2020 InfluxData. All rights reserved. 76 _time tag _field _value 0001 foo example 98.3 0002 foo example 89.1 0003 foo example 72.9 0004 foo example 100.6 0005 foo example 210.4 _time tag _field _value 0001 bar example 56.1 0002 bar example 49.3 0003 bar example 67.6 0004 bar example 69.6 0005 bar example 131.4 tag _field _value foo example 114.26 tag _field _value bar example 74.8 |> average()
  77. 77. Example custom functions using reduce()
  78. 78. © 2020 InfluxData. All rights reserved. 78 minMaxMean = (tables=<-) => tables |> reduce( identity: {count: 0, sum: 0.0, min: 0.0, max: 0.0, mean:0.0}, fn: (r, accumulator) => ({ count: accumulator.count + 1, sum: r._value + accumulator.sum, min: if accumulator.count == 0 then r._value else if r._value < accumulator.min then r._value else accumulator.min, max: if accumulator.count == 0 then r._value else if r._value > accumulator.max then r._value else accumulator.max, mean: if accumulator.count == 0 then r._value else (r._value + accumulator.sum) / float(v: accumulator.count + 1) }) ) |> drop(columns: ["count", "sum"])
  79. 79. © 2020 InfluxData. All rights reserved. 79 candlestick = (tables=<-) => tables |> reduce( identity: { index: 0, first: 0.0, last: 0.0, max: 0.0, min: 0.0 }, fn: (r, accumulator) => ({ index: accumulator.index + 1, first: if accumulator.index == 0 then r._value else accumulator.first, last: r._value, max: if accumulator.index == 0 then r._value else if r._value > accumulator.max then r._value else accumulator.max, min: if accumulator.index == 0 then r._value else if r._value < accumulator.min then r._value else accumulator.min }) ) |> drop(columns: ["index"])
  80. 80. © 2020 InfluxData. All rights reserved. 80 import "strings" commaSeparatedList = (tables=<-) => tables |> reduce( identity: {values: ""}, fn: (r, accumulator) => ({ values: accumulator.values + "${string(v: r._value)}," }) ) |> map(fn: (r) => ({ r with values: strings.trimRight(v: r.values, cutset: ",") }))
  81. 81. © 2020 InfluxData. All rights reserved. 81 Common gotchas ● reduce() does not support the with operator. ● Columns not in the group key or explicitly mapped in the reduce operation are dropped. ● Mathematical operands must be of the same data type. ● reduce() is “destructive” and will only output a single row per table.
  82. 82. Contribute custom functions to Flux Contribute & Share
  83. 83. © 2020 InfluxData. All rights reserved. 83 Contribute to Flux 1. Fork and clone the Flux repo – github.com/influxdata/flux 2. Navigate to flux/stdlib/contrib and create a new directory with your GitHub username. flux/stdlib/contrib/sanderson 1. Create directories for each sub-package. flux/stdlib/contrib/sanderson/finance 4. Add a package definition file. flux/stdlib/contrib/sanderson/finance/finance.flux
  84. 84. © 2020 InfluxData. All rights reserved. 84 package finance
  85. 85. © 2020 InfluxData. All rights reserved. 85 package finance import "math" import "regexp"
  86. 86. © 2020 InfluxData. All rights reserved. 86 package finance import "math" import "regexp" option currency = "$"
  87. 87. © 2020 InfluxData. All rights reserved. 87 package finance import "math" import "regexp" option currency = "$" nyse = ["A","AA","AACG","AAL","AAMC","AAME" ... ]
  88. 88. © 2020 InfluxData. All rights reserved. 88 package finance import "math" import "regexp" option currency = "$" nyse = ["A","AA","AACG","AAL","AAMC","AAME" ... ] // Comments that explain what functions do and how they work candlestick = (tables=<-) => tables |> ... // Calculate profit and loss profitAndLoss = (tables=<-) => tables |> ... // Calculate the days until retirement retireIn = (tables=<-, goal) => tables |> ...
  89. 89. © 2020 InfluxData. All rights reserved. 89 5. Add tests for each function. flux/stdlib/contrib/sanderson/finance/retireIn_test.f lux Contribute to Flux
  90. 90. © 2020 InfluxData. All rights reserved. 90 package retireIn_test import "testing" import "contrib/sanderson/finance" inData= "...some annotated CSV..." outData = "...some more annotated CSV..." t_retireIn = (table=<-) => table |> range(start: 2020-06-01T00:00:00Z, stop: 2020-06-05T00:00:00Z) |> finance.retireIn(goal: 5000000) test _retireIn = () => ({ input: testing.loadStorage(csv: inData), want: testing.loadMem(csv: outData), fn: t_retireIn })
  91. 91. © 2020 InfluxData. All rights reserved. 91 6. Generate the standard library. make or go generate ./stdlib 6. Push your changes to GitHub. 7. Submit a pull request on the Flux repo. 8. Use your contributed package. Contribute to Flux 5. Add tests for each function. flux/stdlib/contrib/sanderson/finance/retireIn_test.f lux
  92. 92. © 2020 InfluxData. All rights reserved. 92 import "contrib/sanderson/finance" from(bucket: "financials") |> range(start: -2y) |> filter(fn: (r) => r._measurement =~ /assets|liabilities|stocks/) |> finance.retireIn(goal: 5000000)
  93. 93. © 2020 InfluxData. All rights reserved. 93 6. Generate the standard library. make or go generate ./stdlib 6. Push your changes to GitHub. 7. Submit a pull request on the Flux repo. 8. Use your contributed package. 9. Share your contribution. Contribute to Flux 5. Add tests for each function. flux/stdlib/contrib/sanderson/finance/retireIn_test.f lux
  94. 94. © 2020 InfluxData. All rights reserved. 94 Thank you! Email: scott@influxdata.com Github: @sanderson InfluxData Community: @scott InfluxDB Community Slack: @Scott Anderson

×