Successfully reported this slideshow.

Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021

0

Share

1 of 41
1 of 41

Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021

0

Share

Download to read offline

In this talk, Tim and Ryan will provide an InfluxDB roadmap and engineering update. This will also include what you can expect in the future in terms of InfluxDB and Flux capabilities.

In this talk, Tim and Ryan will provide an InfluxDB roadmap and engineering update. This will also include what you can expect in the future in terms of InfluxDB and Flux capabilities.

More Related Content

Similar to Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021

Related Books

Free with a 14 day trial from Scribd

See all

Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021

  1. 1. Ryan Betts, VP Engineering Tim Hall, VP Products InfluxDB Roadmap & Engineering Update
  2. 2. © 2021 InfluxData. All rights reserved. 2 | Agenda ● InfluxDB Enterprise ● InfluxData Community ● InfluxDB Cloud
  3. 3. Managed by you... InfluxDB Enterprise
  4. 4. © 2021 InfluxData. All rights reserved. 4 | ● 2x Growth ● Management Depth ● Deployment & Build Automation Team Growth and Security Posture ● SOC 2 Type II ● Penetration Testing ● Code Signing ● Digital Supply Chain
  5. 5. © 2021 InfluxData. All rights reserved. 5 | These Engineering Investments Enable... Delivery of new data access & processing while not significantly altering current deployment Continue the success of deployments managed by you
  6. 6. © 2021 InfluxData. All rights reserved. 6 |InfluxDB Enterprise 1.9 ● Flux ○ Upgraded from 0.65 -> 0.113 including feature advancements and improvements already available in InfluxDB Cloud ○ Query management controls ■ Place limits on resource usage ■ Log queries which exceed these limits ○ Performance Improvements ■ last(), min(), max() ● Expanded metrics for managing InfluxDB Enterprise at scale ○ data size on disk per measurement (also in 1.8.5) ○ cardinality per measurement ○ data written metrics by measurement and/or username ○ series creation metrics ● InfluxQL ○ Add WITH KEY syntax to show tag keys ○ Performance improvement for sorted merge iterator ● Other ○ Updated prometheus remote protocol ○ Enable security integration with Kapacitor via LDAP ○ Node labels: add key/value meta data to either meta nodes and/or data nodes. Note: There is no companion InfluxDB 1.9 OSS release paired with InfluxDB Enterprise 1.9 Also, 32-bit support is also no longer available.
  7. 7. © 2021 InfluxData. All rights reserved. 7 | Key Theme: Continue the success of your on-prem deployments ● Leverage Node Labels for shard ownership/placement ● Streamline backup process including object store targets ● Prevent shard creation outside designated time range ● Upgrade internal hashing algorithm to support FedRAMP/FIPS 140-2 ● Reduce frequency of OOM conditions via InfluxQL ○ Ex: Require time range Need something? Let’s Talk InfluxDB Enterprise 1.10 and beyond
  8. 8. © 2021 InfluxData. All rights reserved. 8 |Kapacitor 1.6 ● Goal: enable a path for Kapacitor users to continue to leverage that investment with InfluxDB 2.x ● Introduce Flux task engine to Kapacitor ○ Allow definition, deployment, scheduling and management of Flux-based tasks ○ Leverage the v2 API for tasks ○ Flux helper functions - ease conversion of TICKscript to Flux tasks ○ Add queryFlux node for TICKscripts ○ Alert Handler additions: Zenoss ● Backport Kapacitor Enterprise Security feature to OSS ○ Allows Kapacitor, when paired with InfluxDB Enterprise to leverage the same security store.
  9. 9. © 2021 InfluxData. All rights reserved. 9 |Chronograf 1.9 • Data Explorer – Set database context for meta queries such as DELETE/DROP – Filter fields • Dashboards – Flux-based template variables • LogViewer Improvements • Eliminate phantom query refresh • Enhance data precision via time in visualization • Enable time selector • Improve load performance • Option to hide histogram panel • Kapacitor Integration – Support configuration for ServiceNow, Big Pandas, and Zenoss, retire HipChat – Add pagination for TICKscripts • Chronograf HA – Allow for migration script to be run against secure etcd cluster – Improved integration with InfluxDB Enterprise for administrative functions
  10. 10. InfluxData Community
  11. 11. © 2021 InfluxData. All rights reserved. 11 | ● Due to limited address space, large databases cause InfluxDB to crash. ● What’s large? > 3GB ● Typically retention policies can be used to restrict the amount of data stored on these smaller footprint instances. ● Community to the rescue ○ A number of community members have collaborated on addressing this and building a 32-bit edition of InfluxDB to address this. ● Thanks to @fluffynukeit and @simonvetter ● Builds: https:/ /github.com/simonvetter/influxdb/ We do not plan on producing a 32-bit build for InfluxDB 2.x Community can build for this platform, if desired. InfluxDB 1.x OSS on 32-bit Systems
  12. 12. © 2021 InfluxData. All rights reserved. 12 | OSS 1.x → OSS 2.0 How to take the limo ride? IN-PLACE UPGRADE ● Updates Configuration ● Write Data Into 2.0 Buckets ● Create DBRP Mappings ● Migrate Users, Metadata ● Build TSI (Time-series index) ● Includes Docker-based installs as of 2.0.4 https:/ /docs.influxdata.com/influxdb/v2.0/upgrade/v1-to-v2/ INFLUXDB OSS 2.0
  13. 13. © 2021 InfluxData. All rights reserved. 13 | Continuous Queries (CQ) Replace with Flux Tasks. Leverage docs 1.x Supported Protocols (Graphite, CollectD, OpenTSDB, UDP, Prometheus Remote Write) Use Telegraf Kapacitor 1.x (Subscription API) Dual write to InfluxDB 2.0 and Kapacitor 1.x Chronograf / Grafana (Dashboards) Existing dashboards will continue to work with InfluxQL compatibility API Authentication Auth required by default Admin users Re-assess and manually create admin users in OSS 2.0 Platform Support Docker, Windows, ARM builds are available now! A Few Things To Watch For... INFLUXDB OSS 2.0 https:/ /docs.influxdata.com/influxdb/v2.0/upgrade/v1-to-v2/
  14. 14. © 2021 InfluxData. All rights reserved. 14 | Flux: TICKscript Compatibility Package ● Most TICKscript functions have similar counterparts in Flux. ● The tickscript package simplifies conversion of TICKscripts to InfluxDB 2.x tasks. ● import ts "contrib/bonitoo-io/tickscript" ● Provides a set of functions making conversion easier. ● Allows for creation of custom checks executed as Flux tasks and triggering notifications in InfluxDB v2.x
  15. 15. © 2021 InfluxData. All rights reserved. 15 | 2.0 OSS Alpha 2.1 ● User Interface ○ Add Notebooks support ○ Add Dashboard annotations ● Flux performance improvements ○ first/last ○ count/sum ○ window grouping ○ and more... ● Operational Improvements ○ Additional Query Log Levels ● Update to Go 1.16 Q3 ● API Improvement ○ Add/Standardize Pagination across APIs ● Functions: Flux ● User Interface ○ Editing Tasks w/ Notebooks ○ Telegraf Config enhancements ○ Static legends in Graphs ○ Secrets management ○ Custom base path ● Connection to InfluxDB Cloud ○ Status ● Continue Flux performance improvements based on feedback 2.x Q4 ● User Interface ○ InfluxDB Templates integration ○ Alert building w/ Notebooks ● Connection to InfluxDB Cloud ○ Data Replication ○ Remote Management ○ Single Login InfluxDB OSS 2.x 2022
  16. 16. © 2021 InfluxData. All rights reserved. 16 | In Nov 2020, we invited 3 community members to participate in PR reviews and more. ● Sven Rebhan - @srebhan, @DocLambda ● Thomas Casteleyn - @Hipska ● Paweł Żak - @zak-pawel, @p-zak Their contributions have been invaluable in terms of helping move the Telegraf project forward! Thank you!!! Telegraf Community Contributions
  17. 17. © 2021 InfluxData. All rights reserved. 17 | Telegraf Enhancements in 2020 – Dialect of Python Specification has details about the syntax and available functions – Execution cannot access file system, network, system resources – Use Cases: • Math operations • Logic operations • Some string operations • Add or remove tags/fields/metrics External Plugins via Execd Starlark Processor Telegraf Inputs Processors Aggregators Outputs Execd Execd Execd Execd External Input External Processor External Aggregator External Output
  18. 18. © 2021 InfluxData. All rights reserved. 18 |Telegraf 1.18.0 Released - March 17, 2021 New Inputs ● Beat Input Plugin ● CS:GO Input Plugin ● Directory Monitoring Input Plugin* ● RavenDB Input Plugin ● NFS Input Plugin* New Outputs ● Google BigQuery Output Plugin ● Grafana Loki Output Plugin ● Sensu Output Plugin ● SignalFX Output Plugin New External Plugins ● GeoIP Processor Plugin ● Plex Webhook Input Plugin ● SMCIPMITool Input Plugin New Aggregators ● Derivative Aggregator Plugin ● Quantile Aggregator Plugin New Processors ● AWS EC2 Metadata Processor Plugin New Parsers ● XML Parser Plugin New Serializers ● MessagePack Serializer Plugin *community requests New Features ● Updated to Go 1.16.2 ● Code Signing for Windows & Mac ● More SNMP v3 authentication protocols including SHA-512 ● 140+ commits from 78 contributors
  19. 19. © 2021 InfluxData. All rights reserved. 19 | JSON Parsing Improvements Telegraf: What’s Next? Prometheus Remote Write Parsing Dynamic Configuration
  20. 20. InfluxDB Cloud
  21. 21. © 2021 InfluxData. All rights reserved. 21 | Global Distribution Run where you are / Pay how you want Multi-Cloud + Global Distribution Now: US (Oregon) US(Virginia) EU(Frankfurt) US (Iowa) EU (Belgium) US (Virginia) EU (Netherlands)
  22. 22. © 2021 InfluxData. All rights reserved. 22 | Global Distribution Run where you are / Pay how you want Next: • Continue expansion across all super regions (N. America, Europe, APAC) – Next: AWS AP (Sydney) – Azure Europe (North) – AWS Europe (Ireland) – Other APAC* Multi-Cloud + Global Distribution Now: US (Oregon) US(Virginia) EU(Frankfurt) US (Iowa) EU (Belgium) US (Virginia) EU (Netherlands)
  23. 23. © 2021 InfluxData. All rights reserved. 23 | • Usage-based billing • Social sign-on integration with Google and Microsoft (Azure AD) • Eligible to retire CSP committed consumption Billing Options Cloud Service Provider Marketplace Offers Now: Later: Annual contracts
  24. 24. © 2021 InfluxData. All rights reserved. 24 | IoT: Geotemporal Available Now! Telegraf Plugins S2 Geo Processor: adds tags with an S2 cell ID token of a specified cell level. Specify lat and lon field values with WGS-84 coordinates in decimal degrees. GeoIP Processor: looks up IP addresses in the MaxMind GeoLite2 database and adds the respective ISO country code, city name, latitude and longitude.
  25. 25. © 2021 InfluxData. All rights reserved. 25 | IoT: Geotemporal Available Now! Filtering • filterRows() • gridFilter() • strictFilter() Aggregate • groupByArea() • asTracks() Transformation • s2CellIDToken() • toRows() Supported Shapes • box - defined by: minLat, maxLat, minLon, maxLon • circle - defined by: lat, lon, radius • polygon - array of points: lat, lon Telegraf Plugins S2 Geo Processor: adds tags with an S2 cell ID token of a specified cell level. Specify lat and lon field values with WGS-84 coordinates in decimal degrees. GeoIP Processor: looks up IP addresses in the MaxMind GeoLite2 database and adds the respective ISO country code, city name, latitude and longitude.
  26. 26. © 2021 InfluxData. All rights reserved. 26 | IoT: Geotemporal Available Now! Filtering • filterRows() • gridFilter() • strictFilter() Aggregate • groupByArea() • asTracks() Transformation • s2CellIDToken() • toRows() Supported Shapes • box - defined by: minLat, maxLat, minLon, maxLon • circle - defined by: lat, lon, radius • polygon - array of points: lat, lon Telegraf Plugins S2 Geo Processor: adds tags with an S2 cell ID token of a specified cell level. Specify lat and lon field values with WGS-84 coordinates in decimal degrees. GeoIP Processor: looks up IP addresses in the MaxMind GeoLite2 database and adds the respective ISO country code, city name, latitude and longitude. Map Visualisation
  27. 27. © 2021 InfluxData. All rights reserved. 27 | Key Themes in 2021 Teams & Collaboration Tenant Observability Functions & Reuse Ease Data Movement
  28. 28. © 2021 InfluxData. All rights reserved. 28 | Notebooks Teams & Collaboration ● Goal is to provide a single consistent building experience in our platform ● Designed to allow teams to incrementally build queries or background processing tasks together ● Flexible enough to guide new users and unblock power users ● View multiple related resources in a single view Now
  29. 29. © 2021 InfluxData. All rights reserved. 29 | Dashboard Annotations Teams & Collaboration ● Goal is share context in dashboards ● Add, edit, delete as needed ● Annotations never expire ● Available in your InfluxDB Cloud accounts soon ● Ranged annotations, custom colors, support for more graph types coming soon Now
  30. 30. © 2021 InfluxData. All rights reserved. 30 | Tenant Observability ● Goal is to provide self-service access to usage and operations: ○ Usage ○ Cardinality ○ Rate limit events ○ Write errors ○ Query Log ● User configurable templates for dashboards & alerts ● Available on all cloud accounts shortly Next
  31. 31. © 2021 InfluxData. All rights reserved. 31 | Evolution of Tasks Schedule Tasks: Flux Now
  32. 32. © 2021 InfluxData. All rights reserved. 32 | Evolution of Tasks Functions Schedule Trigger (via Events) Tasks: Flux Parameterized / Reusable Remote Invocation Task (Via Schedule) Web hook (via API) Flux-based...to Start Next
  33. 33. © 2021 InfluxData. All rights reserved. 33 | Functions: Flux ● Goal is code reusability ○ Define Flux code once ○ Register as a Function with defined parameters ○ Invoke, as desired, with different values supplied as arguments ● Power tasks, triggers, dashboard cells with common functions Next Task = Function (Schedule, Arguments)
  34. 34. © 2021 InfluxData. All rights reserved. 34 | Triggers ● Goal is to use events to invoke functions ● Events can be “write to a bucket” ● Use cases: ○ Late arrivals of data ○ Streamlined alerting Next QUERY ACT Scheduled Task / Alert check Notificatio n systems WRITE Notificatio n systems WRITE WRITE ACT Triggers
  35. 35. © 2021 InfluxData. All rights reserved. 35 | Functions: Polyglot Programming ● Goal is make the platform do more for you ● Two main use cases: ○ Pre-process data before writing into InfluxDB ○ Post-process rows returned from querying InfluxDB ● MVP - Run your python code as function that can be invoked remotely Next Python code as a Function
  36. 36. © 2021 InfluxData. All rights reserved. 36 |Make data movement easier Enable users to define object store locations within their CSP account. Bulk Import Line protocol CSV TSM files Parquet Next
  37. 37. © 2021 InfluxData. All rights reserved. 37 |Make data movement easier Enable users to define object store locations within their CSP account. Bulk Import Line protocol CSV TSM files Parquet Self-Service Backup Bucket backup (restore to OSS or Cloud) -- or -- Export to Parquet Next
  38. 38. © 2021 InfluxData. All rights reserved. 38 |Make data movement easier Enable users to define object store locations within their CSP account. Bulk Import Line protocol CSV TSM files Parquet Self-Service Backup Bucket backup (restore to OSS or Cloud) -- or -- Export to Parquet Next Cloud Native Ingest Connect with cloud native messaging services • MQTT, • AWS Kinesis • Azure EventHub
  39. 39. © 2021 InfluxData. All rights reserved. 39 |Bonus: Explicit Schema Define Buckets with Explicit Schema ● Blocks non-compliant writes ● Allow for additions Next
  40. 40. © 2021 InfluxData. All rights reserved. 40 | InfluxDB Enterprise ○ Continue the success of deployments managed by you ○ Deliver latest improvements for data access & processing InfluxData Community ○ Limo door is open...and ready to take you on the ride to InfluxDB 2.x ○ Telegraf advancements makes it the most powerful data acquisition tool ever InfluxDB Cloud ○ Teams & Collaboration ○ Functions & Reuse ○ Tenant Observability ○ Ease Data Movement Summary
  41. 41. Questions

×