Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021

1. Ryan Betts, VP Engineering Tim Hall, VP Products InﬂuxDB Roadmap & Engineering Update

3. Managed by you... InﬂuxDB Enterprise

4. © 2021 InﬂuxData. All rights reserved. 4 | ● 2x Growth ● Management Depth ● Deployment & Build Automation Team Growth and Security Posture ● SOC 2 Type II ● Penetration Testing ● Code Signing ● Digital Supply Chain

5. © 2021 InﬂuxData. All rights reserved. 5 | These Engineering Investments Enable... Delivery of new data access & processing while not signiﬁcantly altering current deployment Continue the success of deployments managed by you

6. © 2021 InfluxData. All rights reserved. 6 |InfluxDB Enterprise 1.9 ● Flux ○ Upgraded from 0.65 -> 0.113 including feature advancements and improvements already available in InfluxDB Cloud ○ Query management controls ■ Place limits on resource usage ■ Log queries which exceed these limits ○ Performance Improvements ■ last(), min(), max() ● Expanded metrics for managing InfluxDB Enterprise at scale ○ data size on disk per measurement (also in 1.8.5) ○ cardinality per measurement ○ data written metrics by measurement and/or username ○ series creation metrics ● InfluxQL ○ Add WITH KEY syntax to show tag keys ○ Performance improvement for sorted merge iterator ● Other ○ Updated prometheus remote protocol ○ Enable security integration with Kapacitor via LDAP ○ Node labels: add key/value meta data to either meta nodes and/or data nodes. Note: There is no companion InfluxDB 1.9 OSS release paired with InfluxDB Enterprise 1.9 Also, 32-bit support is also no longer available.

7. © 2021 InfluxData. All rights reserved. 7 | Key Theme: Continue the success of your on-prem deployments ● Leverage Node Labels for shard ownership/placement ● Streamline backup process including object store targets ● Prevent shard creation outside designated time range ● Upgrade internal hashing algorithm to support FedRAMP/FIPS 140-2 ● Reduce frequency of OOM conditions via InfluxQL ○ Ex: Require time range Need something? Let’s Talk InfluxDB Enterprise 1.10 and beyond

8. © 2021 InfluxData. All rights reserved. 8 |Kapacitor 1.6 ● Goal: enable a path for Kapacitor users to continue to leverage that investment with InfluxDB 2.x ● Introduce Flux task engine to Kapacitor ○ Allow definition, deployment, scheduling and management of Flux-based tasks ○ Leverage the v2 API for tasks ○ Flux helper functions - ease conversion of TICKscript to Flux tasks ○ Add queryFlux node for TICKscripts ○ Alert Handler additions: Zenoss ● Backport Kapacitor Enterprise Security feature to OSS ○ Allows Kapacitor, when paired with InfluxDB Enterprise to leverage the same security store.

9. © 2021 InfluxData. All rights reserved. 9 |Chronograf 1.9 • Data Explorer – Set database context for meta queries such as DELETE/DROP – Filter fields • Dashboards – Flux-based template variables • LogViewer Improvements • Eliminate phantom query refresh • Enhance data precision via time in visualization • Enable time selector • Improve load performance • Option to hide histogram panel • Kapacitor Integration – Support configuration for ServiceNow, Big Pandas, and Zenoss, retire HipChat – Add pagination for TICKscripts • Chronograf HA – Allow for migration script to be run against secure etcd cluster – Improved integration with InfluxDB Enterprise for administrative functions

10. InﬂuxData Community

11. © 2021 InfluxData. All rights reserved. 11 | ● Due to limited address space, large databases cause InfluxDB to crash. ● What’s large? > 3GB ● Typically retention policies can be used to restrict the amount of data stored on these smaller footprint instances. ● Community to the rescue ○ A number of community members have collaborated on addressing this and building a 32-bit edition of InfluxDB to address this. ● Thanks to @fluffynukeit and @simonvetter ● Builds: https:/ /github.com/simonvetter/influxdb/ We do not plan on producing a 32-bit build for InfluxDB 2.x Community can build for this platform, if desired. InfluxDB 1.x OSS on 32-bit Systems

12. © 2021 InfluxData. All rights reserved. 12 | OSS 1.x → OSS 2.0 How to take the limo ride? IN-PLACE UPGRADE ● Updates Configuration ● Write Data Into 2.0 Buckets ● Create DBRP Mappings ● Migrate Users, Metadata ● Build TSI (Time-series index) ● Includes Docker-based installs as of 2.0.4 https:/ /docs.influxdata.com/influxdb/v2.0/upgrade/v1-to-v2/ INFLUXDB OSS 2.0

13. © 2021 InfluxData. All rights reserved. 13 | Continuous Queries (CQ) Replace with Flux Tasks. Leverage docs 1.x Supported Protocols (Graphite, CollectD, OpenTSDB, UDP, Prometheus Remote Write) Use Telegraf Kapacitor 1.x (Subscription API) Dual write to InfluxDB 2.0 and Kapacitor 1.x Chronograf / Grafana (Dashboards) Existing dashboards will continue to work with InfluxQL compatibility API Authentication Auth required by default Admin users Re-assess and manually create admin users in OSS 2.0 Platform Support Docker, Windows, ARM builds are available now! A Few Things To Watch For... INFLUXDB OSS 2.0 https:/ /docs.influxdata.com/influxdb/v2.0/upgrade/v1-to-v2/

14. © 2021 InfluxData. All rights reserved. 14 | Flux: TICKscript Compatibility Package ● Most TICKscript functions have similar counterparts in Flux. ● The tickscript package simplifies conversion of TICKscripts to InfluxDB 2.x tasks. ● import ts "contrib/bonitoo-io/tickscript" ● Provides a set of functions making conversion easier. ● Allows for creation of custom checks executed as Flux tasks and triggering notifications in InfluxDB v2.x

15. © 2021 InfluxData. All rights reserved. 15 | 2.0 OSS Alpha 2.1 ● User Interface ○ Add Notebooks support ○ Add Dashboard annotations ● Flux performance improvements ○ first/last ○ count/sum ○ window grouping ○ and more... ● Operational Improvements ○ Additional Query Log Levels ● Update to Go 1.16 Q3 ● API Improvement ○ Add/Standardize Pagination across APIs ● Functions: Flux ● User Interface ○ Editing Tasks w/ Notebooks ○ Telegraf Config enhancements ○ Static legends in Graphs ○ Secrets management ○ Custom base path ● Connection to InfluxDB Cloud ○ Status ● Continue Flux performance improvements based on feedback 2.x Q4 ● User Interface ○ InfluxDB Templates integration ○ Alert building w/ Notebooks ● Connection to InfluxDB Cloud ○ Data Replication ○ Remote Management ○ Single Login InfluxDB OSS 2.x 2022

16. © 2021 InﬂuxData. All rights reserved. 16 | In Nov 2020, we invited 3 community members to participate in PR reviews and more. ● Sven Rebhan - @srebhan, @DocLambda ● Thomas Casteleyn - @Hipska ● Paweł Żak - @zak-pawel, @p-zak Their contributions have been invaluable in terms of helping move the Telegraf project forward! Thank you!!! Telegraf Community Contributions

17. © 2021 InfluxData. All rights reserved. 17 | Telegraf Enhancements in 2020 – Dialect of Python Specification has details about the syntax and available functions – Execution cannot access file system, network, system resources – Use Cases: • Math operations • Logic operations • Some string operations • Add or remove tags/fields/metrics External Plugins via Execd Starlark Processor Telegraf Inputs Processors Aggregators Outputs Execd Execd Execd Execd External Input External Processor External Aggregator External Output

18. © 2021 InﬂuxData. All rights reserved. 18 |Telegraf 1.18.0 Released - March 17, 2021 New Inputs ● Beat Input Plugin ● CS:GO Input Plugin ● Directory Monitoring Input Plugin* ● RavenDB Input Plugin ● NFS Input Plugin* New Outputs ● Google BigQuery Output Plugin ● Grafana Loki Output Plugin ● Sensu Output Plugin ● SignalFX Output Plugin New External Plugins ● GeoIP Processor Plugin ● Plex Webhook Input Plugin ● SMCIPMITool Input Plugin New Aggregators ● Derivative Aggregator Plugin ● Quantile Aggregator Plugin New Processors ● AWS EC2 Metadata Processor Plugin New Parsers ● XML Parser Plugin New Serializers ● MessagePack Serializer Plugin *community requests New Features ● Updated to Go 1.16.2 ● Code Signing for Windows & Mac ● More SNMP v3 authentication protocols including SHA-512 ● 140+ commits from 78 contributors

20. InﬂuxDB Cloud

21. © 2021 InﬂuxData. All rights reserved. 21 | Global Distribution Run where you are / Pay how you want Multi-Cloud + Global Distribution Now: US (Oregon) US(Virginia) EU(Frankfurt) US (Iowa) EU (Belgium) US (Virginia) EU (Netherlands)

22. © 2021 InﬂuxData. All rights reserved. 22 | Global Distribution Run where you are / Pay how you want Next: • Continue expansion across all super regions (N. America, Europe, APAC) – Next: AWS AP (Sydney) – Azure Europe (North) – AWS Europe (Ireland) – Other APAC* Multi-Cloud + Global Distribution Now: US (Oregon) US(Virginia) EU(Frankfurt) US (Iowa) EU (Belgium) US (Virginia) EU (Netherlands)

23. © 2021 InﬂuxData. All rights reserved. 23 | • Usage-based billing • Social sign-on integration with Google and Microsoft (Azure AD) • Eligible to retire CSP committed consumption Billing Options Cloud Service Provider Marketplace Oﬀers Now: Later: Annual contracts

24. © 2021 InfluxData. All rights reserved. 24 | IoT: Geotemporal Available Now! Telegraf Plugins S2 Geo Processor: adds tags with an S2 cell ID token of a specified cell level. Specify lat and lon field values with WGS-84 coordinates in decimal degrees. GeoIP Processor: looks up IP addresses in the MaxMind GeoLite2 database and adds the respective ISO country code, city name, latitude and longitude.

25. © 2021 InfluxData. All rights reserved. 25 | IoT: Geotemporal Available Now! Filtering • filterRows() • gridFilter() • strictFilter() Aggregate • groupByArea() • asTracks() Transformation • s2CellIDToken() • toRows() Supported Shapes • box - defined by: minLat, maxLat, minLon, maxLon • circle - defined by: lat, lon, radius • polygon - array of points: lat, lon Telegraf Plugins S2 Geo Processor: adds tags with an S2 cell ID token of a specified cell level. Specify lat and lon field values with WGS-84 coordinates in decimal degrees. GeoIP Processor: looks up IP addresses in the MaxMind GeoLite2 database and adds the respective ISO country code, city name, latitude and longitude.

26. © 2021 InfluxData. All rights reserved. 26 | IoT: Geotemporal Available Now! Filtering • filterRows() • gridFilter() • strictFilter() Aggregate • groupByArea() • asTracks() Transformation • s2CellIDToken() • toRows() Supported Shapes • box - defined by: minLat, maxLat, minLon, maxLon • circle - defined by: lat, lon, radius • polygon - array of points: lat, lon Telegraf Plugins S2 Geo Processor: adds tags with an S2 cell ID token of a specified cell level. Specify lat and lon field values with WGS-84 coordinates in decimal degrees. GeoIP Processor: looks up IP addresses in the MaxMind GeoLite2 database and adds the respective ISO country code, city name, latitude and longitude. Map Visualisation

28. © 2021 InﬂuxData. All rights reserved. 28 | Notebooks Teams & Collaboration ● Goal is to provide a single consistent building experience in our platform ● Designed to allow teams to incrementally build queries or background processing tasks together ● Flexible enough to guide new users and unblock power users ● View multiple related resources in a single view Now

29. © 2021 InﬂuxData. All rights reserved. 29 | Dashboard Annotations Teams & Collaboration ● Goal is share context in dashboards ● Add, edit, delete as needed ● Annotations never expire ● Available in your InﬂuxDB Cloud accounts soon ● Ranged annotations, custom colors, support for more graph types coming soon Now

30. © 2021 InﬂuxData. All rights reserved. 30 | Tenant Observability ● Goal is to provide self-service access to usage and operations: ○ Usage ○ Cardinality ○ Rate limit events ○ Write errors ○ Query Log ● User conﬁgurable templates for dashboards & alerts ● Available on all cloud accounts shortly Next

32. © 2021 InﬂuxData. All rights reserved. 32 | Evolution of Tasks Functions Schedule Trigger (via Events) Tasks: Flux Parameterized / Reusable Remote Invocation Task (Via Schedule) Web hook (via API) Flux-based...to Start Next

33. © 2021 InfluxData. All rights reserved. 33 | Functions: Flux ● Goal is code reusability ○ Define Flux code once ○ Register as a Function with defined parameters ○ Invoke, as desired, with different values supplied as arguments ● Power tasks, triggers, dashboard cells with common functions Next Task = Function (Schedule, Arguments)

34. © 2021 InfluxData. All rights reserved. 34 | Triggers ● Goal is to use events to invoke functions ● Events can be “write to a bucket” ● Use cases: ○ Late arrivals of data ○ Streamlined alerting Next QUERY ACT Scheduled Task / Alert check Notificatio n systems WRITE Notificatio n systems WRITE WRITE ACT Triggers

35. © 2021 InfluxData. All rights reserved. 35 | Functions: Polyglot Programming ● Goal is make the platform do more for you ● Two main use cases: ○ Pre-process data before writing into InfluxDB ○ Post-process rows returned from querying InfluxDB ● MVP - Run your python code as function that can be invoked remotely Next Python code as a Function

37. © 2021 InfluxData. All rights reserved. 37 |Make data movement easier Enable users to define object store locations within their CSP account. Bulk Import Line protocol CSV TSM files Parquet Self-Service Backup Bucket backup (restore to OSS or Cloud) -- or -- Export to Parquet Next

38. © 2021 InfluxData. All rights reserved. 38 |Make data movement easier Enable users to define object store locations within their CSP account. Bulk Import Line protocol CSV TSM files Parquet Self-Service Backup Bucket backup (restore to OSS or Cloud) -- or -- Export to Parquet Next Cloud Native Ingest Connect with cloud native messaging services • MQTT, • AWS Kinesis • Azure EventHub

40. © 2021 InfluxData. All rights reserved. 40 | InfluxDB Enterprise ○ Continue the success of deployments managed by you ○ Deliver latest improvements for data access & processing InfluxData Community ○ Limo door is open...and ready to take you on the ride to InfluxDB 2.x ○ Telegraf advancements makes it the most powerful data acquisition tool ever InfluxDB Cloud ○ Teams & Collaboration ○ Functions & Reuse ○ Tenant Observability ○ Ease Data Movement Summary

41. Questions

Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021

You are reading a preview.

Check these out next

Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021

More Related Content

Slideshows for you (20)

Similar to Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021 (20)

More from InfluxData (20)

Recently uploaded (20)