Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

InfluxEnterprise Architecture Patterns by Tim Hall & Sam Dillard

222 views

Published on

In this InfluxDays NYC 2019 presentation, InfluxData VP of Products Tim Hall and Sales Engineer Sam Dillard discuss architecture patterns with InfluxEnterprise time series platform. They cover an overview of InfluxEnterprise, features, ingestion and query rates, deployment examples, replication patterns, and general advice. Presentation highlights include InfluxEnterprise cluster architecture and how to determine if you're ready for adopting InfluxEnterprise.

Published in: Technology
  • Login to see the comments

  • Be the first to like this

InfluxEnterprise Architecture Patterns by Tim Hall & Sam Dillard

  1. 1. Sam Dillard Sales Engineering InfluxEnterprise Architectural Patterns Tim Hall VP, Products
  2. 2. What we will be covering ✓ Enterprise Overview ✓ Other Features ✓ Ingestion & Query Rates ✓ Deployment Examples ✓ Replications Patterns ✓ General Advice
  3. 3. Why InfluxEnterprise?
  4. 4. InfluxEnterprise • Open Source Core • High Availability • Scalability • Fine Grained Authorization • Support from InfluxData • OnPremise/Cloud Deployment Options
  5. 5. Signs You’re Ready for InfluxEnterprise 6. Your CPU average is >=70% 1. The sales team starts calling you on weekends 5. Increasing throughput causing write drops errors 4. Sprawling number of single node deployments 3. Horizontal scaling not providing further benefit 2. Data durability matters
  6. 6. What Problem Are You Trying to Solve?
  7. 7. What are you dealing with? • Metrics • Events • Log Data • Sensors • Apps • Servers • Long-Term Storage • Vendor Replacement • Time-Series Alerts • Visualization • Network Data • Custom Solution • Real-Time Analytics • Virtualization Monitoring • Managed Service (InfluxCloud)
  8. 8. InfluxEnterprise Overview
  9. 9. InfluxEnterprise Cluster Architecture: Meta Nodes
  10. 10. InfluxEnterprise Clustering: Data Nodes
  11. 11. InfluxEnterprise Cluster Architecture
  12. 12. InfluxEnterprise Cluster Architecture
  13. 13. Features
  14. 14. Security • LDAP Support – Enterprise customers can configure the database to use LDAP as a backing authentication source for users, roles and permissions. – Connection between DB and LDAP server secured once connected • Fine-grained authorization – Used to control access at a measurement or series level (compared to limiting access at the database level) – Enable authentication in your configuration file – Create users through the query API – Grant users explicit read and/or write privileges – Set restrictions which define a combination of database, measurement, and tags which cannot be accessed without an explicit grant
  15. 15. © 2018 InfluxData. All rights reserved.© 2017 InfluxData. All rights reserved. Eventual Consistency • Anti-Entropy Service – Expands on capabilities to detect and copy full shards – Now allows for detection and repair of inconsistent shards • Hinted-handoff Queue – Queue inbound points destined to land on other nodes in the cluster which may currently be down – Stored by node and shard (10GB - default)
  16. 16. Backup and Restore • Useful for: Disaster recovery, Debugging, Restoring clusters to a consistent state • What it does: Creates a copy of the metastore and the shard data • Backup is compressed and is not human readable • Export is not compressed but is human readable • OSS and InfluxEnterprise ARE NOW compatible – aka portable • Full or partial backup options • Move data into a new database (with new Retention Policies, etc)
  17. 17. Ingestion & Query Rates
  18. 18. Cluster Data Node 1 Data Node 2 Data Node 3 Data Node 4 Database X (Replication=1) Shard 1 Shard 2 Shard 3 Shard 4 a b c d X ≈ 4x ingest rate ≤ 1x concurrent query rate a b c d
  19. 19. Cluster Data Node 1 Data Node 2 Data Node 3 Data Node 4 Database X (Replication=4) Shard 1 Shard 2 Shard 3 Shard 4 a a’ a’’ a’’’ X ≈ 1x ingest rate replication ≈ 4x concurrent query rate a b b b’ b’’ b’’’
  20. 20. Cluster Data Node 1 Data Node 2 Data Node 3 Data Node 4 Database X (Replication=2) Shard 1 Shard 2 Shard 3 Shard 4 a a’ b b’ X ≈ 2x ingest rate replication replication ≤ 2x concurrent query rate a b
  21. 21. Deployment Examples
  22. 22. How does InfluxEnterprise Fit?
  23. 23. Example 1: Mothership Data Center 1 Kapacitor Telegraf InfluxDB Ent Enterprise Cluster Data Node 1 Data Node 2 Data Node 3 Data Node n Firewall/ LoadBalancer Telegraf Telegraf Chronograf Chronograf Kapacitor Data Center 2 Kapacitor Telegraf InfluxDB EntTelegraf Telegraf Chronograf
  24. 24. Example 2: Durable Data Ingest Telegraf Cluster Telegraf or other source Kafka Queue LoadBalancer InfluxDB Cluster Telegraf or other source Telegraf or other source Telegraf or other sources Telegraf Telegraf Telegraf Telegraf Put each Telegraf instance in the same Kafka Consumer Group How Fast is Fast? (ex): Six datanodes at 2.5M values per second
  25. 25. Example 3: Influx with ElasticSearch InfluxDB Cluster • Discover trends before and during the Error from metrics • Perform Root Cause Analysis from Logs LoadBalancer Telegraf ElasticSearch Include common Session ID or other UID Kapacitor You Metrics Logs Query using the common Session ID or UID received form Alert
  26. 26. Replication Patterns
  27. 27. How Are You Using InfluxDB?
  28. 28. Data Replication Generally there are two types of data that we care about replicating: • New Data – Data which is coming form our raw sources • Derived Data – The output of a SELECT INTO, or TICK script
  29. 29. Replication of New Data – Pattern 1 Cluster 1 Firewall/ Load Balancer Data Node 1 Data Node 2 Data Node 3 Data Node n Telegraf Cluster 2 Firewall/ Load Balancer Data Node 1 Data Node 2 Data Node 3 Data Node n Telegraf
  30. 30. Replication of New Data – Pattern 2 Cluster 1 Firewall/ Load Balancer Data Node 1 Data Node 2 Data Node 3 Data Node n Telegraf Cluster 2 Firewall/ Load Balancer Data Node 1 Data Node 2 Data Node 3 Data Node n Telegraf Kafka Queue
  31. 31. Replication of Derived Data – Pattern 3 Cluster 1 Load Balancer Cluster 2 Data Nodes Data Nodes Data Nodes Data Nodes Data Nodes Data Nodes Data Nodes Kapacitor Load Balancer Data Nodes Data Nodes Data Nodes Data Nodes Data Nodes Data Nodes Data Nodes Kapacitor Uses output of Kapacitor to other cluster Telegraf Telegraf
  32. 32. General Advice
  33. 33. General Cluster Advice • Batch your writes! • The number of data nodes should be a multiple of your replication factor • Use a single node of InfluxDB to monitor your cluster • Put a load balancer in front of each of your data nodes • Higher replication factors result in higher query concurrency, but higher write latency. • Use Fine Grained Authorization instead of multiple databases
  34. 34. Thank You!

×