SlideShare a Scribd company logo
1 of 14
Confidential
Time Series Data Management Evolution at Capital
One
Building a highly resilient Enterprise Influx across multiple regions
Monitoring Intelligence
October 23, 2018
Confidential
Agenda
2
1. Influx at Capital One
2. Architecture
How Influx Architecture evolved
3. Resiliency
Complete protection against entire region failure
4. Performance Metrics
Critical Platform Metrics
5. Q & A
Confidential
Influx at Capital One
3
• Business Transaction Metrics
• Infrastructure Health metrics
• Application Performance Metrics
• Service Adoption Metrics
Confidential
Agenda
4
1. Influx at Capital One
2. Architecture
How Influx Architecture evolved
3. Resiliency
Complete protection against entire region failure
4. Performance Metrics
Critical Platform Metrics
5. Q & A
Confidential
Architecture – Gen1
5
InfluxDB
( Active )
InfluxDB
(Standby)
DR Site
Visualization
https
Primary Site
LB
Backup/Restore
Features
1. Grafana for visualization
Challenges
1. High Data Retention ( > 400 days )
2. Unstable DR Solution
Splunk
Direct API
Telegraf
Confidential
Architecture – Gen2
6
InfluxDB
( Active )
InfluxDB
(Standby)
DR Site
Visualization
Primary Site
LB
AWS S3
Data Lake
ML Model
Execution
Model
Governance
ML Model
Dev/Train
Backup/Restore
Daily batch
Features
1. Grafana for visualization
2. Raw Data Exported Daily to One Lake
3. Raw Data available for ML
Challenges
High Data Retention ( > 400 days )
2. Unstable DR Solution
A
B. ReviewC. Deploy
E
D
Splunk
Direct API
Telegraf
https
Confidential
Architecture – Current
7
InfluxDB
( Active )
InfluxDB
(Passive)
DR Site
Visualization
Primary Site
LB
AWS S3
Data Lake
ML Model
Execution
Model
Governance
ML Model
Dev/Train
AWS S3
ExportImport Mini batch
Features
1. Grafana for visualization
2. Raw Data Exported every 30 minutes
to One Lake
3. Raw Data available for ML
4. Stable DR Solution with Passive
Cluster
Challenges
High Data Retention ( > 400 days )
Unstable DR Solution
A
B. ReviewC. Deploy
E
D
Splunk
Direct API
Telegraf
https
Confidential
Agenda
8
1. Influx at Capital One
2. Architecture
How Influx Architecture evolved
3. Resiliency
Complete protection against entire region failure
4. Performance Metrics
Critical Platform Metrics
5. Q & A
Confidential
Resiliency – Region 1 Active
9
https
D
M
D
M
D
M
Region 1 ( Active)
Zone 1 Zone 2 Zone 3
LB
S3
Cluster 1
D
M
D
M
D
M
Region 2
(DR)
Zone 1 Zone 2 Zone 3
LB
S3
Cluster 2
Route53
(DNS Switch)
Cross Region
Replication
Admin
A
ASGASG
M Meta Node D Data Node A Admin
Node
3
A
Route53
(DNS Switch)
1
4
2
Splunk
Direct API
Telegraf
All Traffic routed to Region 11
Influx Export Script every 15 min2
Data Replicated to Region 23
Influx Import Script every 15 min4
Confidential
Resiliency – Region 2 Active
10
https
D
M
D
M
D
M
Region 1 ( DR)
Zone 1 Zone 2 Zone 3
LB
S3
Cluster 1
D
M
D
M
D
M
Region 2 (Active)
Zone 1 Zone 2 Zone 3
LB
S3
Cluster 2
Route53
(DNS Switch)
Cross Region
Replication
Admin
A
ASGASG
M Meta Node D Data Node A Admin
Node
3
A
Route53
(DNS Switch)
1
2
4
Splunk
Direct API
Telegraf
All Traffic routed to Region 21
Influx Export Script every 15 min2
Data Replicated to Region 13
Influx Import Script every 15 min4
Confidential
Resiliency – Export/Import
11
Export*
Import*
* High level pseudo code , Not the actual code
Confidential
Agenda
12
1. Influx at Capital One
2. Architecture
How Influx Architecture evolved
3. Resiliency
Complete protection against entire region failure
4. Performance Metrics
Critical Platform Metrics
5. Q & A
Confidential
Performance Metrics
Performance Metrics collected using Telegraf
Confidential
Q & A

More Related Content

What's hot

Enterprise Architecture: The role of the Design Authority
Enterprise Architecture:The role of the Design AuthorityEnterprise Architecture:The role of the Design Authority
Enterprise Architecture: The role of the Design AuthorityInvestnet
 
Building a more cohesive organisation using business architecture
Building a more cohesive organisation using business architectureBuilding a more cohesive organisation using business architecture
Building a more cohesive organisation using business architectureCraig Martin
 
Agile Transformation Strategy
Agile Transformation StrategyAgile Transformation Strategy
Agile Transformation StrategySemen Arslan
 
Enterprise Agile Transformation
Enterprise Agile TransformationEnterprise Agile Transformation
Enterprise Agile TransformationPooja Wandile
 
Agile transformation Explained: Agile 2017 Session
Agile transformation Explained: Agile 2017 SessionAgile transformation Explained: Agile 2017 Session
Agile transformation Explained: Agile 2017 SessionLeadingAgile
 
Transitioning Compute Models: Hadoop MapReduce to Spark
Transitioning Compute Models: Hadoop MapReduce to SparkTransitioning Compute Models: Hadoop MapReduce to Spark
Transitioning Compute Models: Hadoop MapReduce to SparkSlim Baltagi
 
Business Agility 2017 (final)
Business Agility 2017 (final)Business Agility 2017 (final)
Business Agility 2017 (final)Fabio Armani
 
Lean and Agile SAP
Lean and Agile SAPLean and Agile SAP
Lean and Agile SAPJason Fair
 
SAFe (Scaled Agile Framework) 5 mins overview - Roni Tamari
SAFe (Scaled Agile Framework) 5 mins overview - Roni TamariSAFe (Scaled Agile Framework) 5 mins overview - Roni Tamari
SAFe (Scaled Agile Framework) 5 mins overview - Roni TamariAgileSparks
 
Beyond CIO - Will there still be Architecture Management in 2025
Beyond CIO - Will there still be Architecture Management in 2025Beyond CIO - Will there still be Architecture Management in 2025
Beyond CIO - Will there still be Architecture Management in 2025LeanIX GmbH
 
Value analysis with Value Stream and Capability modeling
Value analysis with Value Stream and Capability modelingValue analysis with Value Stream and Capability modeling
Value analysis with Value Stream and Capability modelingCOMPETENSIS
 
Align IT and Enterprise Operating Models.pdf
Align IT and Enterprise Operating Models.pdfAlign IT and Enterprise Operating Models.pdf
Align IT and Enterprise Operating Models.pdfJoelRodriguze
 
Capability-based Business Model Transformation
Capability-based Business Model TransformationCapability-based Business Model Transformation
Capability-based Business Model TransformationIlia Bider
 
How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...
How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...
How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...DataWorks Summit
 

What's hot (20)

Enterprise Architecture: The role of the Design Authority
Enterprise Architecture:The role of the Design AuthorityEnterprise Architecture:The role of the Design Authority
Enterprise Architecture: The role of the Design Authority
 
MAPPING TOGAF® ADM AND AGILE APPROACH
MAPPING TOGAF® ADM AND AGILE APPROACHMAPPING TOGAF® ADM AND AGILE APPROACH
MAPPING TOGAF® ADM AND AGILE APPROACH
 
Business agility
Business agilityBusiness agility
Business agility
 
Building a more cohesive organisation using business architecture
Building a more cohesive organisation using business architectureBuilding a more cohesive organisation using business architecture
Building a more cohesive organisation using business architecture
 
Agile Transformation Strategy
Agile Transformation StrategyAgile Transformation Strategy
Agile Transformation Strategy
 
Enterprise Agile Transformation
Enterprise Agile TransformationEnterprise Agile Transformation
Enterprise Agile Transformation
 
SAP Overview
SAP OverviewSAP Overview
SAP Overview
 
Agile transformation Explained: Agile 2017 Session
Agile transformation Explained: Agile 2017 SessionAgile transformation Explained: Agile 2017 Session
Agile transformation Explained: Agile 2017 Session
 
Agile Fundamentals
Agile FundamentalsAgile Fundamentals
Agile Fundamentals
 
Transitioning Compute Models: Hadoop MapReduce to Spark
Transitioning Compute Models: Hadoop MapReduce to SparkTransitioning Compute Models: Hadoop MapReduce to Spark
Transitioning Compute Models: Hadoop MapReduce to Spark
 
Business Agility 2017 (final)
Business Agility 2017 (final)Business Agility 2017 (final)
Business Agility 2017 (final)
 
Introduction to sap 1
Introduction to sap 1Introduction to sap 1
Introduction to sap 1
 
Project to product mindset
Project to product mindsetProject to product mindset
Project to product mindset
 
Lean and Agile SAP
Lean and Agile SAPLean and Agile SAP
Lean and Agile SAP
 
SAFe (Scaled Agile Framework) 5 mins overview - Roni Tamari
SAFe (Scaled Agile Framework) 5 mins overview - Roni TamariSAFe (Scaled Agile Framework) 5 mins overview - Roni Tamari
SAFe (Scaled Agile Framework) 5 mins overview - Roni Tamari
 
Beyond CIO - Will there still be Architecture Management in 2025
Beyond CIO - Will there still be Architecture Management in 2025Beyond CIO - Will there still be Architecture Management in 2025
Beyond CIO - Will there still be Architecture Management in 2025
 
Value analysis with Value Stream and Capability modeling
Value analysis with Value Stream and Capability modelingValue analysis with Value Stream and Capability modeling
Value analysis with Value Stream and Capability modeling
 
Align IT and Enterprise Operating Models.pdf
Align IT and Enterprise Operating Models.pdfAlign IT and Enterprise Operating Models.pdf
Align IT and Enterprise Operating Models.pdf
 
Capability-based Business Model Transformation
Capability-based Business Model TransformationCapability-based Business Model Transformation
Capability-based Business Model Transformation
 
How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...
How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...
How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...
 

Similar to Why Architecting for Disaster Recovery is Important for Your Time Series Data by Capital One

Flink at netflix paypal speaker series
Flink at netflix   paypal speaker seriesFlink at netflix   paypal speaker series
Flink at netflix paypal speaker seriesMonal Daxini
 
ScyllaDB V Developer Deep Dive Series: Resiliency and Strong Consistency via ...
ScyllaDB V Developer Deep Dive Series: Resiliency and Strong Consistency via ...ScyllaDB V Developer Deep Dive Series: Resiliency and Strong Consistency via ...
ScyllaDB V Developer Deep Dive Series: Resiliency and Strong Consistency via ...ScyllaDB
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesJosef Adersberger
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesQAware GmbH
 
Meetup Microservices Commandments
Meetup Microservices CommandmentsMeetup Microservices Commandments
Meetup Microservices CommandmentsBill Zajac
 
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scaleVelocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scaleMichael Kehoe
 
Architectural Commandments for Building & Running Microservices at Scale
Architectural Commandments for Building & Running Microservices at ScaleArchitectural Commandments for Building & Running Microservices at Scale
Architectural Commandments for Building & Running Microservices at ScaleBrian Wilson
 
cncf overview and building edge computing using kubernetes
cncf overview and building edge computing using kubernetescncf overview and building edge computing using kubernetes
cncf overview and building edge computing using kubernetesKrishna-Kumar
 
App Modernization with Aerospike as an Intraday System of Record_Venkat Thamm...
App Modernization with Aerospike as an Intraday System of Record_Venkat Thamm...App Modernization with Aerospike as an Intraday System of Record_Venkat Thamm...
App Modernization with Aerospike as an Intraday System of Record_Venkat Thamm...Aerospike, Inc.
 
Flink forward-2017-netflix keystones-paas
Flink forward-2017-netflix keystones-paasFlink forward-2017-netflix keystones-paas
Flink forward-2017-netflix keystones-paasMonal Daxini
 
Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...
Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...
Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...Sumeet Singh
 
Agile integration cloud native developement
Agile integration   cloud native developementAgile integration   cloud native developement
Agile integration cloud native developementChristina Lin
 
QCON 2015: Gearpump, Realtime Streaming on Akka
QCON 2015: Gearpump, Realtime Streaming on AkkaQCON 2015: Gearpump, Realtime Streaming on Akka
QCON 2015: Gearpump, Realtime Streaming on AkkaSean Zhong
 
Beaming flink to the cloud @ netflix ff 2016-monal-daxini
Beaming flink to the cloud @ netflix   ff 2016-monal-daxiniBeaming flink to the cloud @ netflix   ff 2016-monal-daxini
Beaming flink to the cloud @ netflix ff 2016-monal-daxiniMonal Daxini
 
Monal Daxini - Beaming Flink to the Cloud @ Netflix
Monal Daxini - Beaming Flink to the Cloud @ NetflixMonal Daxini - Beaming Flink to the Cloud @ Netflix
Monal Daxini - Beaming Flink to the Cloud @ NetflixFlink Forward
 
Patterns of-streaming-applications-qcon-2018-monal-daxini
Patterns of-streaming-applications-qcon-2018-monal-daxiniPatterns of-streaming-applications-qcon-2018-monal-daxini
Patterns of-streaming-applications-qcon-2018-monal-daxiniMonal Daxini
 
Spark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike FreedmanSpark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike FreedmanSpark Summit
 
Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...
Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...
Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...Flink Forward
 
Stream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream SharingStream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream Sharingconfluent
 

Similar to Why Architecting for Disaster Recovery is Important for Your Time Series Data by Capital One (20)

Flink at netflix paypal speaker series
Flink at netflix   paypal speaker seriesFlink at netflix   paypal speaker series
Flink at netflix paypal speaker series
 
ScyllaDB V Developer Deep Dive Series: Resiliency and Strong Consistency via ...
ScyllaDB V Developer Deep Dive Series: Resiliency and Strong Consistency via ...ScyllaDB V Developer Deep Dive Series: Resiliency and Strong Consistency via ...
ScyllaDB V Developer Deep Dive Series: Resiliency and Strong Consistency via ...
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
 
Meetup Microservices Commandments
Meetup Microservices CommandmentsMeetup Microservices Commandments
Meetup Microservices Commandments
 
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scaleVelocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
 
Architectural Commandments for Building & Running Microservices at Scale
Architectural Commandments for Building & Running Microservices at ScaleArchitectural Commandments for Building & Running Microservices at Scale
Architectural Commandments for Building & Running Microservices at Scale
 
cncf overview and building edge computing using kubernetes
cncf overview and building edge computing using kubernetescncf overview and building edge computing using kubernetes
cncf overview and building edge computing using kubernetes
 
App Modernization with Aerospike as an Intraday System of Record_Venkat Thamm...
App Modernization with Aerospike as an Intraday System of Record_Venkat Thamm...App Modernization with Aerospike as an Intraday System of Record_Venkat Thamm...
App Modernization with Aerospike as an Intraday System of Record_Venkat Thamm...
 
Flink forward-2017-netflix keystones-paas
Flink forward-2017-netflix keystones-paasFlink forward-2017-netflix keystones-paas
Flink forward-2017-netflix keystones-paas
 
Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...
Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...
Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...
 
Agile integration cloud native developement
Agile integration   cloud native developementAgile integration   cloud native developement
Agile integration cloud native developement
 
QCON 2015: Gearpump, Realtime Streaming on Akka
QCON 2015: Gearpump, Realtime Streaming on AkkaQCON 2015: Gearpump, Realtime Streaming on Akka
QCON 2015: Gearpump, Realtime Streaming on Akka
 
Beaming flink to the cloud @ netflix ff 2016-monal-daxini
Beaming flink to the cloud @ netflix   ff 2016-monal-daxiniBeaming flink to the cloud @ netflix   ff 2016-monal-daxini
Beaming flink to the cloud @ netflix ff 2016-monal-daxini
 
Monal Daxini - Beaming Flink to the Cloud @ Netflix
Monal Daxini - Beaming Flink to the Cloud @ NetflixMonal Daxini - Beaming Flink to the Cloud @ Netflix
Monal Daxini - Beaming Flink to the Cloud @ Netflix
 
Patterns of-streaming-applications-qcon-2018-monal-daxini
Patterns of-streaming-applications-qcon-2018-monal-daxiniPatterns of-streaming-applications-qcon-2018-monal-daxini
Patterns of-streaming-applications-qcon-2018-monal-daxini
 
Spark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike FreedmanSpark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike Freedman
 
Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...
Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...
Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...
 
dA Platform Overview
dA Platform OverviewdA Platform Overview
dA Platform Overview
 
Stream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream SharingStream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream Sharing
 

More from InfluxData

Announcing InfluxDB Clustered
Announcing InfluxDB ClusteredAnnouncing InfluxDB Clustered
Announcing InfluxDB ClusteredInfluxData
 
Best Practices for Leveraging the Apache Arrow Ecosystem
Best Practices for Leveraging the Apache Arrow EcosystemBest Practices for Leveraging the Apache Arrow Ecosystem
Best Practices for Leveraging the Apache Arrow EcosystemInfluxData
 
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...InfluxData
 
Power Your Predictive Analytics with InfluxDB
Power Your Predictive Analytics with InfluxDBPower Your Predictive Analytics with InfluxDB
Power Your Predictive Analytics with InfluxDBInfluxData
 
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base InfluxData
 
Build an Edge-to-Cloud Solution with the MING Stack
Build an Edge-to-Cloud Solution with the MING StackBuild an Edge-to-Cloud Solution with the MING Stack
Build an Edge-to-Cloud Solution with the MING StackInfluxData
 
Meet the Founders: An Open Discussion About Rewriting Using Rust
Meet the Founders: An Open Discussion About Rewriting Using RustMeet the Founders: An Open Discussion About Rewriting Using Rust
Meet the Founders: An Open Discussion About Rewriting Using RustInfluxData
 
Introducing InfluxDB Cloud Dedicated
Introducing InfluxDB Cloud DedicatedIntroducing InfluxDB Cloud Dedicated
Introducing InfluxDB Cloud DedicatedInfluxData
 
Gain Better Observability with OpenTelemetry and InfluxDB
Gain Better Observability with OpenTelemetry and InfluxDB Gain Better Observability with OpenTelemetry and InfluxDB
Gain Better Observability with OpenTelemetry and InfluxDB InfluxData
 
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...InfluxData
 
How Delft University's Engineering Students Make Their EV Formula-Style Race ...
How Delft University's Engineering Students Make Their EV Formula-Style Race ...How Delft University's Engineering Students Make Their EV Formula-Style Race ...
How Delft University's Engineering Students Make Their EV Formula-Style Race ...InfluxData
 
Introducing InfluxDB’s New Time Series Database Storage Engine
Introducing InfluxDB’s New Time Series Database Storage EngineIntroducing InfluxDB’s New Time Series Database Storage Engine
Introducing InfluxDB’s New Time Series Database Storage EngineInfluxData
 
Start Automating InfluxDB Deployments at the Edge with balena
Start Automating InfluxDB Deployments at the Edge with balena Start Automating InfluxDB Deployments at the Edge with balena
Start Automating InfluxDB Deployments at the Edge with balena InfluxData
 
Understanding InfluxDB’s New Storage Engine
Understanding InfluxDB’s New Storage EngineUnderstanding InfluxDB’s New Storage Engine
Understanding InfluxDB’s New Storage EngineInfluxData
 
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDBStreamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDBInfluxData
 
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...InfluxData
 
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022InfluxData
 
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022InfluxData
 
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...InfluxData
 
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022InfluxData
 

More from InfluxData (20)

Announcing InfluxDB Clustered
Announcing InfluxDB ClusteredAnnouncing InfluxDB Clustered
Announcing InfluxDB Clustered
 
Best Practices for Leveraging the Apache Arrow Ecosystem
Best Practices for Leveraging the Apache Arrow EcosystemBest Practices for Leveraging the Apache Arrow Ecosystem
Best Practices for Leveraging the Apache Arrow Ecosystem
 
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
 
Power Your Predictive Analytics with InfluxDB
Power Your Predictive Analytics with InfluxDBPower Your Predictive Analytics with InfluxDB
Power Your Predictive Analytics with InfluxDB
 
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
 
Build an Edge-to-Cloud Solution with the MING Stack
Build an Edge-to-Cloud Solution with the MING StackBuild an Edge-to-Cloud Solution with the MING Stack
Build an Edge-to-Cloud Solution with the MING Stack
 
Meet the Founders: An Open Discussion About Rewriting Using Rust
Meet the Founders: An Open Discussion About Rewriting Using RustMeet the Founders: An Open Discussion About Rewriting Using Rust
Meet the Founders: An Open Discussion About Rewriting Using Rust
 
Introducing InfluxDB Cloud Dedicated
Introducing InfluxDB Cloud DedicatedIntroducing InfluxDB Cloud Dedicated
Introducing InfluxDB Cloud Dedicated
 
Gain Better Observability with OpenTelemetry and InfluxDB
Gain Better Observability with OpenTelemetry and InfluxDB Gain Better Observability with OpenTelemetry and InfluxDB
Gain Better Observability with OpenTelemetry and InfluxDB
 
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...
 
How Delft University's Engineering Students Make Their EV Formula-Style Race ...
How Delft University's Engineering Students Make Their EV Formula-Style Race ...How Delft University's Engineering Students Make Their EV Formula-Style Race ...
How Delft University's Engineering Students Make Their EV Formula-Style Race ...
 
Introducing InfluxDB’s New Time Series Database Storage Engine
Introducing InfluxDB’s New Time Series Database Storage EngineIntroducing InfluxDB’s New Time Series Database Storage Engine
Introducing InfluxDB’s New Time Series Database Storage Engine
 
Start Automating InfluxDB Deployments at the Edge with balena
Start Automating InfluxDB Deployments at the Edge with balena Start Automating InfluxDB Deployments at the Edge with balena
Start Automating InfluxDB Deployments at the Edge with balena
 
Understanding InfluxDB’s New Storage Engine
Understanding InfluxDB’s New Storage EngineUnderstanding InfluxDB’s New Storage Engine
Understanding InfluxDB’s New Storage Engine
 
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDBStreamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB
 
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...
 
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022
 
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022
 
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
 
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
 

Recently uploaded

What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditSkynet Technologies
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Scott Andery
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPathCommunity
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 

Recently uploaded (20)

What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance Audit
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to Hero
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 

Why Architecting for Disaster Recovery is Important for Your Time Series Data by Capital One

  • 1. Confidential Time Series Data Management Evolution at Capital One Building a highly resilient Enterprise Influx across multiple regions Monitoring Intelligence October 23, 2018
  • 2. Confidential Agenda 2 1. Influx at Capital One 2. Architecture How Influx Architecture evolved 3. Resiliency Complete protection against entire region failure 4. Performance Metrics Critical Platform Metrics 5. Q & A
  • 3. Confidential Influx at Capital One 3 • Business Transaction Metrics • Infrastructure Health metrics • Application Performance Metrics • Service Adoption Metrics
  • 4. Confidential Agenda 4 1. Influx at Capital One 2. Architecture How Influx Architecture evolved 3. Resiliency Complete protection against entire region failure 4. Performance Metrics Critical Platform Metrics 5. Q & A
  • 5. Confidential Architecture – Gen1 5 InfluxDB ( Active ) InfluxDB (Standby) DR Site Visualization https Primary Site LB Backup/Restore Features 1. Grafana for visualization Challenges 1. High Data Retention ( > 400 days ) 2. Unstable DR Solution Splunk Direct API Telegraf
  • 6. Confidential Architecture – Gen2 6 InfluxDB ( Active ) InfluxDB (Standby) DR Site Visualization Primary Site LB AWS S3 Data Lake ML Model Execution Model Governance ML Model Dev/Train Backup/Restore Daily batch Features 1. Grafana for visualization 2. Raw Data Exported Daily to One Lake 3. Raw Data available for ML Challenges High Data Retention ( > 400 days ) 2. Unstable DR Solution A B. ReviewC. Deploy E D Splunk Direct API Telegraf https
  • 7. Confidential Architecture – Current 7 InfluxDB ( Active ) InfluxDB (Passive) DR Site Visualization Primary Site LB AWS S3 Data Lake ML Model Execution Model Governance ML Model Dev/Train AWS S3 ExportImport Mini batch Features 1. Grafana for visualization 2. Raw Data Exported every 30 minutes to One Lake 3. Raw Data available for ML 4. Stable DR Solution with Passive Cluster Challenges High Data Retention ( > 400 days ) Unstable DR Solution A B. ReviewC. Deploy E D Splunk Direct API Telegraf https
  • 8. Confidential Agenda 8 1. Influx at Capital One 2. Architecture How Influx Architecture evolved 3. Resiliency Complete protection against entire region failure 4. Performance Metrics Critical Platform Metrics 5. Q & A
  • 9. Confidential Resiliency – Region 1 Active 9 https D M D M D M Region 1 ( Active) Zone 1 Zone 2 Zone 3 LB S3 Cluster 1 D M D M D M Region 2 (DR) Zone 1 Zone 2 Zone 3 LB S3 Cluster 2 Route53 (DNS Switch) Cross Region Replication Admin A ASGASG M Meta Node D Data Node A Admin Node 3 A Route53 (DNS Switch) 1 4 2 Splunk Direct API Telegraf All Traffic routed to Region 11 Influx Export Script every 15 min2 Data Replicated to Region 23 Influx Import Script every 15 min4
  • 10. Confidential Resiliency – Region 2 Active 10 https D M D M D M Region 1 ( DR) Zone 1 Zone 2 Zone 3 LB S3 Cluster 1 D M D M D M Region 2 (Active) Zone 1 Zone 2 Zone 3 LB S3 Cluster 2 Route53 (DNS Switch) Cross Region Replication Admin A ASGASG M Meta Node D Data Node A Admin Node 3 A Route53 (DNS Switch) 1 2 4 Splunk Direct API Telegraf All Traffic routed to Region 21 Influx Export Script every 15 min2 Data Replicated to Region 13 Influx Import Script every 15 min4
  • 11. Confidential Resiliency – Export/Import 11 Export* Import* * High level pseudo code , Not the actual code
  • 12. Confidential Agenda 12 1. Influx at Capital One 2. Architecture How Influx Architecture evolved 3. Resiliency Complete protection against entire region failure 4. Performance Metrics Critical Platform Metrics 5. Q & A