SlideShare a Scribd company logo
Monitoring and Alerting
as Code with Terraform
and New Relic
http://bit.ly/terraform-newrelic
January 30, 2019
Richard Bullington-McGuire
Principal Architect, Modus Create
richard@moduscreate.com
@obscurerichard
My New Relic & Terraform Experience
2014: Used New Relic to improve performance 40x in public safety training
system - supported $MM exit - Java, Windows
2014-2017: Used New Relic to improve performance 10x in hospitality system -
saving the client’s reputation - .NET, SQL Server, Windows, JMeter load tests
2015: Took over ProductShop project - Terraform
2017: Used Terraform & New Relic to migrate 14 critical systems to AWS for a
large education company.
2017: Taught real estate information software firm how to do cloud migration
right with Terraform and New Relic.
2018: Built out devops-infra-demo Terraform / Packer / Ansible / Jenkins
Configuring monitoring by hand is fragile and tough
Use Infrastructure as Code techniques to make it repeatable
Load test to ensure that monitoring & alerting actually work
Work around chicken or egg problems
Monitoring and Alerting as Code
The old way:
Set up New Relic by hand on each server.
Tweak configs until they match.
Standing up a new environment or application?
Copy and paste and pray
Configuring monitoring by hand: just say no
● Set up your application stack and monitoring & alerting with
the same toolkit
● Leverage tools such as Ansible to reduce boilerplate
● Use an image bakery pattern - consider immutable
infrastructure or a hybrid pattern.
● Consider AWS CodeDeploy for legacy migration - easier than
dockerizing everything!
● Use Jenkins or another CI process to drive the bakery
Use Infrastructure as Code for repeatability
● ab is good, JMeter is better
● Use New Relic to monitor load tests and check validity
● Build a representative test resembling production peak load
● Run load tests through CI at scale or use BlazeMeter
● jmeter-ec2 or its forks can run massive load tests
○ But the code is ugly and not maintained much and you
will have to hack it
Load test to ensure that it all actually works
● Terraform can only define resources for items that exist
○ So how do you define monitoring for a stack that is not
yet created?
● count to the rescue
● See this Gruntwork blog article on Terraform Tips & Tricks
Work around chicken or egg problems
Case Study:
Terraform & New Relic & JMeter at work
Driven by Jenkins CI
Case Study:
Terraform & New Relic & JMeter at work
Driven by Jenkins CI
● Education company cloud migration (4mo -> prod)
● Apps w/> 30,000 RPM at peak measured with New Relic
● Production with 80+ sizeable AMIs baseline
● Auto Scaling to 200+ AMIs under heavy load
● Multiple environments: dev, qa, staging, prod
● Terabyte-scale MySQL Aurora cluster, 50+ TB in S3
● Jenkins, Terraform, Ansible, Packer, CodeDeploy,
JMeter load tests, New Relic monitoring
Implementation
http://bit.ly/terraform-newrelic
Local Development
CI Environment
Jenkins
Elastic Load
Balancer
EC2 Auto Scaling
Group - Web App
Terraform Provision
Packer
Provision
Demo
http://bit.ly/terraform-newrelic
Credit where Credit is Due
● Sergio Bruder (@sdbruder) built the JMeter test harness and
CodeDeploy app used in the demo
● Don Anderson (@dranderson) integrated New Relic APM,
Infrastructure, and Dashboards for this demo
I could not have done this without their help!
Thank You!
http://bit.ly/terraform-newrelic
richard@moduscreate.com
@obscurerichard

More Related Content

What's hot

Introduction to CI/CD
Introduction to CI/CDIntroduction to CI/CD
Introduction to CI/CD
Steve Mactaggart
 
Continuous Delivery, Continuous Integration
Continuous Delivery, Continuous Integration Continuous Delivery, Continuous Integration
Continuous Delivery, Continuous Integration
Amazon Web Services
 
Code Quality Lightning Talk
Code Quality Lightning TalkCode Quality Lightning Talk
Code Quality Lightning Talk
Jonathan Gregory
 
Neoload
Neoload Neoload
Neoload
Kumar Gupta
 
Easy Cloud Native Transformation with Nomad
Easy Cloud Native Transformation with NomadEasy Cloud Native Transformation with Nomad
Easy Cloud Native Transformation with Nomad
Bram Vogelaar
 
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdfOSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
NETWAYS
 
Test automation
Test automationTest automation
Test automation
Xavier Yin
 
Infrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using PrometheusInfrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using Prometheus
Marco Pas
 
Observability
Observability Observability
Observability
Enes Altınok
 
Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKS
Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKSMigrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKS
Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKS
Weaveworks
 
CICD with Jenkins
CICD with JenkinsCICD with Jenkins
DevOps Monitoring and Alerting
DevOps Monitoring and AlertingDevOps Monitoring and Alerting
DevOps Monitoring and Alerting
Khairul Zebua
 
CI/CD
CI/CDCI/CD
CI/CD
AmitDhodi
 
How to Monitoring the SRE Golden Signals (E-Book)
How to Monitoring the SRE Golden Signals (E-Book)How to Monitoring the SRE Golden Signals (E-Book)
How to Monitoring the SRE Golden Signals (E-Book)
Siglos
 
Transforming Organizations with CI/CD
Transforming Organizations with CI/CDTransforming Organizations with CI/CD
Transforming Organizations with CI/CD
Cprime
 
Observability
ObservabilityObservability
Observability
Ebru Cucen Çüçen
 
Using Jenkins and Jmeter to build a scalable Load Testing solution
Using Jenkins and Jmeter to build a scalable Load Testing solutionUsing Jenkins and Jmeter to build a scalable Load Testing solution
Using Jenkins and Jmeter to build a scalable Load Testing solution
Ruslan Strazhnyk
 
Introduction to Prometheus
Introduction to PrometheusIntroduction to Prometheus
Introduction to Prometheus
Julien Pivotto
 
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
DevOps.com
 
Spotify architecture - Pressing play
Spotify architecture - Pressing playSpotify architecture - Pressing play
Spotify architecture - Pressing play
Niklas Gustavsson
 

What's hot (20)

Introduction to CI/CD
Introduction to CI/CDIntroduction to CI/CD
Introduction to CI/CD
 
Continuous Delivery, Continuous Integration
Continuous Delivery, Continuous Integration Continuous Delivery, Continuous Integration
Continuous Delivery, Continuous Integration
 
Code Quality Lightning Talk
Code Quality Lightning TalkCode Quality Lightning Talk
Code Quality Lightning Talk
 
Neoload
Neoload Neoload
Neoload
 
Easy Cloud Native Transformation with Nomad
Easy Cloud Native Transformation with NomadEasy Cloud Native Transformation with Nomad
Easy Cloud Native Transformation with Nomad
 
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdfOSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
 
Test automation
Test automationTest automation
Test automation
 
Infrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using PrometheusInfrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using Prometheus
 
Observability
Observability Observability
Observability
 
Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKS
Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKSMigrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKS
Migrating from Self-Managed Kubernetes on EC2 to a GitOps Enabled EKS
 
CICD with Jenkins
CICD with JenkinsCICD with Jenkins
CICD with Jenkins
 
DevOps Monitoring and Alerting
DevOps Monitoring and AlertingDevOps Monitoring and Alerting
DevOps Monitoring and Alerting
 
CI/CD
CI/CDCI/CD
CI/CD
 
How to Monitoring the SRE Golden Signals (E-Book)
How to Monitoring the SRE Golden Signals (E-Book)How to Monitoring the SRE Golden Signals (E-Book)
How to Monitoring the SRE Golden Signals (E-Book)
 
Transforming Organizations with CI/CD
Transforming Organizations with CI/CDTransforming Organizations with CI/CD
Transforming Organizations with CI/CD
 
Observability
ObservabilityObservability
Observability
 
Using Jenkins and Jmeter to build a scalable Load Testing solution
Using Jenkins and Jmeter to build a scalable Load Testing solutionUsing Jenkins and Jmeter to build a scalable Load Testing solution
Using Jenkins and Jmeter to build a scalable Load Testing solution
 
Introduction to Prometheus
Introduction to PrometheusIntroduction to Prometheus
Introduction to Prometheus
 
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
 
Spotify architecture - Pressing play
Spotify architecture - Pressing playSpotify architecture - Pressing play
Spotify architecture - Pressing play
 

Similar to Monitoring and alerting as code with Terraform and New Relic

Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Richard Bullington-McGuire
 
Hewlett Packard Entreprise | Stormrunner load | Game Changer
Hewlett Packard Entreprise | Stormrunner load | Game ChangerHewlett Packard Entreprise | Stormrunner load | Game Changer
Hewlett Packard Entreprise | Stormrunner load | Game Changer
Jeffrey Nunn
 
Using Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High ScalabilityUsing Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High Scalability
mabuhr
 
Keptn: Unbreakable Continuous Delivery - Berlin CI/CD Meetup
Keptn: Unbreakable Continuous Delivery - Berlin CI/CD MeetupKeptn: Unbreakable Continuous Delivery - Berlin CI/CD Meetup
Keptn: Unbreakable Continuous Delivery - Berlin CI/CD Meetup
Jürgen Etzlstorfer
 
Containerised Testing at Demonware : PyCon Ireland 2016
Containerised Testing at Demonware : PyCon Ireland 2016Containerised Testing at Demonware : PyCon Ireland 2016
Containerised Testing at Demonware : PyCon Ireland 2016
Thomas Shaw
 
MLOps and Reproducible ML on AWS with Kubeflow and SageMaker
MLOps and Reproducible ML on AWS with Kubeflow and SageMakerMLOps and Reproducible ML on AWS with Kubeflow and SageMaker
MLOps and Reproducible ML on AWS with Kubeflow and SageMaker
Provectus
 
Cloud-Native Fundamentals: Accelerating Development with Continuous Integration
Cloud-Native Fundamentals: Accelerating Development with Continuous IntegrationCloud-Native Fundamentals: Accelerating Development with Continuous Integration
Cloud-Native Fundamentals: Accelerating Development with Continuous Integration
VMware Tanzu
 
Hands-on monitoring with Prometheus
Hands-on monitoring with PrometheusHands-on monitoring with Prometheus
Hands-on monitoring with Prometheus
Brice Fernandes
 
Effective Kubernetes - Is Kubernetes the new Linux? Is the new Application Se...
Effective Kubernetes - Is Kubernetes the new Linux? Is the new Application Se...Effective Kubernetes - Is Kubernetes the new Linux? Is the new Application Se...
Effective Kubernetes - Is Kubernetes the new Linux? Is the new Application Se...
Wojciech Barczyński
 
Building Autonomous Operations for Kubernetes with keptn
Building Autonomous Operations for Kubernetes with keptnBuilding Autonomous Operations for Kubernetes with keptn
Building Autonomous Operations for Kubernetes with keptn
Johannes Bräuer
 
YEGOR MAKSYMCHUK «Using Kubernetes for organization performance tests»
YEGOR MAKSYMCHUK «Using Kubernetes for organization performance tests»YEGOR MAKSYMCHUK «Using Kubernetes for organization performance tests»
YEGOR MAKSYMCHUK «Using Kubernetes for organization performance tests»
QADay
 
From 0 to DevOps in 80 Days [Webinar Replay]
From 0 to DevOps in 80 Days [Webinar Replay]From 0 to DevOps in 80 Days [Webinar Replay]
From 0 to DevOps in 80 Days [Webinar Replay]
Dynatrace
 
Capacity Planning Infrastructure for Web Applications (Drupal)
Capacity Planning Infrastructure for Web Applications (Drupal)Capacity Planning Infrastructure for Web Applications (Drupal)
Capacity Planning Infrastructure for Web Applications (Drupal)
Ricardo Amaro
 
Chicago DevOps Meetup Nov2019
Chicago DevOps Meetup Nov2019Chicago DevOps Meetup Nov2019
Chicago DevOps Meetup Nov2019
Mike Villiger
 
Performance Testing in the Cloud
Performance Testing in the CloudPerformance Testing in the Cloud
Performance Testing in the Cloud
stuartmoncrieff
 
5 Habits of High-Velocity Teams Using Kubernetes
5 Habits of High-Velocity Teams Using Kubernetes5 Habits of High-Velocity Teams Using Kubernetes
5 Habits of High-Velocity Teams Using Kubernetes
Codefresh
 
Regain Control Thanks To Prometheus
Regain Control Thanks To PrometheusRegain Control Thanks To Prometheus
Regain Control Thanks To Prometheus
Etienne Coutaud
 
How to push a react js application in production and sleep better
How to push a react js application in production and sleep betterHow to push a react js application in production and sleep better
How to push a react js application in production and sleep better
Emanuele Rampichini
 
Running OpenStack in Production
Running OpenStack in ProductionRunning OpenStack in Production
Running OpenStack in Production
Tesora
 
Planning for a (Mostly) Hassle-Free Cloud Migration | VTUG 2016 Winter Warmer
Planning for a (Mostly) Hassle-Free Cloud Migration | VTUG 2016 Winter WarmerPlanning for a (Mostly) Hassle-Free Cloud Migration | VTUG 2016 Winter Warmer
Planning for a (Mostly) Hassle-Free Cloud Migration | VTUG 2016 Winter Warmer
Joe Conlin
 

Similar to Monitoring and alerting as code with Terraform and New Relic (20)

Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
 
Hewlett Packard Entreprise | Stormrunner load | Game Changer
Hewlett Packard Entreprise | Stormrunner load | Game ChangerHewlett Packard Entreprise | Stormrunner load | Game Changer
Hewlett Packard Entreprise | Stormrunner load | Game Changer
 
Using Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High ScalabilityUsing Grid Technologies in the Cloud for High Scalability
Using Grid Technologies in the Cloud for High Scalability
 
Keptn: Unbreakable Continuous Delivery - Berlin CI/CD Meetup
Keptn: Unbreakable Continuous Delivery - Berlin CI/CD MeetupKeptn: Unbreakable Continuous Delivery - Berlin CI/CD Meetup
Keptn: Unbreakable Continuous Delivery - Berlin CI/CD Meetup
 
Containerised Testing at Demonware : PyCon Ireland 2016
Containerised Testing at Demonware : PyCon Ireland 2016Containerised Testing at Demonware : PyCon Ireland 2016
Containerised Testing at Demonware : PyCon Ireland 2016
 
MLOps and Reproducible ML on AWS with Kubeflow and SageMaker
MLOps and Reproducible ML on AWS with Kubeflow and SageMakerMLOps and Reproducible ML on AWS with Kubeflow and SageMaker
MLOps and Reproducible ML on AWS with Kubeflow and SageMaker
 
Cloud-Native Fundamentals: Accelerating Development with Continuous Integration
Cloud-Native Fundamentals: Accelerating Development with Continuous IntegrationCloud-Native Fundamentals: Accelerating Development with Continuous Integration
Cloud-Native Fundamentals: Accelerating Development with Continuous Integration
 
Hands-on monitoring with Prometheus
Hands-on monitoring with PrometheusHands-on monitoring with Prometheus
Hands-on monitoring with Prometheus
 
Effective Kubernetes - Is Kubernetes the new Linux? Is the new Application Se...
Effective Kubernetes - Is Kubernetes the new Linux? Is the new Application Se...Effective Kubernetes - Is Kubernetes the new Linux? Is the new Application Se...
Effective Kubernetes - Is Kubernetes the new Linux? Is the new Application Se...
 
Building Autonomous Operations for Kubernetes with keptn
Building Autonomous Operations for Kubernetes with keptnBuilding Autonomous Operations for Kubernetes with keptn
Building Autonomous Operations for Kubernetes with keptn
 
YEGOR MAKSYMCHUK «Using Kubernetes for organization performance tests»
YEGOR MAKSYMCHUK «Using Kubernetes for organization performance tests»YEGOR MAKSYMCHUK «Using Kubernetes for organization performance tests»
YEGOR MAKSYMCHUK «Using Kubernetes for organization performance tests»
 
From 0 to DevOps in 80 Days [Webinar Replay]
From 0 to DevOps in 80 Days [Webinar Replay]From 0 to DevOps in 80 Days [Webinar Replay]
From 0 to DevOps in 80 Days [Webinar Replay]
 
Capacity Planning Infrastructure for Web Applications (Drupal)
Capacity Planning Infrastructure for Web Applications (Drupal)Capacity Planning Infrastructure for Web Applications (Drupal)
Capacity Planning Infrastructure for Web Applications (Drupal)
 
Chicago DevOps Meetup Nov2019
Chicago DevOps Meetup Nov2019Chicago DevOps Meetup Nov2019
Chicago DevOps Meetup Nov2019
 
Performance Testing in the Cloud
Performance Testing in the CloudPerformance Testing in the Cloud
Performance Testing in the Cloud
 
5 Habits of High-Velocity Teams Using Kubernetes
5 Habits of High-Velocity Teams Using Kubernetes5 Habits of High-Velocity Teams Using Kubernetes
5 Habits of High-Velocity Teams Using Kubernetes
 
Regain Control Thanks To Prometheus
Regain Control Thanks To PrometheusRegain Control Thanks To Prometheus
Regain Control Thanks To Prometheus
 
How to push a react js application in production and sleep better
How to push a react js application in production and sleep betterHow to push a react js application in production and sleep better
How to push a react js application in production and sleep better
 
Running OpenStack in Production
Running OpenStack in ProductionRunning OpenStack in Production
Running OpenStack in Production
 
Planning for a (Mostly) Hassle-Free Cloud Migration | VTUG 2016 Winter Warmer
Planning for a (Mostly) Hassle-Free Cloud Migration | VTUG 2016 Winter WarmerPlanning for a (Mostly) Hassle-Free Cloud Migration | VTUG 2016 Winter Warmer
Planning for a (Mostly) Hassle-Free Cloud Migration | VTUG 2016 Winter Warmer
 

Recently uploaded

DevOps Consulting Company | Hire DevOps Services
DevOps Consulting Company | Hire DevOps ServicesDevOps Consulting Company | Hire DevOps Services
DevOps Consulting Company | Hire DevOps Services
seospiralmantra
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
ICS
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
dakas1
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
Remote DBA Services
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
kgyxske
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Paul Brebner
 
Using Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query PerformanceUsing Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query Performance
Grant Fritchey
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
sjcobrien
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
Alina Yurenko
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
ISH Technologies
 
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
XfilesPro
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
ervikas4
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
ShulagnaSarkar2
 
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
Luigi Fugaro
 
Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we workMicroservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
Sven Peters
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Peter Muessig
 
Kubernetes at Scale: Going Multi-Cluster with Istio
Kubernetes at Scale:  Going Multi-Cluster  with IstioKubernetes at Scale:  Going Multi-Cluster  with Istio
Kubernetes at Scale: Going Multi-Cluster with Istio
Severalnines
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
Peter Muessig
 
Boost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management AppsBoost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management Apps
Jhone kinadey
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Julian Hyde
 

Recently uploaded (20)

DevOps Consulting Company | Hire DevOps Services
DevOps Consulting Company | Hire DevOps ServicesDevOps Consulting Company | Hire DevOps Services
DevOps Consulting Company | Hire DevOps Services
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
 
Using Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query PerformanceUsing Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query Performance
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
 
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
 
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
 
Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we workMicroservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
 
Kubernetes at Scale: Going Multi-Cluster with Istio
Kubernetes at Scale:  Going Multi-Cluster  with IstioKubernetes at Scale:  Going Multi-Cluster  with Istio
Kubernetes at Scale: Going Multi-Cluster with Istio
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
 
Boost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management AppsBoost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management Apps
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
 

Monitoring and alerting as code with Terraform and New Relic

  • 1. Monitoring and Alerting as Code with Terraform and New Relic http://bit.ly/terraform-newrelic January 30, 2019 Richard Bullington-McGuire Principal Architect, Modus Create richard@moduscreate.com @obscurerichard
  • 2.
  • 3. My New Relic & Terraform Experience 2014: Used New Relic to improve performance 40x in public safety training system - supported $MM exit - Java, Windows 2014-2017: Used New Relic to improve performance 10x in hospitality system - saving the client’s reputation - .NET, SQL Server, Windows, JMeter load tests 2015: Took over ProductShop project - Terraform 2017: Used Terraform & New Relic to migrate 14 critical systems to AWS for a large education company. 2017: Taught real estate information software firm how to do cloud migration right with Terraform and New Relic. 2018: Built out devops-infra-demo Terraform / Packer / Ansible / Jenkins
  • 4. Configuring monitoring by hand is fragile and tough Use Infrastructure as Code techniques to make it repeatable Load test to ensure that monitoring & alerting actually work Work around chicken or egg problems Monitoring and Alerting as Code
  • 5. The old way: Set up New Relic by hand on each server. Tweak configs until they match. Standing up a new environment or application? Copy and paste and pray Configuring monitoring by hand: just say no
  • 6. ● Set up your application stack and monitoring & alerting with the same toolkit ● Leverage tools such as Ansible to reduce boilerplate ● Use an image bakery pattern - consider immutable infrastructure or a hybrid pattern. ● Consider AWS CodeDeploy for legacy migration - easier than dockerizing everything! ● Use Jenkins or another CI process to drive the bakery Use Infrastructure as Code for repeatability
  • 7. ● ab is good, JMeter is better ● Use New Relic to monitor load tests and check validity ● Build a representative test resembling production peak load ● Run load tests through CI at scale or use BlazeMeter ● jmeter-ec2 or its forks can run massive load tests ○ But the code is ugly and not maintained much and you will have to hack it Load test to ensure that it all actually works
  • 8. ● Terraform can only define resources for items that exist ○ So how do you define monitoring for a stack that is not yet created? ● count to the rescue ● See this Gruntwork blog article on Terraform Tips & Tricks Work around chicken or egg problems
  • 9. Case Study: Terraform & New Relic & JMeter at work Driven by Jenkins CI
  • 10. Case Study: Terraform & New Relic & JMeter at work Driven by Jenkins CI ● Education company cloud migration (4mo -> prod) ● Apps w/> 30,000 RPM at peak measured with New Relic ● Production with 80+ sizeable AMIs baseline ● Auto Scaling to 200+ AMIs under heavy load ● Multiple environments: dev, qa, staging, prod ● Terabyte-scale MySQL Aurora cluster, 50+ TB in S3 ● Jenkins, Terraform, Ansible, Packer, CodeDeploy, JMeter load tests, New Relic monitoring
  • 13. CI Environment Jenkins Elastic Load Balancer EC2 Auto Scaling Group - Web App Terraform Provision Packer Provision
  • 15. Credit where Credit is Due ● Sergio Bruder (@sdbruder) built the JMeter test harness and CodeDeploy app used in the demo ● Don Anderson (@dranderson) integrated New Relic APM, Infrastructure, and Dashboards for this demo I could not have done this without their help!