SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engineering

•Download as PPTX, PDF•

2 likes•321 views

How often have you heard stories where someone thought they had a disaster strategy, never tested it and it fails when you need it the most? LinkedIn has evolved from serving live traffic out of one data center to four data centers spread geographically. Serving live traffic from four data centers at the same time has taken the company from a disaster recovery model to a disaster avoidance model, where an unhealthy data center can be taken out of rotation and its traffic redistributed to the healthy data centers within minutes, with virtually no visible impact to users. As LinkedIn transitioned from big monolithic applications to microservices, it was difficult to determine capacity constraints of individual services to handle extra load during disaster scenarios. Stress testing individual services using artificial load in a complex microservices architecture wasn’t sufficient to provide enough confidence in data center’s capacity. To solve this problem, LinkedIn moves live traffic to services site-wide by shifting traffic between datacenters to simulate a disaster every business day!

Engineering

Michael Kehoe
/USR/BIN/WHOAMI
• Staff Site Reliability Engineer @ LinkedIn
• Production-SRE Team
• Funny accent = Australian + 4 years
American
• Former Network Engineer at the
University of Queensland

Who are we?
PRODUCTION-SRE TEAM AT LINKEDIN
• Disaster Recovery Planning and
Automation
• Incident Response and Automation
• Visibility Engineering
• Reliability Principles

LinkedIn
EVOLUTION OF THE INFRASTRUCTURE
2003 2010 2011 2013 2014 2015
Active &
Passive
Active &
Active
Multi-colo 3-
way Active &
Active
Multi-colo n-
way Active &
Active

LinkedIn
2018
4 Data Centers 21 PoPs 1000+ services

What is Resilience Engineering?
• Projects that directly demand increased
resilience from our applications and
infrastructure.
• Application Injection Failure
• Infrastructure Injection Failure
• Full Disaster-Recovery Tests

How often have you heard stories where someone
thought they had a disaster strategy, never tested it and
it fails when you need it the most?

Problem Statement
• How do we ensure that we always have
disaster recovery ability without incident?
• How do we consistently test for disaster
recovery ability without disrupting the
company?

Project Overview
1
• Build a process (with Automation) to facilitate disaster recovery
• Operate the process on regular cadence
• Provide reporting on outcomes of tests with engineering executives

What is Load Testing?
5x a week Peak hour traffic Fixed SLA

LinkedIn Traffic-Tier
Border
Router IPVS ATS ATS Frontend
EDGE FABRIC
Stickyrouting

LinkedIn Traffic-Tier
Fabric
Buckets
1
91
2 3 10
92 93 100

LinkedIn Traffic-Tier
EDGE FABRIC
DC1
DC2
DC1 in Cookie
Got DC2 as secondary fabric
Gets
secondary
fabric for userStickyrouting

TrafficShift Architecture
Web
application
Salt master
Stickyrouting
ServiceCouchbase Backend Worker
Processes
FABRIC
BUCKETS

Load Testing
FABRIC
DC3
DC1 DC2
60%
Traffic
Percentage

Benefits of Load-testing
Capacity
Planning
Identify Bugs Confidence

Benefits of Load-testing
CAPACITY PLANNING
• Through this process, we continuously validate our infrastructure
capacity
• This is the best signal we can possibly get since we’re simulating a
real disaster

Benefits of Load-testing
IDENTIFY BUGS
2
• Some bugs are only found at high load (under duress)
• Helps find inefficiency’s that otherwise may not be found until it’s too late
• Gives us clues on how to make our code more resilient to potential failure

Benefits of Load-testing
CONFIDENCE
2
• Through load-testing, we’ve built confidence in our disaster recovery
strategy
• We understand exactly:
• What process to follow
• How long it takes to avert disaster
• What are the risks associated with a disaster incident

Key Takeaways
• Resilience Engineering is a must for
LinkedIn
• Design infrastructure to facilitate disaster
recovery
• Disaster-test regularly to avoid surprises
• Automate your testing/ process to reduce
engagement time

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engineering

Much attention is focused to the technology aspects of DevOps, including how to automate security testing. But DevOps is as much a cultural movement as anything else, with a strong focus on feedback loops and continuous improvement. How can organizations implement these aspects of DevOps culture when integrating security, given the massive shortfall in skilled information security personnel? This talk discusses organizational and cultural aspects of DevOps with an emphasis on the role of “security champions”—developers cross-trained in information security basics—in executing a successful DevSecOps transformation.

Continuous Delivery

Mike McGarr

Security & DevOps- Ways To Make Sure Your Apps & Infrastructure Are Secure

Puppet

This webinar on DevOps and security will cover the definition of DevOps, common security challenges with the DevOps model, and how to take a "SecDevOps" approach to embed security into the development process. The presenters will discuss recommendations like increasing trust between development and security teams, using a continuous delivery pipeline to incrementally improve security, and including security as acceptance criteria for user stories. Questions from attendees will be answered at the end.

DevSecOps and the CI/CD Pipeline

James Wickett

All organizations want to go faster and decrease friction in their cloud software delivery pipeline. Infosec has an opportunity to change their classic approach from blocker to enabler. This talk will discuss hallmarks of CI/CD and some practical examples for adding security testing across different organizations. The talk will cover emergent patterns, practices and toolchains that bring security to the table. Presented at OWASP NoVA, Sept 25th, 2018

NYIT DSC/ Spring 2021 - Introduction to DevOps (CI/CD)

Hui (Henry) Chen

DevSecCon London 2017: How far left do you want to go with security? by Javie...

DevSecCon

The document discusses how security practices can be integrated further left into the software development lifecycle using a DevSecOps approach. It outlines how traditional security operated in silos separate from development. DevSecOps aims to break down these silos by integrating security activities like policy, reviews, and audits directly into development practices like coding, continuous integration, and deployment. This is achieved through automation and tools as well as collaborative environments to bring together stakeholders from security, development, and operations. The document concludes that a DevSecOps approach following principles from Agile and DevOps can help redefine security and establish new baselines.

Security As A Code :

ทวิร พานิชสมบัติ

Security testing has traditionally been siloed from development, operations, and quality assurance teams. However, a new approach called "Security as a Code" aims to break down these barriers by integrating security directly into coding practices similar to how unit testing, infrastructure as code, and acceptance testing have previously integrated other functions. This would allow security to work in parallel with development rather than as a separate phase, improving feedback cycles and collaboration between teams.

QA in DevOps: Transformation thru Automation via Jenkins

Tatyana Kravtsov

This document outlines the agenda for a Jenkins World Tour 2015 presentation in Washington D.C. on QA in DevOps through automation using Jenkins. The presentation discusses the definition of DevOps and provides a 10 step process to DevOps transformation focusing on continuous integration, automated testing, code quality metrics, environment testing, and automated reporting. The presenter is Tanya Kravtsov, founder of the DevOpsQA NJ Meetup group.

As companies have adopted faster development methodologies a new constraint has emerged in the journey to digital transformation: data. Data has long been the neglected discipline, the weakest link in the tool chain, with provisioning times still counted in days, weeks, or even months. In addition, most companies are still using decades-old processes to manage and deploy database changes, further anchoring development teams.

ATAGTR2017 Security Testing / IoT Testing in Real World

Agile Testing Alliance

DevOps the Big Picture for Testers by Joseph Ours

QA or the Highway

DevOps is a set of lean development principles, delivery automation tools, and cloud practices that helps technology teams solve challenges around delivery speed and cost containment. DevOps provides visibility, enhances productivity, improves quality, and increases delivery cadence. DevOps touches every aspect of IT delivery including architecture, continuous deployment, delivery, operations, testing, integration, infrastructure, security, culture, and planning. For testers, DevOps presents opportunities to contribute to increased velocity through automation skills and incremental testing approaches, while being ready to take on more testing responsibilities.

Continuous Testing in DevOps

TechWell

This document provides an overview of continuous testing in DevOps presented by Michael Hackett of LogiGear Corporation. It discusses what DevOps is and how testing fits into the continuous delivery pipeline. It emphasizes the need for testing at every step of development and deployment through test automation. It also provides examples of challenges companies face in implementing continuous testing and DevOps practices.

Scaling Enterprise DevOps with CloudBees

DevOps.com

This document discusses how CloudBees Jenkins Enterprise can help organizations achieve continuous delivery at scale. It introduces CloudBees Jenkins Enterprise as a scalable, secure, and manageable continuous delivery platform built on a cloud-native architecture. Key benefits highlighted include elastic scalability, multi-tenancy, pipeline templates, monitoring and alerting, role-based access control, and resiliency capabilities like automated failover and backups. Personas of different roles within an organization are presented along with their challenges that CloudBees Jenkins Enterprise aims to address.

HOW TO OPTIMIZE NON-CODING TIME, ORI KEREN, LinearB

DevOpsDays Tel Aviv

Github Copilot and tools that help us code better are cool. But I’m lucky if I spend 90 minutes a day writing code. We really need to optimize the hours we spend reviewing code, updating tickets and tracing where our code is deployed. Learn how I save an hour a day streamlining non-coding tasks. This talk is unique because 99% of developer productivity tools and hacks are about coding faster, better, smarter. And yet the vast majority of our time is spent doing all of this other stuff. After I started focusing on optimizing the 10 hours I spend every day on non-coding tasks, I found I my productivity went up and my frustration at annoying stuff went way down. I cover how to save time by reducing cognitive load and by cutting menial, non-coding tasks that we have to perform 10-50 times every day. For example: Bug or hotfix comes through and you want to start working on it right away so you create a branch and start fixing. What you don’t do is create a Jira ticket but then later your boss/PM/CSM yells at your due to lack of visibility. I share how I automated ticket creation in Slack by correlating Github to Jira. You have 20 minutes until your next meeting and you open a pull request and start a review. But you get pulled away half way through and when you come back the next day you forgot everything and have to start over. Huge waste of time. I share an ML job I wrote that tells me how long the review will take so I can pick PRs that fit the amount of time I have. You build. You ship it. You own it. Great. But after I merge my code I never know where it actually is. Did the CI job fail? Is it release under feature flag? Did it just go GA to everyone? I share a bot I wrote that personally tells me where my code is in the pipeline after it leaves my hands so I can actually take full ownership without spending tons of time figuring out what code is in what release.

Building DevOps Toolchain

IBM UrbanCode Products

DevOps and All the Continuouses w/ Helen Beal

Sonatype

DevOps promises to make better software faster and more safely and many organizations begin by practicing Continuous Integration and moving on to Continuous Delivery and sometimes even extending as far as Continuous Deployment - but this is only the tip of the iceberg. DevOps demands a fundamental shift in the way we work and requires all participants in an organization to live its principles. It’s much more than a tool chain. When you are delivering software in an Agile manner in fortnightly sprints, are you still funding in an annual manner? Are you adhering to The Third Way? I.e. are you practicing Continuous Experimentation? Continuous Learning? How are you doing Continuous Testing? Are you including security in that? Have you have Continuous Improvement in your organization for years? When does Continuous Everything turn into Continuous Apathy?

Reliability (R)evolution: Turning the DevOps World Upside Down (Again).

Hannes Lenke

Join Mirko Novakovic, Co-founder and CEO of Instana and Hannes Lenke, CEO of Checkly as they discuss the Reliability (R)evolution. Hear how freeing DevOps teams from complexity, empowers them to scale and accelerate with end-to-end testing and monitoring. Testing and monitoring was previously seen as slow, flaky, and costly, but this is no longer the case. Technologies like Headless, Puppeteer, and Jamstack are changing the way we ensure reliability. Combined with the ability to integrate APM into testing, you can also monitor the transactions simulated. This enables DevOps teams to push into production quicker with more reliability.

The Art of Container Monitoring

Derek Chen

According to service scale, there are hundreds or thousands of running containers in your service. Should we monitor each container by microscope or monitor each microservice by magnifier? This depends which granularity can help us find and solve the problems. In this sharing, I will introduce how to use cAdvisor, Icinga2, InfluxDB and Grafana to build a self-hosted monitoring system. In addition, I also discuss with how to embrace open source and share some practical experiences.

Drive Continuous Delivery With Continuous Testing

CA Technologies

Silos. Lack of visibility. Some agile teams… some not. Manual handoffs. Bottlenecks. This summer, it’s time to get outside (your old processes) and take some time off (your application release cycle). Take back your weekends and spend more time by the pool. We’ll show you how to automate, orchestrate, and facilitate continuous everything – and that includes continuous testing – one of the biggest bottlenecks of all. You’ll learn how to: Automatically shift quality left: Orchestrate and automate testing in every phase of the SDLC with automated promotion and feedback loops Accelerate testing in the cloud: Test web and mobile apps in parallel – achieve up to 10X improvement in testing time. Use tools of choice while optimizing every aspect of your complex, interdependent multi-application pipelines. Get started in less than 1 hour…. and for free! Achieve truly automated, continuous delivery (including continuous testing!!!) in the cloud with CA and Sauce Labs. Try Continuous Delivery Director free: https://cddirector.io/#/home Try Sauce Labs free: https://saucelabs.com/

A True Story of Why QA Loves DevOps

IBM UrbanCode Products

Kku2011

ทวิร พานิชสมบัติ

This document discusses software development methodologies like Waterfall and Agile. It emphasizes that while Waterfall is still dominant, Agile is better suited for small organizations because it focuses on delivering working software frequently. It also stresses the importance of good programming practices like using intention-revealing names, avoiding side effects, and test-driven development (TDD) to produce understandable and testable code. Finally, it outlines various tools that can be used for project management, version control, testing, continuous integration, and monitoring in the development process.

Secure your Azure and DevOps in a smart way

Eficode

¿Qué es DevOps y por qué es importante en el Ciclo de Software? por michelada.io

Software Guru

Engineering Trust in Your Automated Tests

Jyoti Mittal

Automated testing is important for achieving speed and quality in software development. However, flaky tests can undermine trust in test results. The document discusses strategies for addressing flaky tests, including quarantining them, understanding their causes, improving test code quality, and rewriting tests if needed. The goal is to create reliable automated tests that build trust in the testing process.

Where Testers & QA Fit in the Story of DevOps

QASymphony

Where Testers and QA Fit in the Story of DevOps Continuous delivery. CI. GitHub. Scrum. CD. Jenkins. Continuous testing. Continuous integration. These are just some terms that are supposed to describe the word soup that is DevOps. Chances are that you have heard some or all of these words being passed around at your daily stand ups or company meetings. However, where does QA and testing fit into the story of DevOps? Some would say that developers and operations teams are all you need for a successful DevOps pipeline, while others show that Dev, Test and Ops need to be included to ensure quality at every step in your pipeline. In this webinar, Ryan Yackel, QASymphony's Director of Product Marketing, and Sunil Sehgal, Managing Partner at TechArcis, will share their experiences as they navigate you through the DevTestOps waters. In this webinar you will learn: Overview of the State of DevOps Common misconceptions of DevOps and QA How testers must adapt to the DevOps process The tools testers need for continuous testing Can't make the webinar? Sign up and we will send you the recording.

Designing for the internet - Page Objects for the Real World

Qualitest

We explored Page Object design pattern to some of the more common, and sometimes frustrating, object configurations found on the internet. Learn how proper application of this pattern enables you to leverage Selenium’s power to produce concise, readable, and maintainable automated tests. We tackled challenging DOM configurations such as Messy tables Frames Random identifiers Third part frameworks like JQuery and Moment HTML5 video players and more with Java and Selenium 3. Learn how solving these tricky problems with the correct techniques leads to more robust tests while saving scripting time! For more information, please visit www.QualiTestGroup.com

Why Serverless is scary without DevSecOps and Observability

Eficode

This document discusses the security challenges of serverless computing without proper DevSecOps practices and observability. It notes that serverless applications are often seen as more secure since the cloud provider manages the infrastructure, but they can still be vulnerable to events, libraries, and code issues. The document recommends implementing DevSecOps with a focus on permissions, security analysis, public scrutiny of practices, and crowdsourcing security through bug bounties and hackathons. It also stresses the importance of observability tools to monitor serverless applications and catch issues.

The Next Wave of Reliability Engineering

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engineering

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engineering

Similar to SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engineering (20)

More from Michael Kehoe

More from Michael Kehoe (20)

Recently uploaded

Recently uploaded (20)

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engineering

Editor's Notes