Building Resilience: How Outages Shaped Etsy's Systems

•

0 likes•673 views

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1iyFyTo. Avleen Vig presents some of the most unexpected, confusing, hilarious and face-palming events during Etsy's outages to show what can be learnt from their problems to build more resilient systems. Filmed at qconlondon.com. Avleen Vig is a Staff Operations Engineer at Etsy, where he spends much of his time growing the infrastructure for selling knitted gloves and cross-stitch periodic tables. Before joining Etsy he worked at several large tech companies, including EarthLink and Google, as well as a number of small successful startups.

Technology Business

InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/resiliency-etsy

Presented at QCon London
www.qconlondon.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

Building resilience
How outages shaped Etsy’s systems

Quick! Be resilient!
http://www.ﬂickr.com/photos/niaid/11854196633/sizes/l/

Quick! Be resilient!
• Actually, it’s a slow process
• Iterative
• Introspective
• Horizontal and vertical
development

Quick! Be resilient!
http://www.ﬂickr.com/photos/ogcodes/6091644301/sizes/l/

Quick! Be resilient!
http://www.ﬂickr.com/photos/studio360/1150744342/sizes/o/

Quick! Be resilient!
http://www.ﬂickr.com/photos/studio360/1150744368/sizes/o/

Quick! Be resilient!
Current generation
Next generation

Quick! Be resilient!
http://www.ﬂickr.com/photos/jurvetson/8671257096/

Quick! Be resilient!
http://cudebi.wordpress.com/2012/09/19/tah-pagh-tahbe-o-el-reconocimiento-de-william-shakespeare-en-el-universo-de-star-trek/

Resilience Engineering
http://www.ﬂickr.com/photos/freefoto/728651045/sizes/o/

Resilience Engineering
• “To Engineer is Human” 
“To Forgive Design” 
- Henry Petroski
• “The Field Guide to Understanding Human Error” 
“Just Culture” 
- Sidney Dekker

Building resilience at Etsy
• Continuous deployment
• Metrics, metrics, metrics
• Peer review
• Postmortems

Building resilience at Etsy
• Continuous deployment
• Metrics, metrics, metrics
• Peer review
• Postmortems
}Culture

• No blame
• Open discussion
• Focus on improvements
Constructive cultures

• No blame
• Open discussion
• Focus on improvements
}Culture
Constructive cultures

–Japanese proverb
“The nail that sticks up, 
gets hammered down”
Destructive cultures

• #23: Fortune’s “Top 50 best small and medium
businesses to work for”
• Rapid code iterations and deploys
• Lasting relationships
• Generousity of spirit
• …and much more

Doing postmortems?
Get Morgue
http://github.com/etsy/morgue

Forkistan
• Mean time to detect: 0 min
• Mean time to recover: 10 mins

Yo Dawg, I Heard You
Like Errors..
• Mean time to detect: 2 mins
• Mean time to recover: 15 mins

Smashing INT for Fun
and Proﬁt
• Mean time to detect: 0 min
• Mean time to recover: 4 hrs 52 mins

Apache Amnesia
• Mean time to detect: 2 hours
• Mean time to recover: 5 mins

Continuously Upgrading
Databases
• Mean time to detect: 2 mins
• Mean time to recover: 1 hour (but, not really..)

Q & A
Avleen Vig
Staﬀ Operations Engineer
Etsy, Inc
@avleen

Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/resiliency
-etsy

Viewers also liked

Digital Sellsheet Vn

Vincent Nelson

Me em

miltonpaucaraticona

Martha López Pimentel

Be mean and sing powder 348002

Tony Burt

Hitung pph

Mukhliansyah

Libre office

Michelle1701

$H:\Soft Media\Software Media$ $H:\Soft Media\Software Media$

H:\Soft Media\Software Media

Nicte Garcia

Apres.Ppoint.Orient.Didat.Prof.CiêNcias.Op2

Albano Novaes

Viewers also liked (8)

Digital Sellsheet Vn

Me em

Be mean and sing powder 348002

Hitung pph

Libre office

$H:\Soft Media\Software Media$ $H:\Soft Media\Software Media$

H:\Soft Media\Software Media

Apres.Ppoint.Orient.Didat.Prof.CiêNcias.Op2

More from C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/39NIjLV. Akhilesh Gupta does a technical deep-dive into how Linkedin uses the Play/Akka Framework and a scalable distributed system to enable live interactions like likes/comments at massive scale at extremely low costs across multiple data centers. Filmed at qconlondon.com. Akhilesh Gupta is the technical lead for LinkedIn's Real-time delivery infrastructure and LinkedIn Messaging. He has been working on the revamp of LinkedIn’s offerings to instant, real-time experiences. Before this, he was the head of engineering for the Ride Experience program at Uber Technologies in San Francisco.

Streaming a Million Likes/Second: Real-Time Interactions on Live Video

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2x0Fav8. Jose Nino guides the audience through the journey of Mobile APIs at Lyft. He focuses on how the team has reaped the benefits of API generation to experiment with the network transport layer. He also discusses recent developments the team has made with Envoy Mobile and the roadmap ahead. Filmed at qconlondon.com. Jose Nino works as a Software Engineer at Lyft.

Next Generation Client APIs in Envoy Mobile

C4Media

How do we cope with an environment that has been radically disrupted, where people are suddenly thrust into remote work in a chaotic state? What are the emerging good practices and new ideas that are shaping the way in which software development teams work? What can we do to make the workplace a more secure and diverse one while increasing the productivity of our teams? This report aims to assist technical leaders in making mid- to long-term decisions that will have a positive impact on their organisations and teams and help individual contributors find the practices, approaches, tools, techniques, and frameworks that can help them get a better experience at work - irrespective of where they are working from.

Software Teams and Teamwork Trends Report Q1 2020

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2QCmmJ0. Mark Stoodley examines some of the strengths and weaknesses of the different Java compilation technologies, if one was to apply them in isolation. Stoodley discusses how production JVMs are assembling a combination of these tools that work together to provide excellent performance across the large spectrum of applications written in Java and JVM based languages. Filmed at qconsf.com. Mark Stoodley joined IBM Canada to build Java JIT compilers for production use and led the team that delivered AOT compilation in the IBM SDK for Java 6. He spent the last five years leading the effort to open source nearly 4.3 million lines of source code from the IBM J9 Java Virtual Machine to create the two open source projects Eclipse OMR and Eclipse OpenJ9, and now co-leads both projects.

Understand the Trade-offs Using Compilers for Java Applications

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2y2yPiS. Colin McCabe talks about the ongoing effort to replace the use of Zookeeper in Kafka: why they want to do it and how it will work. He discusses the limitations they have found and how Kafka benefits both in terms of stability and scalability by bringing consensus in house. He talks about their progress, what work is remaining, and how contributors can help. Filmed at qconsf.com. Colin McCabe is a Kafka committer at Confluent, working on the scalability and extensibility of Kafka. Previously, he worked on the Hadoop Distributed Filesystem and the Ceph Filesystem.

Kafka Needs No Keeper

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2SXXXiD. Katharina Probst talks about what it means to act like an owner and why teams need ownership to be high-performing. When team members, regardless of whether they have a formal leadership role or not, act like owners, magical things can happen. She shares ideas that we can apply to our own work, and talks about how to recognize when we don’t live up to our own expectations of acting like an owner. Filmed at qconsf.com. Katharina Probst is a Senior Engineering Leader, Kubernetes & SaaS at Google. Before this, she was leading engineering teams at Netflix, being responsible for the Netflix API, which helps bring Netflix streaming to millions of people around the world. Prior to joining Netflix, she was in the cloud computing team at Google, where she saw cloud computing from the provider side.

High Performing Teams Act Like Owners

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2T04Lw4. Sergey Kuksenko talks about the performance benefits inline types bring to Java and how to exploit them. Inline/value types are the key part of experimental project Valhalla, which should bring new abilities to the Java language. Filmed at qconsf.com. Sergey Kuksenko is a Java Performance Engineer at Oracle working on a variety of Java and JVM performance enhancements. He started working as Java Engineer in 1996 and as Java Performance Engineer in 2005. He has had a passion for exploring how Java works on modern hardware.

Does Java Need Inline Types? What Project Valhalla Can Bring to Java

C4Media

Do you need service meshes in your tech stack? This on-line guide aims to answer pertinent questions for software architects and technical leaders, such as: what is a service mesh?, do I need a service mesh?, how do I evaluate the different service mesh offerings? In software architecture, a service mesh is a dedicated infrastructure layer for facilitating service-to-service communications between microservices, often using a sidecar proxy.

Service Meshes- The Ultimate Guide

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2UgQ3BU. Christie Wilson describes what to expect from CI/CD in 2019, and how Tekton is helping bring that to as many tools as possible, such as Jenkins X and Prow. Wilson talks about Tekton itself and performs a live demo that shows how cloud native CI/CD can help debug, surface and fix mistakes faster. Filmed at qconsf.com. Christie Wilson is a software engineer at Google, currently leading the Tekton project. Over the past decade, she has worked in the mobile, financial and video game industries. Prior to working at Google she led a team of software developers to build load testing tools for AAA video game titles, and founded the Vancouver chapter of PyLadies.

Shifting Left with Cloud Native CI/CD

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2S7lDiS. Sasha Rosenbaum shows how a CI/CD pipeline for Machine Learning can greatly improve both productivity and reliability. Filmed at qconsf.com. Sasha Rosenbaum is a Program Manager on the Azure DevOps engineering team, focused on improving the alignment of the product with open source software. She is a co-organizer of the DevOps Days Chicago and the DeliveryConf conferences, and recently published a book on Serverless computing in Azure with .NET.

CI/CD for Machine Learning

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/36epVKg. Todd Montgomery discusses the techniques and lessons learned from implementing Aeron Cluster. His focus is on how Raft can be implemented on Aeron, minimizing the network round trip overhead, and comparing single process to a fully distributed cluster. Filmed at qconsf.com. Todd Montgomery is a networking hacker who has researched, designed, and built numerous protocols, messaging-oriented middleware systems, and real-time data systems, done research for NASA, contributed to the IETF and IEEE, and co-founded two startups. He currently works as an independent consultant and is active in several open source projects.

Fault Tolerance at Speed

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2FWc5Sk. Ben Sigelman talks about "Deep Systems", their common properties and re-introduces the fundamentals of control theory from the 1960s, including the original conceptualizations of Observability & Controllability. He uses examples from Google & other companies to illustrate how deep systems have damaged people's ability to observe software, and what needs to be done in order to regain control. Filmed at qconsf.com. Ben Sigelman is a co-founder and the CEO at LightStep, a co-creator of Dapper (Google’s distributed tracing system), and co-creator of the OpenTracing and OpenTelemetry projects (both part of the CNCF). His work and interests gravitate towards observability, especially where microservices, high transaction volumes, and large engineering organizations are involved.

Architectures That Scale Deep - Regaining Control in Deep Systems

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/39SddUL. Victor Dibia provides a friendly introduction to machine learning, covers concrete steps on how front-end developers can create their own ML models and deploy them as part of web applications. He discusses his experience building Handtrack.js - a library for prototyping real time hand tracking interactions in the browser. Filmed at qconsf.com. Victor Dibia is a Research Engineer with Cloudera’s Fast Forward Labs. Prior to this, he was a Research Staff Member at the IBM TJ Watson Research Center, New York. His research interests are at the intersection of human computer interaction, computational social science, and applied AI.

ML in the Browser: Interactive Experiences with Tensorflow.js

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2s9T3Vl. Colin Eberhardt looks at some of the internals of WebAssembly, explores how it works “under the hood”, and looks at how to create a (simple) compiler that targets this runtime. Filmed at qconsf.com. Colin Eberhardt is the Technology Director at Scott Logic, a UK-based software consultancy where they create complex application for their financial services clients. He is an avid technology enthusiast, spending his evenings contributing to open source projects, writing blog posts and learning as much as he can.

Build Your Own WebAssembly Compiler

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2S9tOgy. Satyajit Thadeshwar provides useful insights on how Netflix implemented a secure, token-agnostic, identity solution that works with services operating at a massive scale. He shares some of the lessons learned from this process, both from architectural diagrams and code. Filmed at qconsf.com. Satyajit Thadeshwar is an engineer on the Product Edge Access Services team at Netflix, where he works on some of the most critical services focusing on user and device authentication. He has more than a decade of experience building fault-tolerant and highly scalable, distributed systems.

User & Device Identity for Microservices @ Netflix Scale

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2Ezs08q. Justin Ryan talks about Netflix’ scalability issues and some of the ways they addressed it. He shares successes they’ve had from unintuitively partitioning computation into multiple services to get better runtime characteristics. He introduces us to useful probabilistic data structures, innovative bi-directional data passing, open-source projects available from Netflix that make this all possible. Filmed at qconsf.com. Justin Ryan is Playback Edge Engineering at Netflix. He works on some of the most critical services at Netflix, specifically focusing on user and device authentication. Years of building developer tools has also given him a healthy set of opinions on developer productivity.

Scaling Patterns for Netflix's Edge

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2Z4ZJjn. Kilian Valkhof discusses the process of making an Electron app feel at home on all three platforms: Windows, MacOS and Linux, making devs aware of the pitfalls and how to avoid them. Filmed at qconsf.com. Kilian Valkhof is a Front-end Developer & User-experience Designer at Firstversionist. He writes about various topics, from design to machine learning, on his personal website, kilianvalkhof.com and is a frequent contributer to open source software. He is part of the Electron governance team that oversees the development of the Electron framework.

Make Your Electron App Feel at Home Everywhere

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/344PnB1. Steve Klabnik goes over the deep details of how async/await works in Rust, covering concepts like coroutines, generators, stack-less vs stack-ful, "pinning", and more. Filmed at qconsf.com. Steve Klabnik is on the core team of Rust, leads the documentation team, and is an author of "The Rust Programming Language." He is a frequent speaker at conferences and is a prolific open source contributor, previously working on projects such as Ruby and Ruby on Rails.

The Talk You've Been Await-ing For

C4Media

Future of Data Engineering

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2rm4hFD. Yevgeniy Brikman talks about how to write automated tests for infrastructure code, including the code written for use with tools such as Terraform, Docker, Packer, and Kubernetes. Topics covered include: unit tests, integration tests, end-to-end tests, dependency injection, test parallelism, retries and error handling, static analysis, property testing and CI / CD for infrastructure code. Filmed at qconsf.com. Yevgeniy Brikman is the co-founder of Gruntwork, a company that provides DevOps as a Service. He is the author of two books published by O'Reilly Media: Hello, Startup and Terraform: Up & Running. Previously, he worked as a software engineer at LinkedIn, TripAdvisor, Cisco Systems, and Thomson Financial.

Automated Testing for Terraform, Docker, Packer, Kubernetes, and More

C4Media

More from C4Media (20)

Streaming a Million Likes/Second: Real-Time Interactions on Live Video

Next Generation Client APIs in Envoy Mobile

Software Teams and Teamwork Trends Report Q1 2020

Understand the Trade-offs Using Compilers for Java Applications

Kafka Needs No Keeper

High Performing Teams Act Like Owners

Does Java Need Inline Types? What Project Valhalla Can Bring to Java

Service Meshes- The Ultimate Guide

Shifting Left with Cloud Native CI/CD

CI/CD for Machine Learning

Fault Tolerance at Speed

Architectures That Scale Deep - Regaining Control in Deep Systems

ML in the Browser: Interactive Experiences with Tensorflow.js

Build Your Own WebAssembly Compiler

User & Device Identity for Microservices @ Netflix Scale

Scaling Patterns for Netflix's Edge

Make Your Electron App Feel at Home Everywhere

The Talk You've Been Await-ing For

Future of Data Engineering

Automated Testing for Terraform, Docker, Packer, Kubernetes, and More

Recently uploaded

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Manulife - Insurer Innovation Award 2024

The Digital Insurer

MINDCTI Revenue Release Quarter One 2024

MIND CTI

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Increase engagement and revenue with Muvi Live Paywall! In this presentation, we will explore the five key benefits of using Muvi Live Paywall to monetize your live streams. You'll learn how Muvi Live Paywall can help you: Monetize your live content easily: Set up pay-per-view access to your live streams and start generating revenue from your content. Increase audience engagement: Provide exclusive, premium content behind the paywall to keep your viewers engaged. Gain valuable viewer insights: Track viewer data and analytics to better understand your audience and tailor your content accordingly. Reduce content piracy: Muvi Live Paywall's security features help protect your content from unauthorized distribution. Streamline your workflow: The all-in-one platform simplifies the process of managing and monetizing your live streams. With Muvi Live Paywall, you can take control of your live stream monetization and create a sustainable business model for your content. Learn more about Muvi Live Paywall and start generating revenue from your live streams today!

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Roshan Dwivedi

Scaling API-first – The story of a global engineering organization

Radu Cotescu

This presentation explores the impact of HTML injection attacks on web applications, detailing how attackers exploit vulnerabilities to inject malicious code into web pages. Learn about the potential consequences of such attacks and discover effective mitigation strategies to protect your web applications from HTML injection vulnerabilities. for more information visit https://bostoninstituteofanalytics.org/category/cyber-security-ethical-hacking/

HTML Injection Attacks: Impact and Mitigation Strategies

Boston Institute of Analytics

Recently uploaded (20)

Boost Fertility New Invention Ups Success Rates.pdf

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

2024: Domino Containers - The Next Step. News from the Domino Container commu...

AWS Community Day CPH - Three problems of Terraform

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Partners Life - Insurer Innovation Award 2024

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

How to Troubleshoot Apps for the Modern Connected Worker

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Manulife - Insurer Innovation Award 2024

MINDCTI Revenue Release Quarter One 2024

Artificial Intelligence Chap.5 : Uncertainty

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Scaling API-first – The story of a global engineering organization

HTML Injection Attacks: Impact and Mitigation Strategies

Building Resilience: How Outages Shaped Etsy's Systems

2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /resiliency-etsy

3. Presented at QCon London www.qconlondon.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide

4. Building resilience How outages shaped Etsy’s systems

5. Act 1

6. Quick! Be resilient! http://www.ﬂickr.com/photos/niaid/11854196633/sizes/l/

7. Quick! Be resilient! • Actually, it’s a slow process • Iterative • Introspective • Horizontal and vertical development

8. Quick! Be resilient! http://www.ﬂickr.com/photos/ogcodes/6091644301/sizes/l/

9. Quick! Be resilient! http://www.ﬂickr.com/photos/studio360/1150744342/sizes/o/

10. Quick! Be resilient! http://www.ﬂickr.com/photos/studio360/1150744368/sizes/o/

11. Quick! Be resilient! http://www.ﬂickr.com/photos/ogcodes/6091644301/sizes/l/

12. Quick! Be resilient! Current generation Next generation

13. Quick! Be resilient! http://www.ﬂickr.com/photos/jurvetson/8671257096/

14. Quick! Be resilient! http://cudebi.wordpress.com/2012/09/19/tah-pagh-tahbe-o-el-reconocimiento-de-william-shakespeare-en-el-universo-de-star-trek/

15. Resilience Engineering http://www.ﬂickr.com/photos/freefoto/728651045/sizes/o/

16. Resilience Engineering • “To Engineer is Human”  “To Forgive Design”  - Henry Petroski • “The Field Guide to Understanding Human Error”  “Just Culture”  - Sidney Dekker

17. Act 2

18. Building resilience at Etsy • Continuous deployment • Metrics, metrics, metrics • Peer review • Postmortems

19. Building resilience at Etsy • Continuous deployment • Metrics, metrics, metrics • Peer review • Postmortems }Culture

20. Or: How to win at failing Postmortems

21. • No blame • Open discussion • Focus on improvements Constructive cultures

22. • No blame • Open discussion • Focus on improvements }Culture Constructive cultures

23. –Japanese proverb “The nail that sticks up,  gets hammered down” Destructive cultures

24. The result?

25. • #23: Fortune’s “Top 50 best small and medium businesses to work for” • Rapid code iterations and deploys • Lasting relationships • Generousity of spirit • …and much more

26. Act 3

27. Doing postmortems? Get Morgue http://github.com/etsy/morgue

28. Morgue

29. Morgue

30. Morgue

31. Forkistan • Mean time to detect: 0 min • Mean time to recover: 10 mins

32. Yo Dawg, I Heard You Like Errors.. • Mean time to detect: 2 mins • Mean time to recover: 15 mins

33. Smashing INT for Fun and Proﬁt • Mean time to detect: 0 min • Mean time to recover: 4 hrs 52 mins

34. Apache Amnesia • Mean time to detect: 2 hours • Mean time to recover: 5 mins

35. Continuously Upgrading Databases • Mean time to detect: 2 mins • Mean time to recover: 1 hour (but, not really..)

36. Q & A Avleen Vig Staﬀ Operations Engineer Etsy, Inc @avleen

37.

38. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/resiliency -etsy

Building Resilience: How Outages Shaped Etsy's Systems

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (8)

More from C4Media

More from C4Media (20)

Recently uploaded

Recently uploaded (20)

Building Resilience: How Outages Shaped Etsy's Systems