Scaling Deployment at Etsy

•

12 likes•8,656 views

Daniel Schauenberg

How we keep deployments fast at Etsy presented at Scaleconf 2013

Technology

Scaling Deployment at Etsy
Daniel Schauenberg
dschauenberg@etsy.com
@mrtazz
Thursday, April 18, 13

the marketplace we make
together
Thursday, April 18, 13

February 2013
•1.49 billion page views
•4,215,169 items sold
•$94.7 million of goods sold
•22+ million members
•800,000+ active shops
http://www.etsy.com/blog/news/2013/etsy-statistics-february-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things
Thursday, April 18, 13

LAMMP
Item by TheBackPackShoppe
Thursday, April 18, 13

Item by FrankelPhotos
Monolithic App
Thursday, April 18, 13

No Branching
Item by NurseryWallArt
Thursday, April 18, 13

First Day
Item by ﬂowersandﬂeurons
Thursday, April 18, 13

DeveloperVMs
•KVM
•Dev version of full Etsy stack
•Chef
•DevTools
Thursday, April 18, 13

Item by codecards
Thursday, April 18, 13

% review -r dschauenberg
Thursday, April 18, 13

automatically assignedautomatically assigned
Thursday, April 18, 13

Try
Item by CSSDesign
Thursday, April 18, 13

CI - The Old Way
•~10 buildtest servers, single SSD
•running multiple Jenkins executors
•tests pegging IO
•Jenkins is much better at running single executors
Thursday, April 18, 13

The Bobs
Item by Signz
Thursday, April 18, 13

The Bobs
•LXC containers on buildtests
•Multiple SSDs
•Labels for heavy/any execution
•One heavy executor per disk
Thursday, April 18, 13

CI/Try
•~200 Bobs
•Mostly for try
•Constant monitoring for slow tests
Thursday, April 18, 13

push train
Item by decomodwalls
Thursday, April 18, 13

#push
•IRC channel to organize push trains
•Join a train if you want to deploy changes
•Schedule is planned via the channel topic
•First in the train is the driver (controls the deploy)
•Opening hours: 7am - 10pm NYC time
Thursday, April 18, 13

#push
<prod> kseever* + jameslee | jpaul | DanielConvissor (c)
Thursday, April 18, 13

#push
bateman* + krunal* + enorris* | tristan (c) + jameslee (c)
+ jlaster (c) | dawa + corey + sandosh + jklein + magera
+ seth_home + mpascual + nathan | bateman | russp (c)
Thursday, April 18, 13

pushbot
•.join
•.in
•.good
•.done
Thursday, April 18, 13

https://github.com/etsy/PushBot
Thursday, April 18, 13

Item by EsalonPhotography
Thursday, April 18, 13

version checks
buttons disabledbuttons disabled
Thursday, April 18, 13

lock down deploys
Thursday, April 18, 13

https://github.com/etsy/deployinator
Thursday, April 18, 13

Dashboards
deploy linesdeploy lines
Thursday, April 18, 13

https://github.com/etsy/dashboard
Thursday, April 18, 13

https://github.com/etsy/supergrep
Thursday, April 18, 13

Summary
•Current setup has scaled to ~150 people
•Constantly trying to improve the speed of deployment
•Find weak parts in the process and make them more
robust/faster
•Bring Dev closer to Prod
•Not being able to deploy has the same status as the site
being down
Thursday, April 18, 13

codeascraft.etsy.com
www.etsy.com/codeascraft/talks
etsy.github.com
www.etsy.com/careers
Thursday, April 18, 13

More from Daniel Schauenberg

Our daily work takes place in a myriad of systems. They are comprised of software, hardware and humans. And everybody who has worked with complex systems at any scale knows: Failure is not an option, it's inevitable. At Etsy we are embracing the fact that failures happen and that the only way to understand how the accident happened is to investigate it without blaming the humans involved. This is why we have a blameless postmortem for every outage that occurs. It is an open meeting and everybody is invited to join and find out what happened and how we can make the system safer. This talk will explain how postmortems at Etsy are conducted and how we maintain and scale the process as the team grows and new people start. It will go over the tools we built and utilize to make postmortems efficient and also share the learnings from each one with all the people in the company.

Human Factors and PostMortems

Daniel Schauenberg

At Etsy we are continuously deploying software with about 50 deploys per day on average. We have about 150 engineers that work together to add features, improve the site, solve problems, and figure out outages. We are constantly working on improving ways to collaborate and have the time to invest into research and new projects instead of fire fighting. However this hasn't been like this from the beginning. In the Dark Ages of Etsy we had a vastly different software architecture, a myriad of silos with a less than ideal amount of communication and a not so stable site. So what happened? How did we fix this? We will briefly go over how we changed the architecture and culture to make site operations and stability better. After that we will go into detail about what it meant for us to maintain that culture of collaboration and trust and things we learned from that. And finally take a look at the state of things in the present and give a comprehensive picture of how we arrived here and where to go from there.

Deploy, Collaborate and Listen

Daniel Schauenberg

At Etsy we are big fans of graphing and monitoring all the things. We deploy our main site several times a day and our monitoring provides us with the tight feedback loop we need to make this possible. The same goes for changes in the infrastructure which are deployed in a similar fashion of small and frequent changes. For this to work we have build up monitoring that tracks changes and possible problems in every nook and cranny of the Etsy stack, be it a network change, systems or application level performance or how bad the last week of on-call rotation was. The flipside of monitoring all the things however is that we have a myriad of graphs and alerts that can potentially be important and page the on-call engineer at any given time. The risk of running into alert fatigue and a normalization of deviance through not properly scoped checks is rising with this ever increasing size of the monitoring system. This is why we also continuously monitor our monitoring system and ask questions about whether we have all the information at hand when we get paged, if certain alerts actually need to wake someone up or if they are needed at all. I will give a quick overview of how our monitoring stack is built and then give insights into how we gather data about it and use it to make things better. For site operations and more importantly for the human getting paged when something does go wrong.

Data Driven Monitoring

Daniel Schauenberg

Development, Deployment & Collaboration at Etsy

Daniel Schauenberg

Mobile CI at Etsy

Daniel Schauenberg

It's no secret that at Etsy we are big fans of small, incremental and frequent changes and tight feedback loops. This is how we make it possible to deploy changes to our main codebase more than 50 times a day and also safely apply changes to our infrastructure in a continuous fashion. It enables us to rapidly fix bugs and roll out features in our application stack and infrastructure. This however would not be possible without a tight feedback loop and a myriad of monitoring tools that keep us informed about changes and possible problems in every nook and cranny of the Etsy stack, no matter if it's a network change event, systems or application level performance or how bad the last week of on-call rotation was.

A Whirlwind Tour of Etsy's Monitoring Stack

Daniel Schauenberg

At Etsy about 150 engineers deploy a single monolithic application more than 60 times a day. This process of deploying small changesets continuously enables us to build up and release robust features and detect and fix bugs extremely fast. All while serving over a billion page views per month. Developing and deploying at such a high velocity however only works because product developers and designers, infrastructure and operations engineers and the security team work closely together. We have an extremely open culture of sharing (inside and outside the company) and make sure we run into as few surprises as possible by bringing everybody on the same page about changes.

Development, Deployment and Collaboration at Etsy

Daniel Schauenberg

Scaling Deployment at Etsy

Daniel Schauenberg

At Etsy we run our production stack exclusively on physical hardware and have (including developer and CI VMs) about 1000 nodes managed with a single Chef server running the same cookbooks across development and production nodes. We have built the knife-spork plugin to support our workflow and have 30 engineers making changes about 20 times a day. While this setup keeps configuration changes and package installations consistent across our network, it makes it harder to gradually roll out and test changes on a per node or role basis. In the talk our general workflow will be introduced and explained and how we enable engineers to frequently roll out small changes. Following the general introduction, our workflow for testing changes will be discussed and the library we wrote to enable config flags and gradual roll outs for our infrastructure within our chef recipes.

Feature Flagging your Infrastructure for Fun and Profit

Daniel Schauenberg

StatsD Workshop Monitorama 2013

Daniel Schauenberg

Etsy chef-workflow

Daniel Schauenberg

More from Daniel Schauenberg (11)

Human Factors and PostMortems

Deploy, Collaborate and Listen

Data Driven Monitoring

Development, Deployment & Collaboration at Etsy

Mobile CI at Etsy

A Whirlwind Tour of Etsy's Monitoring Stack

Development, Deployment and Collaboration at Etsy

Scaling Deployment at Etsy

Feature Flagging your Infrastructure for Fun and Profit

StatsD Workshop Monitorama 2013

Etsy chef-workflow

Recently uploaded

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Manulife - Insurer Innovation Award 2024

The Digital Insurer

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Scaling API-first – The story of a global engineering organization

Radu Cotescu

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

Recently uploaded (20)

Axa Assurance Maroc - Insurer Innovation Award 2024

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

AWS Community Day CPH - Three problems of Terraform

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Data Cloud, More than a CDP by Matt Robison

Apidays New York 2024 - The value of a flexible API Management solution for O...

Manulife - Insurer Innovation Award 2024

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

A Domino Admins Adventures (Engage 2024)

Strategies for Landing an Oracle DBA Job as a Fresher

Boost Fertility New Invention Ups Success Rates.pdf

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Automating Google Workspace (GWS) & more with Apps Script

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Artificial Intelligence Chap.5 : Uncertainty

Scaling API-first – The story of a global engineering organization

Boost PC performance: How more available memory can improve productivity

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Exploring the Future Potential of AI-Enabled Smartphone Processors

Scaling Deployment at Etsy

1. Scaling Deployment at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz Thursday, April 18, 13

2. Thursday, April 18, 13

3. the marketplace we make together Thursday, April 18, 13

4. February 2013 •1.49 billion page views •4,215,169 items sold •$94.7 million of goods sold •22+ million members •800,000+ active shops http://www.etsy.com/blog/news/2013/etsy-statistics-february-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things Thursday, April 18, 13

5. LAMMP Item by TheBackPackShoppe Thursday, April 18, 13

6. Item by FrankelPhotos Monolithic App Thursday, April 18, 13

7. No Branching Item by NurseryWallArt Thursday, April 18, 13

8. Deploy Frequency Thursday, April 18, 13

9. First Day Item by ﬂowersandﬂeurons Thursday, April 18, 13

10. Thursday, April 18, 13

11. IRC Thursday, April 18, 13

12. Thursday, April 18, 13

13. DeveloperVMs •KVM •Dev version of full Etsy stack •Chef •DevTools Thursday, April 18, 13

14. Thursday, April 18, 13

15. Item by codecards Thursday, April 18, 13

16. % review -r dschauenberg Thursday, April 18, 13

17. automatically assignedautomatically assigned Thursday, April 18, 13

18. Try Item by CSSDesign Thursday, April 18, 13

19. Actually ... Thursday, April 18, 13

20. CI - The Old Way •~10 buildtest servers, single SSD •running multiple Jenkins executors •tests pegging IO •Jenkins is much better at running single executors Thursday, April 18, 13

21. The Bobs Item by Signz Thursday, April 18, 13

22. The Bobs •LXC containers on buildtests •Multiple SSDs •Labels for heavy/any execution •One heavy executor per disk Thursday, April 18, 13

23. CI/Try •~200 Bobs •Mostly for try •Constant monitoring for slow tests Thursday, April 18, 13

24. push train Item by decomodwalls Thursday, April 18, 13

25. #push •IRC channel to organize push trains •Join a train if you want to deploy changes •Schedule is planned via the channel topic •First in the train is the driver (controls the deploy) •Opening hours: 7am - 10pm NYC time Thursday, April 18, 13

26. #push <prod> kseever* + jameslee | jpaul | DanielConvissor (c) Thursday, April 18, 13

27. #push bateman* + krunal* + enorris* | tristan (c) + jameslee (c) + jlaster (c) | dawa + corey + sandosh + jklein + magera + seth_home + mpascual + nathan | bateman | russp (c) Thursday, April 18, 13

28. pushbot •.join •.in •.good •.done Thursday, April 18, 13

29. pushbot Thursday, April 18, 13

30. https://github.com/etsy/PushBot Thursday, April 18, 13

31. Item by EsalonPhotography Thursday, April 18, 13

32. Deployinator Thursday, April 18, 13

33. Thursday, April 18, 13

34. Thursday, April 18, 13

35. stale commits Thursday, April 18, 13

36. version checks buttons disabledbuttons disabled Thursday, April 18, 13

37. version checks Thursday, April 18, 13

38. lock down deploys Thursday, April 18, 13

39. https://github.com/etsy/deployinator Thursday, April 18, 13

40. Dashboards deploy linesdeploy lines Thursday, April 18, 13

41. https://github.com/etsy/dashboard Thursday, April 18, 13

42. Supergrep Thursday, April 18, 13

43. https://github.com/etsy/supergrep Thursday, April 18, 13

44. Thursday, April 18, 13

45. Summary •Current setup has scaled to ~150 people •Constantly trying to improve the speed of deployment •Find weak parts in the process and make them more robust/faster •Bring Dev closer to Prod •Not being able to deploy has the same status as the site being down Thursday, April 18, 13

46. codeascraft.etsy.com www.etsy.com/codeascraft/talks etsy.github.com www.etsy.com/careers Thursday, April 18, 13

47. Questions? Thursday, April 18, 13

48. Scaling Deployment at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz Thursday, April 18, 13

Scaling Deployment at Etsy

Recommended

Recommended

More Related Content

More from Daniel Schauenberg

More from Daniel Schauenberg (11)

Recently uploaded

Recently uploaded (20)

Scaling Deployment at Etsy