SlideShare a Scribd company logo
1 of 65
Download to read offline
Continuously Deploying
         Culture
 Scaling Culture at
    Michael Rembetsy, Director of Operations Engineering (@mrembetsy)
            Patrick McDonnell, Senior Operations Engineer (@mcdonnps)
                                                        October 4, 2012
                                                        Velocity Europe
About Us
• Operations team at Etsy totals twelve
• Engineering totals 125
• Company totals about 350
• 15 million members in over 150 countries
• Over 800,000 sellers
• 1.4 billion page views monthly
Disclaimer


A + B != Culture
The Beginning
Silos and Barriers
• Etsy employs 30 to 35
 • About half are engineers
• Siloed culture, barriers to collaboration in engineering
• “Sprouter”
 • Designed to prevent engineers from directly touching databases
Management Shake-up
• Maria Thomas promoted to CEO
 • Understands that community is of utmost importance
 • Begins to prioritize culture that supports community
• Chad Dickerson brought on as CTO
 • Starts to bring focus to engineering team
Bring the Pain
Opening Communications
• fix.etsy.com
 • First time exposing engineering to community
  • Scheduled maintenance posted ahead of time
  • Outage follow-ups
 • Culturally, technical debt needs to be paid off
Opening Communications
• fix.etsy.com
 • First time exposing engineering to community
  • Scheduled maintenance posted ahead of time
  • Outages followed upon
 • Culturally, technical debt needs to be paid off
2008 Takeaways
• WTF did I get myself into?
 • Low or no impact projects
 • Downtime is accepted / needed
 • Community is most important
 • Lucky to make it through holiday season
2008 End of Year Snapshot

• Gross Merchandise Sales: $87.3 million
• Visits: 163 million
• Unique Visits:  47.7 million
2008 Action Items
• Gain support from the top and bottom to change culture
• Increase transparency both within the organization and to the public
• Pay back technical debt as soon as possible
Sea Change
• Major shift in culture
 • People leave or are let go
 • Too many remotes
  • Teams centralized in Brooklyn
  • Need “DevOps” culture before remotes are manageable
Building the Foundation
•   DevTools team created

    •   Develops first iteration of Deployinator

        •   Stabilizes the deploy chain through dev, QA, and prod environments

•   Infrastructural overhaul

    •   Move from lighttpd to Apache starts

    •   Added monitoring and graphing

        •   Better metrics exposure to non-technical divisions
Building the Foundation
•   DevTools team created

    •   Develops first iteration of Deployinator

        •   Stabilizes the deploy chain through dev, QA, and prod environments

•   Infrastructural overhaul

    •   Move from lighttpd to Apache starts

    •   Added monitoring and graphing

        •   Better metrics exposure to non-technical divisions
Hiring Push
• Only two operations engineers and too few local developers
 • Concentration on sourcing talent to keep pace with growth
 • Move from Downtown Brooklyn to larger space in DUMBO
  • More accessible to non-Brooklynites
  • More mature tech and art community
Internal Improvement
• Stand-up meetings
 • Time-consuming, but necessary to improve communication
• Inter-team collaboration
 • Ops involved in decision-making for dev projects and vice versa
• Internal stability much stronger, damage control period ends
• Network solidified, infrastructural foundation allows for future growth
Stability Arrives
• Everyone wants to collaborate from top down and bottom up
 • Not just upper management pushing down
 • Everyone shapes culture, suggestions always welcome
• People happy to come to work, contribute beyond their job description
• No more scheduled downtime, site remains up as much as possible
• Master database purchased as a stopgap for holiday capacity
2009 Takeaways
• Foundation solidified
 • Technology
 • Human capital
 • Beginning of “DevOps” culture
  • Berlin Wall falls
2009 End of Year Snapshot

• Gross Merchandise Sales:  $177 million (102.6%)
• Visits: 320 million (96.8%)
• Unique Visits: 92.9 million (94.8%)
• Page Views: 6.45 billion
2009 Action Items
• Stabilize the parts of your organization which create thrashing
• Hire staff that will make a difference
• Think about collaborative processes
• When dealing with staffing constraints, prioritize projects by impact
• Get things done (later “Just Ship”)
Renewed Energy
.etsy.com
40

30

20

10




     Very end of 2009
                        Today
Standardization Effort
• PHP
 • Use it and nothing else
 • Everyone should be able to read and rewrite your code
• MySQL becomes database of choice
 • Backfill of PostgreSQL tables to MySQL shards begins
If it Moves, Graph it
• Graphing tools
 • Ganglia, Graphite, FITB, Weathermap
• Monitoring
 • Nagios, Naglite
• Increased focus on work/life balance
 • We removed alerts! WHAT?
If it Moves, Graph it
• Graphing tools
 • Ganglia, Graphite, FITB, Weathermap
• Monitoring
 • Nagios, Naglite
• Increased focus on work/life balance
 • We removed alerts! WHAT?
Management Ideals
• Accept failures but not lower standards
• Доверяй, но проверяй (trust, but verify)
• Blameless post-mortems
• Welcome one-on-ones (http://bit.ly/cCWMqr)
• Career planning
• Happy company = happy community
Engineering Processes
• Developer on call
• A/B Testing
• Prototypes
• Feature flags and ramp-ups
• Schema Change Thursday
2010 Takeaways
• Paring down the number of technologies used for development
• Focus on technical visibility throughout the organization
• Developers take responsibility for code release
• Freedom to hire as needed
• New focus on work life balance
2010 End of Year Snapshot

• Gross Merchandise Sales: $307 million (73.4%)
• Visits: 534 million (66.7%)
• Unique Visits: 147 million (58.2%)
• Page Views: 9.25 billion (43.5%)
2010 Action Items
• Don’t guess at what’s wrong; graph it, monitor it, and find out
• Empower developers with responsibilities
• Have clear, documented standards and practices
• Keep human management a priority
The Reaping
• Death of non-standard technologies
 • Mongo, Scala, CoffeeScript, etc. removed from production systems
• Sprouter is eradicated
 • No more Python
 • Removal of a major barrier to full developer accountability
 • Check out Ross Snyder’s talk at Surge 2011 (http://bit.ly/po8zIj)
The Reaping
• Death of non-standard technologies
 • Mongo, Scala, CoffeeScript, etc. removed from production systems
• Sprouter is eradicated
 • No more Python
 • Removal of a major barrier to full developer accountability
 • Check out Ross Snyder’s talk at Surge 2011 (http://bit.ly/po8zIj)
Organizational Change
• Management becomes more engineering focused
 • Chad becomes CEO
 • Kellan is promoted to CTO
 • John is promoted to SVP of Technical Operations
Technical Contribution
• Big push to contribute to open source and technical community
 • Deployinator, Statsd, Logster, many more
• Three annual goals for every engineer
 • Speak at a conference
 • Write a blog post
 • Open source code
Still Scaling
Configuration Management
And Still Scaling Culture
•   Engineering culture decides git is better for our working style than svn

•   Product development efforts are refocused on high-impact products such as search
    ads

•   Ops works to improve signal-to-noise ratio of alerts and develops internal tools such
    as Schemanator

•   Focus on security

    •   SCRAM team, Security and Fraud team

    •   Check out Nick Galbreath’s talk at DevOpsDays Austin (http://slidesha.re/IMaavq)
Happy Holidays

• Game days to test failures before they happen
• Capacity planning is easier
 • More dashboards (framework now on GitHub!)
 • Improved weekly financial reporting
Happy Holidays

• Game days to test failures before they happen
• Capacity planning is easier
 • More dashboards (framework now on GitHub!)
 • Improved weekly financial reporting
2011 Takeaways
• Year of the tool
  • Statsd, Deployinator, Supergrep v2, Logster, Schemanator, FITB
  • DevTools adds three engineers
• Automation reaches maturity, few surprises
• Focus on security allows PCI infrastructure to be completed in 6
    weeks

•   Engineering matures, both platform and people
2011 End of Year Snapshot

• Gross Merchandise Sales: $526 million (71.4%)
• Visits: 895 million (67.7%)
• Unique Visits: 237 million (61.1%)
• Page Views: 12.9 billion (39.3%)
2011 Action Items
• Senior management at a tech company should be technology-
  focused

• Implement configuration management and automation as soon as
  possible (saves headaches later)

• As technical staff increases, continue to focus on projects that
  matter

• New technical challenges do not dictate cultural shift
Massive Growth
•   Explosive growth in hiring

•   Major organizational changes around product

    •   Increased focus on community

    •   High-impact products (shipping labels, gift cards)

    •   Ramp-up of internationalization projects

•   Unique engineering challenges force non-standard technology

    •   Redis

    •   Virtualized CI test slaves
Spreading Information
• Invitation to other teams to stop by the office for a chat
• Ops becomes more involved in external informational exchange
 • Code as Craft blog posts
• Rapid scaling of personnel, no one knows everyone
 • Remotes come in to Brooklyn more often
Work in Progress
• Developer boredom curbed by allowing transfers between teams
 • Even between divisions (such as engineering -> product)
• Developers now read data from prod databases for development
 • Removes one of the last anomalies between development and
    production

• Front End Performance team forms to minimize site load times
2012 Takeaways
• Organizational resilience allows for total focus on community
 • Improvement on international product
 • Increased transparency to the public
• Focus on informational exchange, both internally and externally
• Open source all the things
2012 Action Items
• Know when not to try something
• Focus on performance early
• Allow dynamic allocation of resources
• Don’t allow size to dictate culture
Future Predictions
• “Wait till you hit 500 people, that’s when everything falls apart”
 • Etsy will not fall apart, but will change
  • Management supports and responds to cultural shifts
  • Freedom of movement between job functionalities
  • Trust that change is for the better
Future Predictions
• Will create better and innovative ways of communicating as we scale
 • Building tools is the Etsy way
• Continue to influence corporate culture
 • B Corp
 • Employee happiness
Office Hours
    Sponsor Pavilion Table A
   2:00 to Last Call at the Bar

Michael Rembetsy (@mrembetsy)
Patrick McDonnell (@mcdonnps)

   https://github.com/etsy

More Related Content

What's hot

DevOps & Security: Here & Now
DevOps & Security: Here & NowDevOps & Security: Here & Now
DevOps & Security: Here & NowCheckmarx
 
Difference between Github vs Gitlab vs Bitbucket
Difference between Github vs Gitlab vs BitbucketDifference between Github vs Gitlab vs Bitbucket
Difference between Github vs Gitlab vs Bitbucketjeetendra mandal
 
DevOps 101 - an Introduction to DevOps
DevOps 101  - an Introduction to DevOpsDevOps 101  - an Introduction to DevOps
DevOps 101 - an Introduction to DevOpsRed Gate Software
 
Continuous Integration/Deployment with Gitlab CI
Continuous Integration/Deployment with Gitlab CIContinuous Integration/Deployment with Gitlab CI
Continuous Integration/Deployment with Gitlab CIDavid Hahn
 
How to implement DevOps in your Organization
How to implement DevOps in your OrganizationHow to implement DevOps in your Organization
How to implement DevOps in your OrganizationDalibor Blazevic
 
CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton Araf Karsh Hamid
 
DevSecOps Training Bootcamp - A Practical DevSecOps Course
DevSecOps Training Bootcamp - A Practical DevSecOps CourseDevSecOps Training Bootcamp - A Practical DevSecOps Course
DevSecOps Training Bootcamp - A Practical DevSecOps CourseTonex
 
¿Qué es DevOps y por qué es importante en el Ciclo de Software? por michelada.io
¿Qué es DevOps y por qué es importante en el Ciclo de Software? por michelada.io¿Qué es DevOps y por qué es importante en el Ciclo de Software? por michelada.io
¿Qué es DevOps y por qué es importante en el Ciclo de Software? por michelada.ioSoftware Guru
 
CI/CD (DevOps) 101
CI/CD (DevOps) 101CI/CD (DevOps) 101
CI/CD (DevOps) 101Hazzim Anaya
 
Devsecops con azure devops en global azure bootcamp 2019
Devsecops con azure devops en global azure bootcamp 2019Devsecops con azure devops en global azure bootcamp 2019
Devsecops con azure devops en global azure bootcamp 2019Luciano Moreira da Cruz
 
DevOps Powerpoint Presentation Slides
DevOps Powerpoint Presentation SlidesDevOps Powerpoint Presentation Slides
DevOps Powerpoint Presentation SlidesSlideTeam
 
Introduction to DevOps
Introduction to DevOpsIntroduction to DevOps
Introduction to DevOpsYosef Tavin
 
Beyond the spotify model - Team Topologies - Agile Scotland 2019-03-11 - Matt...
Beyond the spotify model - Team Topologies - Agile Scotland 2019-03-11 - Matt...Beyond the spotify model - Team Topologies - Agile Scotland 2019-03-11 - Matt...
Beyond the spotify model - Team Topologies - Agile Scotland 2019-03-11 - Matt...Matthew Skelton
 

What's hot (20)

Devops Mindset Essentials
Devops Mindset EssentialsDevops Mindset Essentials
Devops Mindset Essentials
 
DevOps & Security: Here & Now
DevOps & Security: Here & NowDevOps & Security: Here & Now
DevOps & Security: Here & Now
 
Difference between Github vs Gitlab vs Bitbucket
Difference between Github vs Gitlab vs BitbucketDifference between Github vs Gitlab vs Bitbucket
Difference between Github vs Gitlab vs Bitbucket
 
DevOps 101 - an Introduction to DevOps
DevOps 101  - an Introduction to DevOpsDevOps 101  - an Introduction to DevOps
DevOps 101 - an Introduction to DevOps
 
Continuous Integration/Deployment with Gitlab CI
Continuous Integration/Deployment with Gitlab CIContinuous Integration/Deployment with Gitlab CI
Continuous Integration/Deployment with Gitlab CI
 
DevSecOps
DevSecOpsDevSecOps
DevSecOps
 
How to implement DevOps in your Organization
How to implement DevOps in your OrganizationHow to implement DevOps in your Organization
How to implement DevOps in your Organization
 
DevOps Foundation
DevOps FoundationDevOps Foundation
DevOps Foundation
 
DevOps - A Gentle Introduction
DevOps - A Gentle IntroductionDevOps - A Gentle Introduction
DevOps - A Gentle Introduction
 
CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton
 
DevSecOps Training Bootcamp - A Practical DevSecOps Course
DevSecOps Training Bootcamp - A Practical DevSecOps CourseDevSecOps Training Bootcamp - A Practical DevSecOps Course
DevSecOps Training Bootcamp - A Practical DevSecOps Course
 
¿Qué es DevOps y por qué es importante en el Ciclo de Software? por michelada.io
¿Qué es DevOps y por qué es importante en el Ciclo de Software? por michelada.io¿Qué es DevOps y por qué es importante en el Ciclo de Software? por michelada.io
¿Qué es DevOps y por qué es importante en el Ciclo de Software? por michelada.io
 
The Journey to DevSecOps
The Journey to DevSecOpsThe Journey to DevSecOps
The Journey to DevSecOps
 
CI/CD (DevOps) 101
CI/CD (DevOps) 101CI/CD (DevOps) 101
CI/CD (DevOps) 101
 
CI/CD with GitHub Actions
CI/CD with GitHub ActionsCI/CD with GitHub Actions
CI/CD with GitHub Actions
 
Devsecops con azure devops en global azure bootcamp 2019
Devsecops con azure devops en global azure bootcamp 2019Devsecops con azure devops en global azure bootcamp 2019
Devsecops con azure devops en global azure bootcamp 2019
 
DevOps Powerpoint Presentation Slides
DevOps Powerpoint Presentation SlidesDevOps Powerpoint Presentation Slides
DevOps Powerpoint Presentation Slides
 
Introducing DevOps
Introducing DevOpsIntroducing DevOps
Introducing DevOps
 
Introduction to DevOps
Introduction to DevOpsIntroduction to DevOps
Introduction to DevOps
 
Beyond the spotify model - Team Topologies - Agile Scotland 2019-03-11 - Matt...
Beyond the spotify model - Team Topologies - Agile Scotland 2019-03-11 - Matt...Beyond the spotify model - Team Topologies - Agile Scotland 2019-03-11 - Matt...
Beyond the spotify model - Team Topologies - Agile Scotland 2019-03-11 - Matt...
 

Similar to Continuously Deploying Culture: Scaling Culture at Etsy - Velocity Europe 2012

An evening with... DevOps
An evening with... DevOpsAn evening with... DevOps
An evening with... DevOpsArkhotech
 
Scaling r&d org while maintaining quality
Scaling r&d org while maintaining qualityScaling r&d org while maintaining quality
Scaling r&d org while maintaining qualityAviran Mordo
 
DOES16 London - Jonathan Fletcher - Re-imagining Hiscox IT: A DevOps Story
DOES16 London - Jonathan Fletcher - Re-imagining Hiscox IT: A DevOps StoryDOES16 London - Jonathan Fletcher - Re-imagining Hiscox IT: A DevOps Story
DOES16 London - Jonathan Fletcher - Re-imagining Hiscox IT: A DevOps StoryGene Kim
 
Oscon 2016: open source lessons from the todo group
Oscon 2016: open source lessons from the todo groupOscon 2016: open source lessons from the todo group
Oscon 2016: open source lessons from the todo groupBen VanEvery
 
Agile and Lean Software Development
Agile and Lean Software DevelopmentAgile and Lean Software Development
Agile and Lean Software DevelopmentTathagat Varma
 
Open Source Lessons from the TODO Group
Open Source Lessons from the TODO GroupOpen Source Lessons from the TODO Group
Open Source Lessons from the TODO GroupChris Aniszczyk
 
Technology Planning for River Groups
Technology Planning for River GroupsTechnology Planning for River Groups
Technology Planning for River GroupsSean Larkin
 
Open World Forum - The Agile and Open Source Way
Open World Forum - The Agile and Open Source WayOpen World Forum - The Agile and Open Source Way
Open World Forum - The Agile and Open Source WayAlexis Monville
 
French Scrum User Group @Google - The Agile and Open Source Way
French Scrum User Group @Google - The Agile and Open Source WayFrench Scrum User Group @Google - The Agile and Open Source Way
French Scrum User Group @Google - The Agile and Open Source WayAlexis Monville
 
How to Ease Scaleup Growing Pains - from Startup to Scaleup without the pain
How to Ease Scaleup Growing Pains - from Startup to Scaleup without the painHow to Ease Scaleup Growing Pains - from Startup to Scaleup without the pain
How to Ease Scaleup Growing Pains - from Startup to Scaleup without the painNeerventure.com
 
(SPOT205) 5 Lessons for Managing Massive IT Transformation Projects
(SPOT205) 5 Lessons for Managing Massive IT Transformation Projects(SPOT205) 5 Lessons for Managing Massive IT Transformation Projects
(SPOT205) 5 Lessons for Managing Massive IT Transformation ProjectsAmazon Web Services
 
Webinar: The Acceleration of Everything
Webinar: The Acceleration of EverythingWebinar: The Acceleration of Everything
Webinar: The Acceleration of EverythingPerforce
 
Open Source: How to empower your technical teams in Digital Transformation pr...
Open Source: How to empower your technical teams in Digital Transformation pr...Open Source: How to empower your technical teams in Digital Transformation pr...
Open Source: How to empower your technical teams in Digital Transformation pr...Kangaroot
 
Accelerate using DevOps and CI/CD.
Accelerate using DevOps and CI/CD.Accelerate using DevOps and CI/CD.
Accelerate using DevOps and CI/CD.Rick Hightower
 
Engineering Culture and Infrastructure
Engineering Culture and InfrastructureEngineering Culture and Infrastructure
Engineering Culture and InfrastructureSchubert Zhang
 
Mage Titans USA 2016 - Mathew Beane - Edit Fully Stacked: Less OOPS, More OPS...
Mage Titans USA 2016 - Mathew Beane - Edit Fully Stacked: Less OOPS, More OPS...Mage Titans USA 2016 - Mathew Beane - Edit Fully Stacked: Less OOPS, More OPS...
Mage Titans USA 2016 - Mathew Beane - Edit Fully Stacked: Less OOPS, More OPS...Stacey Whitney
 
Building enterprise platforms - off the beaten path - SharePoint User Group U...
Building enterprise platforms - off the beaten path - SharePoint User Group U...Building enterprise platforms - off the beaten path - SharePoint User Group U...
Building enterprise platforms - off the beaten path - SharePoint User Group U...Andy Talbot
 
Continuous Delivery at Wix, Yaniv Even Haim
Continuous Delivery at Wix, Yaniv Even HaimContinuous Delivery at Wix, Yaniv Even Haim
Continuous Delivery at Wix, Yaniv Even HaimDevOpsDays Tel Aviv
 
Story of Multnomah County: Migrating from Vignette and Building a Drupal Ecos...
Story of Multnomah County: Migrating from Vignette and Building a Drupal Ecos...Story of Multnomah County: Migrating from Vignette and Building a Drupal Ecos...
Story of Multnomah County: Migrating from Vignette and Building a Drupal Ecos...Acquia
 

Similar to Continuously Deploying Culture: Scaling Culture at Etsy - Velocity Europe 2012 (20)

An evening with... DevOps
An evening with... DevOpsAn evening with... DevOps
An evening with... DevOps
 
Scaling r&d org while maintaining quality
Scaling r&d org while maintaining qualityScaling r&d org while maintaining quality
Scaling r&d org while maintaining quality
 
Computing DevOp Summit
Computing DevOp SummitComputing DevOp Summit
Computing DevOp Summit
 
DOES16 London - Jonathan Fletcher - Re-imagining Hiscox IT: A DevOps Story
DOES16 London - Jonathan Fletcher - Re-imagining Hiscox IT: A DevOps StoryDOES16 London - Jonathan Fletcher - Re-imagining Hiscox IT: A DevOps Story
DOES16 London - Jonathan Fletcher - Re-imagining Hiscox IT: A DevOps Story
 
Oscon 2016: open source lessons from the todo group
Oscon 2016: open source lessons from the todo groupOscon 2016: open source lessons from the todo group
Oscon 2016: open source lessons from the todo group
 
Agile and Lean Software Development
Agile and Lean Software DevelopmentAgile and Lean Software Development
Agile and Lean Software Development
 
Open Source Lessons from the TODO Group
Open Source Lessons from the TODO GroupOpen Source Lessons from the TODO Group
Open Source Lessons from the TODO Group
 
Technology Planning for River Groups
Technology Planning for River GroupsTechnology Planning for River Groups
Technology Planning for River Groups
 
Open World Forum - The Agile and Open Source Way
Open World Forum - The Agile and Open Source WayOpen World Forum - The Agile and Open Source Way
Open World Forum - The Agile and Open Source Way
 
French Scrum User Group @Google - The Agile and Open Source Way
French Scrum User Group @Google - The Agile and Open Source WayFrench Scrum User Group @Google - The Agile and Open Source Way
French Scrum User Group @Google - The Agile and Open Source Way
 
How to Ease Scaleup Growing Pains - from Startup to Scaleup without the pain
How to Ease Scaleup Growing Pains - from Startup to Scaleup without the painHow to Ease Scaleup Growing Pains - from Startup to Scaleup without the pain
How to Ease Scaleup Growing Pains - from Startup to Scaleup without the pain
 
(SPOT205) 5 Lessons for Managing Massive IT Transformation Projects
(SPOT205) 5 Lessons for Managing Massive IT Transformation Projects(SPOT205) 5 Lessons for Managing Massive IT Transformation Projects
(SPOT205) 5 Lessons for Managing Massive IT Transformation Projects
 
Webinar: The Acceleration of Everything
Webinar: The Acceleration of EverythingWebinar: The Acceleration of Everything
Webinar: The Acceleration of Everything
 
Open Source: How to empower your technical teams in Digital Transformation pr...
Open Source: How to empower your technical teams in Digital Transformation pr...Open Source: How to empower your technical teams in Digital Transformation pr...
Open Source: How to empower your technical teams in Digital Transformation pr...
 
Accelerate using DevOps and CI/CD.
Accelerate using DevOps and CI/CD.Accelerate using DevOps and CI/CD.
Accelerate using DevOps and CI/CD.
 
Engineering Culture and Infrastructure
Engineering Culture and InfrastructureEngineering Culture and Infrastructure
Engineering Culture and Infrastructure
 
Mage Titans USA 2016 - Mathew Beane - Edit Fully Stacked: Less OOPS, More OPS...
Mage Titans USA 2016 - Mathew Beane - Edit Fully Stacked: Less OOPS, More OPS...Mage Titans USA 2016 - Mathew Beane - Edit Fully Stacked: Less OOPS, More OPS...
Mage Titans USA 2016 - Mathew Beane - Edit Fully Stacked: Less OOPS, More OPS...
 
Building enterprise platforms - off the beaten path - SharePoint User Group U...
Building enterprise platforms - off the beaten path - SharePoint User Group U...Building enterprise platforms - off the beaten path - SharePoint User Group U...
Building enterprise platforms - off the beaten path - SharePoint User Group U...
 
Continuous Delivery at Wix, Yaniv Even Haim
Continuous Delivery at Wix, Yaniv Even HaimContinuous Delivery at Wix, Yaniv Even Haim
Continuous Delivery at Wix, Yaniv Even Haim
 
Story of Multnomah County: Migrating from Vignette and Building a Drupal Ecos...
Story of Multnomah County: Migrating from Vignette and Building a Drupal Ecos...Story of Multnomah County: Migrating from Vignette and Building a Drupal Ecos...
Story of Multnomah County: Migrating from Vignette and Building a Drupal Ecos...
 

Continuously Deploying Culture: Scaling Culture at Etsy - Velocity Europe 2012

  • 1. Continuously Deploying Culture Scaling Culture at Michael Rembetsy, Director of Operations Engineering (@mrembetsy) Patrick McDonnell, Senior Operations Engineer (@mcdonnps) October 4, 2012 Velocity Europe
  • 2. About Us • Operations team at Etsy totals twelve • Engineering totals 125 • Company totals about 350 • 15 million members in over 150 countries • Over 800,000 sellers • 1.4 billion page views monthly
  • 3. Disclaimer A + B != Culture
  • 5. Silos and Barriers • Etsy employs 30 to 35 • About half are engineers • Siloed culture, barriers to collaboration in engineering • “Sprouter” • Designed to prevent engineers from directly touching databases
  • 6. Management Shake-up • Maria Thomas promoted to CEO • Understands that community is of utmost importance • Begins to prioritize culture that supports community • Chad Dickerson brought on as CTO • Starts to bring focus to engineering team
  • 8.
  • 9. Opening Communications • fix.etsy.com • First time exposing engineering to community • Scheduled maintenance posted ahead of time • Outage follow-ups • Culturally, technical debt needs to be paid off
  • 10.
  • 11. Opening Communications • fix.etsy.com • First time exposing engineering to community • Scheduled maintenance posted ahead of time • Outages followed upon • Culturally, technical debt needs to be paid off
  • 12. 2008 Takeaways • WTF did I get myself into? • Low or no impact projects • Downtime is accepted / needed • Community is most important • Lucky to make it through holiday season
  • 13. 2008 End of Year Snapshot • Gross Merchandise Sales: $87.3 million • Visits: 163 million • Unique Visits:  47.7 million
  • 14. 2008 Action Items • Gain support from the top and bottom to change culture • Increase transparency both within the organization and to the public • Pay back technical debt as soon as possible
  • 15. Sea Change • Major shift in culture • People leave or are let go • Too many remotes • Teams centralized in Brooklyn • Need “DevOps” culture before remotes are manageable
  • 16. Building the Foundation • DevTools team created • Develops first iteration of Deployinator • Stabilizes the deploy chain through dev, QA, and prod environments • Infrastructural overhaul • Move from lighttpd to Apache starts • Added monitoring and graphing • Better metrics exposure to non-technical divisions
  • 17.
  • 18.
  • 19. Building the Foundation • DevTools team created • Develops first iteration of Deployinator • Stabilizes the deploy chain through dev, QA, and prod environments • Infrastructural overhaul • Move from lighttpd to Apache starts • Added monitoring and graphing • Better metrics exposure to non-technical divisions
  • 20. Hiring Push • Only two operations engineers and too few local developers • Concentration on sourcing talent to keep pace with growth • Move from Downtown Brooklyn to larger space in DUMBO • More accessible to non-Brooklynites • More mature tech and art community
  • 21. Internal Improvement • Stand-up meetings • Time-consuming, but necessary to improve communication • Inter-team collaboration • Ops involved in decision-making for dev projects and vice versa • Internal stability much stronger, damage control period ends • Network solidified, infrastructural foundation allows for future growth
  • 22. Stability Arrives • Everyone wants to collaborate from top down and bottom up • Not just upper management pushing down • Everyone shapes culture, suggestions always welcome • People happy to come to work, contribute beyond their job description • No more scheduled downtime, site remains up as much as possible • Master database purchased as a stopgap for holiday capacity
  • 23. 2009 Takeaways • Foundation solidified • Technology • Human capital • Beginning of “DevOps” culture • Berlin Wall falls
  • 24. 2009 End of Year Snapshot • Gross Merchandise Sales:  $177 million (102.6%) • Visits: 320 million (96.8%) • Unique Visits: 92.9 million (94.8%) • Page Views: 6.45 billion
  • 25. 2009 Action Items • Stabilize the parts of your organization which create thrashing • Hire staff that will make a difference • Think about collaborative processes • When dealing with staffing constraints, prioritize projects by impact • Get things done (later “Just Ship”)
  • 27.
  • 29. 40 30 20 10 Very end of 2009 Today
  • 30. Standardization Effort • PHP • Use it and nothing else • Everyone should be able to read and rewrite your code • MySQL becomes database of choice • Backfill of PostgreSQL tables to MySQL shards begins
  • 31. If it Moves, Graph it • Graphing tools • Ganglia, Graphite, FITB, Weathermap • Monitoring • Nagios, Naglite • Increased focus on work/life balance • We removed alerts! WHAT?
  • 32.
  • 33.
  • 34. If it Moves, Graph it • Graphing tools • Ganglia, Graphite, FITB, Weathermap • Monitoring • Nagios, Naglite • Increased focus on work/life balance • We removed alerts! WHAT?
  • 35.
  • 36. Management Ideals • Accept failures but not lower standards • Доверяй, но проверяй (trust, but verify) • Blameless post-mortems • Welcome one-on-ones (http://bit.ly/cCWMqr) • Career planning • Happy company = happy community
  • 37. Engineering Processes • Developer on call • A/B Testing • Prototypes • Feature flags and ramp-ups • Schema Change Thursday
  • 38. 2010 Takeaways • Paring down the number of technologies used for development • Focus on technical visibility throughout the organization • Developers take responsibility for code release • Freedom to hire as needed • New focus on work life balance
  • 39. 2010 End of Year Snapshot • Gross Merchandise Sales: $307 million (73.4%) • Visits: 534 million (66.7%) • Unique Visits: 147 million (58.2%) • Page Views: 9.25 billion (43.5%)
  • 40. 2010 Action Items • Don’t guess at what’s wrong; graph it, monitor it, and find out • Empower developers with responsibilities • Have clear, documented standards and practices • Keep human management a priority
  • 41. The Reaping • Death of non-standard technologies • Mongo, Scala, CoffeeScript, etc. removed from production systems • Sprouter is eradicated • No more Python • Removal of a major barrier to full developer accountability • Check out Ross Snyder’s talk at Surge 2011 (http://bit.ly/po8zIj)
  • 42.
  • 43. The Reaping • Death of non-standard technologies • Mongo, Scala, CoffeeScript, etc. removed from production systems • Sprouter is eradicated • No more Python • Removal of a major barrier to full developer accountability • Check out Ross Snyder’s talk at Surge 2011 (http://bit.ly/po8zIj)
  • 44. Organizational Change • Management becomes more engineering focused • Chad becomes CEO • Kellan is promoted to CTO • John is promoted to SVP of Technical Operations
  • 45. Technical Contribution • Big push to contribute to open source and technical community • Deployinator, Statsd, Logster, many more • Three annual goals for every engineer • Speak at a conference • Write a blog post • Open source code
  • 48.
  • 49. And Still Scaling Culture • Engineering culture decides git is better for our working style than svn • Product development efforts are refocused on high-impact products such as search ads • Ops works to improve signal-to-noise ratio of alerts and develops internal tools such as Schemanator • Focus on security • SCRAM team, Security and Fraud team • Check out Nick Galbreath’s talk at DevOpsDays Austin (http://slidesha.re/IMaavq)
  • 50. Happy Holidays • Game days to test failures before they happen • Capacity planning is easier • More dashboards (framework now on GitHub!) • Improved weekly financial reporting
  • 51.
  • 52. Happy Holidays • Game days to test failures before they happen • Capacity planning is easier • More dashboards (framework now on GitHub!) • Improved weekly financial reporting
  • 53. 2011 Takeaways • Year of the tool • Statsd, Deployinator, Supergrep v2, Logster, Schemanator, FITB • DevTools adds three engineers • Automation reaches maturity, few surprises • Focus on security allows PCI infrastructure to be completed in 6 weeks • Engineering matures, both platform and people
  • 54. 2011 End of Year Snapshot • Gross Merchandise Sales: $526 million (71.4%) • Visits: 895 million (67.7%) • Unique Visits: 237 million (61.1%) • Page Views: 12.9 billion (39.3%)
  • 55. 2011 Action Items • Senior management at a tech company should be technology- focused • Implement configuration management and automation as soon as possible (saves headaches later) • As technical staff increases, continue to focus on projects that matter • New technical challenges do not dictate cultural shift
  • 56. Massive Growth • Explosive growth in hiring • Major organizational changes around product • Increased focus on community • High-impact products (shipping labels, gift cards) • Ramp-up of internationalization projects • Unique engineering challenges force non-standard technology • Redis • Virtualized CI test slaves
  • 57. Spreading Information • Invitation to other teams to stop by the office for a chat • Ops becomes more involved in external informational exchange • Code as Craft blog posts • Rapid scaling of personnel, no one knows everyone • Remotes come in to Brooklyn more often
  • 58.
  • 59.
  • 60. Work in Progress • Developer boredom curbed by allowing transfers between teams • Even between divisions (such as engineering -> product) • Developers now read data from prod databases for development • Removes one of the last anomalies between development and production • Front End Performance team forms to minimize site load times
  • 61. 2012 Takeaways • Organizational resilience allows for total focus on community • Improvement on international product • Increased transparency to the public • Focus on informational exchange, both internally and externally • Open source all the things
  • 62. 2012 Action Items • Know when not to try something • Focus on performance early • Allow dynamic allocation of resources • Don’t allow size to dictate culture
  • 63. Future Predictions • “Wait till you hit 500 people, that’s when everything falls apart” • Etsy will not fall apart, but will change • Management supports and responds to cultural shifts • Freedom of movement between job functionalities • Trust that change is for the better
  • 64. Future Predictions • Will create better and innovative ways of communicating as we scale • Building tools is the Etsy way • Continue to influence corporate culture • B Corp • Employee happiness
  • 65. Office Hours Sponsor Pavilion Table A 2:00 to Last Call at the Bar Michael Rembetsy (@mrembetsy) Patrick McDonnell (@mcdonnps) https://github.com/etsy