1.
Continuously Deploying Culture Scaling Culture at Michael Rembetsy, Director of Operations Engineering (@mrembetsy) Patrick McDonnell, Senior Operations Engineer (@mcdonnps) October 4, 2012 Velocity Europe

2.
About Us• Operations team at Etsy totals twelve• Engineering totals 125• Company totals about 350• 15 million members in over 150 countries• Over 800,000 sellers• 1.4 billion page views monthly

3.
DisclaimerA + B != Culture

4.
The Beginning

5.
Silos and Barriers• Etsy employs 30 to 35 • About half are engineers• Siloed culture, barriers to collaboration in engineering• “Sprouter” • Designed to prevent engineers from directly touching databases

6.
Management Shake-up• Maria Thomas promoted to CEO • Understands that community is of utmost importance • Begins to prioritize culture that supports community• Chad Dickerson brought on as CTO • Starts to bring focus to engineering team

7.
Bring the Pain

8.
Opening Communications• fix.etsy.com • First time exposing engineering to community • Scheduled maintenance posted ahead of time • Outage follow-ups • Culturally, technical debt needs to be paid off

9.
Opening Communications• fix.etsy.com • First time exposing engineering to community • Scheduled maintenance posted ahead of time • Outages followed upon • Culturally, technical debt needs to be paid off

10.
2008 Takeaways• WTF did I get myself into? • Low or no impact projects • Downtime is accepted / needed • Community is most important • Lucky to make it through holiday season

11.
2008 End of Year Snapshot• Gross Merchandise Sales: $87.3 million• Visits: 163 million• Unique Visits: 47.7 million

12.
2008 Action Items• Gain support from the top and bottom to change culture• Increase transparency both within the organization and to the public• Pay back technical debt as soon as possible

13.
Sea Change• Major shift in culture • People leave or are let go • Too many remotes • Teams centralized in Brooklyn • Need “DevOps” culture before remotes are manageable

14.
Building the Foundation• DevTools team created • Develops first iteration of Deployinator • Stabilizes the deploy chain through dev, QA, and prod environments• Infrastructural overhaul • Move from lighttpd to Apache starts • Added monitoring and graphing • Better metrics exposure to non-technical divisions

15.
Building the Foundation• DevTools team created • Develops first iteration of Deployinator • Stabilizes the deploy chain through dev, QA, and prod environments• Infrastructural overhaul • Move from lighttpd to Apache starts • Added monitoring and graphing • Better metrics exposure to non-technical divisions

16.
Hiring Push• Only two operations engineers and too few local developers • Concentration on sourcing talent to keep pace with growth • Move from Downtown Brooklyn to larger space in DUMBO • More accessible to non-Brooklynites • More mature tech and art community

17.
Internal Improvement• Stand-up meetings • Time-consuming, but necessary to improve communication• Inter-team collaboration • Ops involved in decision-making for dev projects and vice versa• Internal stability much stronger, damage control period ends• Network solidified, infrastructural foundation allows for future growth

18.
Stability Arrives• Everyone wants to collaborate from top down and bottom up • Not just upper management pushing down • Everyone shapes culture, suggestions always welcome• People happy to come to work, contribute beyond their job description• No more scheduled downtime, site remains up as much as possible• Master database purchased as a stopgap for holiday capacity

19.
2009 Takeaways• Foundation solidified • Technology • Human capital • Beginning of “DevOps” culture • Berlin Wall falls

20.
2009 End of Year Snapshot• Gross Merchandise Sales: $177 million (102.6%)• Visits: 320 million (96.8%)• Unique Visits: 92.9 million (94.8%)• Page Views: 6.45 billion

21.
2009 Action Items• Stabilize the parts of your organization which create thrashing• Hire staff that will make a difference• Think about collaborative processes• When dealing with staffing constraints, prioritize projects by impact• Get things done (later “Just Ship”)

22.
Renewed Energy

23.
.etsy.com

24.
40302010 Very end of 2009 Today

25.
Standardization Effort• PHP • Use it and nothing else • Everyone should be able to read and rewrite your code• MySQL becomes database of choice • Backfill of PostgreSQL tables to MySQL shards begins

26.
If it Moves, Graph it• Graphing tools • Ganglia, Graphite, FITB, Weathermap• Monitoring • Nagios, Naglite• Increased focus on work/life balance • We removed alerts! WHAT?

27.
If it Moves, Graph it• Graphing tools • Ganglia, Graphite, FITB, Weathermap• Monitoring • Nagios, Naglite• Increased focus on work/life balance • We removed alerts! WHAT?

28.
Management Ideals• Accept failures but not lower standards• Доверяй, но проверяй (trust, but verify)• Blameless post-mortems• Welcome one-on-ones (http://bit.ly/cCWMqr)• Career planning• Happy company = happy community

29.
Engineering Processes• Developer on call• A/B Testing• Prototypes• Feature flags and ramp-ups• Schema Change Thursday

30.
2010 Takeaways• Paring down the number of technologies used for development• Focus on technical visibility throughout the organization• Developers take responsibility for code release• Freedom to hire as needed• New focus on work life balance

31.
2010 End of Year Snapshot• Gross Merchandise Sales: $307 million (73.4%)• Visits: 534 million (66.7%)• Unique Visits: 147 million (58.2%)• Page Views: 9.25 billion (43.5%)

32.
2010 Action Items• Don’t guess at what’s wrong; graph it, monitor it, and find out• Empower developers with responsibilities• Have clear, documented standards and practices• Keep human management a priority

33.
The Reaping• Death of non-standard technologies • Mongo, Scala, CoffeeScript, etc. removed from production systems• Sprouter is eradicated • No more Python • Removal of a major barrier to full developer accountability • Check out Ross Snyder’s talk at Surge 2011 (http://bit.ly/po8zIj)

34.
The Reaping• Death of non-standard technologies • Mongo, Scala, CoffeeScript, etc. removed from production systems• Sprouter is eradicated • No more Python • Removal of a major barrier to full developer accountability • Check out Ross Snyder’s talk at Surge 2011 (http://bit.ly/po8zIj)

35.
Organizational Change• Management becomes more engineering focused • Chad becomes CEO • Kellan is promoted to CTO • John is promoted to SVP of Technical Operations

36.
Technical Contribution• Big push to contribute to open source and technical community • Deployinator, Statsd, Logster, many more• Three annual goals for every engineer • Speak at a conference • Write a blog post • Open source code

37.
Still Scaling

38.
Configuration Management

39.
And Still Scaling Culture• Engineering culture decides git is better for our working style than svn• Product development efforts are refocused on high-impact products such as search ads• Ops works to improve signal-to-noise ratio of alerts and develops internal tools such as Schemanator• Focus on security • SCRAM team, Security and Fraud team • Check out Nick Galbreath’s talk at DevOpsDays Austin (http://slidesha.re/IMaavq)

40.
Happy Holidays• Game days to test failures before they happen• Capacity planning is easier • More dashboards (framework now on GitHub!) • Improved weekly financial reporting

41.
Happy Holidays• Game days to test failures before they happen• Capacity planning is easier • More dashboards (framework now on GitHub!) • Improved weekly financial reporting

42.
2011 Takeaways• Year of the tool • Statsd, Deployinator, Supergrep v2, Logster, Schemanator, FITB • DevTools adds three engineers• Automation reaches maturity, few surprises• Focus on security allows PCI infrastructure to be completed in 6 weeks• Engineering matures, both platform and people

43.
2011 End of Year Snapshot• Gross Merchandise Sales: $526 million (71.4%)• Visits: 895 million (67.7%)• Unique Visits: 237 million (61.1%)• Page Views: 12.9 billion (39.3%)

44.
2011 Action Items• Senior management at a tech company should be technology- focused• Implement configuration management and automation as soon as possible (saves headaches later)• As technical staff increases, continue to focus on projects that matter• New technical challenges do not dictate cultural shift

45.
Massive Growth• Explosive growth in hiring• Major organizational changes around product • Increased focus on community • High-impact products (shipping labels, gift cards) • Ramp-up of internationalization projects• Unique engineering challenges force non-standard technology • Redis • Virtualized CI test slaves

46.
Spreading Information• Invitation to other teams to stop by the office for a chat• Ops becomes more involved in external informational exchange • Code as Craft blog posts• Rapid scaling of personnel, no one knows everyone • Remotes come in to Brooklyn more often

47.
Work in Progress• Developer boredom curbed by allowing transfers between teams • Even between divisions (such as engineering -> product)• Developers now read data from prod databases for development • Removes one of the last anomalies between development and production• Front End Performance team forms to minimize site load times

48.
2012 Takeaways• Organizational resilience allows for total focus on community • Improvement on international product • Increased transparency to the public• Focus on informational exchange, both internally and externally• Open source all the things

49.
2012 Action Items• Know when not to try something• Focus on performance early• Allow dynamic allocation of resources• Don’t allow size to dictate culture

50.
Future Predictions• “Wait till you hit 500 people, that’s when everything falls apart” • Etsy will not fall apart, but will change • Management supports and responds to cultural shifts • Freedom of movement between job functionalities • Trust that change is for the better

51.
Future Predictions• Will create better and innovative ways of communicating as we scale • Building tools is the Etsy way• Continue to influence corporate culture • B Corp • Employee happiness

52.
Office Hours Sponsor Pavilion Table A 2:00 to Last Call at the BarMichael Rembetsy (@mrembetsy)Patrick McDonnell (@mcdonnps) https://github.com/etsy