Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Continuously Deploying 
Culture 2.0 
A Story Of Scaling Engineering Culture at Etsy 
Rich Smith 
Director of Security, Ets...
Who Am I ? 
• Rich Smith, Brooklyn NYC 
• Director of Security Engineering at Etsy 
• Cover the AppSec, NetSec, Risk Engin...
@iodboi 
WTF is ! 
• Online community for 
handcrafted & vintage 
items 
• Human scale 
manufacturing 
• Marketplace for s...
@iodboi 
Now (FY 2013) 
• Gross Marketplace Sales (GMS) $1.35 Billion 
• 40 million members, 1 million active sellers 
• 2...
@iodboi 
Core Engineering Principles 
• Empower the edges 
• Trustworthy not trusted 
• Every engineer can push to prod at...
Very end of 2009 Today 
@iodboi 
Pushes Per Day
But How Did We Get 
There?
What’s Etsy’s Story of 
Engineering Culture? 
@iodboi
Continuously Deploying Culture v1 
• Mike Rembetsy & Patrick McDonnell 
• Gave a talk at the Velocity conferences in 2011/...
Disclaimer 
A + B != Culture
2006 - 2008 Silos and Barriers 
• Etsy 4 person startup grows to employ 30 - 35 FTE’s 
• Around 15 engineers 
• A very sil...
Management Changes 
• Maria Thomas from NPR promoted to CEO 
• Brings a clear understanding that community is very importa...
@iodboi 
2006-2008 Takeaways 
• Downtime was an accepted fact of life 
• It was even expected to a degree! 
• Engineering ...
@iodboi 
2008 End of Year Snapshot 
Employees Engineers 
35 15 
$87.5 Million 
GMS
2009 Internal Improvements 
• As teams grow, big efforts in good communication 
• Daily standups begun 
• Much better cros...
2009 Takeaways 
• Big growth year 
• Built solid foundations: 
• Infrastructure 
• Invested in human capital 
• ‘DevOps’ c...
@iodboi 
2009 End of Year Snapshot 
GMS 
$180.6 Million
2010 Renewed Energy
2010 Standardisation & Graphs 
• Moved to PHP & MySQL for * 
• It almost doesn’t matter what you choose, just stick to it ...
**
Management Ideals Engineering Ideals 
@iodboi 
2010 Following our ideals ….. 
• Blameless PostMortems 
• 1:1’s as a core m...
2010 Takeaways 
• Reduce number of technologies used in development 
• Focus on technical visibility throughout the org 
•...
@iodboi 
2010 End of Year Snapshot 
GMS 
$314.3 Million
2011 Tech highlights 
• End of long tail legacy silo holdovers (Sprouter gone!) 
• Non-Standard technologies removed from ...
2011 - Organisational Changes 
• Snr. management to become more Engineering focused 
• Chad to CEO 
• Kellan to CTO 
• All...
2011 Takeaways 
• Year of the Open Sourced tool 
• Statsd, Logster, Deployinator, Supergrep, Schemanator …. 
• Overall mat...
@iodboi 
2011 End of Year Snapshot 
GMS 
$525.6 Million
Let’s catch our breath!
2011-2012 - A Focus on Security 
• Security alongside Dev & Ops as being integral to culture 
• Applying our ‘DevOps’ prin...
2012 - Growth + Foster Our Values 
• Explosive growth in hiring, allow easy transfers 
• Some major changes around product...
What’s a B-Corp ? 
• Aim to use the power of business to solve social & 
environmental issues 
• Impacts engineering in ne...
2012 - Technical Achievements 
• Create wholly separate payments environment 
• Allows PCI compliance without disrupting t...
2012 Takeaways 
• Do what’s needed to sustain long term & not just keep 
the lights on 
• More headcount than required all...
@iodboi 
2012 End of Year Snapshot 
GMS 
$895.1 Million
2012 Action Items 
• Security is integral DevOps lifecycle and culture 
• Know when to flick sights from short to longer t...
2013 - An Interesting Year! 
• Had many of the hard engineering wins taken care of … 
• Time to focus internally 
• No eng...
2013 - Technical 
• Morgue tool created to capture and aid postmortems 
• Moved to Vertica for BI data & metrics 
• Superb...
2013 Takeaways 
• Democratisation of data is made easier with tooling that 
levels access and allows interrogation by ALL ...
@iodboi 
2013 End of Year Snapshot 
Employees Engineers 
615 >33% 
$1.35 Billion 
GMS
2013 Action Items 
• Datasets grow, evaluate how they can be accessed, 
evaluated and contextualized 
• Have you reached a...
2014 - Organisational changes 
• Everyone pushes on their 1st day, not just Engineers 
• Yearly planning is restructured 
...
Cultural Acquisition 
• As part of growth, Paris based A Little Market acquired 
• Integrating another engineering culture...
2014 - Technical 
• Move away from Splunk and to ElasticSearch/Logstash/ 
Kibana (ELK) 
• Mobile CI infrastructure embedde...
Logstash Lessons 
• Replacing Splunk with LogStash taught many lessons 
• Changing of a core tool require huge comms inves...
API First 
• Supporting the Mobile First push & diversity of clients 
• No longer assume LAMP, decoupling required 
• Adds...
Mobile First 
• Applying your principles and culture to the changing 
tech landscape is key 
• Continuous Deployment hard ...
Continuous Deployment Continuous Delivery 
Frequent checkins 
directly to mainline ✓ ✓ 
Automated build & test 
cycle ✓ ✓ ...
Why This Approach? 
• Continuous integration, Continuous Delivery 
• Build your apps in a reproducible way after each push...
Single 
release 
Many 
releases 
50K LOC/month 
Few opportunities for failure 
Wide surface area (50,000 LOC) 
High MTTR 
...
@iodboi 
Sleep Tracking 
• Experiment with fitbands & Ops 
• Collect sleep data for on-call 
• Analyse in a variety of man...
2014 Takeaways 
• Another year of big growth, also now M&A 
• Integrating other engineering cultures inside your own 
is a...
2014 Action Items 
• Culture is still king despite growth or M&A activity 
• It takes effort to keep it so however 
• Ensu...
Credit for the awesome graph: Patrick Koch
Conclusions 
• Culture doesn’t come for free, it takes continuous work 
• Iterate & improve - Even when you think you have...
Never. Stop. Pushing.
@iodboi 
Questions?
@iodboi 
Links / References 
Continuously Deploying Culture 
(Mike Rembetsy, Patrick McDonnell) 
Slides: http://slidesha.r...
Contiuously Deploying Culture 2.0 - Agile Ísland
Contiuously Deploying Culture 2.0 - Agile Ísland
Contiuously Deploying Culture 2.0 - Agile Ísland
Contiuously Deploying Culture 2.0 - Agile Ísland
Contiuously Deploying Culture 2.0 - Agile Ísland
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
Continuous Deployment at Etsy — TimesOpen NYC
Next
Upcoming SlideShare
Continuous Deployment at Etsy — TimesOpen NYC
Next
Download to read offline and view in fullscreen.

6

Share

Contiuously Deploying Culture 2.0 - Agile Ísland

Download to read offline

Presentation discussing the story of engineering culture at Etsy and the lessons learnt of maintaining a genuine and engaging culture in a rapidly growing technology company.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Contiuously Deploying Culture 2.0 - Agile Ísland

  1. 1. Continuously Deploying Culture 2.0 A Story Of Scaling Engineering Culture at Etsy Rich Smith Director of Security, Etsy @iodboi Agile Iceland 2014
  2. 2. Who Am I ? • Rich Smith, Brooklyn NYC • Director of Security Engineering at Etsy • Cover the AppSec, NetSec, Risk Engineering teams • Also building the security organisation & security culture • Co-Founder of here in Iceland ! @iodboi
  3. 3. @iodboi WTF is ! • Online community for handcrafted & vintage items • Human scale manufacturing • Marketplace for small businesses
  4. 4. @iodboi Now (FY 2013) • Gross Marketplace Sales (GMS) $1.35 Billion • 40 million members, 1 million active sellers • 26 million active listings • 200+ Countries Performing Transactions • >615 Employees • Offices in 8 countries
  5. 5. @iodboi Core Engineering Principles • Empower the edges • Trustworthy not trusted • Every engineer can push to prod at any time • ‘Just Ship’ - Get things done • ‘If it moves graph it’ - Let the data lead you
  6. 6. Very end of 2009 Today @iodboi Pushes Per Day
  7. 7. But How Did We Get There?
  8. 8. What’s Etsy’s Story of Engineering Culture? @iodboi
  9. 9. Continuously Deploying Culture v1 • Mike Rembetsy & Patrick McDonnell • Gave a talk at the Velocity conferences in 2011/12 • Etsy’s engineering culture evolution 2006-2011/12 • Slides here: http://slidesha.re/1xYxZrG Watch it here: http://vimeo.com/51310058 • Today we are extending those lessons up to the @iodboi
  10. 10. Disclaimer A + B != Culture
  11. 11. 2006 - 2008 Silos and Barriers • Etsy 4 person startup grows to employ 30 - 35 FTE’s • Around 15 engineers • A very siloed culture, creates barriers to engineering collaboration • Bred initiatives like Sprouter - ‘Middleware of distrust!’ • Project dedicated to stopping engineers touching databases @iodboi
  12. 12. Management Changes • Maria Thomas from NPR promoted to CEO • Brings a clear understanding that community is very important • Prioritises a culture that supports community • Chad Dickerson brought on as CTO • Brings a clearer focus to the engineering team • ‘This Silo’d culture cannot work, we need to start over’ @iodboi
  13. 13. @iodboi 2006-2008 Takeaways • Downtime was an accepted fact of life • It was even expected to a degree! • Engineering projects were often low impact • Community needs to be a technical focus • Survived the holiday season … just!
  14. 14. @iodboi 2008 End of Year Snapshot Employees Engineers 35 15 $87.5 Million GMS
  15. 15. 2009 Internal Improvements • As teams grow, big efforts in good communication • Daily standups begun • Much better cross-team collaboration between Ops & others • Network solidified and provided basis for future growth • Moved from Downtown Brooklyn to DUMBO @iodboi
  16. 16. 2009 Takeaways • Big growth year • Built solid foundations: • Infrastructure • Invested in human capital • ‘DevOps’ culture begins in earnest ….. • A lot of reflection and finding an Engineering identity @iodboi
  17. 17. @iodboi 2009 End of Year Snapshot GMS $180.6 Million
  18. 18. 2010 Renewed Energy
  19. 19. 2010 Standardisation & Graphs • Moved to PHP & MySQL for * • It almost doesn’t matter what you choose, just stick to it • ‘If it moves Graph it’ • Graphite, Ganglia, FITB, Weathermap, Nagios, Naglite ….. • Starting to use this data for work/life balance as well as technical/systems reasons @iodboi
  20. 20. **
  21. 21. Management Ideals Engineering Ideals @iodboi 2010 Following our ideals ….. • Blameless PostMortems • 1:1’s as a core mgmt tool • Eng career planning (Reverb) • Accept failures, but not low standards • Developer on-call • Use of A/B testing • Lots of Prototypes • FeatureFlags & Ramp Up
  22. 22. 2010 Takeaways • Reduce number of technologies used in development • Focus on technical visibility throughout the org • Developers responsible for code release (start of DevOps) • Member support rotations for all • Work hard at work/life balance & have data to support @iodboi
  23. 23. @iodboi 2010 End of Year Snapshot GMS $314.3 Million
  24. 24. 2011 Tech highlights • End of long tail legacy silo holdovers (Sprouter gone!) • Non-Standard technologies removed from production • Engineers receive 3 annual goals: • Speak at a conference • Write a blog post • Release open source software @iodboi
  25. 25. 2011 - Organisational Changes • Snr. management to become more Engineering focused • Chad to CEO • Kellan to CTO • Allspaw to SVP of Operations • Consolidates importance of engineering culture to the very top of Etsy and increases stability @iodboi
  26. 26. 2011 Takeaways • Year of the Open Sourced tool • Statsd, Logster, Deployinator, Supergrep, Schemanator …. • Overall maturing of engineering - platform & people • Automation & config management solidified (Chef everything) • Security starts becoming a 1st class citizen • ‘Security Culture at Etsy’ begins to be chased & discussed @iodboi
  27. 27. @iodboi 2011 End of Year Snapshot GMS $525.6 Million
  28. 28. Let’s catch our breath!
  29. 29. 2011-2012 - A Focus on Security • Security alongside Dev & Ops as being integral to culture • Applying our ‘DevOps’ principles & learnings to security • Emphasis on security being a facilitator not a blocker • Security often ‘enforced’ with terrible cultural impact • Build a human and effective security organization @iodboi
  30. 30. 2012 - Growth + Foster Our Values • Explosive growth in hiring, allow easy transfers • Some major changes around product • Increased focus on community • Internationalisation • High impact products (Shipping Labels, Gift Cards) • Became a certified B-Corp - not just the bottom line @iodboi
  31. 31. What’s a B-Corp ? • Aim to use the power of business to solve social & environmental issues • Impacts engineering in new and interesting ways: • Waste, Recycling, Compost, Flushes (Yes we graph them!) • Efficiency of our tech, data centre usage & partners • ‘Make the world more like Etsy’ - Extending the culture @iodboi
  32. 32. 2012 - Technical Achievements • Create wholly separate payments environment • Allows PCI compliance without disrupting the culture • Interface with the webstack via a restricted Internet facing API • Get serious on Data Science • Dedicated Hadoop cluster for full time data scientists • Taking some chances and broadening of our engineers @iodboi
  33. 33. 2012 Takeaways • Do what’s needed to sustain long term & not just keep the lights on • More headcount than required allows us to take chances • Focus on info exchange, internally & externally with communities • Open source all of the things @iodboi
  34. 34. @iodboi 2012 End of Year Snapshot GMS $895.1 Million
  35. 35. 2012 Action Items • Security is integral DevOps lifecycle and culture • Know when to flick sights from short to longer term goals • Pursue dynamic engineering resource allocation • Do not allow increasing org size to dictate culture @iodboi
  36. 36. 2013 - An Interesting Year! • Had many of the hard engineering wins taken care of … • Time to focus internally • No engineer can know everything any longer • Need to maintain the culture of transparency & trust • Really was the year of internal tooling to achieve this @iodboi
  37. 37. 2013 - Technical • Morgue tool created to capture and aid postmortems • Moved to Vertica for BI data & metrics • Superbit allowed simple querying of Vertica & big data by anyone who knows SQL • Catapult launched to relate metrics to experiments • Begin a refocus on a Mobile/API First product vision @iodboi
  38. 38. 2013 Takeaways • Democratisation of data is made easier with tooling that levels access and allows interrogation by ALL • Conscious effort on internal tooling to minimize the pain of large & complex stack • Engineering invested in transparency & trust • The world doesn’t wait, mobile is the future @iodboi
  39. 39. @iodboi 2013 End of Year Snapshot Employees Engineers 615 >33% $1.35 Billion GMS
  40. 40. 2013 Action Items • Datasets grow, evaluate how they can be accessed, evaluated and contextualized • Have you reached a point where no one can know everything? • While tooling can’t create culture it can help you support it • Be free to apply your culture in new ways • Inward focus cannot lead to outward blindness, tech changes fast @iodboi
  41. 41. 2014 - Organisational changes • Everyone pushes on their 1st day, not just Engineers • Yearly planning is restructured • Take account of a growing Etsy • San Francisco opens as 1st non-Brooklyn Eng hub • Acquire & integrate A Little Market with Etsy @iodboi
  42. 42. Cultural Acquisition • As part of growth, Paris based A Little Market acquired • Integrating another engineering culture can be tough • Etsy’s culture is ‘different’ & this can be a big step • Language, timezone and human cultural differences • Can be very successful, but don’t underestimate @iodboi
  43. 43. 2014 - Technical • Move away from Splunk and to ElasticSearch/Logstash/ Kibana (ELK) • Mobile CI infrastructure embedded & ramped up • API First a huge effort and development push • Mobile First as an increasing product focus • Technical work for quality of life - on-call sleep tracking @iodboi
  44. 44. Logstash Lessons • Replacing Splunk with LogStash taught many lessons • Changing of a core tool require huge comms investment • Without it enclaves & silos can form to resist change • Explain the whys not just in terms of technicals or $$ • Fully understanding all use cases, not just the main ones • Don’t settle for a half complete end goal, go the distance @iodboi
  45. 45. API First • Supporting the Mobile First push & diversity of clients • No longer assume LAMP, decoupling required • Adds security & agility • Embeds fundamental future resilience • Capacity planning becomes more challenging @iodboi
  46. 46. Mobile First • Applying your principles and culture to the changing tech landscape is key • Continuous Deployment hard in the ‘App Store world’ • Continuous Integration still applies of course • Continuous Deployment becomes Continuous Delivery • Still use API to enable feature flag driven native apps @iodboi
  47. 47. Continuous Deployment Continuous Delivery Frequent checkins directly to mainline ✓ ✓ Automated build & test cycle ✓ ✓ Keep the build green, always ready to release ✓ ✓ One button deploys ✓ ✓ Business dictates when to deploy ✓ Every passing build deployed to prod ✓ All enhancements gated by feature flag ✓ ?
  48. 48. Why This Approach? • Continuous integration, Continuous Delivery • Build your apps in a reproducible way after each push to git • Identify bugs, missing dependencies early & often • Integrate security testing throughout lifecycle • Improve Mean Time To Recovery • Stop stressing about releases! @iodboi
  49. 49. Single release Many releases 50K LOC/month Few opportunities for failure Wide surface area (50,000 LOC) High MTTR ! All of the bugs we’ve written More opportunities for failure Narrow surface area (< 100 LOC) Low MTTR ! A fraction of the bugs we’ve written per release Imagine that we’ll write
  50. 50. @iodboi Sleep Tracking • Experiment with fitbands & Ops • Collect sleep data for on-call • Analyse in a variety of manners • Sleep lost when on-call/pagerduty • Alert on VPN/SSH logins while asleep • Focus on data for quality of life
  51. 51. 2014 Takeaways • Another year of big growth, also now M&A • Integrating other engineering cultures inside your own is a challenge you should prepare for • Core tooling changes require great thought & comms • Mobile focus does not mean the end of always pushing • Tooling for happiness & W/L balance is a win for all @iodboi
  52. 52. 2014 Action Items • Culture is still king despite growth or M&A activity • It takes effort to keep it so however • Ensure your API is up to the job of supporting Mobile 1st • Ensure core tooling changes are understood & embraced by all • Communicate your Eng culture & history to new hires @iodboi
  53. 53. Credit for the awesome graph: Patrick Koch
  54. 54. Conclusions • Culture doesn’t come for free, it takes continuous work • Iterate & improve - Even when you think you have ‘it’ • Don’t give in to potential disruptors like growth & security and let them destroy your culture • Get smart and use them to test, support and improve it @iodboi
  55. 55. Never. Stop. Pushing.
  56. 56. @iodboi Questions?
  57. 57. @iodboi Links / References Continuously Deploying Culture (Mike Rembetsy, Patrick McDonnell) Slides: http://slidesha.re/1xYxZrG Video: http://vimeo.com/51310058 Scaling Etsy, what went wrong, what went right (Ross Snyder) Slides & Video: http://bit.ly/po8zIj Etsy’s journey to continuous integration for mobile apps (Nassim Kammah) Blog post: http://bit.ly/1yiGWwc Mean time to sleep (Ryan Frantz, Laurie Denness) Slides, Blog post, code: http://ryanfrantz.com/mtts/
  • pesmou

    Jan. 8, 2017
  • HYDN

    Nov. 28, 2016
  • rakeshnagar

    Jan. 18, 2016
  • unforgettable1

    Dec. 1, 2015
  • juliamarieee

    Dec. 15, 2014
  • apatiev

    Nov. 11, 2014

Presentation discussing the story of engineering culture at Etsy and the lessons learnt of maintaining a genuine and engaging culture in a rapidly growing technology company.

Views

Total views

3,102

On Slideshare

0

From embeds

0

Number of embeds

19

Actions

Downloads

40

Shares

0

Comments

0

Likes

6

×