Managing Experimentation in a Continuously Deployed Environment

737 views
515 views

Published on

Video and slides synchronized, mp3 and slide download available at http://bit.ly/12Xi2lC.

Wil Stuckey explains how Etsy manages to deploy nearly ~10,000 changes in one year, and how they run A/B experiments in the midst of continual code change.Filmed at qconnewyork.com.

Wil Stuckey is Software Engineer at Etsy.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
737
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Managing Experimentation in a Continuously Deployed Environment

  1. 1. Wil Stuckey @quiiver
  2. 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /etsy-deploy
  3. 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. 4. 25 Million Members
  5. 5. 25 Million Members 900,000 Shops `
  6. 6. 25 Million Members 900,000 Shops 1.4B Page Views / Month
  7. 7. Managing Experimentation
  8. 8. Managing Experimentation
  9. 9. Managing Launches
  10. 10. 1. Experiments 2. Rampups 3. Communications
  11. 11. Life Before Continuous Deployments
  12. 12. Initial Idea & Planning
  13. 13. Geocities-ize
  14. 14. Initial Idea & Planning Design & Development
  15. 15. Initial Idea & Planning Deploy & Launch! Design & Development
  16. 16. Pain?
  17. 17. Testing?
  18. 18. Communication?
  19. 19. How can we do better?
  20. 20. Living in a Continuous Deployment World
  21. 21. Total Deploys: 4,040 Total Commits: 45,447 Total Unique Authors: 220 2013 ... So far
  22. 22. Total Deploys: 4,040 Total Commits: 45,447 Total Unique Authors: 220 Average Deploys per day: 35.75 Average Authors per deploy: 7.01 Average Commits per deploy: 11.25 2013 ... So far
  23. 23. That’s a lot of people
  24. 24. We are only halfway through the year!
  25. 25. CHAOS
  26. 26. Config System $server_config['my_awesome_feature'] = [ 'enabled' => 10, 'admin' => 'on', 'groups' => [1234], 'users' => ['wstuckey', 'dmckinley'], ];
  27. 27. Branching in Code if (Feature::isEnabled('my_awesome_feature')) { // display my_awesome_feature... } else { // continue on }
  28. 28. Typical Launch Cycle
  29. 29. Initial Idea & Planning
  30. 30. Initial Idea & Planning Design & Development Launch!
  31. 31. Initial Idea & Planning Design & Development Launch! Internal Admin Launch
  32. 32. Initial Idea & Planning Design & Development Launch! Internal Admin Launch Public Prototype
  33. 33. Initial Idea & Planning Design & Development Launch! Internal Admin Launch Public Prototype 50% Experiment
  34. 34. • Small iterative changes
  35. 35. • Small iterative changes • Product exists in prod as soon as possible
  36. 36. • Small iterative changes • Product exists in prod as soon as possible • Can be available for small groups for testing and feedback.
  37. 37. • Small iterative changes • Product exists in prod as soon as possible • Can be available for small groups for testing and feedback. • Easy to communicate milestones because they happen as part of the development process.
  38. 38. Because of the constant change you have to constantly communicate.
  39. 39. Start Simple
  40. 40. Wiki + Email
  41. 41. • Works great in the beginning Wiki
  42. 42. • Works great in the beginning • Quickly becomes overwhelming as you increase your concurrent launches. Wiki
  43. 43. • Works great in the beginning • Quickly becomes overwhelming as you increase your concurrent launches. • Requires manual labor to keep stakeholders informed Wiki
  44. 44. FRUSTRATION
  45. 45. We can do better!
  46. 46. Frank Harris @hirefrank
  47. 47. Launch Calendar
  48. 48. Launch Calendar • Simple web app • Collect structured meta-data about the launch • Central location for upcoming, current and past launch information
  49. 49. Launch Calendar Collect launch specific meta-data
  50. 50. Launch Calendar • Owner • Launch Date • Internal Launch Date • Team • Expected Reaction Collect launch specific meta-data
  51. 51. Launch Calendar ✉ Make your email work for you
  52. 52. ✉ Launch Calendar • Automated emails to product owners around deadlines • Daily emails to product groups containing upcoming and recently launched launches Make your email work for you
  53. 53. Launch Calendar Make your email work for you ✉
  54. 54. Launch Calendar • Highly utilized across the organization • During the year+ that it was running we recorded and tracked over 275 launches How did it go?
  55. 55. “This beats the crap out of the wiki page in every conceivable way.” - real feedback
  56. 56. What could we do better?
  57. 57. What would a more integrated product look like?
  58. 58. Can we automate more?
  59. 59. We can do better!
  60. 60. Catapult A unified launch management tool.
  61. 61. “I believe that by changing the ‘add to cart’ button to blue we will increase conversion rate.”
  62. 62. Catapult
  63. 63. $server_config['my_awesome_button'] = [ 'enabled' => 50, 'admin' => 'on', ]; Remember this?
  64. 64. Hack, Deploy, Hack, Hack Deploy!
  65. 65. Integration is awesome.
  66. 66. Communication is awesome.
  67. 67. Start Simple.
  68. 68. Build process that enable you to Ship.
  69. 69. Never stop deploying!
  70. 70. Thank You!
  71. 71. Wil Stuckey @quiiver

×