Managing experimentation

7,695 views
7,336 views

Published on

Published in: Technology, Real Estate
0 Comments
15 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,695
On SlideShare
0
From Embeds
0
Number of Embeds
885
Actions
Shares
0
Downloads
56
Comments
0
Likes
15
Embeds 0
No embeds

No notes for slide

Managing experimentation

  1. 1. Wil Stuckey @quiiver
  2. 2. 25 Million Members
  3. 3. 25 Million Members 900,000 Shops `
  4. 4. 25 Million Members 900,000 Shops 1.4B Page Views / Month
  5. 5. Managing Experimentation
  6. 6. Managing Experimentation
  7. 7. Managing Launches
  8. 8. 1. Experiments 2. Rampups 3. Communications
  9. 9. Life Before Continuous Deployments
  10. 10. Initial Idea & Planning
  11. 11. Geocities-ize
  12. 12. Initial Idea & Planning Design & Development
  13. 13. Initial Idea & Planning Deploy & Launch! Design & Development
  14. 14. Pain?
  15. 15. Testing?
  16. 16. Communication?
  17. 17. How can we do better?
  18. 18. Living in a Continuous Deployment World
  19. 19. Total Deploys: 4,040 Total Commits: 45,447 Total Unique Authors: 220 2013 ... So far
  20. 20. Total Deploys: 4,040 Total Commits: 45,447 Total Unique Authors: 220 Average Deploys per day: 35.75 Average Authors per deploy: 7.01 Average Commits per deploy: 11.25 2013 ... So far
  21. 21. That’s a lot of people
  22. 22. We are only halfway through the year!
  23. 23. CHAOS
  24. 24. Config System $server_config['my_awesome_feature'] = [ 'enabled' => 10, 'admin' => 'on', 'groups' => [1234], 'users' => ['wstuckey', 'dmckinley'], ];
  25. 25. Branching in Code if (Feature::isEnabled('my_awesome_feature')) { // display my_awesome_feature... } else { // continue on }
  26. 26. Typical Launch Cycle
  27. 27. Initial Idea & Planning
  28. 28. Initial Idea & Planning Design & Development Launch!
  29. 29. Initial Idea & Planning Design & Development Launch! Internal Admin Launch
  30. 30. Initial Idea & Planning Design & Development Launch! Internal Admin Launch Public Prototype
  31. 31. Initial Idea & Planning Design & Development Launch! Internal Admin Launch Public Prototype 50% Experiment
  32. 32. • Small iterative changes
  33. 33. • Small iterative changes • Product exists in prod as soon as possible
  34. 34. • Small iterative changes • Product exists in prod as soon as possible • Can be available for small groups for testing and feedback.
  35. 35. • Small iterative changes • Product exists in prod as soon as possible • Can be available for small groups for testing and feedback. • Easy to communicate milestones because they happen as part of the development process.
  36. 36. Because of the constant change you have to constantly communicate.
  37. 37. Start Simple
  38. 38. Wiki + Email
  39. 39. • Works great in the beginning Wiki
  40. 40. • Works great in the beginning • Quickly becomes overwhelming as you increase your concurrent launches. Wiki
  41. 41. • Works great in the beginning • Quickly becomes overwhelming as you increase your concurrent launches. • Requires manual labor to keep stakeholders informed Wiki
  42. 42. FRUSTRATION
  43. 43. We can do better!
  44. 44. Frank Harris @hirefrank
  45. 45. Launch Calendar
  46. 46. Launch Calendar • Simple web app • Collect structured meta-data about the launch • Central location for upcoming, current and past launch information
  47. 47. Launch Calendar Collect launch specific meta-data
  48. 48. Launch Calendar • Owner • Launch Date • Internal Launch Date • Team • Expected Reaction Collect launch specific meta-data
  49. 49. Launch Calendar ✉ Make your email work for you
  50. 50. ✉ Launch Calendar • Automated emails to product owners around deadlines • Daily emails to product groups containing upcoming and recently launched launches Make your email work for you
  51. 51. Launch Calendar Make your email work for you ✉
  52. 52. Launch Calendar • Highly utilized across the organization • During the year+ that it was running we recorded and tracked over 275 launches How did it go?
  53. 53. “This beats the crap out of the wiki page in every conceivable way.” - real feedback
  54. 54. What could we do better?
  55. 55. What would a more integrated product look like?
  56. 56. Can we automate more?
  57. 57. We can do better!
  58. 58. Catapult A unified launch management tool.
  59. 59. “I believe that by changing the ‘add to cart’ button to blue we will increase conversion rate.”
  60. 60. Catapult
  61. 61. $server_config['my_awesome_button'] = [ 'enabled' => 50, 'admin' => 'on', ]; Remember this?
  62. 62. Hack, Deploy, Hack, Hack Deploy!
  63. 63. Integration is awesome.
  64. 64. Communication is awesome.
  65. 65. Start Simple.
  66. 66. Build process that enable you to Ship.
  67. 67. Never stop deploying!
  68. 68. Thank You!
  69. 69. Wil Stuckey @quiiver

×