Wil Stuckey
@quiiver
25 Million Members
25 Million Members
900,000 Shops
`
25 Million Members
900,000 Shops
1.4B Page Views / Month
Managing
Experimentation
Managing
Experimentation
Managing
Launches
1. Experiments
2. Rampups
3. Communications
Life Before Continuous
Deployments
Initial Idea &
Planning
Geocities-ize
Initial Idea &
Planning
Design &
Development
Initial Idea &
Planning
Deploy &
Launch!
Design &
Development
Pain?
Testing?
Communication?
How can we do better?
Living in a
Continuous
Deployment World
Total Deploys: 4,040
Total Commits: 45,447
Total Unique Authors: 220
2013 ... So far
Total Deploys: 4,040
Total Commits: 45,447
Total Unique Authors: 220
Average Deploys per day: 35.75
Average Authors per de...
That’s a lot of people
We are only halfway
through the year!
CHAOS
Config System
$server_config['my_awesome_feature'] = [
'enabled' => 10,
'admin' => 'on',
'groups' => [1234],
'users' => ['w...
Branching in Code
if (Feature::isEnabled('my_awesome_feature')) {
// display my_awesome_feature...
} else {
// continue on...
Typical Launch
Cycle
Initial Idea &
Planning
Initial Idea &
Planning
Design &
Development
Launch!
Initial Idea &
Planning
Design &
Development
Launch!
Internal Admin Launch
Initial Idea &
Planning
Design &
Development
Launch!
Internal Admin Launch
Public Prototype
Initial Idea &
Planning
Design &
Development
Launch!
Internal Admin Launch
Public Prototype
50% Experiment
• Small iterative changes
• Small iterative changes
• Product exists in prod as soon as possible
• Small iterative changes
• Product exists in prod as soon as possible
• Can be available for small groups for
testing and...
• Small iterative changes
• Product exists in prod as soon as possible
• Can be available for small groups for
testing and...
Because of the
constant change you
have to constantly
communicate.
Start Simple
Wiki + Email
• Works great in the
beginning
Wiki
• Works great in the
beginning
• Quickly becomes
overwhelming as you
increase your
concurrent launches.
Wiki
• Works great in the
beginning
• Quickly becomes
overwhelming as you
increase your
concurrent launches.
• Requires manual ...
FRUSTRATION
We can do better!
Frank Harris
@hirefrank
Launch Calendar
Launch Calendar
• Simple web app
• Collect structured meta-data about the
launch
• Central location for upcoming, current
...
Launch Calendar
Collect launch specific meta-data
Launch Calendar
• Owner
• Launch Date
• Internal Launch Date
• Team
• Expected Reaction
Collect launch specific meta-data
Launch Calendar
✉
Make your email work for you
✉
Launch Calendar
• Automated emails to product owners
around deadlines
• Daily emails to product groups containing
upcomi...
Launch Calendar
Make your email work for you
✉
Launch Calendar
• Highly utilized across the organization
• During the year+ that it was running we
recorded and tracked o...
“This beats the crap out
of the wiki page in every
conceivable way.” - real
feedback
What could we do better?
What would a more
integrated product look
like?
Can we automate more?
We can do better!
Catapult
A unified launch management tool.
“I believe that by changing
the ‘add to cart’ button to
blue we will increase
conversion rate.”
Catapult
$server_config['my_awesome_button'] = [
'enabled' => 50,
'admin' => 'on',
];
Remember this?
Hack, Deploy, Hack, Hack
Deploy!
Integration is
awesome.
Communication is
awesome.
Start Simple.
Build process that
enable you to Ship.
Never stop
deploying!
Thank You!
Wil Stuckey
@quiiver
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Upcoming SlideShare
Loading in...5
×

Managing experimentation

6,066

Published on

Published in: Technology, Real Estate
0 Comments
15 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,066
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
52
Comments
0
Likes
15
Embeds 0
No embeds

No notes for slide

Managing experimentation

  1. 1. Wil Stuckey @quiiver
  2. 2. 25 Million Members
  3. 3. 25 Million Members 900,000 Shops `
  4. 4. 25 Million Members 900,000 Shops 1.4B Page Views / Month
  5. 5. Managing Experimentation
  6. 6. Managing Experimentation
  7. 7. Managing Launches
  8. 8. 1. Experiments 2. Rampups 3. Communications
  9. 9. Life Before Continuous Deployments
  10. 10. Initial Idea & Planning
  11. 11. Geocities-ize
  12. 12. Initial Idea & Planning Design & Development
  13. 13. Initial Idea & Planning Deploy & Launch! Design & Development
  14. 14. Pain?
  15. 15. Testing?
  16. 16. Communication?
  17. 17. How can we do better?
  18. 18. Living in a Continuous Deployment World
  19. 19. Total Deploys: 4,040 Total Commits: 45,447 Total Unique Authors: 220 2013 ... So far
  20. 20. Total Deploys: 4,040 Total Commits: 45,447 Total Unique Authors: 220 Average Deploys per day: 35.75 Average Authors per deploy: 7.01 Average Commits per deploy: 11.25 2013 ... So far
  21. 21. That’s a lot of people
  22. 22. We are only halfway through the year!
  23. 23. CHAOS
  24. 24. Config System $server_config['my_awesome_feature'] = [ 'enabled' => 10, 'admin' => 'on', 'groups' => [1234], 'users' => ['wstuckey', 'dmckinley'], ];
  25. 25. Branching in Code if (Feature::isEnabled('my_awesome_feature')) { // display my_awesome_feature... } else { // continue on }
  26. 26. Typical Launch Cycle
  27. 27. Initial Idea & Planning
  28. 28. Initial Idea & Planning Design & Development Launch!
  29. 29. Initial Idea & Planning Design & Development Launch! Internal Admin Launch
  30. 30. Initial Idea & Planning Design & Development Launch! Internal Admin Launch Public Prototype
  31. 31. Initial Idea & Planning Design & Development Launch! Internal Admin Launch Public Prototype 50% Experiment
  32. 32. • Small iterative changes
  33. 33. • Small iterative changes • Product exists in prod as soon as possible
  34. 34. • Small iterative changes • Product exists in prod as soon as possible • Can be available for small groups for testing and feedback.
  35. 35. • Small iterative changes • Product exists in prod as soon as possible • Can be available for small groups for testing and feedback. • Easy to communicate milestones because they happen as part of the development process.
  36. 36. Because of the constant change you have to constantly communicate.
  37. 37. Start Simple
  38. 38. Wiki + Email
  39. 39. • Works great in the beginning Wiki
  40. 40. • Works great in the beginning • Quickly becomes overwhelming as you increase your concurrent launches. Wiki
  41. 41. • Works great in the beginning • Quickly becomes overwhelming as you increase your concurrent launches. • Requires manual labor to keep stakeholders informed Wiki
  42. 42. FRUSTRATION
  43. 43. We can do better!
  44. 44. Frank Harris @hirefrank
  45. 45. Launch Calendar
  46. 46. Launch Calendar • Simple web app • Collect structured meta-data about the launch • Central location for upcoming, current and past launch information
  47. 47. Launch Calendar Collect launch specific meta-data
  48. 48. Launch Calendar • Owner • Launch Date • Internal Launch Date • Team • Expected Reaction Collect launch specific meta-data
  49. 49. Launch Calendar ✉ Make your email work for you
  50. 50. ✉ Launch Calendar • Automated emails to product owners around deadlines • Daily emails to product groups containing upcoming and recently launched launches Make your email work for you
  51. 51. Launch Calendar Make your email work for you ✉
  52. 52. Launch Calendar • Highly utilized across the organization • During the year+ that it was running we recorded and tracked over 275 launches How did it go?
  53. 53. “This beats the crap out of the wiki page in every conceivable way.” - real feedback
  54. 54. What could we do better?
  55. 55. What would a more integrated product look like?
  56. 56. Can we automate more?
  57. 57. We can do better!
  58. 58. Catapult A unified launch management tool.
  59. 59. “I believe that by changing the ‘add to cart’ button to blue we will increase conversion rate.”
  60. 60. Catapult
  61. 61. $server_config['my_awesome_button'] = [ 'enabled' => 50, 'admin' => 'on', ]; Remember this?
  62. 62. Hack, Deploy, Hack, Hack Deploy!
  63. 63. Integration is awesome.
  64. 64. Communication is awesome.
  65. 65. Start Simple.
  66. 66. Build process that enable you to Ship.
  67. 67. Never stop deploying!
  68. 68. Thank You!
  69. 69. Wil Stuckey @quiiver
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×