Continuous Delivery and Rapid Experimentation

1,151 views

Published on

A presentation I gave at Agile Australia 2013 on Continuous Delivery

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Continuous Delivery and Rapid Experimentation

  1. 1. Continuous Delivery The Path to Rapid Experimentation Paul Coia
  2. 2. Deploy Experiment Example Lessons
  3. 3. Global platform for independent artists and designers to sell their work more easily, more oen.
  4. 4. 10M designs on 40M products 500,000 artists all over the world. 40M pageviews from 7M Users/month
  5. 5. Deploy Experiment Example Lessons
  6. 6. 2000 releases 4 years 700 in 2012 Average 3 per day
  7. 7. cap production deploy
  8. 8. 1. Continuous Integration success 2. Deploy code via Capistrano 3. Monitor vital signs
  9. 9. Jenkins Continuous Integration Server
  10. 10. New Relic Application Monitoring
  11. 11. New Relic Key Transaction Monitoring
  12. 12. Airbrake Error Reporting
  13. 13. Circonus and Statsd Business Metrics Monitoring
  14. 14. StatsdClient.increment("Payment OK")
  15. 15. if enrolled_in?(“new”).variant? // show variant else // show normal end
  16. 16. Minimal cost of deploying Rapid pace of change possible Confidence make lots of changes Try ideas quicker Why we like it
  17. 17. Deploy Experiment Example Lessons
  18. 18. 1.Observe 2.Form a hypothesis 3.Devise an experiment to test it 4.Draw conclusions Scientific Method
  19. 19. Quantitive Research - What? Qualitative Research - Why?
  20. 20. “If the SEARCH box was bigger, then users will be more likely to search”
  21. 21. Split users at point of variation Isolate experiments Persist the decision across visits Exclude crawlers and ‘old’ users Enrollment
  22. 22. // ON LANDING PAGE if new_visitor() @guinea_pig = true end // ON EXPERIMENT PAGE if @guinea_pig && !retrieve_enrolment() experiment = enrol_in_experiment(“new”) persist_enrolment(experiment) end if enrolled_in?(“new”).variant? // experiment logic else // normal logic end
  23. 23. RUNNING_EXPERIMENTS = { ... new: {description: 'New feature', enrolment_percent: 20}, ... }
  24. 24. Google Analytics Data Collection
  25. 25. ‘R’ Statistical Language Google Analytics API
  26. 26. Respect Statistics Establish your test duration up front Wait for the test to complete Results
  27. 27. Model your Product funnel Greater volume earlier in the funnel Higher conversion rate between steps Add lots of instrumentation Useful metrics
  28. 28. Homepage Sign up page Signed up
  29. 29. 1% Homepage Sign up page Signed up
  30. 30. 1% Homepage Sign up page Signed up 20% 5%
  31. 31. ‘Landing Page’ Optimizers Client-side changes only Simple experiments
  32. 32. Trivial deployment Feature toggle Experimentation platform = Trivial feature experimentation
  33. 33. Deploy Experiment Example Lessons
  34. 34. 22% less likely to Add to Cart 50% less likely to Checkout Mobile Visitors
  35. 35. No change to Checkout Completion
  36. 36. Improvement in Checkout Completion
  37. 37. Increase in Add to Cart rate
  38. 38. Further Increase in Add to Cart rate
  39. 39. Multiple lightweight experiments A handful of weeks Verified 30% increase in Purchasing
  40. 40. Deploy Experiment Example Lessons
  41. 41. Invest in your deployment tools
  42. 42. Experiment on the small scale
  43. 43. Expect to be wrong
  44. 44. Build on the successes
  45. 45. Design for incremental change
  46. 46. Thank You Paul Coia paul@redbubble.com @pjcoia h p://www.redbubble.com/jobs

×