Continuous Delivery and Rapid Experimentation
Upcoming SlideShare
Loading in...5
×
 

Continuous Delivery and Rapid Experimentation

on

  • 707 views

A presentation I gave at Agile Australia 2013 on Continuous Delivery

A presentation I gave at Agile Australia 2013 on Continuous Delivery

Statistics

Views

Total Views
707
Views on SlideShare
707
Embed Views
0

Actions

Likes
0
Downloads
12
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Continuous Delivery and Rapid Experimentation Continuous Delivery and Rapid Experimentation Presentation Transcript

  • Continuous Delivery The Path to Rapid Experimentation Paul Coia
  • Deploy Experiment Example Lessons
  • Global platform for independent artists and designers to sell their work more easily, more oen.
  • 10M designs on 40M products 500,000 artists all over the world. 40M pageviews from 7M Users/month
  • Deploy Experiment Example Lessons
  • 2000 releases 4 years 700 in 2012 Average 3 per day
  • cap production deploy
  • 1. Continuous Integration success 2. Deploy code via Capistrano 3. Monitor vital signs
  • Jenkins Continuous Integration Server
  • New Relic Application Monitoring
  • New Relic Key Transaction Monitoring
  • Airbrake Error Reporting
  • Circonus and Statsd Business Metrics Monitoring
  • StatsdClient.increment("Payment OK")
  • if enrolled_in?(“new”).variant? // show variant else // show normal end
  • Minimal cost of deploying Rapid pace of change possible Confidence make lots of changes Try ideas quicker Why we like it
  • Deploy Experiment Example Lessons
  • 1.Observe 2.Form a hypothesis 3.Devise an experiment to test it 4.Draw conclusions Scientific Method
  • Quantitive Research - What? Qualitative Research - Why?
  • “If the SEARCH box was bigger, then users will be more likely to search”
  • Split users at point of variation Isolate experiments Persist the decision across visits Exclude crawlers and ‘old’ users Enrollment
  • // ON LANDING PAGE if new_visitor() @guinea_pig = true end // ON EXPERIMENT PAGE if @guinea_pig && !retrieve_enrolment() experiment = enrol_in_experiment(“new”) persist_enrolment(experiment) end if enrolled_in?(“new”).variant? // experiment logic else // normal logic end
  • RUNNING_EXPERIMENTS = { ... new: {description: 'New feature', enrolment_percent: 20}, ... }
  • Google Analytics Data Collection
  • ‘R’ Statistical Language Google Analytics API
  • Respect Statistics Establish your test duration up front Wait for the test to complete Results
  • Model your Product funnel Greater volume earlier in the funnel Higher conversion rate between steps Add lots of instrumentation Useful metrics
  • Homepage Sign up page Signed up
  • 1% Homepage Sign up page Signed up
  • 1% Homepage Sign up page Signed up 20% 5%
  • ‘Landing Page’ Optimizers Client-side changes only Simple experiments
  • Trivial deployment Feature toggle Experimentation platform = Trivial feature experimentation
  • Deploy Experiment Example Lessons
  • 22% less likely to Add to Cart 50% less likely to Checkout Mobile Visitors
  • No change to Checkout Completion
  • Improvement in Checkout Completion
  • Increase in Add to Cart rate
  • Further Increase in Add to Cart rate
  • Multiple lightweight experiments A handful of weeks Verified 30% increase in Purchasing
  • Deploy Experiment Example Lessons
  • Invest in your deployment tools
  • Experiment on the small scale
  • Expect to be wrong
  • Build on the successes
  • Design for incremental change
  • Thank You Paul Coia paul@redbubble.com @pjcoia h p://www.redbubble.com/jobs