Your SlideShare is downloading. ×
Experimenting Your Way to Success: Applying Lean Startup Principles to Product Development at IMVU
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Experimenting Your Way to Success: Applying Lean Startup Principles to Product Development at IMVU

4,513
views

Published on

How IMVU uses experiments to ship products that customers love, and to improve the way we build those products. Key points include company culture and methods of learning, including 5 Whys, project …

How IMVU uses experiments to ship products that customers love, and to improve the way we build those products. Key points include company culture and methods of learning, including 5 Whys, project postmortems, and sprint retrospectives. Discussion is framed within the Lean Startup Build-Measure-Learn loop. Roots of the Lean Startup method in the scientific method are discussed.

Published in: Technology, Business

2 Comments
15 Likes
Statistics
Notes
  • I think this is the missing link. I would want to encorporate in the PhD work I am doing together with Stanford's BJ Fogg. I think you are on to something.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • this is presentation is fascinating, why have i never heard of 5 Whys before?!?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
4,513
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
0
Comments
2
Likes
15
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • I’m James Birchler, Engineering Director at IMVU, and today we’re going to talk about how IMVU uses experiments and Lean Startup methods to build products our customers love, and to innovate and improve our product development process.
  • What are experiments, anyway? Experiments are one way to learn—about the world, or about your business. What makes scientific experimentation different is that you pay attention to a lot of details—like how you set up your experiment to ensure your testing the right things, or whether your results just look good, or if they statistically significant.
  • IMVU uses experiments to help inform product feature development (e.g., what type of user interface works better for managing a large inventory.
  • We also use process experiments to inform how we build those features (e.g., does it work better to have a small team of 4 engineers or a larger team of 12?).
  • Sadly, Copernicus inspired the wrath of the “folks in charge”, and was persecuted by the Roman Catholic Inquisition.
  • Giordano Bruno suffered a worse fate: he was burned at the stake.
  • Galileocrystallized the concept that experimentation is at the very heart of the scientific method.
  • Their findings were good, but their news was bad. What is it like to share bad news in your company?
  • Happily, this whole “science” and “experimentation” thing has caughton, certainly at IMVU.
  • Here is why you should really care, though: experimentation may help you understand your customers better, and help your business succeed.
  • Obviously, at IMVU we feel that experimentation is good.
  • But how does it really work? Let’s start with a quick refresher on The scientific method – from 7th grade science class!This is structured learning at its finest. The scientific method helps you remember the context surrounding your experiment so you can repeat your success—and control the variables that can help you avoid failure.Now imagine that instead of a linear process, the scientific could be applied as a method of continuous learning and improvement.
  • At IMVU, we approach the scientific method as an ongoing process.
  • The Lean Startup Build-Measure-Learn loop is based on the scientific method.
  • You can apply the duck test if you’re not sure… if it looks like a duck, swims like a duck, and sounds like a duck, it probably is a duck!
  • If your company does this already, you likely have a good Culture of Experimentation, too.
  • We like to keep things simple, because running experiments should be fun and easy.
  • Make sure it is simple, even trivial, for your team to run experiments. The easier it is, the more likely you’ll be running lots of experiments and learning more quickly.
  • Make it easy to manage your experiments. If it’s easy to manage which experiments are on and off, and which customers should be participating, your product team will run more experiments and you’ll learn more faster.
  • Viewing and sharing data should also be simple. You might notice that some of these metrics are red. Not all experiments succeed, but happily at IMVU, our culture supports both success and failure…
  • Everyone in the company knows that sharing data and findings is important, whether the results are expected or not, good or “bad”.
  • Another tipwesubscribe to at IMVU…
  • Now lets talk about experimenting with the way we get things done.
  • I’ll frame this within the Lean Startup Build-Measure-Learn loop.
  • Think delivering value to customers in your current sprint is most important?Learning is actually the more important.
  • Think delivering value to customers in your current sprint is most important?Learning is actually the more important.
  • Measure and Learn: 5 Whys Root Cause AnalysisTaiichiOhno, the architect of the Toyota Production System (which includes both management philosophy and production practices) described the 5 whys method as "the basis of Toyota's scientific approach . . . by repeating why five times, the nature of the problem as well as its solution becomes clear." IMVU uses this technique to understand root causes of failures, not just symptoms. Here is how we do it:1. Ask “why” 5 levels deep for each issue2. Identify corrective measures for each cause3. Respond with appropriate level of investment
  • Remember a few thingsabout these approaches: What works now may not work in the future.So be cautious about the kinds of conclusions that you draw from our results.We do the same ourselves, and consciously work to avoid being dogmatic about approaches.
  • Story points: our teams argued for hours about story points, so proceed with caution.
  • Transcript

    • 1. Experimenting
      your way toSUCCESS
      applying lean startup principles to product development at imvu
      James Birchler
      Engineering Director, IMVU
      GDC, San Francisco, March 3, 2011
    • 2. #imvugdc
      @jamesbirchler
    • 3. Experiments
      Product Features
      product development
    • 4. The
      is basedon
      scientific method
    • 5. experimentation
    • 6. Inventory UI
    • 7.
    • 8. Renaissance Astronomer
      Heliocentrism
      Experimental observation
      @copernicus
    • 9. Renaissance Astronomer
      Heliocentrism
      Experimental observation
      #inquisition
    • 10. Renaissance Astronomer
      Heliocentrism
      Experimental observation
      Afraid to publish until on deathbed
      @copernicus
    • 11. Persecuted
      Prosecuted
      Burned at the stake
      “Copernicus FTW!”
      @Giordano_Bruno
    • 12. Persecuted
      Accused of heresy
      House arrest until his death
      “Father of Modern science”
      “Experimentation FTW!”
      Improvement over Bruno.
      @Galileo
    • 13. Question: Why was it so hard for these guys to share their data and findings?
      Answer: It’s complicated, but I think we can all agree that the folks in charge didn’t support hearing “bad news.”
    • 14. Experiments FTW!
      Share results freely!
      Rapid iteration and learning!
      Promoted to Director
      GDC Speaker GIG
      @Jamesbirchler
    • 15. Profitable
      >$40M annualized revenue run rate
      World’s largest virtual goods catalog >6M items
      2M active users in the last 30 days
      Experiments may help your company succeed!
      $
      IMVU Revenue Growth
    • 16. Experimentation
      IS GOOD
    • 17. Got Science?
      Ask Question
      Do Research
      Hypothesis
      Test
      Analyze Data
      Conclusion
      Report Results
    • 18. Rapid iteration = Rapid learning
      Scientific Method
    • 19. “Lean Startup”
    • 20. Quack.
      Scientific Method
      #leanstartup
      An Alleged “duck”
    • 21. culture of
      Experimentation
    • 22. Okay, I’ll run an experiment!
      Okay, I’ll run an experiment!
      Okay, I’ll run an experiment!
      Okay, I’ll run an experiment!
      Okay, I’ll run an experiment!
      Okay, I’ll run an experiment!
      Let’s keep this simple!
      Okay, I’ll run an experiment!
      Okay, I’ll run an experiment!
      Prove your idea is the best!
    • 23. IMVU
      Running experiments is fun!
    • 24. IMVU Feature Experiment “How-To”
      Form hypothesis to test
      Write code, test on dev machine
      Test in production as QA/admin
      Roll out to a % of customers
      Results, conclusion
      Share learning
      Ask Question
      Do Research
      Hypothesis
      Test
      Analyze Data
      Conclusion
      Report Results
      Looks a lot like the scientific method (it is!)…
    • 25. Keep itsimple
    • 26. if( setup_experiment(...) == "control" ) {
        // do it the old way
      } else {
       // do it the new way
      }
      Simple
      To code
    • 27. Simple
      To manage
    • 28. Cool
      Stats
      Porn
      Uh-oh.
      Simple
      To share
    • 29. embracefailure
      Suits
      Company
      Teams
    • 30. The “highest paid person’s opinion” (HiPPO) is not assumed to be correct.
    • 31. “Embrace failure as an opportunity to learn.”
      how you respond to failure says a lot about your culture…
    • 32. Recap:
      Culture is key.
    • 33. Experimentwith process
    • 34. 1
      build
    • 35. 1
      build
      2-3 week sprints
      Adjust process each sprint
      Agile and XP methods
      Aha moment…
    • 36. Agile & XP Methods
      Ask Question
      Embrace change (and try to make it painless)
      Prefer flexibility to perfection
      Iterate, improve continuously
      Feed data and insight back into the process
      Split-test (A/B) experimentation
      Test Driven Development (TDD)
      Use defects to drive improvements
      Rapid, continuous code integration and deployment
      Software cluster immune system: prevent defects
      Pair Programming
      Just-in-time scalability
      Refactoring
      Scientific Method
      Do Research
      Hypothesis
      “Change, flexibility, iteration, continuous improvement...”
      Learning:
      Test
      Agile and XP support experimentation.
      Analyze Data
      Conclusion
      Report Results
    • 37. 2
      Measure
    • 38. 2
      Projects / stories completed
      Time spent on tasks
      Story points delivered
      Unplanned vs. planned work completed
      How productive and happy do we feel?
      Measure
      Value delivered to customers
    • 39. 3
      learn
    • 40. 3
      learn
      Project postmortems
      Sprint retrospectives
      5 Whys root cause anaylsis
      Support open communication…
    • 41. Engineering project managersMatrix management Scrum of ScrumsTeam swapsOpen floor planPlace value on communication
      Oh yeah! Promote & support Learning every day
    • 42. Postmortems & Retrospectives:
      Let’s see it in action!
      Meeting roles
      Metrics
      Action items
    • 43. Appoint a skilled facilitator.
    • 44. Foster communication and engagement.
    • 45. Metrics: Projects, Days, Story points, Misc...
    • 46. Misc.
      Keep
      Stop
      Start
      Action Items
    • 47. 5whys
    • 48. 5whys
      Y1: Why were logins failing?A1: The database has too many connections.
      Y2: Why did the Database have too many connections?A2: A new feature had slow-running queries.
      Y3: Why were there slow queries?A3: They were implemented with a cache time of zero.
      Y4: Why was cache time set to zero?A4: James didn’t know that slow queries must be cached.
      Y5: Why didn’t James know about caching slow queries?A5: He’s new, and we didn’t cover that in our training.
      Root Cause: Fix this!
    • 49. 1. Fix root causes.
      2. Make the size of the fix commensurate with the size of the problem.
      A couple notes about action items:
    • 50. Recap:
      Learning is key.
    • 51. ProcessExperiments
    • 52. Dogma
    • 53. Scrum: A project management methodology, works well with Agile/XP.
    • 54. This GDC Online 2010 talk focused on how IMVU uses Scrum and Agile/XP.
    • 55. IMVU Scrum v1.0…
      1.
      Team org, short sprints, daily standups FTW!
      2.
      Stay flexible, don’t be dogmatic.
    • 56. Story Pointsvs.Ideal Days
      Used to measure progress and value delivered to customers
    • 57. !
    • 58. Story Points at IMVU have an interesting history. Things that worked for us included that they gave team members a reason to engage and discuss projects and tasks, and planning meetings got shorter after an initial period of calibration.
      Some things didn’t work: it was hard to get team buy-in for what story points mean, swapping team members means recalibrating, and the focus on points can be distracting.
      Overall, though, a success.
    • 59. Story Points at IMVU have an interesting history. Things that worked for us included that they gave team members a reason to engage and discuss projects and tasks, and planning meetings got shorter after an initial period of calibration.
      Some things didn’t work: it was hard to get team buy-in for what story points mean, swapping team members means recalibrating, and the focus on points can be distracting.
      Overall, though, a success.
      1.
      Agree on a definition.
      2.
      Shorter planning meetings.
      3.
      Big win for engagement.
    • 60. IdealDays
    • 61. IdealDays
      1.
      Easy to understand.
      2.
      Accurate.
      3.
      Caution: Technical debt.
    • 62. Story Points at IMVU have an interesting history. Things that worked for us included that they gave team members a reason to engage and discuss projects and tasks, and planning meetings got shorter after an initial period of calibration.
      Some things didn’t work: it was hard to get team buy-in for what story points mean, swapping team members means recalibrating, and the focus on points can be distracting.
      Overall, though, a success.
      Both Ideal Days and Story Points foster engagement and discussion, and that’s the real win.
    • 63. 1.
      Focus on work, not time.
      2.
      Short planning meetings.
      3.
      Caution: Reduced ability to predict progress.
      Reduced focus on time spent
      Increased focus on the work
      Short planning meetings
      Engineers love focus on work, and not on task time!
      No ability to predict pace of progress
      Less insight into task details ahead of time
    • 64. Scrum technology…
      Simple + Flexiblebeats Complex + Rigid
    • 65. Sub-teams…
      1.
      4 engineers = sweet spot
      2.
      Don’t share infrastructure
    • 66. Hack Week @IMVU…
      This blog was a Hack Week project.
      100% self-directed.
      100% supported.
      100% awesome.
    • 67. Top Secret: IMVU Garage
      Patio umbrella
      Caffeine
      Bar fridges
      Power
      Flats of water
    • 68. Recap…
      1.
      Experimentation works.
      2.
      Culture is key.
      3.
      Learning is key.
    • 69. We’re Hiring!
      www.imvu.com/jobs
    • 70. credits
      1. flickr/The Chemistry of Inversion/[f]oxymoron
      2. flickr/Mouths/The Wandering Angel
      3. flickr/Culture Tubes/Hey Paul
      4. flickr/Test Tubes/HåkanDahlström
      5. flickr/Tablet packet macro/nick@
      6. flickr/11-08-09/idovermani
      7. flickr/Gauge/Hey Paul
      8. flickr/Cell Culture/kaibara87
      9. IMVU Top Secret Garage/Photo courtesy of Robert Otani
      10. dukenostalgia.com/Mad Scientist Game
      11. flickr/Acceleration of a bouncing ball is constant/dullhunk
      12. commons.wikimedia.org/blue ribbon/Gothika
      13. commons.wikimedia.org/ Ebers7766.jpg/ignacioicke
      14. Vito’s Dancers/Photo courtesy of Ken Patterson
      15. Build-Measure-Learn courtesy of Eric Ries http://www.startuplessonslearned.com/
      16. flickr.com/books/rodrigogalindez
      17. flickr.com/cowboys/anyjazz65
      18. slideshare.net/jamesbirchler Scaling Product Development at a Lean Startup: Agile at IMVU
      19. flickr.com/neanderthal/hamed saber
      20. flickr.com/Clock/JörgWeingrill
      21. flickr.com/Open Tattered Book/playingwithbrushes
      22. Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO, by Ron Kohavi, Randal M. Henne, and Dan Sommerfield, @Microsoft, KDD 2007.
      Thanks
      Jesse Imbach and Josh Adlin provided excellent advice, ideas, laughs, and gourmet food while I was preparing this talk—thanks you guys!
      Matt Danzig made paper float--awesome graphic design skillZ, Matt! Brett Durrett, Chad Austin, Steven Peterson, and Roland Blanton were kind enough to provide helpful comments and feedback.
      Finally, thanks to all of my colleagues and friends at IMVU who make all this possible—you rock!
      James Birchler
      Engineering Director, IMVU
      @jamesbirchler
      james@imvu.com
      #imvugdc