• Save
Experimenting Your Way to Success: Applying Lean Startup Principles to Product Development at IMVU
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Experimenting Your Way to Success: Applying Lean Startup Principles to Product Development at IMVU

  • 4,889 views
Uploaded on

How IMVU uses experiments to ship products that customers love, and to improve the way we build those products. Key points include company culture and methods of learning, including 5 Whys, project......

How IMVU uses experiments to ship products that customers love, and to improve the way we build those products. Key points include company culture and methods of learning, including 5 Whys, project postmortems, and sprint retrospectives. Discussion is framed within the Lean Startup Build-Measure-Learn loop. Roots of the Lean Startup method in the scientific method are discussed.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • I think this is the missing link. I would want to encorporate in the PhD work I am doing together with Stanford's BJ Fogg. I think you are on to something.
    Are you sure you want to
    Your message goes here
  • this is presentation is fascinating, why have i never heard of 5 Whys before?!?
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
4,889
On Slideshare
4,421
From Embeds
468
Number of Embeds
21

Actions

Shares
Downloads
0
Comments
2
Likes
14

Embeds 468

http://productdevelopmentinpractice.blogspot.com 340
http://www.usmansheikh.com 55
http://www.parisoma.com 14
http://blog.fabernovel.com 13
http://paper.li 12
http://productdevelopmentinpractice.blogspot.fr 6
https://www.linkedin.com 6
http://productdevelopmentinpractice.blogspot.in 4
http://www.linkedin.com 3
http://productdevelopmentinpractice.blogspot.sg 3
http://productdevelopmentinpractice.blogspot.com.es 2
http://bakedweb.net 1
http://productdevelopmentinpractice.blogspot.pt 1
http://feedproxy.google.com 1
http://productdevelopmentinpractice.blogspot.it 1
http://productdevelopmentinpractice.blogspot.ca 1
http://productdevelopmentinpractice.blogspot.de 1
http://productdevelopmentinpractice.blogspot.ie 1
http://www.netvibes.com 1
http://translate.googleusercontent.com 1
http://static.slidesharecdn.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • I’m James Birchler, Engineering Director at IMVU, and today we’re going to talk about how IMVU uses experiments and Lean Startup methods to build products our customers love, and to innovate and improve our product development process.
  • What are experiments, anyway? Experiments are one way to learn—about the world, or about your business. What makes scientific experimentation different is that you pay attention to a lot of details—like how you set up your experiment to ensure your testing the right things, or whether your results just look good, or if they statistically significant.
  • IMVU uses experiments to help inform product feature development (e.g., what type of user interface works better for managing a large inventory.
  • We also use process experiments to inform how we build those features (e.g., does it work better to have a small team of 4 engineers or a larger team of 12?).
  • Sadly, Copernicus inspired the wrath of the “folks in charge”, and was persecuted by the Roman Catholic Inquisition.
  • Giordano Bruno suffered a worse fate: he was burned at the stake.
  • Galileocrystallized the concept that experimentation is at the very heart of the scientific method.
  • Their findings were good, but their news was bad. What is it like to share bad news in your company?
  • Happily, this whole “science” and “experimentation” thing has caughton, certainly at IMVU.
  • Here is why you should really care, though: experimentation may help you understand your customers better, and help your business succeed.
  • Obviously, at IMVU we feel that experimentation is good.
  • But how does it really work? Let’s start with a quick refresher on The scientific method – from 7th grade science class!This is structured learning at its finest. The scientific method helps you remember the context surrounding your experiment so you can repeat your success—and control the variables that can help you avoid failure.Now imagine that instead of a linear process, the scientific could be applied as a method of continuous learning and improvement.
  • At IMVU, we approach the scientific method as an ongoing process.
  • The Lean Startup Build-Measure-Learn loop is based on the scientific method.
  • You can apply the duck test if you’re not sure… if it looks like a duck, swims like a duck, and sounds like a duck, it probably is a duck!
  • If your company does this already, you likely have a good Culture of Experimentation, too.
  • We like to keep things simple, because running experiments should be fun and easy.
  • Make sure it is simple, even trivial, for your team to run experiments. The easier it is, the more likely you’ll be running lots of experiments and learning more quickly.
  • Make it easy to manage your experiments. If it’s easy to manage which experiments are on and off, and which customers should be participating, your product team will run more experiments and you’ll learn more faster.
  • Viewing and sharing data should also be simple. You might notice that some of these metrics are red. Not all experiments succeed, but happily at IMVU, our culture supports both success and failure…
  • Everyone in the company knows that sharing data and findings is important, whether the results are expected or not, good or “bad”.
  • Another tipwesubscribe to at IMVU…
  • Now lets talk about experimenting with the way we get things done.
  • I’ll frame this within the Lean Startup Build-Measure-Learn loop.
  • Think delivering value to customers in your current sprint is most important?Learning is actually the more important.
  • Think delivering value to customers in your current sprint is most important?Learning is actually the more important.
  • Measure and Learn: 5 Whys Root Cause AnalysisTaiichiOhno, the architect of the Toyota Production System (which includes both management philosophy and production practices) described the 5 whys method as "the basis of Toyota's scientific approach . . . by repeating why five times, the nature of the problem as well as its solution becomes clear." IMVU uses this technique to understand root causes of failures, not just symptoms. Here is how we do it:1. Ask “why” 5 levels deep for each issue2. Identify corrective measures for each cause3. Respond with appropriate level of investment
  • Remember a few thingsabout these approaches: What works now may not work in the future.So be cautious about the kinds of conclusions that you draw from our results.We do the same ourselves, and consciously work to avoid being dogmatic about approaches.
  • Story points: our teams argued for hours about story points, so proceed with caution.

Transcript

  • 1. Experimenting
    your way toSUCCESS
    applying lean startup principles to product development at imvu
    James Birchler
    Engineering Director, IMVU
    GDC, San Francisco, March 3, 2011
  • 2. #imvugdc
    @jamesbirchler
  • 3. Experiments
    Product Features
    product development
  • 4. The
    is basedon
    scientific method
  • 5. experimentation
  • 6. Inventory UI
  • 7.
  • 8. Renaissance Astronomer
    Heliocentrism
    Experimental observation
    @copernicus
  • 9. Renaissance Astronomer
    Heliocentrism
    Experimental observation
    #inquisition
  • 10. Renaissance Astronomer
    Heliocentrism
    Experimental observation
    Afraid to publish until on deathbed
    @copernicus
  • 11. Persecuted
    Prosecuted
    Burned at the stake
    “Copernicus FTW!”
    @Giordano_Bruno
  • 12. Persecuted
    Accused of heresy
    House arrest until his death
    “Father of Modern science”
    “Experimentation FTW!”
    Improvement over Bruno.
    @Galileo
  • 13. Question: Why was it so hard for these guys to share their data and findings?
    Answer: It’s complicated, but I think we can all agree that the folks in charge didn’t support hearing “bad news.”
  • 14. Experiments FTW!
    Share results freely!
    Rapid iteration and learning!
    Promoted to Director
    GDC Speaker GIG
    @Jamesbirchler
  • 15. Profitable
    >$40M annualized revenue run rate
    World’s largest virtual goods catalog >6M items
    2M active users in the last 30 days
    Experiments may help your company succeed!
    $
    IMVU Revenue Growth
  • 16. Experimentation
    IS GOOD
  • 17. Got Science?
    Ask Question
    Do Research
    Hypothesis
    Test
    Analyze Data
    Conclusion
    Report Results
  • 18. Rapid iteration = Rapid learning
    Scientific Method
  • 19. “Lean Startup”
  • 20. Quack.
    Scientific Method
    #leanstartup
    An Alleged “duck”
  • 21. culture of
    Experimentation
  • 22. Okay, I’ll run an experiment!
    Okay, I’ll run an experiment!
    Okay, I’ll run an experiment!
    Okay, I’ll run an experiment!
    Okay, I’ll run an experiment!
    Okay, I’ll run an experiment!
    Let’s keep this simple!
    Okay, I’ll run an experiment!
    Okay, I’ll run an experiment!
    Prove your idea is the best!
  • 23. IMVU
    Running experiments is fun!
  • 24. IMVU Feature Experiment “How-To”
    Form hypothesis to test
    Write code, test on dev machine
    Test in production as QA/admin
    Roll out to a % of customers
    Results, conclusion
    Share learning
    Ask Question
    Do Research
    Hypothesis
    Test
    Analyze Data
    Conclusion
    Report Results
    Looks a lot like the scientific method (it is!)…
  • 25. Keep itsimple
  • 26. if( setup_experiment(...) == "control" ) {
      // do it the old way
    } else {
     // do it the new way
    }
    Simple
    To code
  • 27. Simple
    To manage
  • 28. Cool
    Stats
    Porn
    Uh-oh.
    Simple
    To share
  • 29. embracefailure
    Suits
    Company
    Teams
  • 30. The “highest paid person’s opinion” (HiPPO) is not assumed to be correct.
  • 31. “Embrace failure as an opportunity to learn.”
    how you respond to failure says a lot about your culture…
  • 32. Recap:
    Culture is key.
  • 33. Experimentwith process
  • 34. 1
    build
  • 35. 1
    build
    2-3 week sprints
    Adjust process each sprint
    Agile and XP methods
    Aha moment…
  • 36. Agile & XP Methods
    Ask Question
    Embrace change (and try to make it painless)
    Prefer flexibility to perfection
    Iterate, improve continuously
    Feed data and insight back into the process
    Split-test (A/B) experimentation
    Test Driven Development (TDD)
    Use defects to drive improvements
    Rapid, continuous code integration and deployment
    Software cluster immune system: prevent defects
    Pair Programming
    Just-in-time scalability
    Refactoring
    Scientific Method
    Do Research
    Hypothesis
    “Change, flexibility, iteration, continuous improvement...”
    Learning:
    Test
    Agile and XP support experimentation.
    Analyze Data
    Conclusion
    Report Results
  • 37. 2
    Measure
  • 38. 2
    Projects / stories completed
    Time spent on tasks
    Story points delivered
    Unplanned vs. planned work completed
    How productive and happy do we feel?
    Measure
    Value delivered to customers
  • 39. 3
    learn
  • 40. 3
    learn
    Project postmortems
    Sprint retrospectives
    5 Whys root cause anaylsis
    Support open communication…
  • 41. Engineering project managersMatrix management Scrum of ScrumsTeam swapsOpen floor planPlace value on communication
    Oh yeah! Promote & support Learning every day
  • 42. Postmortems & Retrospectives:
    Let’s see it in action!
    Meeting roles
    Metrics
    Action items
  • 43. Appoint a skilled facilitator.
  • 44. Foster communication and engagement.
  • 45. Metrics: Projects, Days, Story points, Misc...
  • 46. Misc.
    Keep
    Stop
    Start
    Action Items
  • 47. 5whys
  • 48. 5whys
    Y1: Why were logins failing?A1: The database has too many connections.
    Y2: Why did the Database have too many connections?A2: A new feature had slow-running queries.
    Y3: Why were there slow queries?A3: They were implemented with a cache time of zero.
    Y4: Why was cache time set to zero?A4: James didn’t know that slow queries must be cached.
    Y5: Why didn’t James know about caching slow queries?A5: He’s new, and we didn’t cover that in our training.
    Root Cause: Fix this!
  • 49. 1. Fix root causes.
    2. Make the size of the fix commensurate with the size of the problem.
    A couple notes about action items:
  • 50. Recap:
    Learning is key.
  • 51. ProcessExperiments
  • 52. Dogma
  • 53. Scrum: A project management methodology, works well with Agile/XP.
  • 54. This GDC Online 2010 talk focused on how IMVU uses Scrum and Agile/XP.
  • 55. IMVU Scrum v1.0…
    1.
    Team org, short sprints, daily standups FTW!
    2.
    Stay flexible, don’t be dogmatic.
  • 56. Story Pointsvs.Ideal Days
    Used to measure progress and value delivered to customers
  • 57. !
  • 58. Story Points at IMVU have an interesting history. Things that worked for us included that they gave team members a reason to engage and discuss projects and tasks, and planning meetings got shorter after an initial period of calibration.
    Some things didn’t work: it was hard to get team buy-in for what story points mean, swapping team members means recalibrating, and the focus on points can be distracting.
    Overall, though, a success.
  • 59. Story Points at IMVU have an interesting history. Things that worked for us included that they gave team members a reason to engage and discuss projects and tasks, and planning meetings got shorter after an initial period of calibration.
    Some things didn’t work: it was hard to get team buy-in for what story points mean, swapping team members means recalibrating, and the focus on points can be distracting.
    Overall, though, a success.
    1.
    Agree on a definition.
    2.
    Shorter planning meetings.
    3.
    Big win for engagement.
  • 60. IdealDays
  • 61. IdealDays
    1.
    Easy to understand.
    2.
    Accurate.
    3.
    Caution: Technical debt.
  • 62. Story Points at IMVU have an interesting history. Things that worked for us included that they gave team members a reason to engage and discuss projects and tasks, and planning meetings got shorter after an initial period of calibration.
    Some things didn’t work: it was hard to get team buy-in for what story points mean, swapping team members means recalibrating, and the focus on points can be distracting.
    Overall, though, a success.
    Both Ideal Days and Story Points foster engagement and discussion, and that’s the real win.
  • 63. 1.
    Focus on work, not time.
    2.
    Short planning meetings.
    3.
    Caution: Reduced ability to predict progress.
    Reduced focus on time spent
    Increased focus on the work
    Short planning meetings
    Engineers love focus on work, and not on task time!
    No ability to predict pace of progress
    Less insight into task details ahead of time
  • 64. Scrum technology…
    Simple + Flexiblebeats Complex + Rigid
  • 65. Sub-teams…
    1.
    4 engineers = sweet spot
    2.
    Don’t share infrastructure
  • 66. Hack Week @IMVU…
    This blog was a Hack Week project.
    100% self-directed.
    100% supported.
    100% awesome.
  • 67. Top Secret: IMVU Garage
    Patio umbrella
    Caffeine
    Bar fridges
    Power
    Flats of water
  • 68. Recap…
    1.
    Experimentation works.
    2.
    Culture is key.
    3.
    Learning is key.
  • 69. We’re Hiring!
    www.imvu.com/jobs
  • 70. credits
    1. flickr/The Chemistry of Inversion/[f]oxymoron
    2. flickr/Mouths/The Wandering Angel
    3. flickr/Culture Tubes/Hey Paul
    4. flickr/Test Tubes/HåkanDahlström
    5. flickr/Tablet packet macro/nick@
    6. flickr/11-08-09/idovermani
    7. flickr/Gauge/Hey Paul
    8. flickr/Cell Culture/kaibara87
    9. IMVU Top Secret Garage/Photo courtesy of Robert Otani
    10. dukenostalgia.com/Mad Scientist Game
    11. flickr/Acceleration of a bouncing ball is constant/dullhunk
    12. commons.wikimedia.org/blue ribbon/Gothika
    13. commons.wikimedia.org/ Ebers7766.jpg/ignacioicke
    14. Vito’s Dancers/Photo courtesy of Ken Patterson
    15. Build-Measure-Learn courtesy of Eric Ries http://www.startuplessonslearned.com/
    16. flickr.com/books/rodrigogalindez
    17. flickr.com/cowboys/anyjazz65
    18. slideshare.net/jamesbirchler Scaling Product Development at a Lean Startup: Agile at IMVU
    19. flickr.com/neanderthal/hamed saber
    20. flickr.com/Clock/JörgWeingrill
    21. flickr.com/Open Tattered Book/playingwithbrushes
    22. Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO, by Ron Kohavi, Randal M. Henne, and Dan Sommerfield, @Microsoft, KDD 2007.
    Thanks
    Jesse Imbach and Josh Adlin provided excellent advice, ideas, laughs, and gourmet food while I was preparing this talk—thanks you guys!
    Matt Danzig made paper float--awesome graphic design skillZ, Matt! Brett Durrett, Chad Austin, Steven Peterson, and Roland Blanton were kind enough to provide helpful comments and feedback.
    Finally, thanks to all of my colleagues and friends at IMVU who make all this possible—you rock!
    James Birchler
    Engineering Director, IMVU
    @jamesbirchler
    james@imvu.com
    #imvugdc