Experimenting Your Way to Success: Applying Lean Startup Principles to Product Development at IMVU

5,907 views

Published on

How IMVU uses experiments to ship products that customers love, and to improve the way we build those products. Key points include company culture and methods of learning, including 5 Whys, project postmortems, and sprint retrospectives. Discussion is framed within the Lean Startup Build-Measure-Learn loop. Roots of the Lean Startup method in the scientific method are discussed.

Published in: Technology, Business
2 Comments
15 Likes
Statistics
Notes
  • I think this is the missing link. I would want to encorporate in the PhD work I am doing together with Stanford's BJ Fogg. I think you are on to something.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • this is presentation is fascinating, why have i never heard of 5 Whys before?!?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
5,907
On SlideShare
0
From Embeds
0
Number of Embeds
1,209
Actions
Shares
0
Downloads
0
Comments
2
Likes
15
Embeds 0
No embeds

No notes for slide
  • I’m James Birchler, Engineering Director at IMVU, and today we’re going to talk about how IMVU uses experiments and Lean Startup methods to build products our customers love, and to innovate and improve our product development process.
  • What are experiments, anyway? Experiments are one way to learn—about the world, or about your business. What makes scientific experimentation different is that you pay attention to a lot of details—like how you set up your experiment to ensure your testing the right things, or whether your results just look good, or if they statistically significant.
  • IMVU uses experiments to help inform product feature development (e.g., what type of user interface works better for managing a large inventory.
  • We also use process experiments to inform how we build those features (e.g., does it work better to have a small team of 4 engineers or a larger team of 12?).
  • Sadly, Copernicus inspired the wrath of the “folks in charge”, and was persecuted by the Roman Catholic Inquisition.
  • Giordano Bruno suffered a worse fate: he was burned at the stake.
  • Galileocrystallized the concept that experimentation is at the very heart of the scientific method.
  • Their findings were good, but their news was bad. What is it like to share bad news in your company?
  • Happily, this whole “science” and “experimentation” thing has caughton, certainly at IMVU.
  • Here is why you should really care, though: experimentation may help you understand your customers better, and help your business succeed.
  • Obviously, at IMVU we feel that experimentation is good.
  • But how does it really work? Let’s start with a quick refresher on The scientific method – from 7th grade science class!This is structured learning at its finest. The scientific method helps you remember the context surrounding your experiment so you can repeat your success—and control the variables that can help you avoid failure.Now imagine that instead of a linear process, the scientific could be applied as a method of continuous learning and improvement.
  • At IMVU, we approach the scientific method as an ongoing process.
  • The Lean Startup Build-Measure-Learn loop is based on the scientific method.
  • You can apply the duck test if you’re not sure… if it looks like a duck, swims like a duck, and sounds like a duck, it probably is a duck!
  • If your company does this already, you likely have a good Culture of Experimentation, too.
  • We like to keep things simple, because running experiments should be fun and easy.
  • Make sure it is simple, even trivial, for your team to run experiments. The easier it is, the more likely you’ll be running lots of experiments and learning more quickly.
  • Make it easy to manage your experiments. If it’s easy to manage which experiments are on and off, and which customers should be participating, your product team will run more experiments and you’ll learn more faster.
  • Viewing and sharing data should also be simple. You might notice that some of these metrics are red. Not all experiments succeed, but happily at IMVU, our culture supports both success and failure…
  • Everyone in the company knows that sharing data and findings is important, whether the results are expected or not, good or “bad”.
  • Another tipwesubscribe to at IMVU…
  • Now lets talk about experimenting with the way we get things done.
  • I’ll frame this within the Lean Startup Build-Measure-Learn loop.
  • Think delivering value to customers in your current sprint is most important?Learning is actually the more important.
  • Think delivering value to customers in your current sprint is most important?Learning is actually the more important.
  • Measure and Learn: 5 Whys Root Cause AnalysisTaiichiOhno, the architect of the Toyota Production System (which includes both management philosophy and production practices) described the 5 whys method as "the basis of Toyota's scientific approach . . . by repeating why five times, the nature of the problem as well as its solution becomes clear." IMVU uses this technique to understand root causes of failures, not just symptoms. Here is how we do it:1. Ask “why” 5 levels deep for each issue2. Identify corrective measures for each cause3. Respond with appropriate level of investment
  • Remember a few thingsabout these approaches: What works now may not work in the future.So be cautious about the kinds of conclusions that you draw from our results.We do the same ourselves, and consciously work to avoid being dogmatic about approaches.
  • Story points: our teams argued for hours about story points, so proceed with caution.
  • Experimenting Your Way to Success: Applying Lean Startup Principles to Product Development at IMVU

    1. 1. Experimenting <br />your way toSUCCESS<br />applying lean startup principles to product development at imvu<br />James Birchler<br />Engineering Director, IMVU<br />GDC, San Francisco, March 3, 2011<br />
    2. 2. #imvugdc<br />@jamesbirchler<br />
    3. 3. Experiments<br />Product Features<br />product development<br />
    4. 4. The <br />is basedon<br />scientific method<br />
    5. 5. experimentation<br />
    6. 6. Inventory UI<br />
    7. 7.
    8. 8. Renaissance Astronomer<br />Heliocentrism<br />Experimental observation<br />@copernicus<br />
    9. 9. Renaissance Astronomer<br />Heliocentrism<br />Experimental observation<br />#inquisition<br />
    10. 10. Renaissance Astronomer<br />Heliocentrism<br />Experimental observation<br />Afraid to publish until on deathbed<br />@copernicus<br />
    11. 11. Persecuted<br />Prosecuted<br />Burned at the stake<br />“Copernicus FTW!”<br />@Giordano_Bruno<br />
    12. 12. Persecuted<br />Accused of heresy<br />House arrest until his death<br />“Father of Modern science”<br />“Experimentation FTW!”<br />Improvement over Bruno.<br />@Galileo<br />
    13. 13. Question: Why was it so hard for these guys to share their data and findings?<br />Answer: It’s complicated, but I think we can all agree that the folks in charge didn’t support hearing “bad news.”<br />
    14. 14. Experiments FTW!<br />Share results freely!<br />Rapid iteration and learning!<br />Promoted to Director<br />GDC Speaker GIG<br />@Jamesbirchler<br />
    15. 15. Profitable<br />>$40M annualized revenue run rate<br />World’s largest virtual goods catalog >6M items<br />2M active users in the last 30 days<br />Experiments may help your company succeed!<br />$<br />IMVU Revenue Growth<br />
    16. 16. Experimentation<br />IS GOOD<br />
    17. 17. Got Science?<br />Ask Question<br />Do Research<br />Hypothesis<br />Test<br />Analyze Data<br />Conclusion<br />Report Results<br />
    18. 18. Rapid iteration = Rapid learning<br />Scientific Method<br />
    19. 19. “Lean Startup”<br />
    20. 20. Quack.<br />Scientific Method<br />#leanstartup<br />An Alleged “duck”<br />
    21. 21. culture of<br />Experimentation<br />
    22. 22. Okay, I’ll run an experiment!<br />Okay, I’ll run an experiment!<br />Okay, I’ll run an experiment!<br />Okay, I’ll run an experiment!<br />Okay, I’ll run an experiment!<br />Okay, I’ll run an experiment!<br />Let’s keep this simple!<br />Okay, I’ll run an experiment!<br />Okay, I’ll run an experiment!<br />Prove your idea is the best!<br />
    23. 23. IMVU<br />Running experiments is fun!<br />
    24. 24. IMVU Feature Experiment “How-To”<br />Form hypothesis to test<br />Write code, test on dev machine<br />Test in production as QA/admin<br />Roll out to a % of customers<br />Results, conclusion<br />Share learning<br />Ask Question<br />Do Research<br />Hypothesis<br />Test<br />Analyze Data<br />Conclusion<br />Report Results<br />Looks a lot like the scientific method (it is!)…<br />
    25. 25. Keep itsimple<br />
    26. 26. if( setup_experiment(...) == "control" ) {<br />  // do it the old way<br />} else {<br /> // do it the new way<br />}<br />Simple<br />To code<br />
    27. 27. Simple<br />To manage<br />
    28. 28. Cool<br />Stats<br />Porn<br />Uh-oh.<br />Simple<br />To share<br />
    29. 29. embracefailure<br />Suits<br />Company<br />Teams<br />
    30. 30. The “highest paid person’s opinion” (HiPPO) is not assumed to be correct.<br />
    31. 31. “Embrace failure as an opportunity to learn.”<br />how you respond to failure says a lot about your culture…<br />
    32. 32. Recap: <br />Culture is key.<br />
    33. 33. Experimentwith process<br />
    34. 34. 1<br />build<br />
    35. 35. 1<br />build<br />2-3 week sprints<br />Adjust process each sprint<br />Agile and XP methods<br />Aha moment…<br />
    36. 36. Agile & XP Methods<br />Ask Question<br />Embrace change (and try to make it painless)<br />Prefer flexibility to perfection<br />Iterate, improve continuously<br />Feed data and insight back into the process<br />Split-test (A/B) experimentation<br />Test Driven Development (TDD)<br />Use defects to drive improvements<br />Rapid, continuous code integration and deployment<br />Software cluster immune system: prevent defects<br />Pair Programming<br />Just-in-time scalability<br />Refactoring<br />Scientific Method<br />Do Research<br />Hypothesis<br />“Change, flexibility, iteration, continuous improvement...”<br />Learning: <br />Test<br />Agile and XP support experimentation.<br />Analyze Data<br />Conclusion<br />Report Results<br />
    37. 37. 2<br />Measure<br />
    38. 38. 2<br />Projects / stories completed<br />Time spent on tasks<br />Story points delivered<br />Unplanned vs. planned work completed<br />How productive and happy do we feel?<br />Measure<br />Value delivered to customers<br />
    39. 39. 3<br />learn<br />
    40. 40. 3<br />learn<br />Project postmortems<br />Sprint retrospectives<br />5 Whys root cause anaylsis<br />Support open communication…<br />
    41. 41. Engineering project managersMatrix management Scrum of ScrumsTeam swapsOpen floor planPlace value on communication<br />Oh yeah! Promote & support Learning every day<br />
    42. 42. Postmortems & Retrospectives: <br />Let’s see it in action!<br />Meeting roles<br />Metrics<br />Action items<br />
    43. 43. Appoint a skilled facilitator.<br />
    44. 44. Foster communication and engagement.<br />
    45. 45. Metrics: Projects, Days, Story points, Misc...<br />
    46. 46. Misc.<br />Keep<br />Stop<br />Start<br />Action Items<br />
    47. 47. 5whys<br />
    48. 48. 5whys<br />Y1: Why were logins failing?A1: The database has too many connections.<br />Y2: Why did the Database have too many connections?A2: A new feature had slow-running queries.<br />Y3: Why were there slow queries?A3: They were implemented with a cache time of zero.<br />Y4: Why was cache time set to zero?A4: James didn’t know that slow queries must be cached.<br />Y5: Why didn’t James know about caching slow queries?A5: He’s new, and we didn’t cover that in our training.<br />Root Cause: Fix this!<br />
    49. 49. 1. Fix root causes.<br />2. Make the size of the fix commensurate with the size of the problem.<br />A couple notes about action items:<br />
    50. 50. Recap: <br />Learning is key.<br />
    51. 51. ProcessExperiments<br />
    52. 52. Dogma<br />
    53. 53. Scrum: A project management methodology, works well with Agile/XP.<br />
    54. 54. This GDC Online 2010 talk focused on how IMVU uses Scrum and Agile/XP.<br />
    55. 55. IMVU Scrum v1.0…<br />1.<br />Team org, short sprints, daily standups FTW!<br />2.<br />Stay flexible, don’t be dogmatic.<br />
    56. 56. Story Pointsvs.Ideal Days<br />Used to measure progress and value delivered to customers<br />
    57. 57. !<br />
    58. 58. Story Points at IMVU have an interesting history. Things that worked for us included that they gave team members a reason to engage and discuss projects and tasks, and planning meetings got shorter after an initial period of calibration.<br /> Some things didn’t work: it was hard to get team buy-in for what story points mean, swapping team members means recalibrating, and the focus on points can be distracting.<br /> Overall, though, a success. <br />
    59. 59. Story Points at IMVU have an interesting history. Things that worked for us included that they gave team members a reason to engage and discuss projects and tasks, and planning meetings got shorter after an initial period of calibration.<br /> Some things didn’t work: it was hard to get team buy-in for what story points mean, swapping team members means recalibrating, and the focus on points can be distracting.<br /> Overall, though, a success. <br />1.<br />Agree on a definition.<br />2.<br />Shorter planning meetings.<br />3.<br />Big win for engagement.<br />
    60. 60. IdealDays<br />
    61. 61. IdealDays<br />1.<br />Easy to understand.<br />2.<br />Accurate.<br />3.<br />Caution: Technical debt.<br />
    62. 62. Story Points at IMVU have an interesting history. Things that worked for us included that they gave team members a reason to engage and discuss projects and tasks, and planning meetings got shorter after an initial period of calibration.<br /> Some things didn’t work: it was hard to get team buy-in for what story points mean, swapping team members means recalibrating, and the focus on points can be distracting.<br /> Overall, though, a success. <br />Both Ideal Days and Story Points foster engagement and discussion, and that’s the real win.<br />
    63. 63. 1.<br />Focus on work, not time.<br />2.<br />Short planning meetings.<br />3.<br />Caution: Reduced ability to predict progress.<br />Reduced focus on time spent<br />Increased focus on the work<br />Short planning meetings<br />Engineers love focus on work, and not on task time!<br />No ability to predict pace of progress<br />Less insight into task details ahead of time<br />
    64. 64. Scrum technology…<br />Simple + Flexiblebeats Complex + Rigid<br />
    65. 65. Sub-teams…<br />1.<br />4 engineers = sweet spot<br />2.<br />Don’t share infrastructure<br />
    66. 66. Hack Week @IMVU…<br />This blog was a Hack Week project. <br />100% self-directed.<br />100% supported.<br />100% awesome.<br />
    67. 67. Top Secret: IMVU Garage<br />Patio umbrella<br />Caffeine<br />Bar fridges<br />Power<br />Flats of water<br />
    68. 68. Recap…<br />1.<br />Experimentation works.<br />2.<br />Culture is key.<br />3.<br />Learning is key.<br />
    69. 69. We’re Hiring!<br />www.imvu.com/jobs<br />
    70. 70. credits<br />1. flickr/The Chemistry of Inversion/[f]oxymoron<br />2. flickr/Mouths/The Wandering Angel<br />3. flickr/Culture Tubes/Hey Paul<br />4. flickr/Test Tubes/HåkanDahlström<br />5. flickr/Tablet packet macro/nick@<br />6. flickr/11-08-09/idovermani<br />7. flickr/Gauge/Hey Paul<br />8. flickr/Cell Culture/kaibara87<br />9. IMVU Top Secret Garage/Photo courtesy of Robert Otani<br />10. dukenostalgia.com/Mad Scientist Game<br />11. flickr/Acceleration of a bouncing ball is constant/dullhunk<br />12. commons.wikimedia.org/blue ribbon/Gothika<br />13. commons.wikimedia.org/ Ebers7766.jpg/ignacioicke<br />14. Vito’s Dancers/Photo courtesy of Ken Patterson<br />15. Build-Measure-Learn courtesy of Eric Ries http://www.startuplessonslearned.com/<br />16. flickr.com/books/rodrigogalindez<br />17. flickr.com/cowboys/anyjazz65<br />18. slideshare.net/jamesbirchler Scaling Product Development at a Lean Startup: Agile at IMVU<br />19. flickr.com/neanderthal/hamed saber<br />20. flickr.com/Clock/JörgWeingrill<br />21. flickr.com/Open Tattered Book/playingwithbrushes<br />22. Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO, by Ron Kohavi, Randal M. Henne, and Dan Sommerfield, @Microsoft, KDD 2007.<br />Thanks<br />Jesse Imbach and Josh Adlin provided excellent advice, ideas, laughs, and gourmet food while I was preparing this talk—thanks you guys! <br />Matt Danzig made paper float--awesome graphic design skillZ, Matt! Brett Durrett, Chad Austin, Steven Peterson, and Roland Blanton were kind enough to provide helpful comments and feedback. <br />Finally, thanks to all of my colleagues and friends at IMVU who make all this possible—you rock! <br />James Birchler<br />Engineering Director, IMVU<br />@jamesbirchler<br />james@imvu.com<br />#imvugdc<br />

    ×