Continuous Innovation

1. Con8nuous Innova8on In a Rapidly Evolving Market Experimen*ng and being agile with products, processes, … Chris Dolezalek – October, 2012 Chris Dolezalek – October, 2012

2. The path to enlightenment lies not in the footsteps of another What is the one mantra to follow religiously? Follow no mantra religiously 初心 Beginner’s Mind Chris Dolezalek – October, 2012

3. The Pivot That Elusive Hockey S*ck BroadVision, Inc. pivots to become fastest growing soDware company on Nasdaq SoDlab, GmbH pivots to become Germany’s 2nd fastest growing independent soDware company Google $0 to $37.9B in 10 years ”Innova(ng at a breakneck pace; rolling out a new service capability prac(cally every month1.” PowerSoD developed PowerBuilder in 1991, went public in 1993 and was acquired for ~$900 million in 1995 1. The Ten Faces of Innova*on by Thomas Kelley & Jonathan LiRman hRps://itunes.apple.com/us/book/the-‐ten-‐faces-‐of-‐innova*on/id421032598?mt=11 hRp://www.sunarsports.com/products/Easton-‐Stealth-‐S19-‐Sr.-‐Composite-‐Hockey-‐S*ck Chris Dolezalek – October, 2012

4. Lean Staffing Hiring Your Most Valuable Resources Hire for Tomorrow not Today • The top 10 in-‐demand jobs in 2010 did not exist in 2004 “Did You Know? Shift Happens (August 2011)“ http://www.youtube.com/watch?v=EJpSqeQbh4o Skills vs Curiosity & Passion • "CQ + PQ > IQ” Curiosity & Passion Quo:ents are more important than Intelligence Quo:ent Thomas Friedman, The World is Flat APtude MaQers • “It is not their ap:tude but their aFtude that will determine their al:tude.” Jesse Jackson, Washington Post 21 May 1978 Chris Dolezalek – October, 2012

5. Don’t stop at Hiring Invest in spinning up At IMVU, we dedicate a mentor to every new engineer that will sit next to them and spend 6 weeks spinning them up according to a boot camp guide -‐ including a day-‐one push of a change to produc*on. Spinning Marathon voor WAR CHILD -‐ Some rights reserved By isafmedia (cropped) Chris Dolezalek – October, 2012

6. Manage Your Most Valuable Resources What makes for the most produc*ve & innova*ve employee? Happiness = Passion, Engagement • Support your team Perspec8ve: Managers work for the engineers to enable them to be produc8ve • Sell vs Tell = No show, No go Winning the hearts and minds bring engagement as opposed to bodies in seats. 5-‐Whys: Company Direc8ves Alignment = What are you working on and Why is it important? • Be Transparent & Invite Challenges Share business objec8ves and reward those who ques8ons them • Let them pick their team & project LiRle startups/laboratories • Let them Hack Hack weeks once a quarter • Let them experiment • Let them choose their tools • Let them determine when done is Story Point Sizing • Keep teams small, agile and fun (BVSN 5 man teams vs whole companies) Chris Dolezalek – October, 2012

7. • Run Experiments • Minimal Viable Products • Con*nuous Deployment • Smart Test Coverage • Use an Immune System • Post Mortems, & Retrospec*ves & 5 Whys Chris Dolezalek – October, 2012

8. Number of uses of new feature Con8nuous Innova8on Validate Hypotheses with Data Time: Experiment A: up ~6% over 2 weeks prior Chris Dolezalek – October, 2012

9. Number of uses of new feature Con8nuous Innova8on Validate Hypotheses with the Right Data Time: Subgroup A1: up ~48% over 2 weeks prior Subgroup A2: down ~68% over 2 weeks prior Chris Dolezalek – October, 2012

10. Experiment as individuals but also as teams 5 Teams 5 Diﬀerent Processes IMVU con*nually runs process experiments as a way to improve our processes and to avoid falling into dysfunc*onal rou*nes. Chris Dolezalek – October, 2012

11. Minimal Viable Products • Diﬀerent Kinds of MVPs hRp://www.rise.net/blog/minimum-‐viable-‐product – A heavily used system (IMVU’s Landing Pages Millions of monthly hits) – New Feature Sets (inventory management into exis*ng product) – A brand new subsystem (a new game with a new game/fun loop) – A paradigm shiD (an iPhone App) Chris Dolezalek – October, 2012

12. Deploy, Iterate & Learn at New Speeds Production releases How ofen do you deploy? • Waterfall Development : every 6-‐12 months • Agile Methodology: every 1-‐3 weeks • Con8nuous Deployment: every day • IMVU every 7 Minutes Chris Dolezalek – October, 2012

13. Test-‐Driven Development Where/How to start? Test Driven Development hRp://www.ﬂickr.com/photos/brunobord/3987593006/lightbox Some rights reserved by brunobord Chris Dolezalek – October, 2012

14. Immune Systems Small Change-‐Sets = Easy Debugging Chris Dolezalek – October, 2012

15. 5 Whys Turning Losses into Wins 1. Why did the server grind to a halt? DB query took forever. 2. Why did the query take so long? Un-‐indexed query on large table. 3. Why didn’t the engineer realize the query would be slow? Local tests ran fine against a small test db. 4. Why didn’t the engineer know to run a “try” against produc:on? It was his first database change as a new hire. 5. Why did his mentor not tell him? The mentor had leD for the day. 6. Why did the new hire not know to only check-‐in with mentor? This was the first mentee for that mentor and he hadn’t told him. Follow-‐Up 1) Update Spin-‐Up Doc for Mentor and Mentee to start with a clear statement that mentee should not check into produc*on without mentor or without having reached that stage. Follow-‐Up 2) Add notes to DB Query sec*on of spin-‐up doc on use of “try” Post-‐Mortems = Teachable Moments ; Fix what needs fixing Ques*on mark sign photo by Colin_K on Flickr Chris Dolezalek – October, 2012

16. Requirement Opacity Granularity versus deadlines: It’s ok not to have the nth degree of detail of what you’re building in 6 months’ 8me. Chris Dolezalek – October, 2012

17. Con8nuous Deployment Flow Hypothesis Build on VMs hRp://www.slideshare.net/bgdurreR/con*nuous-‐deployment-‐at-‐lean-‐la Chris Dolezalek – October, 2012

18. Experiments & Alterna8ves • Experiments: Vo*ng by doing, requires func*onality • User Sessions: Hard to get good sample size quickly • Customer Advocates: Par*cipates in forums, user groups, … • Anthropologist: Do ﬁeld research to observe interac*ons • Surveys: What people say rather than do • Dog-‐fooding: Use your product • Mavens: Good insights into established market hRp://www.gladwell.com/*ppingpoint/index.html Chris Dolezalek – October, 2012

19. An Enhancement / Story is Born • Project Brief Goes to a Exec Checkpoint – User stories – Success criteria by date x… (engagement, revenue, …) • Preview Mee*ng by Product Owner – To: Tech Lead, QA Lead, Data Analyst • Tech Design, Test Plan, Data Design • Planning Mee*ng – Story point sizing – High and low es*mate jus*ﬁca*on – Risk buckets for stories vs risk included in task es*mates Chris Dolezalek – October, 2012

20. Delivering Innova*on to Risk Averse, Enterprise Customers • In 1985, I came to Edinburgh from Munich to help sell the Bank of Scotland on transi*oning to our new product that resulted from our second pivot at SoDlab. • What unfolded was not quite according to script. • They were not at all excited about adop*ng innova*on, having in 8 previous systems suppor*ng 180 of their IT staﬀ. But, they did… Chris Dolezalek – October, 2012

21. Apply Beginner’s Mind and Con8nuously Innovate… Your Product Direc*on, Your Company Direc*on, 初心 Beginner’s Mind Your Development Processes, Your Resource Management, Your Marke*ng Strategy, Your Technology Choices, Your Analy*cs Approaches, Understanding Your Customers & Prospects, .. Everything You Do and Every Approach You Take. Chris Dolezalek Chris Dolezalek – October, 2012

22. Thank You! Chris Dolezalek Chris Dolezalek – October, 2012 22

23. References & Other Links • Eric Ries, IMVU, former CTO of Engineering -‐ The Lean Startup How Today's Entrepreneurs Use Con*nuous Innova*on to Create Radically Successful Businesses hRp://www.amazon.com/Lean-‐Startup-‐Entrepreneurs-‐Con*nuous-‐Innova*on/dp/0307887898/ref=sr_1_1?ie=UTF8&qid=1318352510&sr=8-‐1 hRp://itunes.apple.com/us/book/the-‐lean-‐startup/id422540072?mt=11 • BreQ DurreQ, IMVU, CEO – Con8nuous Deployment… Con8nuous Deployment: Possibility or Pipe Dream? hRp://bos*nnova*on.com/2011/11/21/con*nuous-‐deployment-‐possibility-‐or-‐pipe-‐dream Con8nuous Deployment & Learning Fast with A/B Tes8ng hRp://www.slideshare.net/bgdurreR/con*nuous-‐deployment-‐at-‐lean-‐la hRp://www.slideshare.net/bgdurreR/learning-‐fast-‐with-‐ab-‐tes*ng-‐and-‐con*nuous-‐deployment hRp://www.mediabistro.com/The-‐Challenges-‐of-‐Con*nuous-‐Deployment-‐Social-‐Developer-‐Summit-‐467-‐ondemandvideo.html • • • • • James Birchler, IMVU, VP Engineering Learning from Experiments hRp://www.slideshare.net/jamesbirchler/learning-‐from-‐experiments-‐at-‐imvu Flickr, 10+ Deploys Per Day: Dev and Ops Coopera*on at Flickr hRp://www.slideshare.net/jallspaw/10-‐deploys-‐per-‐day-‐dev-‐and-‐ops-‐coopera*on-‐at-‐flickr Carol Dweck, Mindset: The New Psychology of Success hRp://www.amazon.com/Mindset-‐Psychology-‐Success-‐Carol-‐Dweck/dp/0345472322/ref=sr_1_1?s=books&ie=UTF8&qid=1318359458&sr=1-‐1 hRp://itunes.apple.com/us/book/mindset/id422549774?mt=11 hRp://www.youtube.com/watch?v=XHW9l_sCEyU Did You Know (a.k.a ShiD Happens) Blog Post, Slides, Videos hRp://www.lps.k12.co.us/schools/arapahoe/fisch/fischbowlpresenta*ons.htm hRp://shiDhappens.wikispaces.com Paul Stoltz, Peak Learning hRp://www.peaklearning.com – Erik Weihenmayer & Paul Stoltz, The Adversity Advantage hRp://itunes.apple.com/us/book/the-‐adversity-‐advantage/id381515439?mt=11 hRp://www.amazon.com/Adversity-‐Advantage-‐Everyday-‐Struggles-‐Greatness/dp/1439199493/ref=sr_1_1?s=books&ie=UTF8&qid=1318360603&sr=1-‐1 – Paul Stoltz, Adversity Quo*ent at Work hRp://itunes.apple.com/ca/book/adversity-‐quo*ent-‐work/id385756581?mt=11 hRp://www.amazon.com/Adversity-‐Quo*ent-‐Work-‐Finding-‐Capacity/dp/0060937211/ref=sr_1_sc_1?s=books&ie=UTF8&qid=1318360551&sr=1-‐1-‐spell • • • Michael Jordan, "Failure" Nike Commercial hRp://www.youtube.com/watch?v=45mMioJ5szc Malcolm Gladwell, Outliers & Tipping Point hRp://www.gladwell.com/outliers/index.html hRp://itunes.apple.com/us/book/outliers/id357396748?mt=11 hRp://www.gladwell.com/*ppingpoint/index.html hRp://itunes.apple.com/us/book/the-‐*pping-‐point/id357658331?mt=11 Is Con8nuous Innova8on Too Risky? hRp://www.forbes.com/sites/stevedenning/2012/02/10/is-‐radical-‐management-‐too-‐risky Chris Dolezalek – October, 2012

24. References on Experimen8ng • Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, Ya Xu http://www.exp-platform.com/Documents/puzzlingOutcomesInControlledExperiments.pdf • Puzzling outcomes in A/B testing - Greg Linden http://glinden.blogspot.co.uk/2012/07/puzzling-outcomes-in-ab-testing.html • Microsoft EXP - Experimentation Platform - Ronny Kohavi Accelerating software innovation through trustworthy experimentation http://www.exp-platform.com/Pages/default.aspx • • What types of things does Netflix A/B test aside from member sign-up? http://www.quora.com/What-types-of-things-does-Netflix-A-B-test-aside-from-member-sign-up From Zero to a Million Users - Dropbox and Xobni lessons learned - Adam Smith http://www.slideshare.net/adamsmith1/from-zero-to-a-million-users-dropbox-and-xobni-lessons-learned • Data-‐Driven Startups -‐ July 23, 2010 http://davidcancel.com/data-driven-startups • Analy8cs Maturity hRp://www.forbes.com/sites/piyankajain/2012/06/22/what-‐is-‐your-‐organiza*ons-‐analy*cs-‐maturity/ Chris Dolezalek – October, 2012

25. Appendix, Drill Down Slides Chris Dolezalek – October, 2012

26. Prototype, Test, Refactor Defined Milestones & Iteration No Meeting No Meeting Mondays Mondays … Interrupt Lanes/ Engineers Risk Buckets vs Risk factored into task estimates "It is one of the happy incidents of the federal system, that a single courageous state may, if its citizens choose, serve as a laboratory; and try novel social and economic experiments without risk to the rest of the country.” -‐ Louis D. Brandeis, 1932 Supreme Court Jus*ce Experiment as individuals but also as teams IMVU con*nually runs process experiments as a way to improve our processes and to avoid falling into dysfunc*onal rou*nes. Chris Dolezalek – October, 2012

27. Zynga Numbers • Zynga’s daily highs of concurrent players is equivalent to everyone in the city of Paris playing together at the same *me. That happens every day. The games are always changing and the devices they’re played on change constantly. Zynga releases over 100 updates across all its games every day. The company once released more than 1000 updates in one week. Zynga players make a million ac*ons per second. • Zynga runs about 130 experiments in its games every day. ADer releasing a new feature Zynga can ﬁnd out within minutes if players enjoyed them. Beyond fun is social — what maRer is that friends and family con*nue to play. This is a player’s Ac*ve Social Network (ASN). It’s a true barometer of how social a game is. hRp://www.insidesocialgames.com/2012/06/26/live-‐from-‐zynga-‐unleashed Chris Dolezalek – October, 2012

28. Flickr’s Perspec*ve hQp://www.slideshare.net/jallspaw/10-‐deploys-‐per-‐day-‐dev-‐and-‐ops-‐coopera8on-‐at-‐ﬂickr Chris Dolezalek – October, 2012

29. What’s Under Your Hood? Improving, replacing and dele*ng old code can vital, but when and how? hRp://www.commercialmotor.com/big-‐lorry-‐blog/now-‐thats-‐more-‐like-‐it-‐for-‐biglorryblog-‐one-‐horsepower-‐truck-‐exclusive-‐picture Chris Dolezalek – October, 2012

30. VALVE SoDware’s Game Design Process hRp://www.valvesoDware.com/publica*ons/2009/GDC2009_ValvesApproachToPlaytes*ng.pdf hRp://www.geekwire.com/2011/experiments-‐video-‐game-‐economics-‐valves-‐gabe-‐newell Valve is the company that created the Steam Game Distribu*on system. Chris Dolezalek – October, 2012

31. Con*nuous Improvement -‐ Iterate 1. 2. 3. 4. 5. Release Minimal Viable Products (MVPs), Get Customer Feedback / Collect Metrics (don’t presume you know what’ll work), Analyze Changes, Make Itera*ve Changes Go back to Step 2 hRp://buu700.com/steverant : Also note that he men*ons a "ship early and iterate" ethos at Amazon, and also how a clear order from the top was what successfully drove them into building SOA pla|orms despite the high ini*al cost. Chris Dolezalek – October, 2012

32. Counter Innova8on • How to make your oﬀering less appealing to… – Fraudsters – Scammers – Spammers – Cyber-‐Bullies Knowing your users may need to go beyond knowing the ones you want to keep. Chris Dolezalek – October, 2012

33. Priori8za8on When priori*zing, deciding what’s at the top of the list, though hard, oDen maRers less than what’s at the boRom… 1. iOS App 2. Android App 3. Infrastructure Upgrade 4. Enhance Adver*sing interface Chris Dolezalek – October, 2012

34. On Le}ng Go Shoo8ng the dogs (lePng go): • Loss aversion and the “sunk cost fallacy” Many people have strong misgivings about "was*ng" resources (loss aversion). They may feel they've passed a point of no return. Economists would label this behavior "irra*onal": it is ineﬃcient because it misallocates resources by depending on informa*on that is irrelevant to the decision being made. Colloquially, this is known as "throwing good money aDer bad“. • Sunk Cost Dilemma The dilemma of having to choose between con*nuing a project of uncertain prospects already involving considerable sunk costs, or discon*nuing the project. Given this choice between the certain loss of the sunk costs when stopping the project versus possible – even if unlikely – long-‐term proﬁtability when going on, policy makers tend to favor uncertain success over certain loss. h<p://en.wikipedia.org/wiki/Sunk_costs#Loss_aversion_and_the_sunk_cost_fallacy h<p://en.wikipedia.org/wiki/Sunk_cost_dilemma Chris Dolezalek – October, 2012

35. Mobile App Release verses Con8nuous Deployment  pp Stores require extended *me to market A + obile users expect higher ﬁdelity apps M = ismatch from con*nuous web-‐deployment M Reaching back to trusted prac*ces: Clearer Requirements Release Candidates Stable Libraries Code Freezes Merging Changes Rigorous Release Tes8ng Bug Triages Release Sign-‐Oﬀs Release Reference Copies Risk-‐Awareness Down8me Avoidance Server Isola8ons Chris Dolezalek – October, 2012

Continuous Innovation

Continuous Innovation

Recommended

More Related Content

What's hot

What's hot(20)

Similar to Continuous Innovation

Similar to Continuous Innovation(20)

Recently uploaded

Recently uploaded(20)

Continuous Innovation

Editor's Notes