Continuous Innovation
Upcoming SlideShare
Loading in...5
×
 

Continuous Innovation

on

  • 719 views

Continuous Innovation In a Rapidly Evolving Market

Continuous Innovation In a Rapidly Evolving Market
Experimenting and being agile with products, processes, …

Statistics

Views

Total Views
719
Views on SlideShare
719
Embed Views
0

Actions

Likes
0
Downloads
8
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • One of my interview questions for an Engineering Manager: What is the one thing you should be religious about when exercising “agile” software development?The answer I’m looking for: The one mantra to be religious about is to not be “religious” about anything – that’s why it’s called “agile”.No business plan, not product vision, no strategy, no process is perfect in it’s initial conception. Likewise, none is successful without adjusting and course correcting along the way.There is no silver bullet: Judgment trumps process, process serves the organization and not vice versa.
  • +7? At Softlab, as architect, I architected three key technology pivots that didn’t leave behind the existing technology or customers though the business had initially said that would be ok.This technology Pivot (actually two) that Enabled other pivots as wellAs VP, Eng at BroadVision, Inc, before, during and after the pivot… Also a Technology Pivot that Enabled other pivots as well (segment pivot, …) ultimately from 1 to 7 product linesBroadVision also did an early market and technology pivot in 1996 from BroadBand to Internet – same basic premise, radically different technology and market.At Cooperative Solutions, Inc when PowerSoft entered the space and did the pivot “for” us… -> PowerSoft: someone else ran with the pivot… Industry's largest acquisition to date…“PowerBuilder was originally developed by PowerSoft in 1991. PowerSoft went public in 1993 and was acquired by Sybase for $904 million in Sybase stock in 1995.”
  • +1 In order to enable continuous innovation, you need a team that will embrace it. Start with the people.Hire for potential not current state knowledge: your employees need to be able and eager to adapt to changing markets and changing processes.Hiring a structural engineer for a software engineering position: At IMVU we hire for passion and ability to collaborate and learn, and we manage to inspire.When we interview at IMVU, we don’t probe for knowledge so much; we test for problem solving, intelligence and ability to ask the right questions to understand the problem.Manage:What makes for the most productive employee?Happiness, Passion, Engagement – Consider also what Google does for it’s employee and how one of the first things Marissa did when she came to Yahoo was to institute the same sort of benefits. Everybody is a Genius. But if you judge a fish by its ability to climb a tree, it will spend its whole life believing that it is stupid. ~ Albert Einstein If you want to climb trees, hire monkeys; if you want to innovate, hire people with innovative mindsets… Company performance based bonuses
  • We continually improve the spin up guide with each new hireMore on this later
  • +2 Engineering manager interview question: What would you say is the single most important contributing factor in making an engineer productive?The answer I’m looking for in one form or another: A happy engineer. Happy engineers are motivated, passionate, engaged and productive. Quoted on 1-on-1 question: are you happy?5 whys of a one-on-one: What are you working on? Why are you working on that? Why does that matter? Which company goal does that map to? Why does that company goal matter?Engineers need to believe in and care about the final outcome for product and company.Transparency
  • Run Experiments: see above In order to continuously deploy, it is vital to have good test coverage. If you don’t start with it, how do you get there – where to start adding coverage? Use an immune system to catch abnormalities in any changes that you roll out in production Start with MVP’s to learn fast, fail fast, and start iterating and continuous innovation asap How to learn from what went wrong in production release, processes, … find the teachable moment, the take-away from any situation or outcome.
  • We hire smart people to come up with good hypothesis on what will make our customers happy. However, we also validate those hypotheses (not every can be a Steve Jobs)In this case, looking at the data, it appears the impact of the change was minor.Label y axis
  • +3 We hire smart people to come up with good hypothesis on what will make our customers happy. However, we also validate those hypotheses (not everyone can be a Steve Jobs)In this case, looking at the data, it appears the impact of the change was minor.Also compare click-through rates to new-paying-user conversions to LTVsWhich data to look at - Photo Stream example of engagement vs buy from picture vs downstream buy vs buy from same creator vs buy to generate better photos and collect accolades
  • As the company grows, the mission and vision evolves, and new technologies and tools appear, IMVU too will continue to evolve rather than stagnate.Each team chooses the processes and tools that work best for it as discovered through experimentation followed by retrospectives, and that knowledge is shared across teams.1. Existing experience: Millions of monthly hits on landing page = easy to quickly see the impact of a change; however, no longer solving for early adopters = higher bar2. Infra-Maint: Balanced Priorities, Graphs, Interrupt Engineers & Lanes/Stories3. New Paradigm: Design>User-Test>Project-Brief>Develop MVP>Iterate4. A New Game: Start with no users, Story Board and user test, prototype and beta test, iterate, validate, release, iterate5. New Infrastructure: TDD
  • +4 What it means to release an MVP differs fairly significantly with what you are introducing, to whom, where and what you hope to learn…e.g forgiving early adopters vs existing/mainstream, free vs paying users, funnel conversion vs LTVKnow your audience: eBay PM at IMVU – why people come to your site influences what they are tolerant of – they don’t just come to IMVU to buy/sell (credits/vg), primary focus is social interaction In a rapidly changing and increasingly diverse global market, it’s hard to know what will flyGet products in front of customers asapTakes RisksIterate based on feedback and dataOr, shoot the dog(aka Fail-Fast) & LearnSee also:http://www.rise.net/blog/minimum-viable-product
  • Intuit (from 6-12 months to 2-3 weeks), Yahoo (from 2-3 weeks to 1-2x daily), IMVU (every 40 minutes to every 7)Benefits of small changes for product and for engineering – see also Immune system and continuous integration
  • If you’re starting with a brand new company that believes in TDD, that’s one thing…If you’re an existing company and want to add tests in hindsight, where do you start?
  • IMVU’s Immune System allows a new change to be rolled out to one or two servers first. The system monitors memory usage, cpu usage, customer usage, … If any threshold of change is exceeded, the system automatically rolls back the change and alerts the developer(s) that contributed to that change.With continuous deployment, this change-set is usually very small and easy to debug.We have have extensive monitoring of production systems to alert us when something goes awry.If something does go wrong in production, there is a post-mortem with follow-ups to be addressed within the next sprint.
  • +5 Use this as an example of 5 (4-6) whys, lessons learned, good experience Mentoring Testing Monitoring and Immune System
  • +6?Immune SystemHypothesis BuildsRollbacks and Finding the Issue
  • MalcolmGladwell’s The Tipping PointIMVU Forum Moderators and other Mavens provide good insights into what will resonate with existing user base.Stipso, Steven Drost – Lunch conversation about the value and relevance of traditional survey results.
  • http://planningpoker.com/
  • Root Cause Analysis – Titanic (Cause Mapping) http://www.youtube.com/watch?v=GOVeO5_0qD05 Whys Fun Skit http://www.youtube.com/watch?v=5od0T_Fhk9oSocial Intelligence:http://itunes.apple.com/gb/book/social-intelligence/id436993236?mt=11 http://www.amazon.com/Social-Intelligence-Science-Human-Relationships/dp/0553803522
  • As the company grows, the mission and vision evolves, and new technologies and tools appear, IMVU too will continue to evolve rather than stagnate.Each team and individual may innovate and evolve separately as needed
  • Yahoo Media’s approach vFlickr’s even though Yahoo acquired Flickr…
  • We often rip out or rewrite entire systems that have grown organically over time. We learn from refactoring existing code and take learnings from that into our review process for new code (often written by the same engineer just years later)
  • “Issues with Traditional Methods:Artificial gameplay sessions- Many potential biases- Distorted data (interpreted behavior)- Lack of empiricism- Missing elements of objectivity- Sometimes difficult to establish emotions, baselines, and independence
  • Imagine going to the movies, buying tickets and 10 minutes deciding it’s a really crappy movie; would you still stay to the end just because you’re already there and paid?Impermanence (split this slide into 2)

Continuous Innovation Continuous Innovation Presentation Transcript

  • Con8nuous  Innova8on   In  a  Rapidly  Evolving  Market   Experimen*ng  and  being  agile   with  products,  processes,  …   Chris  Dolezalek  –  October,  2012   Chris  Dolezalek  –  October,  2012  
  • The path to enlightenment lies not in the footsteps of another What is the one mantra to follow religiously? Follow no mantra religiously   初心 Beginner’s  Mind   Chris  Dolezalek  –  October,  2012  
  • The  Pivot   That  Elusive  Hockey  S*ck   BroadVision,  Inc.  pivots  to   become  fastest  growing   soDware  company  on  Nasdaq   SoDlab,  GmbH  pivots  to  become   Germany’s  2nd  fastest  growing   independent  soDware  company     Google $0 to $37.9B in 10 years   ”Innova(ng  at  a  breakneck        pace;  rolling  out  a  new        service  capability        prac(cally  every  month1.” PowerSoD  developed  PowerBuilder   in  1991,  went  public  in  1993  and  was   acquired  for  ~$900  million  in  1995   1. The  Ten  Faces  of  Innova*on  by  Thomas  Kelley  &  Jonathan  LiRman   hRps://itunes.apple.com/us/book/the-­‐ten-­‐faces-­‐of-­‐innova*on/id421032598?mt=11   hRp://www.sunarsports.com/products/Easton-­‐Stealth-­‐S19-­‐Sr.-­‐Composite-­‐Hockey-­‐S*ck   Chris  Dolezalek  –  October,  2012  
  • Lean Staffing Hiring Your Most Valuable Resources Hire  for  Tomorrow  not  Today   •  The  top  10  in-­‐demand  jobs  in  2010     did  not  exist  in  2004   “Did You Know? Shift Happens (August 2011)“ http://www.youtube.com/watch?v=EJpSqeQbh4o Skills  vs  Curiosity  &  Passion   •  "CQ  +  PQ  >  IQ”     Curiosity  &  Passion  Quo:ents     are  more  important  than     Intelligence  Quo:ent    Thomas  Friedman,  The  World  is  Flat   APtude  MaQers   •  “It  is  not  their  ap:tude      but  their  aFtude      that  will  determine  their  al:tude.”   Jesse  Jackson,  Washington  Post  21  May  1978   Chris  Dolezalek  –  October,  2012  
  • Don’t  stop                                    at  Hiring   Invest  in  spinning  up   At  IMVU,  we  dedicate  a  mentor  to   every  new  engineer  that  will  sit  next   to  them  and  spend  6  weeks  spinning   them  up  according  to  a  boot  camp   guide  -­‐  including  a  day-­‐one  push  of  a   change  to  produc*on.   Spinning  Marathon  voor  WAR  CHILD  -­‐  Some  rights  reserved  By  isafmedia  (cropped)   Chris  Dolezalek  –  October,  2012  
  • Manage  Your  Most  Valuable  Resources   What  makes  for  the  most  produc*ve  &  innova*ve  employee?   Happiness  =  Passion,  Engagement   •  Support  your  team   Perspec8ve:  Managers  work  for  the  engineers  to  enable  them  to  be  produc8ve   •  Sell  vs  Tell  =  No  show,  No  go   Winning  the  hearts  and  minds  bring  engagement  as  opposed  to  bodies  in  seats.     5-­‐Whys:    Company  Direc8ves  Alignment  =  What  are  you  working  on  and  Why  is  it  important?   •  Be  Transparent  &  Invite  Challenges    Share  business  objec8ves  and  reward  those  who  ques8ons  them   •  Let  them  pick  their  team  &  project   LiRle  startups/laboratories   •  Let  them  Hack   Hack  weeks  once  a  quarter   •  Let  them  experiment   •  Let  them  choose  their  tools   •  Let  them  determine  when  done  is   Story  Point  Sizing   •  Keep  teams  small,  agile  and  fun   (BVSN  5  man  teams  vs  whole  companies)   Chris  Dolezalek  –  October,  2012  
  • • Run  Experiments       • Minimal  Viable  Products     • Con*nuous  Deployment     • Smart  Test  Coverage     • Use  an  Immune  System     • Post  Mortems,  &  Retrospec*ves  &  5  Whys     Chris  Dolezalek  –  October,  2012  
  • Number  of  uses  of  new  feature   Con8nuous  Innova8on     Validate  Hypotheses  with  Data   Time:  Experiment  A:  up  ~6%  over  2  weeks  prior   Chris  Dolezalek  –  October,  2012  
  • Number  of  uses  of  new  feature   Con8nuous  Innova8on     Validate  Hypotheses  with  the  Right  Data   Time:  Subgroup  A1:              up  ~48%  over  2  weeks  prior                        Subgroup  A2:  down  ~68%  over  2  weeks  prior   Chris  Dolezalek  –  October,  2012  
  • Experiment  as   individuals  but   also  as  teams   5  Teams   5  Different   Processes   IMVU  con*nually  runs  process  experiments  as  a  way  to  improve  our   processes  and  to  avoid  falling  into  dysfunc*onal  rou*nes.   Chris  Dolezalek  –  October,  2012  
  • Minimal  Viable  Products   •  Different  Kinds  of  MVPs   hRp://www.rise.net/blog/minimum-­‐viable-­‐product   – A  heavily  used  system     (IMVU’s  Landing  Pages  Millions  of  monthly  hits)   – New  Feature  Sets     (inventory  management  into  exis*ng  product)   – A  brand  new  subsystem     (a  new  game  with  a  new  game/fun  loop)   – A  paradigm  shiD   (an  iPhone    App)   Chris  Dolezalek  –  October,  2012  
  • Deploy,  Iterate  &  Learn  at  New  Speeds   Production releases How  ofen  do  you  deploy?   •  Waterfall  Development  :   every  6-­‐12  months   •  Agile  Methodology:     every  1-­‐3  weeks   •  Con8nuous  Deployment:   every  day   •  IMVU   every  7  Minutes   Chris  Dolezalek  –  October,  2012  
  • Test-­‐Driven  Development    Where/How  to  start?   Test  Driven  Development  hRp://www.flickr.com/photos/brunobord/3987593006/lightbox  Some  rights  reserved  by  brunobord   Chris  Dolezalek  –  October,  2012  
  • Immune  Systems   Small  Change-­‐Sets  =  Easy  Debugging   Chris  Dolezalek  –  October,  2012  
  • 5  Whys     Turning  Losses  into  Wins   1.  Why  did  the  server  grind  to  a  halt?   DB  query  took  forever.   2.  Why  did  the  query  take  so  long?   Un-­‐indexed  query  on  large  table.   3.  Why  didn’t  the  engineer  realize  the  query  would  be  slow?   Local  tests  ran  fine  against  a  small  test  db.   4.  Why  didn’t  the  engineer  know  to  run  a  “try”  against  produc:on?   It  was  his  first  database  change  as  a  new  hire.   5.  Why  did  his  mentor  not  tell  him?   The  mentor  had  leD  for  the  day.   6.  Why  did  the  new  hire  not  know  to  only  check-­‐in  with  mentor?     This  was  the  first  mentee  for  that  mentor  and  he  hadn’t  told  him.       Follow-­‐Up  1)  Update  Spin-­‐Up  Doc  for  Mentor  and  Mentee  to  start  with  a  clear   statement  that  mentee  should  not  check  into  produc*on  without  mentor  or   without  having  reached  that  stage.   Follow-­‐Up  2)  Add  notes  to  DB  Query  sec*on  of  spin-­‐up  doc  on  use  of  “try”   Post-­‐Mortems  =  Teachable  Moments  ;  Fix  what  needs  fixing   Ques*on  mark  sign  photo  by  Colin_K  on  Flickr   Chris  Dolezalek  –  October,  2012  
  • Requirement  Opacity   Granularity  versus  deadlines:      It’s  ok  not  to  have  the  nth  degree  of  detail  of  what  you’re  building  in  6  months’  8me.   Chris  Dolezalek  –  October,  2012  
  • Con8nuous  Deployment  Flow   Hypothesis   Build  on   VMs   hRp://www.slideshare.net/bgdurreR/con*nuous-­‐deployment-­‐at-­‐lean-­‐la   Chris  Dolezalek  –  October,  2012  
  • Experiments  &  Alterna8ves   •  Experiments:  Vo*ng  by  doing,  requires  func*onality   •  User  Sessions:  Hard  to  get  good  sample  size  quickly   •  Customer  Advocates:  Par*cipates  in  forums,  user  groups,  …   •  Anthropologist:  Do  field  research  to  observe  interac*ons   •  Surveys:  What  people  say  rather  than  do   •  Dog-­‐fooding:  Use  your  product   •  Mavens:  Good  insights  into  established  market   hRp://www.gladwell.com/*ppingpoint/index.html   Chris  Dolezalek  –  October,  2012  
  • An  Enhancement  /  Story  is  Born   •  Project  Brief  Goes  to  a  Exec  Checkpoint   –  User  stories   –  Success  criteria  by  date  x…  (engagement,  revenue,  …)   •  Preview  Mee*ng  by  Product  Owner   –  To:  Tech  Lead,  QA  Lead,  Data  Analyst   •  Tech  Design,  Test  Plan,  Data  Design   •  Planning  Mee*ng   –  Story  point  sizing   –  High  and  low  es*mate  jus*fica*on   –  Risk  buckets  for  stories  vs  risk  included  in  task  es*mates   Chris  Dolezalek  –  October,  2012  
  • Delivering  Innova*on  to  Risk  Averse,   Enterprise  Customers   •  In  1985,  I  came  to  Edinburgh  from  Munich  to  help  sell   the  Bank  of  Scotland  on  transi*oning  to  our  new   product  that  resulted  from  our  second  pivot  at  SoDlab.   •  What  unfolded  was  not  quite  according  to  script.   •  They  were  not  at  all  excited  about  adop*ng  innova*on,   having  in  8  previous  systems  suppor*ng  180  of  their  IT   staff.  But,  they  did…   Chris  Dolezalek  –  October,  2012  
  • Apply  Beginner’s  Mind  and  Con8nuously  Innovate…   Your  Product  Direc*on,   Your  Company  Direc*on,   初心 Beginner’s  Mind   Your  Development  Processes,   Your  Resource  Management,   Your  Marke*ng  Strategy,   Your  Technology  Choices,   Your  Analy*cs  Approaches,   Understanding  Your  Customers  &  Prospects,  ..   Everything  You  Do  and  Every  Approach  You  Take.   Chris  Dolezalek   Chris  Dolezalek  –  October,  2012  
  • Thank You! Chris Dolezalek Chris  Dolezalek  –  October,  2012   22  
  • References  &  Other  Links   •  Eric  Ries,  IMVU,  former  CTO  of  Engineering  -­‐  The  Lean  Startup   How  Today's  Entrepreneurs  Use  Con*nuous  Innova*on  to  Create  Radically  Successful  Businesses   hRp://www.amazon.com/Lean-­‐Startup-­‐Entrepreneurs-­‐Con*nuous-­‐Innova*on/dp/0307887898/ref=sr_1_1?ie=UTF8&qid=1318352510&sr=8-­‐1 hRp://itunes.apple.com/us/book/the-­‐lean-­‐startup/id422540072?mt=11   •    BreQ  DurreQ,  IMVU,  CEO  –  Con8nuous  Deployment…   Con8nuous  Deployment:  Possibility  or  Pipe  Dream?  hRp://bos*nnova*on.com/2011/11/21/con*nuous-­‐deployment-­‐possibility-­‐or-­‐pipe-­‐dream   Con8nuous  Deployment  &  Learning  Fast  with  A/B  Tes8ng    hRp://www.slideshare.net/bgdurreR/con*nuous-­‐deployment-­‐at-­‐lean-­‐la   hRp://www.slideshare.net/bgdurreR/learning-­‐fast-­‐with-­‐ab-­‐tes*ng-­‐and-­‐con*nuous-­‐deployment   hRp://www.mediabistro.com/The-­‐Challenges-­‐of-­‐Con*nuous-­‐Deployment-­‐Social-­‐Developer-­‐Summit-­‐467-­‐ondemandvideo.html   •  •  •  •  •  James  Birchler,  IMVU,  VP  Engineering   Learning  from  Experiments  hRp://www.slideshare.net/jamesbirchler/learning-­‐from-­‐experiments-­‐at-­‐imvu   Flickr,  10+  Deploys  Per  Day:  Dev  and  Ops  Coopera*on  at  Flickr   hRp://www.slideshare.net/jallspaw/10-­‐deploys-­‐per-­‐day-­‐dev-­‐and-­‐ops-­‐coopera*on-­‐at-­‐flickr   Carol  Dweck,  Mindset:  The  New  Psychology  of  Success   hRp://www.amazon.com/Mindset-­‐Psychology-­‐Success-­‐Carol-­‐Dweck/dp/0345472322/ref=sr_1_1?s=books&ie=UTF8&qid=1318359458&sr=1-­‐1   hRp://itunes.apple.com/us/book/mindset/id422549774?mt=11   hRp://www.youtube.com/watch?v=XHW9l_sCEyU   Did  You  Know  (a.k.a  ShiD  Happens)  Blog  Post,  Slides,  Videos   hRp://www.lps.k12.co.us/schools/arapahoe/fisch/fischbowlpresenta*ons.htm   hRp://shiDhappens.wikispaces.com   Paul  Stoltz,  Peak  Learning     hRp://www.peaklearning.com   –  Erik  Weihenmayer  &  Paul  Stoltz,  The  Adversity  Advantage   hRp://itunes.apple.com/us/book/the-­‐adversity-­‐advantage/id381515439?mt=11   hRp://www.amazon.com/Adversity-­‐Advantage-­‐Everyday-­‐Struggles-­‐Greatness/dp/1439199493/ref=sr_1_1?s=books&ie=UTF8&qid=1318360603&sr=1-­‐1   –  Paul  Stoltz,  Adversity  Quo*ent  at  Work   hRp://itunes.apple.com/ca/book/adversity-­‐quo*ent-­‐work/id385756581?mt=11   hRp://www.amazon.com/Adversity-­‐Quo*ent-­‐Work-­‐Finding-­‐Capacity/dp/0060937211/ref=sr_1_sc_1?s=books&ie=UTF8&qid=1318360551&sr=1-­‐1-­‐spell   •  •  •  Michael  Jordan,  "Failure"  Nike  Commercial     hRp://www.youtube.com/watch?v=45mMioJ5szc   Malcolm  Gladwell,  Outliers  &  Tipping  Point   hRp://www.gladwell.com/outliers/index.html                        hRp://itunes.apple.com/us/book/outliers/id357396748?mt=11   hRp://www.gladwell.com/*ppingpoint/index.html        hRp://itunes.apple.com/us/book/the-­‐*pping-­‐point/id357658331?mt=11   Is  Con8nuous  Innova8on  Too  Risky?   hRp://www.forbes.com/sites/stevedenning/2012/02/10/is-­‐radical-­‐management-­‐too-­‐risky   Chris  Dolezalek  –  October,  2012  
  • References  on  Experimen8ng   •  Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, Ya Xu http://www.exp-platform.com/Documents/puzzlingOutcomesInControlledExperiments.pdf •  Puzzling outcomes in A/B testing - Greg  Linden http://glinden.blogspot.co.uk/2012/07/puzzling-outcomes-in-ab-testing.html •  Microsoft EXP - Experimentation Platform - Ronny Kohavi Accelerating software innovation through trustworthy experimentation http://www.exp-platform.com/Pages/default.aspx •  •  What types of things does Netflix A/B test aside from member sign-up? http://www.quora.com/What-types-of-things-does-Netflix-A-B-test-aside-from-member-sign-up From Zero to a Million Users - Dropbox and Xobni lessons learned - Adam Smith http://www.slideshare.net/adamsmith1/from-zero-to-a-million-users-dropbox-and-xobni-lessons-learned •  Data-­‐Driven  Startups  -­‐  July  23,  2010 http://davidcancel.com/data-driven-startups •  Analy8cs  Maturity hRp://www.forbes.com/sites/piyankajain/2012/06/22/what-­‐is-­‐your-­‐organiza*ons-­‐analy*cs-­‐maturity/ Chris  Dolezalek  –  October,  2012  
  • Appendix,   Drill  Down  Slides   Chris  Dolezalek  –  October,  2012  
  • Prototype, Test, Refactor Defined Milestones & Iteration No Meeting No Meeting Mondays Mondays … Interrupt Lanes/ Engineers Risk Buckets vs Risk factored into task estimates "It is one of the happy incidents of the federal system, that a single courageous state may, if its citizens choose, serve as a laboratory; and try novel social and economic experiments without risk to the rest of the country.” -­‐ Louis  D.  Brandeis,  1932   Supreme  Court  Jus*ce   Experiment  as   individuals  but   also  as  teams   IMVU  con*nually  runs  process  experiments  as  a  way  to  improve  our   processes  and  to  avoid  falling  into  dysfunc*onal  rou*nes.   Chris  Dolezalek  –  October,  2012  
  • Zynga  Numbers   •  Zynga’s  daily  highs  of  concurrent  players  is  equivalent  to   everyone  in  the  city  of  Paris  playing  together  at  the  same   *me.  That  happens  every  day.  The  games  are  always  changing   and  the  devices  they’re  played  on  change  constantly.  Zynga   releases  over  100  updates  across  all  its  games  every  day.  The   company  once  released  more  than  1000  updates  in  one  week.   Zynga  players  make  a  million  ac*ons  per  second.   •  Zynga  runs  about  130  experiments  in  its  games  every  day.   ADer  releasing  a  new  feature  Zynga  can  find  out  within   minutes  if  players  enjoyed  them.  Beyond  fun  is  social  —  what   maRer  is  that  friends  and  family  con*nue  to  play.  This  is  a   player’s  Ac*ve  Social  Network  (ASN).  It’s  a  true  barometer  of   how  social  a  game  is.   hRp://www.insidesocialgames.com/2012/06/26/live-­‐from-­‐zynga-­‐unleashed     Chris  Dolezalek  –  October,  2012  
  • Flickr’s  Perspec*ve   hQp://www.slideshare.net/jallspaw/10-­‐deploys-­‐per-­‐day-­‐dev-­‐and-­‐ops-­‐coopera8on-­‐at-­‐flickr   Chris  Dolezalek  –  October,  2012  
  • What’s  Under  Your  Hood?    Improving,  replacing  and  dele*ng  old  code  can  vital,  but  when  and  how?   hRp://www.commercialmotor.com/big-­‐lorry-­‐blog/now-­‐thats-­‐more-­‐like-­‐it-­‐for-­‐biglorryblog-­‐one-­‐horsepower-­‐truck-­‐exclusive-­‐picture     Chris  Dolezalek  –  October,  2012  
  • VALVE  SoDware’s     Game  Design  Process   hRp://www.valvesoDware.com/publica*ons/2009/GDC2009_ValvesApproachToPlaytes*ng.pdf   hRp://www.geekwire.com/2011/experiments-­‐video-­‐game-­‐economics-­‐valves-­‐gabe-­‐newell   Valve  is  the  company  that  created  the  Steam  Game  Distribu*on  system.   Chris  Dolezalek  –  October,  2012  
  • Con*nuous  Improvement  -­‐  Iterate   1.  2.  3.  4.  5.  Release  Minimal  Viable  Products  (MVPs),   Get  Customer  Feedback  /  Collect  Metrics  (don’t  presume  you  know  what’ll  work),   Analyze  Changes,   Make  Itera*ve  Changes     Go  back  to  Step  2   hRp://buu700.com/steverant  :  Also  note  that  he  men*ons  a  "ship  early  and  iterate"  ethos  at  Amazon,   and  also  how  a  clear  order  from  the  top  was  what  successfully  drove  them  into  building  SOA  pla|orms   despite  the  high  ini*al  cost.   Chris  Dolezalek  –  October,  2012  
  • Counter  Innova8on   •  How  to  make  your  offering  less  appealing  to…   – Fraudsters   – Scammers   – Spammers   – Cyber-­‐Bullies   Knowing  your  users  may  need  to  go  beyond   knowing  the  ones  you  want  to  keep.   Chris  Dolezalek  –  October,  2012  
  • Priori8za8on   When  priori*zing,  deciding  what’s  at  the  top  of   the  list,  though  hard,  oDen  maRers  less  than   what’s  at  the  boRom…   1.  iOS  App   2.  Android  App   3.  Infrastructure  Upgrade   4.  Enhance  Adver*sing  interface   Chris  Dolezalek  –  October,  2012  
  • On  Le}ng  Go   Shoo8ng  the  dogs  (lePng  go):   •  Loss  aversion  and  the  “sunk  cost  fallacy”   Many  people  have  strong  misgivings  about  "was*ng"  resources   (loss  aversion).    They  may  feel  they've  passed  a  point  of  no   return.  Economists  would  label  this  behavior  "irra*onal":  it  is   inefficient  because  it  misallocates  resources  by  depending  on   informa*on  that  is  irrelevant  to  the  decision  being  made.   Colloquially,  this  is  known  as  "throwing  good  money  aDer  bad“.   •  Sunk  Cost  Dilemma   The  dilemma  of  having  to  choose  between  con*nuing  a  project   of  uncertain  prospects  already  involving  considerable  sunk   costs,  or  discon*nuing  the  project.  Given  this  choice  between   the  certain  loss  of  the  sunk  costs  when  stopping  the  project   versus  possible  –  even  if  unlikely  –  long-­‐term  profitability  when   going  on,  policy  makers  tend  to  favor  uncertain  success  over   certain  loss.   h<p://en.wikipedia.org/wiki/Sunk_costs#Loss_aversion_and_the_sunk_cost_fallacy   h<p://en.wikipedia.org/wiki/Sunk_cost_dilemma   Chris  Dolezalek  –  October,  2012  
  • Mobile  App  Release  verses   Con8nuous  Deployment     pp  Stores  require  extended  *me  to  market   A +  obile  users  expect  higher  fidelity  apps   M =  ismatch  from  con*nuous  web-­‐deployment   M Reaching  back  to  trusted  prac*ces:   Clearer  Requirements   Release  Candidates   Stable  Libraries   Code  Freezes   Merging  Changes   Rigorous  Release  Tes8ng   Bug  Triages   Release  Sign-­‐Offs   Release  Reference  Copies   Risk-­‐Awareness   Down8me  Avoidance   Server  Isola8ons   Chris  Dolezalek  –  October,  2012