Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Disruptive Innovation: how do you use these theories to manage your IT?


Published on

The term disruptive innovation was popularized by Harvard professor Clayton Christensen in his 1997 book “The Innovator’s Dilemma.” Nearly 20 years later “Disrupt!” is a popular leadership mantra that is more frequently uttered than experienced. You can't productize it. You can't always control it – at least what effects it has in practice. You aren't necessarily going to like every product of innovation. So are you sure you want it? If so, how do you promote a culture in which innovation can flower – and, potentially, thrive? Because that's probably the best that you can do.
Perhaps there's a better framing for innovation than just "disruption.“ This session is an overview of commmoditization and innovation theories followed by basic things you can do to apply that theory to your daily job architecting, choosing and managing a data environment in your company.

Published in: Technology

Disruptive Innovation: how do you use these theories to manage your IT?

  1. 1. Disruptive Innovation: Past, Present, Future (how to use these theories to manage your IT) February, 2016 Mark Madsen - @markmadsen -
  2. 2. © Third Nature Innovation: The Cargo Cult of Management Consulting
  3. 3. © Third Nature Max Gurvitz
  4. 4. © Third Nature The go-to innovation company is Google You can’t get fired for doing what everyone else did (but you can get fired for not getting the results they did)
  5. 5. © Third Nature If you do what google did you could: Make a data center out of shipping containers. ▪ That didn’t work out. Build your own servers. ▪ Made out of “razor blades and hate” ▪ Start fires in the data center Build your own environmental cooling data center ▪ Generating fog and rain inside the data center As Dan Luu noted, at least copy current engineering practices rather than things done in 1999. By the way, are you using MapReduce?
  6. 6. © Third Nature Saying there’s a process you can follow to be innovative is like saying there’s a recipe that will make you a chef.
  7. 7. © Third Nature You keep using that word. I do not think it means what you think it means. Innovation?
  8. 8. © Third Nature Innovation is not “add new features”
  9. 9. © Third Nature “Better experiences, not more features.” Roland Rust “When technology delivers basic needs, user experience dominates” Don Norman
  10. 10. © Third Nature Value is not in the product, it’s in the practice Innovation is not a characteristic of things
  11. 11. © Third Nature Innovation is change. Change is often not appreciated.
  12. 12. © Third Nature Paradox: Innovation becomes best practice Innovation isn’t reproducible. Only the conditions that permit it are
  14. 14. © Third Nature Commoditization of Computing Technology is the Driver “There is no reason anyone would want a computer in their home.” Ken Olson, CEO of DEC, 1977 “…by 2008 we will be producing one billion transistors for every man, woman and child on earth” Semiconductor Industry Association, 2007
  15. 15. © Third Nature How significant is the computing improvement? 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 1010 10 9 10 8 107 106 105 104 103 102 101 10 10-1 01-2 10-3 10-4 10-5 10-6 Calculationspersecondper$1000 Data: Ray Kurzweil, 2001 10,000 X improvement DW architecture and methods start here in the mid 80s Term “BI” coined Mechanical Relay Vacuum tube Transistor Integrated circuit
  16. 16. © Third Nature Don’t worry about performance, just buy more hardware 10,000 X performance improvement, soon 100K
  17. 17. © Third Nature There are always limits “If the automobile had followed the same development as the computer, a Rolls-Royce would today cost $100, get a million miles per gallon, and explode once a year killing everyone inside.” Robert Cringely Time Anything Reality
  18. 18. © Third Nature RIP Moore’s Law. Data is growing faster than compute. This forces an architectural shift.
  19. 19. © Third Nature We’ve reached another generational technology shift 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 1010 10 9 10 8 107 106 105 104 103 102 101 10 10-1 01-2 10-3 10-4 10-5 10-6 Calculationspersecondper$1000 Mechanical Relay Vacuum tube Transistor Integrated circuit Data: Ray Kurzweil, 2001 Multicore and networked parallelism is the next wave
  20. 20. © Third Nature What’s different? Parallelism We’re not getting more CPU power, but more CPUs. There are too many CPUs relative to other resources, creating an imbalance in hardware platforms. Most software is designed for a single worker, not high degrees of parallelism and won’t scale well.
  21. 21. © Third Nature Reality: you must assume distributed architecture Why by default? Because the upgrade between single node and distributed is a major change to designs. It carries new component linkages and complexities. A new ecosystem. The future holds cloud provisioning, software-defined environments and a lot less single-server provisioning. Slide 21Copyright Third Nature, Inc. (a) Scaling up with a larger server (b) Scaling out with many small servers (aka “++ungood)” (aka “the future”) The future is already here, it just isn’t evenly distributed yet.
  22. 22. © Third Nature An important cloud computing benefit Scalability is free (if you have the right software) If your task requires 10 units of work, you can decide when you want the results: 10 servers, 1 unit of time Cost is the same. Not true of the conventional IT model Time 1 server, 10 units of time X X
  23. 23. © Third Nature We are in a transitional phase in IT architecture Then State of Practice Now, forward Architecture Timeshare Client/server Cloud Data Core TXs All TXs, some events, docs All data Rate of change Slow Rapid Continuous Uses Few Many Everything Latency Daily+++ < daily to minutes Immediate Data platform Uniprocessor SMP, cluster Shared nothing
  24. 24. © Third Nature Majority use of computing over time 1930s-1950s: Calculate 1960s-1980s: Automate 1990s-2010s: Informate 2010s+: Analyze and Actuate Computing technology has become a tool of observation Risingorganizationalcomplexity
  25. 25. © Third Nature Evolution of data 50s-60s: data as product 70s-80s: data as byproduct 90s-00s: data as asset 2010s +: data as substrate The real data revolution is in business structure and processes and how they use information.
  26. 26. © Third Nature Not hype: another round of infrastructure change Mainframe  c/s  cloud Batch  online  event driven Infrastructure takes a long time. Value is driven by new capabilities used to do new things, less by doing old things better or cheaper
  27. 27. © Third Nature Disruption Time The Internet forced a new architectural evolution. IT has had a hard time keeping up, and new entrants in many markets are taking advantage of the new architecture to change how IT work is done. Any time you have a backlog of resisted innovations, the pressure will eventually force wholesale change.
  28. 28. © Third Nature UNDERSTANDING INNOVATION AND COMMODITIZATION PROCESSES* This is how things change. An amalgam of Everett Rogers, Yochai Benkler, Geoffrey Moore, Clayton Christensen, Stephen Gould, Eric von Hippel and others. aka I like S curves *the very short version
  29. 29. © Third Nature COMMODITIZATION
  30. 30. © Third Nature Four phases of technology commoditization Early adopters show results Market growth Innovation Development Maturation Saturation Time Early mainstream starts to pay attention Mainstream buy-in This axis could be considered market penetration, adoption, product maturity Invention / discovery
  31. 31. © Third Nature Time Characteristics of software as it evolves Innovation Unique Custom built High value High cost Differentiator Not well understood High rate of change Few vendors
  32. 32. © Third Nature Time Oddity  known Focus on integration not building, customizable products High value Lowered costs or cost/speed/fit tradeoffs Point of competition Better understood Slowed rate of change Growing then shrinking vendor count Characteristics of software as it evolves Maturation
  33. 33. © Third Nature Time Ubiquitous Configurable product High to low value* Low cost Barrier to entry Purpose and limitations well understood Negligible rate of change Few, large vendors Characteristics of software as it evolves Saturation
  34. 34. © Third Nature Time Compete on differentiating value Vendor strategies (in general) vary by phase Maturation Compete on product & features Compete on process SaturationInnovation Market growth
  35. 35. © Third Nature Should you be a first mover or fast follower? Time Little product substitution is possible here. Few competitive bids or RFPs. Maturation Uncertain tradeoffs here. Competitive bids for unlike products. Early it’s less “what feature” and more “how to accomplish my task”, later it’s the opposite. Predictable cost and feature comparison until practices change. That change can take a long time to occur. SaturationInnovation Market growth
  36. 36. © Third Nature Time Few product choices The vendor landscape changes over time Many, expanding product choices Many, contracting product choices Relatively few product choices Market growth A particularly dangerous time to pick vendors
  37. 37. © Third Nature We see this pattern in evolutionary processes Saturation,competition,env.constraints Evolution in most complex systems goes through periods of rapid change followed by periods of general stability, referred to as “punctuated equilibrium” Invertebrates Vertebrates Bacteria Insects Innovation – Adaptive radiation – Selection – Convergence 3.5b Bacteria (Cell) 2.5b Sponge (Body) 0.7b Clams (Nerves) 0.5b Trilobites (Brains) 0.1b Mammals Timing
  38. 38. © Third Nature The same model can be applied to technology Saturation,competition,env.constraints Copyright Third Nature, Inc. Innovation – Adaptive radiation – Selection – Convergence
  39. 39. © Third Nature With a long view a pattern emerges Evolution in most complex systems goes through periods of rapid change followed by periods of general stability, referred to as “punctuated equilibrium” New technologies take the place of old, establishing new ecosystems which are in turn disrupted by newer technologies and ecosystems. Chaotic Stable Chaotic Stable Chaotic Time
  40. 40. © Third Nature Activities, products and practices evolve over time Source: Simon Wardley
  41. 41. © Third Nature Technology doesn’t just fulfill a need. It generates new needs and new problems. Business practices and technology co-evolve.
  42. 42. © Third Nature As practices evolve based on new capabilities… A new level of complexity develops over top of the older, now better understood processes, leading to new needs.
  43. 43. © Third Nature Evolution and the Salaman-Story Paradox Source: Simon Wardley
  44. 44. © Third Nature Evolution and the Salaman-Story Paradox ”Survival requires efficient exploration of current competencies and ‘coherence, coordination and stability’; whereas innovation requires discovery and development of new competencies and this requires the loosening and replacement of these erstwhile virtues” Source: Simon Wardley
  45. 45. © Third Nature As a technology moves from emerging to commodity the nature of acquiring, using and managing it should change Generate options Innovation Novel practice Maximize value Maturation Standardize / minimize choice Acquisition Best practice Minimize costs SaturationInnovation e.g. BI which went from many tools to a few vendors, now being disrupted by new technologies and capabilities Constrain choices Adaptation Good practice Optimize
  46. 46. © Third Nature In terms of technology, we are in a tough position because the ecosystem is in a disjoint state Maturation SaturationInnovation Big data and analytics is here BI / DW is here
  48. 48. The Enterprise IT Adoption Cycle Wardley IT adoption reality Adoption cycle graphic © 2012 Simon Wardley and CC BY-SA 3.0 ****
  49. 49. © Third Nature The Incredible Rate of Technology Change Big data? OMG!
  50. 50. © Third Nature The Incredible Rate of Technology Change We told you about it in 2004…
  51. 51. © Third Nature Time Adoption Rate Some Innovation Adoption Theory End of LifeNew innovation Time Adoption Rate End of LifeNew innovation
  52. 52. © Third Nature Adopter Categories Innovators Late Majority Early Majority Early Adopters Laggards
  53. 53. © Third Nature Ability to adopt is governed by people & organizations Innovators Late Majority Early Majority Early Adopters Late adopters People here tend to view a technology as a means to capitalize on future opportunities* e.g. big new projects, process change People here tend to view technology as a means to resolve present problems. e.g. more focused projects, process improvement Copyright Third Nature, Inc. *adopter status is based on the person/org and a given technology, it’s not a blanket statement
  54. 54. © Third Nature Slowing it down: innovation is gated by ability to adopt No technology stands entirely alone – these dependencies slow adoption, stretching the maturation phase. This ecosystem effect is what creates technology regimes that can last decades. Copyright Third Nature, Inc. Younger companies have a relative advantage when it comes to absorbing new infrastructure.
  55. 55. © Third Nature Time Cumulative Adoption Market Adoption Hard work Tipping point
  56. 56. © Third Nature Product Maturity Some Ideas Aren’t That Good End of LifeTimeNew innovation Some ideas aren’t that good, like object databases in the 1990s
  57. 57. © Third Nature These Curves Can Explain a Lot Time Product Maturity Analyst revenue predictions Executive interest“Gartner Gap”
  58. 58. © Third Nature The “experts” often have a foreshortened view “Open source is not worth paying attention to.” A Gartner analyst talking about the database and analytics market, January, 2006. Where the analysts are on the adoption curve
  59. 59. © Third Nature Crossing the Chasm (1991)
  60. 60. © Third Nature Geoffrey Moore’s Ideas Built on Rogers’ ideas, extended them to tech marketing and product management. The original focus was on the development of technology (gray). Just say no Stick with the proven Stick with the herd Stay ahead of the herd Just try it
  61. 61. © Third Nature Core BI / DW technology is mainstream-stable The data management market has many segments, some new, some mature, some being rejuvenated. Platforms (this should scare everyone) Databases* Reporting & ETL and DI Analytics
  62. 62. © Third Nature Product evolution in software markets PC 1 2 3 4 5 6 Image: Geoffrey Moore, Dealing With Darwin”
  63. 63. © Third Nature INNOVATION
  64. 64. © Third Nature Innovation and Commoditization This section isn’t really a summary
  65. 65. © Third Nature Image: Harvard Business Review, “Skate to Where the Money Will Be” Theory of Disruptive Innovation i.e. you don’t pay attention and do what you always did and the other guy eats your market from below
  66. 66. © Third Nature Disproving Christensen a) 9% of the cases fit the model b) Disruptive innovation <> success; banks disruptively innovated debt products and we know how that turned out c) The model fails to predict failure too: In 2007, Christensen told Business Week that “the prediction of the theory would be that Apple won’t succeed with the iPhone,” adding, “History speaks pretty loudly on that.” In its first five years, the iPhone generated a hundred and fifty billion dollars of revenue. In the preface to the 2011 edition of “The Innovator’s Dilemma,” Christensen reports that, since the book’s publication, in 1997, “the theory of disruption continues to yield predictions that are quite accurate.” d) Oh
  67. 67. © Third Nature Types of innovation Incremental or “sustaining” ▪ Incremental is based on existing concepts, framing; smaller changes within the same general framework Disruptive ▪ Based on new concepts, science, principles; requires new knowledge, skills; over time has significant consequences to market Architectural – the third path ▪ Changes how the parts are related. It devalues advantage of experience, knowledge, usefulness of prior knowledge, but doesn’t affect the existing knowledge. (Christensen missed this one)
  68. 68. © Third Nature Adoption and decline – everything gets old For most businesses, nearly 80% of IT budget is dedicated to basic infrastructure …and more than 60% of IT labor cost goes to keep things running, i.e. basic operations and support. Strategic Commodity
  69. 69. © Third Nature It Wasn’t Always This Way As technologies mature and spread to competitors, they cease to be differentiators. Unfortunately, this is what packaged software vendors do to your “best practice.” CommodityCommodity The old advantages becomes the new focus of cost reduction. For example, your data warehouse. Strategic Strategic
  70. 70. © Third Nature Adoption and decline Rarely does anyone talk about the core problem: preexisting conditions You have something new. How does it affect the old? ▪ Replaces it? ▪ Adds something new? ▪ Overlaps it, forcing you to make hard decisions about what parts to keep, change, throw away? The heart of this problem is the process of architecture: integrating changes to systems over time. The integration is not purely technical, it’s practices of use, operation, deployment.
  71. 71. © Third Nature Most data tech is a commodity, a cost of doing business
  72. 72. © Third Nature Adopting new things: there’s a problem with your budget
  73. 73. © Third Nature How IT strategies evolved with commoditization Time Equipment Expensive: outsource to reduce equipment cost Labor Affordable: insource for control, innovation Dirt cheap: outsource to reduce labor cost 76
  74. 74. © Third Nature The cost flip in the business intelligence world Cost factors traded positions 1990 - 2010 Equipment Software 77 Cost Labor For small to mid-sized organizations it’s very affordable
  75. 75. © Third Nature TCO and BI What can you control? ▪ Labor effort is almost identical across BI products. ▪ Hardware use by BI tools is similar across products. ▪ You can negotiate the software costs. 3 Year BI TCO Cost Categories Source: Third Nature Open Source cost study
  76. 76. © Third Nature BI Market: Cost is normally driven out by commodities, not increased 79
  77. 77. © Third Nature This is an old problem BI tools are better, but the model being applied in most organizations is not different from the past.
  78. 78. Slide 81November 2010 Mark Madsen If BI is a commodity, why does it cost so much? Processes Applications Data Integration Storage EDM / BRM Delivery Consumers Purchasing Distribution Manufacturing Sales & Service ERP Data warehouse ODS Stream db / cache Content store Identify Analyze Debt<10% of Income Debt=0% Good Credit Risks Bad Credit Risks Good Credit Risks Yes YesYes NO NONO Income>$40K Predict Batch ETL EII SCM SFA CRM ESB EDR Monitor Explore Data mart Low-lat ETL BPM/Workflow BRE CEP Prescribe Data services Transaction services Manual feedback Automated feedback
  79. 79. © Third Nature Lessons to take from this 1. Business intelligence is still expensive for many organizations, with the largest proportion of cost being labor. 2. Business intelligences is not a technology problem, or the failure rate and costs wouldn’t be so high. 3. BI tools being a commodity does not make BI a commodity. 4. Architecture has an outsized impact on your ability to adopt and adapt. 5. What you remove is as important as what you add. 82
  81. 81. © Third Nature Questions to ask Why innovate? ▪ Usual answer: profit ▪ Proper answer: solve a problem Innovation for what? ▪ A product or service you are selling to customers ▪ Internal products and services, how you run your business or department
  82. 82. © Third Nature Reinforcing relationships resist change, despite radical technology and practice shifts Note how only one third is tech Architectural Regime MethodologyTechnology Organization Organization defines where the work is done and the roles. Technology defines what work can be done in a given area. Methodology defines how work is done and what that work is. Slide 85
  83. 83. © Third Nature Designing for data: monolithic vendor technology- based classifications of the ecosystem won’t help These types of eye charts provide a categorization of what’s available, not what you need. They ignore the contexts of use that are most important. 86
  84. 84. © Third Nature It’s tough making it decisions in a turbulent market Maturation SaturationInnovation If you’re here you probably don’t want to be making long term technology or vendor commitments.
  85. 85. © Third Nature Today: repeating the experience of the 80s & 90s This is the turbulent phase of the market as it goes through rapid development, then product and service changes. Copyright Third Nature, Inc. The Internet combined with commodity computing is forcing a new architectural evolution, already well underway. Maturation SaturationInnovation
  86. 86. © Third Nature Time Rule of thumb: when a product is in phase… Maturation SaturationInnovation Market growth Build Integrate Buy
  87. 87. © Third Nature Methods change too, one size doesn’t fit all Maturation SaturationInnovation Agile & exploratory methods 6 Sigma & efficiency methods
  88. 88. © Third Nature How procurement decisions are made Deliberation ▪ Actions are consciously chosen. Don’t attribute to malice what you can attribute to stupidity, and don’t attribute to stupidity what you can attribute to laziness. Rationality ▪ People make logical decisions. Sure they do. Order ▪ System are understandable and the results of actions predictable.
  89. 89. © Third Nature 90% of EVERYTHING is crap “Sturgeon’s Revelation”
  90. 90. © Third Nature “Choose Boring Technology” You only get so many chances to make big changes at a company. Don’t waste them. You can spend X time focusing on the goal and worry less about the known tech, or you can spend X time learning the new tech and less time focusing on the goal. The important thing is not the choice of tech, it’s knowing when the time is right to make a new tech choice.
  91. 91. © Third Nature Beware unintended consequencesUnintended consequences
  92. 92. © Third Nature In other words… Software is like puppies. Getting a puppy is easy, raising one is hard. “The short term benefits of using a new [type of] database exceed the long term cost of operating it.” Dan Mckinley
  93. 93. © Third Nature Where does innovation come from? “It has long been assumed that product innovations are typically developed by product manufacturers. …it now appears that this basic assumption is often wrong.” Eric von Hippel
  94. 94. © Third Nature How to find “innovative” solutions N.W.A. Answer: steal them from somewhere else.
  95. 95. © Third Nature Being innovative and culture Myth of process – there is no “process for innovation”, only principles and exceptions e.g. “It’s best to work in small teams, keep them crowded and foster serendipitious connections.” – Eric Schmidt It depends on creative problem solving, and solving problems people care about. Removing deviance removes change, so you have to be careful about best practices.
  96. 96. © Third Nature No silver bullet It’s culture-dependent, and creative and messy and idiosyncratic and slow and hard, no process, just survival bias and heuristics and principles.
  97. 97. © Third Nature “The future, according to some scientists, will be exactly like the past, only far more expensive.” ~ John Sladek
  98. 98. © Third Nature Further Reading Further Reading: Manager’s Theories About Innovation, Salaman & Storey, 2002 Democratizing Innovation, Eric von Hippel, Sources of Innovation, Eric von Hippel, The Wealth of Networks, Yocahi Benkler An introduction to value chain mapping, introduction-to-wardley-value-chain.html The diffusion of infrastructure dependent technologies: A simple model http://www.dime- Architectural innovation: the reconfiguration of existing product technologies and the failure of established firms What the Gospel of Innovation Gets Wrong How Useful Is the Theory of Disruptive Innovation? useful-is-the-theory-of-disruptive-innovation/ Slide 101
  99. 99. © Third Nature Image Attributions Thanks to the people who supplied the images used in this presentation: indonesian angry mask phone - Erik De Castro Reuters.jpg egg_face1.jpg - chicken_head2.jpg - snail1.jpg - wheat_field.jpg -
  100. 100. © Third Nature About Third Nature Third Nature is a consulting and advisory firm focused on new and emerging technology and practices in information architecture, analytics, business intelligence and data management. If your question is related to data, analytics, information strategy and technology infrastructure then you‘re at the right place. Our goal is to help organizations solve problems using data. We offer education, consulting and research services to support business and IT organizations as well as technology vendors. We specialize in information strategy and architecture, so we look at emerging technologies and markets, evaluating how technologies are applied to solve problems.
  101. 101. © Third Nature About the Presenter Mark Madsen is president of Third Nature, a technology research and consulting firm focused on business intelligence, analytics and performance management. Mark is an award-winning author, architect and former CTO whose work has been featured in numerous industry publications. During his career Mark received awards from the American Productivity & Quality Center, TDWI, Computerworld and the Smithsonian Institute. He is an international speaker, contributing editor at Intelligent Enterprise, and manages the open source channel at the Business Intelligence Network. For more information or to contact Mark, visit