Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

IBM iForum - Rethinking Innovation Measurement by Dean Spitzer


Published on

Slide deck courtesy from Dean Spitzer on recent IBM's iForum event in Zurich about "Capitalising on Innovation" talking around the subject of improving performance measurement of innovation and social computing altogether.

Published in: Business, Technology
  • Dean, thanks a lot for your experience to be a guiding force the time to come
    Are you sure you want to  Yes  No
    Your message goes here
  • I'm glad that so many people have found this presentation valuable. If anyone in interested in this slide deck or other materials on measuring innovation, feel free to let me know. My e-mail address is:

    I believe that measuring innovation is a much neglected area and is one of the keys to unlocking innovation potential in organizations.
    Are you sure you want to  Yes  No
    Your message goes here
  • Excellent ideas and very good way for communicating them
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi Larry, thanks for the heads up and for the feedback comments! Glad you enjoyed the slides; unfortunately, I no longer have them in my hands and would not be able to share a copy of them, since they are now in the possession of the original author. Apologies for the inconvenience.

    Are you sure you want to  Yes  No
    Your message goes here
  • This is a very good presentation.
    Wish I could down load it or freeze it on some slide for my reading speed.
    Larry 2010 Aug
    Are you sure you want to  Yes  No
    Your message goes here

IBM iForum - Rethinking Innovation Measurement by Dean Spitzer

  1. 1. Rethinking the Measurement of Innovation Dr. Dean Spitzer Performance Measurement Thought Leader IBM Research Presentation at the IBM Innovation Forum Zurich, November 27, 2007
  2. 2. WARNING: There is no “magic metric” for innovation
  3. 3. Pre-assessment: How well is your organization currently measuring innovation? <ul><li>Does your organization have a clear operational definition of ‘innovation’? </li></ul><ul><li>How much holistic visibility is there around innovation in your organization? </li></ul><ul><li>Is your organization really measuring “innovation that matters”? </li></ul><ul><li>Is your organization increasing its understanding of innovation? </li></ul><ul><li>How much has innovation measurement in your organization has changed and been improved in the past few years? </li></ul><ul><li>What proportion of your innovation measures are ‘leading’ versus ‘lagging’ indicators? </li></ul><ul><li>How well is your organization measuring and managing ideas and creativity? </li></ul><ul><li>How useful for managing innovation is the data you are currently collecting? </li></ul><ul><li>How confident are you that your current innovation measurement system will help you drive and sustain real innovation today and in the future? </li></ul><ul><li>Does your innovation measurement system help stakeholders collaborate to co-create optimal value? </li></ul>On a scale of 0 percent to 100 percent, to what extent do you think your organization measures and manages its entire portfolio of innovation assets?
  4. 4. Innovation is too important not to be measured well
  5. 5. What experts are saying about the measurement of innovation <ul><li>“ Measurement is both fundamental and critical to success with innovation.” [Davila, Epstein, Shelton, Making Innovation Work , Wharton, 2005] </li></ul><ul><li>“In a recent survey… more than half rated their performance measurement system for innovation as poor or less than adequate .” [ Making Innovation Work ] </li></ul><ul><li>“ Few companies actually measure their innovation efforts , and those that do tend to look at a handful of financial measures at best.” [Kuczmarski, Middlebrooks & Swaddling, Innovating the Corporation ] </li></ul><ul><li>“ Innovation is widely undermeasured , and few firms—even those that attempt to track innovation rigorously—are confident that they’re doing it right.” [Boston Consulting Group, Measuring Innovation 2006 ] </li></ul><ul><li>“ The potential for most companies to improve their measurement practices—and as a result, boost their return on innovation spending—is sizable.” [Boston Consulting Group, Measuring Innovation 2006 ] </li></ul><ul><li>“ Measurement is critical if a company hopes to improve its innovation effectiveness. Companies can’t improve performance if they don’t measure it.” [ Innovating the Corporation ] </li></ul><ul><li>“ Measurement is critically important to a healthy innovation system.” [ Innovating the Corporation ] </li></ul><ul><li>“Without question, innovation is being asked to play a greater role in the CEO’s organic growth strategy. Yet to date, innovation metrics have been vague at best , and more often elusive.” [Pat LaPointe] </li></ul>
  6. 6. Few understand just how powerful measurement is! <ul><li>&quot;Measurement always improves performance.&quot; [Buckingham & Coffman, First, Break All the Rules , p. 236] </li></ul><ul><li>&quot;Good data, properly distributed, transform organizations. &quot; [Whitely, The Customer-Driven Company , p. 175] </li></ul><ul><li>&quot;Measures have great power, almost like genetic code, to shape action and performance...Change the measures, and you change the organism.&quot; [ Epstein & Birchard, Counting What Counts , p. 145] </li></ul><ul><li>&quot;Most often when we see illogical behavior, the fault is in the measurement system, not in the employees.&quot; [Brian Joiner, Fourth Generation Management , p. 242] </li></ul><ul><li>&quot;Changing the way we measure changes everything.&quot; [Meador, The Dance of Change , p. 299] </li></ul><ul><li>&quot;An organization's measurement system strongly affects the behavior of people both inside and outside the organization.&quot; [Kaplan & Norton, The Balanced Scorecard , p. 21] </li></ul><ul><li>&quot;The essence of a corporate culture is the firm's measurement system.&quot; [Strassman, The Business Value of Computers , p. 73] </li></ul><ul><li>&quot;The mere action of defining measures of success will change behavior positively or otherwise.&quot; [Thorp, The Information Paradox , p. 164] </li></ul><ul><li>&quot;Metrics are to a business what the five senses are to humans - systems of feedback that improve our capacity to adapt and excel over the long run. &quot; [Tachi Kiuchi, &quot;What We Learned in the Rainforest,&quot; Barrett-Koehler, 2002, pp. 152-153] </li></ul>“ One cannot reliably manage what one cannot measure. This is not my axiom. This is an unimpeachable law of nature. It cannot be broken…ever.” J.B. Thoreson Employees will do what you ‘inspect,’ not necessarily what you ‘expect.’
  7. 7. Measurement is ubiquitous and it is a mess! In The Agenda , Michael Hammer puts it this way: “ A company's measurement systems typically deliver a blizzard of nearly meaningless data that quantifies practically everything in sight, no matter how unimportant; that is devoid of any particular rhyme or reason; that is so voluminous as to be unusable; that is delivered so late as to be virtually useless; and that then languishes in printouts and briefing books, without being put to any significant purpose.... In short, measurement is a mess.&quot; And even fewer realize how broken performance measurement is! 2 3 4 5 6 7 8 9 10
  8. 8. Management is based on measurement, and all other organizational systems are dependent on the measurement system No organization can be any better than its measurement system The importance of performance measurement, as the foundational management system, is not widely appreciated. And, it is ironic that the measurement of innovation is one of the least effective and innovative forms of measurement.
  9. 9. One of the biggest problems in organizations is lack of alignment among ‘measurement systems’ Organizations with poor performance measurement will be poorly aligned, with functions pursuing their own self-interest and often working at cross-purposes Supply Chain Information Technology Not only don’t most organizations have a single integrated measurement system, but the disparate measurement systems don’t even communicate! Budgeting Sales Operations Research & Development Human Resources Marketing Finance Learning
  10. 10. Most organizations are dissatisfied with their performance measurement
  11. 11. Measurement is the elephant in the room The innovation measurement paradox: While everyone knows it’s inhibiting progress, few want to deal with it!
  12. 12. That’s why I wrote my latest book, “Transforming Performance Measurement” <ul><li>&quot;This is one of the best books on performance measurement that I have ever read ! If you only read one book on performance measurement this year, make it this one.&quot; – Andy Neely, Professor and Chair, Centre for Business Performance, Cranfield University School of Management, and author of The Performance Prism </li></ul><ul><li>&quot; Dean Spitzer takes the mystery out of performance measurement with this practical guide to how doing measurement right can transform any organization. Dean's insight into transformational measurement has been extremely helpful at Wachovia as we transform our learning measurement to better assist our businesses in achieving their goals.&quot; – Dean Williams, Chief Learning Officer, Wachovia Corporation </li></ul><ul><li>&quot; Dean Spitzer’s concept of transformational measures will radically change the way individuals and organizations view performance measurement .” – Gregg Mauro, Executive Director, Government Executives Network </li></ul>
  13. 13. Performance measurement needs to be transformed Changing measurement is one of the highest leverage things any organization can do.
  14. 14. How people “experience” measurement is the key to how they will respond to it Context
  15. 15. Measurement is typically used to monitor and control, which does not encourage innovation Context Too often evaluation and its consequences undermine learning from measurement.
  16. 16. So much measurement is being used to report on the past and for self-serving purposes Context “ There are too many patents in the world today that have been filed simply because it made somebody look good, but never really presented any value opportunities.” [Goran Roos] Measurement done to prove will rarely improve !
  17. 17. Context The most important aspects of measurement are the social and organizational, not the technical, factors. Almost nothing has been done to address the “context of measurement” Context
  18. 18. The biggest problem with the context of measurement is the lack of Measurement Leadership <ul><li>While most managers publicly extol the value of measurement, few actually use it systematically and well. </li></ul><ul><li>Measurement is everybody’s job and therefore nobody’s job! </li></ul><ul><li>Organizational leaders are more than happy to delegate measurement to &quot;measurement specialists.&quot; They don't realize how strategic measurement is, and how much management attention it requires to do it right. Measurement is one of the most under-appreciated organizational activities, and measurement leadership is one of the least appreciated leadership roles. </li></ul>Context Who is leading Innovation Measurement in your organization?
  19. 19. Measures are the lens through which most performance is viewed Focus Once measures have been selected, they color what we see and how we act.
  20. 20. The measures you select will determine what you ‘see’ and can greatly bias the perceptions of the entire organization! Focus
  21. 21. Most organizations are full of “routine measures” that have been accepted as “standard operating metrics” and rarely, if ever, change Focus Performance measures are viewed as hard-wired into the organization. Unfortunately most organizations tend to default to easy-to-measure things, for which data is readily available. Routine Measures
  22. 22. Is your organization’s measurement system differentiating? <ul><li>Unfortunately, most leaders don't differentiate between the critical few measures that will have the greatest impact and the hundreds, or thousands, of other measures -- the trivial many -- that permeate every area of their organizations. </li></ul><ul><li>Too many executives and other managers think it is sufficient just to track generic or standard industry-approved measures. These are what I call &quot;routine measures&quot; and they are satisfactory for maintaining the status quo , but not for taking the organization to the next level. </li></ul><ul><li>If you measure what everyone else is measuring, your business will be well on the way to commoditization. However, if your organization strives for more than that, you might want to consider rethinking the measures that are currently populating your measurement system </li></ul><ul><li>In today's highly competitive and increasingly services-oriented marketplace, it is vital to be measuring more than just the same old, routine measures. Not only is almost all measurement routine, but almost all of it has derived from a commodity manufacturing mindset. It is amazing how few new measures have been developed since the industrial revolution! </li></ul>Focus
  23. 23. Traditional innovation measurement and management has resulted in enormous waste 4% hit rate 96% of innovation projects fail Focus
  24. 24. What does a rear view mirror tell us about innovation? Focus The balance sheet is supposed to describe value, but it ignores almost all intangible sources of future value.
  25. 25. Most innovation measurement systems don’t encourage innovative thinking Focus Traditional metrics tend to reinforce the status quo. Old performance measures are a box that constrains performance. Traditional Metrics
  26. 26. There is a tendency to measure what is easiest and safest to measure, not necessarily what is most important Focus “ Too often measurement systems are driven by what is available rather than what is needed.” [KPMG, The Search for New Insights, 2001]
  27. 27. Measures of innovation are among the least innovative measures “ Most measurements of performance are geared to the need of the 20th-century manufacturing companies. Times have changed. Metrics must change as well.” [Lowell L. Bryan, “The new metrics of corporate performance,” McKinsey Quarterly , 2007] What we measure is what we get. If we keep measuring the same things, we will get more of the same. Focus Although organizations have many creative people involved in innovation, almost no creative people are working on improving the innovation measurement system! “ You can’t Six Sigma your way to innovation.” [Russ Nussbaum]
  28. 28. Measures need to change Focus “ If we measure the new with the tools of the old, we will not ‘see’ the new.” [Karl-Erik Sveiby]
  29. 29. Innovative organizations must not be afraid to experiment with measurement Focus “ To change our perspective, we must change our measures.” [ Source: D.R. Spitzer , Transforming Performance Measurement ]
  30. 30. Emergent measures often require qualitative measurement, estimating, and subjectivity Focus &quot;A high barrier stands between us and the habit of making rough estimates -- the fear of getting the 'wrong' answer. There is nothing wrong with educated guesses as long as the uncertainty is acknowledged and managed. Contrary to what most of us have learned in school … an inexact answer is almost always good enough. “ [Arno Penzias] “ Measurement is assessing the degree to which a variable is present … .Notice there is no reference to numbers in that definition. ” [D. Lynn Kelley] Extremely Honest Not Honest At All 1 2 3 4 5 6 7 8 9 Extremely Honest Not Honest At All 1 2 3 4 5 6 7 8 9 Extremely Honest Not Honest At All 1 2 3 4 5 6 7 8 9 Extremely Honest Not Honest At All 1 2 3 4 5 6 7 8 9 Everything can be measured in some way. Imperfect measurement is better than none, as long as we acknowledge the ‘confidence level.’ In measuring intangibles, we need to improve our ability to estimate.
  31. 31. Developing emergent measures is an iterative process Focus One of the keys to emergent measurement is the “socialization process” that occurs as the construct and its measurement are discussed.
  32. 32. Sample Emergent Measures: <ul><li>Innovative Climate </li></ul><ul><li>Innovation Talent </li></ul><ul><li>Employee Engagement in Innovation </li></ul><ul><li>Innovation Network Strength </li></ul><ul><li>Innovation Knowledge Flow </li></ul><ul><li>Learning from Failure </li></ul><ul><li>Learning Speed </li></ul><ul><li>Innovation Agility </li></ul><ul><li>Innovation Collaboration </li></ul><ul><li>Reputation for Innovation </li></ul><ul><li>Innovation Partner Relationships </li></ul><ul><li>Organizational Trust </li></ul><ul><li>Innovation Leadership </li></ul><ul><li>Strategic Alignment of Innovation </li></ul><ul><li>IP Strength </li></ul><ul><li>Client Success from Innovation </li></ul><ul><li>Innovation Impact </li></ul><ul><li>Innovation Intensity </li></ul><ul><li>Innovation Freshness </li></ul><ul><li>Idea Pipeline </li></ul><ul><li>Innovation Pipeline Quality </li></ul><ul><li>Innovation Velocity </li></ul><ul><li>Innovation Value-Add Time </li></ul><ul><li>Innovation Process Maturity </li></ul>Focus “… the most predictive measure of whether an organization will be innovative is the level of trust between people in the organization.” [Dauphinais, Means & Price, The Wisdom of the CEO ]
  33. 33. <ul><li>Characteristics </li></ul><ul><li>Commitment: Commitment to organizational goals and operations. Work perceived as stimulating and engaging. </li></ul><ul><li>Freedom: Opportunities to make own decisions, seek information, and show initiative. Freedom from tight supervision. </li></ul><ul><li>Idea-support: People encouraged for putting forward ideas and suggesting improvements. </li></ul><ul><li>Positive Relationships: People trust and get on well with one another in a climate. Absence of personal conflicts. </li></ul><ul><li>Dynamism: Dynamic and exciting atmosphere. </li></ul><ul><li>Playfulness: People laugh and joke with one another. </li></ul><ul><li>Idea-proliferation: People are perceived as having creative ideas and varied perspectives towards their work. </li></ul><ul><li>Stress: People generally feel overburdened and under pressure at work. </li></ul><ul><li>Risk-taking: New ideas may be implemented; people prepared to take risks. </li></ul><ul><li>Idea-time: People have the time to generate and consider new ideas. </li></ul><ul><li>Shared View: There are open and adequate communications between more and less senior employees. </li></ul><ul><li>Pay Recognition: People are satisfied with their remuneration. </li></ul><ul><li>Work Recognition: People receive praise for their achievements. </li></ul>Emergent Measure: Innovation Climate Questionnaire Source: Innovation Centre Europe
  34. 34. <ul><li>Executives model innovative behavior </li></ul><ul><li>I am encouraged to use my creativity </li></ul><ul><li>I am personally excited about innovation </li></ul><ul><li>Innovation is a high priority in this organization </li></ul><ul><li>My supervisor talks with me about innovation </li></ul><ul><li>Managers sponsor innovation </li></ul><ul><li>Employees feel empowered to be innovative </li></ul><ul><li>Cross-functional teams are empowered to be innovative </li></ul><ul><li>Diversity of ideas is encouraged </li></ul><ul><li>The organization has a well-articulated innovation strategy </li></ul><ul><li>There is a sense of urgency about innovation </li></ul><ul><li>The customer is the focus of innovation </li></ul><ul><li>I take advantage of opportunities to network with other innovative people </li></ul><ul><li>I regularly access resources (expertise, knowledge, education, etc.) that help me become more innovative </li></ul><ul><li>Creative ideas are responded to promptly </li></ul><ul><li>Employees feel empowered to take risks </li></ul><ul><li>There is appropriate tolerance for failure </li></ul><ul><li>Learning from mistakes is a formal part of work </li></ul><ul><li>Developing employees’ innovation-related knowledge and skills are a priority in this organization </li></ul><ul><li>Employees understand how innovation is measured </li></ul><ul><li>Creativity is recognized </li></ul><ul><li>Time is provided for creative efforts </li></ul><ul><li>I am involved in at least one innovative project </li></ul>Emergent Measure: Employee Engagement in Innovation Scale: (4) To a great extent. (3) To some extent. (2) To a little extent. (1) To no extent. Developed by Dr. Dean Spitzer, IBM Corporation
  35. 35. innovation is extremely complex and multi-faceted and requires integrative thinking &quot;As a student of innovation for some twenty odd years, I still find it amazing just how hard innovation continues to be.“ [John Seely Brown] It is important, first and foremost, to at least begin to develop a clear operational definition of the meaning of “innovation”; it is impossible to measure what one can’t define! Integration Innovation
  36. 36. Innovation is becoming even more complex and difficult to manage <ul><li>Types of Innovation </li></ul><ul><li>Business model </li></ul><ul><li>Management system </li></ul><ul><li>Networks and alliances </li></ul><ul><li>Core process </li></ul><ul><li>Enabling process </li></ul><ul><li>Product </li></ul><ul><li>Product performance </li></ul><ul><li>Product system </li></ul><ul><li>Service </li></ul><ul><li>Channel </li></ul><ul><li>Brand </li></ul><ul><li>Customer experience </li></ul>“ Innovation is not a simple matter of execution, nor is becoming an effective innovator a simple task. In truth, the personal capabilities that generate innovation are untidy, unruly, disruptive, and unpredictable capabilities that often bring with them confusion, ambiguity, surprise, vagary, conflict, and failure–experiences that few of us actively seek out.” [Roger Martin & Hilary Austen] “ The biggest insight we got from all that research, 750 CEOs, was that it would be in the business model, not their products and services. The product and services, that is where everybody always thinks about innovation…but that kind of innovation can be easily copied. What can’t be easily copied is your business model.” - Ginni Rometty, SVP, IBM GBS “ Management innovation has produced the most profound shifts in business productivity.” – Gary Hamel Integration
  37. 37. Measurement must enable management to drive value creation and avoid value destruction Integration Innovation is one of the keys to value creation in the 21 st century, but how much innovation is being measured? How much value destruction is being perpetuated by poor measurement and management of the innovation ecosystem? What are the innovation drivers in your organization?
  38. 38. The Balanced Scorecard was a step toward integration Integration Integration But it is typically used to classify existing metrics. Unfortunately the Learning & Growth perspective (where innovation resides) has been referred to as “the missing measures.”
  39. 39. The key to effective measurement is asking the right questions How much of valuable “potential innovation” is being missed? What aspects of innovation should we be measuring? Integration What are the organization’s most crucial measures of innovation and their relationships? What happens to ideas? Most waste occurs in the early stages of innovation. Excellence in innovation begins with excellence in creativity management. What is the best operational definition of “innovation” for our organization?
  40. 40. Innovation measurement requires looking beyond what we are currently measuring, making new connections, and crossing functional silos Focus Integration “ Every organization has hidden caches of knowledge.” [Siebel & Malone] Integrated performance measurement requires a context of collaboration, trust, and leadership. When people become creative, it is usually in spite of the formal education or workplace performance measures. Creative answers lie outside of the traditional curriculum. Measuring the amount of content absorbed does not measure innovation or the capacity for it.
  41. 41. Innovation is a social process. How much are we doing to measure and manage the complex web of interrelationships? How widespread is invention? What innovation collaboration is taking place? Who are new inventors? Who are potential inventors? Who should be interacting with whom? Who should be ‘innovation mentors’? Who are crucial innovation ‘partners’? “ Innovation is without doubt a social process.” [Mark W. McElroy] Integration
  42. 42. It has been found that the ‘major roadblocks’ to innovation are culture and climate. Are you measuring and managing them? Integration
  43. 43. A integrated framework for measuring and managing innovation is needed Integration Integration Leading Lagging
  44. 44. Many ‘internal outcomes’ of innovation become ‘inputs’ in the next cycles of innovation Integration Integration
  45. 45. Performance Measurement Cycle Innovation measurement should be holistic Integration Innovation Capability Innovation Capacity Innovation Effectiveness Innovation Efficiency
  46. 46. The purpose of measurement is not to collect data but to increase knowledge and wisdom Interactivity Organizational Intelligence Creating knowledge and wisdom from data requires more than technical measurement skills. Interactivity
  47. 47. Performance Measurement Cycle Interactivity Performance measurement should be interactive and iterative “ Measurement is critical important to a healthy innovation system. But what is really important is what gets done with the information….The ultimate goal should be to have innovation measurement and acting on the information become a natural part of how business gets done.” [ Innovating the Corporation ] Performance Measurement Cycle
  48. 48. Performance Measurement Cycle How well is your organization’s innovation measurement system working? Interactivity
  49. 49. Automation can be a blessing and a curse! Interactivity “ Technology has given us all a bad case of information overload.” - C.D. Hobbs False precision: What does it mean in terms of innovation for a dashboard metric to be green, yellow, or red?
  50. 50. Performance Measurement Cycle Dialogue is the key to strengthening the performance measurement cycle Measurement needs to be treated less like ‘a project’ and more like an ongoing ‘dialogue.’ It is important to have a robust and continuing dialogue around the measurement of innovation. Interactivity
  51. 51. Performance Measurement Cycle Interactivity
  52. 52. Questions to promote dialogue about your organization’s innovation measurement <ul><li>How well does our organization understand innovation and what drives it? </li></ul><ul><li>What is your innovation measurement system really telling us? </li></ul><ul><li>Are we measuring the right things? Are we making progress toward measuring more of the right things? </li></ul><ul><li>How well is our innovation measurement system aligned with our strategy? </li></ul><ul><li>How well are we doing in managing our innovation potential for the future? </li></ul><ul><li>What assumptions and mental models is our innovation measurement system based on? Are they the right assumptions? Are they causing us to miss any potentially valuable sources of innovation? </li></ul><ul><li>Are any current measures or ways of measuring inhibiting innovation? </li></ul><ul><li>Are the right people being recognized and rewarded for the right things? </li></ul><ul><li>How much measurement is used for self-serving purposes (versus for learning and improvement)? How much manipulation and ‘game playing’ is there? </li></ul><ul><li>How serious is the problem of the “silo-ization” (fragmentation) of innovation measurement? </li></ul><ul><li>Are we becoming an more agile organization with respect of innovation? </li></ul><ul><li>How much high-quality dialogue is there around measuring innovation? </li></ul><ul><li>Is innovation measurement being continuously improved? </li></ul><ul><li>How innovative are we in measuring innovation? </li></ul>
  53. 53. Innovation measurement needs to be transformed Changing innovation measurement is one of the highest leverage things any organization that aspires of leadership in innovation can do.
  54. 54. Context The most important aspects of measurement are the social and organizational, not the technical, factors. Before innovation measurement can significantly change, something must be done about the “context of measurement”
  55. 55. So now let’s get innovative about measuring innovation! Please remember that it isn’t just about the metrics…