IBM iForum - Rethinking Innovation Measurement by Dean Spitzer

Rethinking the Measurement of Innovation Dr. Dean Spitzer Performance Measurement Thought Leader IBM Research Presentation at the IBM Innovation Forum Zurich, November 27, 2007

WARNING: There is no “magic metric” for innovation

Pre-assessment: How well is your organization currently measuring innovation?

Does your organization have a clear operational definition of ‘innovation’?

How much holistic visibility is there around innovation in your organization?

Is your organization really measuring “innovation that matters”?

Is your organization increasing its understanding of innovation?

How much has innovation measurement in your organization has changed and been improved in the past few years?

What proportion of your innovation measures are ‘leading’ versus ‘lagging’ indicators?

How well is your organization measuring and managing ideas and creativity?

How useful for managing innovation is the data you are currently collecting?

How confident are you that your current innovation measurement system will help you drive and sustain real innovation today and in the future?

Does your innovation measurement system help stakeholders collaborate to co-create optimal value?

On a scale of 0 percent to 100 percent, to what extent do you think your organization measures and manages its entire portfolio of innovation assets?

Innovation is too important not to be measured well

What experts are saying about the measurement of innovation

“ Measurement is both fundamental and critical to success with innovation.” [Davila, Epstein, Shelton, Making Innovation Work , Wharton, 2005]

“In a recent survey… more than half rated their performance measurement system for innovation as poor or less than adequate .” [ Making Innovation Work ]

“ Few companies actually measure their innovation efforts , and those that do tend to look at a handful of financial measures at best.” [Kuczmarski, Middlebrooks & Swaddling, Innovating the Corporation ]

“ Innovation is widely undermeasured , and few firms—even those that attempt to track innovation rigorously—are confident that they’re doing it right.” [Boston Consulting Group, Measuring Innovation 2006 ]

“ The potential for most companies to improve their measurement practices—and as a result, boost their return on innovation spending—is sizable.” [Boston Consulting Group, Measuring Innovation 2006 ]

“ Measurement is critical if a company hopes to improve its innovation effectiveness. Companies can’t improve performance if they don’t measure it.” [ Innovating the Corporation ]

“ Measurement is critically important to a healthy innovation system.” [ Innovating the Corporation ]

“Without question, innovation is being asked to play a greater role in the CEO’s organic growth strategy. Yet to date, innovation metrics have been vague at best , and more often elusive.” [Pat LaPointe]

Few understand just how powerful measurement is!

"Measurement always improves performance." [Buckingham & Coffman, First, Break All the Rules , p. 236]

"Good data, properly distributed, transform organizations. " [Whitely, The Customer-Driven Company , p. 175]

"Measures have great power, almost like genetic code, to shape action and performance...Change the measures, and you change the organism." [ Epstein & Birchard, Counting What Counts , p. 145]

"Most often when we see illogical behavior, the fault is in the measurement system, not in the employees." [Brian Joiner, Fourth Generation Management , p. 242]

"Changing the way we measure changes everything." [Meador, The Dance of Change , p. 299]

"An organization's measurement system strongly affects the behavior of people both inside and outside the organization." [Kaplan & Norton, The Balanced Scorecard , p. 21]

"The essence of a corporate culture is the firm's measurement system." [Strassman, The Business Value of Computers , p. 73]

"The mere action of defining measures of success will change behavior positively or otherwise." [Thorp, The Information Paradox , p. 164]

"Metrics are to a business what the five senses are to humans - systems of feedback that improve our capacity to adapt and excel over the long run. " [Tachi Kiuchi, "What We Learned in the Rainforest," Barrett-Koehler, 2002, pp. 152-153]

“ One cannot reliably manage what one cannot measure. This is not my axiom. This is an unimpeachable law of nature. It cannot be broken…ever.” J.B. Thoreson Employees will do what you ‘inspect,’ not necessarily what you ‘expect.’

Measurement is ubiquitous and it is a mess! In The Agenda , Michael Hammer puts it this way: “ A company's measurement systems typically deliver a blizzard of nearly meaningless data that quantifies practically everything in sight, no matter how unimportant; that is devoid of any particular rhyme or reason; that is so voluminous as to be unusable; that is delivered so late as to be virtually useless; and that then languishes in printouts and briefing books, without being put to any significant purpose.... In short, measurement is a mess." And even fewer realize how broken performance measurement is! 2 3 4 5 6 7 8 9 10

Management is based on measurement, and all other organizational systems are dependent on the measurement system No organization can be any better than its measurement system The importance of performance measurement, as the foundational management system, is not widely appreciated. And, it is ironic that the measurement of innovation is one of the least effective and innovative forms of measurement.

One of the biggest problems in organizations is lack of alignment among ‘measurement systems’ Organizations with poor performance measurement will be poorly aligned, with functions pursuing their own self-interest and often working at cross-purposes Supply Chain Information Technology Not only don’t most organizations have a single integrated measurement system, but the disparate measurement systems don’t even communicate! Budgeting Sales Operations Research & Development Human Resources Marketing Finance Learning

Most organizations are dissatisfied with their performance measurement

Measurement is the elephant in the room The innovation measurement paradox: While everyone knows it’s inhibiting progress, few want to deal with it!

That’s why I wrote my latest book, “Transforming Performance Measurement”

"This is one of the best books on performance measurement that I have ever read ! If you only read one book on performance measurement this year, make it this one." – Andy Neely, Professor and Chair, Centre for Business Performance, Cranfield University School of Management, and author of The Performance Prism

" Dean Spitzer takes the mystery out of performance measurement with this practical guide to how doing measurement right can transform any organization. Dean's insight into transformational measurement has been extremely helpful at Wachovia as we transform our learning measurement to better assist our businesses in achieving their goals." – Dean Williams, Chief Learning Officer, Wachovia Corporation

" Dean Spitzer’s concept of transformational measures will radically change the way individuals and organizations view performance measurement .” – Gregg Mauro, Executive Director, Government Executives Network

Performance measurement needs to be transformed Changing measurement is one of the highest leverage things any organization can do.

How people “experience” measurement is the key to how they will respond to it Context

Measurement is typically used to monitor and control, which does not encourage innovation Context Too often evaluation and its consequences undermine learning from measurement.

So much measurement is being used to report on the past and for self-serving purposes Context “ There are too many patents in the world today that have been filed simply because it made somebody look good, but never really presented any value opportunities.” [Goran Roos] Measurement done to prove will rarely improve !

Context The most important aspects of measurement are the social and organizational, not the technical, factors. Almost nothing has been done to address the “context of measurement” Context

The biggest problem with the context of measurement is the lack of Measurement Leadership

While most managers publicly extol the value of measurement, few actually use it systematically and well.

Measurement is everybody’s job and therefore nobody’s job!

Organizational leaders are more than happy to delegate measurement to "measurement specialists." They don't realize how strategic measurement is, and how much management attention it requires to do it right. Measurement is one of the most under-appreciated organizational activities, and measurement leadership is one of the least appreciated leadership roles.

Context Who is leading Innovation Measurement in your organization?

Measures are the lens through which most performance is viewed Focus Once measures have been selected, they color what we see and how we act.

The measures you select will determine what you ‘see’ and can greatly bias the perceptions of the entire organization! Focus

Most organizations are full of “routine measures” that have been accepted as “standard operating metrics” and rarely, if ever, change Focus Performance measures are viewed as hard-wired into the organization. Unfortunately most organizations tend to default to easy-to-measure things, for which data is readily available. Routine Measures

Is your organization’s measurement system differentiating?

Unfortunately, most leaders don't differentiate between the critical few measures that will have the greatest impact and the hundreds, or thousands, of other measures -- the trivial many -- that permeate every area of their organizations.

Too many executives and other managers think it is sufficient just to track generic or standard industry-approved measures. These are what I call "routine measures" and they are satisfactory for maintaining the status quo , but not for taking the organization to the next level.

If you measure what everyone else is measuring, your business will be well on the way to commoditization. However, if your organization strives for more than that, you might want to consider rethinking the measures that are currently populating your measurement system

In today's highly competitive and increasingly services-oriented marketplace, it is vital to be measuring more than just the same old, routine measures. Not only is almost all measurement routine, but almost all of it has derived from a commodity manufacturing mindset. It is amazing how few new measures have been developed since the industrial revolution!

Focus

Traditional innovation measurement and management has resulted in enormous waste 4% hit rate 96% of innovation projects fail Focus

What does a rear view mirror tell us about innovation? Focus The balance sheet is supposed to describe value, but it ignores almost all intangible sources of future value.

Most innovation measurement systems don’t encourage innovative thinking Focus Traditional metrics tend to reinforce the status quo. Old performance measures are a box that constrains performance. Traditional Metrics

There is a tendency to measure what is easiest and safest to measure, not necessarily what is most important Focus “ Too often measurement systems are driven by what is available rather than what is needed.” [KPMG, The Search for New Insights, 2001]

Measures of innovation are among the least innovative measures “ Most measurements of performance are geared to the need of the 20th-century manufacturing companies. Times have changed. Metrics must change as well.” [Lowell L. Bryan, “The new metrics of corporate performance,” McKinsey Quarterly , 2007] What we measure is what we get. If we keep measuring the same things, we will get more of the same. Focus Although organizations have many creative people involved in innovation, almost no creative people are working on improving the innovation measurement system! “ You can’t Six Sigma your way to innovation.” [Russ Nussbaum]

Measures need to change Focus “ If we measure the new with the tools of the old, we will not ‘see’ the new.” [Karl-Erik Sveiby]

Innovative organizations must not be afraid to experiment with measurement Focus “ To change our perspective, we must change our measures.” [ Source: D.R. Spitzer , Transforming Performance Measurement ]

Emergent measures often require qualitative measurement, estimating, and subjectivity Focus "A high barrier stands between us and the habit of making rough estimates -- the fear of getting the 'wrong' answer. There is nothing wrong with educated guesses as long as the uncertainty is acknowledged and managed. Contrary to what most of us have learned in school … an inexact answer is almost always good enough. “ [Arno Penzias] “ Measurement is assessing the degree to which a variable is present … .Notice there is no reference to numbers in that definition. ” [D. Lynn Kelley] Extremely Honest Not Honest At All 1 2 3 4 5 6 7 8 9 Extremely Honest Not Honest At All 1 2 3 4 5 6 7 8 9 Extremely Honest Not Honest At All 1 2 3 4 5 6 7 8 9 Extremely Honest Not Honest At All 1 2 3 4 5 6 7 8 9 Everything can be measured in some way. Imperfect measurement is better than none, as long as we acknowledge the ‘confidence level.’ In measuring intangibles, we need to improve our ability to estimate.

Developing emergent measures is an iterative process Focus One of the keys to emergent measurement is the “socialization process” that occurs as the construct and its measurement are discussed.

Sample Emergent Measures:

Innovative Climate

Innovation Talent

Employee Engagement in Innovation

Innovation Network Strength

Innovation Knowledge Flow

Learning from Failure

Learning Speed

Innovation Agility

Innovation Collaboration

Reputation for Innovation

Innovation Partner Relationships

Organizational Trust

Innovation Leadership

Strategic Alignment of Innovation

IP Strength

Client Success from Innovation

Innovation Impact

Innovation Intensity

Innovation Freshness

Idea Pipeline

Innovation Pipeline Quality

Innovation Velocity

Innovation Value-Add Time

Innovation Process Maturity

Focus “… the most predictive measure of whether an organization will be innovative is the level of trust between people in the organization.” [Dauphinais, Means & Price, The Wisdom of the CEO ]

Characteristics

Commitment: Commitment to organizational goals and operations. Work perceived as stimulating and engaging.

Freedom: Opportunities to make own decisions, seek information, and show initiative. Freedom from tight supervision.

Idea-support: People encouraged for putting forward ideas and suggesting improvements.

Positive Relationships: People trust and get on well with one another in a climate. Absence of personal conflicts.

Dynamism: Dynamic and exciting atmosphere.

Playfulness: People laugh and joke with one another.

Idea-proliferation: People are perceived as having creative ideas and varied perspectives towards their work.

Stress: People generally feel overburdened and under pressure at work.

Risk-taking: New ideas may be implemented; people prepared to take risks.

Idea-time: People have the time to generate and consider new ideas.

Shared View: There are open and adequate communications between more and less senior employees.

Pay Recognition: People are satisfied with their remuneration.

Work Recognition: People receive praise for their achievements.

Emergent Measure: Innovation Climate Questionnaire Source: Innovation Centre Europe

Executives model innovative behavior

I am encouraged to use my creativity

I am personally excited about innovation

Innovation is a high priority in this organization

My supervisor talks with me about innovation

Managers sponsor innovation

Employees feel empowered to be innovative

Cross-functional teams are empowered to be innovative

Diversity of ideas is encouraged

The organization has a well-articulated innovation strategy

There is a sense of urgency about innovation

The customer is the focus of innovation

I take advantage of opportunities to network with other innovative people

I regularly access resources (expertise, knowledge, education, etc.) that help me become more innovative

Creative ideas are responded to promptly

Employees feel empowered to take risks

There is appropriate tolerance for failure

Learning from mistakes is a formal part of work

Developing employees’ innovation-related knowledge and skills are a priority in this organization

Employees understand how innovation is measured

Creativity is recognized

Time is provided for creative efforts

I am involved in at least one innovative project

Emergent Measure: Employee Engagement in Innovation Scale: (4) To a great extent. (3) To some extent. (2) To a little extent. (1) To no extent. Developed by Dr. Dean Spitzer, IBM Corporation

innovation is extremely complex and multi-faceted and requires integrative thinking "As a student of innovation for some twenty odd years, I still find it amazing just how hard innovation continues to be.“ [John Seely Brown] It is important, first and foremost, to at least begin to develop a clear operational definition of the meaning of “innovation”; it is impossible to measure what one can’t define! Integration Innovation

Innovation is becoming even more complex and difficult to manage

Types of Innovation

Business model

Management system

Networks and alliances

Core process

Enabling process

Product

Product performance

Product system

Service

Channel

Brand

Customer experience

“ Innovation is not a simple matter of execution, nor is becoming an effective innovator a simple task. In truth, the personal capabilities that generate innovation are untidy, unruly, disruptive, and unpredictable capabilities that often bring with them confusion, ambiguity, surprise, vagary, conflict, and failure–experiences that few of us actively seek out.” [Roger Martin & Hilary Austen] “ The biggest insight we got from all that research, 750 CEOs, was that it would be in the business model, not their products and services. The product and services, that is where everybody always thinks about innovation…but that kind of innovation can be easily copied. What can’t be easily copied is your business model.” - Ginni Rometty, SVP, IBM GBS “ Management innovation has produced the most profound shifts in business productivity.” – Gary Hamel Integration

Measurement must enable management to drive value creation and avoid value destruction Integration Innovation is one of the keys to value creation in the 21 st century, but how much innovation is being measured? How much value destruction is being perpetuated by poor measurement and management of the innovation ecosystem? What are the innovation drivers in your organization?

The Balanced Scorecard was a step toward integration Integration Integration But it is typically used to classify existing metrics. Unfortunately the Learning & Growth perspective (where innovation resides) has been referred to as “the missing measures.”

The key to effective measurement is asking the right questions How much of valuable “potential innovation” is being missed? What aspects of innovation should we be measuring? Integration What are the organization’s most crucial measures of innovation and their relationships? What happens to ideas? Most waste occurs in the early stages of innovation. Excellence in innovation begins with excellence in creativity management. What is the best operational definition of “innovation” for our organization?

Innovation measurement requires looking beyond what we are currently measuring, making new connections, and crossing functional silos Focus Integration “ Every organization has hidden caches of knowledge.” [Siebel & Malone] Integrated performance measurement requires a context of collaboration, trust, and leadership. When people become creative, it is usually in spite of the formal education or workplace performance measures. Creative answers lie outside of the traditional curriculum. Measuring the amount of content absorbed does not measure innovation or the capacity for it.

Innovation is a social process. How much are we doing to measure and manage the complex web of interrelationships? How widespread is invention? What innovation collaboration is taking place? Who are new inventors? Who are potential inventors? Who should be interacting with whom? Who should be ‘innovation mentors’? Who are crucial innovation ‘partners’? “ Innovation is without doubt a social process.” [Mark W. McElroy] Integration

It has been found that the ‘major roadblocks’ to innovation are culture and climate. Are you measuring and managing them? Integration

A integrated framework for measuring and managing innovation is needed Integration Integration Leading Lagging

Many ‘internal outcomes’ of innovation become ‘inputs’ in the next cycles of innovation Integration Integration

Performance Measurement Cycle Innovation measurement should be holistic Integration Innovation Capability Innovation Capacity Innovation Effectiveness Innovation Efficiency

The purpose of measurement is not to collect data but to increase knowledge and wisdom Interactivity Organizational Intelligence Creating knowledge and wisdom from data requires more than technical measurement skills. Interactivity

Performance Measurement Cycle Interactivity Performance measurement should be interactive and iterative “ Measurement is critical important to a healthy innovation system. But what is really important is what gets done with the information….The ultimate goal should be to have innovation measurement and acting on the information become a natural part of how business gets done.” [ Innovating the Corporation ] Performance Measurement Cycle

Performance Measurement Cycle How well is your organization’s innovation measurement system working? Interactivity

Automation can be a blessing and a curse! Interactivity “ Technology has given us all a bad case of information overload.” - C.D. Hobbs False precision: What does it mean in terms of innovation for a dashboard metric to be green, yellow, or red?

Performance Measurement Cycle Dialogue is the key to strengthening the performance measurement cycle Measurement needs to be treated less like ‘a project’ and more like an ongoing ‘dialogue.’ It is important to have a robust and continuing dialogue around the measurement of innovation. Interactivity

Performance Measurement Cycle Interactivity

Questions to promote dialogue about your organization’s innovation measurement

How well does our organization understand innovation and what drives it?

What is your innovation measurement system really telling us?

Are we measuring the right things? Are we making progress toward measuring more of the right things?

How well is our innovation measurement system aligned with our strategy?

How well are we doing in managing our innovation potential for the future?

What assumptions and mental models is our innovation measurement system based on? Are they the right assumptions? Are they causing us to miss any potentially valuable sources of innovation?

Are any current measures or ways of measuring inhibiting innovation?

Are the right people being recognized and rewarded for the right things?

How much measurement is used for self-serving purposes (versus for learning and improvement)? How much manipulation and ‘game playing’ is there?

How serious is the problem of the “silo-ization” (fragmentation) of innovation measurement?

Are we becoming an more agile organization with respect of innovation?

How much high-quality dialogue is there around measuring innovation?

Is innovation measurement being continuously improved?

How innovative are we in measuring innovation?

Innovation measurement needs to be transformed Changing innovation measurement is one of the highest leverage things any organization that aspires of leadership in innovation can do.

Context The most important aspects of measurement are the social and organizational, not the technical, factors. Before innovation measurement can significantly change, something must be done about the “context of measurement”

So now let’s get innovative about measuring innovation! Please remember that it isn’t just about the metrics…