1) Several studies have found that innovation measurement through surveys is highly susceptible to questionnaire design effects, compromising international comparability as countries differ in their survey methods. 2) Randomized experiments manipulating factors like survey length, format, and voluntary/mandatory status found significant differences in reported innovation rates. For example, short forms and voluntary surveys yielded higher rates. 3) The document recommends more systematic testing of design effects and revising current guidelines that assume short and long forms provide equivalent measurements of innovation. Standardizing certain design features across countries could improve comparability.

1. State of the art insights in capturing,
measuring and reporting firm-level
innovation indicators
Machteld Hoskens, Julie Delanote,
Koen Debackere and Laura Verheyden
OECD Blue Sky Forum
Ghent, September 2016

2. Background
• Oslo Manual provides guidelines for measuring innovation at
the firm level
• Throughout its 3 editions (1992, 1997, 2005) most attention
went to clarifying and expanding innovation concepts
• Questionnaire design received limited attention
• For example: just 2 pages out of 163 discussing questionnaire design
issues in 3rd ed., and only 1 paragraph in this section on issues to pay
attention to when running international innovation surveys (§ 455)

3. Concerns
• A small number of empirical studies done so far suggest
innovation measurement as we have been doing it for years, is
very vulnerable to questionnaire design effects.

4. Studies
• OECD (2012) study using innovation survey metadata suggested
some innovation survey design features negatively impacted
the comparability of results between countries.

5. Studies (ctd.)
• The Netherlands (CBS, 2012) went from a paper form to a web
form design in 2011.
• The overall innovation rate increased from 35% to 48%, i.e., a
more than 10% increase.
• “Break in series might partially reflect true increase in some
innovating enterprises, but more important for explaining the
break was the change in survey format.” (CBS, 2012, p. 177)

6. Studies (ctd.)
• Norway performed a number of randomized experiments in
2011 (Wilhelmsen, 2012). They manipulated 2 factors.
• Innovation rates were lower when the survey was mandatory
than when it was voluntary;
o e.g., resp. 38% vs. 45% for product and/or process
innovation, a 7% increase.
• Innovation rates were lower in a combined R&D – innovation
survey vs. in a stand-alone innovation survey
o e.g., resp. 24% vs. 36% for product and/or process
innovation, an increase of more than 10%.

7. Studies (ctd.)
• In Belgium we performed a randomized experiment in 2015.
We manipulated questionnaire length.
• The innovation rates increased in the short form compared to
the regular long form, and sometimes almost doubled.
Form
Product Process Organisational Marketing
1 Long (paper/web) .17 .23 .25 .22
2 Short (paper/web) .32 .35 .43 .33
3 Short (phone nrsp) .37 .33 .29 .27
Difference ( 2 - 1 ) .15 .12 .18 .10
Difference ( 3 - 1 ) .21 .10 .04 .05
Innovation Type

8. Studies (ctd.)
• In a World Bank study (Cirera & Muzi, 2016) a short innovation
form was administered first, followed by a long innovation form
for a random subset of initial respondents.
• Here too, generally higher innovation rates were obtained with
the short form.
o e.g., resp. 51% and 39% for product innovation for the short
vs. long form, a difference of more than 10%.
• Potential explanations for the findings in the different studies:
effects during cognitive processing of survey questionnaire
forms: satisficing, unpacking effect, selection bias, mode
effects, …

9. Conclusions
• Innovation measurement at the firm level in its current form is
like attitude measurement, highly susceptible to method
effects
• This seriously compromises international comparability, as
countries currently differ widely in the methods they use
• E.g., within EU, variation in:
o voluntary vs. mandatory status of the survey;
o use of short form surveys for non-response adjustments;
o response mode: paper format, web format, phone interview,
face-to-face interview
o combined vs. stand-alone innovation survey

10. Recommendations
• There is a clear need for more systematic scientific
experimentation with and validation of potential innovation
survey design effects
• As innovation measurement occurs in a multinational,
multiregional and multicultural context (3MC surveys), we
should gain from lessons learned in other 3MC projects such as
ESS, PIRLS, PISA, TIMMS, … (see e.g., Harkness, Braun, Edwards,
Johnson, Lyberg, Mohler, Pennell & Smith, 2010)
o E.g., translation issues should at least be looked into
o E.g., not just cognitive testing but also pretesting in multiple
countries of questions in harmonized EU innovation form

11. Recommendations (ctd.)
• E.g., More unpacked (list based) measurement of innovation:
“The Oslo Manual presents a rather marked distinction between
functional and other forms of changes to products and processes.
Future Oslo revision work should attempt to make more explicit
what those criteria are and formulate them in survey
environments. Specific questions asking firms to describe the
various dimensions of novelty may assist in this process. The
experience accumulated from this and other related projects on
the analysis of design can help formulate these in a more concrete
way.” (Galindo-Rueda & Van Cruysen, 2016, § 41, p. 13)

12. Recommendations (ctd.)
• Revise section below currently in the Oslo Manual, as evidence
showed short and long form innovation surveys do not provide
equivalent measurements:
• ”For many small units and units in sectors with little innovation
activity, the response burden for a full innovation questionnaire may
be quite large relative to their innovation activity. Unit non-response
rates may also be higher for these units. In such cases, shorter survey
questionnaires that focus on a set of core questions can be useful.
Short-form questionnaires can also be used in surveying units that
have not reported innovation activity in previous innovation surveys.
Conversely, for individual units in the above-mentioned groups (small
units or less innovative sectors) which have previously reported
substantial innovation activity, full questionnaires may be used.”
(OECD, 2005, pp. 124-125, § 456)

13. Recommendations (ctd.)
• Revise recommendation currently in Eurostat methodological
guidelines, as evidence showed short and long form innovation
surveys do not provide equivalent measurements:
• ”If non-respondents, as an un-weighted percentage of all relevant
enterprises in the sampling frame, exceed 30%, then a simple
random sample of at least 10% of the non-respondents (excluding
non-relevant enterprises) should be selected. The questions to be
included in the non-response survey are specified in Annex 7.”
(Eurostat, 2014, p. 5)
(Annex 7 specifies 8 core questions, i.e., an abbreviated version of
the regular long form)