- A study conducted innovation surveys in 15 developing countries using both a short 9-question survey and a longer, more in-depth survey to test if the short survey accurately captured firm-level innovation. - The results showed large differences between innovation rates reported in the short vs long surveys, with over 30% of firms changing their responses. The short survey significantly overestimated innovation rates. - Explanations for the differences included cognitive challenges understanding innovation concepts, respondent framing biases, and difficulties recalling innovation over the period between surveys. The quality of the interview and time elapsed between surveys impacted response accuracy.

1. MEASURING FIRM-LEVEL INNOVATION USING SHORT
QUESTIONNAIRES.
EVIDENCE FROM AN EXPERIMENT IN DEVELOPING
COUNTRIES
Xavier Cirera and Silvia Muzi
September 21, 2016
OECD Blue Sky Conference
Ghent, Belgium

2. 1. MOTIVATION
Little evidence on extent and impact of firm level innovation in developing
countries
o Lack of comparable innovation data
o Issues with reliability: subjective nature of some key concepts is a
challenge, especially in developing countries (more incremental and
less radical innovation)
o Implementing innovation surveys is costly, especially in developing
countries
>>As a result, there is a complete lack of longitudinal data
1

3. 2. OBJECTIVES
WBG-DECEA launched the Innovation Enterprise Surveys with the aim to:
• Increase the coverage of innovation surveys in developing countries
• Assessing the robustness of the Oslo framework (testing short form
questionnaire vs traditional long questionnaire):
o Which survey instrument is preferable?
o Are the innovation data collected through these two survey
instruments different? If so, why are they different?
Main finding: Short questionnaire severely mis-measures innovation
outcomes
2

4. 3. THE EXPERIMENT
Two stage approach to data collection.
• First stage: a short innovation section of 9 selected questions (short
questionnaire) is incorporated into the main ES questionnaire.
• Second stage, an in-depth innovation module (long questionnaire) is
administered to 75% randomly selected ES respondents.
Both questionnaires are based on the standard Community Innovation
Survey (CIS) but the long questionnaire was expanded to
o capture more details on innovation outcomes – when innovation was
reported
o to integrate other important questions on organization, management
practices, use of ICT, and innovation inputs.
3

5. 4. THE DATA
• The dataset covers more than 11,000 firms in 15 countries –
• 11 in Sub-Saharan Africa - DRC, Ghana, Kenya, Namibia, Nigeria, Sudan,
South Sudan, Tanzania, Uganda, Malawi and Zambia
• 4 in South Asia - Bangladesh, India, Pakistan, and Nepal
• Firms in manufacturing and services
• Only formal (registered) firms with 5+ employees (ES classification small (5-19),
medium (20-99) and large (100+) - micro firms excluded
4

6. 5. DIFFERENCES ON INNOVATION OUTCOMES BETWEEN THE
TWO SURVEYS
• Large differences between innovation rates reported in the two questionnaires
• In more than 30% of cases firms change their mind regarding innovation
outcomes between the two questionnaires
5
BGD
DRC
GHA
IND
KEN
MWI
NAM
NGANPL
PAK
SDN
SSDTZA
UGA
ZMB
0
20406080
0 20 40 60 80
Long questionnaire
Product Innovation
BGD
DRC
GHA
IND
KEN
MWI
NAM
NGA
NPL
PAK
SDN
SSD
TZA
UGA
ZMB
0
20406080
Shortquestionnaire
0 20 40 60 80
Long questionnaire
Process Innovation
BGD
DRC
GHA IND
KEN
MWI
NAM
NGA NPL
PAK
SDN
SSD
TZA
UGA
ZMB
0
20406080
0 20 40 60 80 100
Long questionnaire
Marketing Innovation
BGD
DRCGHA
IND
KEN
MWI
NAM
NGA
NPL
PAK
SDN
SSD
TZA
UGA
ZMB
0
20406080
Shortquestionnaire
0 20 40 60 80
Long questionnaire
R&D

7. 5. DIFFERENCES ON INNOVATION OUTCOMES BETWEEN THE
TWO SURVEYS
• More importantly, both surveys overstate the extend of innovation – when this
can be verified looking at the explanation of the innovation introduced –
• Short questionnaire severely overestimates for most countries – but still very
high innovation rates once cleaned!!!
6

8. 5. DIFFERENCES ON INNOVATION OUTCOMES BETWEEN THE
TWO SURVEYS

9. 5. DIFFERENCES ON INNOVATION OUTCOMES BETWEEN THE
TWO SURVEYS
8

10. 5. DIFFERENCES ON INNOVATION OUTCOMES BETWEEN THE
TWO SURVEYS
• How similar are firms in these groups? No-no likely non-innovators, but other
non-innovators we could not say
9

11. 5. DIFFERENCES ON INNOVATION OUTCOMES BETWEEN THE
TWO SURVEYS
• How similar are firms in these groups? No-no likely non-innovators, but other
non-innovators we could not say
10

12. 6. EXPLAINING DIFFERENCES BETWEEN THE TWO SURVEYS
Three sets of elements may affect respondents’ answers to innovation questions
• Cognitive problems due to different understanding of innovation and quality of the
interview
• Respondent framing - firm characteristics; context in which firms operate; quality of
the interview
• Recall period – time elapsed between the two interviews conducted for the
experiment
Estimation based on
- probit model (dependent variable = 1 if the answer in the two surveys is different and 0
otherwise).
- multinomial logit model, (Yi = 0: no difference between the two surveys; Yi =1 firm i reports
innovation in the short questionnaire and no innovation in the long questionnaire; Yi =2: firm i
reports no innovation in short questionnaire and innovation in the long questionnaire)
11

13. 6. EXPLAINING
DIFFERENCES BETWEEN
THE TWO SURVEYS
• Discrepancies between
surveys:
(–) same respondent;
size; potential innovator;
report bribes payment
(+) time interval; quality
of the interview
• No effect of explanatory
cards
12

14. 6.
ACCURACY
SHORT
QUESTION
NAIRE–
controlling
for selection
•(+) quality of the
interview
•(-) size, report
bribes payment
•no explanatory card
effects,
13

15. 7. MAIN FINDINGS
• The results from the survey experiment are clear: a few short questions in a more general
firm-level survey do not provide an accurate picture of firm-level innovation.
• More context to the questions is likely to be needed; although the innovation rates found in the
longer innovation questionnaire also requires substantial cleaning and present innovation
rates that are unreasonably very high for the level of development in these countries – we
need a new approach to better measure innovation outcomes and to disentangle “new”
from “significantly improved”
Also:
• The results show that lack of information does not seem to play a key factor explaining
inaccuracies, and neither the use of explanatory cards nor the experience of the respondent
seem to matter.
• The quality of the interview is likely to influence the accuracy of responses (lower quality
interview> inaccurate responses), while whether the firm reports to pay informal payments and
the size of the firm are likely to reduce inaccuracies.
• In the case of explaining discrepancies between surveys, the time elapsed between interviews
is a strong predictor of differences, likely the result of the recall effect.
14

16. THANK YOU
15