Parallel sessions 5 - 20 September 2016

1. Holistic and timely monitoring of STI system
through an annual panel survey to experts
and researchers
— an experience in Japan
20 September, 2016
Masatsura IGAMI
National Institute of Science and
Technology Policy (NISTEP), Japan
OECD Blue Sky Forum
STI actors: the potential of direct surveys
1

2. • Japan’s science and technology policies are promoted based on a
Science and Technology Basic Plan (STBP) that is formulated every
five years.
• The fourth STBP (FY2011 to FY2015).
– Identification of important issues that must be tackled.
– The integrated promotion of science and technology/innovation
policies.
• The effective implementation of the plan-do-check-act cycle in
science, technology, and innovation (STI) policies requires
evidence regarding how Japanese STI has changed over time.
Background
1. Introduction
2

3. • Continuity
– Status of STI does not change overnight, thereby necessitating
continuous monitoring in order to identify changes.
• Comprehensive understanding
– Assessing the national STI system requires a comprehensive point of
view.
• Measurability
– There are often instances wherein quantitative data documenting
STI’s status are unavailable.
• Timeliness
– Timely monitoring of the impact of various policies is needed.
– Annual budgeting process.
Challenges in monitoring the status of STI
1. Introduction
3

4. • NISTEP conducted an annual panel survey that targeted Japan’s
leading researchers and experts (NISTEP Teiten survey).
• Their views were assessed by a set of questions intended to
measure their levels of satisfaction regarding issues related to STI.
• This survey can be considered as a health monitoring system of
STI.
Our strategy
1. Introduction
STI system in
Japan
Subject of monitoring Sensors: recognition of the country’s leading
researchers and experts (i.e., combined
wisdom).
University
Public research
institution
Private firm
4

5. • Status of Japanese STI and its situational changes were monitored
by 57 questions asked annually, in addition to questions specific
to certain years.
Question items
• Respondents answer each question
based on a six-point scale ranging
from “insufficient” to “sufficient.”
• Previous year’s responses were fed
back to respondents.
• A short description indicating why
they changed their position.
Composition of the 57 NISTEP Teiten
survey questions
2. Overview of NISTEP Teiten surveys conducted during the fourth STBP
5

6. Two groups of respondents
1) University / Public Research Institution (PRI) group (about 1,000)
2) Innovation overview group (about 500)
• Presidents of universities / PRIs
• Researchers who were nominated by a head of department or subordinate
agency of universites / PRIs
• Directors of the World Premier International (WPI)
research center
• Principal Investigators of the Funding Program for World-Leading Innovative
R&D on Science and Technology (FIRST)
• Executives of private firms who are responsible for R&D
• Representatives of startup companies, venture capital
• Heads of academic-industrial alliance offices in universities
• Persons who are working on S&T at a think tank / mass media
• Directors of a hospital
2. Overview of NISTEP Teiten surveys conducted during the fourth STBP
6

7. Timeline of the survey and response rate
86% Mean of 5 years (FY2011-2015)
About 1,000Respondents who responded 5 years
in a row
2. Overview of NISTEP Teiten surveys conducted during the fourth STBP
Launch of the fourth
STBP
Launch of the fifth
STBP
Feb. – Apr. 2012
FY2011
Oct. 2012 – Jan. 2013
FY2012
Sept. – Dec. 2013
FY2013
Sept. – Dec. 2014
FY2014
Sept. – Dec. 2015
FY2015
Apr. 2016
FY2016
1st survey 2nd survey 3rd survey 4th survey 5th survey
Discussion for planning of the fifth
STBP at MEXT (Summer of 2014)
Discussion for planning of the
fifth STBP at Council for Science,
Technology and Innovation
(Winter of 2014)
7

8. Examples of questions and their results
2. Overview of NISTEP Teiten surveys conducted during the fourth STBP
Index
change
Index
change
-0.47
(-0.1)
Science
-1.06
(-0.46)
-1.57
(-0.77)
Engineering
-0.53
(0.07)
Agriculture
-0.22
(-0.08)
G1
-0.97
(-0.15)
Health
-0.24
(-0.06)
G2
-0.55
(-0.2)
G3
-0.44
(-0.23)
G4
-0.17
(0.07)
Q1-18: Status of the baseline funding for executing research and development at universities and
PRIs
University
Fieldofdepartment
(University)
Public research
institute
Universitygroup
3 4 5 6
Index Index
2 2
Enough
Not-enough
Enough
Not-enough
3
Attributes Attributes
4 5 6
2.7(748) 3.0(108)
4.0(122) 3.1(248)
1.7(83)
2.9(148) 2.5(236)
2.2(237)
2.2(160)
3.7(203)
2.6(713) 2.9(105)
3.8(117) 2.9(236)
1.5(78)
2.6(134) 2.3(221)
2.1(230)
2.1(153)
3.5(196)
2.5(720) 2.6(105)
3.4(117) 2.8(235)
1.4(82)
2.3(134) 2.3(223)
2.0(235)
2.1(153)
3.5(196)
2.4(725) 2.4(101)
3.2(114) 2.5(239)
1.6(75)
2.1(134) 2.3(227)
1.9(227)
2.0(158)
3.4(206)
2.3(689) 2.0(96)
2.4(109) 2.6(218)
1.5(75)
2.0(121) 2.3(219)
1.7(221)
1.7(143)
3.5(204)
Note1: Categorization of university group was done by scientific publication share in Japan. Group 1: 5% or more; Group 2: 1% or more
and less than 5%; Group 3: 0.5% or more and less than 1%; Group 4: 0.05% or more and less than 0.5%.
Note2: The satisfaction index values are on a scale of zero (insufficient) to 10 (sufficient). Values of 5.5 or greater are considered
“unproblematic” ( ), 4.5–5.4 “not generally problematic” ( ), 3.5–4.4 “insufficient” ( ), 2.5–3.4 “generally insufficient”
( ), and less than 2.5 “extremely insufficient” ( ).
4.0(122)
3.8(117)
3.4(117)
3.2(114)
2.4(109)
Result of 2011 survey
Result of 2015 survey
Index
Respondents
8

9. Free answers of reasons for opinion change
Reasons for increased satisfaction Reasons for decreased satisfaction
 Increased funding made available by the
president or dean of university
 Initiatives by the university’s executive
directors.
 Changes in circumstances owing to a
respondent transferring to a different
institution.
 Management expenses grants decreased
considerably over time, necessitating
budget cuts.
 Sufficient money for research cannot be
procured through facility maintenance
funds alone.
 Insufficient funding for conducting
experiment based research activities.
 Increased cost of electricity and
consumption taxes have forced
researchers to bear an increasing
proportion of expenses related to the use
of common facilities.
 Funds are being depleted to cover just
printing and postage costs.
2. Overview of NISTEP Teiten surveys conducted during the fourth STBP
Free answers provide us with contexts which underline
changes of index.
9

10. Rank
Question
No.
Category Question Index change
Index
value
2015
1 Q1-19
Usability of research expenses in Grants-in-Aid for Scientific Research
(KAKENHI)
0.79
(0.13) 5.4
2 Q1-22
Fostering and securing of specialized personnel to handle operations
necessary for the smooth execution of research activities (URAs) (in
universities and PRIs)
0.35
(0.09) 2.4
3 Q3-04
Cooperation beyond the boundaries of the natural sciences to address
technical issues for resolving important issues
0.34
(0.07) 3.6
4 Q3-12
Unified public-private efforts to implement Japanese technology and
systems to overseas
0.32
(0.04) 2.8
5 Q3-03
Concentration on core competence in government-led research and
development
0.30
(0.10) 3.9
6 Q3-02
Implementation of strategies and national projects to resolve important
issues through cooperation between industry, academia, and government
0.24
(0.03) 3.6
10
Top 6 questions in terms of positive change with
respect to satisfaction between 2011–2015
Note1: Darker shades in the “index change” column indicate greater change in satisfaction. The top and bottom figures in each row
indicate the degree of change in satisfaction between 2011–2015 and 2014–2015, respectively.
Note2: The satisfaction index values are on a scale of zero (insufficient) to 10 (sufficient). Values of 5.5 or greater are considered
“unproblematic” ( ), 4.5–5.4 “not generally problematic” ( ), 3.5–4.4 “insufficient” ( ), 2.5–3.4 “generally insufficient”
( ), and less than 2.5 “extremely insufficient” ( ).
3. Situational changes of Japanese STI system during the fourth STBP
Research personnel
Research environment
Industry-academia-government collaboration
Innovation policyBasic research

11. Rank
Question
No.
Category Question Index change
Index
value
2015
1 Q1-18
Baseline funding for executing research and development at universities
and PRIs
-0.62
(-0.19) 2.3
2 Q1-06
Whether or not individuals with the required capabilities are applying to
doctoral programs
-0.57
(-0.17) 3.0
3 Q1-24
Research facilities and equipment for innovative and advanced R&D; and
the training of high-quality human resources (in universities and PRIs)
-0.49
(-0.07) 4.4
4 Q2-22
Degree of diversity in Japanese basic research as the source of future
innovation
-0.43
(-0.14) 3.0
5 Q2-23
Degree of originality in Japanese basic research as the source of future
innovation
-0.40
(-0.16) 3.0
6 Q2-17
Amount of indirect funding related to the government's public appeal-
based research funds (competitive research funds, etc.) (in universities
and PRIs)
-0.36
(-0.07) 4.0
11
Top 6 questions in terms of negative change with
respect to satisfaction between 2011–2015
3. Situational changes of Japanese STI system during the fourth STBP
Note1: Darker shades in the “index change” column indicate greater change in satisfaction. The top and bottom figures in each row
indicate the degree of change in satisfaction between 2011–2015 and 2014–2015, respectively.
Note2: The satisfaction index values are on a scale of zero (insufficient) to 10 (sufficient). Values of 5.5 or greater are considered
“unproblematic” ( ), 4.5–5.4 “not generally problematic” ( ), 3.5–4.4 “insufficient” ( ), 2.5–3.4 “generally insufficient”
( ), and less than 2.5 “extremely insufficient” ( ).
Research personnel
Research environment
Industry-academia-government collaboration
Innovation policyBasic research

12. -0.9
-0.8
0.2
1.0
2.6
2.9
3.0
3.0
-1.4
-1.0
0.0
-0.6
2.0
2.4
2.3
2.4
-5.0 -4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0
Innovation overview group University and PRI groupDecreasing
Increasing
Changes in research activities at universities and
PRIs over the past ten years by respondent group
Research for the direct purposes of resolving social issues and
creating economic value
Research to fulfill organizational goals (e.g., regional or social
contributions)
Research to achieve integration between different disciplines
Research in pursuit of temporary trends
Research on basic technologies required by industry (material
testing, etc.)
Research that grasps fragmented knowledge from a comprehensive
viewpoint
Ground-breaking research to produce new research fields
Exploratory research to find new research themes
Note: Respondents were asked to indicate the degree of change compared to 2005 by selecting “decreased greatly,” “decreased,” “no
change,” “increased,” and “increased greatly.” The results obtained after indexation were as follows: “decreased greatly” (-10
points), “decreased” (-5 points), “no change” (0 points), “increased” (5 points), “increased greatly” (10 points).
12
3. Situational changes of Japanese STI system during the fourth STBP

13. • The results of the NISTEP Teiten survey showed that respondents
felt that some progress was made in resolving important issues
during the fourth STBP.
– Implementations of strategies and national projects.
– Concentration on core competence in government-led research and
development
– Increase of research for the direct purposes of resolving social issues
and creating economic value.
• However, there is growing concern regarding fundamental
research activities at universities and PRIs.
– Baseline funding in universities / PRIs.
– Degree of diversity in Japanese basic research.
13
Changes in the status of Japanese STI
between 2011–2015
3. Situational changes of Japanese STI system during the fourth STBP

14. • This paper presented a method for qualitatively determining the
status of STI through a panel survey to researchers and experts
(NISTEP Teiten survey).
• NISTEP Teiten survey is effective in comprehensive monitoring the
STI system’s status.
• The data provided by these surveys are useful in formulating
science and technology policies, and have been referenced in
official documents by various governmental councils and
committees.
• Utilizing qualitative data from the NISTEP Teiten surveys along
with R&D statistics would promote a fuller understanding of STI
systems.
Conclusions
5. Conclusions and future works
14

15. Other facts of the NISTEP Teiten survey
5. Conclusions and future works
13 reports in five years
2 workshops
2 infographics
16 newspaper articles
2.5 million characters in FAs
(equivalent to 25 paperbacks)
15

16. • Fifth STBP started from FY2016.
• Preparation for a new survey is
undergoing.
• Fifth STBP identified numerical
targets/indicators that will be
monitored during the plan.
• Original evidence that NISTEP
Teiten survey provides is getting
more important in order to
understand a context of changes
of the indicators.
Future works
5. Conclusions and future works
16