Most people assume that highly cited papers are "innovative". Using survey results we show that most highly cited papers exemplify normal progress rather than innovation. We also attempt to correlate various indicators with those papers classified as innovative by their authors. Most of these correlations are very weak.

1. Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities
Indicators of Innovative Research
2014 Science & Technology Indicator Conference
Leiden University
The Netherlands
Sept 3, 2014
Richard Klavans (presenter)
Kevin W. Boyack
Henry Small
Aaron A. Sorensen
John P.A. Iaonnidis
SciTech Strategies Berwyn, PA.
SciTech Strategies, Albuquerque, NM
SciTech Strategies, Bala Cynwd, PA.
Temple University School of Medicine, Philadelphia, PA.
Stanford University School of Medicine, Stanford, CA.

2. 2
Motivation
“What gets us into trouble
is not what we don't know.
It's what we know for sure
that just ain't so.”
Mark Twain
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

3. 3
Motivation
Ioannidis et al [Nature, 2013] assumes that
highly cited papers are more innovative
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

4. 4
Motivation
Ioannidis et al [Nature, 2013] assumes that
highly cited papers are more innovative
We hope that the ‘distance’ between
a citing paper & its references will
correlate with innovativeness
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

5. Ioannidis et al [Nature, 2013] assumes that
highly cited papers are more innovative
We hope that the ‘distance’ between
a citing paper & its references will
correlate with innovativeness
Uzzi et al. [Science, 2013]
argue that one needs a
combination of
typical & atypical
knowledge relationships
in order to be innovative.
He also assumes that
innovative papers are
highly cited.
5
Motivation
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

6. 6
Is this so?
 Invited 400 biomedical researchers to participate in a
survey
» Invitation came from Ioannidis (Stanford University)
» Researchers had over 25,000 citations and H index > 75
» Participation rate was 30.8% (123 out of 400)
 Researchers were asked to rate their top 10 (most cited)
papers
» Papers had to be published between 2005-2008 (& in the Scopus
database)
» Ratings were along six dimensions of impact
 Survey results will be compared with bibliometric
indicators
» Traditional Indicators
» Indicators suggested by K&B [2013]
» Indicators suggested by Uzzi et al. [2013]
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

7. 7
Survey Design
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

8. 8
Survey Analysis
 Assigned Papers to Impact Categories
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

9. 9
Bibliometric Data*
Common Sense
1. Review Paper?
2. # of References
3. Team Science (# of Authors)
K&B [2013]
4. Distance between citing paper and cited references
USMJ [2013]
5. Typical Knowledge Relationships
6. Atypical Knowledge Relationships
* Scopus Data
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

10. 10
Finding #1:
 Most High Impact Papers represent
‘Normal Science’
Type of Impact # of Papers % of Sample
Progress 335.4 28.8%
Synthesis 262.9 21.3%
Broader Interest 220.0 17.8%
Innovation 195.9 15.9%
Surprise 99.3 8.0%
Difficulty 41.6 3.4%
Unassigned 58.0 4.7%
68.9%
27.3%
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

11. 11
Finding #2:
 Traditional indicators work
(but ‘team science’ is associated with the less
innovative papers!)
Type of
Impact
% Review # Ref #Authors
Progress 12.5 51.7 17.8
Synthesis 26.1 71.0 16.2
Broader
17.3 59.8 15.5
Interest
Innovation 6.3 42.9 12.3
Surprise 4.5 41.7 11.4
Difficulty 12.7 51.1 9.8
Unassigned 12.1 55.0 14.9
F-stat 13.8 9.63 1.90
Probability .0000 .0000 .0773
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

12. 12
Finding #3:
 ‘Distance’ does not identify innovative
papers
Type of Impact Distance
Progress 2.05
Synthesis 2.01
Broader Interest 2.13
Innovation 2.05
Surprise 2.04
Difficulty 1.98
Unassigned 2.04
F-Stat .42
Probability .867
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

13. 13
Finding #4:
 Only atypical knowledge relationships
works
Type of
Impact
Atypical Typical
Progress .491 .511
Synthesis .506 .480
Broader
.574 .475
Interest
Innovation .636 .467
Surprise .608 .500
Difficulty .620 .451
Unassigned .672 .500
F-stat 4.61 .41
Probability .0001 .875
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

14. 14
Finding #5
 And the Overall Explanatory Value is very low
(R2~.033)
Regression Equation
Dependent Innovation or Surprise (0,1 variable)
Variable:
t P>|t|
Independent Review Paper (0,1) -5.14 .000
Variables: References Log(# ref) -2.08 .037
Team Log(# authors) -1.81 .071
Atypical (0,1) 3.87 .000
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

15. 15
highly cited papers are not necessarily innovative.
Implications
‘distance’ didn’t work.
Atypical knowledge
relationships works….
but why?
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

16. 16
Implications
 And we still don’t know how to identify
innovative papers
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities

17. 17
Thank-you for your attention!
Physics Chemistry Engineering Computer Earth Biology Disease Medicine Brain Health Social Humanities