Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Measure up!
The extent author-level bibliometric indicators are
appropriate measures of individual researcher
performance
...
OVERVIEW
1. Background
2. Characteristics and effects of ALI
(Papers 1, 2, 4)
3. The appropriateness of ALI across differe...
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
“ Bibliometrics is the
application of vario...
4
1980s 1990s 2000s 2010s1970s1960s1950s
de Solla Price
growth model
Namilov
interdisciplinary
approach to indexing
scient...
5
Managerial
Teaching
Outreach
Societal
Technical Skills
Funding/Grants
Productivity
Quality
Relevance
Visibility
Reputati...
DEFINTITIONS OF AU, P & C IN ALI
Authorship, publication and citation is different from
discipline to discipline, from tim...
EVALUATION
OUTPUT
DATA
ALI
MODEL
MOTIVATION TO PUBLISH
MOTIVATION TO CITE
INTERPRETATION OF ALI
LEGITIMATE LINK TO REAL
WO...
1. What are the characterisitics of ALI of academic
performance?
2. To what extent are ALI appropriate in the
evaluation o...
9
ALI GIVE A SNAP-SHOT OF SELF
IMAGE AND CORE PERSONALITY
TRAITS
COMPARING RESEARCHERS
CAN EXPOSE THE
INDIVIDUAL
INDIVIDUA...
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
INTERNAL CHARACTERISTICS OF ALI
ALI are designed to measure particular aspects of
the effect of a researcher’s work – over time, to
field, as quality, ran...
FAMILY CHARACTERISTICS
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
METHOD:
Compute simp...
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
FAMILY CHARACTERISTICS
DISCIPLINE CENTRAL I...
METHOD:
Publication and citation data for 512 researchers
22,143 papers, 423, 371 citations (WOS)
52,227 papers, 746,985 c...
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
hg produced highest level of agreement in r...
hg fulfills its potential as a ranking indicator (Alonso et al
2010) yet this tells us nothing about the excellence of the...
METHOD:
Same data as previous studies
Explorative statistical study of distribution; two step cluster;
ordinal regression:...
CLUSTER 1
below IQR, high nnc, sc, low
P & C. Adjusting for academic
age mg, m.quotient,
currency of cites improved
scores...
ASTRO HEALTHPHILENVIRO
h2 SUM PP
TOP
PROP
Q2 e
GROUPING CHARACTERISTICS
Ordinal regression:
Academic age is statistically ...
1. Simple ALI are complex!
2. Basic descriptive statistics can be more informative
3. There are families of indicators
4. ...
Knowing all these practical challenges, what
properties should a well constructed indicator
contain?
METHOD:
Based on pape...
THE 3 CRITERIA OF PROFESSOR YVES GINGRAS
1. Adequacy of the indicator to the object it
measures
-quantity of items a resea...
10
28
HOMOGENEITY
ADEQUACY INERTIA
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
17 did n...
OPERATIONALIZATION OF AU, P, C
”THE LOGIC GRID”
METHOD:
• Extract definitions and hypotheses in ALI
• Matrix of definition...
HIRSCH QUALITY IMPACT USE OVER TIME IDEA TRANSFER
CITATION
6 6 6 5 2
PERFORMANCE EFFECT INFLUENCE POPULARITY REWARD NO DEF...
LOGIC GRID
http://tinyurl.com/ofm7h8s
• ALI are designed for specific author and
publication types
• Majority of ALI designed for papers in WoS
• Some indicator...
1. Transparency
Know which data is used to compute the indicator
Understand the math and the inferences
2. Demographics
Be...
RQ 1: What are the characterisitics of ALI of academic
performance?
ALI are complex in character. They use different:
• di...
RQ 2: To what extent are ALI appropriate in the
evaluation of researchers from different disciplines and
different academi...
RQ 3: To what extent are the concepts being measured
defined in the construction of ALI?
• Many indicators are poorly or o...
THANK YOU!!
ACUMEN partners: especially
Judit Bar-Ilan, Frank van der
Most & Paul Wouters.
My supervisors: Jesper
Schneide...
Upcoming SlideShare
Loading in …5
×

Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance

513 views

Published on

PhD defense about the appropriate use and validity of bibliometric indicators at the individual level. The presentation considers the familial relationships between indicators, the extent concepts being measured are captured by the indicator model as well as the general characteristics of the indicators and the profile of the researchers they are designed to measure

Published in: Leadership & Management
  • Be the first to comment

Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance

  1. 1. Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance Lorna Wildgaard Royal School of Library and Information Science Faculty of the Humanities, CPH U 27th November 2015
  2. 2. OVERVIEW 1. Background 2. Characteristics and effects of ALI (Papers 1, 2, 4) 3. The appropriateness of ALI across different disciplines and different academic seniorities (Papers 3, 5, 6, 7) 4. Concept definintion in the construction of ALI (Chapter 6 & Appendix B, http://tinyurl.com/nj4mvca) 5. Conclusions Background Preliminary Analyses Empirical Analyses Validation Study Conclusion CORRECTION
  3. 3. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion “ Bibliometrics is the application of various statistical analyses to study patterns of authorship, publication, and literature use.” (Lancaster, 1977). “[…] to analyse the structure of science, measure science, and indicate the production, citations and collaboration of researchers, institutions and countries.” (De Bellis 2014; Pritchard 1969).
  4. 4. 4 1980s 1990s 2000s 2010s1970s1960s1950s de Solla Price growth model Namilov interdisciplinary approach to indexing scientific literature Merton Social structure of scientific communication Haitun; Yablonsky distribution of citations Garfield SCI & IF h-index Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  5. 5. 5 Managerial Teaching Outreach Societal Technical Skills Funding/Grants Productivity Quality Relevance Visibility Reputation Scientific Counting, adding, dividing, multiplying an authors’ publications & citations Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  6. 6. DEFINTITIONS OF AU, P & C IN ALI Authorship, publication and citation is different from discipline to discipline, from time to time and location to location and has implications in indicator development (Bošnjak and Marušić, 2012) What constitutes a publication [author and citation] should be clearly defined to ensure representative operationalization in the indicator and the extraction of meaningful relationships (Wouters 1999; Skupin 2009; Colledge 2012) Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  7. 7. EVALUATION OUTPUT DATA ALI MODEL MOTIVATION TO PUBLISH MOTIVATION TO CITE INTERPRETATION OF ALI LEGITIMATE LINK TO REAL WORLD PERFORMANCE OF RESEARCHER IN SYSTEM REDEFINES WHAT SUCCESS IS EXOGENOUS VARIABLES WHAT IS MISSING INDEXING POLICY Background Preliminary Analyses Empirical Analyses Validation Study Conclusion THE PROBLEM
  8. 8. 1. What are the characterisitics of ALI of academic performance? 2. To what extent are ALI appropriate in the evaluation of researchers from different disciplines and different academic seniorities? 3. To what extent are the concepts being measured defined in the construction of ALI? Background Preliminary Analyses Empirical Analyses Validation Study Conclusion RESEARCH QUESTIONS
  9. 9. 9 ALI GIVE A SNAP-SHOT OF SELF IMAGE AND CORE PERSONALITY TRAITS COMPARING RESEARCHERS CAN EXPOSE THE INDIVIDUAL INDIVIDUALS USE ANY DATA TO INCREASE ALI SCORES, TO INCREASE THEIR SUBJECT VALIDITY & SELF- WORTH ALI BRING OBJECTIVITY TO THE EVALUATION & REDUCE GENDER; CULTURAL AND RACIAL BIAS DOCUMENTING BEING OUT- PERFORMED IS DETRIMENTAL TO A RESEARCHER’S SELF-DEFINITION ALI DO NOT ADD CONTEXT BUT CAN ADD REDUNDANT INFORMATION SUCCESS IS DEFINED AS WELL IN THE SYSTEM EXTERNAL CHARACTERISTICS OF ALI
  10. 10. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion INTERNAL CHARACTERISTICS OF ALI
  11. 11. ALI are designed to measure particular aspects of the effect of a researcher’s work – over time, to field, as quality, ranking all or selected works, co- authorship etc. Judgements based on ALI can lead to assumptions about the productivity and impact of a researcher, which can be unsubstantiated, and affect the psychological character of the individual. ALI have to be methodologically sound PRELIMINARY ANALYSIS: RESULTS Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  12. 12. FAMILY CHARACTERISTICS Background Preliminary Analyses Empirical Analyses Validation Study Conclusion METHOD: Compute simple indicators on publication and citation data of 750 researchers (585 m, 165w) 4 disciplines, 5 academic seniorities Correlation (Kendall’s Tau) as measure of association between seniority and ALI, discipline by discipline ALSCAL IBM SPSS v.19 ENDPOINT Suitability of indicators in different disciplines and seniorities
  13. 13. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion FAMILY CHARACTERISTICS DISCIPLINE CENTRAL INDICATOR S-STRESS % VARIANCE ASTRONOMY hg 0.37 25 ENVIRONMENTAL SCIENCE h, h2 0.37 24 PHILOSOPHY IQP 0.38 47 PUBLIC HEALTH g 0.49 38
  14. 14. METHOD: Publication and citation data for 512 researchers 22,143 papers, 423, 371 citations (WOS) 52,227 papers, 746,985 citations (GS) For each researcher 17 ALI calculated Agreement: determined by matching rank positions Variability of rank position determined by standard deviation of the difference in scholar rank position, calculated from the matched pairs END POINTS: agreement in ranking between WOS & GS RANKING CHARACTERISTICS Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  15. 15. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion hg produced highest level of agreement in rankings in WOS & GS, 0.8-0.9 tau across disciplines and seniorities (Patel 2013)
  16. 16. hg fulfills its potential as a ranking indicator (Alonso et al 2010) yet this tells us nothing about the excellence of the researcher. h and g cannot be rationally combined to produce indications of research excellence (Franceschini & Maisano 2011). Indicator rankings inform of the researcher’s visibility in the citation index not in their academic community (Bar-Ilan 2008; De Battisti 2012; Farhadi et al 2013). Background Preliminary Analyses Empirical Analyses Validation Study Conclusion RANKING CHARACTERISTICS
  17. 17. METHOD: Same data as previous studies Explorative statistical study of distribution; two step cluster; ordinal regression: odds ratios IBM SPSS v.22 ENDPOINT: Identify disciplinary & seniority appropriate indicators Background Preliminary Analyses Empirical Analyses Validation Study Conclusion GROUPING CHARACTERISTICS
  18. 18. CLUSTER 1 below IQR, high nnc, sc, low P & C. Adjusting for academic age mg, m.quotient, currency of cites improved scores. CLUSTER 2 Median IQR, high scores on collaboration indicators CLUSTER 3 Top IQR, highest scores normalized for field, one SIG paper CLUSTER 4 Extreme outliers, highest C, cited quickly, highest htype scores
  19. 19. ASTRO HEALTHPHILENVIRO h2 SUM PP TOP PROP Q2 e GROUPING CHARACTERISTICS Ordinal regression: Academic age is statistically significant for cluster placement Within cluster rank position determined by the ratio P:h
  20. 20. 1. Simple ALI are complex! 2. Basic descriptive statistics can be more informative 3. There are families of indicators 4. ALI scores are highly influenced by the database, academic age and the fit of the data to the model 5. ALI are more appropriate in some disciplines than others 6. Different ALI produce stable rankings within and across databases. 7. ALI can be easily manipulated 8. Scores can be due to chance rather than practical importance. EMPIRICAL ANALYSIS: CONCLUSIONS Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  21. 21. Knowing all these practical challenges, what properties should a well constructed indicator contain? METHOD: Based on papers 1-7 Validate ALI using Gingras’ criteria Analyze operationalization of AU, P, C in ALI ENDPOINT: Recommendation of a set of disciplinary dependent indicators of scientific publication output Background Preliminary Analyses Empirical Analyses Validation Study Conclusion VALIDITY ANALYSIS
  22. 22. THE 3 CRITERIA OF PROFESSOR YVES GINGRAS 1. Adequacy of the indicator to the object it measures -quantity of items a researcher publishes is an indicator of production 2. Sensitivity to the intrinsic inertia of the object -changing values in the time interval correspond to the speed and direction of change in the object 3. Homogeneity of the dimensions of the indicator -h-index is a heterogeneous indicator: it mixes quantity of papers published with citation counts Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  23. 23. 10 28 HOMOGENEITY ADEQUACY INERTIA Background Preliminary Analyses Empirical Analyses Validation Study Conclusion 17 did not fulfill any of the criteria n=69 ALI 4 1 7 20
  24. 24. OPERATIONALIZATION OF AU, P, C ”THE LOGIC GRID” METHOD: • Extract definitions and hypotheses in ALI • Matrix of definitions of AU, P, C & aim of ALI AIM: Identify in a simple way the properties of ALI Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  25. 25. HIRSCH QUALITY IMPACT USE OVER TIME IDEA TRANSFER CITATION 6 6 6 5 2 PERFORMANCE EFFECT INFLUENCE POPULARITY REWARD NO DEF. 2 1 1 1 1 20 PUBLISHED SCIENTIST AUTHOR AWARD WINNER PUBLISHED & CITED SENIORITY NO DEF. AUTHOR 12 11 10 7 5 2 5 PAPER IN WOS PAPER IN OTHER INDEX PAPER OBJECTS WITH CITATION EXPRESSION PUBLICATIONS 30 15 6 1 1 EXCELLENCE EFFECT INDEPENDENCE COMPARE RANK CURRENCY MEASURE 9 7 7 6 6 4 GROWTH QUALITY DISTRIBUTION QUANTITY & QUALITY CAREER DURABILITY 3 3 2 2 1 1
  26. 26. LOGIC GRID http://tinyurl.com/ofm7h8s
  27. 27. • ALI are designed for specific author and publication types • Majority of ALI designed for papers in WoS • Some indicators are purely theoretical • Inherent bias in all ALI is that they are designed for researcher profiles befitting the hard sciences • Lack of definitions = methodological deficiency If developers cannot define the variables they are measuring, how can we? Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  28. 28. 1. Transparency Know which data is used to compute the indicator Understand the math and the inferences 2. Demographics Be aware if demographics affect the ALI scores 3. Motive ALI must fit the objectives of the evaluation 4. Diversity Choose ALI that fit the discipline 5. Openess Make the limitations of ALI explicit, use supplementary ALI PROPERTIES OF A WELL CONSTRUCTED ALI Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  29. 29. RQ 1: What are the characterisitics of ALI of academic performance? ALI are complex in character. They use different: • disciplinary perspectives, • objectives, • operationalization of variables, • requirements to data, and • mathematical models that favour different disciplines and seniorities. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion CONCLUSIONS one size does NOT fit all.
  30. 30. RQ 2: To what extent are ALI appropriate in the evaluation of researchers from different disciplines and different academic seniorities? • No seniority ALI were identified • There is a disconnection between the performance of researcher on the CV and in ALI. • Some ALI are more appropriate in some disciplines than others as grouping and ranking indicators or as indicators that isolate unique information Background Preliminary Analyses Empirical Analyses Validation Study Conclusion CONCLUSIONS
  31. 31. RQ 3: To what extent are the concepts being measured defined in the construction of ALI? • Many indicators are poorly or only partially defined • Decisions based on these ALI could turn out to be less effective than expected • If we do not know how the developer has done the measurement, we cannot repeat it, reflect on the meaning associated with the concept and compare results to previous findings Background Preliminary Analyses Empirical Analyses Validation Study Conclusion CONCLUSIONS
  32. 32. THANK YOU!! ACUMEN partners: especially Judit Bar-Ilan, Frank van der Most & Paul Wouters. My supervisors: Jesper Schneider, Peter Ingwersen & Birger Larsen PhD/Masters group: Jesper, Niels-Peder, Maria, Ole, Rikke, Sara, Sille, Lisa Sutherland & BADASS Researcher/teacher support: Johanne Maibohm, Susanne Acevedo, Pia Dithmar & Ragnhild Riis. My boys: Kim, Zander & Balthazar My opponents: Henk Moed Rodrigo Costas & Jeppe Nicolaisen

×