CWTS is a research institute at Leiden University Faculty of Social Sciences.
CWTS is/was based mainly on contract research, but still produces roughly 10 papers in scientific journals per year.
Currently in a transformation process, based on a mix of public funding and contract research, focused on both fundamental research in the field and service contracts.
The work of CWTS
CWTS conducts from the early 1990’s large scale bibliometric research performance analyses that accompany research assessments by review committees
Many of these studies had a disciplinary character, initiated by the VSNU.
A revision of the evaluation protocol (SEP) in 2003 caused a shift of the analyses to university- or institute-initiated studies.
Introduction of bibliometrics
Bibliometrics can be defined as the quantitative analysis of science and technology performance and the cognitive and organizational structure of science and technology.
Basic for these analyses is the scientific communication between scientists through (mainly) journal publications .
Key concepts in bibliometrics are output and impact , as measured through publications and citations .
Important starting point in bibliometrics: scientists express, through citations in their scientific publications, a certain degree of influence of others on their own work.
By large scale quantification, citations indicate influence or (inter)national visibility of scientific activity .
CWTS data system
CWTS has a full bibliometric license from Thomson Reuters Scientific to conduct evaluation studies using the Web of Science
Our database covers the period 1981-2007.
500.000.000 million citation relations between source papers
48.000.000 authors (incl variations)
28.000.000 million addresses, some 90% cleaned up over the last 10 years.
Contains reference sets for journal and field citation data
Part I: Various databases
Differences between Citation Indexes: implications for bibliometric studies Martijn S. Visser and Henk F. Moed
Multidisciplinary Citation Indexes
Web of Science
since 1963, formerly produced by ISI
ca. 9,000 Journals are indexed
launched by Elsevier in 2004
ca. 15,000 journals, conf papers and other
launched in 2004
Disciplinary distribution of journals in WoS
Roughly 5.000 journals from natural-, life-, medical- and technical sciences.
Roughly 2.500 journals from the social- and behavioral sciences.
Roughly 1.500 journals from the humanities.
Comparing Web of Science and Scopus on a paper by paper basis
RAE 2001: Coverage differences in a practical situation
Implications for bibliometric studies: case study in the field of oncology (SCImago)
Comparing WoS & Scopus
Time period: 1996 – 2006
Snapshot in time
Only citeable documents
Scopus Matching WoS with Scopus on a paper by paper basis Intersection 9.4 M Web of Science Papers 1996 – 2006: In Scopus 14.7 M In WoS 10.5 M Overall overlap 59% WoS overlap with Scopus 90% Scopus overlap with WoS 64%
I: From a WoS perspective
To what extent are citeable documents (articles, letters, notes and reviews) in journals processed for the WoS covered by Scopus?
Scopus coverage of WoS papers increases over time
Incomplete coverage of WoS journals decreases over time % WoS Journals In 2005 82% of WoS journals were completely covered in Scopus In1996 55% of WoS journals were completely covered in Scopus WoS journals not covered by Scopus at all
II: From a Scopus perspective
To what extent are citeable documents (articles, notes, letters, reviews, conference papers) in sources processed for the Scopus not covered by Web of Science? (=Scopus Surplus)
NATURAL AND LIFE SCIENCES
ENGINEERING SCIENCES, SOCIAL SCIENCES & HUMANITIES
III: From an external perspective
To what extent are documents in an external file processed for the WoS and/or Scopus?
Coverage differences in a practical case: 2001 RAE papers
Up to 4 publications of every active faculty staff member
Time Period 1994/1996 – 2000
Papers assigned to units of assessment
Only small overall coverage differences Domain Total nr Papers in Wos in Scopus +/- Science 95,056 84.1% 84.4% +0.3 Mathematics 6,634 81.8% 80.1% -1.7 Social Science and Humanities 91,324 24.9% 25.9% +1.0
Coverage differences for Units of Assessments (Sciences)
Scopus –WoS > 3%
Other Studies and professions allied to Medicine
Mineral and Mining Engineering
Wos – Scopus > 3%
Food Science and Technology
Why are the differences relatively small?
‘ Best’ papers are more likely to be published in high impact journals which tend to be processed by both indexes
British academics may not often use sources solely processed by Scopus
Need to characterize the surplus coverage of Scopus
Case study for journals in Oncology
Cooperation between SCImago and CWTS (Carmen Lopez Illescas )
Comparison between Scopus and Web of Science
Comparison on a journal-by-journal basis
Overlap from a WoS perspective: WoS cancer categories No. WoS journals WoS journals in Scopus Oncology 126 112 (89%) In canc catg 14 (11%) In other catg 0 (0%) Not in Scopus Total no. journals 126
Overlap from a Scopus perspective: Scopus cancer catgs No. Scopus journals In both categs Scopus journals in WoS Oncology 167 75 112 (48%) Canc catgs 13 (6%) Other catgs Cancer Research 139 106 (46%) Not in WoS Total no. journals 231
Characterization of Scopus surplus: Language and Publisher country Data source: Ulrich Journals in WoS and Scopus (n=112) Journals in Scopus but not in WoS (n=106) English language 94% 72% Number of different languages 4 15 Publisher in: USA; UK; Netherlands 78% 51%
Characterization of Scopus surplus: other journal properties Data source: Ulrich Journals in WoS and Scopus (n=112) Journals in Scopus but not in WoS (n=106) Nº % Nº % Started year bet. 1996-2006 17 15.2 57 53.8 Media online free 1 0.9 10 9.4 Refereed 108 96.4 72 67.9 Frequency 12 or more a year 69 61.6 31 29.2
Main characteristics Scopus surplus
More journals in non-English languages
More recently established
Less often refereed
Relatively low impact factors
Implications for bibliometric studies
Web of Science tends to select only journals with sufficient high impact (research front)
Scopus tends to be more representative of the total of scientific literature
The position of the Western World in Scopus is probably less dominant
Strong relationship between nr WoS and Scopus papers for the 50 most productive countries
Average Citation rate in WoS and Scopus for the 50 most productive countries
Negative relationship between number of papers in Scopus surplus and average citation rate
Effect of Scopus surplus:
Countries that profit most in terms of percentage of published documents tend to show a decline in their average citation rate
Keeping in mind the RAE outcomes, the effect of the extension is also that the countries publishing relatively often in non-English journals, show a decline in their average citatioin rate
Conclusion: ‘ More is not necessarily better !’
Citation indexes (will) adopt a more inclusive coverage policy in which citation impact is less important as a criterion for selection.
This will have implications
for the way bibliometric assessments of research performance have to be carried out
for the interpretation of bibliometric indicators and rankings derived from these databases
Should bibliometric studies aim for widest possible coverage?
- It depends on what you want to measure.
How should one deal with possible bias as a result of changing coverage policies?
Define sub-universes (journal sets) of publications and citations (e.g. national /international)
Adequacy of citation indexes : implications for bibliometric studies
How to tackle this issue ?
We conduct analyses on the adequacy of the citation indexes across disciplines based on reference behavior of researchers themselves.
The degree of referring towards other indexed literature indicates the importance of journal literature in the scientific communication process.
The medical & Life sciences
The natural sciences
The technical sciences
The social– and behavioral sciences
Overall WoS coverage by main field EXCELLENT (> 80%) VERY GOOD (60-80%) GOOD(40-60%) Biochem & Mol Biol Appl Phys & Chem Mathematics Biol Sci – Humans Biol Sci – Anim & Plants Economics Chemistry Psychol & Psychiat Engineering Clin Medicine Geosciences MODERATE (<40 %) Phys & Astron Soc Sci ~ Medicine Other Soc Sci Humanities & Arts
Conclusions on adequacy issue
We can clearly conclude that the application of bibliometric techniques, solely based on WoS (but very likely also Scopus) will not be valid for some of the ‘soft’ fields in the social sciences and the humanities.
That is why the tool box has to be extended !
Part II: Bibliometric indicators one encounters in the field
Some basic indicators are …
P : number of publications in journals processed for the Web
C : number of received citations, excl. self-citations.
CPP: mean number of citations per publication, excl. self-
Pnc: percentage of the publications not cited (within a
certain time-frame !!!)
% SC : percentage self-citations related to an output set.
ISI Impact Factors: calculation and validity
Methodology: ISI’s classical IF
The ISI Impact Factor (IF) is defined as the number of citations received by a journal in year t, divided by the number of citeable documents in that same journal in the years t-1 and t-2,
Or, as a Formula:
Citations in year t Number of ‘citeable documents’ in t -1 & t -2
Practical exercise on IF
Here we get to the first practical exercise of the afternoon
Read the Science commentary on Impact Factors.
Calculate the Impact Factor for ‘the Lancet’
Discuss the results.
Share ‘citations-for-free’ for The Lancet
Article 784 2986
Note 144 593
Review 29 232
Sub-total 957 (a) 7959 (b)
Letter 4181 4264
Editorial 1313 905
Other 1421 909
Total 7872 14037 (c)
ISI Method :
Citations in 2000 .
Citeable documents in ‘98 and ‘99
CWTS Method :
Citations to Art/Not/Rev in 2000 .
Art/Not/Rev in ‘98 and ‘99
Citations to Art/Let/Not/Rev in 2000 .
Art/Let/Not/Rev in ‘98 and ‘99
ISI Impact Factors
From 1995 onwards CWTS has analyzed the uses and validity ISI Journal Impact Factor (IF).
Most important points of criticism were:
Calculated erroneously .
Not sensitive for the composition of the journal in terms of the document types.
Not sensitive for the science fields a journal is attached to …
Based on too short ‘citation windows’ .
Distribution of citations used for the calculationof the IF value of The Lancet
The IF-score of The Lancet is seriously ‘overrated’ by the scientific ‘audience’ of the journal.
The red area indicates citations ‘for free’, while the blue area indicates ‘correct citations ’
Impact Factoren voor Br. J. Clin. Pharm. en Clin. Pharm. & Ther.
The graph shows the correct and erroneous impact factors of BJCP and CPT
In the case of CPT , citations to published meeting abstracts are included, while BJCP has stopped publishing of meeting abstracts !
Document types and fields Field Journal IF JFIS The IF is for ‘02, JFIS covers ‘98-‘02 IMMUNOLOGY ANN REV IMMUNOL 50.49 1 5.18 1 BIOCHEM & MOLECULAR BIOL ANN REV BIOCHEM 34.61 1 4.10 3 PHARMACOL & PHARMACY PHARMACOLOGICAL REV 27.74 1 4.75 1 CELL BIOL ANN REV CELL & DEVELOPM BIOL 27.53 1 1.72 13 DEVELOPMENTAL BIOL ANN REV CELL & DEVELOPM BIOL 27.53 1 1.72 3 PHYSIOLOGY PHYSIOLOGICAL REV 24.82 1 3.18 1 CELL BIOLOGY NATURE REV MOL CELL BIOL 22.21 4 2.76 8 ENDOCRINOL & METABOLISM ENDOCRINE REV 21.98 1 2.87 1 NEUROSCIENCES ANN REV NEUROSCIENCE 21.89 1 3.12 4 PHYSICS REV MODERN PHYSICS 20.14 1 5.02 1 CHEMISTRY CHEMICAL REV 19.67 1 2.89 2
International rankings of university performance
Various academic rankings
ARWU or Shanghai Ranking
CHE Excellence Ranking
CWTS University Ranking
Composition of Shanghai ranking
Nobel prizes and field medals by alumni 10% .
Nobel prizes and field medals by staff 20%
Highly cited staff in 21 disciplines 20%
Articles published in Nature & Science 20%
Articles published in citation indexes 20%
Per capita performance on those indicators 10%
Composition of THES ranking
Academic peer review 40% .
Employer review 10%
Faculty – Student ratio 20%
Citation per faculty 20%
International faculty 5%
International students 5%
Composition of CHE ranking
Size indicator : output volume in citation indexes.
Perception indicator : citations (in relation to an international standard)
Beacon indicator: Number of often-cited staff & Nobel prize winners at the university
Europe indicator : number of projects in the Marie Curie research promotion programme of the EU
No weights are indicated, rankings are applied on four fields (biology, chemistry, mathematics, & physics), data delivered by CWTS.
Important CWTS standard indicators
CPP/JCSm : ratio between real, actual impact, and mean journal impact.
CPP/FCSm : ratio between real, actual impact, and mean field impact.
JCSm/FCSm: ratio between journal impact, and field impact, indicative for the ‘quality’ of the journal package in the field
Composition of CWTS ranking
Yellow ranking by size, the number of publications .
Green ranking by size-independent, field-normalized citation impact (‘Crown indicator’).
Orange ranking by the size-dependent ‘brute force’ impact indicator, the multiplication of P with the university’s field-normalized average impact
Blue ranking by the ‘simple’ citations-per-publication indicator ( CPP ) 0%
Pink ranking by the ‘simple’ citations-per-publication indicator ( CPP ) for the top-50 ranking institutes on size !
Practical exercise on rankings
Here we get to the third practical exercise of the afternoon
Which disadvantages can you think of, when considering the indicators used in the various rankings ?
Discuss the results.
The H-Index and its limitations
The H-Index, defined as …
The H-Index is the score that indicates the position at which a publication in a set, the number of received citations is equal to the ranking position of that publication.
Idea of an American physicist, J. Hirsch, who published about this index in the Proc. NAS USA.
Examples of Hirsch-index values
Environmental biologist, output of 188 papers, cited 4,788 times in the period 80-04.
Hirsch-index value of 31
Clinical psychologist, output of 72 papers, cited 760 time sin the period 80-04.
Hirsch-index value of 14
Problems with the H-Index
For serious evaluation of scientific performance, the H-Index is as indicator not suitable, as the index:
Is insensitive to field specific characteristics (e.g., difference in citation cultures between medicine and other disciplines).
Does not take into account age and career length of scientists, a small oeuvre leads necessarily to a low H-Index value.
Actual versus field normalized impact (CPP/FCSm) displayed against the output.
Large output can be combined with a relatively low impact
H-Index displayed against the output.
Larger output is strongly correlated with a high H-Index value.
Part III Methodological issues
Journal & Field Normalization
Network of publications (nodes ) linked by citations (edges ) Lower citation-density Higher citation - density e.g., applied research, e.g., basic natural social sciences medical research FCSm JCSm CPP Values for normalization