Paper 9: How to increase one's ranking efficiently? (Xu)Presentation Transcript
How to increase one's ranking “e fficiently ” ? - Decision Pathway Sensitivity Analysis Fang Xu Kent Business School, UKC Centre for the Evaluation of Research Performance (CERP) 3 rd Workshop on IWIPM, 1 st July, 2010, UKC
If “iceberg tip” type of data (like Nobel Prize numbers, top awards, top journals etc.) are given heavy weights in a ranking system, the results often are quite sensitive to the data perturbation – one extra award may just change the rankings so much.
An example of CAS International Institution Ranking Research
Great change of scores can be found in the results due to data perturbation.
Final Results Change Rate Institutes Data Perturbation (+/-) 5% Institute 1 64.86% Institute 2 56.02% Institute 3 19.78% Institute 4 22.41% Institute 5 36.05% Institute 6 10.16% Institute 7 13.54% Institute 8 17.63% …… . …… .
One of the possible remedy…
To use data from different part of the “iceberg” – from the base to middle to tip – important is the shape of the iceberg, not necessary the most visible part – tip.
E.g. in CAS study, indicators aimed to describe the middle part of the “iceberg” have been applied, such as top 1% publications , top 10% publications etc.
More details have been introduced by Prof. Steve Liu .
On the other hand…
This sensitivity does raise another issue:
it may be possible to increase one’s rank rapidly by invest on these sensitive areas;
E.g. recruit a small number of certain type of personnel ( N obel prize winners etc)
Influence to decision makers
In this talk, we show some initial analysis in this regard.
Decision pathway analysis
To simulate such decision scenarios and analyze how one’s rank change;
From the comparison and analysis of the simulation results, it is possible to find out “efficient” way to increasing one’s rank for an organization.
We will illustrate our idea using Academic Ranking for World Universities (ARWU) produced by Shanghai Jiaotong University and World Famous University (WFU) produced by Wuhan University.
Raw data is necessary for simulation analysis;
Raw data can be recovered via a sample (e.g. Harvard University) using the methods stated;
However, Răzvan Florian show that accurate raw data is hardly recovered in some ranking scheme s .
( Răzvan Florian, Ad Astra 5, 2006,
Several scenarios are considered
Single indicator perturbation
Dual indicators perturbation
Chain influence of multi-indicators perturbation (Decision Pathway sensitivity analysis)
One more Award winner Two more Award winners Two more HICI researchers One more HICI researcher Change in Rank? One more N&S Pubs Two more N&S Pubs
Criteria and weights of ARWU ranking
Criteria Indicator Code Weight Quality of Education Alumni of an institution winning Nobel Prizes and Fields Medals Alumni 10% Quality of Faculty Staff of an institution winning Nobel Prizes and Fields Medals Award 20% Highly cited researchers in 21 broad subject categories HiCi 20% Research Output Papers published in Nature and Science N&S 20% Papers indexed in Science Citation Index-expanded and Social Science Citation Index PUB 20% Per Capita Performance Per capita academic performance of an institution PCP 10% Total 100% For institutions specialized in humanities and social sciences such as London School of Economics, N&S is not considered, and the weight of N&S is relocated to other indicators.
Data sensitivity of ARWU ranking --- Single indicator perturbation
Influence of sample universities’ ranks in the case of single indicator perturbation in an interval .
Change Rate HICI PUB A University B University A University B University 2.50% 2 2 11 7 5.00% 7 5 16 14 7.50% 14 7 25 22 10.00% 16 8 30 26 12.50% 28 13 36 32
Data sensitivity of ARWU ranking --- multi- indicators perturbation
Influence of A university’s ranks in the case of multi indicators perturbation in an interval .
Data sensitivity of ARWU ranking --- Chain Influence of multi indicators perturbation Scenarios Decision Pathway Description Change in Rank A University B University Scenario 0 0 Award, 0 HICI winner, 0 N&S Pubs - - Scenario 1 0 Award,1 HICI winner 11 5 Scenario 2 0 Award, 1 HICI winner, 2 N&S Pubs (SSCI) 15 8 Scenario 3 1 Award & HICI winner 70 45 Scenario 4 1 Award & HICI winner, 2 N&S Pubs (SSCI) 72 60 Scenario 5 1 Award & HICI winner, 6 N&S Pubs (SSCI) 83 82 Scenario 6 2 Award & HICI winners 118 82 Scenario 7 2 Award & HICI winners, 8 N&S Pubs (SSCI) 130 110 Scenario 8 2 Award & HICI winners, 16 N&S (SSCI), 20 other SSCI Pubs 144 127
Criteria of World Famous University (WFU) Description Indicators Weights Research productivity Total publications 0.3 Total citations 0.2 Research impact Highly cited papers 0.2 Number of subjects in the rank 0.05 Research innovation patents 0.05 Hot papers 0.1 Research development Percentage of highly cited papers 0.1
Data sensitivity of WFU ranking --- Chain Influence of multi indicators perturbation Scenarios Decision Pathway Description Change in Rank of C University Scenario 1 1 more Highly Cited paper 11 Scenario 2 1 more Highly Cited paper, 1 more Hot paper 48 Scenario 3 1 more Highly Cited paper, 1 more Hot paper & 1 more patents 48 Scenario 4 2 more Highly Cited papers 66 Scenario 5 2 more Highly Cited papers, 2 more Hot papers 97 Scenario 6 2 more Highly Cited papers, 2 more Hot papers & 2 more patents 98
Conclusions & Limitations
Data sensitivity has been estimated of two ranking schemes, significant changes have been detected in the case of single indicator perturbation, and dual indicators perturbation.
Particularly, more attention has been paid to decision pathway sensitivity analysis, where three universities have been selected for an illustration.
Unavoidable errors exist due to difficulties in data recovery in some ranking scheme s .
Thanks to the useful discussions with AWRU and WFU groups!