Paper 9: How to increase one's ranking efficiently? (Xu)
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Paper 9: How to increase one's ranking efficiently? (Xu)

on

  • 1,116 views

 

Statistics

Views

Total Views
1,116
Views on SlideShare
1,111
Embed Views
5

Actions

Likes
0
Downloads
8
Comments
0

1 Embed 5

http://blogs.kent.ac.uk 5

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Paper 9: How to increase one's ranking efficiently? (Xu) Presentation Transcript

  • 1. How to increase one's ranking “e fficiently ” ? - Decision Pathway Sensitivity Analysis Fang Xu Kent Business School, UKC Centre for the Evaluation of Research Performance (CERP) 3 rd Workshop on IWIPM, 1 st July, 2010, UKC
  • 2. Introduction
    • Input data sensitivity is an important issue for any ranking methods.
    • Results reliability
    • Influence of decision making
    • Normally we wish that a ranking system has a good stability with respect the data perturbation so to have stable ranking results.
  • 3.
    • If “iceberg tip” type of data (like Nobel Prize numbers, top awards, top journals etc.) are given heavy weights in a ranking system, the results often are quite sensitive to the data perturbation – one extra award may just change the rankings so much.
  • 4. An example of CAS International Institution Ranking Research
    • Great change of scores can be found in the results due to data perturbation.
    Final Results Change Rate Institutes Data Perturbation (+/-) 5% Institute 1 64.86% Institute 2 56.02% Institute 3 19.78% Institute 4 22.41% Institute 5 36.05% Institute 6 10.16% Institute 7 13.54% Institute 8 17.63% …… . …… .
  • 5. One of the possible remedy…
    • To use data from different part of the “iceberg” – from the base to middle to tip – important is the shape of the iceberg, not necessary the most visible part – tip.
    • E.g. in CAS study, indicators aimed to describe the middle part of the “iceberg” have been applied, such as top 1% publications , top 10% publications etc.
    • More details have been introduced by Prof. Steve Liu .
  • 6. On the other hand…
    • This sensitivity does raise another issue:
    • it may be possible to increase one’s rank rapidly by invest on these sensitive areas;
    • E.g. recruit a small number of certain type of personnel ( N obel prize winners etc)
    • Influence to decision makers
    • In this talk, we show some initial analysis in this regard.
  • 7. Decision pathway analysis
    • To simulate such decision scenarios and analyze how one’s rank change;
    • From the comparison and analysis of the simulation results, it is possible to find out “efficient” way to increasing one’s rank for an organization. 
    • We will illustrate our idea using Academic Ranking for World Universities (ARWU) produced by Shanghai Jiaotong University and World Famous University (WFU) produced by Wuhan University.
  • 8. Methodology
    • Data recovery
    • Raw data is necessary for simulation analysis;
    • Raw data can be recovered via a sample (e.g. Harvard University) using the methods stated;
    • However, Răzvan Florian show that accurate raw data is hardly recovered in some ranking scheme s .
    • ( Răzvan Florian, Ad Astra 5, 2006,
    • www.ad-astra.ro/journal )
  • 9.
    • Several scenarios are considered
    • Single indicator perturbation
    • Dual indicators perturbation
    • Chain influence of multi-indicators perturbation (Decision Pathway sensitivity analysis)
    One more Award winner Two more Award winners Two more HICI researchers One more HICI researcher Change in Rank? One more N&S Pubs Two more N&S Pubs
  • 10. Criteria and weights of ARWU ranking
    • …… ..
    Criteria Indicator Code Weight Quality of Education Alumni of an institution winning Nobel Prizes and Fields Medals Alumni 10% Quality of Faculty Staff of an institution winning Nobel Prizes and Fields Medals Award 20% Highly cited researchers in 21 broad subject categories HiCi 20% Research Output Papers published in Nature and Science N&S 20% Papers indexed in Science Citation Index-expanded and Social Science Citation Index PUB 20% Per Capita Performance Per capita academic performance of an institution PCP 10% Total 100% For institutions specialized in humanities and social sciences such as London School of Economics, N&S is not considered, and the weight of N&S is relocated to other indicators.
  • 11. Data sensitivity of ARWU ranking --- Single indicator perturbation
    • Influence of sample universities’ ranks in the case of single indicator perturbation in an interval .
    Change Rate HICI PUB A University B University A University B University 2.50% 2 2 11 7 5.00% 7 5 16 14 7.50% 14 7 25 22 10.00% 16 8 30 26 12.50% 28 13 36 32
  • 12. Data sensitivity of ARWU ranking --- multi- indicators perturbation
    • Influence of A university’s ranks in the case of multi indicators perturbation in an interval .
    Change In Final Scores HICI 2.50% 5.00% 7.50% 10.00% 12.50% PUB 2.50% 13 15 18 25 34 5.00% 16 23 25 33 40 7.50% 25 28 34 36 51 10.00% 33 34 38 48 59 12.50% 36 42 48 56 65
  • 13. Data sensitivity of ARWU ranking --- Chain Influence of multi indicators perturbation Scenarios Decision Pathway Description Change in Rank A University B University Scenario 0 0 Award, 0 HICI winner, 0 N&S Pubs - - Scenario 1 0 Award,1 HICI winner 11 5 Scenario 2 0 Award, 1 HICI winner, 2 N&S Pubs (SSCI) 15 8 Scenario 3 1 Award & HICI winner 70 45 Scenario 4 1 Award & HICI winner, 2 N&S Pubs (SSCI) 72 60 Scenario 5 1 Award & HICI winner, 6 N&S Pubs (SSCI) 83 82 Scenario 6 2 Award & HICI winners 118 82 Scenario 7 2 Award & HICI winners, 8 N&S Pubs (SSCI) 130 110 Scenario 8 2 Award & HICI winners, 16 N&S (SSCI), 20 other SSCI Pubs 144 127
  • 14. Criteria of World Famous University (WFU) Description Indicators Weights Research productivity Total publications 0.3 Total citations 0.2 Research impact Highly cited papers 0.2 Number of subjects in the rank 0.05 Research innovation patents 0.05 Hot papers 0.1 Research development Percentage of highly cited papers 0.1
  • 15. Data sensitivity of WFU ranking --- Chain Influence of multi indicators perturbation Scenarios Decision Pathway Description Change in Rank of C University Scenario 1 1 more Highly Cited paper 11 Scenario 2 1 more Highly Cited paper, 1 more Hot paper 48 Scenario 3 1 more Highly Cited paper, 1 more Hot paper & 1 more patents 48 Scenario 4 2 more Highly Cited papers 66 Scenario 5 2 more Highly Cited papers, 2 more Hot papers 97 Scenario 6 2 more Highly Cited papers, 2 more Hot papers & 2 more patents 98
  • 16. Conclusions & Limitations
    • Data sensitivity has been estimated of two ranking schemes, significant changes have been detected in the case of single indicator perturbation, and dual indicators perturbation.
    • Particularly, more attention has been paid to decision pathway sensitivity analysis, where three universities have been selected for an illustration.
    • Unavoidable errors exist due to difficulties in data recovery in some ranking scheme s .
  • 17.
    • Thanks to the useful discussions with AWRU and WFU groups!
    • Thanks for your patience!