Nov 2, 2012 Presentation slides for JAMJE meeting

1. The future of research evaluation:
Trends and prospects in research assessment
Syun Tutiya
National Institution for Academic Degrees and University Evaluation
土屋俊
大学評価・学位授与機構
Novermber 2, 2012
JAMJE

2. Outline
• Types of “research assessment”(研究評価の種類)
• Currently in Japan(日本における状況)
• Problems recognized(その問題点)
• Institutional assessment of research(研究の機関別
評価)
• Alternative philosophies and methods(代替的評価)
• Prepublication review and research assessment
(査読と研究評価)

3. Types of “research assessment”
• By purpose and assessment agent(目的と評価主体)
– POLISY MAKING => policy maker
– EVECTIVENESS => funder
– EMPOYMENT/PROMOTION/INCENTIVE => management – university, research
institute
• By assessment unit(評価対象の単位)
– institution, department, research unit
– project, researcher
– “product” – article, protocol, database, dataset, software/program
• By transparency(客観性)
– self
– external(peer/stakeholder)
– third party(peer/stakeholder)
• By viewpoint(観点)
– process -- “how many articles?” (output)
– quality -- “how many citations?” (outcomes)
– impact -- “how much money?, what product?”
• By timing(時期)
– prior
– interim
– posterior

4. Current research assessment practices
in Japan
• Project reviews
– Big Grant-in-Aid projects(JSPS and others)
– JST grant projects
• Institutional reviews
– National University Corporation Evaluation by NIAD-UE => by
third party peers
• Personal assessment
– employment/promotion
– funding
• National trends
– JST
– NISTEP
– Elsevier

5. Problems of current practices
• Peer review is OK, but NO standards, NO benchmarks(研究評価はピア
レビューで、しかし比較を全然しないのは?)
– Typically, Nat Uni Corporation Evaluation(国立大学法人評価は、その典
型)
• “Quantitative indicators”
– Conventional: citation, “Journal impact factor”
– h-index(Hirsch) => researcher-level(productivity and impact) => group-
level (but co-authorship, fields, youth discriminated, etc, etc)
– Altmetrics => article-level
• Commercialization
– Thomson Reuters, Elsevier(Scival Spotlight)
• “World University Ranking”
– ARWU by Shanghai Jiao Tong University
– THE-WUR by Times Higher Education(Thomson-Reuters)
– QS-WUR by Quacquarelli Symonds(Scopus) and more

6. What if RAE/REF in Japan?
• REF: Research Excellence Framework(UK)
– For selective allocation of research funding, accountability,
benchmarking information and reputational yardstick
– Assess the quality of research in UK HEI
– By expert review, 36 units(4 main panel)
– 5 levels(4 * to unclassified)
– outputs(65%) in terms of “originality, significance and regour”,
impact(20%) in terms of “reach” and “significance”
• Any use?
– Close to “research” category of Nat U C Assessment
– bibliometric approach? => “citation information is not sufficiently
robust to be used formulaically or as a primary indicator of quality,
but there is considerable scope for it to inform and enhance the
process of expert review.”

7. Altmetrics
• Usually measuring article level impact
– views, downloads
– mentions in social media
– correlation with citation counts
• I.e. post-publication impact/pre-citation
– Light weight review by PLoS ONE => making it
sustainable, though
– Scientific Reports review process growing heavier
• Any use for article-level measure?

8. eLife’s new approach
“Media policy” Oct 29
1. Presenting and discussing the work prior to publication
a. Prior to publication authors are encouraged to present their findings to
their peers
b. When there is media interest..., we encourage the author to deposit the
accepted version of the manuscript in an open-access repository,
c. ...include a reference to eLife and/or elifesciences.org.
“NO Ingelfinger rule! NO embargo!”
1. Promotion of published content
a. Every published eLife paper will have a short, plain-language summary (the
eLife Digest).
b. Papers in eLife will be promoted to the media and to interested readers on
the day of publication.
c. We will issue press releases for some papers on the day of publication.