Evaluation, Quality and Impact of Open Access Publications Chair: Julianne Nyhan (European Science Foundation) Susan Murray (Director, African Journals online) Christiane Fritze (Berlin-Brandenburg Academy of Sciences and Humanities)

Chair: Julianne Nyhan (European Science Foundation)

Susan Murray (Director, African Journals online)

Christiane Fritze (Berlin-Brandenburg Academy of Sciences and Humanities)

European Reference Index for the Humanities – ERIH Better access to and assessment of Humanities research in Europe and beyond Dr Julianne Nyhan [email_address]

European Reference Index

for the Humanities – ERIH

Better access to and assessment of Humanities research in Europe and beyond

Dr Julianne Nyhan

[email_address]

European Humanities research– the challenges Difficulty to identify and compare Humanities excellence - vis- à -vis other sciences - across all languages at a supra-national (European) level Poor international visibility of research output in national languages (considered as necessary vehicles for Humanities research) Multiplicity of formats for research output (monographs, edited volumes, journals [less than 1/3 !], conference proceedings, web-based content and data, outreach)

Difficulty to identify and compare Humanities excellence

- vis- à -vis other sciences

- across all languages at a supra-national (European) level

Poor international visibility of research output in national languages

(considered as necessary vehicles for Humanities research)

Multiplicity of formats for research output (monographs, edited volumes, journals [less than 1/3 !], conference proceedings, web-based content and data, outreach)

ERIH – the origins SCH Workshop on evaluation of scientific production in Humanities (Budapest 2001): Existing citation indices (AHCI, SSCI) have unsatisfactory coverage of European Humanities research Need for a European citation index for the Humanities as additional tool for research assessment ESF requested to compile lists of reference journals in the first instance

SCH Workshop on evaluation of scientific production in Humanities (Budapest 2001):

Existing citation indices (AHCI, SSCI) have unsatisfactory coverage of European Humanities research

Need for a European citation index for the Humanities as additional tool for research assessment

ESF requested to compile lists of reference journals in the first instance

ERIH – objectives / methods Benchmarking tool for comparisons at aggregate level Focus on format used in other sciences ( journals ) > achieve a degree of initial comparability BUT : methodology needed for other formats Encourage ’best practice’ in the publication of journals in the Humanites Peer-review : discursive identification of excellence > across all fields of the Humanities > across Europe < national consultations European Research Area : Improve access to European Humanities research across all languages (basis for VLE?) ERIH as a tool for the strengthening of Humanities scholarship in Europe created by scholars for scholars

Benchmarking tool for comparisons at aggregate level

Focus on format used in other sciences ( journals )

> achieve a degree of initial comparability

BUT : methodology needed for other formats

Encourage ’best practice’ in the publication of journals in the Humanites

Peer-review : discursive identification of excellence

> across all fields of the Humanities

> across Europe < national consultations

European Research Area :

Improve access to European Humanities research across all languages (basis for VLE?)

ERIH as a tool for the strengthening of Humanities scholarship in Europe created by scholars for scholars

ERIH – process Set up Steering Committee Define the first 15 disciplines to be included Define categories of journals: A, B, C Compile guidelines for inter/national panels Approves Expert Panels (members suggested by MO’s, SCH, StComm, in the future: ENCoPs) Validates work of expert panels Produce and process material Input from ESF MO’s (national panels / reference tools) according to guidelines Initially 15 international panels who compile categorised lists of good peer-reviewed research journals in 15 disciplines

Set up Steering Committee

Define the first 15 disciplines to be included

Define categories of journals: A, B, C

Compile guidelines for inter/national panels

Approves Expert Panels (members suggested by MO’s, SCH, StComm, in the future: ENCoPs)

Validates work of expert panels

Produce and process material

Input from ESF MO’s (national panels / reference tools) according to guidelines

Initially 15 international panels who compile categorised lists of good peer-reviewed research journals in 15 disciplines

ERIH – current coverage Index Coverage (publication criteria): Peer-reviewed journals; ISSN; bibliographic data for cited references; author data Index coverage (discipline criteria): 15 subject groups defined, some covering areas between Humanities and Social Sciences (anthropology; education; psychology) Index coverage (internationality criteria): Cat. A (<25%)/B: international journals (defined by status: reputation, readership, authors, citations), Cat. C: important European journals with more regional circulation (ESF MO Europe: not Russia) All journals included – whether A-B-C - to be considered good research journals

Index Coverage (publication criteria):

Peer-reviewed journals; ISSN; bibliographic data

for cited references; author data

Index coverage (discipline criteria):

15 subject groups defined, some covering areas

between Humanities and Social Sciences

(anthropology; education; psychology)

Index coverage (internationality criteria):

Cat. A (<25%)/B: international journals (defined by status: reputation, readership, authors, citations),

Cat. C: important European journals with more regional circulation (ESF MO Europe: not Russia)

All journals included – whether A-B-C - to be considered good research journals

ERIH – categories Precision of criteria for international journals a genuine, varied and regular international cohort of contributors and readership a consistently high-quality scholarly content a broad consensus within the field concerning international status and visibility, insofar as possible a quality control mechanism, normally through peer-review Plus some, not necessarily all, of the following characteristics: Active international advisory board Openness to unsolicited contributions Highly discriminating and selective in the choice of articles published Publication on time and to an agreed schedule Degree to which these characteristics are applied could be added to the criteria distinguishing journals in categories ‘A’ and ‘B’

Precision of criteria for international journals

a genuine, varied and regular international cohort of contributors and readership

a consistently high-quality scholarly content

a broad consensus within the field concerning international status and visibility, insofar as possible

a quality control mechanism, normally through peer-review

Plus some, not necessarily all, of the following characteristics:

Active international advisory board

Openness to unsolicited contributions

Highly discriminating and selective in the choice of articles published

Publication on time and to an agreed schedule

Degree to which these characteristics are applied could be added to the criteria distinguishing journals in categories ‘A’ and ‘B’

ERIH – current disciplines/panels ANTHROPOLOGY Archaeology Art and Art History Classical Studies Gender Studies History History & Philosophy of Science Linguistics Literary Studies Musicology Oriental & African Studies Pedagogical & Educational Research Philosophy Psychology Religious Studies & Theology Possible Additions for updates : Archives, Library & Museum Studies Film, Media & Cultural Studies General Humanities

ANTHROPOLOGY

Archaeology

Art and Art History

Classical Studies

Gender Studies

History

History & Philosophy

of Science

Linguistics

Literary Studies

Musicology

Oriental & African Studies

Pedagogical & Educational Research

Philosophy

Psychology

Religious Studies

& Theology

Possible Additions for updates :

Archives, Library & Museum Studies

Film, Media & Cultural Studies

General Humanities

ERIH – towards the « initial lists » 2003/04: MO’s provide input based on previous national consultation (panels; reference tools) 2005/II: Expert Panels work - define field / remit (“scope notes”), - analyse and assess input received, - suggest circles of consultation, - consult where gaps are identified (eliminate / add), - discuss categories (multiple listing; single listing), - produce draft list, explain methods and problems. 2006 (mid): Wide consultation of - ESF MO’s, - (European) subject associations, - specialist research centres 2007 / 2008: Publication of the lists in three batches 2008: Update / feedback Online form (quantitative information): contact publishers, editors, European subject associations, national subject associations (through ENCoPs)

2003/04: MO’s provide input based on previous national consultation (panels; reference tools)

2005/II: Expert Panels work

- define field / remit (“scope notes”),

- analyse and assess input received,

- suggest circles of consultation,

- consult where gaps are identified (eliminate / add),

- discuss categories (multiple listing; single listing),

- produce draft list, explain methods and problems.

2006 (mid): Wide consultation of

- ESF MO’s,

- (European) subject associations,

- specialist research centres

2007 / 2008: Publication of the lists in three batches

2008: Update / feedback

Online form (quantitative information): contact publishers, editors, European subject associations, national subject associations (through ENCoPs)

Peer review at work… Layers : Input: National panels / scientific communities Selection: Expert Panels define scope, analyse and assess input, produce lists Consultation: MOs, subject associations (European level and some national), specialist research centres Calibrate/harmonise: ERIH Steering Committee Feedback: open process leading to updates in 2008 Challenges : Wide differences in quality of lists received from MOs Domain-specific (e.g.: cult./soc./evol. anthropology; philosophy and ethics) Language (e.g.: “forum languages” in art history; “international languages” in Oriental & African studies Some panels more reluctant to overrule authority of (own) national panels Outside peer pressure during consultation phases

Layers :

Input: National panels / scientific communities

Selection: Expert Panels define scope, analyse and assess input, produce lists

Consultation: MOs, subject associations (European level and some national), specialist research centres

Calibrate/harmonise: ERIH Steering Committee

Feedback: open process leading to updates in 2008

Challenges :

Wide differences in quality of lists received from MOs

Domain-specific (e.g.: cult./soc./evol. anthropology; philosophy and ethics)

Language (e.g.: “forum languages” in art history; “international languages” in Oriental & African studies

Some panels more reluctant to overrule authority of (own) national panels

Outside peer pressure during consultation phases

ERIH insights – quality through peer review? Peer review recognised as the only practicable method in basic research (standard method used in scientific communications themselves) Peer review can overrule weight of numbers for better (detect originality) or for worse (defend conservatism) Peer review introduces comparability into discussions of national discourses in Humanities scholarship

Peer review recognised as the only practicable method in basic research (standard method used in scientific communications themselves)

Peer review can overrule weight of numbers for better (detect originality) or for worse (defend conservatism)

Peer review introduces comparability into discussions of national discourses in Humanities scholarship

Evolution of ERIH Lists 1st batch of published initial lists are highlighted in yellow Discipline Initial submission 1st draft lists Consultation 1a Consultation 1b Initial lists Mar-Apr 2006 Spring 2006 Summer 2006 2007 Anthropology (social and evolutionary) 752 154 133 17 242 Archaeology 1310 524 290 25 419 Art and Art History 938 445 292 16 472 Classical Studies 619 256 321 11 251 Gender Studies 155 181 37 6 119 History 1419 874 508 71 907 History and Philosophy of Science 806 145 64 4 166 Linguistics 1093 680 391 34 586 Literature 1453 1481 706 10 802 Music and Musicology 204 n/a 187 4 166 Oriental and African Studies 196 588 386 14 505 Pedagogical and Educational Research 666 404 271 92 470 Philosophy 658 320 153 22 305 Psychology 1198 1201 159 4 634 Religious Studies and Theology 745 n/a 580 10 371

ERIH – Update in 2009 Recomposition of Expert panels: panel rotation mechanism and inclusion of new experts, document setting out methodology of recomposition Summary and Process of Methodology that underpinned work on ERIH “initial Lists” Summary and Process of Methodology that underpinned work of ERIH Steering Committee Panel meetings to take place from October 2008 – February 2009. ERIH Lists to be published incrementally in 2008 and 2009

Recomposition of Expert panels: panel rotation mechanism and inclusion of new experts, document setting out methodology of recomposition

Summary and Process of Methodology that underpinned work on ERIH “initial Lists”

Summary and Process of Methodology that underpinned work of ERIH Steering Committee

Panel meetings to take place from October 2008 – February 2009. ERIH Lists to be published incrementally in 2008 and 2009

ERIH – Update in 2009 Feedback on ERIH “initial List” Anthropology (2007/08) Of crucial importance in the updating of the “initial Lists” ERIH office has been contacting Editors and Publishers of relevant journals to ask that they submit quantitative data about their journals ERIH Feedback form: https://www2.esf.org/feedback.asp Difficult process because no central database of contact details exists ( [email_address] etc) Only 26% of journals on “initial List” have been commented upon by Editors / Publishers

Feedback on ERIH “initial List” Anthropology (2007/08)

Of crucial importance in the updating of the “initial Lists”

ERIH office has been contacting Editors and Publishers of relevant journals to ask that they submit quantitative data about their journals

ERIH Feedback form: https://www2.esf.org/feedback.asp

Difficult process because no central database of contact details exists ( [email_address] etc)

Only 26% of journals on “initial List” have been commented upon by Editors / Publishers

ERIH – futures? Open Source - ERIH is creating consensus on excellence (> priority list for inclusions) - ERIH includes publications of ESF MO’s, incl. learned societies (> institutional incentive?) - link of access / assess may create incentive to use OA [perhaps further stimulated by universities through support for IR’s]

Open Source

- ERIH is creating consensus on excellence

(> priority list for inclusions)

- ERIH includes publications of ESF MO’s, incl. learned societies

(> institutional incentive?)

- link of access / assess may create incentive to use OA

[perhaps further stimulated by universities through support for IR’s]

ERIH – futures? Open source futures? - ERIH aims to create a consensus on excellence (> priority list for inclusions) - ERIH includes publications of ESF MO’s, incl. learned societies (> institutional incentive?) - link of access / assess may create incentive to use OA [perhaps further stimulated by universities through support for IR’s]

Open source futures?

- ERIH aims to create a consensus on excellence

(> priority list for inclusions)

- ERIH includes publications of ESF MO’s, incl. learned societies

(> institutional incentive?)

- link of access / assess may create incentive to use OA

[perhaps further stimulated by universities through support for IR’s]

ERIH – futures? Open Questions : - Measuring influence of OA journals when not cited: downloads and citation refer to different phases in research process - OA portal appropriate for a highly diverse field? - How to link OA to IR’s [additional institutional stakeholders]? - OA as linked to budget model of learned societies? - How to integrate current knowledge about delayed impact in most Humanities [6-month-rule not applicable?]?

Open Questions :

- Measuring influence of OA journals when not cited: downloads and citation refer to different phases in research process

- OA portal appropriate for a highly diverse field?

- How to link OA to IR’s [additional institutional stakeholders]?

- OA as linked to budget model of learned societies?

- How to integrate current knowledge about delayed impact in most Humanities [6-month-rule not applicable?]?