Niamh Brennan (Trinity College Dublin) – CERIFy

MahendraMahey (UKOLN)
Stephanie Taylor (UKOLN)
Niamh Brennan (TCD)
Kevin Kiely (TCD)
http://cerify.ukoln.ac.uk

Current and past CRIS/CERIF/RIM projects
Research Information ManagementProcess Mapping and Gap Analysis
TCD CRIS
Questionnaires Site Visits
Business Process Analysis‘as is’ and ‘to be’
4 Priority Processes
User requirements elicitation
Prioritization of RIM Processes / Identification,
Mapping, Gap and Stakeholder Analysis
Data from at least 2 partners
Thomson CRIS
Research in View and InCites
Increasing level of Engagement with a CRIS / CERIF
Evaluation of CERIF and CRIS
Import and Export of Data and Facilitation of data exchange
CERIFy Approach
People Centered
Socio Cognitive
TCD – CERIF People and Publications
Institutions see value of CERIF and CRIS, Tell their regional institutions at regional and national events.CERIF and CRIS Engagement Increases
Priority
Process Areas – ‘as is’ and ‘to be’
CERIF
TCD
XML
2008
Mappings and Cross Walks relevant CERIF Schema
InCites
CERIF
TCD
XML
2002
CERIF Health
Check
Other, e.g. PURE?
Reducing duplication
Research in View
Data Surgery
Mahendra Mahey
Stephanie Taylor
Rosemary Russell
Michael Day
Talat Chaudhri
NiamhBrennan
Kevin Kiely
Jimmy Tang
Roisin Croker

InCites Exchange of Data

How Is This Used?
RAE requirement
Comparison with other universities (e.g. Russell Group)
Global comparisons (updated annually)
Citation information generated from this tool provides a lot of key information

Collection of Data
Two-way process between Thomson Reuters & individual institution
Requires a lot of work by institution
Institutional data collected in local database of staff research information

User Issues
“There is a lot of effort involved in understanding the data - there would be much value in standardising the data.”
E.g. - problems with author ID - duplicates & null fields

User Requirements
“If the data was in a standardised format it would make life much easier!”
Nightly updates
Internationalised data sets
Relational database with mapping research output onto staff ID number for returned data
Better mapping to institutional schools structure

Measures of Esteem

How Is This Used?
RAE requirement
Personal review for promotion process
Assessment & benchmarking - internal & external
As key information in the assessment in judging the overall quality of research outputs
Drafting REF documentation
Part of institution-wide annual planning cycle

Collection of Data
From individual researchers on adhoc basis
Systematic for RAE
Continuous tracking for REF
Ask individual schools to provide this on regular basis

User Issues
Process is ‘woolly’ - difficult to provide meaningful data
Perception varies depending on given point in a career path
Citations alone are not accurate enough
Context is needed to make sense of raw data
Defining this process is very difficult
Researchers can undersell themselves and don’t make information available

User Requirements
Systematic way of capturing this is required
Bringing a bibliometrics perspective to esteem-related information
Take into account that one measurement doesn’t fit all
Acknowledge differences in different subjects
When one size doesn’t fit all, find where to plug the gaps in information
Personalised audit tool to be built into RIM system

Thomson Reuters data exchange – what’s InCites?

Why CERIFy it?
The community expressed interest in this process
Current data is unsatisfactory ; the dataset is a specific view
Institutional Citation Report is ‘historic’: based on affiliation at the time of authorship.
Does not show Schools, Research Centres etc. (WoS API will not provide this either).
Author, institution name variants (no automatic DAI in WoS/InCites)
Missing papers (aside from general coverage issues)
But.. institutions want the metrics + institutional structure in InCites + to embed data in local CRIS/IR:
Sample
Current exchange process is unsatisfactory: non-standard schema, heavy demands on institution, disappointing results
Opportunity to create a CERIF data model for exchange of people, institutional affiliations, publication data & metrics
Opportunity to demonstrate a 360⁰ data exchange using CERIF
If successful, it should help to show value and help to engage institutions with CERIF
Model can be re-used for other similar exchange processes

Process 1: InCites‘as is’ –Based on Queen’s University Belfast

Data drilldown
Creation of CERIF Data Model for ‘InCites’ Exchange
Process 1: InCites ‘to be’ –Generic (based on CERIFy Data Surgery)
3 Institutional Data automatically matched with TR records for authors & publications
4. InCites 'Institutional Profile' established and updated automatically
2. TR import and update deltas
Thomson Reuters
EXTERNAL
[6] 5 TR send report to institution on results of matching process (including matched Author ID)
1. Online export of CERIF-based data to TR from RIM/CRIS via web services (nightly updates)
MIS/CRIS developers
[7] TR data integrated with CRIS data (if required) Note: may use TR API instead or alongside
Research Office / Library
Business Intelligence Officers/Institutional Analysts
5. Reporting on Research
8.1 Internal Reporting 8.2 External Reporting
EXTERNAL
DATA EXCHANGE PARTNERS

Over to Kevin…

Where we are – results
CERIF Data model
Data exchange next week!
Includes: extended ‘publication types’ list
Multiple identifiers (Researcher ID; Institutional ID, HESA ID, Publication ID etc
Full metrics
= standardisation
Next steps
QUB
Thomson Reuters are working on automating the process using Web Services…

Notes on Esteem & REF
“‘Esteem’ is no longer included as a distinct element in the assessment”

67. All information provided by HEIs in submissions to the 2014 REF must be capable of verification [note: can CERIF help provide the reference or source of data?]

Under ‘Definition of Impact for the REF’:

“a. Impacts on research or the advancement of academic knowledge within the higher education sector (whether in the UK or internationally) are excluded. (The submitted unit’s contribution to academic research and knowledge is assessed within the ‘outputs’ and ‘environment’ elements of REF.)”
Part 3 Section 5: Environment template (REF5)
183. Information is required about the research environment for each submitting unit relating to the period 1 January 2008 to 31 July 2013. Each submission must include a single completed REF5 form, consisting of the following sections:
• Overview.
• Research strategy.
• People, including:
– Staffing strategy and staff development.
– Research students.
• Income, infrastructure and facilities.
• Collaboration and contribution to the discipline.
From: Assessment framework and guidance on submissions, HEFCE, July 2011

http://www.hefce.ac.uk/research/ref/pubs/2011/02_11/02_11.pdf
REF5 ‘Environment’:
Maximum score reduced from 30%
(RAE 2008) to 15% (REF 2014)
Indicators of Esteem = Collaboration &
Contribution to the Discipline

Process 1: Esteem ‘as is’ –Based on Aberystwyth University
1 Completes ‘Other Activities’ annual form on Research Activity Database (RAD) SharePoint interface
Academic
4. Hardcopies printed for Research Monitoring meetings
2. Details of form transferred to RAD
3. Pdf report produced
Research Monitoring Team
5. Electronic copy deposited within RAD SharePoint site Departmental document folder
HoD/DDR
6. Access electronic copy deposited within RAD SharePoint site Departmental folder
Deans
7. Hardcopies printed for Research Monitoring meetings
DATA EXCHANGE PARTNERS

Process 1: Esteem ‘to be’ –CERIFy Data Surgery
1. CRIS/RIMacademic profile is automatically populated with elements of 'Measures of Esteem'
MIS/Library
Academic
2. Academic updates CRIS/RIM with other elements of Measures of Esteem
3. CRIS / RIM system data is queried and reported by Research Monitoring Team
Research Monitoring Team
HoD/DDR
4. Reports generated by HoDs, Deans etc.
Deans
5. Reports generated by HoDs, Deans etc.
External agencies/Funders
6. Data Exported to external agencies as required

Indicators of Esteem:
based on CERIFy Data
Surgery, current practice
In partner institutions +
a number of external sources

Indicators of Esteem -
the extended version:
based on original CERIFy IE,
With additions to
publications & outputs
Please see spreadsheet
for latest list CERIFy Indicators of
Esteem

CERIFy Esteem Indicators Model – based on MICE (Impact) Note: current focus for Esteem is more on Indicators rather than on Measures
Esteem Indicator
language table
Link tables
cfOrgUnitId etc.cfEsteemIndicatorId
cfClassID
cfClassSchemeId
cfStartDate
cfEndDate
Esteem Indicator table
OrgUnit
Project
Person
Publ.
Etc.
OrgUnit
Project
Person
Publ.
Etc.
Link table
Esteem Measure table
cfEsteemMeasureIdcfEsteemIndicatorId
cfStartDate
cfEndDate
cfEsteemMeasureIdcfEsteemMeasureKind
cfCountInteger
cfValueFloatingPoint
cfValueJudgementalNumeric
cfCountIntegerChange
cfValueFloatingPointChange
cfValueJudgementalChange
cfValueJudgementalText
Link tables
Link tables
Link tables
cfClassID
cfClassSchemeId
cfStartDate
cfEndDate
cfClassID
cfClassSchemeId
cfStartDate
cfEndDate
cfClassID
cfClassSchemeId
cfStartDate
cfEndDate
NB: this adaptation has yet to be finally validated by Brigitte

Esteem Indicators mapped to UK RAE 2008, Australian ERA

Esteem Indicators for REF5: Contribution to Discipline/Collaborations

The Issue with Esteem Indicators [as with Impact]
The processes associated with the capture and reporting of Esteem Indicators can benefit more if there is an effort to standardise at least some of the data required.
Most of the data is not currently available from sources other than narratives or reports supplied by members of academic staff
To reduce time and effort & assist with verification:
Where data can be imported from external sources it should.
‘Authority-controlled’, taxonomies and definitions lists should be agreed and built into the system
NOTE: the RAE 2008 submissions under ‘Research Environment, Measures of Esteem’ provide an extremely valuable source of information on the type of information provided by institutions in this area. This body of knowledge is not available for other REF-relevant areas such as ‘Impact’. The RAE 2008 ‘Measures of Esteem’ is a particularly rich source of data from the disciplinary viewpoint.
The analysis of these ‘texts’ has great potential to assist with developing the type of ‘authority lists’/taxonomies described above.
The following slides show the results from analysing the RAE 2008 University of Bath ‘Measures of Esteem’ submission in the field of Pharmacy. This text was analysed using Many Eyes and Open Calais. The results of using Open Calais for semantic analysis and entity extraction even for this small sample are very positive.

Acknowledgements
The CERIFY Team & Partners
Brigitte Jorge, EuroCRIS
Thomson Reuters InCitesCERIFy team, Philadelphia
Bo Alroe, Atira /UK Pure User Group (Esteem Indicators)

Niamh Brennan (Trinity College Dublin) – CERIFy

by Repository Fringe on Aug 10, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 164

Statistics

Niamh Brennan (Trinity College Dublin) – CERIFy — Presentation Transcript