The University of Oxford faces significant challenges for research assessment due to its large size, organizational complexity, and decentralized structure. It has over 5,800 academic staff across 4 divisions and 44 separate colleges. Data is scattered across different departments and systems, and collecting consistent information for research assessment takes around 2.5 years. Assigning outputs to the correct units of assessment is difficult when research spans many disciplines. Ensuring consistent quality of case studies across the university's 31 units of assessment is also a challenge. Fully capturing the university's impact, which may occur in broad fields across departments, requires comprehensive data collection. Adopting common standards like ORCIDs could help integrate data but implementing them across the university's complex structures takes significant effort

1. Organizational complexity as a
challenge to research assessment
Sally Rumsey
The Bodleian Libraries
University of Oxford
A case study of the University of Oxford

2. Complexity
A result of
• Size of the institution
and the amount of
research produced
(research intensive)
Coupled with
• Organisational
complexity of the
institution
(administrative
structure; governance;
etc)

3. University of Oxford
Facts and figures*
• > 5800 academic and research staff
• > 5600 research students
• 4 university academic divisions (> 70 departments)
– Humanities: 9 faculties + Ruskin School of Art
– Social Sciences: 16 departments
– MPLS (Mathematical, Physical & Life Sciences): 12 departments
– Medical Sciences: 16 departments
– Plus cross disciplinary institutes eg Oxford Martin School and others
• 44 colleges and PPHs (Permanent Private Halls) (separate legal
entities to University)
• External research funding £478.3m
• Approx 4700 active research awards
• Estimated 16,000 journal articles published p.a.
• > 120 libraries
* 2013 and 2013/14 figures: Annual Review 2013-14

4. Democracy
• Wellington Square as the
centre of ‘the University’
• Democratic
• Decision-making devolved
to the four Academic
Divisions and their
departments/faculties
• Decision-making can
progress slowly
• Example: Symplectic
adoption for REF 2014
• REF OA Service design
project (for REF 2020)
Cppyright Clare Byers

5. Colleges
• Separate legal
entities to the
university
• Dual appointments
• College-only
appointed staff
• Not on HR system
• Submitted to REF
Copyright Jason Partridge

6. Governance and infrastructure
• Complex committee structure
• Routes and timing of committees
• Systems integration (WIP)
• RIM data quality
• Lack of unique identifiers and controlled vocabularies
across the entire institution (Divisions, Colleges, NHS
etc)
• Devolved model
• Internal matters also relevant to external integration
• Compliance with standards (eg RIOXX)

7. Facts and figures
Oxford REF 2014 submission
Submitted to
31 Units of
Assessment
(UoA)*
(out of 36)
c.2400
research staff
submitted
c. 8500
outputs
submitted
Returned
380,000 data
fields
REF 2008
resulted in £1.9
billion research
income over 6
year period
Reported
4892.5 doctoral
degrees
awarded
Took 2½ years
to manage and
compile return
* http://www.ref.ac.uk/panels/unitsofassessment/

8. REF complexity illustrative case
OII – Oxford Internet Institute
• A department of Social Sciences Division
• Multi/cross disciplinary
• Submitted under 9 UoAs (inc psychology,
economics, law, geography)
• Difficulty in identifying which are the ‘best’
outputs. Best for whom?

9. Quality control for REF Case studies
• Quality control was a challenge
• 31 UoAs split across 4 academic divisions
• Needed consistency across all submitted case
studies across:
– Unit
– Within the division
– Across all 4 divisions

10. REF complexity illustrative case:
Impact
• Medical sciences – broad areas such as
‘cancer’ or ‘neuroscience’ over a large
academic division
• Do not ‘sit’ in any one department
• Administrative staff cannot know all the
details to report impact
• Impact can be reported back 20 years
previously. Difficult to identify and track back
so far – people leave. Generally cite only 10
years

11. The effect on assessment
• Locating data required for
assessment
• Dealing with a variety of
systems
• Lack of single location for
each data type
• Data entry/import
(quantity)
• Consistency
• Size of the operation and
numbers of people involved
Copyright Clare Byers

12. • REF OA Service
Design Project
• Pilot Jan – June 2015
• Simple, short
message
– For individuals
– For support staff
(including directive)
• Comms is key
• RS/Bod/Divisional
staff

13. ORCIDs
• How ORCIDs should help
– De facto standard
– Common across all units of the University
– Supports integration externally
– To be required for REF 2020?
– Ultimately link individuals, publications, funding
– Assist automation
• Difficulties eg College staff not on HR/CUD system
• Limitations
• ORCIDs at Oxford service

14. ORCIDs at Oxford service is live
Released 12 May 2015

15. Points for discussion
• Data quality
• Systems integration
• Standards
compliance
• Common framework
between agencies
• Reduce
administrative
burden
• Automation to cope
with scale
Sally Rumsey, the Bodleian Libraries, 2015