Presentation by Dr Ian McMahon, Director, Research Group of Eight at the Where is the evidence conference, National Library of Australia, Canberra, 10 October 2012.

1. MAKING GREY LITERATURE
COUNT
ERA, impact and other mechanisms
for assessing and valuing grey
literature
Ian McMahon
Director, Research Group of Eight Ltd
Canberra
Email: ian.mcmahon@go8.edu.au

2. Assessment of University
Research Performance
• Promotion
• Internal university assessments
• Grant applications
• Excellence in Research For Australia
(ERA)
• Assessment of Research Impact Trial

3. The Excellence in Research for
Australia (ERA)
• aims to identify and promote excellence
across the full spectrum of research
activity
• evaluate university research

6. ERA Indicators
research quality
•publishing behaviour, citation analysis, ERA
peer review, and peer reviewed Australian
and international research income.
research volume and activity
research application
esteem

8. ERA Eligible Research Output
Types
Traditional research outputs
•Books—Authored Research;
•Book—Chapters in Research Book;
•Journal Articles—Refereed, Scholarly Journal;
and
•Conference Publications—Full Paper
Refereed.

9. ERA ‘non-traditional’ research
output
• Original Creative Works;
• Live Performance of Creative Works;
• Recorded/Rendered Creative Works; and
• Curated or Produced Substantial Public
Exhibitions and Events.

10. Original Creative Works
• Visual art work
• Design/Architectural work
• Textual work novel or art review. Exhibition
catalogues and catalogue entries
• Other: For example, scholarly editions,
scholarly translations and public policy reports
may be submitted under this category, provided
they meet the relevant eligibility criteria, including
meeting the definition of research.

11. Research Statement for ERA
Peer Review of Non-Traditional
Research Outputs
For non-traditional research outputs which are
nominated for ERA peer review, a statement
identifying the research component of the output
must be provided as part of the submission of an
institution. The statement must be no more than
2000 characters (around 250 words)

12. ATN – Go8 Excellence in Innovation for
Australia (EIA) Trial
• Trial of an assessment of the impact of
research based on case studies.
• Jointly conducted by ATN and Go8 with the
involvement of three other universities
• Recognise work in UK REF – pilot in 2010, full
REF in 2014
• DIISRTE feasibility study into assessment of
impact

13. Design Principles of the Trial
The EIA Trial aims to :
a) Identify good impacts of research
b) Justify the claims being made by institutions of the
impacts
c) Identify the research underpinning the impact
d) Communicate to government, business and the
community
e) Advisory Board including: Phil Clark, Robin
Batterham, Patricia Kelly, Laurie Hammond, Craig
Roy, David Sweeney + uni representatives

14. Guidelines for the Trial
 Four digit SEO; Four broad sectors, Defence,
Economic Development, Society, Environment
 Time period – impacts from 1st January 2007,
underpinning research from 1st January, 1992
 Unit of evaluation – the impact itself, not
discipline, academic unit or individual
 A maximum of five case studies for each of the
four SEO Sectors (i.e. 20 in total)
 A minimum of two case studies in three of the four
SEO Sectors (i.e. 6 in total)

15. The mechanics of submissions
 Case Studies on prescribed templates ≤ 15
pages
 Information sought on
 Context
 Details of the impact
 Underpinning research and research publications
– submissions to draw the link to research quality
 Income and other relevant metrics
 Validation material available
 People involved in research and impact

16. Evidence of Impact
 In making submissions universities need to
make provide evidence of impact
 Nature of evidence depends on nature of
impact
 Could include all forms of ‘grey literature’
 Underpinning research could also have been
published in ‘grey literature’
 Confidentiality issues in assessment

17. The assessment of submissions
• Each case study rated A-E based on reach &
significance of impact
• 7 assessment panels
o 1 x Defence
o 2 x Economic Development
o 2 x Society
o 2 x Environment
• Majority of panel members will be non-
academics

18. Use of the results
A consolidated assessment report will be prepared for each
institution on results for those units submitted from their own
institution
A summary report of results spanning all institutions will be
made available to each participating institution
A summary EIA Trial report will be prepared for the peak EIA
Governance Group and released to the public following
endorsement. This report will evaluate the process of the EIA
Trial and make recommendations concerning its potential more
widespread adoption

19. Higher Education Research
Data Collection (HERDC)
• Collects research income and research
output data for use by DIISRTE in funding
research block grants
• Research outputs: refereed Journal
Articles, Research Books, Book Chapters
and Conference Papers
• Initial HERDC collections included other
categories
• Internal university measures could include
other categories eg policy reports

20. A Books
Authored-
A1 Research
Authored-
A2 Other
A3 Edited
Revision/New
A4 Edition
B Book Chapter
C Journal Articles
Article in
Scholarly
Refereed
C1 Journal
Other
Contribution to
Refereed
C2 Journal
Non-refereed
C3 Articles
C4 Letter or Note
D Major Reviews
E Conference Publications
E1 Full Written Paper- Refereed Proceedings
E2 Full Written Paper- Non-refereed Proceedings
E3 Extract of Paper
E4 Edited Volume of Conference Proceedings
F Audio-Visual Recordings
G Computer Software
H Technical Drawing/Architectural & Industrial Design/Working Model
I Patents
J Other Creative Works
J1 Major Written or Recorded Work
J2 Minor Written or Recorded Work
J3 Individual Exhibition of Original Art
J4 Representation of Original Art

21. Promotion criteria
The following factors may be taken into account in
assessing performance in research and/or creative activity:
•Publications
•IP
•evidence of professional activity in the visual and
performing arts
•evidence of consultancy or professional work which
contributes to advancement of knowledge, or scholarship
within a discipline or professional field
•evidence of success in securing external funding for
research
•evidence of a significant role in collaborative research
projects

22. Promotion criteria
The following factors may be taken into account in
assessing performance in research and/or creative activity:
•quality and impact of research and/or creative activity as
evidenced by honours (prizes, higher degrees, professional
qualifications, fellowships etc)
•quality and impact of research and/or creative activity as
evidenced by extent and nature of citations of work
•quality and impact of research and/or creative activity as
evidenced by successful uptake of intellectual property
•quality and impact of creative activity as evidenced by
representation in publications and collections and
performance of works by other artists.

23. www.go8.edu.au
The Group of Eight Limited Level 2, Group of Eight House Tel +61 2 6239 5844
PO Box 6229 101 Northbourne Avenue Fax +61 2 6239 5808
O’Connor ACT 2602 Turner ACT 2612 executive.director@go8.edu.au

25. Esteem Measures
• Editorship of a Prestigious Work of
Reference
• Fellowship of a Learned Academy and
Membership of AIATSIS at 5.7.2.4
• Nationally Competitive Research
Fellowships
• Membership of a Statutory Committee
• Australia Council Grants or Australia
Council Fellowships

26. Indicators of research
application
• research commercialisation income
• patents
• Plant Breeder‘s Rights
• registered designs
• NHMRC endorsed guidelines