This document discusses strategies for measuring the impact of arts and humanities research, which often involves non-traditional outputs. It notes that current systems focus too narrowly on citations and publications. To develop an inclusive system, it recommends considering four questions: how the research benefits society, who finds it valuable, how to communicate its value, and how to prove others find it valuable. It provides examples answering these questions for two projects and emphasizes gathering diverse evidence of impact from the beginning to communicate research value to different stakeholders.

1. Niamh NicGhabhann
Course Director, MA Festive Arts
Irish World Academy of Music and Dance
University of Limerick
niamh.nicghabhann@ul.ie /
@Niamh_NicGhabh

2. Designing a connected
research impact strategy
for arts and humanities disciplines

3. Communicating
the impact of non-traditional
research outputs

4. Traditional
vs.
Non-traditional
?

5. What is traditionally measured as
‘impactful’
vs
Research that is not reflected within
current impact measurement systems

6. What is traditionally
measured as ‘impactful’:
Books with prestigious
academic presses
Articles in high ranking
journals

7. Books and articles are central to arts
& humanities research
Institutions must continue to support
researchers in:
• Developing journal articles
• Selecting appropriate journals
• Preparing monographs/ edited volumes
• https://academicbookfuture.org/

8. Beyond publications…
Films
Performances
Exhibitions
Projects
Creative writing
Media appearances
Teaching/ lectures
Policy contribution
Sitting on advisory boards
Artworks
Public engagement
Festival contribution… etc!

9. We need to develop a research impact
measurement system that responds
to ALL of the different kinds of research
within arts and humanities disciplines

10. Current impact
measurement system:
Citation-based (H-Index)
Developed for STEM disciplines

11. H-Index citation system:
Does not reflect or capture all the
citations even in the most
‘traditional’ academic outputs
– a hardback book sitting on a
library shelf

12. What does impact mean?
What does it mean in
an Irish academic
context?

13. “A common position regarding
impact allows for an agreed version
to be used for policy purposes and to
assist in funding applications”
Irish Humanities Alliance (2015) report on
impact:
http://www.irishhumanities.com/assets/Uplo
ads/IMPACT-and-the-Humanities.pdf

14. “A recorded or otherwise auditable
occasion of influence from academic
research on another actor or
organisation”
Public Policy Group, London School of Economics
(2011) report on maximising research impacts :
http://eprints.lse.ac.uk/35758/1/Handbook_PDF_for_the_LSE_im
pact_blog_April_2011.pdf

15. ‘Academic impacts’ – usually measured as
citations in other authors’ work
&
‘External impacts’ – including influences
on actors outside academia, and could
include business, government, civil society
and the media.

16. ‘Beneficial changes that will happen in the
real world (beyond the world of
researchers) as a result of your research’
Mark Reed,
Research Impact Handbook (2016)
Blog: http://www.fasttrackimpact.com/

17. ‘An effect on, change or benefit to, the
economy, society, culture, public policy or
services, health, the environment or
quality of life, beyond academia’
Research Excellence Framework (REF)
definition of impact
http://www.hefce.ac.uk/rsrch/refimpact/

18. Key terms emerging from the 2014
REF around impact:
• Public engagement
• Knowledge transfer
• Impact on policy

19. Key criteria for assessment
• Reach: the spread or breadth of
influence or effect on relevant
constituencies
• Significance: the intensity of the
influence or effect.

20. Diversifying impact:
• Both within and without academia
• Moving far beyond the H-Index as a
measure of impact

21. 4 key questions to answer when
planning an impact strategy:
1. How is your research valuable to the world?
2. Who will find this valuable?
3. How can I best communicate this value?
4. How can I prove that others find my
research valuable?

22. 4 key questions to answer when
planning an impact strategy:
1. How is your research valuable to the world?
2. Who will find this valuable?
3. How can I best communicate this value?
4. How can I prove that others find my
research valuable?

23. Structural Issues around Impact
1. Important from individual and institutional
perspectives
2. Requires time, effort and skills
3. Requires investment and support from the
institution

24. Impact for the individual academic
1. Evidence of impact increasingly necessary for career
progression
2. Required as information on promotion/ progression
forms
3. Early career academics – need to evidence both research
activity and research impact
4. Required for grant applications

25. Impact for the institution
1. Research impact – contributes to position in university
rankings
2. Contributes to international visibility (attracting
students/ faculty)
3. Linked to ongoing funding – grants & core funding
4. Necessary for the ongoing health & viability of the
institution

26. Institutions and individuals need to evidence
research impact
Investment required in:
1. Research impact infrastructure
2. Research
3. Training
4. Support staff
5. Advocacy
6. Technology

27. We need to get better at communicating the
full range of arts and humanities research
impacts
Exclusive emphasis on metrics – wasting a huge number
of opportunities to represent the real contribution of Irish
higher education to culture, economy and society.
Disadvantages graduates – translatable aspects of
research often not represented/ visible to potential
employers or collaborators

28. Measuring the impact of the humanities:
often linked to the debate on defending the
value of arts and humanities research within
higher education

29. “How can a philosophy PhD be as ‘valuable’ to the
economy or to society as a computer science
student building a start-up company, or a medical
researcher exploring new treatment delivery
technologies?”

30. Key humanities skills and capacities include:
• Engaging with issues of identity
• intercultural relationships
• cultural values and understandings
• historical and comparative perspectives
• critical reasoning and nuanced deliberation

31. Key humanities skills and capacities:
Critically needed in relation to a range of urgent national
security issues/ issues of global importance such as
climate change, nationalism, terrorism, cultural integration
etc.
The issue of ‘free speech’ – currently an issue central to
security and public safety – a philosophical question that
requires skilled interpretation and leadership.

33. Arts and humanities disciplines need to be
proactive about designing a measurement
system that reflects modes of working and
researching.
If not, we risk being subject to systems that are
built for very different disciplines.

34. Key issue: Pace of impact
• Greater tendency towards single-author works
• Slower process around publication – journals
tend to have smaller staff numbers/ slower
turnaround/ longer articles in terms of peer
review – not unusual for articles to take 1 year, 1
½ years to be published (making associated/
aligned outputs even more important).
• Creative work/ events – immediate and often
ephemeral – alternative set of challenges in
capturing impact

35. Important critiques of measuring research
impact:
• too much emphasis on policy change
• potential to constrain academic freedoms and blue-sky
research
• concept imposes a neoliberal accountability regime with a
narrow focus on instrumentality
• idea of research impact suggests a smooth and uncomplicated
transition from academia to the rest of the world, when the
reality is often much more complex
• Irish system has the opportunity to learn from these critiques &
develop a more nuanced system

36. Considering the 4 key questions in relation to 2
research projects:
1. How is your research valuable to the world?
2. Who will find this valuable?
3. How can I best communicate this value?
4. How can I prove that others find my research valuable?
Examples: World Within Walls/ IU Culture Lab
Certainly not perfect examples, but useful!

37. World Within Walls
Multi-strand project
commissioned by the HSE
Advertised via public e-tender
Successful application by Stair:
An Irish Public History Company
Ltd. (research team: Dr Anne
Mac Lellan/ Fiona Byrne/ Niamh
NicGhabhann)
Outputs:
• Exhibition at Monaghan County
Museum (May 2015 - February
2016)
• History book
• Articles
• Conference presentations
• Heritage Week events
• Public talks at museum/ library
• Oral history collection
• Schools outreach project
• Website

38. http://stdavnets.wixsite.com/worldwithinwalls
(Project website developed by Fiona Byrne)

39. IU Culture Lab
‘Think-tank’ lab commissioned by
Limerick 2020
Took place in Limerick city over 6
weeks in summer 2015
Led by: Grainne Hassett/ Stephen
Kinsella/ Annmarie Ryan/ Niamh
NicGhabhann
Outputs:
• Article in Cultural Trends, peer-
reviewed journal
• Book of ideas
• 3 short films
• 3 policy proposals
• Series of public consultations
• Final stakeholder presentation
• Possible influence on newly released
DoSP/ DoA policy
• Newspaper articles on lab
• Blog posts on interdisciplinary working
process

40. 1. How is your research valuable to the world?
• Requires some confidence and thought
• Helpful to think about ‘short-range’ and ‘long-range’
values

41. 1. How is your research valuable to the world?
• Requires some confidence and thought
• Helpful to think about ‘short-range’ and ‘long-range’ values
• ‘Short-range’ – those working in the same field as you/ closely
related etc.
• ‘Long-range’ – how it contributes in a broader sense to the field of
inquiry – does it bring something new to research methods/ uses of
sources/ engagement with a specific community etc.

42. World Within Walls: how is this research valuable to
the world?
‘Short-range’ answers:
• It contributes to the growing field of medical history research
around mental health in an Irish context;
• It is one of the first major public history projects around a specific
mental health institution;
• It explores an important part of Monaghan’s local history;
• It explores a history that many people have experience of – either as
a current or former service user or staff member, or as somebody
connected to either of these groups.

43. World Within Walls: how is this research valuable to
the world?
‘Long-range’ answers:
• It provides an example of the use of exhibitions as a research outcome
• It provides an example of a humanities-based start-up – academic
entrepreneurship
• It creates opportunities for public engagement with previously under-researched
aspects of Irish history, and therefore has creative industries potential.
• It engages with ideas around stigma and mental health
• It provides historical comparison for contemporary mental health care, including
aspects of design for mental health care and the history and perception of various
different kinds of treatment
• It provides insight into the lived experiences of those involved in mental health care
– both as service providers and as a service user.

44. 1. How is your research valuable to the world?
• May seem like an obvious thing to do, but articulating key value
points is a necessary step in communicating that value
• Academics tend to be used to this within processes of writing
articles/ conference papers
• However – slight change of focus here – this is less about what the
research IS and more about what it DOES.
• Thinking about the ‘long-range’ values also helps to prompt a range
of contributions that could go unnoticed –could form the basis of an
article articulating those cross-disciplinary perspectives.

45. Key points from Question 1
• Be clear about what is new about your work
• Don’t be afraid to think beyond your immediate field of
research when thinking about value

46. 2. Who will find this valuable?
• Use the answers to Q.1 as a prompt
• Brainstorm as many different stakeholders as possible (drawing on
project management skills)

47. World Within Walls: who will find this valuable?
Initial stakeholder brainstorm
HSE
Medical historians
Irish historians
Museum studies scholars
Museum professionals
Creative industries professionals interested in creating engaging experiences around
history and heritage
Mental health professionals
Architects interested in designing for healthcare
Architectural restoration professionals
Local communities interested in history and heritage
Former service users and former staff members
Current service users and current staff members
Policy makers interested in issues around stigma and mental health
Policy makers interested in mental health and health communications generally

48. World Within Walls: who will find this valuable?
HSE listed at the top – commissioned the research
Engaging with key stakeholders early in the research process – can be a
key to impactful research, as researcher has the opportunity to respond
to the real concerns/ questions of the stakeholder

49. World Within Walls: who will find this valuable?
Grouping the stakeholders into priority groups
Engaging with stakeholders – time consuming. Important to identify the
priority groups, and plan effective and appropriate communications
strategies.

50. World Within Walls: who will find this valuable?
For World Within Walls – priority groups included:
• HSE
• Public interested in the history of the site
• Museum professionals and museum studies scholars
• Medical professionals and mental health professionals
These are the groups that we felt were most important in terms of
sharing research findings/ making them aware of the project.

51. Key points from Question 2
• Brainstorm at the beginning - is there a stakeholder that
could get involved as a project partner?
• Be strategic – decide on the most important groups to
communicate with

52. 3. How can I best communicate this value to them?
• Books & Articles – great for academics, not wonderful for everyone
else – may include discipline-specific jargon/ literature reviews that
won’t be accessible or relevant
• Try to connect with the most commonly used method of
communication used by the group that you are trying to target

53. 3. How can I best communicate this value to them?
• HSE – project reports (1-4-20) – 1 page summary/ 4 page executive
summary with key points/ 20 page project report
• Highlight recommendations for best practice/ policy implications.

54. 3. How can I best communicate this value to them?
• Members of the public interested in the site – public exhibition in
Monaghan County Museum
• Engaging with existing museum audience/ carried out public
consultation events/ talks/ workshops
• Heritage Week event – huge turnout! Guided tour of the site.

63. 3. NB re public communications – ETHICS & CONSENT
• If you are creating a public presence for your work and research – be
sure that you are not in breach of any ethics guidelines – make sure
that you have full ethical clearance for any of the research data that
is being used within your project.
• If you are taking images - particularly of children – consent forms/
guardian consent forms are key.
• Communicate with your project leader/ any project partners about
creating a public presence for your work.

64. 3. How can I best communicate this value to them?
• Museum professionals and museum studies scholars
• Be proactive in making sure that these communities have access to
your work
• Send free eprint link of articles/ sample of book to key individuals
• Share work via social media using key community hashtag -
#museumhour
• Write a project review for their industry magazine/ website/ blog
• Give conference paper/ poster presentation at their annual meeting

65. 3. Useful tip re communications
• Make a habit of bringing your camera – take good quality images of
research in action and get consent to reproduce them
• get name of photographer if not yourself, keep full image caption
• Keep a project blog as you go – write short chatty summaries of
work in progress – creating content ready to share while memories
are fresh rather than having to think back at the end of the project
when everyone is tired and stressed!
• Project blogs are very cost effective and easy to share – project blog
on World Within Walls turned into a guest blogging spot on the
British Medical Journal Medical Humanities blog – wide readership.

67. Key points from Question 3
• Be proactive – make sure the right people see your
research – email them/ send them a hard copy!
• Make sure you have designed the most effective way of
communicating
• Collect strong images/ video if possible
• Get buy-in from all project partners before going public.

68. 4. How can I prove that others find my research
valuable?
• Probably the most challenging question to answer
• Focus on prestige indicators/ peer review/ use and uptake by
communities
• Be proactive about gathering evidence
• Be proactive about getting heard – sign up as an available speaker
on www.womenonair.ie / write and send press releases with help
and advice from university communications department.

69. Prestige indicators/ peer review:
• Gather media coverage where possible (Irish Times/ Nationwide etc.)
• Keep track of visitor numbers - over 13,000 visitors to the exhibition,
over 250 at Heritage Week event
• Get project partners to write an evaluation of the project to form part
of an final research impact case study – even a short quotation can
be helpful.

74. Gather evidence – use library infrastructure
• Use online institutional repository (IR) where available in your
institution to safely store material that might not otherwise be
published in a journal or book.
• IR will collect usage statistics – a useful form of evidence.
• Compile a portfolio of event reviews/ images/ examples of
promotional material/ feedback for live performance events – create
a mini-archive per event and post on library IR.
• Share material with other archives – physical or digital – if
appropriate.

78. Institutional Repositories allows less traditional research outputs –
such as an exhibition – to be brought back into more ‘traditional’
formats
If not made available online in a searchable format – easy to be
overlooked by future scholars.
• Translation of non-traditional research back into more traditional
research outputs – cited in Dr. Brendan Kelly’s new volume on the
history of psychiatry in Ireland
• Cited in On Patienthood: Searching for the patient's voice in the Irish
asylums Brendan D Kelly
Med Humanities 2016;42:2 87-91 Published Online First: 5 January
2016 doi:10.1136/medhum-2015-010825

79. http://www.campusengage.ie/content/engaged-research
World Within Walls project listed in the first national audit of engaged research
Engaged research – a further metric of impact to
capture and communicate this kind of research output.

80. 4. How can I prove that others find my research
valuable?
• Policy change - a cautionary tale! Can be messy & indirect!

81. Presenting evidence of impact
• REF – used a case study system – all available online – v valuable
resource
(http://impact.ref.ac.uk/CaseStudies/CaseStudy.aspx?Id=35238
• http://impact.ref.ac.uk/CaseStudies/CaseStudy.aspx?Id=28190
•
• UCD – also starting case studies – have a humanities example (Dr.
Emily Mark Fitzgerald –
https://www.ucd.ie/t4cms/CASE_STUDY16_Emily%20Mark-
Fitzgerald.pdf
• UL – also provide an impact case study template/ impact podcast
series
• UL Engage – Implications for policy template (copies with handout)

82. Alternatives to the H-Index System?
REF – used a star-based scoring system

83. Alternatives to the H-Index System?
Similar challenge to assessing the complex impacts of events – Compass/ 360 model
presented by Donald Getz – could this be adapted?

84. KEY PERFORMANCE INDICATORS AND METHODS
Prof. D. Getz, 2016 84

85. ORGANISATION
PLANNING
DESIGN
ENVIRONMENT
AL OUTOMES
ECONOMIC
OUTCOME
S
EVENT COMPASS EVALUATION SYSTEM
-A Continuous Improvement Process
-Goal Attainment (the scale refers to targets for each theme and KPI)
-Can be used for certification of events and organisations
0
20
40
60
80
100
Prof. D. Getz, 2016 85
MARKETIN
G
RISK
SOCIO-
CULTURAL
OUTCOME
S

86. Conclusion: How can research impact communications be supported?
• Social media and media training for academics
• Research blogs that highlight research across the disciplines
• Short research videos and podcasts highlighting research projects across the
disciplines
• Creating and facilitating meaningful links between policy-makers and academics
• Creating and facilitating meaningful opportunities for inter/ multidisciplinary
research.
• Showcasing examples of impactful research from across the disciplines
• Showcase examples of policy explicitly citing academic research to encourage
collaboration and use
•
• Use the library to highlight research by keyword – a simple thing that can be done.
Making research themes and shared research concerns visible across the
university, or across the third level sector, would really highlight areas for potential
collaboration.
• Invest in, use, reward and support an alternative metrics system that reflects the
full spectrum of research and work in the university.