The document discusses challenges in capturing patient and staff experiences in evaluations. It notes the importance of using a realist approach to understand what works for whom under what circumstances. It then summarizes a Nuffield Trust evaluation of six admission avoidance schemes that sought to understand user and staff experiences through mixed methods. Key challenges discussed include hidden attitudes of staff and how to properly capture user experiences through appropriate tools and methods.

1. © Nuffield Trust30 June 2015
Reflections on the challenges of capturing patient
and staff thoughts in evaluations
Ruth Thorlby

2. © Nuffield Trust
What is the Nuffield Trust’s approach to research and
evaluation?
“.. There is a real danger of circularity here – opinion sources are
quoted as evidence, and subsequent readers may fail to look at
the original source, assuming that what is quoted rests on good
quality evidence”
“I could write the Nuff's report on the back of a fag packet
between the Hobnobs and a cuppa-builders. So could you”

3. © Nuffield Trust
Evaluations: how ‘realist’ can we be?
‘What works, for whom, and under what circumstances’
Nick Tilley (2000)
http://evidence-basedmanagement.com/wp-content/uploads/2011/11/nick_tilley.pdf
• Theory (why might it work?)
• Embedded (professional, organisational and individuals)
• Active (requires people to do something)
• Open systems (other things happening at the same time) (Pawson and Tilley)
“A central distinguishing tenet of critical realism is that there are
both visible and hidden forces at play in the generation of
outcomes”
Jagosh et al;; Res Syn Meth 2013

4. © Nuffield Trust
Capturing user and staff experiences of admission
avoidance schemes
Evaluation of six volunteer-delivered projects across England
Expectations of commissioners: reducing hospital use and
improving patient experience
Mixed methods evaluation
User experience?
Staff/volunteer experience?

5. © Nuffield Trust
Cabinet Office

6. © Nuffield Trust
Project Sites
Age UK, Cornwall
Royal Volunteer
Society, Leicester
Age UK / British Red
Cross, Leeds
Westbank, Devon
Age UK, South
Lakelands
British Red Cross,
Derbyshire
Age UK, Oxfordshire

7. © Nuffield Trust

8. © Nuffield Trust
The visible and the hidden: staff attitudes?
“she’s got some of the relationships in the hospital, so that’s
hugely helpful because it’s quite difficult going to work in an acute
hospital like that where there’s a bit of a blame culture in terms of
delays ... everyone tends to say it’s someone else’s fault and
obviously you’ve got mini-fiefdoms within a big hospital in terms of
who actually gets to control what”
Voluntary sector project officer

9. © Nuffield Trust
Attitudes towards the statutory sector
“.. so they [the GPs] saw that as extra work, more people in the
community identifying more issues that they ought to be dealing
with for which they’re not receiving extra funding, so ‘why would
we support you to go out there and find more patients for us to
deal with?’”
Voluntary sector project officer

10. © Nuffield Trust
Local strategies for collecting user experience?
Warwick Edinburgh Mental Wellbeing Scale
Adult Social Care Outcomes Toolkit (ASCOT)
PHQ-9 (depression screening tool)
Generalised Anxiety Disorder (GAD-7)
Feedback forms
Bespoke forms (“Guided Conversation”; Compass)

11. © Nuffield Trust
Developing a user reported measure of care coordination
Funded by the Aetna Foundation (US)
• King’s Fund
• Picker Europe
• National Voices
• International Foundation for Integrated Care
• Nuffield Trust
• 18 months to develop a user questionnaire to explore whether
care (health and social care) coordinated

12. © Nuffield Trust

13. © Nuffield Trust
Understanding ‘integration’ from a user perspective
• 2011 ‘Narrative’
published, with ‘I
Statements’
• ‘Narrative’ is based on
the statements that
patients and service
users could make if the
care they experienced
was person centred and
coordinated.
• Based on extensive
engagement with
patients and patient
organisations

14. © Nuffield Trust
Example of I Statements: communication domain
But: many of these are not straightforward to
measure!

15. © Nuffield Trust
Methods
• Focus groups with users (over 65), stakeholders (GPs, hospital and
local authority providers)
• Access to National Voices’ latest work with ‘older people’s narrative’
based on in-depth interviews with 75 older people
• Literature review, including policy and organisational documents from
local projects/innovation
• Expert advisory group, with international academic experts
• Cognitive testing n=30
• Piloting: paper questionnaire to approximately 3,000 older people
with one chronic condition, accessed via GP lists in 3-4 areas in
England, including mix of ethnic groups and income distribution

16. © Nuffield Trust
What matters to older people?

17. © Nuffield Trust
Current questionnaire status
39 questions, organised under headings:
• Your health and wellbeing
• Managing your health day to day
• Support from social services
• Planned care
• Urgent care
• Hospital care
+ standard demographic questions: age, gender, ethnicity, health
status

18. © Nuffield Trust
Understanding your own culture in order to ‘see’ patients
Inside the system:
• Hierarchical
• Scientific/bureaucratic
• Rational/hard
• Measurable
• Boundaries are
guarded
• ‘The patient’ as seen
by medicine
• ‘Patients’ as proxies
Patients’ worlds
• Unstructured/messy
• Emotions/feelings
• Real complicated lives
• Things that are hard to deal
with ‘inside’
• The ‘public’- latent patients
Adapted from: Putting patients first: what do we really mean?
Ereaut G and Whiting E Linguistic Landscapes 2013

19. © Nuffield Trust
“Focusing on the patient is difficult because it makes you
confront things you don’t necessarily want to face.”
Hospital consultant cited in PFCC Final report (2014)
http://www.health.org.uk/media_manager/public/75/programme_docs/PFCC
%20final%20report.pdf

20. © Nuffield Trust
Concluding thoughts
Realist approach doesn’t require extensive theoretical expertise,
but..
• Focus on the theories behind the intervention, the reality of the
teams/individuals implementing it, how it might change as it is
implemented and the changes happening around it
• Focus on the visible and the hidden
• Any user experience tools need to be relevant and useable!