Professor Alicia O' Cathain of ScHARR University of Sheffield presents on how qualitative research can contribute to evaluation.

1. How qualitative research
contributes to evaluation
Professor Alicia O’Cathain
ScHARR University of Sheffield
22 June 2015

2. What is qualitative research?
• Normal in evaluation
• Understanding not
measuring
• Set of methods
– Focus groups
– Semi-structured or in
depth interviews
– Non-participant
observation
– Diaries

3. What is evaluation?
• Researcher led
evaluation
• Policy evaluation

4. Researcher-led
• MRC Framework developing
an evaluating complex
interventions
• ACTIF programme
– 5 years
– RCT
– Qualitative at each phase

5. O’Cathain A, Thomas KJ, Drabble SJ, Rudolph A, Hewison J. What can
qualitative research do for randomised controlled trials? A systematic
mapping review. BMJ Open 2013;3:e002889
Intervention
Trial design
and conduct
Outcomes
Measures
Health
conditions

6. Intervention n=254
Develop
n=48
Describe it
n=10
Understand
how it works
n=23
Value and
benefits
n=42Acceptability
in principle
n=32
Feasibility and
acceptability
n=83
Fidelity,
reach and
dose n=12
Implementation in
real world n=4

7. Trial design and conduct n=54
Recruitment
Diversity
Participation in
trials
Acceptability in
principle
Acceptability in
practice
Ethics/informed
consent
Adapting to local
circumstances
Impact on staff,
researchers, patients

8. Potential value
Potential value
Bias Avoidance of measurement bias
Efficiency Faster recruitment
Saves money
Ethics Trials sensitive to human beings
Improved informed consent
Implementation Facilitates replicability of intervention in the real world
Facilitates transferability of findings in the real world
Interpretation Explains trial findings
Relevance Ensures interventions meet the needs of health
professionals and patients
Success Makes a trial successful, feasible, viable
Validity Improves internal validity
Improves external validity

9. Maximising value…
1. Do it early
– 28% pre-trial
• Intervention development 100%
• Acceptability of intervention in principle 25%
• Acceptability of intervention in practice 24%
• Recruitment 18%
• Breadth of outcomes 0%
…otherwise its about future trials

10. 2. Publish learning for specific trial or future trials
3. Think beyond interviews: non-participant
observation
4. Try iterative or dynamic or participatory
approaches at feasibility phase
5. Not just complex interventions
• 38% of 104 data extracted were drugs or devices
6. Think about the range of work

11. Problems with quantitative only
• Null RCTs….explain findings
(context, mechanisms of
action, implementation)
• Failed trials….prevent this at
pilot stage
• It works but what is ‘it’?
…..qualitative can fix

12. Policy evaluation
• Learning from early adopters (feasibility)
• Stakeholder reception (acceptability)
• Service delivered (implementation, workforce)

13. Useful but challenges remain
– Fast evaluation
– When to evaluate
– Moving target
– Replacement of difficult-to-measure outcomes
with understanding of processes

14. Conclusions
• Useful contribution no matter what type of
evaluation – essential due to complexity
• Can help to fix problems faced in researcher-
led evaluation
• Challenges in policy evaluation need reflection