SPHSU lunchtime seminar by Katie Buston, Stephanie Chambers and Matt Maycock

1. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
The value of observational work within
evaluations
Katie Buston, Stephanie Chambers,
Matt Maycock
MRC/CSO Social and Public Health Sciences Unit
Research Skills Seminar
8th December 2015

2. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Seminar overview
• Ethnographic methods and an outline of participant
observation (Matt)
• Illustrated with the following examples:
• PhD in Nepal, Scottish prions - Matt
• School food - Stephanie
• Sex ed, THRIVE, YOIs - Katie

3. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Ethnographic Methods
“From one point of view, that of the textbook, doing
ethnography is establishing rapport, selecting informants,
transcribing texts, taking genealogies, mapping fields,
keeping a diary, and so on. But it is not these things,
techniques and received procedures, that define the
enterprise. What defines it is the kind of intellectual effort
it is.... Ethnography is thick
description”
(Geertz 1973: 6, 9-10)

4. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Participant Observation
Participant Observation (PO)
• ‘Deep hanging out’
• Interviews
• Surveys
• Draw and Talk
• Genealogies
etc etc

5. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Some Practicalities of Participant Observation
• Live/work in study site for extended time
• learn local language and dialect
• participate in wide range of daily activities
• use everyday conversation as interview technique
• informally observe while participating
• record observations in fieldnotes
• continually reflect on experiences and data
• use both explicit and tacit information in analysis
(adapted from Dewalt and Dewalt, 2002)

6. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Strengths of Ethnography
• Produces rich and detailed research material
• Validity of research material:
• Research material does not arise from artificial research
setting
• Reported behaviour can be compared with observed
behaviour
• Actions can be seen in relation to specific social contexts
• Area of enquiry is only partially pre-defined
• inductive

7. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Strengths of Participation Observation
• Greater understanding of how life is experienced
• embodied experience
• tacit knowledge
• Development of trusting relationships
• long-term contact
• solidarity of shared experience
• Almost continually open to new data over long
period
• observe different social contexts
• witness unexpected
• continually testing/ confirming appropriate behaviour

8. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Limitations of Participant Observation
• Very expensive
• very time consuming
• both collecting and analysing data
• requires skilled researchers
• Can create substantial amounts of research
material
• Private behaviours v. difficult to observe
• Difficult to replicate
• Inherently conservative

9. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Subjectivity
• Data and their interpretation inevitably shaped by individual
researcher
• centrality of personal relationships
• selection of data to record
• PO data result from interaction
between researcher and researched
• Explore subjectivity
• don’t pretend objectivity
• be aware of biases and how one interacts
• ‘participant objectification’ (Bourdieu)
• Initial findings are about oneself and social conditions that gave one
one’s culture

10. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Ethics
• Informed consent
• initial introductions but continual process
• fully informed consent undermines main strength of PO
• concealing specific foci
• Confidentiality and reciprocating gossip
• Intervening against local practices
• Maintaining integrity across different groups
• degree of candour about own views

11. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Uses and abuses of Participant
Observation
Matt Maycock
MRC/CSO SPHSU Research Skills Seminar
8 December 2015

12. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
PhD research far-west Nepal, 2013
Masculinity, Modernity and Bonded Labour: Continuity and
Change amongst the Kamaiya of Kailali District, far-west
Nepal (School of International Development, UEA, Norwich)

13. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
PhD fieldwork far-west Nepal 2009
Yearlong fieldwork in Nepal:
• Three month language training and key informant
interviews
• Nine months in two fieldsites
My thesis addressed the the following research questions:
• How have the links between Kamaiya bodies and
Kamaiya masculinities changed following freedom?
• How are working patterns changing following freedom,
and what implications does this have for Kamaiya
masculinities?
• What are the Implications of modernity for Kamaiya
masculinities in family settings?

14. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Methods
• Household survey
• Life History interviews
• Participant observation
• I wore clothes similar to my research participants and made a
conscious effort not to display conspicuous signs of consumption.
• I tried to behave like the men of my age at both fieldwork sites as
far as possible. On occasion this involved doing the work that the
men in Kampur were involved in, although this did not include
driving a rickshaw as the rickshaw drivers found the idea ridiculous.
• I took part in various agricultural and hunting activities.
• I took part in the social life, which posed various difficulties for me.

15. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
My home for nine months…

16. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Participant Observation - Going ‘hunting’

17. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Going ‘hunting’

18. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Issues of Participant Observation/Fieldwork
• Trying to ‘keep up’ with the men in both fieldwork sites in various
ways was problematic and I was unable to do it in many respects,
particularly in relation to drinking, which was an important daily
occurrence
• Domestic Violence seemed to be most likely after a night at a local
bar many village women did not approve of their husbands going
there, especially as it meant they would be spending a large
proportion of the household’s limited income. Therefore, I did not
want to be a part of daily drinking, and felt very uncomfortable about
being associated with it.
• Not going to a local bar on a consistent basis allowed me to form
relationships with other people in the village, not least the women
and older men, who disapproved of the bar and what went on there.
• Isolation
• Incompetence
• Mental health implications of hearing disturbing narratives
• Practical issues with PO, I extensively used scratch notes

19. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Positionality
• As Reinharz (1997) indicates, researchers have multiple
identities apart from those associated with being a researcher;
mine include being white, Welsh, heterosexual, male and, at
the time, unmarried.
• My positionality through the various identities I brought to the
research – my gender, race, class etc. – influenced both how I
collected data and its interpretation (Mullings, 1999), it
influences the sort of PO was able to conduct.
• My position constituted both an advantage and a disadvantage.
West (2003) found that being positioned as an ‘outsider’
brought certain benefits in his research with victims of torture
in Mozambique’s war for independence. It allowed some of his
research subjects to discuss issues that they found it difficult to
speak about with members of their community.

20. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Some implications of my fieldwork…
• I have a long term commitment to these two fieldsites.
After 2009, I visited in 2011, 2013 and 2014 and will
continue to visit the same sites as long as I am able.
• Ethnographic methods have helped me to think differently
about my own identity, the places and people I know
• Having experience of using ethnographic methods is
transferable to other contexts (such as Scottish prisons)
• Sometimes it is hard not to think ‘ethnographically’ and to
not be observing or practicing observing at any given time

21. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Example 2
Prison masculinities in Scotland
(post-doc)

22. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Methods
Data collection:
A mixed methods feasibility and iterative programme
redevelopment study in three Scottish prisons, including:
• Observations of session deliveries (N=55)
• Interviews (N=54)
• Focus groups (N=3) with prisoner participants and prison
staff facilitators
• Objective, self-report and biomarker measures (baseline,
post-programme, 6, 9 and 12 months)
The FIT for LIFE model succeeded in attracting 92 prisoners
enrolled in 6 deliveries across the three prisons

23. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
PO within prison
• I do the Fit for life Programme with the prisoners
• Which means I do a lot of:

24. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
• Building trust and rapport within the prison context is quite
challenging, playing sport, banter and participating in the
programme are an important way of doing this.
• As a man within an extremely hierarchical context, my position in
relation to the hierarchies I am examining, evolved over time, as a
consequence of my changing participation in the programme
• This was manifested in increasing levels of banter, including
frequent ridicule of various jumpers I wore while in the prison. This
served to establish my (external) position in relation to this group of
prisoners and also illustrates the importance of clothing within the
prison context (cf. Ash, 2010).
• In Prison A and B there was an increasing involvement of me within
the group of men participating in the FIT-Prisons programme, which
often included within sport within programme sessions.
My Evolving position

25. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Some adaptations through the four phases
• The programme has been shortened to 10 weeks plus a measurement
week and graduation
• Some initial content (including smoking and alcohol) has been removed
• Setbacks and coping strategies are introduced early on during the
programme
• Sedentary behaviour is a consistent focus in the programme
• An In-cell workout ‘homework’ has been introduced
• There is a flexible approach to peer support
• There is flexibility around graduation (family/not family)
• The central focus on football has changed to include all sports
• Classroom elements are delivered by PEIs, local sports clubs are
involved in 2-10 PA sessions and possibly graduation

26. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Conclusions
• Ethnographic methods (including PO) can contribute illuminating
insights into health and illness within and between cultures
• PO can generate insights into the changes that participants make
within a intervention, that can shape future improvements.
• There are many, often contradictory implications for masculinities in
taking part in programmes to enhance health and wellbeing. PO
enables the collection of nuanced insights into these sometimes
subtle changes in performances of gender.
• PO is quite challenging within the prison context, where there is
constant surveillance. It requires reflection and support following
fieldwork. I am told quite a lot of disturbing and troubling things.

27. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Using participant observation to
understand school food
Stephanie Chambers
MRC/CSO SPHSU Research Skills Seminar
8 December 2015

28. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Participant observation & school food
•Multiple but limited observations
•Thick description
•Doing what your participants do
•Eating what they eat
•Listening to their casual conversation
•Interpreting this totality – to see the
[organisational] culture as a whole, to try
and get at meaning.

29. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
•Live in study site for extended time – not possible,
but good to know the area.
•Learn local language and dialect - terminology
•Participate in wide range of daily activities – when
possible
•Use everyday conversation as interview technique
•Informally observe while participating
•Record observations in fieldnotes
•Continually reflect on experiences and data
•Use both explicit and tacit information in analysis
Participant observation & school food

30. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Project 1 – Intervention development

31. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Project 1 – Intervention development

32. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Project 1 – Intervention development

33. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Project 1 – Intervention development

34. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Project 1 – Intervention development

35. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Project 1 – Intervention development

36. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Project 1 – Intervention development

37. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
One wee boy says his teeth are sore and that he can’t
eat more. He says the pizza is hard but the wedges
soft. The PT says that he wouldn’t forget if he had
sore teeth. She encourages him to eat more wedges
or to have one more bite of pizza. He sees her over at
another table and makes a run for the bin. She sees
him furtively looking around trying to dispose of his
rubbish in the bin, and sends him back to his seat.
She leaves the room after and he’s down at the bin
sensing his opportunity. Lots of the children take this
chance to get rid of their food.
Project 1 – Intervention development

38. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Project 2 – FSM Evaluation

39. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Project 2 – FSM Evaluation

40. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Project 2 – FSM Evaluation

41. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Project 2 – FSM Evaluation

42. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Project 2 – FSM Evaluation

43. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
All the children are gathered round the teacher as she
starts to go through the lunches. She introduces us to the
children and explains that we are interested in what the
children are having for their lunch. She tells the children
what she is having, and then goes through the roast dinner
which she describes as a ‘delicious yellow dinner today.’
She explains to us that that children keep their token in
their tray until lunch time.
She has the envelopes in front of her and goes through
each child’s choice, giving them their token to take to their
tray. Quiche is the other option for the day and for the
children who have chosen that she says ‘I can remember
what the quiche looks like and it’s delicious.’
Project 2 – FSM Evaluation

44. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
When I enter, [the cook] looks at me and says ‘I don’t
have time for this today’. She didn’t know that I was
coming. I know that I arranged with the head teacher, but
this might not have been passed on to the cook. I explain
that I don’t need to speak with anyone, and that I can
stand in the corner out of the way. She says ‘I don’t like
people standing watching me work.’ She says that they
are just back from the holiday and the delivery is coming
in.
I say that I’m happy to be guided by her and can go and
come back to watch the dining hall at lunch time.
Project 2 – FSM Evaluation

45. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
She says I can stay if I want but that she’s had it with the
HT as she never consults with her and she says she
shouldn’t be making appointments on her behalf.
I feel very awkward, but the thought of having driven so
long for nothing is unbearable and I don’t have the time to
come back.
I decide to stand as far into the corner as possible. I don’t
take out my pad for a while so that it doesn’t look
conspicuous but in the end I know I won’t keep it all in my
head. So I take it out and write on the table. I decide I
won’t watch and take notes at the same time.
Project 2 – FSM Evaluation

46. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Project 2 – FSM Evaluation
I think the kitchen need to take the dinners a little bit
more seriously and be more organised, show good
manners, because at times they’re very rude and we’ve
tried to speak to them about how, you know, treat people
how you want to be treated, they want a please and
thank you but they’re not giving it back, you know?
[Class teacher]

47. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Conclusion
What it participant observation added:
For project 1 – highlighted that policy aims not
being realised, limited change, difference in
teaching styles.
For project 2 – highlighted areas of concern not
discussed in interviews, more subtle issues relating
to staff interaction, provides better platform for
interpreting all project data.

48. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
The Value of Observational Work
within Evaluations
8 December 2015
Katie Buston

49. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Observational evaluative work across three
studies
• SHARE: RCT of teacher delivered
sex ed package (1996-2001)
• THRIVE: RCT of parenting
interventions for vulnerable
women (2012 - ongoing)
• Evaluation of parenting
intervention in HMP YOI
Polmont (2015)

50. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Embedding observation in process
evaluations
- Valuable component of Process Evaluations
- As part of RCT can help explain outcomes (overall and for sub-
samples):
- fidelity,
- facilitator characteristics, perceptions and motivation;
- participant engagement;
- different intervention components;
- barriers and facilitating factors around implementation
All of these aspects valuable to observe first hand, as well as
explore through interviews/focus groups (complementary)

51. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Challenges of multi-site, structured
observational work
• Sampling –
• desire to observe ‘everything’, hard to let go and
accept can only see fraction of what going on!
• Need to develop sampling strategy with tight
rationale
• Can supplement obs with self-report forms for whole
sample
• Time likely to be tight and need for focused info great;
likely to be team work context
• pro-forma to collect standardised and specific
information; be realistic re depth of fieldnotes
• Integration of process/outcome work
• Gatekeepers – time consuming to organise; potentially
threatening; need to avoid self-selection particularly in
RCT context

52. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Small scale observations: formative
evaluation work
- Opportunity to get to know participants and facilitators well
- Much more unstructured observation – not always knowing
what you are looking for/going to find interesting
- Likely to be less of a team work context so can follow your
leads, go with the flow!
- Harder to step back – much more likely to get involved
(participant obs)
- Analysis: so far very quick analysis to feedback to provider;
scope for more in depth analysis also
- Challenges: time to write up fieldnotes; what to write up in
fieldnotes; confidentiality; presentation of self; role during
observations
- Ethics (especially round personal/professional boundaries, and
what can be reported in the public domain) and gatekeeping

53. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

54. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

55. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.
Conclusion: Utility of observation in
evaluation work
- In terms of evaluating interventions, single most useful
method imo
- Even stronger if used with other methods (qual and quant)
- Further qual work eg follow up through focused depth
interviews re couple key emergent issues, or in larger scale
work select who/what to observe following interviews
- Obviously must still reflect on status of data you are collecting
– are you changing things by being there and how – but no
substitute to elucidate what actually goes on in the delivery
setting