Your SlideShare is downloading. ×
0
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Centre for population and public health research2
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Centre for population and public health research2

265

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
265
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Stakeholder analysis - Evaluation framework (Wimbush & Watson) Policy-makers (SE) Effectiveness; outcome-oriented evaluations; what works? Strategic planners (DCE) Perfprmance management and monitoring Programme/project managers Objectives-based evaluations; developmental/formative evaluations R&E specialists Knowledge-building; research quality; research utility Service users Quality of service provision, experience/how treated, relevance to needs But this is usually ‘ messy ’ – not everyone with an interest in a program will necessary have the same purpose or questions. Stakeholder management is a key skill for evaluators
  • Questions and discussion What do you think makes a good evaluation? ( data producer/user person)
  • If taking account of and managing different stakeholders views/interests & questions in an evaluation, you can ’ t address all questions all at once SO YOU NEED TO PRIORITISE This is where the above principles ( pragmatic) come into play
  • If taking account of and managing different stakeholders views/interests & questions in an evaluation, you can ’ t address all questions all at once SO YOU NEED TO PRIORITISE This is where the above principles ( pragmatic) come into play
  • Transcript

    • 1. The role of evaluation inmental health andgreenspaceDr Ruth JepsonCo-Director, Centre for Population and PublicHealth Research, University of StirlingLead for Physical Activity, Diet and HealthResearch Programme
    • 2. Outline of talkhe work at the Centre for Public Health and Population HealthResearchhy do evaluation?oints to consider in evaluationome types of evaluation techniques and three examplesood sources of support/toolkits/ training available to projects wishingto evaluate what they do
    • 3. Centre for Public Health & Population Health ResearchProgramme on physical activity and dietFocus on:•Promoting physical and healthy diet as part of everydaybehaviour•Promoting physical activity through health professionalreferrals•Understanding the barriers to physical activity and healthyeating in different population groups (including ethnic minoritygroups)•Encouraging people to use the outdoors to increase theirfeelings of health and wellbeing (including walking andgardening)Three relevant evaluation projects:•Walking groups via a GP practice•Health effects of community gardening•Indoor versus outdoor activities via Exercise Referral Schemes
    • 4. Why do evaluation?Evaluation can be powerful and exciting! (Hmm..)It can help you:•Improve services•Understand what works and what doesn’t•Demonstrate the difference that a project makes•Make decisions about the best use of funding•Have evidence for policy and decision-making (Evaluation Support Scotland) Can be your most important and influential tool for getting new or sustained funding - funders want evidence of what works
    • 5. Who is evaluation for?Evaluation Stakeholders What sort of evaluation is valued?Policy makers Effectiveness; what works?Funders Accountability/value for moneyPlanning & Performance Performance monitoring/targetsManagers Process evaluationsResearchers Knowledge building; research quality and utility Service quality – access, experience,Service users relevance to needs
    • 6. What makes a ‘good’ evaluation? Influences decision making Contributes to the evidence base ifle tu s T a ’s g o d on tti fe ne s mv go n r u u f in d Is obje cti and w ve Shows the e execu ll sH sle up ted value of what we’re doing twu ann dd hte as r cid ef ee wne r ika nr ge m a
    • 7. Before you start an evaluation• Clarify the aims of your project• Think about who you are targeting• Think about how your project will effect change• Identify what you want to achieve in the short, intermediate and long- term• Decide how you will measure what you want to achieve [if possible used well validated measures – don’t attempt to make up your own]• Think about what information you will need to collect [NEVER collect information you don’t intend to use!]
    • 8. Types of evaluationThere are 4 broad types of evaluation:•process (which deals broadly with the processesinvolved in service delivery)•outcome (which determines whether aims havebeen met & how effective the service is)Often carried out at the same time or slightlystaggered•impact (determining the wider implications ofthe service, often comparing with an areas whereno service is provided)•economic (determining whether the service iscost effective)
    • 9. Process evaluationakes place during setting up and/or delivery of the projectrovides guidance to those who are responsible for ensuringand improving the project’s qualityocuses on identifying barriers and facilitators to successfulimplementation/delivery, as well as assessing whether the keyobjectives have been metan be used to refine/modify the delivery of the projectan be used to determine the effects of the project (intended
    • 10. Good process evaluation:1. Is carried out by people not involved in the project (makes it more objective and people may answer more honestly)2. Doesn’t make assumptions about how the project works and what it achieves – sometimes there are unintended consequences3. Uses different types of data to assess the processes Qualitative (talking to people) data can help identify if the project is running as intended; meeting the needs of the participants (staff and users); the experiences (positive and negative) of participants; the changes experienced (intended and unintended); does the project work as it should? Quantitative (numbers) data should be collected to demonstrate that you are attracting the ‘right’ participants (called reach) Minimum data should include all the variables you think are relevant. For example: age, gender, postcode, ? health condition, ?physical activity level for all participants, performance data (e.g. how many referrals were made, activities carried out etc)
    • 11. Outcome Evaluation ims to answer questions such as: oes the project work? oes it provide the benefits to participants you wanted it to make? ow do you measure success? [Where possible VERY IMPORTANT to use questionnaires etc. that have been validated and used in the same type of setting/population] f possible should try and measure change when people start in the project, after a few months and after about one year. old standard would be the randomised controlled trial where you have a control group – you can be more certain that any benefits that are seen are due to the project and not due to other reasons.
    • 12. To do a good outcome evaluation• Be realistic and clear about what your project is likely to achieve – make sure you are measuring the right outcomes (e.g. improving mental health, not curing depression) –ask why you think your project would impact on these outcomes• Collect core information on everyone who comes to the project (baseline)• Consistently ask same questions to all participants• Use proper validated tools to measure outcomes• Don’t collect unnecessary data (just because you think it ‘might be interesting’)• Follow up participants at medium and longer term (e.g. 6 months & 12 months) to see if change/benefit has occurred• Demonstrate effectiveness by using experimental methods (not necessary for all projects) – aspirational but not impossible!
    • 13. Musselburgh Health Walks for sedentary peopleand/or mental health problemsAn evaluation by Roma Robertson, PhD studentrocess evaluation Is it possible to run a programme of health walks 3 times/week? Do health practitioners (HPs) refer onto the service? Do sedentary patients/people with depression utilise the service? Can the model be improved? What positive and
    • 14. Process evaluation1. Is it possible to run a programme of health walks 3 times/week?Yes: The programme of walks ran well for the planned 24 weeks. It wasorganised by CHANGES Community Health Project and supported by 8volunteer walk leaders, many of whom had attended the wellestablished wellbeing walks run by CHANGES over the years. However,without the volunteer walk leaders it would have been difficult toprovide such frequent walks and at low cost.2. Do health practitioners refer to the service?The initial plan was to recruit walk participantsvia GP consultations.Low recruitment numbers led to otheralternative recruitment strategies being introduced(so No!)
    • 15. Process evaluation. Do sedentary patients/people utilise the service?f the 19 who participated, 13 (68%) stayed until the end of the12 weeks and 9 people continued to walk with the groupbeyond the 12 weeks they had agreed to. This suggests that alarge proportion of participants valued the service.. Can the walks be improved?here was a lot of very positive feedback.hen pressed for how the walks could be improved, walkersindicated they would like a greater variety of walks; and aquarter felt the walks were too short. One person found itembarrassing to meet at the health centre and one wasworried in case someone they knew would see them on thewalks. Seven people liked going for coffee afterwards; 5disagreed or were undecided
    • 16. Outcomes evaluations there any evidence of benefits to participants?ollected data at baseline, 6 weeks and 6 monthssed validated measures of health outcomes (IMPORTANT!)ollected data on mental wellbeing, physical activity, social networks, general healthhysical activitymmediately after the intervention and 6 months later more people reported that theyook at least 20 minutes exercise on 3 days each week than before the studyental wellbeingcores for mental health improved after 12 weeks of walks, but themprovement had reduced and was no longer statistically significantmonths later
    • 17. Effects of community gardening onhealth outcomesProject by Di Blackmore, PhD student im to investigate the effect of community gardens on health and elated outcomes. The objectives of this research are to: explore a range of health effects for the individuals explore mechanisms by which the community gardening project affects health etermine how/if the outcomes vary between the different community gardens and other variables such as the amount of time spent in the garden
    • 18. MethodsIntend to recruit participants near the start of their gardeningexperience, and take baseline measures of stress level and physicalhealth: blood pressure, body mass index, activity level and salivarycortisol.In addition, participants will be asked to complete validatedquestionnaires that examine aspects of mental wellbeing, physicalactivity, quality adjusted life years, loneliness, community cohesionand social capital.This data will be collected at baseline, some measures again at 6weeks and all measures at 12 weeks.Also collecting qualitative data to explore participants experiencesof being involved in the projects, and how they felt they benefitedfrom them
    • 19. A feasibility study of Exercise Referral Scheme:indoor versus outdoor activitiesLed by Dr Larry Doi im o test the feasibility, acceptability and effectiveness of randomising patients to ERS in either indoor or outdoor activities. YES a randomised controlled trial!! esearch questions . What are the initial estimations of effectiveness of indoor versus outdoor activities on a range of health outcomes? [OE] . Are there are particular aspects of the outdoor exercise or indoor exercise, delivered via an exercise referral scheme, that confer specific health benefits? [OE] . Do the patients have strong preferences for setting of physical activity? [PE] . What are the main barriers and facilitators to implementing the outdoor intervention successfully? [PE] . What are the underlying mechanisms of action or change (why and how the outdoor and indoor activities have an effect on health outcomes) [PE]
    • 20. A feasibility study of Exercise Referral Scheme:indoor versus outdoor activitiesSetting: Bathgate, West LothianInterventions1) Indoor ERS (normal activities in leisure centre)2) Outdoor ERSIntervention to be developed but will roughly equate with duration and exercise intensify tothe indoor intervention. Components may include:1.Green gym2.Led walks3.Other outdoor activitiesDuration of the intervention period would be 12 weeks. Participants will be asked to only doactivities in the arm of the trial to which they have been allocated. After the 12 weeks allparticipants will be able to continue with the physical activities of their choice.All the outdoor activities (e.g. green gym, led walks) will remain in place for a minimum of48 weeks (making the duration of both interventions a year in total). All participants will befollowed up at one year.
    • 21. Data collected routinely by the exercise referralteameight loss BMI/ Abdominal Girthhysical activity General Practice Physical Activity Questionnairehysical fitness Peak Flowlood pressure Systolic BP/ Diastolic BPental health Hospital Anxiety and Depression Questionnaireeneral health The General Health Questionnaire (GHQ12)atient satisfaction Satisfaction surveydherence Electronic records on attendance at feedback sessionshis data is recorded electronically and is collected at several time
    • 22. Good Evaluation ResourcesEvaluation Support Scotlandhttp://www.evaluationsupportscotland.org.uk/BHF Exercise Referral Toolkithttp://www.bhfactive.org.uk/sites/Exercise-Referral-Toolkit/Standard Evaluation Framework for physical activity interventionshttp://www.noo.org.uk/uploads/doc/vid_16722_SEF_PA.pdfLearning, Evaluation and Planning (LEAP)http://www.scdc.org.uk/what/LEAP/
    • 23. Email: Ruth.jepson@stir.ac.uk

    ×