Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The value of engagement


Published on

Presented by John Young (ODI) and Laura Harper (Wellcome) at the Public Engagement Workshop, 2-5 Dec. 2008, KwaZulu-Natal South Africa,

  • Be the first to comment

  • Be the first to like this

The value of engagement

  1. 1. The value of engagement John Young [email_address] Laura Harper
  2. 2. <ul><li>Buzz: “What do you think would constitute good evidence of the value of public engagement?” </li></ul><ul><li>Presentation: Some approaches to measuring, learning and sharing knowledge. </li></ul><ul><li>Groups: What do we need to learn and share about engagement, and how we’d like to do it? </li></ul>Introduction
  3. 3. Why worry about it <ul><li>Because WT is. </li></ul><ul><li>Because all other research donors are. </li></ul><ul><li>Because we all think it’s important, but don’t really know what works: </li></ul><ul><ul><li>How (exactly) does research influence policy? </li></ul></ul><ul><ul><li>Should we invest energy engaging with legislators or bureaucrats or the media or the public, or schoolchildren? </li></ul></ul><ul><ul><li>Are we even achieving what we’re aiming to achieve? </li></ul></ul>
  4. 4. Evidence <ul><ul><li>Discuss with your neighbours: </li></ul></ul><ul><li>“ What do you think would be convincing evidence of the value of public engagement?” </li></ul><ul><li>Write down whatever you come up with. </li></ul>
  5. 5. The DELIVERI Programme <ul><li>Developing, testing and promoting new forms of animal health services in Indonesia: </li></ul><ul><li>Pilot projects with farmers & field staff </li></ul><ul><li>Training and capacity development for all </li></ul><ul><li>Institutional development </li></ul><ul><li>Quality management </li></ul><ul><li>Communication & advocacy </li></ul><ul><li>“ The DELIVERI programme has developed some useful models of institutional change in the context of decentralisation, making a government service more responsive to the needs of local people” 1 </li></ul>1 DFID Country Strategy Paper for Indonesia Sept. 2000
  6. 6. ODI and RAPID <ul><li>UK’s leading development Think Tank. </li></ul><ul><li>RAPID: Promoting greater use of research-based evidence in development policy & practice </li></ul><ul><ul><li>Research / Advice / Information and Capacity Development </li></ul></ul><ul><ul><li>Working with all stakeholders </li></ul></ul><ul><ul><li>Case studies, frameworks, toolkits </li></ul></ul><ul><ul><li>evidence-based policy in development network (ebpdn) </li></ul></ul> /
  7. 7. Forms of “engagement” <ul><li>With policy-makers, practitioners and communities to: </li></ul><ul><li>Identify problems & issues for research </li></ul><ul><li>Develop research projects and methodologies </li></ul><ul><li>Undertake the research </li></ul><ul><li>Feedback, discuss and validate the results of research </li></ul><ul><li>Formulate solutions – policies and programmes </li></ul><ul><li>Implement the solutions – training and capacity development </li></ul><ul><li>Evaluate their effectiveness </li></ul>
  8. 8. Forms of “engagement” <ul><li>...and we do it through: </li></ul><ul><li>(Literature reviews) </li></ul><ul><li>Telephone calls, e-mail, (and video conferences) </li></ul><ul><li>Face to face 1:1 meetings & field trips </li></ul><ul><li>Meetings, workshops & seminars </li></ul><ul><li>Collaborative work / projects </li></ul><ul><li>Sharing draft outputs for comment </li></ul><ul><li>Web 2 – blogs, wikis, discussion groups </li></ul><ul><li>Print & web publications </li></ul><ul><li>The media </li></ul>
  9. 9. Policy processes are complex Monitoring and Evaluation Agenda Setting Decision Making Policy Implementation Policy Formulation Civil Society Donors Cabinet Parliament Ministries Private Sector
  10. 10. Chronic Poverty in Uganda Kate Bird et al, Fracture Points in Social Policies for Chronic Poverty Reduction, ODI WP242, 2004 (
  11. 11. Factors influencing uptake External Influences Socio-economic and cultural influences, donor policies etc The political context – political and economic structures and processes, culture, institutional pressures, incremental vs radical change. The evidence – credibility, the degree it challenges received wisdom, research approaches and methodology, simplicity of the message, how it is packaged. The links between policy and research communities – networks, relationships, power, competing discourses, trust, knowledge etc.
  12. 12. A Practical Framework political context evidence links Politics and Policymaking Media, Advocacy, Networking Research, learning & thinking External Influences Scientific information exchange & validation Policy analysis, & research Campaigning, Lobbying
  13. 13. Health Care in Tanzania <ul><li>“ The results of household disease surveys informed processes of health service reform which contributed to a 43 and 46 per cent reduction in infant mortality between 2000 and 2003 in two districts in rural Tanzania.” </li></ul>TEHIP Project, Tanzania:
  14. 14. What should you measure? <ul><li>It depends what you’re trying to do…. </li></ul>“ If you don't know where you are going, any road will get you there”
  15. 15. <ul><li>Should be: </li></ul><ul><li>S pecific </li></ul><ul><li>M easurable </li></ul><ul><li>A chievable </li></ul><ul><li>R ealistic </li></ul><ul><li>T ime-bound </li></ul><ul><li>( O bjective) </li></ul>Whatever you measure
  16. 16. <ul><li>...and many projects fail when the inputs cease... </li></ul>Change takes a long time Inputs Activities Outputs Outcomes Impact Other Actors Project Effort Behaviour Change
  17. 17. Focusing on change OUTCOME MAPPING: Building Learning and Reflection into Development Programs Sarah Earl, Fred Carden, and Terry Smutylo
  18. 18. Emphasis on “learning” “… every time we do something again, we should do it better than the last time… ” Goals Results Activities External networks; Colleagues; Information assets; Own knowledge Learn during Learn after Learn before
  19. 19. Starts with the attitude that someone has probably already done what I am about to do. I wonder who?” Learning before: Peer Assist
  20. 20. Learning During: Stories <ul><li>What was the situation? </li></ul><ul><li>What was the challenge? </li></ul><ul><li>What was done? </li></ul><ul><li>What was the result? </li></ul><ul><li>What lessons can be drawn? </li></ul> <ul><li>Most significant change </li></ul><ul><li>Best stories at each level </li></ul><ul><li>Synthesis </li></ul>Stories of change
  21. 21. Horizontal evaluation <ul><li>Peer review </li></ul><ul><ul><li>Choose the moment </li></ul></ul><ul><ul><li>Choose your peers </li></ul></ul><ul><ul><li>Limited criteria </li></ul></ul><ul><ul><li>e.g. ODI Peer Review </li></ul></ul><ul><li>Appreciative enquiry </li></ul><ul><ul><li>Self-evaluation </li></ul></ul><ul><ul><li>CGIAR/CIAT </li></ul></ul><ul><ul><li>Workshop </li></ul></ul>
  22. 22. Learning after: AAR An after action review asks 4 simple questions: 15 minute team debrief, conducted in a “rank-free” environment. <ul><li>What was supposed to happen? </li></ul><ul><li>What actually happened? </li></ul><ul><li>Why was there a difference? </li></ul><ul><li>What can we learn from it? </li></ul>
  23. 23. Case & Episode Studies <ul><li>Classical case studies: how did evidence shape policy decisions? </li></ul><ul><ul><li>e.g. IFPRI & IDRC </li></ul></ul><ul><ul><li>Overestimate the role of research </li></ul></ul> <ul><li>Episode studies: retrospective tracking back from policy change </li></ul><ul><ul><li>e.g. PRSPs, SL, AHC </li></ul></ul><ul><ul><li>Underestimate the role of research </li></ul></ul>
  24. 24. RAPID Outcome Mapping
  25. 25. Social Network Analysis
  26. 26. Other approaches: Public <ul><li>Citations, webstats, media logs etc </li></ul><ul><li>Surveys </li></ul><ul><ul><li>Quantitative </li></ul></ul><ul><ul><li>Qualitative </li></ul></ul><ul><li>Distribution lists and attendance records </li></ul><ul><li>Meeting evaluations </li></ul><ul><li>Logs: </li></ul><ul><ul><li>The expected </li></ul></ul><ul><ul><li>The unexpected </li></ul></ul><ul><ul><li>How you have changed </li></ul></ul>Evaluation: Practical Guidelines, Research Councils UK. 2002
  27. 27. Other approaches: Policy <ul><li>Strategy and direction: Logframes; Social Network Analysis; Impact Pathways; Modular Matrices </li></ul><ul><li>Management : ‘Fit for Purpose’ Reviews; ‘Lighter Touch’ Quality Audits; Horizontal Evaluation; Appreciative Inquiry </li></ul><ul><li>Outputs: Evaluating academic articles and research reports; Evaluating policy and briefing papers; Evaluating websites; Evaluating networks; After Action Reviews </li></ul><ul><li>Uptake: Impact Logs; New Areas for Citation Analysis; User Surveys </li></ul><ul><li>Outcomes and impacts: Outcome Mapping; RAPID Outcome Assessment; Most Significant Change; Innovation Histories; Episode Studies </li></ul>
  28. 28. Learning from each other <ul><li>Face-to-face meetings </li></ul><ul><li>Establishing a network </li></ul><ul><li>E-mail </li></ul><ul><li>Websites </li></ul><ul><li>Collaborative work </li></ul><ul><li>Publications </li></ul><ul><li>D-groups </li></ul> already are a sort of network
  29. 29. Network Functions Facilitators / learners Community builders Investor / providers Convenors Filters Amplifiers Support Agency
  30. 30. Keys to Success <ul><li>Clear governance. </li></ul><ul><li>Strength in numbers. </li></ul><ul><li>Representativeness. </li></ul><ul><li>Quality of evidence. </li></ul><ul><li>Packaging of evidence </li></ul><ul><li>Persistence. </li></ul><ul><li>Membership of key individuals. </li></ul><ul><li>Making use of informal links. </li></ul><ul><li>Complementing official structures. </li></ul><ul><li>Good use of ICTs. </li></ul> Or contact Enrique Mendizabal – [email_address]
  31. 31. Community of practice <ul><li>“ a group of individuals participating in communal activity, and experiencing/continuously creating their shared identity through engaging in and contributing to the practices of their communities” </li></ul>Wenger, Etienne (1998), Communities of Practice: Learning, Meaning, and Identity , Cambridge: Cambridge University Press, ISBN 978-0-521-66363-2 
  32. 32. e vidence b ased p olicy in d evelopment n etwork <ul><li>To promote greater use of research-based evidence in development policy & practice </li></ul><ul><li>Research </li></ul><ul><li>Consultations with CSOs and other stakeholders </li></ul><ul><li>Capacity-development </li></ul><ul><li>Collaborative action-research </li></ul><ul><li>Joint projects </li></ul><ul><li>Mutual learning </li></ul>
  33. 33. e vidence b ased p olicy in d evelopment n etwork <ul><li>To promote greater use of research-based evidence in development policy & practice </li></ul><ul><li>Research </li></ul><ul><li>Consultations with CSOs and other stakeholders </li></ul><ul><li>Capacity-development </li></ul><ul><li>Collaborative action-research </li></ul><ul><li>Joint projects </li></ul><ul><li>Mutual learning </li></ul> <ul><li>Meet annually </li></ul>Interactive map of members
  34. 34. Further information <ul><li>MandE News by Rick Davies: </li></ul><ul><li>Psci-com (practical guides section) by the Wellcome Trust: </li></ul><ul><li>Wellcome Trust Researcher Support Links: </li></ul><ul><li>RAPID Website: </li></ul><ul><li>John Young: [email_address] </li></ul>
  35. 35. Group work – 4 Questions: <ul><li>What sort of evidence do you need in your own project(s) to make sure you are on track (and how to collect it)? </li></ul><ul><li>What sort of evidence would you like to have about other public engagement projects? </li></ul><ul><li>How would you like to get that evidence (and share your own)? </li></ul><ul><li>Who should do what (you, other projects, 3 rd party, Wellcome)? </li></ul>
  36. 36. Process <ul><li>4 groups, each + facilitator & rapporteur </li></ul><ul><li>Split into 2 sub-groups: </li></ul><ul><ul><li>Own evidence </li></ul></ul><ul><ul><li>Evidence from others </li></ul></ul><ul><li>Whole group </li></ul><ul><ul><li>How to get it </li></ul></ul><ul><ul><li>Who should do what </li></ul></ul><ul><li>Max. 3 responses to each question, highlighting the most important. </li></ul><ul><li>Report this back to plenary </li></ul><ul><li>“ Discussion” on D-Groups </li></ul>
  37. 37. Focus and location <ul><li>Policy – in the Auditorium </li></ul><ul><ul><li>Fac: Michelle Jimenez </li></ul></ul><ul><ul><li>Rap: Greer Van Zyl </li></ul></ul><ul><li>Community – in the Boardroom </li></ul><ul><ul><li>Fac: Bella Starling </li></ul></ul><ul><ul><li>Rap: Monica Bonaccorso </li></ul></ul><ul><li>Media – in the canteen </li></ul><ul><ul><li>Fac: Craig Brierly </li></ul></ul><ul><ul><li>Rap: Katrina Nevin Ridley </li></ul></ul><ul><li>Creative/Other – on the Terrace </li></ul><ul><ul><li>Fac: Laura Harper </li></ul></ul><ul><ul><li>Rap: Marina Joubert </li></ul></ul>