Evaluating networks slides_final_monitor3.8.11
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Evaluating networks slides_final_monitor3.8.11

on

  • 547 views

 

Statistics

Views

Total Views
547
Views on SlideShare
535
Embed Views
12

Actions

Likes
0
Downloads
6
Comments
0

1 Embed 12

http://networksguide.wikispaces.com 12

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • EX: World Bank Global and Regional Partnership Program began in mid-1990s – now supporting over 175 partnership programs and spending $3.5 billion annually EX: Climate Works Foundation – organized 10 funders and actors across sectors/geographies as part of a billion dollar networked campaign to fight climate change EX: Grant makers like Packard, Ford, Casey, MacArthur and other Foundations supporting networks as a key part of their portfolio and strategy

Evaluating networks slides_final_monitor3.8.11 Presentation Transcript

  • 1. Monitoring and Evaluating Networks March 5, 2011
  • 2. Seminar Overview
    • Overview of Networks as Complicated and Complex Initiatives - Network Life-Cycle
    • Network Theories of Change and Action
    • Developing a Network Integrated Assessment Framework – Focus on Network Metrics
    • Learning and Reporting Activities for Continuous Improvement
    • Overview of IPARL system Across the Network Life-Cycle
  • 3. Impacts Community of Practice
    • Initial focus on sharing and developing innovative approaches to monitoring and evaluating networks and network approaches – now moving to greater focus on advocacy
    • Includes – Evaluation experts, practitioners, network representatives and funders (in-person and virtual interactions – www.scalingimpact.net)
  • 4. Additional Materials
    • Network Briefs
    • Next Generation Network Evaluation Paper
    • Step-by-Step guide to developing Comparative Constituency Feed Back Surveys for Networks
    • Available at www.scalingimpact.net
  • 5. Introduction and Context
    • Networks and networking have become increasingly and consciously utilized
    • Increased interest and investment in networks from a range of actors
    • Growing demand for more success from public and private initiatives has created explosion of interest for increased and improved M&E/IE
    • Together these trends have created a growing appetite for Network M&E/IE
  • 6. Current Field of Network Monitoring and Evaluation
    • Still in its infancy in terms of theory and even more so in terms of practice
    • Most M&E work using network methodologies to date have been small in size of networks and number of participants
    • Number of metrics, tools and broader approaches developed specifically for networks is growing but not many practical tests/examples
    • Much can be borrowed from other fields like complexity science and systems thinking
  • 7. Gaps in Current Practice
    • Growing number of network metrics but not utilized in practice
    • Most tools and approaches only address one or a few of the specific challenges faced by networks
    • Little explicit focus on revising and refining network’s ToC
    • Few comprehensive frameworks for developing holistic M&E systems
    • Those that do encourage development of broader M&E systems focus more on planning than M&E
  • 8. Networks as Complicated and Complex Initiatives
    • Networks and networking strategies:
        • Involve multiple factors, actors and contexts
        • Are non-linear, path dependent and emergent
    • Network implementation varies even when replication mechanisms are applied because of contextual differences
  • 9. Standardized, Complicated and Complex Initiatives
  • 10. Network Life Cycle (Simplified)
    • Increasing and diversifying network membership and participation
    • Developing network structures (geographically and/or functionally)
    • Spreading and deepening initial key strategies/diversifying/pruning strategies
    • Business model development, diversification and sustainability
    • Enhancing legitimacy and value
    • Building enduring social connectivity/trust
    • Monitoring, learning, continuous improvement and course corrections
    • Implementing a robust network communication platform
    Enhancing and Expanding
    • Agreeing on shared norms and principles
    • Establishing initial network structures and processes (governance, admin, legal form?)
    • Piloting key change strategies and tactics
    • Formulating network theory of change and participant theories of action
    • Designing a user centered network communication platform
    • Business model formulation
    Launching
    • Mapping of issue/problem and key stakeholders, roles and relationships
    • Convening leadership group, key stakeholders and relationship building
    • Defining the initial vision, purposes and value proposition
    • Securing seed funding
    Catalyzing
    • Evaluating network vibrancy, effectiveness and sustainability
    • Deciding to transform or transition
    • If transforming-
      • Refining/redefining network vision, purpose and value
      • Redirecting the issue/problem system and altering core relationships
    • If transitioning-
      • Potential evolution to non-network structures and processes
      • Distribution and utilization of re-usable assets
    Transforming or Transitioning
  • 11. Network Assessment in Summary
    • Networks are Complicated
      • Understand both the network “parts” and “whole”
      • Develop appropriate network metrics
      • Gather comparable and consistent data across the network
    • Networks are Complex
      • Identify impacts
      • Utilize results for real time learning and improvement
      • Manage emergent outcomes
    • Life cycle approach is really helpful in monitoring and evaluation of networks
  • 12. Group Discussion
    • In what ways do networks achieve impact/What are various network “impact pathways”?
    • What types of theory of change approaches are useful for networks?
  • 13. Describe Network Impact Pathways
    • What are the various impact pathways the network and its members will use?
    • Will the network act as a:
      • Filter
      • Amplifier
      • Convener
      • Community Builder
      • Funder
      • Capacity Building
      • Facilitator
    • Most networks will use multiple impact pathways
  • 14. Network Theories of Change and Action
    • Identifies desirable results (What)
    • Articulates the multiple and often interacting pathways for affecting change (How)
    • Maps out the actors that contribute to various pathways (Who)
  • 15. Tools You Can Use
    • Outcome Mapping
    • Causal Mapping
    • Systems Approaches
  • 16. Steps to Developing a Network ToC
    • Articulate the long-term network goal
    •  
    • Describe the various impact pathways the network and its members will utilize
    • Describe the medium and short-term outcomes and outputs the network can affect through its various strategies that will contribute toward the long-term goal
  • 17. Articulate Network Long-Term Goals
    • What is the network’s long-term goal/vision?
    • Opportunity to:
      • review and confirm original long-term goal with members
    • Or…..
      • revise and refine original long-term goal with members
  • 18. Example
    • Global Knowledge Partnership - a network of public, civil society and commercial organizations, works to provide members with access to global knowledge and innovation
    • Filters and disseminates knowledge and innovations gathered from its members and others throughout its network
    • Convenes members and others to share good practices and new knowledge during conferences and smaller workshops
    • Provides grants to members to support their activities
    • Facilitates learning across members through member meetings and joint projects
  • 19. Example
    • Campaign to End Pediatric HIV/AIDS, which focuses on advocacy as a means to increase coverage rates for pediatric HIV/AIDS across six partner countries:
    • Amplifies a particular set of ideas and policy stances on pediatric HIV/AIDS
    • Convenes partners at the global, national and regional level across six partner countries in order to organize a cohesive advocacy campaign
    • Provides resources to each of the official country partners to organize local advocacy efforts
  • 20. Describe Network Outcomes
    • What are the network’s medium and short-term outcomes that contribute toward the long-term goal?
    • Brain-storm preconditions for success
    • Group outcomes into broader categories
    • Identify the specific outcome categories that the network will focus on
  • 21. Steps to Developing Network Theories of Action
    • Examine how (or how not) the members goals align with the network’s long-term goal
    •  
    • Describe the specific member’s impact pathways that will contribute toward the network’s long-term goal
    •  
    • Describe the medium and short-term outcomes and outputs the individual member can affect through its various impact pathways that will contribute toward the networks outcomes and long-term goal
  • 22. Examine Member and Network Goal Alignment
    • How (or how not) do an individual member’s goals align with the network’s long-term goal?
    • Each member examines:
      • How their goals relate to those of the network
      • How/where their activities can best support the network
  • 23. Describe Member Impact Pathways
    • What are the specific pathways a member will use to support the network’s long-term goal?
    • Each member:
      • Reviews the original list drafted by the network
      • Identifies which pathways they will focus on – those that best support their own and the network’s goals
  • 24. Describe Member Outcomes
    • What are the short and medium-term outcomes the individual member can affect that contribute toward the network’s outcomes and long-term goal?
    • Each member
      • Describes the outcomes they will work toward
      • Identifies how these relate to those of the network
      • Note: one member level outcome may contribute toward multiple network level outcomes
  • 25. Describe Member Outputs
    • What are the short-term steps that contribute towards and are necessary for the achievement of member outcomes?
    • Each Member
      • Describes outputs
      • Identifies how they contribute toward their outcomes
  • 26. Group Discussion
    • What types of metrics are particularly applicable for networks?
    • What types of assessment tools are particularly useful for networks?
  • 27. Developing a Network Integrated Assessment Framework
    • Links ongoing monitoring and evaluation with punctuated outcome and impact evaluation
    • Includes:
      • A set of network metrics for monitoring progress tied to the identified outcomes
      • A set of network tools to support the collection of metrics
      • Periodic evaluation including impact evaluation
  • 28. Steps for Developing a Network Integrated Assessment Framework
    • Gather information about members’ current M&E practices and capacity
    • Develop Measures
      • Identify output and outcome indicators
      • Identify vibrancy and connectivity indicators
    • Identify the tools that can be used to support data collection
    • Develop the specific process (who will collect the data, how often, etc…) for collecting and storing data
    • Develop and initial time-line including a schedule for periodic evaluation
    • Develop a system for collecting and recording data
  • 29. Developing Measures
    • Overall network effectiveness is combination of:
      • Network Vibrancy
      • Network Connectivity
      • Network Effects
    • Specific indicator can be developed for each metric
    • Different categories of metrics may be more or less important depending on context
  • 30. Example Network Metrics and Survey/Stakeholder Interview Questions Network Vibrancy Alignment Is there a match between member’s goals/priorities and that of the network? Diversity How diverse is the network? Is the current level of diversity appropriate for the network’s goals? Network Connectivity Participation How active are members? What is the value of participating in the network? Communication How well do the various network parts communicate with each other? How often do network members communicate Network Effects Influence How influential is the network? Value added Is the network fully realizing the advantages of working together?
  • 31. Identifying Output and Outcome (Effects) Indicators
    • What are the network and its members achieving?
    • What are the indicators that can be used to track progress towards outputs and outcomes?
    • Output and outcome indicators should be tied to outputs and outcomes identified in the ToC/ToA
  • 32. Identifying Vibrancy and Connectivity Indicators
    • How well is the “network” itself functioning?
    • What value added does using a network approach bring?
    • Network and members should monitor at least some components within the vibrancy and connectivity categories
    • Vibrancy and Connectivity indicators can help test original assumptions about network approach
  • 33. Selecting Network Tools
    • Tools can be used in combination to support data gathering for specific metrics of interest
    • Identify the indicators and then select the appropriate tool to help gather data
  • 34. Example Vibrancy Tools Tool Purpose Strength Comparative Constituency Feedback Surveys Gather stakeholder perceptions that can then be compared across similar networks Allows for benchmarking and can foster learning across networks. Opportunity to incorporate stakeholder perceptions. Network Effectiveness Diagnostic Tool Short tool to identify the strengths and weaknesses and to explore possible actions to develop or strengthen the network Relatively simple activity requiring limited time and resources. Responses could be compared across stakeholder groups. Partnership Score Card Identifies where partnerships are excelling, where they are breaking down and why. Assesses partnerships by analyzing the tangible and intangible exchanges between the parties.. Provides detail on specific partnerships of interest. Performance targets can be set and measured/assessed at regular intervals
  • 35. Example Connectivity Tools Tool Purpose Strength Social Network Analysis Examines the links, relationships and behaviors between actors Analyzes behavior patterns and group structure - Can be used to asses changes in network structure over time Good for understanding the relationships between actors within the network and overall network structure Value Network Analysis Maps activities and sets of relationships identifying both tangible and intangible transactions and roles within the network Examines what specifically is being “exchanged” within the network . Identifies both tangible and intangible value. Monitoring at the edges Monitor levels of networking that do not directly involve the secretariat Relatively simple activity requiring limited time and resources
  • 36. Example Social Network Analysis Map
  • 37. Example Effects Tools Tool Purpose Strength Strategic Leverage Index Examines how groups come together to achieve their own and broader collective goals - Creates a conversation around values Works to measure stakeholders inter-dependencies Creates a picture of network “health” across stakeholder groups and levels Impact Pathway Evaluation Impact pathway (or ToC) developed at the start of the initiative -Monitoring of indicators aligned withthe ToC used to guide the initiative and revise/refine the ToC in real time - Evaluation seeks to establish plausible links between the initiative’s ToC and subsequent developmental changes Links external punctuated evaluation to continuous internal M&E - Makes explicit, tests and refines an initiative’s ToC Contribution Assessment Understand how and where members are interacting with the network and what their priorities are Relatively simple participatory activity requiring limited time and resources
  • 38. Collecting and Recording Data
    • After indicators and tools have been selected, members should develop an initial time line including:
      • Identifying who will be responsible for collecting data
      • Identifying who will be responsible for recording data
      • Determining how often data will be collected
      • Determining where and how data will be stored
      • Determining how often punctuated evaluation will be conducted
    • Data can be stored in a variety of formats (tables, databases, online, etc…). members, stakeholder should discuss and decide:
      • Who will be the primary users of the data?
      • How often will data be recorded?
      • What needs to be recorded (qualitative or quantitative information)
  • 39. Group Discussion
    • Is it possible to conduct an impact evaluation of networks?
    • How can impact evaluations of networks be rigorously implemented?
  • 40. Impact Evaluation of Networks Some Assumptions and Terminology
    • Bayesian approach to evaluation
    • Evidence requires theory (interpretation)
    • Match design to stage of theory development
    • Approaches/methods vs data collection tools
    • Quantitative and qualitative (data)
    • Data-set vs. causal process observations
  • 41. Network Impact Evaluation
    • Map out Causal Chains
    • Anticipate Heterogeneity (Distributional Impacts)
    • Anticipate Unintended Consequences
    • Understand Context
    • Rigorous Factual Analysis
    • Rigorous Counterfactual Analysis
    • Use Mixed Methods
    • Theory of Change is Touchstone
  • 42. Comparison and Triangulation: Within and Across Evaluation Designs
    • Ethnographic
    • Single case study
    • Comparative (case-study)
    • Statistical/econometric
    • Experimental/quasi-experimental
  • 43. The Platinum Standard Comparison and Triangulation to Address Threats to Validity
    • Construct Validity - how closely a measure reflects the underlying concept of interest
    • Statistical Conclusion Validity - whether, and by how much, presumed causes and effects co-vary
    • Internal Validity - whether observed co-variation represents an actual causal relationship within a specific context
    • External Validity - the extent that a causal inference drawn from one or more contexts/cases applies to other contexts/cases
    • Trade-offs across all evaluation strategies
  • 44. Rigorous Advances in Comparative Method/Case Study Designs
    • Comparative and Case Study Methods/Tools:
    • Process-Tracing
    • Analytic Induction
    • General (and Specific) Elimination
    • Counterfactual Thought Experiments
    • Structured-focused comparisons
    • Boolean Algebra
    • Data-bases of Case Studies
    • … and so many more…
  • 45. Rigorous Advances in Comparative Method and Case Study Designs
    • Case studies still a central “method” and can be made more rigorous with relatively small investments
    • Case studies essential for analysis of:
    • - context – location, time, scale, scope
    • - descriptive inference: construct validity
    • - causal processes (simple, complex, complicated)
    • - multiple results (outputs, outcomes, impacts)
    • - unintended results/unexpected consequences
    • - triangulate qualitative and quantitative data
    • - useful communication and reporting tool
  • 46. Rigorous Advances in Comparative Case Study Designs
    • What is this a Case of?
    • With-In Case Analysis
    • Cross-Case Analysis
  • 47. Rigorous Comparison and Triangulation Advances in Case Study Designs
    • Purposes and types of Case Studies:
    • Generating theory of change – “inductive”
    • Elaborating TOC – “plausability probes”
    • Evaluating TOC – “least likely” and “most likely” “tough tests”, “counterfactual thought experiments”
    • Innovating theory of change – “deviant case studies” of exceptional “outliers”
  • 48. Learning and Reporting Activities for Continuous Improvement
    • What are the unique types of learning and reporting needs/goals for continuous improvement of networks?
    • What types of learning and reporting tools are particularly useful for networks?
  • 49. Continuous Learning and Improvement Activities
    • Structured activities that create space to analyze results in order for the network and its partners to learn and course-correct in real time
    • Lessons learned shared across partners at both the global, regional and national levels to scale-up success and overcome challenges
    • Activities are used as a planning input to future programs and investment as a way to maximize impact
    • Example activities include:
      • Interrogation of monitoring reports, scorecards, and data
      • Stakeholder Ratings
      • Intense Period Debriefs
  • 50. Example Learning Activity: Comparative Constituency Feedback Survey
    • Structured questionnaire to collect comparative feedback from a network’s key constituents
    • Steps:
    • Setting up a cohort – within or across networks
    • Setting up an Advisory Group -
    • Designing the questionnaire
    • Administering the survey
    • Producing comparative reports
    • Follow up
  • 51. Example Learning Activity: Intense Period Debrief
    • Structured reflection activity after intense period of activity
    • Steps:
    • Identify an “intense period” that you want to reflect upon as a team
    • Designate someone to facilitate
    • Either in focus group or individually, facilitator asks participants a set of questions
    • Network members to discuss results of the debrief, what has been learned, and what might be improved for future network efforts
    • Record results of debrief in the evidence of change journal
  • 52. Example Intense Period Debrief Questions
    • What events triggered this intense period?
    • Which elements of the organization’s  response worked well?  Which elements could have been improved?
    • What was the outcome of the intense  period? Was the result positive or negative?
    • What insights will you take away from this experience that might inform your strategies going forward?
  • 53. Developing Stakeholder, Public and Donor Reporting Processes
    • Explicitly sharing evaluation findings with constituents: 
    • Provides new insight about effects on key constituents, can inform the network’s assessments of its efforts, and help it to course correct.
    • Informs the network’s understanding of important, new aspects of its performance.
    • Empowers constituents by amplifying their voices. The process gives constituents a new opportunity to participate in the network and increase their sense of stake in it.
    • Helps the network grow in legitimacy as a result of its visible efforts to be accountable to its constituents.
  • 54. What should be included in a public report?
    • Formats may vary, but public reports should address questions like:
      • What was to be achieved?
      • What was achieved?
      • What caused or hindered success?
      • What’s working well?
      • What should be changed and why?
      • What has been changed and how?
  • 55. Network Impact Planning, Assessment, Learning and Reporting Systems Overview
    • An overall framework for planning, monitoring and evaluating networks
    • Consists of a general set of key components and a menu of tailored options
    • Meets some of the challenges of network evaluation
  • 56. Key Network IPARL Components
    • A clearly articulated theory of change and aligned theories of action
    • An integrated assessment framework composed of:
      • A set of network metrics for monitoring progress and accountability
      • A set of network tools
      • Periodic evaluation – Including impact evaluation
    • Stakeholder, public and donor reporting and feedback
    • Continuous learning and improvement mechanisms
  • 57. Group Discussion
    • Which network monitoring, evaluation and learning ideas and tools seem most helpful and why?
    • How might I use different network monitoring, evaluation and learning tools?
    • How might I use an IPARL system in my work with networks?
  • 58. Additional Materials
    • Network Briefs
    • Next Generation Network Evaluation Paper
    • Step-by-Step guide to developing Comparative Constituency Feed Back Surveys for Networks
    • Available at www.scalingimpact.net