Spreading and deepening initial key strategies/diversifying/pruning strategies
Business model development, diversification and sustainability
Enhancing legitimacy and value
Building enduring social connectivity/trust
Monitoring, learning, continuous improvement and course corrections
Implementing a robust network communication platform
Enhancing and Expanding
Agreeing on shared norms and principles
Establishing initial network structures and processes (governance, admin, legal form?)
Piloting key change strategies and tactics
Formulating network theory of change and participant theories of action
Designing a user centered network communication platform
Business model formulation
Mapping of issue/problem and key stakeholders, roles and relationships
Convening leadership group, key stakeholders and relationship building
Defining the initial vision, purposes and value proposition
Securing seed funding
Evaluating network vibrancy, effectiveness and sustainability
Deciding to transform or transition
Refining/redefining network vision, purpose and value
Redirecting the issue/problem system and altering core relationships
Potential evolution to non-network structures and processes
Distribution and utilization of re-usable assets
Transforming or Transitioning
Network Assessment in Summary
Networks are Complicated
Understand both the network “parts” and “whole”
Develop appropriate network metrics
Gather comparable and consistent data across the network
Networks are Complex
Utilize results for real time learning and improvement
Manage emergent outcomes
Life cycle approach is really helpful in monitoring and evaluation of networks
In what ways do networks achieve impact/What are various network “impact pathways”?
What types of theory of change approaches are useful for networks?
Describe Network Impact Pathways
What are the various impact pathways the network and its members will use?
Will the network act as a:
Most networks will use multiple impact pathways
Network Theories of Change and Action
Identifies desirable results (What)
Articulates the multiple and often interacting pathways for affecting change (How)
Maps out the actors that contribute to various pathways (Who)
Tools You Can Use
Steps to Developing a Network ToC
Articulate the long-term network goal
Describe the various impact pathways the network and its members will utilize
Describe the medium and short-term outcomes and outputs the network can affect through its various strategies that will contribute toward the long-term goal
Articulate Network Long-Term Goals
What is the network’s long-term goal/vision?
review and confirm original long-term goal with members
revise and refine original long-term goal with members
Global Knowledge Partnership - a network of public, civil society and commercial organizations, works to provide members with access to global knowledge and innovation
Filters and disseminates knowledge and innovations gathered from its members and others throughout its network
Convenes members and others to share good practices and new knowledge during conferences and smaller workshops
Provides grants to members to support their activities
Facilitates learning across members through member meetings and joint projects
Campaign to End Pediatric HIV/AIDS, which focuses on advocacy as a means to increase coverage rates for pediatric HIV/AIDS across six partner countries:
Amplifies a particular set of ideas and policy stances on pediatric HIV/AIDS
Convenes partners at the global, national and regional level across six partner countries in order to organize a cohesive advocacy campaign
Provides resources to each of the official country partners to organize local advocacy efforts
Describe Network Outcomes
What are the network’s medium and short-term outcomes that contribute toward the long-term goal?
Brain-storm preconditions for success
Group outcomes into broader categories
Identify the specific outcome categories that the network will focus on
Steps to Developing Network Theories of Action
Examine how (or how not) the members goals align with the network’s long-term goal
Describe the specific member’s impact pathways that will contribute toward the network’s long-term goal
Describe the medium and short-term outcomes and outputs the individual member can affect through its various impact pathways that will contribute toward the networks outcomes and long-term goal
Examine Member and Network Goal Alignment
How (or how not) do an individual member’s goals align with the network’s long-term goal?
Each member examines:
How their goals relate to those of the network
How/where their activities can best support the network
Describe Member Impact Pathways
What are the specific pathways a member will use to support the network’s long-term goal?
Reviews the original list drafted by the network
Identifies which pathways they will focus on – those that best support their own and the network’s goals
Describe Member Outcomes
What are the short and medium-term outcomes the individual member can affect that contribute toward the network’s outcomes and long-term goal?
Describes the outcomes they will work toward
Identifies how these relate to those of the network
Note: one member level outcome may contribute toward multiple network level outcomes
Describe Member Outputs
What are the short-term steps that contribute towards and are necessary for the achievement of member outcomes?
Identifies how they contribute toward their outcomes
What types of metrics are particularly applicable for networks?
What types of assessment tools are particularly useful for networks?
Developing a Network Integrated Assessment Framework
Links ongoing monitoring and evaluation with punctuated outcome and impact evaluation
A set of network metrics for monitoring progress tied to the identified outcomes
A set of network tools to support the collection of metrics
Periodic evaluation including impact evaluation
Steps for Developing a Network Integrated Assessment Framework
Gather information about members’ current M&E practices and capacity
Identify output and outcome indicators
Identify vibrancy and connectivity indicators
Identify the tools that can be used to support data collection
Develop the specific process (who will collect the data, how often, etc…) for collecting and storing data
Develop and initial time-line including a schedule for periodic evaluation
Develop a system for collecting and recording data
Overall network effectiveness is combination of:
Specific indicator can be developed for each metric
Different categories of metrics may be more or less important depending on context
Example Network Metrics and Survey/Stakeholder Interview Questions Network Vibrancy Alignment Is there a match between member’s goals/priorities and that of the network? Diversity How diverse is the network? Is the current level of diversity appropriate for the network’s goals? Network Connectivity Participation How active are members? What is the value of participating in the network? Communication How well do the various network parts communicate with each other? How often do network members communicate Network Effects Influence How influential is the network? Value added Is the network fully realizing the advantages of working together?
Identifying Output and Outcome (Effects) Indicators
What are the network and its members achieving?
What are the indicators that can be used to track progress towards outputs and outcomes?
Output and outcome indicators should be tied to outputs and outcomes identified in the ToC/ToA
Identifying Vibrancy and Connectivity Indicators
How well is the “network” itself functioning?
What value added does using a network approach bring?
Network and members should monitor at least some components within the vibrancy and connectivity categories
Vibrancy and Connectivity indicators can help test original assumptions about network approach
Selecting Network Tools
Tools can be used in combination to support data gathering for specific metrics of interest
Identify the indicators and then select the appropriate tool to help gather data
Example Vibrancy Tools Tool Purpose Strength Comparative Constituency Feedback Surveys Gather stakeholder perceptions that can then be compared across similar networks Allows for benchmarking and can foster learning across networks. Opportunity to incorporate stakeholder perceptions. Network Effectiveness Diagnostic Tool Short tool to identify the strengths and weaknesses and to explore possible actions to develop or strengthen the network Relatively simple activity requiring limited time and resources. Responses could be compared across stakeholder groups. Partnership Score Card Identifies where partnerships are excelling, where they are breaking down and why. Assesses partnerships by analyzing the tangible and intangible exchanges between the parties.. Provides detail on specific partnerships of interest. Performance targets can be set and measured/assessed at regular intervals
Example Connectivity Tools Tool Purpose Strength Social Network Analysis Examines the links, relationships and behaviors between actors Analyzes behavior patterns and group structure - Can be used to asses changes in network structure over time Good for understanding the relationships between actors within the network and overall network structure Value Network Analysis Maps activities and sets of relationships identifying both tangible and intangible transactions and roles within the network Examines what specifically is being “exchanged” within the network . Identifies both tangible and intangible value. Monitoring at the edges Monitor levels of networking that do not directly involve the secretariat Relatively simple activity requiring limited time and resources
Example Social Network Analysis Map
Example Effects Tools Tool Purpose Strength Strategic Leverage Index Examines how groups come together to achieve their own and broader collective goals - Creates a conversation around values Works to measure stakeholders inter-dependencies Creates a picture of network “health” across stakeholder groups and levels Impact Pathway Evaluation Impact pathway (or ToC) developed at the start of the initiative -Monitoring of indicators aligned withthe ToC used to guide the initiative and revise/refine the ToC in real time - Evaluation seeks to establish plausible links between the initiative’s ToC and subsequent developmental changes Links external punctuated evaluation to continuous internal M&E - Makes explicit, tests and refines an initiative’s ToC Contribution Assessment Understand how and where members are interacting with the network and what their priorities are Relatively simple participatory activity requiring limited time and resources
Collecting and Recording Data
After indicators and tools have been selected, members should develop an initial time line including:
Identifying who will be responsible for collecting data
Identifying who will be responsible for recording data
Determining how often data will be collected
Determining where and how data will be stored
Determining how often punctuated evaluation will be conducted
Data can be stored in a variety of formats (tables, databases, online, etc…). members, stakeholder should discuss and decide:
Who will be the primary users of the data?
How often will data be recorded?
What needs to be recorded (qualitative or quantitative information)
Is it possible to conduct an impact evaluation of networks?
How can impact evaluations of networks be rigorously implemented?
Impact Evaluation of Networks Some Assumptions and Terminology
Bayesian approach to evaluation
Evidence requires theory (interpretation)
Match design to stage of theory development
Approaches/methods vs data collection tools
Quantitative and qualitative (data)
Data-set vs. causal process observations
Network Impact Evaluation
Map out Causal Chains
Anticipate Heterogeneity (Distributional Impacts)
Anticipate Unintended Consequences
Rigorous Factual Analysis
Rigorous Counterfactual Analysis
Use Mixed Methods
Theory of Change is Touchstone
Comparison and Triangulation: Within and Across Evaluation Designs
Single case study
The Platinum Standard Comparison and Triangulation to Address Threats to Validity
Construct Validity - how closely a measure reflects the underlying concept of interest
Statistical Conclusion Validity - whether, and by how much, presumed causes and effects co-vary
Internal Validity - whether observed co-variation represents an actual causal relationship within a specific context
External Validity - the extent that a causal inference drawn from one or more contexts/cases applies to other contexts/cases
Trade-offs across all evaluation strategies
Rigorous Advances in Comparative Method/Case Study Designs
Comparative and Case Study Methods/Tools:
General (and Specific) Elimination
Counterfactual Thought Experiments
Data-bases of Case Studies
… and so many more…
Rigorous Advances in Comparative Method and Case Study Designs
Case studies still a central “method” and can be made more rigorous with relatively small investments
Case studies essential for analysis of:
- context – location, time, scale, scope
- descriptive inference: construct validity
- causal processes (simple, complex, complicated)
- multiple results (outputs, outcomes, impacts)
- unintended results/unexpected consequences
- triangulate qualitative and quantitative data
- useful communication and reporting tool
Rigorous Advances in Comparative Case Study Designs
What is this a Case of?
With-In Case Analysis
Rigorous Comparison and Triangulation Advances in Case Study Designs
Purposes and types of Case Studies:
Generating theory of change – “inductive”
Elaborating TOC – “plausability probes”
Evaluating TOC – “least likely” and “most likely” “tough tests”, “counterfactual thought experiments”
Innovating theory of change – “deviant case studies” of exceptional “outliers”
Learning and Reporting Activities for Continuous Improvement
What are the unique types of learning and reporting needs/goals for continuous improvement of networks?
What types of learning and reporting tools are particularly useful for networks?
Continuous Learning and Improvement Activities
Structured activities that create space to analyze results in order for the network and its partners to learn and course-correct in real time
Lessons learned shared across partners at both the global, regional and national levels to scale-up success and overcome challenges
Activities are used as a planning input to future programs and investment as a way to maximize impact
Example activities include:
Interrogation of monitoring reports, scorecards, and data
Intense Period Debriefs
Example Learning Activity: Comparative Constituency Feedback Survey
Structured questionnaire to collect comparative feedback from a network’s key constituents
Setting up a cohort – within or across networks
Setting up an Advisory Group -
Designing the questionnaire
Administering the survey
Producing comparative reports
Example Learning Activity: Intense Period Debrief
Structured reflection activity after intense period of activity
Identify an “intense period” that you want to reflect upon as a team
Designate someone to facilitate
Either in focus group or individually, facilitator asks participants a set of questions
Network members to discuss results of the debrief, what has been learned, and what might be improved for future network efforts
Record results of debrief in the evidence of change journal
Example Intense Period Debrief Questions
What events triggered this intense period?
Which elements of the organization’s response worked well? Which elements could have been improved?
What was the outcome of the intense period? Was the result positive or negative?
What insights will you take away from this experience that might inform your strategies going forward?
Developing Stakeholder, Public and Donor Reporting Processes
Explicitly sharing evaluation findings with constituents:
Provides new insight about effects on key constituents, can inform the network’s assessments of its efforts, and help it to course correct.
Informs the network’s understanding of important, new aspects of its performance.
Empowers constituents by amplifying their voices. The process gives constituents a new opportunity to participate in the network and increase their sense of stake in it.
Helps the network grow in legitimacy as a result of its visible efforts to be accountable to its constituents.
What should be included in a public report?
Formats may vary, but public reports should address questions like:
What was to be achieved?
What was achieved?
What caused or hindered success?
What’s working well?
What should be changed and why?
What has been changed and how?
Network Impact Planning, Assessment, Learning and Reporting Systems Overview
An overall framework for planning, monitoring and evaluating networks
Consists of a general set of key components and a menu of tailored options
Meets some of the challenges of network evaluation
Key Network IPARL Components
A clearly articulated theory of change and aligned theories of action
An integrated assessment framework composed of:
A set of network metrics for monitoring progress and accountability
A set of network tools
Periodic evaluation – Including impact evaluation
Stakeholder, public and donor reporting and feedback
Continuous learning and improvement mechanisms
Which network monitoring, evaluation and learning ideas and tools seem most helpful and why?
How might I use different network monitoring, evaluation and learning tools?
How might I use an IPARL system in my work with networks?
Next Generation Network Evaluation Paper
Step-by-Step guide to developing Comparative Constituency Feed Back Surveys for Networks