How we think about our impactSustainable marketplaceGrowing #s of participating orgsHelping those orgs be more effectiveUsing data to inform the sector as a whole
Michael Luca – Harvard Business School researcherhttp://www.hbs.edu/research/pdf/12-016.pdf+1 star = + 5%-9% RevenueOnly independent restaurants – no effect on chains.Chain restaurant market share declines with Yelp penetration.Yelp covers 70% of restaurants. Seattle Times covers 5% Michelin does not cover Seattle.Individual feedback influences the market.Smart store owners seek feedback and react to it.New markers of quality and reliability.More value created benefitting larger, more local group of restaurants.
This is what we’re trying to achieve.But how to we realize these Yelp-like effects within our marketplace?
Prompting question or image triggers a lived experienceStoryteller self-indexes story, gives meaning (can be done by others)Software detects visual patterns among storiesSoftware enables deeper dive into individual storiesPeople discuss patterns and story clustersAct on patterns, weak signals, phase shiftsStimulate beneficial patterns; dampen undesirable ones
Working! Library of Lived ExperiencesVisualizing Shifting Story PatternsDissemination to Official Aid SectorPossibility of Actionable Constituent VoiceUseful to GlobalGivingPromising!Rolling BaselinesUseful to DonorsUseful for OrganizationsUseful for Beneficiary GroupsInformation Challenges Existing FrameworksCross-Organizational ThinkingWorking on it!Stories facilitate decision-makingStories are basis for evidence-based policy changesFraming the ImpactPeer-to-Peer Knowledge Management
This is for an advocacy initiative. Implementing program work is not significantly different.
Concept of interorganiztaional relationships comes from Heather Creech and UniversaliaFrequently a conflation in the literature between coalition and networkCoalitions are almost always organized around member organizations, whereas networks are split between member networks and individual networks; however, in both cases, it is individuals that do the activitiesNatural gravitation toward increased structure or chaos leads to fluidity among the types of IOR that a single entity will go through.
Important to name these things.
So, up to this point you see a notion of convergence—by and large we are talking about nuance on a theme. Thus, the conclusion is one of nuance as opposed to distinct identity. To test whether this holds up, we must dive into the individual indicators and see if at the end they still lend us to this conclusion.
Not radically different questions for members to consider—they go through a similar decision-making process.
Community: It’s impact on; Increased visibility/knowledge of issue/public willBetter/less fragmented relationships with policy-makers and allies and reduced enemiesNew knowledge/development of good research (increased data) Overcoming important “sticking” points in moving an agenda/policyIncreased coordination of activities and better implementationDevelopment of IOR: Its position (power, centrality, capacity)Increased IOR capacity (e.g., clarity of vision; ability to manage/raise resources; visibility, etc.)Increased quality/prestige/ engagement of membershipIncreased collaboration between IOR members outside the IORMore rapid and organized ability to respondMore important to measure cohesion with network because there is a presupposed cohesion through formalization with coalitionsAbility to promote pair-wise relationshipsDensity as a measure (depth, strength and durability)Organization: Either direct or through individuals enhancing the organization based on experience in the networkNew relationship partnersIndividual: Personal development of skills, sense of belonging/identity, motivation, etc.Increased skills/ capacityIncreased legitimacy/ powerIncreased access to resources (while maintaining autonomy)Increased informationSense of belongingDecreased transaction costsUnit of analysis more likely to center on the individual/organization than the network as a whole.
InterAction - Rockefeller - TCC - GlobalGiving
Advances in Development Evaluation InterAction Forum 2012
Session Overview• Interactive - audience participation• Global Giving’s storytelling work• Evaluation of networks• Collaborative learning activity• Wrap up / Q&A
Introduction• The Rockefeller Foundation Evaluation Office is supporting innovative evaluation approaches, new methods and tools and evaluation capacity development• This session presents the work of two of RF’s grantees who are key contributors to advances in development evaluation
Tell us about something that surprised you at InterAction yesterday. SKILLS OTHER THE IDEAS EXPERIENCE What is your story about? Place it somewhere inside the triangle, or in the “other” box.
Closing the Broken Feedback Loop Feedback makes it safe (and satisfying) to give to riskier (but maybe better performing) orgs. Feedback identifies great NGOs and helps them learn. Resulting data is useful for other funders, NGOs, government agencies, and researchers.
Methodological Logic Story Capture Many Emergence Self-Signification Act on Insights Narratives Contextualized Statistics Visualization Discussion (Theme, Geo, Org, Tim e)
What’s working? • Real-Time Decision-Making Aid for GlobalGiving Working! • Story Patterns Illuminate Needs • Actionable Constituent Voice • Performance Dashboard for NGOs Promising! • Methodology Recognized as Useful • Interoperability with other Monitoring Tools • Globally Scalable Collection Infrastructure Working on it! • Market-Driven Incentives for Story Collection • Open-Source Analysis Tools
EvaluatingCoalitions and Networks Jared Raynor Director of Evaluation InterAction Conference May 2012 strategies to achieve social impact
Logic Model: Advocacy Initiatives Acting Project Inputs/ Strategies: Outcomes: Impact:Organization: Resources: Advocacy Staff: Issue analysis/ •Time, experience and research Improvement in the Quality of Living for the Community expertise Better defined and Change in Social Structure Reflecting Positive Social Justice Shift (administrative, legislativ Nonprofit or Advocacy framed problems coalition as a strategy) Media advocacy/ Organization (uses e, election-related and Public awareness legal) raising •Core skills (analytical, communicatio Issues more clearly set n, research, etc.) Grass roots on the public agenda organizing and gain prominence Funding: (momentum, interest, a •Monetary resources Coalition building/ wareness, etc.) •Non-monetary/ non-staff networking resources (e.g. volunteers, in-kind Policy makers adopt donations) Policy analysis/ new policies based on Coalition/Network research the agenda / do not (as an Actor as compared to a Organizational adopt harmful policies strategy) Reputation: Legal action •Advocacy related •General perception as a Lobbying and direct Policy decisions quality organization policy-maker effectively •Representative for base influence implemented constituency Assure good policy Network: implementation •Policy makers (Administrative/ Impact of new policies •Partner organizations Regulatory is evaluated •Media oversight, TA, monit •Mobilized base oring, etc.) (C) TCC Group 2010
Continuum of Inter-Organizational Relationships (IORs) Mutual Merger Accountability Joint Venture Strategic Alliance Collaboration Formal InformalOrganization Organization Coalition Formal Network Committee Separate AccountabilityInformal Network Adapted from: Wolf, T. (2003). A Practical Approach to Evaluation of Collaborations. In T.E. Backer (Ed.). Evaluating Community Collaborations. New York, NY. Springer Publishing Co. with additional acknowledgement of K. Hobson and M. Hightower King at AEA 2009.
Relationships for What Purpose?• Respond to complexity• Cope with turbulence/complexity• Acquire resources (while maintaining autonomy)• Decrease transaction costs• Gain legitimacy/power Positive Negative Externalities Externalities (Value) (Congestion)
What Are We Measuring? Coalition Organization Individual
An Entanglement of Relations Coalition Organization Organization Number ofIndividual Relationships Organization 2 Person Coalition: 8 Individual Organization 3 Person Coalition: 18 4 Person Coalition: 32 Individual Individual
A Simplified Approach for Evaluation Network/Member Coalition OutcomesCapacity Capacity Image credit: http://www.cugelman.com/research/united-nations-web-network.htm
Member Capacity • What do you want to get out of the network/coalition? • How do you justify your involvement? • How much time can you devote? • What can you offer? Individual Capacity Network CoalitionBoundary spanners Skill/knowledge to work collaborativelyTime/space to engage in network Time and Commitment to working in coalitionRelevant skills and interest Relevant decision-making authorityDesire to be part of something Strategic use of coalitions to fill criticalbigger gaps and leverage resources toward achieving your mission
The Critical Organizational Capacities for Advocacy Initiatives: The Logic Model “Inputs” Initiative Logic ModelLeadership: Management: Technical: Issue analysis/ research Improvement in the Quality of Living for the Community Media advocacy/ Public awareness raising Grass roots organizing Coalition building/ networking Policy analysis/ researchAdaptability: Legal action Lobbying and direct policy- maker influence Access Administrative/ Regulatory window of implementation opportunity/ influence threat: •Offense •Defense Org. Culture (C) TCC Group 2008
A Few Key IOR Capacities Value Goal Proposition Destination(Why is this IOR the (What is the IOR right approach?) trying to achieve?)• Rules, procedures and decision-making• Action/Exchange-oriented• Leadership Entropy Bureaucracy (disorder) (rigidity)
Multiple Levels of Outcome Community Community Impact/ OutcomesDevelopment of IOR Coalition/ Network Organization Benefits to Organizations Individual Benefits to Individuals
Capturing ‘Externalities’ • Sustained networks/relationships (Relationships hurt) • Reduced sense of isolation (Personality conflicts)• Better implementation/coordination of actual programs addressing the issue (More entrenched programming)• Cross-fertilization and Innovative ideas (Group think)• Other specific project goals met
Outcome questions for IORs in Emergency Settings• Has the IOR increased the ability to coordinate during varying stages of the emergency, including preparation?• Did the IOR reduce political complexity?• Did the IOR reduce or increase time and resource constraints?• Has the IOR increased the reliability of information available?• Has the IOR facilitated overcoming infrastructure challenges?• Has the IOR enhanced the well-being of personnel?• Has the IOR facilitated better relationships with affected populations and donors?• Has the IOR increased ability to engage with the media?
Contact Information Jared Raynor Director of Evaluation TCC Group www.tccgrp.com
Leveraging the Evaluator• Navigate power differentials• Use to: • Build (inform development) • Sustain (strengthen existing performance; identify inefficiencies) • Reflect (accountability for resources and prep for future)• Understand how to position yourself in a network• Help articulate goal destination and value proposition
Mobile Phone Principles of Good IOR Behavior1 Getting the Right Plan• What do you need to accomplish?• Do you have appropriate bandwidth (value vs. congestion)?• Right amount of minutes/airtime?
Mobile Phone Principles of Good IOR Behavior 2 Aware of Surroundings • How should others contact you? You contact them? • Are you talking too loudly? Too quietly? • How much background noise is there?Photo source: whatisacellphonejammer.com
Mobile Phone Principles of GoodIOR Behavior 3 • Strategic Use Do you have the right connections? • Are there new apps? • Are you playing too much Angry Birds? • Rate/assess your experience
Mobile Phone Principles of Good IOR Behavior1 Getting the Right 2 Aware of Surroundings • How should others contact Plan you? You contact them?• What do you need to • Are you talking too loudly? accomplish? Too quietly?• Do you have • How much background appropriate bandwidth (value vs. 3 • Strategic Use Do you have the right noise is there? congestion)? connections?• Right amount of • Are there new apps? minutes/airtime? • Are you playing too much Angry Birds? • Rate/assess your experience