Basic Efficiency Resource: A framework for measuring the relative performance of multi-unit programs


Published on

As a solution, the BER model uses a two-dimensional matrix to aid the evaluation of complex multi-unit programs, with quadrants to identify over and underperforming units. The BER model was inspired by portfolio management approaches from the Boston Consulting Group and the General Electric Grid, as well as quadrant analysis by Andreasen (1995). However, its core principles are based on the concept of social return on investment, where output is always compared to input. It provides a relative perspective on performance that allows evaluators to account for impact based on the resources invested in an initiative.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Input:
    Output: may be considered behaviour change in social marketing campaigns, public awareness in marketing campaigns, policy change in advocacy campaigns, reduced inequality in a government equality program, improved environmental health in a community environmental program, or any other
  • Basic Efficiency Resource: A framework for measuring the relative performance of multi-unit programs

    1. 1. BASIC EFFICIENCY RESOURCE: A framework for measuring the relative performance of multi-unit programs Brian Cugelman, PhD Managing Director, AlterSpark 4 October 2010 Canadian Evaluation Society's Annual Conference Optimizing the Practice of Evaluation Toronto, Canada
    2. 2. WHERE BER STARTED Evaluation of Oxfam GB’s Global Climate Change Campaign Solution to evaluation challenges Lot’s of interest Leitmotiv and AlterSpark joint publication on BER 2
    3. 3. EVALUATION CHALLENGES Simplifying complex multi-unit programs Many evaluations are about ROI, but ROI is difficult to assess in social contexts Nothing is good or bad, except in comparison to something else 3
    4. 4. THE BER SOLUTION Provide a simple framework for evaluating complex multi-component programs, campaigns, or activities Build on the basic concepts of SROI to evaluate unit's impact compared to their resources Offers a relative perspective on performance where units of analysis are judged in comparison to their peer units, operating under similar conditions 4
    5. 5. THEORETICAL FOUNDATIONS: MATRIX ANALYSIS FRAMEWORKS • Boston Consulting Group • General Electric Grid • Customer satisfaction quadrant analysis by Andreasen • Bloc modeling techniques used by social network analysts • Multi dimensional scaling 5
    6. 6. THEORETICAL FOUNDATIONS: SROI SROI: efficiency is output relative to input • Input constitutes a program’s resources which may be measured by their budget, number of staff, pool of talent, social capital, or any measure of capacity, concrete or abstract • Output measures a program’s impact, and will vary according to a program's purpose 6
    7. 7. THEORETICAL FOUNDATIONS: SROI 7 Input High Below Average Efficiency Average Efficiency Low Average Efficiency Above Average Efficiency Low High Output
    8. 8. CASE STUDY: OXFAM GB’S GLOBAL CLIMATE CHANGE CAMPAIGN 8 Investment 4.5 4.0 3.5 3.0 Impact Visual stunts & media relations Effective com at policy events Using celebrities Global Oxfam affiliates Research papers Rapid news dissemination Public campaigns/mobilization Policy analysis Partnerships with others Oxfam internal program links Online campaigning Media engagement Lobbying and advocacy Staff in UNFCCC delegations Climate hearings Adopt a negotiator Perceivedimpact Perceived resourcing High HighLow Low
    9. 9. CONDUCTING A BER ANALYSIS The example in following section is fictional and for illustrative purposes. 9
    10. 10. 1. SELECTING UNITS OF ANALYSIS 10 1. Units of Analysis 3. Perceived Input (Budgets) 2. Perceived Output (impact) Online engagement •Most •Average •Least •I don't know •Most •Average •Least •I don't know Research papers •Most •Average •Least •I don't know •Most •Average •Least •I don't know Lobbying and advocacy •Most •Average •Least •I don't know •Most •Average •Least •I don't know
    11. 11. 2. MEASUREMENT TOOLS 11 Input Output • Program budgets (perceived and real) • Number of staff • Number and level of staff • How often a lobbying keyword appeared in policy • Number of widgets produced • Number of people engaged • Perceptions of impact Completely Disagree 1 2 3 4 5 Completely Agree 6 I can't say Online engagement Research papers Lobbying and advocacy
    12. 12. 3. DATA TYPES • Quantitative input data may include budgets, number of staff, or combined multi-dimensional resource measure • Quantitative output data may include process evaluation measures such as the number of people engaged by a campaign or media hits • Qualitative measures can include perceived program investments and perceived output achieved 12
    13. 13. 4. VISUALIZATION APPROACHES 13 High Low Low High
    14. 14. 4. VISUALIZATION APPROACHES 14 Input High • Media relations • Public relations • Lobbying and advocacy • Coalition/partnership building Low • Intra organisational coordination • Online engagement •Research papers •Public mobilization Low High Output
    16. 16. 5. INTERPRETATION Use BER is as a starting point for deeper discussions into the performance of units, their challenges, opportunities, and operating environment Understand the units of analysis and the informants who shared their perceptions Not all units within a program operate under the same conditions Some units contribute indirect effects, by empowering other units 16
    17. 17. LIMITATIONS AND RISKS It is easy to draw conclusions from the simple visualizations that would never stand in the face of a deeper understanding of the reality behind charts Kotler et al. (2005) noted, reliance on matrix approaches prompted a number of companies to sell off strategic assets and plunge into businesses that they lacked the experience to manage 17
    18. 18. FUTURE WORK AND BER DEVELOPMENT Download a copy of the BER whitepaper: • Send comments or examples of BER analyses: • Dr. Brian Cugelman, AlterSpark • Eva Otero, Leitmotiv Share your feedback on Facebook: • 18
    19. 19. THANK YOU Brian Cugelman, PhD Managing Director, AlterSpark Phone: +1 (416) 921-2055 @AlterSpark alterspark alterspark alterspark 19