The monitoring, evaluation and learning design of the MasterCard Foundation Scholars Program. Theory of Change, logic model and learning questions. Design challenges and solutions.

1. Learning Across Borders:
A Monitoring, Evaluation, and Learning
Framework for The MasterCard
Foundation Scholars Program
March 2014

2. Overview of Presentation
 The MasterCard Foundation
 The Scholars Program
 Monitoring, evaluation, and learning (MEL) framework
 MEL challenges, approaches, and lessons learned
2

3. The MasterCard Foundation Scholars Program
3

4. The MasterCard Foundation Scholars Program:
Context and Programmatic Need
Developing countries have achieved a remarkable expansion
in educational enrolment, however:
 Enrolment in secondary and tertiary education remains low, particularly
among poor and rural populations
 There are concerns about access to quality and relevant education,
even as demand for an educated workforce increases
 Many skilled graduates migrate to higher-income countries for better
remuneration and working conditions
 International scholarship programs too often contribute to brain drain
4

5. The MasterCard Foundation Scholars Program
 $500 million, 10-year initiative to educate and support 15,000 young
people – particularly from Africa – to act as “socially transformative
leaders,” driving change and making a positive social impact in their
communities.
 11,000 secondary level Scholarships, 6,000 tertiary
 Target population: youth who have demonstrated academic talent,
social consciousness and an orientation toward leadership, and who
come from highly disadvantaged socioeconomic backgrounds.
 Implemented with and through a global network of 20 NGOs and
secondary and tertiary education institutions, who select and directly
support the Scholars.
5

6. The MasterCard Foundation Scholars Program:
Program Components
6
Comprehensive scholarships
Transition support
21st-century skill building
Give-back component
Alumni network

7. The MasterCard Foundation Scholars Program:
Partners
7

8. The MasterCard Foundation Scholars Program
MEL Framework
8

9. Evaluation and Learning in the Scholars Program
Objectives:
 Continuous program quality improvement
 Evidence of outcomes, impact and best practices…
 For our partners
 For the Foundation’s strategic planning
 For the global community of practitioners
 As the basis of a proactive argument for propagating successful
Program elements and the overall vision of the Program.
9

10. Evaluation and Learning Overview
 Program-wide evaluation: primary end-point 2016
 Implementation study
 Impact study
 Institutional change study
 Longitudinal studies of outcomes and impact
 Outcome study (subset of Scholar cohorts and comparison groups): continued past 5-year point
 Outcome tracking (all cohorts)
 Anticipated outcomes (Stay in / return to Africa, Give-back, career and leadership outcomes,
citizenship outcomes)
 Long-term qualitative work to understand Scholars’ roles in promoting social and economic
development
 Partner-initiated Research
 Rapid studies on emergent issues, aimed at providing actionable intelligence.
10

11. The MEL Framework: The Process
11

12. The MEL Framework: Theory of Change
12

13. Approach
and Inputs
Activities Outputs ImpactsOutcomes
Inputs
Partner
expertise
To Serve Scholars
Academic and
other support
Approach
Provide
Scholars with a
wide range of
opportunities
for educational
support and
enrichment,
give-back, and
leadership
development
At Institutions
Revise recruitment
procedures
Scholars using
academic and
other support
services
Partners
disseminate
lessons learned
Increased
academic
performance
Institutionalized
changes foster
best practices
New cohort of
leaders
supports
social
transformation
and economic
growth,
particularly in
Africa
13
The MEL Framework: Logic Model Components

14. The MEL Framework:
Learning Questions, Methods, and Data Sources
Broad Learning Questions Main Design/Methods Data Sources
Implementation
How was the Scholars
Program implemented?
Mixed methods
Qualitative data,
complemented by
quantitative data
Impacts
In particular, what is the
impact of the Program on
educational, employment,
go-back, and give-
back outcomes?
Comparison group design
• Random assignment
• Regression discontinuity
• Propensity score matching
Survey data, complemented
by qualitative data
Scale-up
Did the Scholars Program
become a catalyst for
broader change within
and beyond partner
institutions?
Qualitative analysis
Qualitative data,
complemented by
quantitative data
14

15. Challenges, Solutions, and Lessons Learned
15

16. Challenges, Solutions, and Lessons Learned:
Designing a Unified MEL Framework
Challenge
 Develop a single unified MEL framework to evaluate The MasterCard
Foundation Scholars Program
Solutions
 Develop the MEL framework from the ground up through a thorough
understanding of partner projects and what they have in common
 Use a flexible approach that allows for different methods
for each partner: Partner-specific comparison group designs allow for a
rigorous evaluation while using different methods to draw the
comparison group
 Random assignment
 Regression discontinuity
 Matched comparison group (via propensity score methods)
16

17. Challenges, Solutions, and Lessons Learned:
Measurement
Challenge
 Locating applicants based on application data may be challenging
 Several key outcomes are difficult to measure
 Key outcomes, including educational attainment, career and service
outcomes, giving back and going back, occur several years in the future
Solutions
 To locate hard-to-find applicants, use their own and their schools’
contact information
 Draw survey items from existing instruments and pilot our
own instruments
 Establish a plan for tracking The MasterCard Foundation Scholars over
time that is appropriate to the context
17

18. Challenges, Solutions, and Lessons Learned:
Measurement—Baseline Data Collection in Uganda
 Sample: 823 applicants spread
around Uganda
 Applicants were located by calling
the phone number provided
in their application
 This is the first of three rounds of data
collection planned
18
92%
8%
Locating applicants
Located
Not located
82%
18%
“Give-back” attitudes:
Helping people in need is
something one should do…
for anyone
for friends and
family
2% 15%
83%
Contact information
No phone
number
1 phone
number
2 phone
numbers

19. Challenge
 Incorporate formative evaluation components that will generate
relevant feedback for partners throughout the evaluation
 Early feedback can then inform mid-course corrections as needed
Solutions
 Engage with partners while developing MEL activities
 Collect quantitative and qualitative data at regular intervals
 Provide partners with feedback from all data collection efforts
19
Challenges, Solutions, and Lessons Learned:
Blending Formative, Summative Evaluation Approaches

20. Phone interviews with 18 graduating tertiary Scholars
 Opportunity to gain early information on Scholar experiences
 This early outcome information will support continuous quality
improvement
Scholars offer suggestions for Program improvement
 Increase flexibility in resource allocation—for example, could leftover
resources for textbooks be used for transportation?
 Coordinate Scholar attendance at academic conferences
to facilitate networking
 Provide orientation to the Program in one intensive session early on,
rather than spread over several sessions
20
Challenges, Solutions, and Lessons Learned:
Regular, Relevant Feedback to Partners

21. Challenges, Solutions, and Lessons Learned:
Regular, Relevant Feedback to Partners
(continued)
Scholars also say what is going well
 The Program gives Scholars “the flexibility, time
and resources to actually do the work you set out
to do”
 “[I’m] trying my best to really use all the
knowledge they’ve given to me and trying to
implement it in myself first and then try to
introduce it to society”
 The Masters in Development program is a “dream
program” that fits well with career goal
of reducing poverty in her home country
21

22. Conclusions
 We have designed MEL activities
that respond to the complex nature
of The MasterCard Foundation Scholars
Program with an approach that is
• Flexible, yet rigorous
• Capable of measuring key outcomes now
and in the future
• Formative and summative in nature
• Collaborative and inclusive, responding
to needs across the partnership
22

23. Contacts
Barry Burciul
 bburciul@mastercardfdn.org
 http://mastercardfdnscholars.org/
Matt Sloan
 msloan@mathematica-mpr.com
 http://www.mathematica-
mpr.com/CIPRE/Projects/MasterCard_Scholars_program.asp
23