SlideShare a Scribd company logo

Evaluation methodology: How do we know what we know? (5th DHG, Helsinki)

Overview of the scoping study on research methodology used in scholarship evaluation, conducted by the Commonwealth Scholarship Commission in the UK. Presented at the 5th Annual Forum of the Donor Harmonisation Conference, 11-13 June, 2014, Helsinki.

1 of 9
Download to read offline
Evaluation methodology:
How do we know what we know?
Aims
…identify trends
…identify omissions / ambiguities
…catalyse dialogue
67Documents
Methodology
Methods
Variables
Data analysis
Thematic issues:
• Counterfactuals
• Value for money
• Harmonisation
Trends
…ex-post evaluation
…mixed-methods dominant
…similar topics analysed
Variables:
Socio-demographics
Return home rates
Employment trajectory
Skill use

Recommended

More Related Content

What's hot

Research output in Irish H.E. academic libraries 2000-2015
Research output in Irish H.E. academic libraries 2000-2015 Research output in Irish H.E. academic libraries 2000-2015
Research output in Irish H.E. academic libraries 2000-2015 Terry O'Brien
 
Graded unit exemplar presentation
Graded unit exemplar presentationGraded unit exemplar presentation
Graded unit exemplar presentationMark Hetherington
 
Designing impactful research in social sciences
Designing impactful research in social sciencesDesigning impactful research in social sciences
Designing impactful research in social sciencesvijay kumar
 
5.3 Musm7034 Conclusion
5.3 Musm7034 Conclusion5.3 Musm7034 Conclusion
5.3 Musm7034 ConclusionGillian King
 
Paper publication, Research/ MM Bagali, HRM, HRD, HR< MBA Research, Manage...
Paper publication, Research/ MM Bagali, HRM, HRD, HR< MBA Research, Manage...Paper publication, Research/ MM Bagali, HRM, HRD, HR< MBA Research, Manage...
Paper publication, Research/ MM Bagali, HRM, HRD, HR< MBA Research, Manage...dr m m bagali, phd in hr
 
Research in language and literature, karpagam university, conference ppt
Research in language and literature, karpagam university, conference pptResearch in language and literature, karpagam university, conference ppt
Research in language and literature, karpagam university, conference pptvijay kumar
 
"Research output in Irish H.E.academic libraries 2000-2015" Terry O’Brien & K...
"Research output in Irish H.E.academic libraries 2000-2015" Terry O’Brien & K..."Research output in Irish H.E.academic libraries 2000-2015" Terry O’Brien & K...
"Research output in Irish H.E.academic libraries 2000-2015" Terry O’Brien & K...CONUL Conference
 
Scholarship of Teaching and Learning - IRB
Scholarship of Teaching and Learning - IRBScholarship of Teaching and Learning - IRB
Scholarship of Teaching and Learning - IRBStaci Trekles
 
Developing a Workplace Health and Safety Action Plan with NVivo
Developing a Workplace Health and Safety Action Plan with NVivoDeveloping a Workplace Health and Safety Action Plan with NVivo
Developing a Workplace Health and Safety Action Plan with NVivoQSR International
 
Assessing quality and bias in studies
Assessing quality and bias in studiesAssessing quality and bias in studies
Assessing quality and bias in studiesILRI
 
RecSys 2015 posters
RecSys 2015 postersRecSys 2015 posters
RecSys 2015 postersJMoralesBSU
 
The Case for Competition: Learning About Evidence-Based Management Through Ca...
The Case for Competition: Learning About Evidence-Based Management Through Ca...The Case for Competition: Learning About Evidence-Based Management Through Ca...
The Case for Competition: Learning About Evidence-Based Management Through Ca...Center for Evidence-Based Management
 
Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?Denise Whitelock
 
Tips and advice june 2010
Tips and advice june 2010Tips and advice june 2010
Tips and advice june 2010deniseturner
 
Integrative Assignment Result
Integrative Assignment ResultIntegrative Assignment Result
Integrative Assignment ResultIvo Neshev, CIPS
 
Crafting a research agenda (in memes)
Crafting a research agenda (in memes) Crafting a research agenda (in memes)
Crafting a research agenda (in memes) George Veletsianos
 

What's hot (20)

Research output in Irish H.E. academic libraries 2000-2015
Research output in Irish H.E. academic libraries 2000-2015 Research output in Irish H.E. academic libraries 2000-2015
Research output in Irish H.E. academic libraries 2000-2015
 
Graded unit exemplar presentation
Graded unit exemplar presentationGraded unit exemplar presentation
Graded unit exemplar presentation
 
Designing impactful research in social sciences
Designing impactful research in social sciencesDesigning impactful research in social sciences
Designing impactful research in social sciences
 
Initial consultation event: Content workshop
Initial consultation event: Content workshopInitial consultation event: Content workshop
Initial consultation event: Content workshop
 
5.3 Musm7034 Conclusion
5.3 Musm7034 Conclusion5.3 Musm7034 Conclusion
5.3 Musm7034 Conclusion
 
Paper publication, Research/ MM Bagali, HRM, HRD, HR< MBA Research, Manage...
Paper publication, Research/ MM Bagali, HRM, HRD, HR< MBA Research, Manage...Paper publication, Research/ MM Bagali, HRM, HRD, HR< MBA Research, Manage...
Paper publication, Research/ MM Bagali, HRM, HRD, HR< MBA Research, Manage...
 
Research in language and literature, karpagam university, conference ppt
Research in language and literature, karpagam university, conference pptResearch in language and literature, karpagam university, conference ppt
Research in language and literature, karpagam university, conference ppt
 
"Research output in Irish H.E.academic libraries 2000-2015" Terry O’Brien & K...
"Research output in Irish H.E.academic libraries 2000-2015" Terry O’Brien & K..."Research output in Irish H.E.academic libraries 2000-2015" Terry O’Brien & K...
"Research output in Irish H.E.academic libraries 2000-2015" Terry O’Brien & K...
 
Scholarship of Teaching and Learning - IRB
Scholarship of Teaching and Learning - IRBScholarship of Teaching and Learning - IRB
Scholarship of Teaching and Learning - IRB
 
Developing a Workplace Health and Safety Action Plan with NVivo
Developing a Workplace Health and Safety Action Plan with NVivoDeveloping a Workplace Health and Safety Action Plan with NVivo
Developing a Workplace Health and Safety Action Plan with NVivo
 
Research presentation
Research presentationResearch presentation
Research presentation
 
Assessing quality and bias in studies
Assessing quality and bias in studiesAssessing quality and bias in studies
Assessing quality and bias in studies
 
RecSys 2015 posters
RecSys 2015 postersRecSys 2015 posters
RecSys 2015 posters
 
The Case for Competition: Learning About Evidence-Based Management Through Ca...
The Case for Competition: Learning About Evidence-Based Management Through Ca...The Case for Competition: Learning About Evidence-Based Management Through Ca...
The Case for Competition: Learning About Evidence-Based Management Through Ca...
 
Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?
 
Assessment Conundrum
Assessment ConundrumAssessment Conundrum
Assessment Conundrum
 
NISO Apr 29 Virtual Conference Keynote Address: The Value of Library-Provided...
NISO Apr 29 Virtual Conference Keynote Address: The Value of Library-Provided...NISO Apr 29 Virtual Conference Keynote Address: The Value of Library-Provided...
NISO Apr 29 Virtual Conference Keynote Address: The Value of Library-Provided...
 
Tips and advice june 2010
Tips and advice june 2010Tips and advice june 2010
Tips and advice june 2010
 
Integrative Assignment Result
Integrative Assignment ResultIntegrative Assignment Result
Integrative Assignment Result
 
Crafting a research agenda (in memes)
Crafting a research agenda (in memes) Crafting a research agenda (in memes)
Crafting a research agenda (in memes)
 

Recently uploaded

Writing Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdf
Writing Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdfWriting Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdf
Writing Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdfMr Bounab Samir
 
Overview of Databases and Data Modelling-1.pdf
Overview of Databases and Data Modelling-1.pdfOverview of Databases and Data Modelling-1.pdf
Overview of Databases and Data Modelling-1.pdfChristalin Nelson
 
UniSC Fraser Coast library self-guided tour
UniSC Fraser Coast library self-guided tourUniSC Fraser Coast library self-guided tour
UniSC Fraser Coast library self-guided tourUSC_Library
 
Exit Essay - Save the Filipino Language by Renz Perez.docx
Exit Essay - Save the Filipino Language by Renz Perez.docxExit Essay - Save the Filipino Language by Renz Perez.docx
Exit Essay - Save the Filipino Language by Renz Perez.docxMYDA ANGELICA SUAN
 
DISCOURSE: TEXT AS CONNECTED DISCOURSE
DISCOURSE:   TEXT AS CONNECTED DISCOURSEDISCOURSE:   TEXT AS CONNECTED DISCOURSE
DISCOURSE: TEXT AS CONNECTED DISCOURSEMYDA ANGELICA SUAN
 
CONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptx
CONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptxCONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptx
CONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptxAnupkumar Sharma
 
HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)
HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)
HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)Rabiya Husain
 
Bayesian Analysis Fundamentals with Examples
Bayesian Analysis Fundamentals with ExamplesBayesian Analysis Fundamentals with Examples
Bayesian Analysis Fundamentals with ExamplesTushar Tank
 
LIPIDS UNIT ONE BIOMOLECULE PRESENTATION
LIPIDS UNIT ONE BIOMOLECULE PRESENTATIONLIPIDS UNIT ONE BIOMOLECULE PRESENTATION
LIPIDS UNIT ONE BIOMOLECULE PRESENTATIONSayali Powar
 
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...EduSkills OECD
 
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...Nguyen Thanh Tu Collection
 
Intuition behind Monte Carlo Markov Chains
Intuition behind Monte Carlo Markov ChainsIntuition behind Monte Carlo Markov Chains
Intuition behind Monte Carlo Markov ChainsTushar Tank
 
Website Fixer-Upper Series to Boost your Online Presence
Website Fixer-Upper Series to Boost your Online PresenceWebsite Fixer-Upper Series to Boost your Online Presence
Website Fixer-Upper Series to Boost your Online PresenceSamantha Russell
 
Food Web SlideShare for Ecology Notes Quiz in Canvas
Food Web SlideShare for Ecology Notes Quiz in CanvasFood Web SlideShare for Ecology Notes Quiz in Canvas
Food Web SlideShare for Ecology Notes Quiz in CanvasAlexandraSwartzwelde
 
Evaluation and management of patients with Dyspepsia.pptx
Evaluation and management of patients with Dyspepsia.pptxEvaluation and management of patients with Dyspepsia.pptx
Evaluation and management of patients with Dyspepsia.pptxgarvitnanecha
 
BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaBTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaFaheema Hasan
 
The Ministry of Utmost Happiness by Arundhati Roy
The Ministry of Utmost Happiness by Arundhati RoyThe Ministry of Utmost Happiness by Arundhati Roy
The Ministry of Utmost Happiness by Arundhati RoyTrushali Dodiya
 
EDL 290F Week 1 - Meet Me at the Start Line.pdf
EDL 290F Week 1 - Meet Me at the Start Line.pdfEDL 290F Week 1 - Meet Me at the Start Line.pdf
EDL 290F Week 1 - Meet Me at the Start Line.pdfElizabeth Walsh
 

Recently uploaded (20)

Writing Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdf
Writing Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdfWriting Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdf
Writing Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdf
 
Overview of Databases and Data Modelling-1.pdf
Overview of Databases and Data Modelling-1.pdfOverview of Databases and Data Modelling-1.pdf
Overview of Databases and Data Modelling-1.pdf
 
UniSC Fraser Coast library self-guided tour
UniSC Fraser Coast library self-guided tourUniSC Fraser Coast library self-guided tour
UniSC Fraser Coast library self-guided tour
 
Exit Essay - Save the Filipino Language by Renz Perez.docx
Exit Essay - Save the Filipino Language by Renz Perez.docxExit Essay - Save the Filipino Language by Renz Perez.docx
Exit Essay - Save the Filipino Language by Renz Perez.docx
 
DISCOURSE: TEXT AS CONNECTED DISCOURSE
DISCOURSE:   TEXT AS CONNECTED DISCOURSEDISCOURSE:   TEXT AS CONNECTED DISCOURSE
DISCOURSE: TEXT AS CONNECTED DISCOURSE
 
CONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptx
CONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptxCONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptx
CONCEPTS OF ENVIRONMENT & ECOSYSTEM.pptx
 
HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)
HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)
HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)
 
Bayesian Analysis Fundamentals with Examples
Bayesian Analysis Fundamentals with ExamplesBayesian Analysis Fundamentals with Examples
Bayesian Analysis Fundamentals with Examples
 
LIPIDS UNIT ONE BIOMOLECULE PRESENTATION
LIPIDS UNIT ONE BIOMOLECULE PRESENTATIONLIPIDS UNIT ONE BIOMOLECULE PRESENTATION
LIPIDS UNIT ONE BIOMOLECULE PRESENTATION
 
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
 
first section physiology laboratory.pptx
first section physiology laboratory.pptxfirst section physiology laboratory.pptx
first section physiology laboratory.pptx
 
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
 
Intuition behind Monte Carlo Markov Chains
Intuition behind Monte Carlo Markov ChainsIntuition behind Monte Carlo Markov Chains
Intuition behind Monte Carlo Markov Chains
 
Website Fixer-Upper Series to Boost your Online Presence
Website Fixer-Upper Series to Boost your Online PresenceWebsite Fixer-Upper Series to Boost your Online Presence
Website Fixer-Upper Series to Boost your Online Presence
 
Food Web SlideShare for Ecology Notes Quiz in Canvas
Food Web SlideShare for Ecology Notes Quiz in CanvasFood Web SlideShare for Ecology Notes Quiz in Canvas
Food Web SlideShare for Ecology Notes Quiz in Canvas
 
Evaluation and management of patients with Dyspepsia.pptx
Evaluation and management of patients with Dyspepsia.pptxEvaluation and management of patients with Dyspepsia.pptx
Evaluation and management of patients with Dyspepsia.pptx
 
BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaBTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic Leukemia
 
Caldecott Medal Book Winners and Media Used
Caldecott Medal Book Winners and Media UsedCaldecott Medal Book Winners and Media Used
Caldecott Medal Book Winners and Media Used
 
The Ministry of Utmost Happiness by Arundhati Roy
The Ministry of Utmost Happiness by Arundhati RoyThe Ministry of Utmost Happiness by Arundhati Roy
The Ministry of Utmost Happiness by Arundhati Roy
 
EDL 290F Week 1 - Meet Me at the Start Line.pdf
EDL 290F Week 1 - Meet Me at the Start Line.pdfEDL 290F Week 1 - Meet Me at the Start Line.pdf
EDL 290F Week 1 - Meet Me at the Start Line.pdf
 

Evaluation methodology: How do we know what we know? (5th DHG, Helsinki)

Editor's Notes

  1. Update colleagues on a piece of analysis that the CSC has been conducting for the last 6 months examining the research methodologies used in evaluating international scholarship schemes for higher education. General notes: Not everyone will run the same kind of Scholarship schemes as us, so try to avoid implying that everyone in the audience is a scholarship provider that hosts full-award scholarships in the donor country.
  2. Purpose of the work: Identify current trends in evaluation design within the sector Note any ambiguities, omissions or contradictions, with the intention of informing our evaluation work and Offer a baseline review for wider dialogue amongst scholarship providers It is, essentially, an analysis of how we know what we know about scholarship outcomes
  3. Involved review mainly of evaluation reports: 67 documents, most published reports, some internal documents, few conference papers and journal articles Looking at: Methodology – by which we mean research design or overarching framework – Methods of data collection, Variables examined in the report, Data analysis strategies, Three issues that we identified early on as important topics in the sector: counterfactuals, value for money, and harmonisation between scholarship providers Share a brief overview of some of the key findings, report is now out so can have a read in more detail if desired
  4. Involved review mainly of evaluation reports: 67 documents, most published reports, some internal documents, few conference papers and journal articles Looking at: Methodology – by which we mean research design or overarching framework – Methods of data collection, Variables examined in the report, Data analysis strategies, Three issues that we identified early on as important topics in the sector: counterfactuals, value for money, and harmonisation between scholarship providers Share a brief overview of some of the key findings, report is now out so can have a read in more detail if desired
  5. Overwhelming majority of evaluation conducted is ex-post: conceived and conducted after recipients have finished their scholarships. Tracer study: identifies alumni, gets in contact (through a variety of means) and asks them to complete a survey instrument detailing their experiences after completing the scholarship Little investment into planned comparison designs Even when tracer studies have been conducted repeatedly (e.g. Fiji), not followed same people repeatedly: no panel studies Of course the degree to which ex-post studies are successful is largely reliant on the effectiveness alumni tracing within scholarship programmes and there has been quite a bit of commentary on this in the field as it can be very difficult! Research methodology is actually not discussed very widely: most commentary at methods level: how will data be collected. Mixed-methods is the dominant approach Surveys dominant instrument: makes sense given global reach Particularly popular are likert-style questions for reflecting on achievements and application of skills and knowledge. Trove of self-report data, but with usual health warnings Qualitative fieldwork common: more expensive and smaller in scale. Interviews with stakeholders, usually recipients of scholarships, but also employers, government officials and others. Evaluators tried to approach employers with surveys, typically poor response rates: interviews much better for that. One of the main drivers for mixed methods designs Evaluations have generally looked at very similar topics: policy objectives of most of our scholarship schemes overlap greatly Socio-demographics, return home rates, post-scholarship employment trajectory, application of knowledge and skills learned during the scholarship, and links maintained to hosts / donors CSC looks at involvement in international development, e.g. contribution to government policy in agriculture, but actually this is uncommon across the sector Some topics are much more complicated – return home: depending on how one defines ‘home’ and for how long one expects scholars to return. Reintegration understudied, particularly social and community reintegration as distinct from reintegration into the labour market.
  6. Overwhelming majority of evaluation conducted is ex-post: conceived and conducted after recipients have finished their scholarships. Tracer study: identifies alumni, gets in contact (through a variety of means) and asks them to complete a survey instrument detailing their experiences after completing the scholarship Little investment into planned comparison designs Even when tracer studies have been conducted repeatedly (e.g. Fiji), not followed same people repeatedly: no panel studies Of course the degree to which ex-post studies are successful is largely reliant on the effectiveness alumni tracing within scholarship programmes and there has been quite a bit of commentary on this in the field as it can be very difficult! Research methodology is actually not discussed very widely: most commentary at methods level: how will data be collected. Mixed-methods is the dominant approach Surveys dominant instrument: makes sense given global reach Particularly popular are likert-style questions for reflecting on achievements and application of skills and knowledge. Trove of self-report data, but with usual health warnings Qualitative fieldwork common: more expensive and smaller in scale. Interviews with stakeholders, usually recipients of scholarships, but also employers, government officials and others. Evaluators tried to approach employers with surveys, typically poor response rates: interviews much better for that. One of the main drivers for mixed methods designs Evaluations have generally looked at very similar topics: policy objectives of most of our scholarship schemes overlap greatly Socio-demographics, return home rates, post-scholarship employment trajectory, application of knowledge and skills learned during the scholarship, and links maintained to hosts / donors CSC looks at involvement in international development, e.g. contribution to government policy in agriculture, but actually this is uncommon across the sector Some topics are much more complicated – return home: depending on how one defines ‘home’ and for how long one expects scholars to return. Reintegration understudied, particularly social and community reintegration as distinct from reintegration into the labour market.
  7. A few challenges for us to consider as part of the discussion Data analysis: are we making the best use of the data we collect? Most evaluation includes numeric data: extensive descriptive analysis has been published: percentages of alumni in specific sectors etc. Inferential statistics less common, only handful of 67 papers used any inferential tests of correlation or differences between means. Generally level of statistical analysis we are conducting is quite basic. Whether it fulfils our purposes is a slightly different question, but there are opportunities for additional avenues of evaluation if we use multivariate analysis more widely More of a problem though, qualitative data analysis often very vague: not clear how evaluators in sector transform dialogues with interviewees into research findings. Lack of comparative data is a concern Widely discussed, but not widely done Ex-post studies tell persuasive stories, but without counterfactual or baseline data cannot give insight into: 1) extent of individual capacity growth, or 2) relative benefit of participation versus non participation Many scholarships ‘catching up’ with alumni, often because emphasis on evaluation more recent and so ex-post evaluation is being conducted. Obviously this prevents effective baseline data collection, but it is a legacy issue: we can revise (some have done already) to improve evaluation going forward Counterfactuals comparing between scholarship recipients and non-recipients seen some use in evaluation (e.g. GMS, USAID LAC), although not widespread, and there is some disagreement over how far comparisons are valid. There are two other types of counterfactual we could consider: comparison between performance of different scholarship schemes or projects with very similar aims (Erasmus Mundus got close to this), and comparison between scholarship schemes and other interventions with similar aims. Harmonisation is nascent, more investment in coordination and joint reporting could be beneficial I mean coordination between scholarship providers, both in scholarship administration generally and evaluation specifically Best description for harmonisation currently is ‘nascent, emerging’. Some evidence of coordination around award administration and joint evaluation strategies (e.g. in Pacific), and DHG of course, but joint provision and evaluation seems to be at a very early stage. Some major gains to be made in this area with more substantial cooperation. Can only address interference and synergy through collaborative analysis (at country level, for instance). All run scholarships aimed at similar objectives, often in same geographical space and / or labour market, but we do not have overarching analysis to show us the compound effect of our work The Fiji example is instructive: According to AusAID, thousands of scholarship places, under dozens of schemes, funded by numerous donors. There must be other such examples in Africa too.
  8. So a few thoughts on the state of the sector and some of the challenges we face to gather reliable and detailed evaluation data about the outcomes of our scholarships. Intention of the exercise has been to catalyse dialogue, so in addition to discussing what our evaluation has told us as part of the remaining discussion I’d be interested to hear thoughts on how we can improve the quality and coverage of our evaluation so it can tell us more. Full report available at CSC website (and maybe paper copies?): encourage to read it and get in touch with any feedback