Ppt analysis of vet results frameworks - franz kehl & markus engler
Brief Analysis ofVSD - Results Frameworkse + i Network MeetingThun, May 15, 2013prepared & presented byFranz Kehl & Markus EnglerKEK-CDC Consultantson behalf of SDC FP e+i & SDC QA
Cooperation Strategies coveredScope of Analysis:13 VSD Result Frameworksout of about 44 Cooperation Strategies• Bangladesh• Bolivia• Bénin• Bosnia Herzegovina• Burkina Faso• Grands Lacs• Kosovo• Mali• Mongolia• Myanmar• Niger• OPT• Uzbekistan
Result Framework: Criteria / Questions for AnalysisColumn 1 Column 2 Column 3• Quality of outcomestatement,(methodologic andthematic)• Nos. of outcomes• Comprehensibility• Informative value ofthe ‘linking’ column• Levels addressed, i.e.linkages, intermediateresults, risks andassumptions• Comprehensibility ofstatements• Traceability• Thematic clarityReference for analysis: - SDC-Guidance for Result Frameworks ( Annex 2 CS-Guidelines)- SDC PCM interactive (http://elearningpcm.ch/)
General Findings & Observations4 Quality of Results Frameworks Roughly 40 % comply well with quality criteria About 40% are fairly well developed About 20% can be improved in important aspectsGiven that the RF has been introduced only recentlyand is a relatively demanding tool, assement shows a Positive picture on the whole, namely SDC-outcomecolumn filled appropriately. Improvements mainly required in column 2 and 3
Column 1: SDC-Outcomes5 Disadvantaged women andmen, and especially youth,have access to relevantvocational skills developmentoffers, to decent employmentand self-employment and earnhigher incomeGood examples: What makes it good practice? Outcome statement describes a (changeof a) situation that should be achieved As short as possible and as long asrequired to provide sufficient detail Differentiated by gender and povertysituation Limited number of indicators (2 – 3) butrepresenting key aspects of the outcome People trained, including thedisadvantaged, have relevantqualifications.Indicators- # of qualifications developed,upgraded, validated & adopted- # of disadvantaged (male-female,poor-non poor, minorities) trainees- % of trainees availing new / betterjobs.Welldone
Column 1: SDC-Outcomes6Examples for improvement: What is wrong with it ? SDC-outcome: ImprovedemployabilityNational outcome: Establishvocational training centers The human resource of thecountry… Réduction du tauxdanalphabétisme Hierarchy of objectives in column 1 and 3inverted. SDC-outcome is above nationaloutcome Enseignement formel primaire Employability strengthened /improved Rapid skills development andimproved employability of menand women with lack ofmarketable skills. Too general, does not specify anyparticularity of SDC supportHow toimprove? The scope of the SDC-outcome is toobroad to be realistic or too general to bemeaningful Describes approach and mixes approachand outcome Same indicators as for National outcomes No target values and baseline
Column 1: SDC-Outcomes: Conclusions 50 % of the Result Frameworks fulfill methodologicalrequirements in column 1. 50 % of RF have some good elements but are not satisfactorythroughout, i.e.- comprehensibility of statements generally okay, but- some incoherence within the result framework. Main improvements required: Outcome statements: Clarification on how to formulate thestatements properly. Includes a clear understanding of what anoutcome is. Provide target values and baselines for indicators
8 Improved quality of technical,vocational and higher education,enhanced enrolment planningand coordination(NDS 4.4. Strategic Goal 4)Country level indicators:9) Number of VET graduates,(NSO) (f/m)10) Ratio of VET vs. highereducation students,(NSO) (f/m)Good examples: What makes it good practice Concrete outcome statements quoted fromNational Strategies Reference to quoted documents given Country level indicators mentioned.Preferably related baselines should beaddedColumn 3: Country Development and Humanitarian Outcomes SDC Outcome: (Column 1)Skills development deliverysystem expanded ... Country outcome indicator: (Column 3)Increased employment targetsfrom 1.8 Mio to 1.9 Mio. jobs Country level outcomes and indicators areselected such that a logical link to the SDCoutcomes is apparent.Welldone
9Examples for improvement: What is wrong with it ?Related to a skillsdevelopment plan, thefollowing indicator is given: Increased overall economicgrowth from x % to y% Outcomes or indicators taken as referenceare too ambitious and/or too general Conducive environment forgrowth and povertyreduction General statement not supported by anyreference to national policies or strategiesHow toimprove?Column 3: Country Development and Humanitarian Outcomes
Availability of national policies and strategies, providing usefulcountry development objectives is still a frequent problem. In their place statements from other sources are used assubstitutes Main improvements required: If substitute statements are used, source needs to be mentionedas well Even if policy papers provide poor outcome statements,thematically they should correspond with SDC outcomesColumn 3: Country Development and Humanitarian OutcomesConclusions
11 Impact hypothesisIf rural young women and menhave access to high-qualityand demand-driven vocationaleducation and training in theirregion, they will be able toobtain suitable employment. Intermediary results:1. TVET institutions haveestablished a system toensure that their training isadapted to changing employerand market needs.2. Number of practical trainingmodules provided by privatecompanies.Good examples: What makes it a good practice? Make use of the proposed levels sincethey provide useful information tounderstand program /project, in particular: Impact hypothesis is supportive tounderstand intervention logic. Intermediate results formulated such that itis obvious how they relate to the outcome.Column 2: Link SDC Contribution and Country OutcomesWelldone
12 Risks:1. Declining competitivenessof … SMEs as a result of themining boom.Good examples: What makes it a good practice? Risks and assumptions: identify the mostcritical respectively relevant ones, becausethey are key to the performance. Explaining the links within the domain aswell as with other programs helps toidentify the much looked for synergiesColumn 2: Link SDC Contribution and Country OutcomesWelldone
Column 2: Link SDC Contribution and Country Outcomes13Examples for improvement: What is wrong with it ? SDC-outcome:People have relevantqualification.related intermediate results:Migration cost decreased,labour protection increased Statements are too short to providemeaningful information, respectively toshow the link between outcome andintermediate result Trained people might migrate Risks and assumptions that seemimportant but actually have no directbearing on the performance of the programHow toimprove?
Column 2: Link SDC Contribution and Country Outcomes In about 50 % of the Result Frameworks the suggested levels/aspects are not described: less clarity on how the SDC outcomes contribute to countryoutcomes. Risks are addressed in most RFs, but not necessarily the mostpertinent risks are mentioned Column with most heterogenous quality (if compared across allRFs) Main improvements required: Point out the importance of this column Emphasize the adequate reflection of pertinent risks andassumptions, as these are critical for the performanceConclusions
Thematic Focus of RFsObservations: The thematic range is fairly broad and diverse: From ‘up-grading matching services’ to ‘quality of training in basic education’ Outcomes at different levels: ‘income generation’ versus ‘training of schooldrop-outs’ Thematic ‘clusters’ identified in the 13 RFs, respectively 34 outcomes: Access to VT / employment, inclusion: 6 outcomes Employability (of young people): 5 Framework conditions 5 Income: 4 Linking market & training; Market-oriented training: 4 Skills development, qualification: 2 Specific mentioning of young people 8 Coherence is observed among programs working in the same region(e.g. Western Balkan, Western Africa) respectively(more likely) sameSDC-division.
Common Outcome IndicatorsAnalysis: Comparison of Indicators used in Result Frameworks with CommonOutcome Indicators: Attribution of RF-indicators to one of the COI fields of observation,respectively where possible to a particular COI if phrasing is similarObservations: About 50 % of RF-indicators are close to the phrasing of specificCommon Outcome Indicators. The other 50 % RF-indicators can only generally be attributed to one ofthe four fields of observation Distribution of RF-indicators among the 4 fields of observation: Income: 9 System: 19 Relevance: 13 Outreach: 24
Summing up / Questions arisingResults FrameworkAlthough VSD-responsibles are doing already quite well on theResult Frameworks, some questions arise for consideration in theplenary discussion:How to go forward regarding the recommended improvements?a) Disseminating the results of this analysis along with a brief hand-out providing guidance for weak points (e.g. formulation ofoutcome statement) ?b) Referring VSD-responsibles to existing (SDC-)guidelines bymeans of internet links ?c) Rely on services of QA for guidance in methdolodical questions?
Summing up / Questions arisingThematic Focus How do we assess the (broad) range of themes? Is there a rationale to go for more focus in terms of themes?E.g. mutual exchange and learning among programs, buildingspecific competence in SDC?Common Outcome Indicators A more detailed analysis of the indicators used in the ResultFrameworks with reference to the COI is recommended. The pertinent question is:How far do the COI reflect the practice or vice versa how far thepractice should be adjusted to COIs?