Ppt analysis of vet results frameworks - franz kehl & markus engler


Published on

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Ppt analysis of vet results frameworks - franz kehl & markus engler

  1. 1. Brief Analysis ofVSD - Results Frameworkse + i Network MeetingThun, May 15, 2013prepared & presented byFranz Kehl & Markus EnglerKEK-CDC Consultantson behalf of SDC FP e+i & SDC QA
  2. 2. Cooperation Strategies coveredScope of Analysis:13 VSD Result Frameworksout of about 44 Cooperation Strategies•  Bangladesh•  Bolivia•  Bénin•  Bosnia Herzegovina•  Burkina Faso•  Grands Lacs•  Kosovo•  Mali•  Mongolia•  Myanmar•  Niger•  OPT•  Uzbekistan
  3. 3. Result Framework: Criteria / Questions for AnalysisColumn 1 Column 2 Column 3•  Quality of outcomestatement,(methodologic andthematic)•  Nos. of outcomes•  Comprehensibility•  Informative value ofthe ‘linking’ column•  Levels addressed, i.e.linkages, intermediateresults, risks andassumptions•  Comprehensibility ofstatements•  Traceability•  Thematic clarityReference for analysis: - SDC-Guidance for Result Frameworks ( Annex 2 CS-Guidelines)- SDC PCM interactive (http://elearningpcm.ch/)
  4. 4. General Findings & Observations4  Quality of Results Frameworks  Roughly 40 % comply well with quality criteria  About 40% are fairly well developed  About 20% can be improved in important aspectsGiven that the RF has been introduced only recentlyand is a relatively demanding tool, assement shows a  Positive picture on the whole, namely SDC-outcomecolumn filled appropriately.  Improvements mainly required in column 2 and 3
  5. 5. Column 1: SDC-Outcomes5  Disadvantaged women andmen, and especially youth,have access to relevantvocational skills developmentoffers, to decent employmentand self-employment and earnhigher incomeGood examples: What makes it good practice?  Outcome statement describes a (changeof a) situation that should be achieved  As short as possible and as long asrequired to provide sufficient detail  Differentiated by gender and povertysituation  Limited number of indicators (2 – 3) butrepresenting key aspects of the outcome  People trained, including thedisadvantaged, have relevantqualifications.Indicators- # of qualifications developed,upgraded, validated & adopted- # of disadvantaged (male-female,poor-non poor, minorities) trainees- % of trainees availing new / betterjobs.Welldone
  6. 6. Column 1: SDC-Outcomes6Examples for improvement: What is wrong with it ?  SDC-outcome: ImprovedemployabilityNational outcome: Establishvocational training centers  The human resource of thecountry…  Réduction du tauxdanalphabétisme  Hierarchy of objectives in column 1 and 3inverted. SDC-outcome is above nationaloutcome  Enseignement formel primaire  Employability strengthened /improved  Rapid skills development andimproved employability of menand women with lack ofmarketable skills.  Too general, does not specify anyparticularity of SDC supportHow toimprove?  The scope of the SDC-outcome is toobroad to be realistic or too general to bemeaningful  Describes approach and mixes approachand outcome  Same indicators as for National outcomes  No target values and baseline
  7. 7. Column 1: SDC-Outcomes: Conclusions  50 % of the Result Frameworks fulfill methodologicalrequirements in column 1.  50 % of RF have some good elements but are not satisfactorythroughout, i.e.- comprehensibility of statements generally okay, but- some incoherence within the result framework.  Main improvements required:  Outcome statements: Clarification on how to formulate thestatements properly. Includes a clear understanding of what anoutcome is.  Provide target values and baselines for indicators
  8. 8. 8  Improved quality of technical,vocational and higher education,enhanced enrolment planningand coordination(NDS 4.4. Strategic Goal 4)Country level indicators:9) Number of VET graduates,(NSO) (f/m)10) Ratio of VET vs. highereducation students,(NSO) (f/m)Good examples: What makes it good practice  Concrete outcome statements quoted fromNational Strategies  Reference to quoted documents given  Country level indicators mentioned.Preferably related baselines should beaddedColumn 3: Country Development and Humanitarian Outcomes  SDC Outcome: (Column 1)Skills development deliverysystem expanded ...  Country outcome indicator: (Column 3)Increased employment targetsfrom 1.8 Mio to 1.9 Mio. jobs  Country level outcomes and indicators areselected such that a logical link to the SDCoutcomes is apparent.Welldone
  9. 9. 9Examples for improvement: What is wrong with it ?Related to a skillsdevelopment plan, thefollowing indicator is given:  Increased overall economicgrowth from x % to y%  Outcomes or indicators taken as referenceare too ambitious and/or too general  Conducive environment forgrowth and povertyreduction  General statement not supported by anyreference to national policies or strategiesHow toimprove?Column 3: Country Development and Humanitarian Outcomes
  10. 10.   Availability of national policies and strategies, providing usefulcountry development objectives is still a frequent problem.  In their place statements from other sources are used assubstitutes  Main improvements required:  If substitute statements are used, source needs to be mentionedas well  Even if policy papers provide poor outcome statements,thematically they should correspond with SDC outcomesColumn 3: Country Development and Humanitarian OutcomesConclusions
  11. 11. 11  Impact hypothesisIf rural young women and menhave access to high-qualityand demand-driven vocationaleducation and training in theirregion, they will be able toobtain suitable employment.  Intermediary results:1. TVET institutions haveestablished a system toensure that their training isadapted to changing employerand market needs.2. Number of practical trainingmodules provided by privatecompanies.Good examples: What makes it a good practice?  Make use of the proposed levels sincethey provide useful information tounderstand program /project, in particular:  Impact hypothesis is supportive tounderstand intervention logic.  Intermediate results formulated such that itis obvious how they relate to the outcome.Column 2: Link SDC Contribution and Country OutcomesWelldone
  12. 12. 12  Risks:1. Declining competitivenessof … SMEs as a result of themining boom.Good examples: What makes it a good practice?  Risks and assumptions: identify the mostcritical respectively relevant ones, becausethey are key to the performance.  Explaining the links within the domain aswell as with other programs helps toidentify the much looked for synergiesColumn 2: Link SDC Contribution and Country OutcomesWelldone
  13. 13. Column 2: Link SDC Contribution and Country Outcomes13Examples for improvement: What is wrong with it ?  SDC-outcome:People have relevantqualification.related intermediate results:Migration cost decreased,labour protection increased  Statements are too short to providemeaningful information, respectively toshow the link between outcome andintermediate result  Trained people might migrate   Risks and assumptions that seemimportant but actually have no directbearing on the performance of the programHow toimprove?
  14. 14. Column 2: Link SDC Contribution and Country Outcomes  In about 50 % of the Result Frameworks the suggested levels/aspects are not described: less clarity on how the SDC outcomes contribute to countryoutcomes.  Risks are addressed in most RFs, but not necessarily the mostpertinent risks are mentioned  Column with most heterogenous quality (if compared across allRFs)  Main improvements required:  Point out the importance of this column  Emphasize the adequate reflection of pertinent risks andassumptions, as these are critical for the performanceConclusions
  15. 15. Thematic Focus of RFsObservations:  The thematic range is fairly broad and diverse:  From ‘up-grading matching services’ to ‘quality of training in basic education’  Outcomes at different levels: ‘income generation’ versus ‘training of schooldrop-outs’  Thematic ‘clusters’ identified in the 13 RFs, respectively 34 outcomes:  Access to VT / employment, inclusion: 6 outcomes  Employability (of young people): 5  Framework conditions 5  Income: 4  Linking market & training; Market-oriented training: 4  Skills development, qualification: 2  Specific mentioning of young people 8  Coherence is observed among programs working in the same region(e.g. Western Balkan, Western Africa) respectively(more likely) sameSDC-division.
  16. 16. Common Outcome IndicatorsAnalysis:  Comparison of Indicators used in Result Frameworks with CommonOutcome Indicators:  Attribution of RF-indicators to one of the COI fields of observation,respectively where possible to a particular COI if phrasing is similarObservations:  About 50 % of RF-indicators are close to the phrasing of specificCommon Outcome Indicators.  The other 50 % RF-indicators can only generally be attributed to one ofthe four fields of observation  Distribution of RF-indicators among the 4 fields of observation:  Income: 9  System: 19  Relevance: 13  Outreach: 24
  17. 17. Summing up / Questions arisingResults FrameworkAlthough VSD-responsibles are doing already quite well on theResult Frameworks, some questions arise for consideration in theplenary discussion:How to go forward regarding the recommended improvements?a)  Disseminating the results of this analysis along with a brief hand-out providing guidance for weak points (e.g. formulation ofoutcome statement) ?b)  Referring VSD-responsibles to existing (SDC-)guidelines bymeans of internet links ?c)  Rely on services of QA for guidance in methdolodical questions?
  18. 18. Summing up / Questions arisingThematic Focus  How do we assess the (broad) range of themes?  Is there a rationale to go for more focus in terms of themes?E.g. mutual exchange and learning among programs, buildingspecific competence in SDC?Common Outcome Indicators  A more detailed analysis of the indicators used in the ResultFrameworks with reference to the COI is recommended.  The pertinent question is:How far do the COI reflect the practice or vice versa how far thepractice should be adjusted to COIs?