workshop cilcv3
•
1 like•326 views
This document describes the Kaleidoscan Project design and impact measurement methodology developed by PSC2, an international development consultancy. The Kaleidoscan is an analytical tool used to assess the effectiveness and performance of organizations. It examines an organization's internal practices, external context, and political/leadership influences. The Kaleidoscan aims to provide a richer understanding of development progress and results to improve project implementation, management and reporting. It uses a systematic, evidence-based and participatory methodology involving multiple stakeholder perspectives. The Kaleidoscan comprises assessment disks, indicators, a benchmark framework, and algorithms to conduct thorough, objective and comparable organizational analyses.
Recommended
More Related Content
What's hot
What's hot (11)
Viewers also liked
Viewers also liked (20)
Similar to workshop cilcv3
Similar to workshop cilcv3 (20)
workshop cilcv3
- 2. PSC² introduc*on • PSC² is a new research consultancy founded by two experienced interna*onal development consultants, Marc van den Muijzenberg and Arjen van Ballegoyen. • We have developed the Kaleidoscan on the basis of our interna*onal development experience over the past 20 years in a great number of countries and different topics. • PSC² conducts both tradi*onal consultancy and advisory work as well as more research-oriented work, such as compara*ve research and system modeling. • PSC² clients include public as well as private sector clients.
- 3. Why did we develop the Kaleidoscan? As prac**oners we no*ced recurring structural issues undermining project performance: 1. Systemic constraints, such as legal framework, finance flows and decision making processes, were oSen outside the scope of the project. 2. Illusion of technocra7c vacuum: “parking” poli*cal issues on paper, even though in reality this is the key to effec*ve and sustainable results; 3. Mismatch of donor and client expecta7ons and assump7ons regarding real objec*ves, project benefits, client capacity, prac*ces, etc. Ø We needed a tool to make such issues visible to decision makers to have a more informed discussion resul*ng in beWer decisions.
- 4. These issues combined lead to warped results measurement frameworks 1. Results frameworks are oBen too ambi7ous: – Ignoring significant parts of the problem while seYng op*mis*c results and impact as if the problems didn’t exist. – Not providing sufficient *me for reforms to be internalized and sustainable. 2. Results frameworks are oBen show ques7onable logic: – The causal connec*on between interven*ons and envisioned results and impact is oSen tenuous and vague. – The envisioned results and impacts are oSen expressed in vague and unmeasurable language (“strengthening” and “contribute”) 3. Results frameworks are oBen too inflexible: – Difficulty in adjus*ng to obstacles or unexpected developments (common if poli*cs and systemic issues are neglected) and shiSing project resources away from stalled project interven*ons to other, beWer performing aspects of the project.
- 5. Why is this a problem • For the beneficiary: disappoin*ng and possibly unsustainable reforms undermining poli*cal posi*on; no or limited impact for ci*zens; poli*cal complica*ons due to technocra*c reform ac*vi*es. • For the donor: tax-payers’ money wasted, chronic underperformance, possibly reduc*on of annual development budget. • For the contractor: impossible targets, bad scores, delayed or non-payment of contracts.
- 6. How to address these issues? 1. Map, measure and address systemic constraints: – Understand the dynamics of systemic constraints – Qualify the impact of systemic constraints on the target organiza*on/ func*on etc. – Measure, track and target systemic constraints for mi*ga*on or removal 2. Explicitly and systema7cally determine poli7cal support for reforms as well as the project: – Determine and track reform support – Adjust ambi*on levels to reality 3. Benchmark development growth: – Add depth to indicators through development milestones – Quan*fying movement across the benchmark to allow beWer progress measurement – (Op*onal) Benchmark results and impact
- 7. How does this help results and impact measurement and repor*ng? The aim of these solu*ons is to provide a richer, deeper and measurable development framework against which to determine reform progress, results and impact. This will allow: • More realis7c and detailed explana7ons and narra7ves for achieved results and impact, including the dynamics behind developments; • BeLer theories of change that have sound logic in what manner, and to what extent project interven*ons lead to results and impact; • The tracking of quan7fied and verifiable effects of reform and development interven*ons over *me; • More accurately capture interven7on results by shiSing away from binary indicators to composite indicators; • Greater adap7veness in project implementa*on and formula*on through enhanced learning and understanding.
- 9. What is the Kaleidoscan • The Kaleidoscan is an analy*cal tool that we use to determine if an organiza*on is performing effec7vely, in a professional manner. • The Kaleidoscan methodology allows us to determine why an organiza*on is or is not performing well. We do this by: – examining the organiza*ons internal prac7ces; – examining the organiza*on’s external context and how well the organiza*on has adapted itself; – examining the poli7cal and leadership pressures affec*ng the organiza*on’s performance and development. • The Kaleidoscan also uncovers different stakeholder perspec*ves on these issues in order to find the “approximate truth” through triangula*on. 9
- 10. The Kaleidoscan approach • The Kaleidoscan assesses and measures things that are difficult, yet important, to measure. • The Kaleidoscan is therefore primarily qualita7ve in nature: the scan relies on the assessment and interpreta*on by experienced and trained consultants. • However, in order to further buWress this qualita*ve assessment the following features have been added: 1. The Kaleidoscan methodology requires a systema7c review of the organiza*on and sector using all the disks and their elements. This ensures that all aspects of the situa*on are thoroughly considered during the assessment; 2. The Kaleidoscan methodology is evidence-based, requiring that the consultant supports the assessment with indicators, suppor*ng documents, etc.; 3. The Kaleidoscan methodology quan7fies assessments to increase objec*vity, consistency and comparability of results; 4. The Kaleidoscan methodology requires key stakeholder par*cipa*on to validate the assessments directly or through triangula*on. 10
- 11. Different methods to conduct a Kaleidoscan • A Kaleidoscan can be conducted in a number of ways: – Fully par7cipatory: with ac*ve par*cipa*on by representa*ve groups of key stakeholders – Par7ally par7cipatory: a consultant leads a small team of key stakeholder representa*ves – Consulta7ve: a consultant does the Kaleidoscan based on interviews and feedback from stakeholders – Coaching: the consultant uses the Kaleidoscan during coaching sessions with the counterpart – Desk research: the consultant uses reports, studies and other documents to complete the Kaleidoscan – Combina7on of the above: combina*ons can be made or different approaches can be used during different stages. • Different methods have different *me and cost implica*ons. They furthermore differ in the degree of validity of the Kaleidoscan results. 11
- 12. Donor use of the Kaleidoscan during procurement and M&E 1. The Kaleidoscan is used for sector analysis and ini7al benchmarking. 2. Kaleidoscan results inform program formula7on and subsequent ToRs through e.g. theory of change. 3. Program and project logical frameworks are based on Kaleidoscan assessments but do not refer or need Kaleidoscan methodology. 4. Project implementa7on repor7ng does not require the Kaleidoscan but repor7ng templates ask for categories of informa7on used by the Kaleidoscan (e.g. periodic detailed poli7cal support updates, system and sector developments). 5. Project evaulau7on is done against the logframe as usual. 6. Program evalua7on uses Kaleidoscan methodology to feed project results into the Kaleidoscan model. 7. The sector analysis is updated using the Kaleidoscan benchmarking. = Kaleidoscan = Kaleidoscan influence = No kaleidoscan Sector Analysis Donor Program formulation Project ToR Logical framework Bid selection & contract awarding Ongoing M&E efforts during implementation Program assessment Updated Sector Analysis Kaleidoscan Informing Informing Kaleidoscan
- 13. Contractor use of the Kaleidoscan during procurement and implementa*on 1. The Kaleidoscan is throughout all the en7re project cycle. 2. Contractors use the kaleidoscan to collect and analyze sector/chain and country knowledge. 3. Kaleidoscan analysis is used to understand and frame project ToRs and provided logical frameworks. 4. Kaleidoscan analysis is used during proposal wri7ng within the framework set by the logframe. 5. Project implementa7on, management and repor7ng is done on the basis of con7nued Kaleidoscan use; the Kaleidoscan results are used to make management decisions and to inform repor7ng against the official log frame. 6. Project post mortems are done to learn about project and interven7on effec7veness. 7. The sector/chain/country analysis is updated using the Kaleidoscan benchmarking for use in future bids. = Kaleidoscan = Kaleidoscan influence = No kaleidoscan Sector Analysis Donor ToR Analysis Bid submission Project implementa7on, management and repor7ng Project post mortem Updated Sector Analysis Kaleidoscan + logical framework Kaleidoscan Kaleidoscan
- 14. Kaleidoscan composi*on The Kaleidoscan comprises a number of components: 1. The Kaleidoscan assessment disks & indicators; 2. The Kaleidoscan benchmark framework; 3. The Kaleidoscan core algorithms; 4. The Kaleidoscan meta algorithms. 14
- 15. Kaleidoscan components: assessment disks The Kaleidoscan contains 4 different assessment categories: 1. System context: a review of the system context in which an organiza*on operates, including the ins*tu*onal structure, decision making authori*es, legal framework and finance flows; 2. Organiza7onal capacity: a review of the internal prac*ces of the organiza*on, including strategy, structure, HR, IT, processes, etc.; 3. Reform readiness: an analysis of the poli*cal support for reform ac*vi*es; 4. Interven7on parameters: an analysis of the alignment of beneficiary, stakeholder and donor views regarding a specific reform program. These four assessment categories are called disks and each disk is subdivided into disk elements that assess a specific aspect. 15 Assessment
- 17. Assessment indicators • For each of the disks and disk elements an assessment will be done to determine the current development or situa*onal stage. • To assist in this assessment and to ensure uniformity in the assessments, each disk and disk element has a number of indicators associated with it, to facilitate determina*on of the correct development or situa*onal stage. • These indicators are regularly reviewed and updated to capture the different itera*ons of the disks an across countries, sectors and cultures. This avoids the indicators from becoming too focused on the dominant prac*ces. 17 Assessment
- 18. Kaleidoscan components: Benchmark framework • The benchmark framework is the score grid against which the organiza*on and the system context is compared and rated. • It is a framework of benchmarks, with different benchmarks for each disk and for each disk element. • All benchmarks are subdivided into different development or situa*onal stages. This makes the benchmarks more dynamic and transparent as it allows the tracking of organiza*onal or context developments over *me across the benchmark stages. • Each benchmark with its stages covers the spectrum of professionalism and/or development. • The assessment indicators are linked to the benchmark stages. 18 Benchmark
- 20. Norma*ve benchmarks • The benchmarks used in the Kaleidoscan for disks 1 and 2 are norma&ve, meaning that the benchmark spectrum scales from less developed to more developed. • The norms used by the benchmark are linked to professionalism and func&onalism in order to increase the effec*veness of an organiza*on to deliver results given its context. • The norms do not include policy, ideology or strategic objec*ve preferences or the focus on the par*cular content of decisions. The norms also do not prescribe big or small government, poli*cal theories or organiza*onal blueprints. • Rather, the norms focus on the quality of how a decision/policy/strategy was made, how effec*vely an organiza*on uses its resources to achieve objec*ves and how linked-up it operates. Ø So, the norms focus on professional standards, but are content-blind. Ø I.e, the norms focus on how things are done, not what is done. 20 Benchmark
- 21. Rela*ve benchmarks • The benchmarks used for disks 3 and 4 are not norma*ve as these disks do not focus on professional development but rather on factual poli*cal posi*ons. • The assessments for these two disks are therefore compara*ve in nature, with rela*ve values. • These posi*ons are however quan*fied and used in the algorithms to determine reform readiness and Benchmark
- 25. Kaleidoscan core algorithms • The core algorithms are the algorithms that generate disk benchmark scores on the basis of the disk element assessments. • The core algorithms are stored in the Kaleidoscan database, which is accessible through the PSC² website. • The algorithms capture the rela*ons between and the rela*ve importance of the various disk elements. • The algorithms are regularly updated to improve accuracy and validity. 25 Algorithms (core)
- 26. Kaleidoscan meta-algorithms • The Kaleidoscan can also generate special purpose reports that highlight par*cular issues as requested by the client. • These special purpose reports are generated by means of meta- algorithms. • Meta-algorithms can draw on all Kaleidoscan informa*on in the Kaleidoscan database as well as other sources. • Some examples include: – Absorp*on capacity (ex ante); – Value for money (ex ante); – Risk assessments; – Sector analysis & progress evalua*on; – Program/project selec*on and evalua*on; – Compara*ve reports; – Special client requests. 26 Algorithms (meta)
- 28. Kaleidoscan database & repor*ng • The Kaleidoscan is supported by a database housing the algorithms that generate the disk, disk element and indicator scores. • The database can generate Kaleidoscan reports populated with the consultant’s input and algorithm results. • The database is accessible to consultants by means of the PSC² website.
- 31. Disk 3: Reform readiness 28% 0% 100%50% CoMSec SG 0.1+ PM CoM 0.1- Ministry of Planning PMAC