Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Module 12:  Managing an Evaluation
Introduction <ul><li>Managing the Design Matrix </li></ul><ul><li>Contracting the Evaluation </li></ul><ul><li>Roles and R...
Evaluation Design Matrix <ul><li>Key to success in evaluation is planning </li></ul><ul><li>Evaluation Design Matrix, link...
Contracting the Evaluation <ul><li>If human resources are not available in the organization, may need to hire one or more ...
Contracting Evaluations <ul><li>Advantages of Hiring </li></ul><ul><li>contractor may have in-depth knowledge of the secto...
Steps for Hiring Contractor <ul><li>Options: </li></ul><ul><li>Develop a request for proposal (RFP) and use a selection pa...
Items for Request for Proposal (RFP) <ul><li>Purposes of  the evaluation </li></ul><ul><li>Background and context for the ...
Selection Process for Contracting <ul><li>Select a panel comprising people with: </li></ul><ul><ul><li>evaluation knowledg...
Possible Criteria for Selecting the Contractor <ul><li>Contractor’s record of related accomplishment </li></ul><ul><li>Evi...
Terms of Reference (TOR) <ul><li>Often stand-alone [no RFP] </li></ul><ul><li>TOR describe the evaluation and establish th...
Definition of  Terms of Reference <ul><li>OECD, DAC Glossary: </li></ul><ul><ul><li>TOR are a written documentation that p...
TOR Typically Include: <ul><li>Short title and description of the intervention </li></ul><ul><li>Purpose of the evaluation...
IPDET © 2009 The good practice Contract or ToR will include: A code of ethics Request for analysis of how intervention has...
TOR Checklists <ul><li>UNDP :  www.undp.org/eo/documents/erc/ Evaluation _ TOR _Description.doc </li></ul><ul><li>UNICEF: ...
Responsibilities of the Contact/TOR Manager <ul><li>Setting clear and  realistic  objectives </li></ul><ul><li>Maintaining...
Roles and Responsibilities  <ul><li>Define and agree to who is responsible for what and when </li></ul><ul><li>Who needs r...
Main Client(s) <ul><li>Meet with main client early to discuss </li></ul><ul><ul><li>the nature and context of the request ...
Key Stakeholders <ul><li>Organizations representing those with “stakes” in the intervention or individuals </li></ul><ul><...
Evaluation Manager Role and  Responsibilities <ul><li>Detailed responsibilities in three areas: </li></ul><ul><ul><li>prep...
Evaluator Skills and Abilities <ul><li>Expertise in the specific subject matter </li></ul><ul><li>Knowledge of key develop...
Managing People <ul><li>People are complicated. They are not machines. Their behavior will change from day to day. </li></...
Conflict Management <ul><li>Communication skills </li></ul><ul><ul><li>use “I” instead of “you” language </li></ul></ul><u...
Teambuilding Skills <ul><li>Listening, questioning, persuading, respecting, helping </li></ul><ul><li>Brainstorming </li><...
Managing Tasks <ul><li>Task map  </li></ul><ul><ul><li>everyone’s assignments </li></ul></ul><ul><ul><li>start and complet...
Example of Task Map IPDET © 2009 Task Name Start Date Date due Conduct review of the literature and identify related inter...
Example of Gantt Chart IPDET © 2009 Task Week  1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Conduct literature review   H...
Managing Budgets <ul><li>Evaluation staff salaries and benefits </li></ul><ul><li>Consultants </li></ul><ul><li>Travel and...
Project Management Process IPDET © 2009 Initiating Planning Executing Controlling Closing Skip to Last Slide
Phase I: Initiating <ul><li>Projects begin by confirming there is a need for the project </li></ul><ul><li>Actions to be u...
Phase II: Planning <ul><li>Managers spend much of their time planning </li></ul><ul><li>13 actions in this phase </li></ul...
Actions for Planning <ul><li>Describe the project scope </li></ul><ul><li>Define and sequence project activities </li></ul...
Actions for Planning  (cont.) <ul><li>Create a formal quality plan </li></ul><ul><li>Create a formal project communication...
Phase III: Executing <ul><li>Manager makes sure the tasks are being completed on schedule </li></ul><ul><li>Action to be u...
Phase IV: Controlling <ul><li>Manager tracks people, activities, money, and scope </li></ul><ul><li>Action to be undertake...
Phase V: Closing <ul><li>Manager finalizes activities, archives information </li></ul><ul><li>Action to be undertaken: Clo...
Carol Weiss: On Building Evaluation Use <ul><li>Who are the primary intended users of the evaluation? </li></ul><ul><li>Wh...
A Good Evaluation: <ul><li>Meets stakeholder needs and requirements </li></ul><ul><li>Is relevant and realistic in scope <...
Kusek and Rist’s Six Characteristics of Quality Evaluations <ul><li>Impartiality </li></ul><ul><li>Usefulness </li></ul><u...
Checklists for Quality <ul><li>The Key Evaluation Checklist (Scriven) </li></ul><ul><li>Program Evaluations Meta-evaluatio...
External Reviewer <ul><li>A person with evaluation and/or subject matter expertise who is not involved in conducting the e...
<ul><li>“ Good management is the art of making problems so interesting and their solutions so constructive that everyone w...
Upcoming SlideShare
Loading in …5
×

Module 12: Managing an Evaluation

751 views

Published on

Published in: Business, Technology
  • useful! thanks for sharing
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Module 12: Managing an Evaluation

  1. 1. Module 12: Managing an Evaluation
  2. 2. Introduction <ul><li>Managing the Design Matrix </li></ul><ul><li>Contracting the Evaluation </li></ul><ul><li>Roles and Responsibilities of Different Players </li></ul><ul><li>Managing People, Tasks, and Budgets </li></ul>IPDET © 2009
  3. 3. Evaluation Design Matrix <ul><li>Key to success in evaluation is planning </li></ul><ul><li>Evaluation Design Matrix, links descriptive, normative, and impact evaluation questions </li></ul><ul><li>Helps to systematically map out the evaluation through an iterative process </li></ul><ul><li>Focuses the attention on each of the major components for each [sub]question </li></ul><ul><li>Can be adapted </li></ul>IPDET © 2009
  4. 4. Contracting the Evaluation <ul><li>If human resources are not available in the organization, may need to hire one or more other people to assist </li></ul><ul><li>Can be conduct the entire study or only parts </li></ul>IPDET © 2009
  5. 5. Contracting Evaluations <ul><li>Advantages of Hiring </li></ul><ul><li>contractor may have in-depth knowledge of the sector or issue and strong evaluation skills </li></ul><ul><li>contractor may speak the local language and have local knowledge </li></ul><ul><li>Disadvantages of Hiring </li></ul><ul><li>contracting can be expensive </li></ul><ul><li>Can be difficult to determine evaluation ca[abilities </li></ul><ul><li>in-house capacity is not increased </li></ul><ul><li>No corporate memory </li></ul>IPDET © 2009
  6. 6. Steps for Hiring Contractor <ul><li>Options: </li></ul><ul><li>Develop a request for proposal (RFP) and use a selection panel to choose the winning proposal (firms) </li></ul><ul><li>Develop terms of reference (TOR) (indivduals) </li></ul>IPDET © 2009
  7. 7. Items for Request for Proposal (RFP) <ul><li>Purposes of the evaluation </li></ul><ul><li>Background and context for the study </li></ul><ul><li>Key information requirements, including evaluation scope and general approach </li></ul><ul><li>Evaluation objectives </li></ul><ul><li>Deliverables required and time frame for them </li></ul><ul><li>Criteria for tender selection </li></ul><ul><li>Contract details for the project manager </li></ul><ul><li>Deadline for the proposal </li></ul><ul><li>Budget and other resources </li></ul>IPDET © 2009
  8. 8. Selection Process for Contracting <ul><li>Select a panel comprising people with: </li></ul><ul><ul><li>evaluation knowledge and experience </li></ul></ul><ul><ul><li>knowledge of the content area </li></ul></ul><ul><ul><li>knowledge of the organizational culture </li></ul></ul><ul><li>Have the panel select the proposal using the criteria in the RFP </li></ul><ul><li>Manage a fair and transparent process </li></ul><ul><li>Keep a record of the selection process </li></ul>IPDET © 2009
  9. 9. Possible Criteria for Selecting the Contractor <ul><li>Contractor’s record of related accomplishment </li></ul><ul><li>Evidence of specific needed competencies </li></ul><ul><li>Willingness to use the design matrix </li></ul><ul><li>RFP adequately addressed </li></ul><ul><li>Quality of T of C model, if requested by RFP </li></ul><ul><li>Detailed explanation of the time commitment of the principal and other staff </li></ul><ul><li>Reasonable cost —specified in detail </li></ul><ul><li>Quality of similar evaluations contractor has done </li></ul>IPDET © 2009
  10. 10. Terms of Reference (TOR) <ul><li>Often stand-alone [no RFP] </li></ul><ul><li>TOR describe the evaluation and establish the responsibilities and timeline for deliverables </li></ul><ul><li>Often used for all members of an evaluation team, whether consultants and/or internal team members </li></ul><ul><li>Helps all understand expectations and context of the evaluation </li></ul>IPDET © 2009
  11. 11. Definition of Terms of Reference <ul><li>OECD, DAC Glossary: </li></ul><ul><ul><li>TOR are a written documentation that present: </li></ul></ul><ul><ul><ul><li>the purpose and scope of the evaluation </li></ul></ul></ul><ul><ul><ul><li>the methods to be used </li></ul></ul></ul><ul><ul><ul><li>the standard against which performance is to be assessed or analyses are to be conducted </li></ul></ul></ul><ul><ul><ul><li>the resources and time allocated </li></ul></ul></ul><ul><ul><ul><li>reporting requirements </li></ul></ul></ul>IPDET © 2009
  12. 12. TOR Typically Include: <ul><li>Short title and description of the intervention </li></ul><ul><li>Purpose of the evaluation and general approach </li></ul><ul><li>The T of C [or this may be the first deliverable] </li></ul><ul><li>The scope, focus, and overall time frame of the evaluation: initial issues and questions for address </li></ul><ul><li>Nature and extent of stakeholder involvement </li></ul><ul><li>Deliverables and schedule, including review process: design matrix, evaluation work plan, interim reports, final report, presentations </li></ul><ul><li>Needed evaluator [or team] qualifications </li></ul><ul><li>Overall budgets for staff and travel </li></ul>IPDET © 2009
  13. 13. IPDET © 2009 The good practice Contract or ToR will include: A code of ethics Request for analysis of how intervention has addressed issues of human rights, gender equity, environmental and social sustainability
  14. 14. TOR Checklists <ul><li>UNDP : www.undp.org/eo/documents/erc/ Evaluation _ TOR _Description.doc </li></ul><ul><li>UNICEF: www.unicef.org/ evaluation /files/TechNote2_ TOR .pdf </li></ul><ul><li>CIDA: http://www.acdicida.gc.ca/INET/IMAGES.NSF/vLUImages/Performancereview4/$file/Tor_sample_text.pdf </li></ul>IPDET © 2009
  15. 15. Responsibilities of the Contact/TOR Manager <ul><li>Setting clear and realistic objectives </li></ul><ul><li>Maintaining ownership of the study </li></ul><ul><li>Monitoring the work and providing timely feedback </li></ul><ul><li>Responsive decision-making </li></ul><ul><li>Openness to renegotiation if scope is expanded or unanticipated problems encountered </li></ul>IPDET © 2009
  16. 16. Roles and Responsibilities <ul><li>Define and agree to who is responsible for what and when </li></ul><ul><li>Who needs roles and responsibilities defined: </li></ul><ul><ul><li>main client </li></ul></ul><ul><ul><li>Key stakeholders </li></ul></ul><ul><ul><li>evaluation manager </li></ul></ul><ul><ul><li>evaluators </li></ul></ul>IPDET © 2009
  17. 17. Main Client(s) <ul><li>Meet with main client early to discuss </li></ul><ul><ul><li>the nature and context of the request </li></ul></ul><ul><ul><li>the need, issue, or interest leading to the evaluation request </li></ul></ul><ul><ul><li>critical timing needs for the evaluation findings </li></ul></ul><ul><ul><li>questions of major importance for the evaluation to address </li></ul></ul><ul><ul><li>communication schedule and frequency </li></ul></ul>IPDET © 2009
  18. 18. Key Stakeholders <ul><li>Organizations representing those with “stakes” in the intervention or individuals </li></ul><ul><li>May be internal or external </li></ul><ul><li>Responsibilities: </li></ul><ul><ul><li>identify issues and questions for the evaluation to consider </li></ul></ul><ul><ul><li>in participatory evaluations, may share responsibilities for planning, conduct, reporting of evaluation </li></ul></ul>IPDET © 2009
  19. 19. Evaluation Manager Role and Responsibilities <ul><li>Detailed responsibilities in three areas: </li></ul><ul><ul><li>preparation </li></ul></ul><ul><ul><li>implementation </li></ul></ul><ul><ul><li>follow-up </li></ul></ul>IPDET © 2009
  20. 20. Evaluator Skills and Abilities <ul><li>Expertise in the specific subject matter </li></ul><ul><li>Knowledge of key development issues, especially those relating to the main goals or the ability to see the “big picture” </li></ul><ul><li>Familiarity with organization’s business and the way such business is conducted </li></ul><ul><li>Evaluation skills in design, data collection, data analysis, and preparing reports </li></ul><ul><li>Skills in the use of information technology </li></ul>IPDET © 2009
  21. 21. Managing People <ul><li>People are complicated. They are not machines. Their behavior will change from day to day. </li></ul><ul><li>The manager is responsible for everything that happens within the manager’s scope of authority. </li></ul>IPDET © 2009
  22. 22. Conflict Management <ul><li>Communication skills </li></ul><ul><ul><li>use “I” instead of “you” language </li></ul></ul><ul><li>Listening skills </li></ul><ul><ul><li>confirm that the listener has correctly understood the speaker </li></ul></ul><ul><li>Most constructive conflicts end up with both parties “winning” </li></ul>IPDET © 2009
  23. 23. Teambuilding Skills <ul><li>Listening, questioning, persuading, respecting, helping </li></ul><ul><li>Brainstorming </li></ul><ul><li>Concept mapping </li></ul>IPDET © 2009
  24. 24. Managing Tasks <ul><li>Task map </li></ul><ul><ul><li>everyone’s assignments </li></ul></ul><ul><ul><li>start and completion dates </li></ul></ul><ul><li>Gantt chart </li></ul><ul><ul><li>chart showing interrelationships of projects and schedules </li></ul></ul>IPDET © 2009
  25. 25. Example of Task Map IPDET © 2009 Task Name Start Date Date due Conduct review of the literature and identify related internal and external issues Anna, Miguel, and Kabir 7/01 7/04 Review program/project documents Kabir 7/05 7/23 Schedule and hold meet ing with client Anna 7/15 7/31 Identify key stakeholders and schedule meetings Kabir and Miguel 7/15 7/17 Write up summary of meetings with client and decisions made Anna 8/01 8/31 Conduct stakeholder meetings – summarize Anna and Miguel 8/05 8/15 Draft initial design for evaluation Anna 7/01 8/31
  26. 26. Example of Gantt Chart IPDET © 2009 Task Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Conduct literature review   Hold stakeholder meetings   Draft theory of change  
  27. 27. Managing Budgets <ul><li>Evaluation staff salaries and benefits </li></ul><ul><li>Consultants </li></ul><ul><li>Travel and per diem </li></ul><ul><li>Communication (postage, telephone, etc.) </li></ul><ul><li>Printing and duplication </li></ul><ul><li>Data processing </li></ul><ul><li>Printed materials </li></ul><ul><li>Supplies and equipment </li></ul><ul><li>Subcontracts </li></ul><ul><li>Overhead (facilities, utilities, etc.) </li></ul>IPDET © 2009
  28. 28. Project Management Process IPDET © 2009 Initiating Planning Executing Controlling Closing Skip to Last Slide
  29. 29. Phase I: Initiating <ul><li>Projects begin by confirming there is a need for the project </li></ul><ul><li>Actions to be undertaken: </li></ul><ul><ul><li>Demonstrate project need and feasibility </li></ul></ul><ul><ul><li>Obtain project authorization </li></ul></ul><ul><ul><li>Obtain authorization for the phase </li></ul></ul>IPDET © 2009
  30. 30. Phase II: Planning <ul><li>Managers spend much of their time planning </li></ul><ul><li>13 actions in this phase </li></ul><ul><li>Some steps are optional, depends on the size and scope of the project </li></ul>IPDET © 2009
  31. 31. Actions for Planning <ul><li>Describe the project scope </li></ul><ul><li>Define and sequence project activities </li></ul><ul><li>Estimate durations for activities and resources required </li></ul><ul><li>Develop a project schedule </li></ul><ul><li>Estimate costs </li></ul><ul><li>Build a budget and spending plan </li></ul><ul><li>(continued on next slide) </li></ul>IPDET © 2009
  32. 32. Actions for Planning (cont.) <ul><li>Create a formal quality plan </li></ul><ul><li>Create a formal project communication plan </li></ul><ul><li>Organize and acquire staff </li></ul><ul><li>Identify risks and plan to respond </li></ul><ul><li>Plan for and acquire outside resources </li></ul><ul><li>Organize the project plan </li></ul><ul><li>Close out the project planning phase </li></ul>IPDET © 2009
  33. 33. Phase III: Executing <ul><li>Manager makes sure the tasks are being completed on schedule </li></ul><ul><li>Action to be undertaken: Execute project activities </li></ul>IPDET © 2009
  34. 34. Phase IV: Controlling <ul><li>Manager tracks people, activities, money, and scope </li></ul><ul><li>Action to be undertake: Control project activities </li></ul>IPDET © 2009
  35. 35. Phase V: Closing <ul><li>Manager finalizes activities, archives information </li></ul><ul><li>Action to be undertaken: Close out project activities </li></ul>IPDET © 2009
  36. 36. Carol Weiss: On Building Evaluation Use <ul><li>Who are the primary intended users of the evaluation? </li></ul><ul><li>Who are the target audiences of the evaluation? </li></ul><ul><li>Which groups or individuals are most likely to be affected by the evaluation? </li></ul><ul><li>Whose actions and/or decisions will be influenced by their engagement with the evaluation process or evaluation findings? </li></ul><ul><li>How can the intended users of the evaluation be involved? </li></ul><ul><li>What challenges/barriers might you face in identifying and involving users? How to overcome them? </li></ul>IPDET © 2009
  37. 37. A Good Evaluation: <ul><li>Meets stakeholder needs and requirements </li></ul><ul><li>Is relevant and realistic in scope </li></ul><ul><li>Uses appropriate methods </li></ul><ul><li>Produces reliable, accurate, and valid data </li></ul><ul><li>Includes appropriate and accurate analysis of data </li></ul><ul><li>Presents impartial conclusions </li></ul><ul><li>Conveys results clearly — in oral or written form </li></ul><ul><li>Meets professional standards </li></ul>IPDET © 2009
  38. 38. Kusek and Rist’s Six Characteristics of Quality Evaluations <ul><li>Impartiality </li></ul><ul><li>Usefulness </li></ul><ul><li>Technical adequacy </li></ul><ul><li>Stakeholder involvement </li></ul><ul><li>Feedback and dissemination </li></ul><ul><li>Value for money </li></ul>IPDET © 2009
  39. 39. Checklists for Quality <ul><li>The Key Evaluation Checklist (Scriven) </li></ul><ul><li>Program Evaluations Meta-evaluation Checklist (Stufflebeam) </li></ul><ul><li>Utilization Focused Evaluation Checklist (Patton) </li></ul><ul><li>Guidelines and Checklist for Constructivist Evaluation (Guba & Lincoln) </li></ul><ul><li>Deliberative Democratic Evaluation Checklist (House & Lowe) </li></ul><ul><li>Guiding Principles Checklist (Stufflebeam) </li></ul>IPDET © 2009
  40. 40. External Reviewer <ul><li>A person with evaluation and/or subject matter expertise who is not involved in conducting the evaluation, but who you can use as a sounding board, advisor, and helpful critic </li></ul>IPDET © 2009
  41. 41. <ul><li>“ Good management is the art of making problems so interesting and their solutions so constructive that everyone wants to get to work and deal with them.“ -- Paul Hawken, Growing a Business </li></ul>IPDET © 2009 Questions?

×