Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evaluation in Education


Published on

Evaluation -What it is? -Definitions
Why it is needed ? - Purposes
How it is done ? - Levels

Published in: Education

Evaluation in Education

  1. 1. EVALUATION IN EDUCATION Dr. Kusum Gaur Asso. Prof. PSM WHO Fellow for IEC 09/07/10
  2. 2. Is that web site good enough to surf?
  3. 3. This one looks good, but How can I” ll be sure?
  4. 4. Think of CARRDSS <ul><li>C REDIBILITY / AUTHORITY </li></ul><ul><li>A CCURACY </li></ul><ul><li>R ELIABILITY </li></ul><ul><li>R ELEVANCE </li></ul><ul><li>D ATE </li></ul><ul><li>S OURCES BEHIND THE TEXT </li></ul><ul><li>S COPE AND PURPOSE </li></ul>
  5. 5. There are other questions in life! <ul><ul><li>Which car should I buy? </li></ul></ul><ul><ul><li>Should I take this medication? </li></ul></ul><ul><ul><li>Should my child have this surgery? </li></ul></ul>
  6. 6. Just as we evaluate our website to surf . . . <ul><li>We have to take decisions </li></ul><ul><li>on the basis of some </li></ul><ul><li>Criteria </li></ul>
  7. 7. Evaluation is important! To draw conclusions and make new predictions.
  8. 8. Evaluation <ul><li>What is evaluation? </li></ul><ul><li>Why to evaluate? </li></ul><ul><li>How it is done? </li></ul>
  9. 9. What is evaluation? <ul><ul><li>Evaluation is systemic determination of merit, worth and significance of something or someone using criteria against a set of standard. ( </li></ul></ul><ul><ul><li>Evaluation is to draw; to assess; to compute an expression ( </li></ul></ul><ul><ul><li>Evaluation is the process of making judgment based on criteria and evidence. ( </li></ul></ul>
  10. 10. What is evaluation? <ul><li>Evaluation is the process of examining a subject and rating it based on </li></ul><ul><li>its important features </li></ul><ul><li>i.e. criteria </li></ul>
  11. 11. Why to Evaluate <ul><li>A ccountability </li></ul><ul><li>V alidating our hypothesis </li></ul><ul><li>C omparison </li></ul><ul><li>K nowing Status </li></ul><ul><li>K nowing Needs </li></ul><ul><li>P lanning further </li></ul><ul><li>P URPOSE ……........... </li></ul>
  12. 12. Why to Evaluate…. <ul><li>M easures the effectiveness of the instructor </li></ul><ul><li>M easures the effectiveness of impact in meeting objectives </li></ul><ul><li>P rovide feedback to students </li></ul><ul><li>P rovides students gratification and motivation ........... </li></ul>
  13. 13. To be Known before How to EVALUATE ? For Quality Evaluation
  14. 14. Quality Evaluation Criteria Questions Techniques M e thods Quality How the evaluation will be done
  15. 15. Criteria <ul><li>Certain standard </li></ul><ul><li>on which the </li></ul><ul><li>achievements of a learner </li></ul><ul><li>is measured </li></ul>
  16. 16. Good Quality Criteria <ul><li>V alidity </li></ul><ul><ul><li>R eliability </li></ul></ul><ul><li>R eproducibility </li></ul><ul><li>S ensitive </li></ul><ul><li>S pecific </li></ul>(VRRSS)
  17. 17. Types of Evaluation <ul><li>As per Quality of Evaluation </li></ul><ul><ul><li>Q uantitative </li></ul></ul><ul><ul><li>Q ualitative </li></ul></ul><ul><li>As per Time of Evaluation </li></ul><ul><ul><li>F ormative </li></ul></ul><ul><ul><li>S ummative </li></ul></ul><ul><ul><li>P re – P ost Evaluation </li></ul></ul>
  18. 18. Quality of Evaluation <ul><li>Quantitative Evaluation </li></ul><ul><ul><li>Provide a quantifiable objective measure </li></ul></ul><ul><ul><li>Expressed in proportions </li></ul></ul><ul><ul><li>Example: </li></ul></ul><ul><ul><ul><li>How many students have got >60%? </li></ul></ul></ul><ul><li>Qualitative Evaluation </li></ul><ul><ul><li>Communicate general expectations </li></ul></ul><ul><ul><li>Expressed in grading </li></ul></ul><ul><ul><li>Open to interpretation </li></ul></ul><ul><ul><li>Examples: </li></ul></ul><ul><ul><ul><li>What about his socio-economic status ? </li></ul></ul></ul>
  19. 19. Time of Evaluation <ul><li>Formative Evaluation </li></ul><ul><ul><li>Ongoing evaluation during an instructional period </li></ul></ul><ul><ul><li>To know the perceptions of the students in comparison to instructor . </li></ul></ul><ul><li>Summative Evaluation </li></ul><ul><ul><li>Conducted at the end course. </li></ul></ul><ul><ul><li>Purpose is to form a judgment about </li></ul></ul><ul><ul><ul><ul><li>Performance of student </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Effectiveness of an instructor </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Effectiveness of the course </li></ul></ul></ul></ul><ul><ul><li>Regularly scheduled at the end of academic terms. </li></ul></ul><ul><ul><li>Pre and post Evaluation </li></ul></ul>
  20. 20. Formative v/s Summative Evaluation Quality Formative Summative Purpose detect strengths & weakness Overall achievements Frequency During or end of unit In end – point of certification, promotion Area covered One unit/no. of units Course content Administrative utility Advisory, not always for permanent record Decisive, for permanent record Feedback to students Done immediately Inform regarding pass or fail Feedback to faculty If significant no. shows error than weakness in instruction Overall pass or fail
  21. 21. Pre- and Post-Evaluation <ul><ul><li>Evaluate in the beginning to asses needs </li></ul></ul><ul><ul><li>Evaluate in the end to assess outcome </li></ul></ul><ul><ul><li>To assess degree of achievement of objectives through pre-post evaluation. </li></ul></ul>
  22. 22. Questions Evaluation Questions should Include Each level of Evaluation
  23. 23. * Levels of Evaluation <ul><ul><li>Level I – Reaction </li></ul></ul><ul><ul><ul><ul><li>H ow did the student react to the class? </li></ul></ul></ul></ul><ul><ul><li>Level - II Learning √ </li></ul></ul><ul><ul><ul><ul><li>W hat has the student learned? </li></ul></ul></ul></ul><ul><ul><li>Level III – Skill √ </li></ul></ul><ul><ul><ul><ul><li>H ow much did the student retain? </li></ul></ul></ul></ul><ul><ul><li>Level – IV Impact √ </li></ul></ul><ul><ul><ul><ul><li>W hat is the final impact or practical application of this learning? </li></ul></ul></ul></ul>
  24. 24. Techniques of Evaluation <ul><ul><li>T eaching dossiers </li></ul></ul><ul><ul><li>S tudent ratings </li></ul></ul><ul><ul><li>P eer observations </li></ul></ul><ul><ul><li>I nterviews </li></ul></ul><ul><ul><li>P ortfolios </li></ul></ul><ul><ul><li>C lassroom A ssessment </li></ul></ul><ul><li>Evaluation should use a combination of strategies to take advantage of their inherent strengths as well as their individual limitations . </li></ul>
  25. 25. Techniques of Evaluation <ul><li>Teaching Dossiers </li></ul><ul><ul><li>Usually done in the end of unit and in the form of some written document. </li></ul></ul><ul><ul><li>As an submission assessment of students </li></ul></ul><ul><li>Student Rating </li></ul><ul><ul><li>Rating of teaching by students </li></ul></ul><ul><li>P eer observations </li></ul><ul><ul><li>Colleagues judge teaching quality by observation </li></ul></ul><ul><ul><li>Peer observation is usually useful for formative evaluation </li></ul></ul><ul><li>I nterviews </li></ul><ul><ul><li>used in teaching award nominations </li></ul></ul><ul><li>P ortfolios </li></ul><ul><ul><li>Appropriateness of course goals and objectives </li></ul></ul><ul><li>C lassroom A ssessment </li></ul><ul><ul><li>Effectiveness of teaching on Learning </li></ul></ul><ul><ul><li>can be used in a timely way to help instructors identify gaps </li></ul></ul>
  26. 26. Methodologies <ul><ul><li>Project Assignments </li></ul></ul><ul><ul><li>Observation </li></ul></ul><ul><ul><ul><ul><li>Simple Observation </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Think-aloud (let the examinee explain) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Constructive Interaction </li></ul></ul></ul></ul><ul><ul><li>Query via </li></ul></ul><ul><ul><ul><ul><li>Interviews </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Focal Group Discussion (FGD) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Questionnaires </li></ul></ul></ul></ul><ul><li>Combinations of methodologies should be used </li></ul>
  27. 27. Summary of Methodologies Type Strengths Weaknesses Project Assignment Creativity Skill Judgment Time Consuming Analysis is Difficult Observational Skill Judgment In Depth Less Planning require Time Consuming Analysis is Difficult Interview Depth High Response Communication skill Extensive Planning Time Consuming Analysis Difficult Focus group discussion (FGD) Group Energy Communication skill Interpersonal Relationship Extensive Planning Analysis Difficult Questionnaire Extensive in Use In Depth Easy Analysis Extensive Planning
  28. 28. Styles of Questions <ul><ul><li>Open-ended questions </li></ul></ul><ul><ul><li>Closed </li></ul></ul><ul><ul><li>Scalar </li></ul></ul><ul><ul><li>Multi-choice </li></ul></ul><ul><ul><li>Ranked </li></ul></ul><ul><ul><li>Combining open-ended and closed questions </li></ul></ul>
  29. 29. Open-ended questions <ul><ul><li>Asks for unprompted opinions </li></ul></ul><ul><ul><li>Good for general subjective information </li></ul></ul><ul><ul><li>But difficult to analyse </li></ul></ul><ul><ul><li>eg: Can you suggest any improvements to the present health care delivery system? </li></ul></ul>
  30. 30. Closed questions <ul><ul><li>Restricts the respondent’s responses by supplying alternative answers </li></ul></ul><ul><ul><li>Can be easily analyzed </li></ul></ul><ul><ul><li>But watch out for hard to interpret responses! </li></ul></ul><ul><ul><li>Alternative answers should be very specific </li></ul></ul>
  31. 31. Scalar <ul><ul><li>Ask user to judge a specific statement on a numeric scale </li></ul></ul><ul><ul><li>Scale usually corresponds with agreement or disagreement with a statement </li></ul></ul>
  32. 32. Multi-choice <ul><ul><li>Respondent offered a choice of explicit responses </li></ul></ul><ul><ul><li>e.g. Which types of presentation is this.? </li></ul></ul><ul><ul><ul><ul><li>Microsoft word </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Microsoft power point </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Microsoft excel </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Microsoft outlook </li></ul></ul></ul></ul>
  33. 33. Ranked <ul><ul><li>Respondent places an ordering on items in a list </li></ul></ul><ul><ul><li>Useful to indicate a user’s preferences </li></ul></ul><ul><ul><li>e.g. Rank the usefulness of 108 services in health sector. </li></ul></ul><ul><ul><ul><ul><li>Useful </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Do not know </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Of no use </li></ul></ul></ul></ul>
  34. 34. Combining open-ended and closed questions <ul><ul><li>Gets specific response, but allows room for user’s opinion </li></ul></ul><ul><ul><li>e.g. Compare and comment on IMR in developing with developed country </li></ul></ul>
  35. 35. Tests <ul><ul><li>Written </li></ul></ul><ul><ul><ul><ul><ul><li>Questionnaire </li></ul></ul></ul></ul></ul><ul><ul><li>Practical </li></ul></ul><ul><ul><ul><ul><ul><li>Project Assignment, Case Study, Observation, Interview, FGD </li></ul></ul></ul></ul></ul><ul><ul><li>Oral </li></ul></ul><ul><ul><ul><ul><ul><li>Interview, FGD </li></ul></ul></ul></ul></ul>
  36. 36. Written Tests <ul><ul><ul><ul><li>Multiple Choice </li></ul></ul></ul></ul><ul><ul><ul><ul><li>True/False </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Matching </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Completion or fill in the blank </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Essay </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Short Notes </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Long Essay </li></ul></ul></ul></ul></ul>
  37. 37. Multiple Choice Tests <ul><ul><li>Common method </li></ul></ul><ul><ul><li>Easy to grade and be objective </li></ul></ul><ul><ul><li>Test construction miscues: </li></ul></ul><ul><ul><ul><ul><li>Using previous questions information </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Negatively worded stems </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Fill in the blank in middle of stem </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Using ‘all of the above” or “none of the above” </li></ul></ul></ul></ul>
  38. 38. True/False Tests <ul><ul><li>Limited to two answers, no gray area </li></ul></ul><ul><ul><li>Difficult to construct in positive voice </li></ul></ul><ul><ul><li>Avoid always or never statements </li></ul></ul><ul><ul><li>Useful tool as a study guide </li></ul></ul>
  39. 39. Matching Tests <ul><ul><li>Works best with definitions and terms </li></ul></ul><ul><ul><li>Difficult to design </li></ul></ul><ul><ul><li>Cautious of multiple matches </li></ul></ul><ul><ul><li>Test directions must be clear </li></ul></ul>
  40. 40. Completion Tests <ul><ul><li>Fill in the blank </li></ul></ul><ul><ul><li>Statements must be clear as to intent of question </li></ul></ul><ul><ul><li>Need to be grammatically correct </li></ul></ul><ul><ul><li>Be aware of size of blank </li></ul></ul><ul><ul><li>Avoid having blank at beginning of sentence </li></ul></ul>
  41. 41. Essay Tests <ul><ul><li>May require long or short answer </li></ul></ul><ul><ul><li>Time consuming and difficult to grade </li></ul></ul><ul><ul><li>Recommended to grade in a group format </li></ul></ul><ul><ul><li>Hand written exams must allow sufficient time </li></ul></ul><ul><ul><li>Shotgun approach.. As much information as possible in hopes of hitting the target </li></ul></ul>
  42. 42. Oral Exams <ul><ul><li>Requires verbal answers by students </li></ul></ul><ul><ul><li>Advantages </li></ul></ul><ul><ul><ul><ul><li>Evaluate quick reaction of student </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Assesses the student thought process </li></ul></ul></ul></ul><ul><ul><li>Disadvantages </li></ul></ul><ul><ul><ul><ul><li>Limited number of students examined at one time </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Difficult to standardize </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Time consuming and labor intensive </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Unexpected distractions </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Unfair emphasis on repeated mistakes </li></ul></ul></ul></ul>
  43. 43. Project Assignments <ul><ul><li>Gets students working outside the class </li></ul></ul><ul><ul><li>In groups, helps develop people skills </li></ul></ul><ul><ul><li>Negatives </li></ul></ul><ul><ul><ul><li>Hard to standardize </li></ul></ul></ul><ul><ul><ul><li>Potential plagiarism </li></ul></ul></ul><ul><ul><ul><li>May measure only end product and not consider the process </li></ul></ul></ul>
  44. 44. Practical Exams <ul><ul><li>Demonstration of a skill in the context of a scenario </li></ul></ul><ul><ul><li>Demonstration of steps of performing a skill </li></ul></ul>
  45. 45. How to EVALUATION Process of Evaluation
  46. 46. Process of Evaluation (9 Steps) Planning Implementation Feedback Step 1. Define Purpose & Scope Step 2. Specify Evaluation Question Step 3. Specify Evaluation Design Step 4. Create Data Collection Action Plan Step 5. Data Collection Step 6. Data analysis Step 7. Finding data Inference Step 8. Disseminate Information Step 9. Feedback for improvement
  47. 47. Data Collection Action Plan <ul><ul><li>D ecide level of Evaluation </li></ul></ul><ul><ul><li>T ype of Evaluation </li></ul></ul><ul><ul><li>T ype of Evaluation Technique </li></ul></ul><ul><ul><li>W hat Evaluation Questions ? </li></ul></ul><ul><ul><li>F rom whom/Data Sources ? </li></ul></ul><ul><ul><li>B y whom ? </li></ul></ul><ul><ul><li>W hen Collected ? </li></ul></ul><ul><ul><li>H ow Collected ? </li></ul></ul><ul><ul><li>H ow Data are to be Analyzed ? </li></ul></ul>
  48. 48. Accreditation Council for Graduate Medical Education (ACGME) <ul><li>Six General Competencies: </li></ul><ul><ul><li>Patient Care </li></ul></ul><ul><ul><li>Medical Knowledge </li></ul></ul><ul><ul><li>Practice-Based Learning </li></ul></ul><ul><ul><li>Interpersonal and Communication Skills </li></ul></ul><ul><ul><li>Professionalism </li></ul></ul><ul><ul><li>Systems-Based Practice </li></ul></ul>
  49. 49. Choosing an Evaluation Method <ul><li>why in process: Purpose of evaluation </li></ul><ul><li>When in process: Formative vs. Summative </li></ul><ul><li>style of evaluation: laboratory vs. field </li></ul><ul><li>type of measures: qualitative vs. quantitative </li></ul><ul><li>level of information: high level vs. low level </li></ul><ul><li>resources available: time, subjects, tools & equipment, expertise etc . </li></ul>
  50. 50. Example <ul><li>Which type of test is most commonly used for state or national certifications? </li></ul><ul><ul><ul><ul><li>True/False </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Matching </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Multiple choice </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Fill in the blank </li></ul></ul></ul></ul>
  51. 51. Goal of Evaluation Status Evaluation Feedback
  52. 52. “ If you can not measure it, you can not improve it.” so Evaluation is important Dr. Kusum Gaur Asso. Prof. PSM WHO Fellow IEC
  53. 53. Thanks