Design Chapter 7 - Testing and Evaluation Techniques


Published on

Published in: Education, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Levels established by educator Donald Kirkpatrick. Take a look at http://
  • Typically, this is done through the use of an evaluation form. For discussion, “when is the best time to have an evaluation form completed”? Some think the very last thing in a class. Unfortunately, so often, the students are in a big hurry to get done and out the door that the evaluations are really not honest.
  • Exams, tests, and skill assessments help an instructors evaluate if what they have taught is actually absorbed by the student and if they are able to adequately recall.
  • Six weeks to six months down the road. Typically, we just don’t do follow up like this. How often have you said that the true evaluation of your course is the safe retirement of the student and possibly even those they have supervised? This should be a good slide for discussion. Can you think of an easy (or even reasonable) way to evaluating how much a student retains six weeks to six months post class?
  • How did the department benefit from the training. Was it a train the trainer scenario? Did the student share the information? How did the student utilize the training for the benefit of the organization?
  • Is the true evaluation completed with a safe retirement of the teacher, the student and all the students of the students? This slide is simply for consideration and discussion. Another thought: A teacher affects eternity; he can never tell where his influence stops.  ~Henry Brooks Adams
  • Compares overall goals, objectives, and content to the student performance. Think about the first day of a Course Delivery class and how the students handle the introduction process. Look at the progress about mid-way and then again at the end.
  • An activity should take place during the discussion of testing. Why not simply take a chapter from this course and ask one group to write a specified number of test questions (with possible exception of essay) and share them with the class. It serves as a good course review and you can evaluate the questions…..hey not to mention, you may use them for future classes. When writing any test questions, use caution with local vernacular wording.
  • All of the above responses can and have been successfully challenged. Choice “A” fits the answer and the rest of the question is not read. Choice “A” is actually a correct answer. None of the above questions tend to trick or cause misreading of questions that suddenly change from looking for positive to looking for negative responses.
  • Hopefully, true/false tests do not become a flip of the coin. Just for FYI, I wrote a 20 question T/F quiz once. The first 19 responses were true (maybe it was false) and the last one was false. I thought the class was going to lynch me.
  • Question: Do you provide more possible answers than there are matches?
  • Don’t forget the term synonym. Does your response have to be exact or would a word with the same meaning be acceptable. You need to know that ahead of grading.
  • Suggest the General Writing Rubric (or one you like better) be handed out as an example. Copy in your “misc” folder.
  • As an instructor, you need to be watching and evaluating the group projects. If you don’t, it is possible that a small number of the group will actually be doing while the others are skating.
  • Rote learning is a learning technique which avoids understanding of a subject and instead focuses on memorization . The major practice involved in rote learning is learning by repetition . The idea is that one will be able to quickly recall the meaning of the material the more one repeats it. Although it has been criticized by some schools of thought , rote learning is commonly used in the areas of mathematics, music, and religion. (Wikipedia). In practical settings this might be an excellent way of doing things. We have learned the “practice makes perfect”. It is likely less beneficial in classroom settings.
  • Situational skills testing can give a more true to life evaluation.
  • Student is being evaluated on using his bunker gear as flotation until he is able to properly remove it all. Check list would be difficult since they are in the water. Students were adequately briefed on what was expected of them prior to their “leap of faith”; stepping off the side of the pool in full bunker gear and air pack.
  • Photo from Extreme Extrication class at Great Florida Fire School 2007.
  • Do not forget, if there is a degree of potential hazard, have a safety officer.
  • Some argument, obviously, could be made for any of these. Oral; however, would have the most negative consequences related to unexpected distractions.
  • Your choice may have been transfer; however, that looks at a much closer time frame of 6 weeks to 6 months.
  • Design Chapter 7 - Testing and Evaluation Techniques

    1. 1. Testing and Evaluation Techniques Chapter 7 Florida State Fire College Ocala, Florida
    2. 2. Testing and Evaluation <ul><li>You had the opportunity to cover part of this chapter in Fire Service Course Delivery </li></ul>
    3. 3. Terminal Objective <ul><li>The student will be able to construct, administer, and evaluate an assessment instrument </li></ul>
    4. 4. Enabling Objectives <ul><li>The student shall be able to: </li></ul><ul><li>Define the four levels of evaluation </li></ul><ul><li>Differentiate between summative and formative evaluation </li></ul><ul><li>Define the different kinds of tests </li></ul><ul><li>Discuss the difference among the various types of tests </li></ul><ul><li>List various sources for tests </li></ul>
    5. 5. Four Levels of Evaluation <ul><li>Level I – reaction </li></ul><ul><li>Level II – learning </li></ul><ul><li>Level III – transfer </li></ul><ul><li>Level IV – business results </li></ul>
    6. 6. Reaction <ul><li>How did the student react to the class? </li></ul><ul><li>Evaluation form </li></ul><ul><li>Focus groups (rarely used) </li></ul><ul><li>Required by most academic institutions </li></ul>
    7. 7. Learning <ul><li>What has the student learned? </li></ul><ul><li>Oral examination </li></ul><ul><li>Written test </li></ul><ul><li>Skills assessment </li></ul>
    8. 8. Transfer <ul><li>How much did the student retain? </li></ul><ul><li>Measured six weeks to six months post coursework – on the job </li></ul><ul><li>Based upon tests, observations, surveys, and interviews </li></ul><ul><li>Not regularly done within the fire service </li></ul>
    9. 9. Business Results <ul><li>An assessment of the financial impact of the training or the return on investment. </li></ul><ul><li>Six months to two years post coursework </li></ul><ul><li>Most difficult to measure </li></ul><ul><ul><li>Training courses do not have business objectives – ie: reduction in accidents </li></ul></ul><ul><ul><li>Methodology for assessment is not yet refined </li></ul></ul>
    10. 10. Do we ever get a final evaluation? In teaching you cannot see the fruit of a day's work.  It is invisible and remains so, maybe for twenty years.  ~Jacques Barzun
    11. 11. Evaluation <ul><li>“An evaluation is a process of making a value judgment based upon one or more sources” </li></ul><ul><li>Evaluation processes examine two components: </li></ul><ul><li>Instruction from the teacher </li></ul><ul><li>Performance of the student on objectives </li></ul>
    12. 12. Purpose of Evaluation <ul><li>Provides feedback to students </li></ul><ul><li>Provide students gratification and motivation </li></ul><ul><li>Measures the effectiveness of the instructor </li></ul><ul><li>Measures the effectiveness of the program in meeting objectives </li></ul>
    13. 13. Formative Evaluation <ul><li>Ongoing evaluation to change or adapt the program </li></ul><ul><li>Compares the objectives to the testing strategy </li></ul><ul><li>Occurs during development of course or test </li></ul><ul><ul><li>Pilot testing is formative evaluation </li></ul></ul>
    14. 14. Summative Evaluation <ul><li>Typically performed at end of program </li></ul><ul><li>Provides students feedback on mastery of subject </li></ul><ul><li>Provides feedback on effectiveness of teaching strategy </li></ul><ul><li>Summative tools </li></ul><ul><ul><li>Course evaluation forms </li></ul></ul><ul><ul><li>Final exams </li></ul></ul><ul><ul><li>Written </li></ul></ul><ul><ul><li>Practical </li></ul></ul>
    15. 15. Formal Evaluation <ul><li>“ the test” </li></ul><ul><ul><li>Did the student attain the course objectives? </li></ul></ul><ul><ul><li>Gives a grade </li></ul></ul><ul><ul><li>Adds stress to the student </li></ul></ul><ul><ul><li>If required to pass, it should not be the first testing of the material </li></ul></ul>
    16. 16. Informal Evaluation <ul><li>Provides student feedback </li></ul><ul><ul><li>With or without a recorded grade </li></ul></ul><ul><ul><li>Helps ID weaknesses and strengths </li></ul></ul><ul><ul><li>Use caution when presenting </li></ul></ul>
    17. 17. Tests <ul><li>Instructor’s last chance to emphasize the important information the student needs to retain </li></ul><ul><ul><li>Written </li></ul></ul><ul><ul><li>Practical </li></ul></ul><ul><ul><li>Oral </li></ul></ul>
    18. 18. Written Tests <ul><li>Multiple Choice </li></ul><ul><li>True/False </li></ul><ul><li>Matching </li></ul><ul><li>Completion or fill in the blank </li></ul><ul><li>Essay </li></ul>
    19. 19. Guidelines for Written Tests <ul><li>Be sure questions relate to objectives </li></ul><ul><li>Allow appropriate amount of time </li></ul><ul><ul><li>Simple multiple choice allow about one minute per question </li></ul></ul><ul><ul><li>Scenarios and essays require more time </li></ul></ul><ul><li>Clear complete directions </li></ul><ul><li>Be sure it has proper grammar and punctuation </li></ul>
    20. 20. Multiple Choice Tests <ul><li>Common method for formal and informal evaluation </li></ul><ul><li>Normally used for national and state certification examinations </li></ul><ul><ul><li>Easy to grade and be objective </li></ul></ul><ul><li>Test construction miscues </li></ul><ul><ul><li>Using previous questions information </li></ul></ul><ul><ul><li>Negatively worded stems </li></ul></ul><ul><ul><li>Fill in the blank in middle of stem </li></ul></ul><ul><ul><li>Using ‘all of the above” or “none of the above” </li></ul></ul>
    21. 21. True/False Tests <ul><li>Limited to two answers, no gray area </li></ul><ul><li>Difficult to construct in positive voice </li></ul><ul><li>Avoid always or never statements </li></ul><ul><li>Useful tool as a study guide </li></ul>
    22. 22. Matching Tests <ul><li>Works best with definitions and terms </li></ul><ul><li>Difficult to design </li></ul><ul><li>Cautious of multiple matches </li></ul><ul><li>Test directions must be clear </li></ul>
    23. 23. Completion Tests <ul><li>Fill in the blank </li></ul><ul><li>Statements must be clear as to intent of question </li></ul><ul><li>Need to be grammatically correct </li></ul><ul><li>Be aware of size of blank </li></ul><ul><li>Avoid having blank at beginning of sentence </li></ul>
    24. 24. Essay Tests <ul><li>May require long or short answer </li></ul><ul><li>Time consuming and difficult to grade </li></ul><ul><li>Recommended to grade in a group format </li></ul><ul><li>Rubric is a useful grading tool </li></ul><ul><li>Hand written exams must allow sufficient time </li></ul><ul><li>Shotgun approach.. As much information as possible in hopes of hitting the target </li></ul>
    25. 25. Heidi Goodrich, a rubrics expert, defines a rubric as &quot;a scoring tool that lists the criteria for a piece of work or 'what counts.'&quot; So a rubric for a multimedia project will list the things the student must have included to receive a certain score or rating. Rubrics help the student figure out how their project will be evaluated. Goodrich quotes a student who said he didn't much care for rubrics because &quot;if you get something wrong, your teacher can prove you knew what you were supposed to do.&quot; Generally rubrics specify the level of performance expected for several levels of quality. These levels of quality may be written as different ratings (e.g., Excellent, Good, Needs Improvement) or as numerical scores (e.g., 4, 3, 2, 1) which are then added up to form a total score which then is associated with a grade (e.g., A, B, C, etc). Many rubrics also specify the level of assistance (e.g., Independently, With Minimal Adult Help; With Extensive Adult Help) for each quality rating. Rubrics can help students and teachers define &quot;quality&quot;. Rubrics can also help students judge and revise their own work before handing in their assignments. What is a Rubric?
    26. 26. Oral Exams <ul><li>Requires verbal answers by students </li></ul><ul><li>Advantages </li></ul><ul><ul><li>Evaluate quick reaction of student </li></ul></ul><ul><ul><li>Assesses the student thought process </li></ul></ul><ul><li>Disadvantages </li></ul><ul><ul><li>Limited number of students examined at one time </li></ul></ul><ul><ul><li>Difficult to standardize </li></ul></ul><ul><ul><li>Time consuming and labor intensive </li></ul></ul><ul><ul><li>Unexpected distractions </li></ul></ul><ul><ul><li>Unfair emphasis on repeated mistakes </li></ul></ul>
    27. 27. Project Assignments <ul><li>Gets students working outside the class </li></ul><ul><li>In groups, helps develop people skills </li></ul><ul><li>Negatives </li></ul><ul><ul><li>Hard to standardize </li></ul></ul><ul><ul><li>Potential plagiarism </li></ul></ul><ul><ul><li>May measure only end product and not consider the process </li></ul></ul>
    28. 28. Practical Exams <ul><li>Situational </li></ul><ul><ul><li>Demonstration of a skill in the context of a scenario </li></ul></ul><ul><ul><li>FFII Exam for forcible entry </li></ul></ul><ul><li>Rote </li></ul><ul><ul><li>“Demonstration of steps of performing a skill” </li></ul></ul><ul><ul><li>FFII Exam donning of SCBA </li></ul></ul>
    29. 29. Practical Skills Evaluation <ul><li>Rote mechanical skills </li></ul><ul><li>No real world stresses </li></ul><ul><li>Situational skills test asks the student to think through a situation </li></ul>
    30. 30. Simple Skill Evaluation <ul><li>Define required skill </li></ul><ul><li>Determine efficiency expected </li></ul><ul><li>Student needs to know what is expected </li></ul><ul><li>Skills check lists are helpful </li></ul>
    31. 31. Performance Evaluations <ul><li>Determine and define </li></ul><ul><ul><li>Expected outcome </li></ul></ul><ul><li>Is performance or decision making more important than the situation? </li></ul><ul><li>How stressful a situation is student prepared to handle? </li></ul>SAFETY
    32. 32. Performance Evaluations <ul><li>Determine standards to be evaluated </li></ul><ul><li>Situation should </li></ul><ul><ul><li>Represent desired outcome </li></ul></ul><ul><ul><li>Realistic scenarios </li></ul></ul><ul><ul><li>Realistic environment </li></ul></ul><ul><ul><li>Realistic in the real world </li></ul></ul>SAFETY
    33. 33. Performance Evaluations <ul><li>Keep situation in perspective </li></ul><ul><ul><li>Remember safety </li></ul></ul><ul><ul><li>Remember legal ramifications </li></ul></ul><ul><li>List all activities to be completed </li></ul>
    34. 34. Reliability <ul><li>Multiple tests </li></ul><ul><ul><li>Compare results </li></ul></ul><ul><ul><li>The closer the scores, the more reliable </li></ul></ul><ul><li>Four questions </li></ul><ul><ul><li>Does it measure consistently on different occasions </li></ul></ul><ul><ul><li>Any influence of the environment </li></ul></ul><ul><ul><li>Any difference with administrators </li></ul></ul><ul><ul><li>Does it discriminate against anyone </li></ul></ul>
    35. 35. Test Resources <ul><li>Written examination resources: </li></ul><ul><ul><li>NFPA </li></ul></ul><ul><ul><li>Publishers’ test banks </li></ul></ul><ul><ul><li>Fire textbooks </li></ul></ul><ul><ul><li>Fire textbook instructor guides </li></ul></ul><ul><ul><li>Textbooks of practice certification examinations </li></ul></ul><ul><ul><li>On-line and computer-based practice certification tests </li></ul></ul><ul><ul><li>Fire Internet sites </li></ul></ul>
    36. 36. Test Resources <ul><li>Practical examination resources: </li></ul><ul><ul><li>NFPA </li></ul></ul><ul><ul><li>Fire Internet sites </li></ul></ul><ul><ul><li>Fire textbooks </li></ul></ul><ul><ul><li>Fire continuing education programs </li></ul></ul>
    37. 37. Test Resources <ul><li>Oral examination resources: </li></ul><ul><ul><li>NFPA </li></ul></ul><ul><ul><li>Fire Internet sites </li></ul></ul>
    38. 38. Conclusion <ul><li>As part of the learning process, the instructor needs a mechanism to evaluate the student’s learning and identify whether or not the student is achieving the objectives and goals of instruction. </li></ul>
    39. 39. Conclusion <ul><li>Evaluate </li></ul><ul><ul><li>Four Levels </li></ul></ul><ul><ul><li>Summative </li></ul></ul><ul><ul><li>Formative </li></ul></ul><ul><li>Tests </li></ul><ul><ul><li>Kinds of test </li></ul></ul><ul><ul><li>Differences among the various types </li></ul></ul><ul><ul><li>Sources for testing materials </li></ul></ul>
    40. 40. <ul><li>Multiple choice tests should avoid what? </li></ul><ul><ul><li>“Which choice is not correct” </li></ul></ul><ul><ul><li>“All of the choices are correct” </li></ul></ul><ul><ul><li>“None of the choices are correct” </li></ul></ul><ul><ul><li>Maybe I just broke the rule!!!!!! </li></ul></ul>
    41. 41. <ul><li>Which type of test is most commonly used for state or national certifications? </li></ul><ul><ul><li>True/False </li></ul></ul><ul><ul><li>Matching </li></ul></ul><ul><ul><li>Multiple choice </li></ul></ul><ul><ul><li>Fill in the blank </li></ul></ul>
    42. 42. <ul><li>Unexpected distractions would likely be a disadvantage in what type of testing? </li></ul><ul><ul><li>Rote </li></ul></ul><ul><ul><li>Situational </li></ul></ul><ul><ul><li>Written </li></ul></ul><ul><ul><li>Oral </li></ul></ul>
    43. 43. <ul><li>Of the four levels of evaluation, which one would likely look at job performance 6 months to a year down the road? </li></ul><ul><ul><li>Reaction </li></ul></ul><ul><ul><li>Business results </li></ul></ul><ul><ul><li>Learning </li></ul></ul><ul><ul><li>Transfer </li></ul></ul>