1740 Ch 9 PowerPoint

1. Fire and Emergency Services
Instructor
Ninth Edition
Chapter 9 — Testing and Evaluation

2. ‣ Discuss two ways to assess a
student’s success.
Learning Objective 1

3. ‣ Certification examinations are intended to determine
mastery of a subject or topic
‣ To pass these exams, students must have a complete
and thorough knowledge of the subject and the ability
to recall and apply the knowledge
‣ Assessing or evaluating student success may be
accomplished either through norm-referenced or
criterion-referenced assessment
Approaches to Student Assessment

4. Norm-Referenced Assessments
‣ Measures One Student Against Another
‣ End of Unit
‣ Exam given
‣ Exam scored
‣ Scores into grades
‣ Grade distribution arranged in bell curve

5. Norm-Referenced Assessments

6. Norm-Referenced Assessments
‣ Used for some evaluations in the fire and emergency
services, particularly in cases where clear
differentiations are made between top performers and
others
‣ For example, norm-referenced assessments are
commonly used during promotional testing and
determining ranking in recruit academies

7. Criterion-Referenced Assessments
‣ Compares Performance with Criteria
‣ Any performance is considered acceptable as long as it
meets the conditions stated in the criteria
‣ Primarily used where many students can succeed at
the same level, such as in a classroom or skills training
setting

8. Criterion-Referenced Assessments
‣ Test scores translate to either a passing or failing
grade, depending on whether the student has met or
failed to meet the criteria
‣ For example, if the criterion for performance on an
end-of-instruction exam is 70 percent, any grade of
70 or higher is a Pass, while anything below 70 is a
No Pass/Fail

9. Criterion-Referenced Assessments
‣ A similar approach may be used to evaluate
psychomotor skills, in which the student either passes or
fails a manipulative skills test
‣ Using a skills checklist and a timer, instructors can
assess whether students have completed the skill quickly
and accurately enough
‣ If they do, they pass, but if they do not, they fail; a
single step may be so critical, incorrect performance
may result in a failing grade

10. ‣ Identify the six classifications of tests.
Learning Objective 2

11. ‣ Both criterion-referenced and norm-referenced tests
can be divided into more specific classifications, based
on the reason for testing and the way the test is
administered
‣ No single test type is suitable for every situation
Test Classifications

12. Test Classifications: Two Types

13. ‣ Given at the beginning of instruction to establish a
student’s current level of knowledge, in order to
compare it to a subsequent summative evaluation
‣ When the tests’ contents are the same or similar in
both iterations, comparing the two scores measures
the amount of learning that has occurred
Purpose Classification:
Prescriptive (Pretest)

14. Purpose Classification:
Formative (Progress)
‣ Quizzes, pop tests, or question/answer periods that
are given throughout the course or session
‣ Typically measure improvement within a small scope
of the class progression and give the instructor and
students feedback on learning progress
‣ When measuring improvement, the test answers the
question: Is the student achieving the objectives?

15. Purpose Classification:
Summative (Comprehensive)
‣ Measure comprehensive knowledge and skills at the
end of a course, or of a major segment of the course
‣ Tests answer the question: Has the student achieved
the course objectives?
‣ Examples: Written and/or practical exams given at
midpoint or end of EMT, basic fire fighting, or
driver/operator courses

16. ‣ Based on how the test is administered, and includes
oral, written, and performance tests
‣ The Level II Instructor or AHJ SOPs may assign this
type of test based on the type of learning that is being
evaluated
‣ The Level I Instructor may be assigned to administer
any of these types of tests
Administration Classification

17. ‣ Student gives verbal answers to
spoken questions during a one-on-one
interaction with the instructor
‣ Not commonly used in the fire and
emergency services, but may be
useful under certain circumstances,
such as determining a student’s
understanding at the end of a lesson
Oral Tests

18. ‣ Administering and scoring oral tests should follow an
established scoring rubric
‣ Instructors must listen carefully to student responses in
order to prevent misunderstanding and be careful not to
make facial expressions that might confuse or mislead
students
‣ Should rarely be used as the sole means of determining
students’ terminal performance for a course or course
segment
Oral Tests

19. ‣ Written tests evaluate if students have met cognitive
learning objectives from the lesson plan
‣ Useful for measuring retention and understanding of
technical subjects, such as fire chemistry, laws and
ordinances, hydraulic principles, and medical protocols
Written Tests

20. Written Tests: Question Types

21. Administering Written Tests
‣ Reading test
aloud
‣ Providing
students a sheet
of questions
‣ Electronic media

22. Performance Tests
‣ Measure students’ ability to perform skills and tasks
as they would on the job, based on standardized
criteria and performance objectives
‣ Students are tested on their current ability to
complete the skill

23. Performance Tests:
Administration Preparation
‣ Determine the
materials, tools, or
equipment that
students must have
available to perform
the skills or
activities and that
they are available
and in working
order

24. Performance Tests:
Administration Preparation
‣ Prepare a skill checklist, along with appropriate time
limits for each of the steps necessary to perform the
skill
‣ Determine the number of test evaluators needed to
observe and measure (by checklist) the performances
of students
‣ Review the operation of unfamiliar equipment with all
instructors and test evaluators

25. ‣ Discuss test bias.
Learning Objective 3

26. ‣ A test is considered biased when members of different
groups (age, cultural and ethnic background, gender)
with the same ability level routinely score differently on
a test
‣ Bias can be difficult to prove because tests are designed
to discriminate between those that know the information
in the test and those that do not
‣ Bias is an indication that the test puts one particular
group at a disadvantage to another
Test Bias

27. Test Bias: Test Qualities

28. Test Bias: Legal Liability for
Employment or Selection Tests
‣ In the United States, the Equal Employment
Opportunity Commission (EEOC) has the responsibility
of investigating any bias on employment tests
‣ In order to avoid any legal liability, Level I Instructors
should follow any scoring rubrics and answer keys that
have been provided to them and teach to the learning
objectives in provided lesson plans

29. ‣ Describe the process of test
administration.
Learning Objective 4

30. Test Administration: Before the Test

31. Test Administration: Day of Test

32. ‣ Ask the testing organization or AHJ to determine whether
there are any specific instructions or protocols for
administering the test
‣ Arrive at the testing location an appropriate amount of
time before testing begins
‣ Maintain test security at all times
‣ Rearrange classroom seating when necessary so that it is
conducive for taking written tests
Administering Written Tests:
Guidelines Before the Test

33. ‣ Eliminate loud talking or noises outside the testing room
‣ Eliminate all potential distractions within the room
‣ Have students place all backpacks, purses, books, and
other unnecessary items out of the way or in an
assigned area
‣ Ensure that all electronic devices are turned off and
secured
Administering Written Tests:
Guidelines Before the Test

34. ‣ Inform students of expectations, including misconduct
rules
‣ Make sure that students do not bring written or
electronic notes into the testing room, unless these are
specifically approved by the examination guidelines
‣ Remind students to follow instructions on the answer
sheet correctly
Administering Written Tests:
Guidelines Before the Test

35. ‣ Watch for signs of academic misconduct
‣ Ensure that the environment remains quiet and safe
for students
‣ If students are allowed to leave the testing area to use
the restroom, refreshment area, or other rehabilitation
facilities, make sure they do not take the test
instrument or paper with them (verify with AHJ prior to
test for policy)
Administering Written Tests:
Guidelines During the Test

36. ‣ Supply extra paper, writing implements, and answer
sheets
‣ If possible, have testing aides on hand to assist with
difficult students or situations, or to perform tasks
such as handing out or collecting tests and evaluation
forms
Administering Written Tests:
Guidelines During the Test

37. ‣ Do not allow students to change their answers once
answer sheets have been submitted
‣ Review the test (at an appropriate time) with students
to clarify any objectives they may not have understood
‣ Maintain the security of tests and answer sheets
Administering Written Tests:
Guidelines After the Test

38. ‣ Return test materials to the proper authority if another
instructor will be scoring and grading them
‣ Identify answer sheets with identical incorrect answers
or sequences of answers on multiple test sheets
Administering Written Tests:
Guidelines After the Test

39. ‣ Ensure that the subject matter of the test matches the
subject matter that is being tested
‣ Provide students with adequate practice time during
class time leading up to the test date
‣ Provide rehabilitation facilities, such as restrooms or
refreshment areas
Administering Performance Tests:
Guidelines Before the Test

40. ‣ Ensure that tests are not biased
through wording, timing, or
unattainable criteria
‣ Include all test administration
rules in the test instructions
‣ Read the instructions aloud to
the students exactly as they are
written
Administering Performance Tests:
Guidelines Before the Test

41. Administering Performance Tests:
Guidelines Before the Test
‣ Provide students with the time limits for each test and
clearly state the times for emphasis
‣ Explain the purpose of each test
‣ Encourage students to ask questions if they do not
understand something

42. Administering Performance Tests:
Guidelines During the Test
‣ Give the test to each student in exactly the same
manner
‣ Record students’ scores on performance checklists as
each one takes the test; do not wait until the end of
the testing period to record scores

43. Administering Performance Tests:
Guidelines After the Test
‣ Keep all test scores confidential

44. ‣ Explain the different processes of
scoring and grading tests.
Learning Objective 5

45. Test Scoring
‣ Scoring is the act of identifying which answers are
right and which are wrong
‣ Scoring should always be objective and based on
established criteria
‣ Grading is the act of assigning a value to the score

46. ‣ Regardless of how tests are scored, the scoring
method should always be indicated on the exam
‣ Composite scoring used for criterion-referenced tests is
based on a point system that encompasses an entire
course
‣ Written tests can be scored either manually or using
an electronic scoring device
Scoring Written Tests

47. ‣ Score the same question on all of the tests before
proceeding to the next question on short-answer or
essay exams
‣ Read through several responses before scoring essay
or short-answer tests
‣ Shuffle the papers before scoring the next question
after scoring one set of questions on all the tests
Manually Scoring Written Tests:
Guidelines

48. ‣ Do NOT attempt to
score large numbers of
tests at one time
‣ Use a checklist to
identify the key points
that should be
addressed in each
essay or short-answer
response
Manually Scoring Written Tests:
Guidelines
Courtesy
of
Oklahoma
State
Fire
Service
Training

49. ‣ Add comments to essay or short-answer questions to
indicate what is missing or congratulate the student on
exceptionally good work
‣ Using lines or arrows, indicate which comments refer
to which words or portions of the answer
‣ Inform students about the meaning of scoring marks
that are used in essay or short-answer question tests
Manually Scoring Written Tests:
Guidelines

50. ‣ Classes in a distance-learning format take much more
preparation time for educators than the same classes
in a face-to-face setting
‣ Student improvement takes longer because feedback
is not as immediate; instructors have to provide
effective and timely feedback
‣ Scoring criteria, marking notations, and instructor
comments must be clear, concise, and constructive
Scoring Written Tests:
Factors for Distance-Learning Courses

51. ‣ Scoring oral tests depends on the purpose of the test,
such as promotional exams, and the type of questions
asked
‣ The instructor should present the questions as they
have been provided on the test and then follow the
scoring guidelines provided
‣ The scoring rubric for an oral test is often very similar
to performance tests
Scoring Oral Tests

52. ‣ Scoring can be very subjective so instructors should
closely follow scoring sheet guidelines
‣ These list the tasks students must perform to complete
the skill, and assign a point value to each task
‣ When the student completes the task, the evaluator
gives the student credit for that task — In some cases,
the instructor may give partial credit
Scoring Performance (Psychomotor)
Tests

53. ‣ Tasks that directly relate to life or safety cannot be
skipped or performed improperly
‣ When possible, use multiple instructors or test evaluators
to observe each student during the test
‣ Students must be given a clearly stated set of objectives
and the scoring criteria
‣ Instructors should provide immediate feedback while
observing the project, especially when safety is a concern
Scoring Performance (Psychomotor)
Tests

54. ‣ The total number of points earned is divided by the total
number of points possible to arrive at a percentage grade;
method usually applies to written tests but may be
applicable to performance tests
‣ Some situations may require students to get every test
item correct in order to pass; these tests should include
questions that attentive students will answer easily as
students are disadvantaged when every question on such
a test has a high level of difficulty
Grading Fire and Emergency Services
Tests

55. ‣ On objective tests such as multiple choice tests, grading
bias is very easy to prove and is considered unethical
conduct
‣ Instructors must be careful to follow scoring sheets on
objective tests
‣ Subjective tests are more challenging for evaluators to
grade because the evaluation is open to some
interpretation; evaluators may inadvertently influence
grades for students that they favor
Grading Bias: Assigning Grades
Based on Factors Other Than Scores

56. ‣ Instructors who evaluate subjective tests should follow
provided scoring rubrics and grading checklists
‣ Should instructors feel that they are pressured to show
bias toward some students, or perceive that another
instructor is showing bias, they should report that opinion
to the AHJ for further evaluation
Grading Bias: Assigning Grades
Based on Factors Other Than Scores

57. ‣ Explain the importance of accurate
grade reporting.
Learning Objective 6

58. ‣ Scores must be recorded and reported in accordance
with local procedures or AHJ policies
‣ Because fire and emergency services students are
graded against a set of specific criteria and not against
each other, grades are recorded in individual student
records and used as feedback for students
‣ Care must be taken to accurately record the grades in
student records
Grade Reporting

59. ‣ An instructor should follow the provided scoring
guidelines to determine a final grade for the course
‣ Students who fail to achieve the minimum required
grade should not be recognized as having completed
the training
Grade Reporting

60. ‣ Testing records are private and confidential, so only
instructors, training division administrators, and the
relevant student should have access to them
‣ Testing results should be retained in individual student
files for the period of time that the AHJ requires and
kept private in accordance with specific
department/AHJ policies and applicable laws
Grade Reporting

61. ‣ The organization must be able to prove whether a
student has met the minimum requirements to
effectively perform a duty or task, especially where
certification is the final intended outcome of training
‣ The test results for all participants in a course provide
the training division or administration with an idea of
the effectiveness of the course or curriculum
Grade Reporting:
Necessary for Two Reasons

62. ‣ Identify guidelines to reduce
academic misconduct.
Learning Objective 7

63. ‣ Essential for all
types of tests
‣ Important to
prevent academic
misconduct
Test Security

64. ‣ Follow security measures when writing, duplicating,
and storing test materials
‣ Never rely solely on questions published in the
textbook or study guide
‣ Revise test questions regularly
Test Security: Guidelines

65. ‣ Use secure data storage systems to prevent
unauthorized access to tests or grades
‣ Require students to use assigned usernames in order
to limit access to computer-administered tests
‣ Number all test sheets, booklets, and answer sheets,
and take inventory after each use
Test Security: Guidelines

66. ‣ Use two or more versions of a single exam to prevent
students from copying from each other during the test
‣ Store all old test sheets in a secure location
‣ Destroy outdated testing materials
Test Security: Guidelines

67. ‣ Discuss the benefits of instructor
feedback on student outcomes.
Learning Objective 8

68. ‣ Level I Instructors must not only possess the ability to
accurately record test results, they must also have the
ability to interpret the subsequent data in order to
gauge the students’ levels of knowledge and
understanding of the course topics
‣ Instructors are encouraged to formally evaluate the
test results to determine which pieces of information
should be reinforced
Evaluation Feedback

69. ‣ Meet individually
with students to
discuss the test
‣ Review incorrect
answers with the
entire class
Conducting Evaluation Feedback