2. L I S T
• What is Assessment?
• Why assess?
• Types of Assessment
- Student evaluation by teaching and learning progress
- Student evaluation based on reference standards
- Example of criteria setting method
5. 측정, 총평, 평가
• Measurement
The process of assigning numbers to specify properties of things
• Assessment Overall rating
Comprehensive analysis of various information of the evaluation target by
mobilizing various methods how to evaluate
• Evaluation
Assigning value judgments to the results of measurements
6. 측정, 총평, 평가
• Measurement 측정
The process of assigning numbers to specify properties of things
• Assessment 총평, 사정
A method of comprehensively evaluating various information of the
evaluation target by using various methods
• Evaluation 평가
Assigning value judgments to the results of measurements
7. (Example) Thoracic Surgery Score
정의적
Affective
(attitudes)
인지적
Cognitive
(knowledge)
심동적
Psychomotor
(skills)
• Attendance (5%)
• Introspection (5%)
• Paper-and-pencil
test (65%)
• Quiz (5%)
• Oral exam (10%)
• OSCE(10%)
학생 자필시험 퀴즈 구술시험 OSCE 출석 자기성찰보고서 총점
A 50 5 8 2 5 5 75
B 55 5 8 3 5 5 81
C 45 4 6 1 4 3 63
D
C
• Measurement 측정
• Assessment 총평, 사정
• Evaluation 평가
8. Assessment vs. Evaluation
Assessment
Is ongoing process
Improves quality of learner
Individualized approach
Provides feedback
It is Process-oriented
It is the means
Evaluation
Provides closure
Judges quality of learner
Applied against standards
Shows shortfalls
It is Product-oriented
Is is the end
9. Assessment and evaluation are often used
interchangeably
Assessment describes the measurement of learning outcomes.
Evaluation describes the measurement of Curriculum, program,
institution, organizational variables, or policy.
10. Why assess?
• To quantify or measure candidate achievement
• To focus and motivate candidate
• To measure the effectiveness of teaching and learning
• Feedback for the student and teacher
• To predict candidate performance when in work-based simulations
(Bullock et al. 2008)
12. Paradigm Shift in Student Assessment
paradigm of the past new paradigm
evaluation
purpose
To evaluate and score learners
Evaluate whether learners have achieved achievement standards
(performances)
Learner: Improve learner's learning
Instructor: Improving Teaching Learning
evaluation
content
Cognitive Characteristics Emphasis on human, psychodynamic and emotional characteristics
Professor,
Learning
Department
Teaching, evaluation at the end
of learning
Seeing educational performance-teaching-learning-assessment as
practiced educational activities
By planning assessment before teaching/learning begins, it is possible
to systematically prepare goals and guidance for performance and
teaching/learning.
evaluation
type
Focus on summative evaluation
Paper-and-pencil evaluation
Formative and summative assessments focus on balance
Occasional evaluation, continuous evaluation, utilization of various
evaluation methods
Emphasis on performance assessment on authentic tasks
Instructor’s
role
Provide feedback for scoring
and correction
Before conducting the assessment, the student's achievement level
should be established in advance.
Student progress should be checked.
Provides feedback and provides guidance to improve student learning.
Use assessment results to improve teaching and learning.
13. Paradigm Shift in Student Assessment
(유효현, 2019)
Evaluation of learning outcomes
assessment for learning
Assessment as Learning
Outcome-oriented evaluation process-oriented assessment
cognitive assessment
Assessment of knowledge, skills, and
attitude domains
uniform evaluation Various evaluation methods
Curriculum-class-student evaluation
disconnection
Curriculum-class-student evaluation
cycle structure
14. Assessment of
learning
How much have
students learned
of at a particular
point in time?
(Summative)
Assessment for
learning
How can we use
assessment to
help students
learn more?
(Formative)
15. Identify
Learning Goals
And Outcomes
Provides
Learning
Opportunities
Reflect upon
The Result
Improve
teaching and
learning
Assess Student
Learning
Learning
Cycle
[Curriculum Management Regulations]
- Objection to student evaluation results and explanation
- Operation of systems that can provide opportunities (K.3.1.4)
- Conduct student evaluation to check the degree
of achievement of intended educational
outcomes (timely outcomes) (K3.2.2)
- Conduct student evaluation in
accordance with the intended
educational performance
(process performance/class
performance) and educational
method (K3.2.1)
- Conduct student assessment
to improve student learning
(K.3.2.3)
- Balanced formative and
summative evaluation (K3.2.4)
- Conduct student evaluation
including knowledge, skills,
and attitude (K3.1.2) using a
variety of evaluation methods
- Conduct student evaluation
(K3.1.3)
[Lecture plan, class plan]
17. Types of Assessment
Student evaluation by teaching and learning progress
• 진단평가 (Diagnostic Assessment)
• 형성평가 (Formative Assessment)
• 총괄평가 (Summative Assessment)
18. 진단평가 Diagnostic Assessment
• Justice
- Assessment that systematically observes, measures, and diagnoses learners' characteristics
before learning begins
• Purpose
- Identification of prior achievement of achievements that are prerequisites for learning
outcomes
- To adjust the appropriate teaching method or class content difficulty
• Evaluation factor
- Cognitive area: Evaluate the degree of pre-learning, analyze the presence and degree of
learning deficits
- Affective Area: Understanding learners' learning motivation, interest, attitude, etc.
19. 형성평가 Formative Assessment
• Justice
- Check the learner's learning level during teaching and learning and provide feedback
- Checking and improving the curriculum and teaching methods
• Role
- Pursuit of individualization of learning
- The role of feedback
- Diagnose learning difficulties
- Motivate learning
20. 형성평가 Formative Assessment
structured formative assessment Unstructured Formative Assessment
How to collect student
information
Homework, exercises, pre-
assessments, quizzes
Behavioral observation (facial expression,
gaze, gestures, etc.)
Verbal Interaction (Q&A)
classroom atmosphere
When to collect
As planned and predicted
Before and after class
voluntary, immediate grasp
Observation/monitoring during class
feedback
Conducted after scoring and
judging collected data
Instantly through interaction
class correction
Implementation after feedback
and interpretation of collected
data
Perform immediately after
understanding the collected data
21. 총괄평가 Summative Assessment
• Justice
- Evaluation that comprehensively determines the achievement of learning outcomes
after learning is completed
• purpose
- Determine whether or not the achievement level has been reached and assign ranks
- The role of granting qualifications
- Comparison between groups
- Deciding whether to implement the program
- Evaluation for decision making
22. 목적에 따른….
형성평가(Formative assessment) 총괄평가(Summative assessment)
• Identify learner’s learning level and performance
ability, provide feedback, and identify areas that
need learning in the future
• Professors work to improve the curriculum and
teaching methods by checking if they are
appropriate
• To measure the achievement of educational
objectives at the end of a learning period or course
and to determine progress in learning
• General public assessment (after unit, course,
semester or academic year)
• To qualify (promotion decisions, medical licensure,
eligibility for graduation)
• Learners self-check their academic achievement
• Providing learners with information about their
academic performance (feedback)
• Creating data that weighs learners' strengths and
weaknesses -> Assessment itself promotes
learning
• Check if learners have met the minimum
qualifications
• Determine whether a particular step can be passed,
whether you can proceed to the next step or need
to repeat the process
• Selection of outstanding students in the relevant
curriculum
23. In terms of feedback…
formative evaluation summative evaluation
• The learner's performance should be identified
and immediate feedback should follow. -> Self-
directed learning UP, Self-esteem UP, learning
motivation UP
• Early detection of learners with learning problems
• To provide feedback to learners, teachers, and
schools during teaching and learning progress
• Feedback: Disclosure of scores, counseling with
low-scoring students, solving test questions
• Officially no feedback.
(Payment degree feedback?)
• However, it can provide
feedback on learners' strengths
and weaknesses, and may
include elements of formative
assessment.
24. formative evaluation summative evaluation
• absolute evaluation
• Atypical, nonjudgmental, continuous, repetitive
• Part of the overall teaching/learning process
• It doesn't have to be reflected in your grades. small
percentage if reflected
• As a purely formative assessment, students do not
value or take the test seriously unless there is a
pass/fail decision and ranking announcement.
• Workplace-based assessment: mini-CEX, DOPS
• A pure summative assessment means that
when a student fails a test, there is no next
test.
• Summative evaluation should include
elements of formative evaluation.
• In real educational settings, summative and
formative assessments may be mixed.
• Mentoring, reflective, change over time
In terms of feedback…
25. Diagnostic evaluation, formative evaluation, summative evaluation
Diagnostic evaluation formative evaluation summative evaluation
Period Before teaching and learning
Teaching and learning in
progress
After teaching and learning
Purpose
Appropriate teaching and
learning input
Appropriateness of teaching and
learning progress
Achievement of learning outcomes
Determining the choice of educational
program
Assessment
methods
Informal and formal evaluation
Informal and formal evaluation
Occasional evaluation
Formal evaluation
Evaluation subject
Professor, educational contents
expert
Professor Curriculum expert, evaluation expert
Evaluation
standard
Governing reference Governing reference Norms or references
Evaluation
question
Questions that meet the
criterion
Questions that meet the
criterion
Standard Reference: Various Difficulties Criteria
Reference: Items Conforming to Criteria
26. “When the cook tastes the soup,
that’s formative assessment”
“When the Customer tastes the soup,
that’s summative assessment”
(brookhart, 1999)
(출처:구글)
27. Types of Assessment
Student evaluation by reference standards
• Normative reference evaluation
• Criterion reference evaluation
28. • Relative evaluation
• Assessment in which the score obtained by a learner or individual is judged by the
relative rank against the standards of the comparison group
• It is characterized by inducing learning motivation through competition
• Not interested in reaching the goal
Normative reference evaluation (Norm-referenced evaluation)
29. • Absolute evaluation
• In light of the criterion of what and how much the learner or individual knows weighing
evaluation
• Provides direct information on the achievement of learning outcomes
• Difficult to establish standards
준거참조평가 Criterion-referenced evaluation
30. 규준참조평가, 준거참조평가
Normative reference evaluation Criterion reference evaluation
emphasis relative order achievement in a specific area
educational beliefs Recognize individual differences complete learning
comparison target individual and individual Criteria and Performance
individual differences maximization try not to maximize
availability
Sorting, sorting, placing
Emphasis on administrative
functions
Qualification
Emphasis on instructional function
32. Why do we establish standards?
evaluation
Evaluation
results
judgment
• Illustration of taking the test (writing,
practical skills, certification evaluation)
• Evaluation
• Judgment on evaluation results
Credible,
Defensible,
Acceptable
How to help you decide
33. • Step 1: Choose a referencing method.
• Step 2: Select a criterion setter.
• Step 3: Describe the performance categories.
• Step 4: Train the standard setters.
• Step 5: Collect the evaluation/judgment results of the criterion setters.
• Step 6: Provide feedback on the evaluation/judgment results and facilitate discussion.
• Step 7: Evaluate the criteria-setting process.
• Step 8: Provide results, materiality and validity evidence to the final decision maker.
8 Steps to Setting Criteria
Yudkowsky,R., Park, Y.S & Doning, S.M(Eds.). (2019). Assessment in health professions education. Routledge
34. What is a good evaluation?
• It should be evaluated according to the intended purpose
(intent). (Purpose purpose)
• Evaluate consistently. (consistency of measurement)
36. To achieve the intended goal (validity)
To increase validity
① 의도한 내용을 선별해서(Content Validity),
② 잘 구성하고(Construct Validity),
③ 피험자가 의도한 반응을 하도록 유지하여(Response Validity),
④ 의도한 결과를 달성하는지 여부를 확인해야 한다.(Result Validity).
For the intended result to be meaningful
⑤ 공인된 결과(Concurrent Validity)나
⑥ 예측한 결과(Predictive Validity)가 유사한 결과를 보여야 한다.
38. In order to obtain consistent results
• Error factors that hinder consistent results must be controlled.
• Evaluation items should be developed to fit the item production principle so as not to cause
mistakes. (underline important words, delete duplicates, etc.)
• Make the evaluation environment similar (instructions, inspection locations, etc.);
• Test subjects should be supported to demonstrate their skills (motivation, test experience, etc.).
• If the rater evaluates, it should be done consistently through clear scoring criteria and
training.
Hallo effect
40. Angoff method(1971)
• The oldest and most used method
• Rrocedure
• All criterion setters discuss borderline students and agree on characteristics for
them.
• Agree on the quality of borderline students, with specific cases in mind.
• Individual criterion setters estimate the percentage a borderline student will
achieve on an item or performance item (0-100%).
• Record the rating results.
• Calculate the evaluation results to determine the passing score (usually sum or
average score)
41. Item check and rating
• Each item is reviewed and rated for each item.
• The criterion setter answers one of the following questions for each item.
• How many out of 100 borderline victims will do this correctly? (0~100%)
• What is the probability that one borderline subject will perform this item
correctly? (0~1.0)
42. Angoff 방법에 의한 준거설정 방법
문항 P 가중 점수 W WP
1 .5 2 1.0
2 .8 3 2.4
3 .7 1 .7
4 .9 1 .9
5 .1 2 .2
전문가마다 문항난이도에 대한 판단 달라지면 준거점수 달라짐
C = 3.0/5 C = 5.2/9
43. Things to note when rating questions
• Student Estimated Probability is “Will”, not “Should”
• How well students will perform, not how well they should perform.
• Items with a difference of more than 20% -> Criterion setters who
gave high and low scores should discuss the item
44. How to establish Jaeger/yes-no Angoff
conformance
<Procedure>
1. Experts analyze the contents
2. Virtually 1 person with minimum ability
3. Determine whether or not to answer each question correctly
4. (multiply if weighted)
5. Add up all the probabilities of answering correctly and use it as a
reference score
47. Take Home Message
• Plan assessments and conduct coursework
• Diagnostic evaluation, formative evaluation, summative evaluation
PLAN
• Absolute evaluation, relative evaluation
• application of various evaluation methods; Peer review, self review,
portfolio
48. Reports
자신이 교수자로서 수업을 가르칠 때 학습자를 평가할 방법을 작성하세요.
(Write how you, as an instructor, will assess learners as you teach a class.)
1) 진단평가, 형성평가, 총괄평가 PLAN
(Diagnostic Assessment, Formative Assessment, Summative Assessment PLAN)
2) 규준참조평가 or 준거참조평가 / 이유
(Norm-referenced evaluation or Criterion-referenced evaluation /reason)
3) 그 외 적용할 평가 방법; 동료평가, 자기평가, 포트폴리오
(Any other evaluation methods to be applied; Peer review, self review, portfolio)
