Assessment planning and quality assurence by Dr. Debjani Sengupta

1. ASSESSMENT PLANNING
AND QUALITY ASSURENCE
RBCW
2nd -4th September
Calcutta National Medical College

2. 1. Necesssity of assessment 2. Essentials in quality
maintenance of assessments
3. Different type of assessment for
theory and practical
4. Hands on demonstration of
formulating assessment tool
Topics covered in
this session

3. Why assessment
• Assessment has a powerful positive steering effect on learning and the
curriculum.
• Motivator of student learning.
• Measuring progress in core knowledge and competencies,
• Assessment may be for elimination,
• Assessment may be for maintaining standard,
• For adequate feedback

4. • In planning and designing assessments, it is essential to recognize the
stakes involved in it.
• The higher the stake, the greater the implications of the outcome of
the assessment
• T he more sophisticated the assessment strategies, the more
appropriate they become for feedback and learning.
• Understanding the necessity of FORMATIVE and SUMMATIVE
assessment

5. Assessement necessity
• In keeping with Millers pyramid the ultimate aim of teaching is to
develop a person competent to deliver the skills acquired /
knowledge gathered to be used gainfully in the society,
• To ensure the the training has reached this optimum benchmark
wherein the subject can be considered to be a professional
,assessment is an essential tool which is quite a high stake for the
society
• Likewise assessment methods should be .......

6. Assessment of medical students...things to consider....
• Formulating an appropriate assessment technique
• Having decided on the mode of assessment decide on type of assessment
• Understand the pro and cons of each technique
• Target optimum relation between feasibility and quality
• Ensure scope for feedback
• Eliminate sources of bias
• Establish gold standard for measurability for student to realize his/ her ultimate goals

7. Essentials of assessment quality maintenance
methods
• Valid for the domain under consideration
• Reliable : yield same result on repeated use
• Precise : address the required domain clearly
• Unambiguous: clear conception of outcome to be measured

8. • Free of scope of subjective error: should be fair
• Measurable against a gold standard
• Should provide scope for feedback as well as self assessment by student
• Different methods may be used
• Feasible

9. • Validity generally refers to how accurately a conclusion,
measurement, or concept corresponds to what is being tested.
• Reliability refers to consistency and repeatability
An assessment that is highly reliable is not necessarily
valid. However, for an assessment to be valid, it must also be reliable.

10. • Often there is a trade off between these elements , some
components are compromised, some are facilitated
• But overall these should be the guiding principles

11. A little more on reliability
• Commonly we use short or spot cases for assessment
• Ever considered why 2-3 short cases and as many as 5 spot cases
• Where as long case is just one
???

12. • With one case the student gets one chance, may be unlucky, gets a
low score---------------- deem as poor performance
• Same student gets another case , common well ,prepared gets a high
score------------------deem as good performance
• Two different results for the same student?
• Why?
The assessment method LONG CASE has poor reliability

13. • How can we improve reliability?
• Use more number of say 10 long caseses---------------- performs
moderately in all--------------deem moderate performance
A good reliability of a method of spot case or short case or long case
comes only if number of cases per student is about 10

14. • But seriously who will go through that -----------about 50 cases per
batch of 5 MD
• Examination will stretch over 10 days!!!!
• No examiner will be available
• So what to do?

15. • We compromise on reliability , (valididity is ok as it measures clinical
diagnostic skill)
• We go for single long case very low reliability
• Spot and short maybe more than one as less time consuming( slightly
better reliability
• We accept what is feasible.
• Trade off -----------
Nothing in life is free !

17. Popular methods of assessment
• Theory or written
 MCQ(multiple choice
questions)
 Essay type questions
 Modified essay type questions
 Short answer type questions
• Practical
objective structured clinical
examination
Long case
Directly observed procedural skills
Mini Clinical examination
360degree evaluation

20. Essay type/ Modified essay tyype

22. Open ended or Close ended

23. Comparison of other methods
Short answer type
• Valid for information
based cognitive
component
• Reliablity low
• Ambiguity may creep
• Subjective error possible
unless answer key is
developed
• Highly feasible for our
settings
Essay type
• All points are applicable
here definitely at a
higher degree
Modified essay type
• Valid for information
based cognitive
component
• Reliability low
• More precise than EQ
• Less ambiguous than EQ
• Scope of subjective
error
• Definitely feasible

30. Multiple choice questions
• Subjective
bias
minimized
• Gold
standard
set
• Reliable as
yields same
outcome
• Valid for
information
based
Can be
tailored for
different
domains
Process
requires time
and effort
Feasible in
most settings
Unambigous
generally

31. Things to remember when desiging an
assessment tool
• Select appropriate
methods
• Ensure reliability by
developing answer key/
gold standard
Avoid non specific
verbs like describe,
evaluate, assess
• To avoid bias ascertain
single person to check
same answer
• Ascertain core
compentcies which should
cover 60% of questions
Avoid ambiguity
• Ensure feedback
• Set a benchmark for
students self assessment
goals
Ensure feasibility

32. When, where, why, what
Necessity of assessment Method of good validity and reliability
Elimination test MCQ
Synthesis of knowledge ( answer key
prepared)
MEQ, SQ
Clinical skill OSCE
Procedural skill DOPS
Formative assessment Any suitable method
Summative assessment Preferably combination of different tecniques

33. Assessment for practical
• Long case
• Objective structured clinical examination
• Direct observation of procedural skills
• Multisource 360
• Checklist evaluation not much in use

34. Long case
• Traditionally used
• Process involves- students examines the pt 30-45 min- presents- unstructured oral
examination
• Logistics demand a single case per student
• Owing to different case the result is not generalizable/ standardized : subjective bias
• Scoring is varied depending on case – no uniformity: not fair
• Reliability can be increased if 8-10 such assessment can be done per student- not
feasible
• Abandoned in western world

35. OSCE
• Candidates rotate through a series of timed stations.
• All candidates move from station to station in sequence on thesame
schedule.
• OSCEs have used standardized patients as the primary assessment
tool

36. OSCE cont..
• The observing faculty or tutor or patient uses either a checklist of
specific behaviours or a global rating form to evaluate the student’s
performance
• This format provides a standardized means to assesse of clinical skills
• A minimum of 10 stations, which the student usually visits overthe
course of 3–4 hours, gives good reliability
• But labour intensive and expensive- feasibility limited in our settings

37. Work place based assessment
Mini_Clinical evaluation exercise
• In the mCEX, a faculty member
observes a trainee–patient
encounterin any healthcare setting
• (about 15 minutes) and focused. The
trainee is expected to conducta
focused history-taking and/or physical
examination during this time and then
provide the assessor with a diagnosis
and treatmentplan. The performance
is scored using a structured form and
educational feedback is provided
Direct observationof procedural skills
• DOPS is a variation on the mCEX in
which the assessor observeswhile the
trainee is doing a procedure (e.g.
venepuncture, givingan injection,
etc.), rates the performance and then
provides feedback

38. For formative assessment
• Formative assessment are year round.
• Compromise on validity and reliability may occur
• Time consuming laborious
• Try to tailor inbuilt formative assessment component in curriculum
designing or class planning
• Provide measurable assessment goals for student motivation

39. For summative assessment to improve
validity and reliability.....
• Ensure questions are based on taught curricula
• Ensure questions are based on standards
• Allow students to demonstrate knowledge/skills in multiple ways
• Ensure a variety of item types (multiple-choice, constructed
response)

40. • Ask questions at varying Depth of Knowledge levels
• Ensure accurate test administration
• Include items that address the full range of standards
• Include multiple items that assess the same standard
• Review scorer reliability, when necessary

41. Take home message