Know how of question bank developmentPresentation Transcript
Question Bank Development: A Probe into its Theoretical and Practical Connotation Faculty Professional Development Programme (FPDP) of UOU Dr. Rajani Ranjan Singh School of Education Uttarakhand Open University
Why we move to Question ?
Question is the index of cognitive Ignition and hence the reflection of Personality as a whole.
It is the question which changes the way of perceiving the world and consequently the course of your life.
"To make my contribution I must Ask Questions, Speak the Truth, Think for Myself, and Care for My Soul,“
"It is better to be a human being dissatisfied than a pig satisfied; better to be Socrates dissatisfied than a fool satisfied. And if the fool, or the pig, are of a different opinion, it is because they only know their own side of the question," says John Stuart Mill in Utilitarianism , Ch.2
Question as catalyst begins the cognitive turbulance.
Take a pause & Think
Question development or item writing is a work of great contemplation which consists a good planning and better execution for sound and objective results concerning with the prognosis and diagnosis of the human potential. So, if you want to have objective result of measurement and evaluation regarding the abilities of an individual, then you must know the technicality of item engineering which is the vital part of test construction and its standardization.
Assumption of Measurement, Assessment & Evaluation
Whatever exists at all, exists in some amount E.L.Thorndike,1918.
It provides the criteria of measurement & assessment.
Anything that exists in amount can be measured or assessed, Mc Call,1939.
That is why, Educational Achievement can be assessed.
Glossary what to be understood before the session
Educands’ Behaviour and their types
Cognitive, Affective and Conative
Objectives and their types
Educational and Instructional Objectives
Taxonomy of Instructional Objectives As Per B.S.Bloom
Discrete bits of information CONCRETE
More complex, organized knowledge
Classifications, categories, principles
Steps to take, how to do something
Determining when to do what
Personalize understanding for the user
Strategic and contextual ABSTRACT
The Cognitive Process Domain The Knowledge Dimension Remember Understand Apply Analyze Evaluate Create Factual Conceptual Procedural Meta Cognitive
How Might This Look? Map out Objectives Instructional activities Assessments Activities and tests should align in order to meet objectives
Question and its relationship with Normal Probability Curve Qu. Battle of Haldighati was fought between Mughals and ………….
Question and its relationship with Normal Probability Curve Qu. The name of the first president of India was……….
Question and its relationship with Normal Probability Curve Q. The first president of India belonged to…………….district of Bihar.
Connotation of Question Bank
Question Bank – A series of questions of different types, on the basis of which some information is sought.
Question Development is a standardized procedure to measure quantitatively or qualitatively one or more than one aspect of traits, by means of sample of verbal or nonverbal behaviour.
A question bank in the form of a test is an organized succession of stimuli designed to measure quantitatively or evaluate qualitatively of some mental processes, traits or characteristics.
Types of Test
Objective test and subjective test
Speed test and power test
Individual test and group test
CRT and NRT
Verbal, Nonverbal, and Performance test
Achievement, Personality, Aptitude, Attitude, and Intelligence test
Diagnostic and Prognostic test
Formative test and summative test
Standardized and Teacher-made test
Tool and Techniques of Educational Assessment
The assessment process produces the data for cognitive, affective and psychomotor objectives. All techniques of assessment can be broadly classified into two categories: (a) Quantitative and (b) Qualitative
(A) The Quantitative techniques are mainly used in educational assessment. These are highly reliable and valid They can be classified into three types (i) Oral (ii) Written and (iii) Practical.
(B) The Qualitative techniques are used for continuous and comprehensive assessment. These techniques are subjective and less reliable, but they are used for assessing the affective objectives. These techniques are like cumulative records, anecdotal records, observation techniques and check list and rating scale.
Tools of the Assessment of Educational Achievement
Diagnostic Test - It aims at identification of academic strengths and weaknesses in a particular content area. It is also helpful in isolation of children with mental handicap as different from other normal children. In a way, diagnostic test is test for identification. It follows a normative models to make comparative evaluations of individuals.
Norm -referenced Test - It is used to determine how much overall knowledge of some subject a particular pupil has achieved. The basic report for a norm referenced test is a count of the number of test questions that are answered correctly. The ‘norm’ in a norm referenced test is the achievement of some specific groups of students on the test. An excellence or deficiency of a particular student’s achievement in learning is judged by that student’s standing among those in the specified group.
Tools of the Assessment of Educational Achievement
Criterion- referenced Test - It is used to determine the certain specified objectives of instruction a particular pupil has attained. The ‘criterion’ in a criterion referenced test indicates the instructional objectives of a content. In contrast to NRT this approach is not concerned with comparison of individuals with a norm or standard. The point of reference is to an absolute standard within an individual rather than a population norm. In target populations, especially individuals with mental handicap, the individual differences are so great that group comparisons are futile. It is also classified as Domain and Objective referenced test as per the type of interpretations
Connotation of an Item
Item- an item is defined as “ a single question or task that is not often broken down into any smaller units.
An arithmetical problem, manipulative task, a mechanical puzzle may be an item.
Lowest common denominator of a test, which is scored is an item.
Structural and functional unit of a test.
General characteristics of an item
Should measure the significant aspect of the subject matter
Should not encourage the guesswork by the subject
Easy in deciphering the intended idea
Incorporation of Independent items in the test
Technical characteristics of an item
Incorporation of sufficient items to make the test reliable
High discriminating power
General Guidelines for Item Writing
Clarity in writing items
Absence of nonfunctional words
Absence of irrelevant accuracies
Moderate difficulty value
Avoidance of the use of stereotyped words
Avoidance of clues
Interlocking or interdependent items should not be there
Item Analysis: A good question or item discriminates between high achiever and low achiever hence the filtration of items must be there while incorporating into question bank or in a test
To analyze constructed test item in terms of effectiveness of the distractors, difficulty index and discrimination index
To summarize the item analysis procedure in tabular form using the evaluation code
Determine the Upper Group (U) and Lower group (L)
Upper group is the 27% of the 50 test papers from the highest score, i.e., 14 test papers from the highest score
Lower group is the 27% of the 50 test papers from the lowest score, i.e., 14 test papers from the lowest score
Estimate Index of Difficulty
I diff = (U + L)/ (N u + N L ) where
U and L are the number of correct
responses in the upper and lower
N u and N L are the number of test
papers from the upper and lower
Index of Discrimination
It might be positive
It might be negative
It might be zero
Formula of DI : Ru/Nu-Rl/Nl
Guessing formula :S=R-W/K-1
Good distractor- More examinees of the lower group
Poor distractor- More examinees of the higher group
Correct response should have positive discrimination index
Distractor should have negative discrimination index
Determine the Index of Difficulty of an Item Interpretation of Index of Difficulty (Ebel, 1965) A test item is: If it has a difficulty index of: Very Easy 0.91 and above Easy 0.76 to 0.90 Optimum Difficulty 0.26 to 0.75 Difficult 0.11 to 0.25 Very Difficult 0.10 and below
Evaluation Code for Item Analysis Item Category is Fair Options Acceptable Discrimination level High or Moderate Difficulty level Optimum or Easy or Difficult Recommendation – Retain the test item
Questions to Ask yourself in Designing a Test
What objectives will (should) I be testing?
What types of items will be included in the test?
How long will the test be in terms of time and number of items?
How much will each objective be worth in terms of weighting and number of items?
Planning a Test
First step: Outline learning objectives or major concepts to be covered by the test
Test should be representative of objectives and material covered
Major student complaint: Tests don’t fairly cover the material that was supposed to be canvassed on the test.
Planning a Test
Second Step: Create a test blueprint
Third Step: Create questions based on blueprint
Match the question type with the appropriate level of learning
Fourth Step: For each check on the blueprint, jot down (might use 3x5 cards) 3-4 alternative question on ideas and item types which will get at the same objective
Fifth Step: Organize questions and/or ideas by item types
Planning a Test
Sixth Step: Eliminate similar questions
Seventh Step: Walk away from this for a couple of days
Eighth Step: Reread all of the items – try doing this from the standpoint of a student
Planning a Test
Ninth Step: Organize questions logically
Tenth Step: Time yourself actually taking the test and then multiply that by about 4 depending on the level of students
Eleventh Step: Analyze the results (item analyses)
Translating Course Objectives/Competencies into Test Items
Specification table- what was taught/weight areas to be tested
Creating a Test Blueprint (see handout)
Blueprint- this is the test plan, i.e., which questions test what concept
Plotting the objectives/competencies against some hierarchy representing levels of cognitive difficulty or depth of processing
Blue Print of a Question Paper Objectives Knowledge Understanding Application Total Form of Ques MC VSA SA MC VSA SA MC VSA SA No.Syst. 1(1) 3(1) 4(2) + & - 2(1) 2(2) 2(2) 4(1) 10(6) x 1(1) 1(1) 2(1) 2(2) 4(1) 10(6) Div. 2(1) 1(1) 1(1) 4(2) 8(5) Frac. 1(1) 2(1) 1(1) 4(3) Measures 1(1) 3(1) 1(1) 1(1) 6(4) Time 1(1) 2(1) 1(1) 4(3) Geo 1(1) 1(1) 2(1) 4(3) Sub Total 3(3) 3(2) 7(4) 5(5) 2(1) 8(4) 6(5) 4(4) 12(4) 50(32) Total 50
Design of a Question Paper
Assigning weightage to objectives
Objective (K+ U+A+A+S+E) with %age Marks
Assigning weightage to content areas
Topics with Marks
Assigning weightage to different types of Questions
Type of Ques./No. of Ques./Marks allotted/Time
Assigning weightage to Difficulty Level
Easy (20%)+Average (50%)+Difficult (25%)
Question Arrangement on a Test
Group by question type
Common instructions will save reading time
Limit the number of times students have to change frame of reference
Patterns on test must be logical
Arrange from a content standpoint
Keep similar concepts together
Group by difficulty (easy to hard)
Selecting the Right Type of evaluation
How do you know what type of question to use and when?
It depends on the skill you are testing.
Evaluation should always match as closely as possible the actual activity you’re teaching.
Examples: Teaching Speech, should evaluate an oral speech
If testing ability to write in Spanish, better give an essay.
Testing reading –MC, TF
Wouldn’t use MC to test creative writing
Question Types verses Cognitive Levels of Learning Knowledge Comprehension Application Analysis Synthesis Evaluation Multiple Choice (MC) True/False (TF) Matching Completion Short Answer MC Short Answer Problems Essay Performance MC Short Answer Essay
Constructing the Test
Types of Test Questions:
Fill-In, Completion or Short-Answer Items
Multiple Choice Items
Extremely versatile-can measure the higher level mental processes (application, analysis, synthesis and evaluation)
A compromise between a short answer/essay and T/F item
Can cover a wide range of content can be sampled by one test
Difficult to construct plausible alternative responses
Types of Multiple Choice Items
Four Basic Types
Incomplete Statement Type
Right Answer Type
Best Answer Type
Which Type is Best?
Question Type vs. Incomplete Statement
Right Answer vs. Best Answer Type
Multiple Choice Items
Writing the stem first:
Be sure the stem asks a clear question
Stems phrased as questions are usually easier to write
Stems should not contain a lot of irrelevant info.
Appropriate reading level/terms
Be sure the stem is grammatically correct
Avoid negatively stated stems
Multiple Choice Items
Writing the correct response
Use same terms/reading level
Avoid too many qualifiers
Assign a random position in the answer sequence
Read the stem and correct response together
Generate the distractors/alternative responses
Multiple Choice Items
Other Tips for Constructing MC Items:
Items should have 3-4 alternatives.
Stem should present a single, clearly formulated problem
Simple, understandable, exclude extraneous words from both stem and alternatives
Include in the stem any word that are repeated in each response
Avoid all of the above (can answer based on partial information)
Avoid none of the above
Multiple Choice Items
Alternative responses/distractors should be plausible and as homogeneous as possible
Response alternatives should not overlap
Two synonymous terms (arithmetic average/mean)
Avoid double negatives
None of the following are part of the brain except which one?
Emphasize negative wording
Each item should be independent of other items in the test
Information in the stem of one item should NOT help answer another item.
True-False Test Items
Best suited for testing 3 kinds of info.:
Knowledge level learning
Understanding of misconceptions
When there are two logical responses
Sample a large amount of learning per unit of student testing time
Tends to be very easy
50-50 chance of guessing
Tends to be low in reliability
Tips for Constructing True/False Items
Tips for constructing True-False Items
Avoid double negatives
Avoid long or complex sentences
Specific determiners (always, never, only, etc.) should be used with caution
Include only one central idea in each statement
Avoid emphasizing the trivial
Exact quantitative (two, three, four) language is better than qualitative (some, few, many)
Avoid a pattern of answers
Objective Test Item Analyses
Evaluating the Effectiveness of Items..
Scientific way to improve the quality of tests and test items
Identify poorly written items which mislead students
Growth, Motivation, Maturation, Development (--------)
Saturn, Mars, earth, Pluto ( ---------------)
Directions: The following item consists of two statements one labelled as ‘Assertion (A)’ and the other as ‘Reason ®’. You are to examine these two statements carefully and select the correct answer using the codes given below:
Both A and R are individually true and R is the correct explanation of A
Both A and R are individually true and R is not the correct explanation of A
A is true but R is false
A is false but R is true
Assertion (A) : Variance of X is always greater than the standard deviation of X
Reason ® : Variance is square of standard deviation
Analogy type item:
There are two situations. The first situation is complete and second is incomplete. It should be completed on the basis of first situation.
1.Socrates : Aristocle :: Aristotle :
2. Mansarovar: Premchand:: Saket :
3.Kolkata: Hooghly:: Hydearbad:
Simple Recall Items:
1.The term Blog has been derived from the word………..
2.The expanded form of the term ‘Wiki’ is……………..
Two Types: (Question and Incomplete Statement)
Easy to construct
Excellent format for measuring who, what, when, and where info.
Guessing in minimized
Student must know the material- rather than simply recognize the answer
Grading can be time consuming
More than one answer can be correct
Short Answer Items
Tips for Constructing Short Answer Items
Better to supply the term and require a definition
For numerical answers, indicate the degree of precision expected and the units in which they are to be expressed.
Use direct questions rather than incomplete statements
Try to phrase items so that there is only one possible correct response
When incomplete statements are used, do not use more than one blank within an item.
1. Enumerate the five functions of measurement.
2. Write down the five examples of learning.
3. Write any five features of Indifference Curve.
4. Write five differences between language and literature.
5. Define term sonnet and differentiate with satire.
6. Indicate the levels or scales of measurement.
7. Enumerate the three technical charecteristics of good measuring tool .
8. What is the relationship between social work and psychology?
Types of Essay Questions
Extended Response Question
Great deal of latitude on how to respond to a question.
Example: Discuss essay and multiple-choice type tests.
Restricted Response Question
More specific, easier to score, improved reliability and validity
Example: Compare and contrast the relative advantages of disadvantages of essay and multiple choice tests with respect to: reliability, validity, objectivity, & usability.
Measures higher learning levels (synthesis, evaluation) and is easier to construct than an objective test item
Students are less likely to answer an essay question by guessing
Require superior study methods
Offer students an opportunity to demonstrate their abilities to:
May limit the sampling of material covered
Tends to reduce validity of the test
Subjective unreliable nature of scoring
“ halo effect” – good or bad student’s previous level of performance
Grammatical and spelling errors
Give students a clear idea of the scope & direction intended for the answer
Might help to start the question with the description of the required behavior (e.g., compare, analyze)
Appropriate language level for students
Construct questions that require students to demonstrate a command of background info, but do not simply repeat that info.
If question calls for an opinion, be sure that the emphasis is not on the opinion but on the way its presented or argued.
Use a larger number of shorter, more specific questions rather than one or two longer questions so that more information can be assessed.
Give students a pair of sample answers to a question of the type you will give on the test.
Sketch out a rubric (grading scheme) for each question before reading the papers OR randomly select a few to read and make up the grading scheme based on those answers
Give students a writing rubric
Detach identifying information and use code numbers instead to avoid letting personality factors influence you.
After grading all the papers on one item, reread the first few to make sure you maintained consistent standards
Be clear to student the extend to which factors other than content (e.g., grammar, handwriting, etc.) will influence the grade.
Tips for constructing Essay Questions
Provide reasonable time limits for each question
“ thinking and writing time”
Avoid permitting students a choice of questions
Will not necessarily get a representative sample of student achievement. Only be requiring all students to answer all questions can their achievement be compared
A definite task should be put forth to the student
Critical words: compare, contrast, analyze, evaluate, etc.
Distinguish between criterion-referenced test and norm-referenced test. Which type of test should be preferred in educational measurement and why?
How can the process of “Management by Learning Objective’s be implemented in the classroom teaching? Discuss the statement with examples.
Describe the forms of objectives type items and illustrate their forms with suitable examples.
Scoring Essay Items
Write an outline of the key points (use outline to design a rubric)
Determine how many points are to be assigned to the question as a whole and to the various parts within it.
If possible, score the test without knowledge of the student’s name
Score all of the answers to one question before proceeding to the next question
A lot of thanks for your precious time and active participation