TEST CONSTRUCTION The Art of Effective Evaluation Presented to the Faculty and Students of UPHSD – Molino CampusPresented by:Arnel O. Riverahttp://www.slideshare.net/ArnelSSI
EMPTY YOUR CUP An Old Buddhist Tale
Bruce Lee, the legendary martial artist, once told a story about a highly- educated man who went to a Zen teacher to ask and acquire knowledge about Zen.
As the Zen teacher began to explain things, the man would frequently interrupt with remarks like, "Oh, yes, we have that too…”
Finally the Zen teacher stopped talking. He began to serve tea. He poured tea into the man’s cup.
He poured tea into the cup until it was full, and then kept pouring until the cup overflowed. "Enough!" the man once more interrupted, "No more can go into the cup!"
“Indeed, I see," answered the Zen teacher, "If you do not first empty the cup, how can you taste my cup of tea?"
A full cup cannot accept anything more. Similarly, a person who believes that he had learnt a lot cannot learn anything else & will stagnate quickly and not move to higher levels. WWW.SIMILIMA.COM
ATTITUDE DETERMINES YOUR ALTITUDE.A SMALL CHANGE IN ATTITUDE MAKES A BIGDIFFERENCE
Why is there a need to study TEST CONSTRUCTION?
13% of students who got lowgrades in exams are causedby faulty test questions. WORLDWATCH The Philadelphia Trumpet August 2005
It is estimated that 90% of all test questions asked in the US are of “Low level” - knowledge and comprehension. (Wilen, W.W., 1992)
“Low level” doesn’t mean easy: Write an essay explaining the decline and fall of the Roman Empire incorporating at least five of the seven causes discussed in class from the writings of Gibbon and Toynbee“High level” doesn’t mean hard: Which movie did you like more, WALL-E or Cars? Why?
Sample Test Question:
Outline: Principles in Test Preparing Matching Construction Type Questions Steps in Preparing Preparing Sentence Test Questions Completion Questions Preparing Multiple Preparing Essay Questions Choice Questions Other types of Test Preparing True or Questions False Questions Wrap-up/Things to Remember
“The evaluation of pupils’ progress is a major aspect of the teacher’s job.“ Evaluating Educational Outcomes (Oriondo & Antonio)
Explain the message of the comic strip.
Our students were trainedto memorize information.After they have memorizedfacts, then what?
Old belief about teaching ..... Education is a process where the notes of the teachers or the contents in the books are transfered to the notebooks of the students, without understanding them. (Wiggins, 2009)
In your lesson plans, did youever bother to ask thisquestion:• How do you know if you have achieved your desired result?
What is testing?“A test is a sample of behavior, products, answers, orperformances from a particular domain” (Carrington, 1994) “… its a systematic method of eliciting performance which is intended to be the basis for some sort of decision making" (Hughes, 1989). “A test will predict performance levels, and the learner will somehow reconstruct its parts in meaningful situations when necessary” (McCann, 2000) “ Testing is generally concerned with turning performance into numbers.” (Baxter, 1997) Guidelines for Test Construction
What are tests for?Inform learners and teachers of the strengths and weaknesses of the processMotivate learners to review or consolidate specific materialGuide the planning/development of the ongoing teaching processCreate a sense of accomplishmentDetermine if the objectives have been achievedEncourage improvement Guidelines for Test Construction
Characteristics of Good Tests Validity – the extent to which the test measures what it intends to measure Reliability – the consistency with which a test measures what it is supposed to measure Usability – the test can be administered with ease, clarity and uniformity
Other Things to Consider Scorability – easy to score Interpretability – test results can be properly interpreted and is a major basis in making sound educational decisions Economical – the test can be reused without compromising the validity and reliability
“To be able to prepare a good test, one has to have a mastery of the subject matter, knowledge of the pupils to be tested, skill in verbal expression and the use of the different test format” Evaluating Educational Outcomes (Oriondo & Antonio)
5 Most Commonly used Test Format1. Multiple Choice2. True or False3. Matching Type4. Fill-in the blanks (Sentence Completion)5. EssaySource: Turn-out of Test Questions in SSI (2003-2007)
General Steps in Test Construction DRAFT PRODUCE A T.O.S. ORDER TEST ANALYZE SUBMISSION
Table of Specifications (TOS) A two way chart that relates the learning outcomes to the course content It enables the teacher to prepare a test containing a representative sample of student behavior in each of the areas tested.
Tips in Preparing the Tableof Specifications (TOS) Don’t make it overly detailed. Its best to identify major ideas and skills rather than specific details. Use a cognitive taxonomy that is most appropriate to your discipline, including non- specific skills like communication skills or graphic skills or computational skills if such are important to your evaluation of the answer.
Tips in Preparing the Tableof Specifications (TOS) Weigh the appropriateness of the distribution of checks against the students level, the importance of the test, the amount of time available. MATCH the question level appropriate to the level of thinking skills
Examples of Student Activities and Verbsfor Bloom’s Cognitive Levels (Jacobs &Chase, 1992:19)Bloom’s Cognitive Student Activity Words to Use in ItemLevel StemKnowledge Remembering facts, Define, list, state, terms, concepts, identify, label, name, definitions, principles who?, when?, where?, what?Comprehension Explaining/interpreting Explain, predict, interpret, the meaning of material infer, summarize, convert, translate, account for, give example, paraphraseApplication Using a concept or Apply, solve, show, make principle to solve a use of, modify, problem demonstrate, compute
Examples of Student Activities and Verbs forBloom’s Cognitive Levels (Jacobs & Chase,1992:19) Bloom’s Cognitive Student Activity Words to Use in Item Level StemAnalysis Breaking material down Differentiate, into its component parts to compare/contrast, see interrelationships/ distinguish ____from hierarchy of ideas ____, how does ____relate to ___, why does ____workSynthesis Producing something new or Design, construct, develop, original from component formulate, imagine, parts create, change, write a poem or short storyEvaluation Making a judgment based Appraise, evaluate, on a pre-established set of justify, judge, which would criteria be better?
Tips in Preparing the Table of Specifications (TOS) The following array shows the most common questions types used at various cognitive levels.Factual Knowledge Application Analysis and EvaluationMultiple Choice Multiple Choice Multiple ChoiceTrue/False Short Answer EssayMatching Type ProblemsS. Completion EssayShort Answer/RRT
General Rules in Writing TestQuestions Number test questions continuously. Keep your test question in each test group uniform. Make your layout presentable. Do not put too many test questions in one test group. T or F: 10 – 15 questions Multiple Choice: max. of 30 questions Matching type: 5 questions per test group Others: 5 – 10 questions
Some additional guidelines to considerwhen writing items are described below:1. Avoid humorous items. Classroom testing is very important and humorous items may cause students to either not take the exam seriously, become confused or anxious.2. Items should measure one’s knowledge of the item context not their level of interest.3. Write items to measure what students know, not what they do not know. (Cohen & Wallack)
Multiple Choice Test
What to Look for onMultiple Choice TestsWhen checking the stems for correctness: Ensure that the stem asks a clear question. Reading level is appropriate to the students The stem is grammatically correct. Negatively stated stems are discouraged.
Anatomy of a Perfect Multiple Choice TestsExample:What is the effect of releasing a ball inpositive gravity?a) It will fall “down.” correctb) It will retain its mass. true but unrelatedc) It will rise. false but relatedd) Its shape will change. false and unrelated
Multiple Choice Questions 1. Use negatively stated stems sparingly and when using negatives such as NOT, underline or bold the print. 2. Use none of the above and all of the above sparingly, and when you do use them, dont always make them the right answer. 3. Only one option should be correct or clearly best.
Multiple Choice Questions:4. All options should be homogenous and nearly equal in length.5. The stem (question) should contain only one main idea.6. Keep all options either singular or plural.7. Have four or five responses per stem (question).
Multiple Choice Questions:7. When using incomplete statements place the blank space at the end.8. When possible organize the responses.9. Reduce wordiness.10. When writing distracters, think of incorrect responses that students might make.
Examples1. Sheldon developed a highly controversial theory of personality based on body type and temperament of the individual. Which of the following is a criticism of Sheldons work? a. He was influenced too much by the Freudian psychoanalysis. b. His rating of physique and temperament were not independent. c. He failed to use empirical approach. d. His research sample was improperly selected.
ExamplesBetter: (Eliminate excessive wording and irrelevant information)1. Which of the following is a criticism of Sheldons theory of personality?
Examples1. The receptors for the vestibular sense are located a. in the fovea. b. in the brain. c. in the middle ear. d. in the inner ear.
ExamplesBetter: (Include in the stem any word(s) that might otherwise be repeated in each option.)1. The receptors for the vestibular senses are located in the _______. a. fovea b. brain c. middle ear d. inner ear
True or False
What to Look for onTrue/False Tests Each statement is clearly true or clearly false. Trivial details should not make a statement false. Statements are written concisely without more elaboration than necessary. Statements are NOT quoted exactly from text.
Tips in Making True/False Tests Give emphasis on the use of quantitative terms than qualitative terms. Avoid using of specific determiners which usually gives a clue to the answer. False = all, always, never, every, none, only True = generally, sometimes, usually, maybe, often Discourage the use of negative statements. Whenever a controversial statement is used, the authority should be quoted. Discourage the use of pattern for answers.
Examples:Find the errors, and/or problems with thefollowing true-false tests.____ 1. Repetition always strengthens the tendency for a response to occur.(Using "always" usually means the answer is false.)
Examples:_____ 2. The process of extinction is seldom immediate but extends over a number of trials.(Words like "seldom" usually indicate a true statement.)
Examples:_____ 3. The mean, median, and mode are measures of central tendency, whereas the standard deviation and range are measures of variability.(Express a single idea in each statement.) e.g.“The mean and standard deviation are measures of central tendency.”
Parts of the Matching Type Test(Vertical Type)Column A Column B(Premise) (Response)
Parts of the Matching Type Test(Horizontal Type) (Response)(Premise)
What to Look for onMatching Type Tests The list of responses should be relatively short. Response options should be arranged alphabetically or numerically. Directions clearly indicate the basis for matching. Can responses be used more than once? Where will you place your answer? Can students infer relationships or are they based on real word logic?
What to Look for onMatching Type TestsPosition of matches should be varied.Avoid using patterns.The choices of each matching setshould be on one pageThere are more responses thanpremises in a single set if responsescannot be used more than once.
What to Look for onMatching Type TestsThe premises are homogeneous as wellas the responses and are grouped asone item. Example: Set A: Provinces in Region I Set B: Provinces in CARIf responses can be used more thanonce, it should be proportional to thenumber of premises (3:5 or 4:10)
Examples:Directions: Match the following.1. Food A. Primary reinforcer2. Psychoanalysis B. Sigmund Freud3. B.F. Skinner C. Operant conditioning4. Standard deviation D. Measure of variability5. Schizophrenia E. Hallucinations
Examples:Better: (Use homogenous material in matching items, andif responses are not to be used more than once, includemore responses than stimuli.)Match the theories in Column A with their proponents inColumn B. Write the letter of the correct answer. Column A Column B___ 1. Psychodynamic Theory A. Albert Bandura___ 2. Trait Theory B. B.F. Skinner___ 3. Behaviorism C. Carl Rogers___ 4. Humanism D. Gordon Allport___ 5. Social Learning Theory E. Karn Horney F. Raymond Cattell G. Sigmund Freud
Sentence Completion /Fill-in the Blanks
What to Look for onSentence Completion TestsOnly significant words are omitted.When omitting words, enough clues areleft so that the student who knows thecorrect answer can supply the correctresponse.Ensure that grammatical clues areavoided.
What to Look for onSentence Completion TestsBlanks are at the end of the statement.The length of the responses are limitedto single words or short phrases.Questions are not lifted as verbatimquotes from text.
Examples:1. An animal with six legs is called _________.The item is so indefinite.It can be completed with answers such as bee, mosquito or any other insectBetter:1. Animals with six legs are called ___________.
Examples:1.The __________ is the answer in _____.Too many key words are omitted. Lines arenot in equal length. Better: 1. The product is the answers in _________.
Examples:1. If a mango weighs 250 grams, 10 mangoeswould weigh ______.There are two possible answers – 250 gramsand .25 kilos.Better:1. If a mango weighs 250 grams, 10 mangoeswould weigh ____ grams.
Essay / Short Answer Test
Types of Essay Items:Extended response type The test may be answered by the examinee in whatever manner he wants Example: Do you think teachers should be allowed to work abroad as domestic helpers? Explain your answer.
Two Types of Essay Items:Restricted response type The test limits the examinees response may be answered by the examinee’s responses in terms of length, content, style or organization. Example: Give and explain three reasons why the government should or should not allow teachers to work abroad as domestic helpers.
What to Look for onEssay TestsThe task is clearly defined. Thestudents are given an idea on the scopeand direction you intended for theanswer to take. The question startswith a description of the requiredbehavior to put them in the correct mindframe.E.g. “Compare” or “Analyze”
What to Look for onEssay TestsThe questions are written in thelinguistic level appropriate to thestudents.Questions require a student todemonstrate command of backgroundinformation, not simply repeatinginformation.
What to Look for onEssay TestsQuestions regarding a student’s opinionon a certain issue should focus not onthe opinion but on the way it ispresented and argued.A larger number of shorter, morespecific questions are better, than, oneor two longer questions.
Proposed Criteria in GradingEssay Test Ideas (20%) Weight of Evidence Presented (40%) Correct Usage (20%) Logical Conclusions drawn from the evidence (20%)
Example:What is wrong with this question? Describe asthma?Better: (Clearly explain what is expected of the student.)Describe asthma. Include in your answer :a. the pathophysiologic features of asthmab. the clinical manifestations associated with an asthma episodec. the management of an asthma episode. (10 points)
Example:What is wrong with this question?Who is better, Rizal or Bonifacio?Better: ( The students are given an idea onthe scope and direction you intended for theanswer to take.)Compare and contrast the method used byRizal and Bonifacio in promotingnationalism. (5 points)
Other types of Test Questions
Restricted Response Test (RRT)Test takers are not given choices as possible answers. Items ask for a specific answer to each questions. Example: Who discovered the Philippines? What are the four elements of the state?
Principles in constructing RRT Do not ask for trivial facts or details. It is not only useless but also frustrates the students. How many balls are used in a 9-ball match? Questions should elicit facts not opinions? What do you think Pres. GMA should do for the country to recover from its’ economic deficit? Minimize questions that call for sheer memory work unless if the answer has important analytical significance. When will the next president be sworn to office?
Chronological Sequencing Test (CST) Test takers are asked to arrange items in a systematic or logical order. Arrange the presidents according to their term of office. _____ Fidel Ramos _____ Joseph Estrada _____ Corazon Aquino _____ Gloria Macapagal - Arroyo
Principles in constructing CST Items should be homogenous and are related to each other. There should not be more than 5 items in each set. Do not number the items. This confuses the students. All items to be arranged should be in the same page. Directions should be clearly stated and that each set should be labeled about their relevance.
What is wrong in this test question? Arrange the following events in their chronological order. 1. Bataan Surrenders 2. The Japanese attacks the US fleet in Pearl Harbor, Hawaii. 3. Hitler invades Poland 4. The US declares war with Japan. 5. Gen. MacArthur escapes to Australia.
Better:21-25.) War in the Pacific Arrange the following events in chronological order. Write the numbers 1-5 on the blanks provided.___ USAFEE forces in Bataan surrender to the Japanese.___ Japanese forces attacks the US fleet in Pearl Harbor, Hawaii.___ Japan breaks diplomatic ties with the US.___ The US declares war with Japan.___ Gen. MacArthur escapes to Australia from Corregidor.
Proposed Arrangement of Test Items True or False Multiple Choice Matching Type Sentence Completion Others (RRT/Analogy/CST) Essay
Things to Remember: Making a good test takes time Teachers have the obligation to provide their students with the best evaluation Tests play an essential role in the life of the students, parents, teachers and other educators Break any of the rules when you have a good reason for doing so! (emphasis mine) (Mehrens, 1973)
POINTS TO PONDER…A good lesson makes a good questionA good question makes a good contentA good content makes a good testA good test makes a good gradeA good grade makes a good studentA good student makes a good COMMUNITY Jesus Ochave Ph.D. VP Research Planning & Development Philippine Normal University
For questions , comments or if youwant to download this file, log-on to:http://www.slideshare.net/ArnelSSI