Downloaded 57 times
Uploaded byMhairi Mcalpine
Item Analysis: Classical and Beyond
This document discusses different methods of item analysis, including classical analysis and latent trait models like Rasch and Item Response Theory (IRT). Classical analysis uses statistics like difficulty, discrimination, and reliability to evaluate test questions based on how a particular group of students performed. Latent trait models aim to measure underlying abilities and provide sample-free measurement of questions. Rasch and IRT models use item characteristic curves and estimate parameters like difficulty, discrimination, and guessing to characterize questions independently of particular test administrations. The document provides an overview of the assumptions, statistics, and software used for different item analysis methods.
More Related Content
Similar to Item Analysis: Classical and Beyond
Item Analysis: Classical and Beyond
- 1. Item Analysis: Classical andBeyond SCROLLA Symposium Measurement Theory and Item Analysis Heriot Watt University 12th February 2003.
- 2. Why is itemanalysis relevant? Item analysis provides a way of measuring the quality of questions - seeing how appropriate they were for the candidates and how well they measured their ability. It also provides a way of re-using items over and over again in different tests with prior knowledge of how they are going to perform.
- 3. What kinds ofitem analysis are there? Item Analysis Classical Latent Trait Models RaschItem Response theory IRT1 IRT2 IRT3 IRT4
- 4. Classical Analysis Classical analysisis the easiest and most widely used form of analysis. The statistics can be computed by generic statistical packages (or at a push by hand) and need no specialist software. Analysis is performed on the test as a whole rather than on the item and although item statistics can be generated, they apply only to that group of students on that collection of items
- 5. Classical Analysis Assumptions Classical testanalysis assumes that any test score is comprised of a “true” value, plus randomised error. Crucially it assumes that this error is normally distributed; uncorrelated with true score and the mean of the error is zero. xobs = xtrue + G(0, σerr)
- 6. Classical Analysis Statistics •Difficulty (item level statistic) • Discrimination (item level statistic) • Reliability (test level statistic)
- 7. Classical Analysis Difficulty The difficultyof a (1 mark) question in classical analysis is simply the proportion of people who answered the question incorrectly. For multiple mark questions, it is the average mark expressed as a proportion. Given on a scale of 0-1, the higher the proportion the greater the difficulty
- 8. Classical Analysis Discrimination The discriminationof an item is the (Pearson) correlation between the average item mark and the average total test mark. Being a correlation it can vary from –1 to +1 with higher values indicating (desirable) high discrimination.
- 9. Classical Analysis Reliability Reliability isa measure of how well the test “holds together”. For practical reasons, internal consistency estimates are the easiest to obtain which indicate the extent to which each item correlates with every other item. This is measured on a scale of 0-1. The greater the number the higher the reliability.
- 10. Classical Analysis vs Latent TraitModels • Classical analysis has the test (not the item as its basis. Although the statistics generated are often generalised to similar students taking a similar test; they only really apply to those students taking that test • Latent trait models aim to look beyond that at the underlying traits which are producing the test performance. They are measured at item level and provide sample-free measurement
- 11. Latent Trait Models •Latent trait models have been around since the 1940s, but were not widely used until the 1960s. Although theoretically possible, it is practically unfeasible to use these without specialist software. • They aim to measure the underlying ability (or trait) which is producing the test performance rather than measuring performance per se. • This leads to them being sample-free. As the statistics are not dependant on the test situation which generated them, they can be used more flexibly
- 12. Rasch vs Item Response Theory Mathematically,Rasch is identical to the most basic IRT model (IRT1), however there are some important differences which makes it a more viable proposition for practical testing • In Rasch the model is superior. Data which does not fit the model is discarded. • Rasch does not permit abilities to be estimated for extreme items and persons. • Rasch eschews the use of bayesian priors to assist parameter setting.
- 13. IRT - thegeneralised model ( ) )( )( 1 )1( gg gg bDa bDa ggg e e ccP − − + −+= θ θ θ Where ag = gradient of the ICC at the point θ (item discrimination) bg = the ability level at which ag is maximised (item difficulty) cg = probability of low candidates correctly
- 14. IRT - ItemCharacteristic Curves •An ICC is a plot of the candidates ability over the probability of them correctly answering the question. The higher the ability the higher the chance that they will respond correctly. c - intercept a - gradient b - ability at max (a)
- 15. IRT - Aboutthe Parameters Difficulty • Although there is no “correct” difficulty for any one item, it is clearly desirable that the difficulty of the test is centred around the average ability of the candidates. • The higher the “b” parameter the more difficult the question - note that this is inversely proportionate to the probability of the question being answered correctly.
- 16. IRT - Aboutthe Parameters Discrimination • In IRT (unlike Rasch) maximal discrimination is sought. Thus the higher the “a” parameter the more desirable the question. • Note however that differences in the discrimination of questions can lead to differences in the difficulties of questions across the ability range.
- 17. IRT - Aboutthe Parameters Guessing • A high “c” parameter suggests that candidates with very little ability may choose the correct answer. • This is rarely a valid parameter outwith multiple choice testing…and the value should not vary excessively from the reciprocal of the number of choices.
- 18. IRT - ParameterEstimation • Before being used (in an item bank or for measurement) items must first be calibrated. That is their parameters must be estimated. • There are two main procedures - Joint Maximal Likelihood and Marginal Maximal Likelihood. JML is most common for IRT1 and 2, while MML is used more frequently for IRT3. • Bayesian estimation and estimated bounds may be imposed on the data to avoid one parameter degrading, or high discrimation items being over valued.
- 19. Resources - ClassicalAnalysis Software • Standard statistical packages (Excel; SPSS; SAS) • ITEMAN (available from www.assess.com) Reading Matlock-Hetzel (1997) Basic Concepts in Item and Test Analysis available at www.ericae.net/ft/tamu/Espy.htm
- 20. Resources - IRT Software •BILOG (available at www.assess.com) • Xcalibre available at www.assess.com Reading • Lord (1980) Applications of Item Response Theory to Practical Testing Problems • Baker, Frank (2001). The Basics of Item Response Theory - available at http://ericae.net/irt/baker/
Editor's Notes
- #2 Facilitation skills offer immediate, practical benefits to any group process. As facilitator, your role is to set the agenda, encourage participation, and guide the pace of the meeting. Use these Dale Carnegie Training® strategies to help make your meeting a success. You can provide handouts to focus the discussion and give attendees a place to record their ideas. If you print handouts with three slides per page, PowerPoint will automatically include blank lines for your meeting participants to take notes.
- #5 I am going to add a note
- #11 Begin with a strong opening, relevant to the topic and your audience. Appeal to your audience’s interests. Your energy and attitude set the tone and provide momentum for the meeting. Facilitation is a process that helps people communicate and work together. Establish an open environment conducive to problem solving and change.
- #12 Bridge to your key topic. State the purpose and importance of your meeting. As facilitator, you are a catalyst for change. Ask questions. Coach participants. Encourage participation. Record comments. Thank participants for their individual contributions, ideas, and insights. You can use PowerPoint’s Meeting Minder to record comments during the meeting.
- #13 Set clear goals for the meeting. Clarify the problem to be solved. Get agreement from the group on the agenda, the problem and the process. As facilitator, guide, inspire, and support participation from all attendees. Solicit personal experiences. Ask for evidence to back up points. Make every participant feel they are valuable with something to contribute.
- #14 Once you have defined the problem, move to solution-finding. Encourage participants to be open-minded, think creatively, and suggest solutions. Respect diverse opinions. Diversity strengthens problem-solving and adds substance to discussions. Record ideas. Consider all suggestions. Keep a positive attitude throughout the process. Bring all participants into the process. List and prioritize options. Use the group process to determine the solution.
- #15 Now that you have a solution, outline your action plan. Using the group process, agree upon an orderly direction to reach your goals. List the action steps. You can use the Meeting Minder to record action items for meeting participants. Identify resources needed. Assign responsibilities. Establish methods of accountability. Define a time line. Identify costs. Define current resources. Also, consider what is needed for success. Training? Specific expertise? Additional information? Special materials? Additional resources? Management buy-in? Proper planning prior to action leads to goal achievement.
- #16 Close your meeting with a brief summary. Thank participants for their input. Review action plans, assignments, and accountability measures. Thank the group for following the process, working together as a team, and committing to problem solving and goal achievement. If you used Meeting Minder to keep track of action items, PowerPoint will automatically add a slide with the action items to the end of your presentation. Use this slide to review the action items and get buy-in from participants. Facilitation is leadership in action. Close your presentation with a strong comment to reinforce the success of the team. Empower the group to work toward goal achievement. Apply effective facilitation skills and you will have productive, profitable meetings.