Using Data to Make Informed Decisions <ul><li>Key Terms Overview and Test Drive </li></ul><ul><li>OAKS </li></ul><ul><li>R...
“ We have wide range of complex achievement targets to assess. We need all the tools we have at our disposal to do this jo...
Key Terms: What does that mean?
ACTIVITY: Picture  the WORD <ul><li>Use the Index Cards and markers on your table </li></ul><ul><li>Draw a word picture th...
Activity: Right Angle Thinking <ul><li>What do you know about OAKS Assessments? </li></ul><ul><li>What questions puzzle yo...
Oregon Assessment of Knowledge and Skills Use and Consequences <ul><li>OAKS—used to measure the extent to which  students ...
Question 1 <ul><li>Scale scores for the OAKS exam are called </li></ul><ul><ul><li>MAX scores </li></ul></ul><ul><ul><li>M...
Measuring Improvement <ul><li>What does the RIT Score mean? </li></ul><ul><li>Vertically-Scaled Score </li></ul><ul><li>Ma...
Question 2 <ul><li>RIT scores can be used to do all of the following except: </li></ul><ul><ul><li>Determine if a student ...
Scale Scores: RIT Scores <ul><li>Oregon reports test scores in a scale score, or RIT score. </li></ul><ul><li>Scale ranges...
Vertical Scale <ul><li>The continuous scale means that a student who improves by 10 points between 3rd and 4th grades (mov...
Vertical Scale
Reading/ Literature Meets Standards RIT score Math Meets Standards RIT score Science Meets Standards RIT score Grade 3 204...
RIT Cut Scores for Meets Standards
A new source of information: growth measures <ul><li>Another way to look at student performance---Did your students grow a...
OAKS Performance Reports
OAKS Scavenger Hunt <ul><li>List the subjects/content areas that are reported on OAKS online </li></ul><ul><li>What years ...
Read the notes in  fine print  ! <ul><li>Teachers should not base decisions and evaluations solely on data displayed in th...
Online Score Report Table
General Overview of Performance by Grade and Subject School School
How would you respond to the following statement?  Visible Thinking Protocol http://www.pz.harvard.edu/vt/VisibleThinking_...
I used to think…about OAKS assessment Visible Thinking Protocol http://www.pz.harvard.edu/vt/VisibleThinking_html_files/03...
Upcoming SlideShare
Loading in …5
×

Understanding Assessments

2,612 views

Published on

In-service presentation for teachers: How and Why Use Assessemnts. Presentation focuses on how to interpret RIT scaled scores of standardized tests.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,612
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Key terminology in education and specifically assessment is a language of acronyms or EDUSPEAK. We hear the words every day, used in the media,in questions from parents, and even with students. We all carry a working definition of the terms, but what do these terms mean? And more importantly, how to we explain these concepts to others?
  • We use OAKS as one piece of the whole child snapshot. It can inform our team. Let’s look at how it does, and how it does not !
  • Key terminology in education and specifically assessment is a language of acronyms or EDUSPEAK. We hear the words every day, used in the media,in questions from parents, and even with students. We all carry a working definition of the terms, but what do these terms mean? And more importantly, how to we explain these concepts to others? For a moment, I want you to put on the hat of the student. You are a student in your
  • Informational slide— Additional Background: accountability exams designed to be used summatively. Limited because in development these exams usually have to balance time for administration (limited) with purpose of getting a valid and reliable score for accountability or evaluation. Most state CRTs were developed due to a call from educators for accountability exams that are built to their state standards (as opposed to the perception that NRTs are not). A CRT by definition is designed to be a deep test of content specific learning objectives. In theory, a CRT is designed to provide information at a deep level to determine students’ level of mastery on specific content, skills and subskills. In reality, due to time and money constraints, state CRTs generally don’t have enough items on any single skill or learning objective to provide the in depth information intended by a CRT. Therefore, the score information at a subscore level is not deep and not intended to diagnose student learning needs. Rather, state CRT scores can be used to classify students and give broad strengths and weaknesses that can be explored through further assessment activities.
  • Living Likert, activity we use to jump start conversation around our existing conceptions and experience with a particular topic. It is like an anticipatory set that we’ll use to informally assess where we are starting from for the concepts and objectives of this event. Stress Why we are doing this….pre-assess, get a sense of perceptions and misconceptions of participants with regards to topic. How does this apply in your practice? Have poster board or chart paper with Strongly agree at one end and strongly disagree at the other end of the room. Make sure there is room to have the group line up on the continuum. This is a process tool that you can use as a leader or even as a classroom teacher to get staff or students to think about what they know and believe about a concept, an opinion, etc. In this case, we’re going to start with a big concept around use of assessments. Talk to each other as you arrange yourself on the continuum. Be prepared to share out why you are standing in the position you are standing relative to the two extremes.
  • OAKS stated purpose is definitive and provided by ODE in numerous locations in print and on the web-based materials.
  • Explain where RIT comes from in terms of the acronym. Rasch unit based on the Rasch scale created using a Rasch model.
  • Oregon reports test results in RIT scores. These are scale scores designed to provide improvement information, specifically, these scores can be used to measure student growth over time in each academic area. These scores can also be averaged and the changes in average performance over time can be viewed. As Oregon settles on a growth model, there will be more opportunity to discuss the difference between measures of improvement and measures of growth.
  • Determine a student’s intervention needs. OAKS is a broad pointer stick. It is designed for accountability. It can give you a direction to further explore to uncover deeper learning and intervention needs.
  • From technical manual: The term RIT score is short for Rasch Unit, a scoring scale named for Georg Rasch, a Danish mathematician. The RIT scale ranges from 150 - 300 and is similar in design to the scale used by the Scholastic Assessment Test (SAT) and American College Testing (ACT) college entrance exams. Since Oregon’s tests are vertically scaled, RIT scores, unlike raw scores, allow student growth to be measured over time. Rasch IRT calibration provides standardization of the item difficulties and a bias correction (Wright&amp; Stone, 1979), while linking new items to the same scale as previously administered items. The RIT scale has a mean of 200 and a standard deviation of 10, and these RIT scores are comparable within the same content area and grade across administrations. A RIT score of 250 from one administration indicates the same level of examinee ability as a score of 250 from another administration. What does it mean to have scores that are vertically scaled? Think about your curriculum standards first. They are developed to represent a vertical continuum of increasing knowledge and skill complexity (cognitive domain) as students move from K-12. Now think about what we said was the purpose of the OAKS—to measure the extent to which students are learning the curriculum established in the Oregon Academic Content Standards. If we are accountable for students’ learning the content standards, and those standards represent a vertical continuum of increasing knowledge and skill complexity, then ideally we want a scale that reflects a measure of progress along that continuum. That is what a vertically scaled exam is designed to do, to measure progress along that continuum and communicate that progress in a scaled score so you can understand the magnitude of the change. RIT scores are vertically scaled. When a student increases in his/her scaled score, you can say that growth has occurred. However, what you can’t say is in what particular set of skills/subskills the student has grown. The score reporting categories give you a general point, but you will need more info for specific areas of growth.
  • From technical manual: The RIT scale ranges from 150 - 300 and is similar in design to the scale used by the Scholastic Assessment Test (SAT) and American College Testing (ACT) college entrance exams. Since Oregon’s tests are vertically scaled, RIT scores, unlike raw scores, allow student growth to be measured over time. Rasch IRT calibration provides standardization of the item difficulties and a bias correction (Wright&amp; Stone, 1979), while linking new items to the same scale as previously administered items. The RIT scale has a mean of 200 and a standard deviation of 10, and these RIT scores are comparable within the same content area and grade across administrations. A RIT score of 250 from one administration indicates the same level of examinee ability as a score of 250 from another administration. What does it mean to have scores that are vertically scaled? Think about your curriculum standards first. They are developed to represent a vertical continuum of increasing knowledge and skill complexity (cognitive domain) as students move from K-12. Now think about what we said was the purpose of the OAKS—to measure the extent to which students are learning the curriculum established in the Oregon Academic Content Standards. If we are accountable for students’ learning the content standards, and those standards represent a vertical continuum of increasing knowledge and skill complexity, then ideally we want a scale that reflects a measure of progress along that continuum. That is what a vertically scaled exam is designed to do, to measure progress along that continuum and communicate that progress in a scaled score so you can understand the magnitude of the change. RIT scores are vertically scaled. When a student increases in his/her scaled score, you can say that growth has occurred. However, what you can’t say is in what particular set of skills/subskills the student has grown. The score reporting categories give you a general point, but you will need more info for specific areas of growth.
  • From technical manual: The RIT scale ranges from 150 - 300 and is similar in design to the scale used by the Scholastic Assessment Test (SAT) and American College Testing (ACT) college entrance exams. Since Oregon’s tests are vertically scaled, RIT scores, unlike raw scores, allow student growth to be measured over time. Rasch IRT calibration provides standardization of the item difficulties and a bias correction (Wright&amp; Stone, 1979), while linking new items to the same scale as previously administered items. The RIT scale has a mean of 200 and a standard deviation of 10, and these RIT scores are comparable within the same content area and grade across administrations. A RIT score of 250 from one administration indicates the same level of examinee ability as a score of 250 from another administration. What does it mean to have scores that are vertically scaled? Think about your curriculum standards first. They are developed to represent a vertical continuum of increasing knowledge and skill complexity (cognitive domain) as students move from K-12. Now think about what we said was the purpose of the OAKS—to measure the extent to which students are learning the curriculum established in the Oregon Academic Content Standards. If we are accountable for students’ learning the content standards, and those standards represent a vertical continuum of increasing knowledge and skill complexity, then ideally we want a scale that reflects a measure of progress along that continuum. That is what a vertically scaled exam is designed to do, to measure progress along that continuum and communicate that progress in a scaled score so you can understand the magnitude of the change. RIT scores are vertically scaled. When a student increases in his/her scaled score, you can say that growth has occurred. However, what you can’t say is in what particular set of skills/subskills the student has grown. The score reporting categories give you a general point, but you will need more info for specific areas of growth.
  • S-3 Cut scores available in Volume 3 of the technical report on the Oregon assessment system. Here is the cut for meets standards for subjects and grades. Supplemental materials too In your OAKS binder, you have the 09/10 Scores for Meets and Exceeds. Look for the BLUE handout.
  • RIT scores are scaled to measure growth. Meets Standards cut points reflect a curvilinear path of increased expected achievement to stay at or above the proficient standard. The Oregon growth model will establish answers to important policy questions.
  • Transition back to OAKS. OAKS Online provides information that gives point in time performance, measures of improvement and measures of growth. This system provides some different views, as well as additional detail not available on the public reporting system.
  • This is a filler that teams can use to test their knowledge of what is available in OAKS.
  • OAKS system online has several small print reminders about how to use the information in the site and the responsibility attached to the privilege of access. Also note that there helpful information in the report interpretation guides provided by the department.
  • This initial report gives state and school or district level summary results for each of the achievement levels, the current year scale score, total count of students, as well as last year’s total count and scale score at the same time of year in prior year. An overview of the # of tests students took and total participation is also provided. The scale score is the RIT score. This report can be drilled down to student level by clicking on the hyperlinked district, institution and personnel links.
  • Start with overall performance and work your way down the funnel to the areas of focus. AT the global level you have the average highest scale score and the margin of error associated with it. The graphic also indicates the performance level that the score falls in. You can drill down to a school and teacher/personnel, or select a report to view a more detailed report on performance levels.
  • Understanding Assessments

    1. 1. Using Data to Make Informed Decisions <ul><li>Key Terms Overview and Test Drive </li></ul><ul><li>OAKS </li></ul><ul><li>RIT </li></ul><ul><li>Vertically Scaled Assessment </li></ul>Presented by Jennifer Shotts
    2. 2. “ We have wide range of complex achievement targets to assess. We need all the tools we have at our disposal to do this job. We can not afford to throw any-including standarized tests-away. Our challenge is to find ways to use all the tools well and use them in balance…” Stiggins, R.J. (1985, October) Improving assessment where it means in the most: in the classrooom. Educational Leadership, pp 69-74.
    3. 3. Key Terms: What does that mean?
    4. 4. ACTIVITY: Picture the WORD <ul><li>Use the Index Cards and markers on your table </li></ul><ul><li>Draw a word picture that helps you identify OAKS, RIT, VERTICALY SCALED. </li></ul><ul><ul><li>Add words that help you define it as we go along. </li></ul></ul><ul><ul><li>Oregon Assessement of Knowledge and Skills </li></ul></ul><ul><ul><li>Summative, Criterion-Based, Vertically Scaled RIT score, Computer-Adaptive </li></ul></ul>
    5. 5. Activity: Right Angle Thinking <ul><li>What do you know about OAKS Assessments? </li></ul><ul><li>What questions puzzle you? </li></ul><ul><li>How can you explore this topic today? </li></ul>On your graphic organizer, brainstorm the Facts (A), and Feelings /Associations (B) about our topic: OAKS Assessments.
    6. 6. Oregon Assessment of Knowledge and Skills Use and Consequences <ul><li>OAKS—used to measure the extent to which students are learning the state’s curriculum as described in the Oregon Academic Content Standards </li></ul><ul><ul><li>Also used to measure Essential skills of Oregon Diploma Project for reading, mathematics and part of writing </li></ul></ul><ul><li>Serves dual purpose of meeting federal and state accountability requirements </li></ul>
    7. 7. Question 1 <ul><li>Scale scores for the OAKS exam are called </li></ul><ul><ul><li>MAX scores </li></ul></ul><ul><ul><li>Mean scores </li></ul></ul><ul><ul><li>RIT scores </li></ul></ul><ul><ul><li>Rater scores </li></ul></ul>
    8. 8. Measuring Improvement <ul><li>What does the RIT Score mean? </li></ul><ul><li>Vertically-Scaled Score </li></ul><ul><li>Margin of Error </li></ul><ul><li>Challenge: How much improvement or lack of improvement is meaningful? </li></ul><ul><li>Still relative—improvement compared to what? </li></ul>
    9. 9. Question 2 <ul><li>RIT scores can be used to do all of the following except: </li></ul><ul><ul><li>Determine if a student meets standards </li></ul></ul><ul><ul><li>Determine if a student has grown in Oregon content and skills </li></ul></ul><ul><ul><li>Determine a student’s strengths and weaknesses in a content area </li></ul></ul><ul><ul><li>Determine a student’s intervention needs </li></ul></ul>
    10. 10. Scale Scores: RIT Scores <ul><li>Oregon reports test scores in a scale score, or RIT score. </li></ul><ul><li>Scale ranges from 150 – 300 </li></ul><ul><li>Scaled to allow for student growth to be measured over time </li></ul><ul><li>RIT scores are comparable within the same content area and grade across administrations </li></ul>
    11. 11. Vertical Scale <ul><li>The continuous scale means that a student who improves by 10 points between 3rd and 4th grades (moving from 204 to 214) has improved just the same as a student who improves by 10 points between 5th and 6th grades (moving from 219 to 229). </li></ul><ul><ul><li>This equal interval feature makes measuring growth of individual students easy and reliable. </li></ul></ul>
    12. 12. Vertical Scale
    13. 13. Reading/ Literature Meets Standards RIT score Math Meets Standards RIT score Science Meets Standards RIT score Grade 3 204 205 Grade 4 211 212 Grade 5 218 218 225 Grade 6 222 221 Grade 7 227 226 Grade 8 231 230 234 Grade 10 236 236 240
    14. 14. RIT Cut Scores for Meets Standards
    15. 15. A new source of information: growth measures <ul><li>Another way to look at student performance---Did your students grow academically during the year? </li></ul><ul><ul><li>RIT scores are scaled to allow for student growth to be measured over time </li></ul></ul><ul><ul><li>RIT scores are comparable within the same content area and grade across administrations </li></ul></ul><ul><ul><li>Policy questions include </li></ul></ul><ul><ul><ul><li>What constitutes expected annual growth? </li></ul></ul></ul><ul><ul><ul><li>What constitutes value added? </li></ul></ul></ul>
    16. 16. OAKS Performance Reports
    17. 17. OAKS Scavenger Hunt <ul><li>List the subjects/content areas that are reported on OAKS online </li></ul><ul><li>What years of test results are available on OAKS online? </li></ul><ul><li>Name two point in time reports on OAKS </li></ul><ul><li>Name an Improvement Report on OAKS </li></ul><ul><li>Name a Longitudinal Report on Oaks </li></ul><ul><li>Name the levels of detail you can drill to on the OAKS system: </li></ul>
    18. 18. Read the notes in fine print ! <ul><li>Teachers should not base decisions and evaluations solely on data displayed in this online score reporting system. </li></ul><ul><li>Comparative data reports are based on the number of students tested at the time and may not be representative of official results. </li></ul><ul><li>FERPA prohibits the release of any personally identifiable information. </li></ul>
    19. 19. Online Score Report Table
    20. 20. General Overview of Performance by Grade and Subject School School
    21. 21. How would you respond to the following statement? Visible Thinking Protocol http://www.pz.harvard.edu/vt/VisibleThinking_html_files/03_ThinkingRoutines/03c_Core_routines/UsedToThink/UsedToThink_Routine.htm
    22. 22. I used to think…about OAKS assessment Visible Thinking Protocol http://www.pz.harvard.edu/vt/VisibleThinking_html_files/03_ThinkingRoutines/03c_Core_routines/UsedToThink/UsedToThink_Routine.htm But now, I think…

    ×