1. PVAAS and PVAAS Teacher
Specific Reporting
Erin Van Guilder
Antietam School District
2. Goal of Session
By the end of the session, participants will be able to
answer the following questions:
1. What is PVAAS and how is it a part of the Teacher
Effectiveness Model?
2. How does PVAAS measure growth?
3. What does PVAAS tell us about growth in our district and
school(s)?
4. What is PVAAS Teacher Specific Reporting?
5. How Specific Reporting is Determined for a Teacher?
6. What and when is Roster Verification?
11. PVAAS Academic Growth is
• PVAAS Growth provides a measure
of the growth of a teacher’s group
of students
12. Where does the PVAAS Growth
Measure come from?
Testing
Subject-Grades
Methodologies
Measurement
Assessment in
Consecutive
Years
Math, 4-8
Reading, 4-8
Growth
Standards
Growth in Math
and Reading
Assessments not
in Consecutive
Years
Writing, 5 & 8
Predictive
Predicted
scaled score
based on each
students’
academic
assessment
history across
subjects
Science- 4&8
KeystonesAlgebra 1,
Biology,
Literature
13. Concept of Measuring Growth in
PVAAS
Prior Achievement for Group
Higher Achievement
• For grades 4-8, Math and Reading - represents the
mean achievement level of the group of student
• For Science, Writing, & Keystones- represents the
mean predicted and observed/actual
achievement level of the group of students
Lower Achievement
F2F Overview 2013
13
14. Concept of Measuring Growth in
PVAAS
Prior Achievement for Group
Higher Achievement
Lower Achievement
Most Recent School Year for Group
Higher Achievement
Lower Achievement
• For grades 4-8, Math and Reading - represents the mean achievement level of the group of
student
F2F Overview 2013
• For Science, Writing, & Keystones- represents the mean predicted and observed/actual
achievement level of the group of students
14
15. Concept of Measuring Growth in
PVAAS
Prior Achievement for Group
Higher Achievement
Lower Achievement
Most Recent School Year for Group
Higher Achievement
Lower Achievement
• For grades 4-8, Math and Reading - represents the mean achievement level of the group of
student
F2F Overview 2013
• For Science, Writing, & Keystones- represents the mean predicted and observed/actual
achievement level of the group of students
15
16. Concept of Measuring Growth in
PVAAS
Prior Achievement for Group
Higher Achievement
Lower Achievement
Most Recent School Year for Group
Higher Achievement
Lower Achievement
• For grades 4-8, Math and Reading - represents the mean achievement level of the group of
student
F2F Overview 2013
• For Science, Writing, & Keystones- represents the mean predicted and observed/actual
achievement level of the group of students
16
17. Concept of Measuring Growth in
PVAAS
Prior Achievement for Group
Higher Achievement
Lower Achievement
Most Recent School Year for Group
Higher Achievement
Lower Achievement
• For grades 4-8, Math and Reading - represents the mean achievement level of the group of
student
F2F Overview 2013
• For Science, Writing, & Keystones- represents the mean predicted and observed/actual
achievement level of the group of students
17
18. Concept of Measuring Growth in
PVAAS
Prior Achievement for Group
Higher Achievement
Lower Achievement
Most Recent School Year for Group
Higher Achievement
Lower Achievement
• For grades 4-8, Math and Reading - represents the mean achievement level of the group of
student
F2F Overview 2013
• For Science, Writing, & Keystones- represents the mean predicted and observed/actual
achievement level of the group of students
18
19. Regardless of the entering
achievement level of the group…
F2F Overview 2013
19
20. Regardless of the entering
achievement level of the group…
• Students should not lose
ground academically
F2F Overview 2013
20
21. Regardless of the entering
achievement level of the group…
• Students should not lose
ground academically
• A reasonable goal would be
to at least maintain the
achievement level of the
group
F2F Overview 2013
21
22. Regardless of the entering
achievement level of the group…
• Students should not lose
ground academically
• A reasonable goal would be
to at least maintain the
achievement level of the
group
• All groups of students can
make growth
F2F Overview 2013
22
23. PVAAS Value-Added
Growth Measures Reading and Math 4-8
1000 1100 1200 1300 1400 1500
Scale Score Points
The Growth Measure is NOT
reported in years and cannot
be referenced in terms of
number of year’s growth.
The Growth Measure is NOT
reported in percentages
and cannot be referenced
in terms of an increase or
decrease in percentage
points.
0 10 20 30 40 50 60 70 80 90 100
NCE Units
Growth is measured in Normal Curve Equivalent (NCE) units as
scale scores cannot be compared from grade to grade and
year to year.
24. PVAAS Value-Added Growth Measures Science, Writing and Keystones
• Growth is measured in scale score points.
• The Growth Measure is NOT reported in
years and cannot be referenced in terms of
number of year’s growth.
• The Growth Measure is NOT reported in
percentages and cannot be referenced in
terms of an increase or decrease in
percentage points.
26. What is PVAAS teacher-specific reporting?
• Estimates the effect of a teacher’s
performance on the academic growth of a
group of students.
• Taking into account both their endpoint and
their entering achievement level.
• By concentrating on growth, PVAAS puts the
emphasis on what educators can influence.
26
27. Purpose of PVAAS Teacher Specific Reporting
• To inform teaching practice in order to
guide replication, changes,
or
• adjustments of teaching practices for
continual professional growth.
28. Who does this affect?
A PA certified educator with full or
partial responsibility for content specific
instruction of the assessed eligible
content as measured by a PA state
assessment
Face to Face
Online Instruction
28
29. PVAAS Teacher Reporting:
Teachers in Which Tested Grades/Subjects/Courses?
• PSSA: With/Without Accommodations (Not PASA)
– Grades 4-8: Reading and Math
– Grades 4 & 8: Science
– Grades 5 & 8: Writing
•Future: Transition into ELA Assessment
• Courses Culminating in Keystone Exams:
With/Without Accommodations
– Algebra I
– Literature
– Biology
Approximately
30-40% of your
teaching staff will
be eligible for a
PVAAS teacher
specific score
29
30. What is content-specific instruction?
• Content Specific Instruction: When a teacher(s) has the
responsibility to
• plan the instruction of the assessed eligible content,
• provide the instruction of the assessed eligible content,
• use the assessment information to evaluate the
effectiveness of the instruction of the assessed eligible
content on a PA state assessment.
Defined on the Framework for Teaching by both Domains 1
and 3
Special education teachers, intervention specialists, reading/math
specialists, ESL teachers, and gifted teachers are eligible for a
PVAAS score
30
34. New Terms: % Instructional Responsibility
% Instructional Responsibility
% Student +
Teacher
Enrollment
Full or
Partial % of
Instruction
• This % is your total Instructional Responsibility for
a student.
• This % reflects how much the student will be
weighted in the PVAAS analyses.
• The % Instructional Responsibility is calculated
by multiplying the (% Student + Teacher
Enrollment) by (Full or Partial % of Instruction).
34
35. Calculating: % Student + Teacher Enrollment
% Student +
Teacher
Enrollment
What proportion of the
grade/subject/course
are the student and
teacher concurrently
enrolled?
Day 1
Grade/Subject/
Course
Day 140
Last Day Before
Testing Window
Student
Enrolled
Teacher
Enrolled
% Student + Teacher
Enrollment = Concurrent
Enrollment
35
36. Calculating: Full or Partial % of Instruction
Full or Partial % of Instruction
Are you the only PA certified educator responsible for the contentspecific instruction (Domains 1 and 3) of the assessed eligible
content in the grade/subject/course?
YES
Full % of Instruction =
100%
NO
Partial % of Instruction = <100%
Proportion of responsibility for the
content-specific instruction of the
assessed eligible content in the
grade/subject/course
(Domains 1 and 3)
36
37. Mrs. Smith’s % Instructional Responsibility
The Total % Instructional
Responsibility for Each
Student
New # of Students in the
TSR Group
40. What is this Verification process all about?
Teachers, school administrators and district administrators
verifying data that is used to yield PVAAS teacher specific
reporting– data that will ultimately be used as part of a
teacher’s evaluation.
The PVAAS rosters are
prepopulated from the
PIMS Course/HQT
submitted by LEAs to PIMS
(Student, Staff, Course
Templates)
40
44. What does my District or School Admin
see?
• The same PVAAS data and reporting for an
individual teacher that a teacher can see
• No additional data or reporting on any
individual teacher!
School Admin. can also:
• See a District/School Teacher Summary for
quick access by school/ grade/ subject/
course
• Search to locate an individual teacher’s
report
45. • Will PVAAS teacher‐specific reporting
be reported publicly?
– Act 82 states that “An employee's
individual rating form shall not be subject
to disclosure
– Reference: Act of February 14, 2008 (P.L. 6, No.3), known as the
"Right‐to‐Know Law." PDE will follow the law on non‐disclosure.
46. Support and Next Steps
Next Steps- Fall Training
Types of Reports for Teacher
• Resources and Supports
Facilitator says:Note the workbook page reference in the top left corner of each slide so you can follow in your participant workbook.By the end of the session, participants will be able to answer the following questions:What is PVAAS?How does PVAAS measure growth?What does PVAAS tell us about growth in our district and school(s)?What does PVAAS tell us about how students are projected to perform on a future state assessment?Which reports are most meaningful for school level planning, course level planning, grade level planning, and student level planning?[CLICK]
In this high-stakes assessment and accountability climate, we need accurate information to guide us in making appropriate decisions for students…decisions about instruction, curriculum, and assessment. PVAAS is one tool to assist schools in this process. PVAAS is a very robust statistical analysis of existing assessment data (PSSA & Keystone assessments) – no additional testing is necessary for PVAAS analyses. PVAAS utilizes longitudinal test scores to produce reliable estimates of academic growth and projections of performance on a future assessment .For the last 12 years, PA has been implementing and piloting PVAAS as the district and school level. We were able to get growth reports and projections on student achievement. Now 13-14 all districts are implementing PVAAS at a district, school and teacher reporting level.
The Educator Effectiveness System has been designed to reform the way we evaluate teachers, educational specialists, and principals, as well as the critical components of training for professional growth.Two decades of research have consistently told us that teachers matter more to student outcomes than any other in-school factor. Improving the effectiveness of teachers is critical to student success, as well as creating safe, nurturing school environments based on the premise of high expectations for all. Through the Educator Effectiveness System, Pennsylvania is actively engaged in improving teaching and learning by implementing better teacher, educational specialist, and principal evaluation systems and providing these professionals with the feedback they need to improve their practice.There are many data collections that go into the TEM: Observational DataAchievement DataGrowth dataStudent Learning Objective DataOther DataThey provide a more accurate, well rounded perspective of the students, school and district for academic achievement and growth. PVAAS is one measure that indicates academic growth of a teachers and group of students.
This year counts as the first year of the 3 year Rolling Average. In the school year 2015-16, the 1st PVAAS 3 Year Rolling Average is available for the TEM rating form.A PVAAS Teacher Specific Report will be created for any grade/course/subject that is measured by PSSA or Keystones grades 4-12. This means:4th grade math, science, and reading, 5th grade reading, math, writing, 6th grade reading, math, 7th grade reading and math, 8th grade reading, math, writing, Keystones in Algebra, Literature, and Biology
We will begin by describing what PVAAS is and clarify what PVAAS is not.
PVAAS provides a wide range of reports, but they fall into two broad categories:The Value-added, or growth, information “looks back”…it helps schools to evaluate the effectiveness of the district and the school. How much growth did groups of students make in the past school year? IN fact, Pennsylvania’s value-added reporting for teachers falls into this looking back side of the PVAAS reporting.The Projection information “looks forward”…it helps schools plan for the future. Are students on a path to proficiency or higher? Both serve different purposes, and both are equally important for continuous school improvement.
For the purposes of today, we will be looking at the value added report which will be associated with your Teacher Effectiveness Model.The Value-added, or growth, information is what “looks back”and helps educators to evaluate the effectiveness of their schools and the influence they have on the academic growth of their students. How much growth did groups of students make in the past school year? This data is available on several different types of reports in PVAAS- so lets look at what PVAAS defines as academic growth.
In Pennsylvania, growth is assessed against the standard for PA Academic Growth. It is based on the philosophy that regardless of the entering achievement level of a group of students, they should not lose ground academically. So as educators, we can ask ourselves, is it not a reasonable goal to maintain the achievement level of a group of students?Is that not a reasonable expectation?In other words, if students are fairly high achieving, is it not reasonable to expect that we, as educators, keep them fairly high achieving?And if students are low achieving, is it not reasonable to expect that we would at least keep those students at that level of achievement (and not have them slip further behind)?We would hope that everyone would agree that this is a reasonable expectation.
PVAAS concept of growth is clear on what it is and what it does not measure.PVAAS does NOT compare the achievement of one group of students in a particular subject and grade level with the previous year’s students in that same subject and grade level. For example, the achievement of the most recent group of students in 8th grade Math is NOT compared to the achievement of the previous year’s group of 8th grade students in Math. In addition, PVAAS does NOT use the percentages of students at various academic performance levels to measure growth. Each performance level contains a range of scores, and students move around within these ranges, as well as between ranges.
PVAAS uses existing assessment data to measure the growth of each group of students as they progress through grade levels over time. It compares the group’s most recent academic performance to their own prior academic performance to monitor and estimate their academic growth. PVAAS measures growth by the comparison of estimates based on multiple test scores. Single test scores, including scale scores, are only one measure and are simply “snapshots in time.” Comparison of these single test scores would NOT be an effective way to measure growth.
It is important to know that PVAAS utilizes two different methodologies for estimating growth measures for groups of students. The Growth Standard and Predictive methodologies are used based on whether the test data are administered in consecutive grades, or not. The Growth Standard Methodology is employed for consecutive grades of assessment data. In Pennsylvania, this means we use this for reporting in grades 4-8 in the subjects of Reading and Math. The Predictive Methodology is used for non-consecutive grades of assessment data. In Pennsylvania, this means we use this for all grade levels of Science and Writing reporting, as well as for all reporting of Keystone content areas (specifically Algebra I, Biology, and Literature).Both serve different purposes, and both are equally important for continuous school improvement. The Predictive Score for Keystone Algebra 1 is based on students PSSA scores in reading, math, and science from Prior Years.The Predictive Score of Writing in 5th grade would be based on your reading and math scores the follow year.( So we are all in this together)
Facilitator says:To estimate the achievement level of this group of students, PVAAS uses all of the student level data from all years, from all grades, for up to 5 years.A mean, or average, achievement level is determined for the group in grades 4-8, reading and math; while in science, writing, or Keystone content areas, it is called a predicted score.[click]
Facilitator says:After the next round of testing, PVAAS uses all the prior data for the group and adds the data from the most recent round of testing…[CLICK]
Facilitator says:…to get a new estimate of the average achievement level of the group. Now we can compare the prior achievement level of the group (or the predicted score of the group in the cases of Science, Writing, or the Keystone content areas) to the new achievement level of the group…[CLICK]
Facilitator says:If the evidence indicates the achievement is roughly at the same point within the distribution of scores as the previous year (or is roughly the same as predicted in the case of Science, Writing, and the Keystone content areas), this would indicate that the group of students maintained their achievement level which in PVAAS would be indicated as Green – evidence that the student group met the standard for PA Academic Growth.[CLICK]
Facilitator says:If there is moderate evidence or significant evidence that the group of students exceeded the standard for PA Academic Growth or increased their average achievement level or performed higher than predicted, this would be indicated with a Light Blue or Dark Blue on the school or district value-added report.[click]
Facilitator says:If there is moderate evidence or significant evidence that the group of students did not met the standard for PA Academic Growth or their average achievement level slipped or they performed lower than predicted, this would be indicated with a Yellow or Red on the school or district value-added report.[CLICK]
Facilitator says:So….regardless of the entering achievement level of the group. Whether the group of students is:Higher achieving, Middle achieving,or Lower achieving
Facilitator says:Students should not lose ground academically or receive a yellow or red color indicator.
Facilitator says:A reasonable goal for all groups of students would be to at least maintain their achievement level or meet their predicted achievement level, or in other words, receive a green color indicator[CLICK]
Facilitator says:And…all groups of students, even higher achieving students can receive a light blue or dark blue color indicator. [CLICK]
For Reading and Math in grades 4-8, Growth is measured in Normal Curve Equivalent (NCE) units as scale scores cannot be compared from grade to grade and year to year.The use of NCEs is simply a way to rescaling scale score points so that scores can be compared across different years.Remember, the Growth Measure is NOT reported in years and cannot be referenced in terms of number of year’s growth. It is also NOT reported in percentages and cannot be referenced in terms of an increase or decrease in percentage points.
Facilitator says:For Science, Writing and Keystone content areas, Growth is measured in scale score points. We can compare scale scores in this case because we are comparing a predicted score to an observed score which is different than how growth is measured for reading and math in grades 4-8.Remember, the Growth Measure is NOT reported in years and cannot be referenced in terms of number of year’s growth. It is also NOT reported in percentages and cannot be referenced in terms of an increase or decrease in percentage points.
So using what you know now about PVAAS and academic growth determination, how does this inform us about the academic growth in our district, school, and class?
PVAAS teacher specific reporting estimates the effect of a teacher’s performance on the academic growth of a group of students. PVAAS provides a measure of academic growth for students by taking into account both their endpoint and their entering achievement level. By concentrating on growth, PVAAS puts the emphasis on what educators can influence. For the first time in Pennsylvania teachers will have access to information regarding their influence on the academic progress of students.
First: What teachers will be considered in the Teacher Specific Reporting based on Grade/Subject/Course?
For the purpose of PVAAS teacher-specific reporting, content specific instruction occurs when a teacher has the responsibilityto plan the instruction of the assessed eligible content, provide the instruction of the assessed eligible content, and use the assessment information to evaluate the effectiveness of the instruction of the assessed eligible content on a PA state assessment.Content specific instruction is defined on the Framework for Teaching by both Domains 1 and 3:Content specific instruction is a determination made locally by the LEA.How many teachers in this auditorium will this affect? Raise your hand if this criteria fits you?
IF you are in the 30-40 percent that will be receiving a Teacher Specific Report, here is an example of how your report will be calculated.
Refer to the first 5 reasons indicated on why these 2 students will not be included in the Teacher Specific Reporting. We are now ready to look at the Teacher’s Instructional Responsibility for the rest of the students.
A few examples
IS …The % of a grade/subject/course that a student and teacher are concurrently enrolledTotal Days Possible: Day 1 through last school day before LEA’s state testing window opens for that grade/subject/course. This enrollment will be calculated by PIMS data- the secretaries and Michelle Lill are working towards dataIS NOT…Attendance
Here are some example:You co teach math which provides equal responsibility for Domain 1 and Domain 3- the percentage of responsibility would be 50%- The total Responsibility would be 50% X 100%= 50%You have a special education student who is gets pull out in language arts for 20%, and 20% for reading support. Your instructional responsibility is 60%. The Total is 60%.
The Total Instructional Responsibility has been determine and the percentages are illustrated above. All students who are less than 10% will not be counted in your teacher specific reporting. So now the class of 27 is down to 22 students and the TSR will be on the academic growth of the group of 22 students up to this point.
Now- on to determine the number of Actual students in the class- this is not live bodies, but rather adding up the parts to equal a whole student. You can see the students that have 50% instructional responsibility. These students are added with others to make a WHOLE number count. The group of students is now down to 19 students. There must be at least 6 students or greater to have a report drawn for teacher specific reporting, so in this case Mrs. Smith’s will be getting a teacher specific report on the academic growth of this growth of students.
Copy the timeline specifications for the Administrators.Teachers will have 2 weeks in May to verify correct tested subjects, grades, and Keystone related contentVerify that correct students are attributed to them for each tested subjects, grade or Keystone.Each teacher verifies % students+ Teacher enrollment and % Full or partial Instruction for each student for each tested subject, grade, or Keystone content areaRoll over will occur on May 18th.
