An Empirical Investigation of Ohio’s Educator Evaluation System (MWERA)

•Download as PPTX, PDF•

1 like•371 views

Presented at the Mid-western Educational Research Association Annual Meeting Nov. 6-9 2013 by Anirudh V.S. Ruhil, Marsha S. Lewis, Lauren Porter, and Emily A. Price

Education Health & Medicine

An Empirical Investigation of Ohio’s
Educator Evaluation System

Authors:

Anirudh V.S. Ruhil (Ohio University) <ruhil@ohio.edu>
Marsha S. Lewis (Ohio University) <lewism5@ohio.edu>
Lauren Porter (The Ohio State University) <porter.700@osu.edu>
Emily A. Price (Ohio University) < ep311508@ohio.edu>

Overview
• Two question motivate our study:
– What does Ohio’s educator evaluation system look like?
– Any systematic patterns (conditionalunconditional)?

•
•
•
•

OTESOPES
eTPES Data
Analysis and Findings
Conclusions

MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

OTES (2014-2015)
(1) Performance Standards (50%)
-- Developed by OH Educator Standards Board
-- Seven Components
1: Students
2: Content
3: Assessment
4: Instruction
5: Learning Environment
6: Collaboration and Communication
7: Professional Responsibility and Growth

(2) Student Growth Measures (50%)
-- Value-Added
-- Approved Vendor Assessments
-- LEA Measures

MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

Student Growth Measures in OTES
(A) Value-Added
-- Grades 4th – 8th, ELA & Math
-- Until June 30, 2014, majority (>25%) of SGM shall
be based on Value-Added
-- On or after July 1, 2014, all (50%) of SGM shall be
based on Value-Added.

(B) Approved Vendor Assessments
-- Terra Nova
-- ACT End-of Course
-- NWEA MAP
-- STAR
-- …

(C) LEA Measures
-- Student Learning Objectives
-- Shared Attribution
-- LEASchool-level Value-Added
-- LEASchool-level SLO
MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

OPES (2014-2015)
(1) Performance Standards
-- Shared vision, establish goals, and
continuous improvement
-- HQ instruction -> increased student
achievement
-- Manage resourcesoperations
-- Establish collaborating learning and
shared leadership
-- Engage parents and community

(2) Student Growth Measures
-- Value-Added
-- Approved Vendor Assessments
-- LEA Measures
MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

Student Growth Measures in OPES
(A) Value-Added
-- School-level Value-Added
(B) Approved Vendor Assessments
-- School-level Composite Measure
-- School-level Aggregate of AVA scores
(C) LEA Measures
-- District SLOs
-- District Value-Added
-- Aggregate of Teachers’ Value-Added Scores
-- Student Achievement Trends
-- Progress on Improvement Plans
-- Student Course-Taking Trend (e.g., AP)
MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

OTESOPES Final Rating Determination
Performance Standards

SGM Rating

4
Above

3

Accomplished Accomplished

2

1

Skilled

Developing

Expected

Skilled

Skilled

Developing Developing

Below

Developing

Developing

Ineffective Ineffective

MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

OTESOPES Data for 2012-2013
•

OTES – 26 LEAs = 23 PSDs + 1 JVSDs + 2 CSs

Ineffective

Skilled

Accomplished

Total

13 (0.6%)
•

Developing
298 (12.8%)

1,580 (67.8%)

441 (18.9%)

2,332 (100%)

OPES – 27 LEAs = 24 PSDs + 1 JVSD + 2 CSs

Ineffective

Skilled

Accomplished

Total

0 (0%)
•

Developing
22 (12.9%)

98 (57.3%)

51 (29.8%)

171 (100%)

After exclusions for LEAs piloting OTESOPES we have
– 24 LEAs ~ 2,001 Teacher records
– 15 LEAs ~ 62 Principal records
– These sub-sample sizes drop further once “Exempt” records are excluded

MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

OTESOPES Final Summative Ratings

MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

OTES: Performance Rating by SGM Rating

MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

OTES:
Distribution
of SGM
Ratings
Across
Performance
Standard
Ratings
MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

OTES: SGM Category by Rating

MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

OTES: Value-Added Weight by Rating

MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

OTES: SLO Weight by Ratings

MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

Initial Conclusions
• Value-Added fairly congruent with other evaluation measures
• Weight placed on Value-Added seems to be of no consequence for final
summative rating
• Our early results in line with NYC (http://bit.ly/16qYVYg) … possibly other
states as well.
• Limitations
– Limited data at hand
– Potentially biased set of LEAs studied
– Value-Added has been well studied; More research is needed for Vendor Assessments &
SLOs
– Questions of OTESOPES reliability are only answerable with multiple waves of data

MWERA Annual Meeting 2013 (Evanston, IL)

11/08/13

This document discusses the University of the Highlands and Islands' efforts to create a single operating framework for its diverse higher education programs, including those involving the Scottish Qualifications Authority (SQA). It notes that UHI holds maximum devolvable powers from SQA for key quality assurance functions. The challenges of coordinating SQA programs across 13 academic partners are discussed. Changes made to streamline SQA quality assurance processes and oversight are outlined, such as combining roles, establishing committees, verifying systems, and revising procedures. Improvements to mitigating circumstances policies, core reporting, and network progression boards are also summarized.

Learning Analytics - Sweden

EduSkills OECD

Naace Strategic Conference 2009: The Next Generation of assessment - GL Asses...

Naace Naace

This document discusses assessments in schools and the evolution from standardized tests to assessing pupils' progress (APP). It summarizes GL Assessment's role in providing both formative and summative assessments online and through their Testwise system. Key points covered include the phasing out of KS3 SATs, implementing APP across all subjects by 2011, and using periodic and transitional assessments to make holistic judgments of student progress.

Governing Complex Education Systems: The Use of Data, Tracey Burns, OECD

EduSkills OECD

"Challenging traditional program design and delivery to better support wideni...

Association of University Administrators

Professor Helen Marshall, Vice-Chancellor of University of Salford, delivered a keynote focused on delivering a better support of widening participation in Universities. Current agendas that seek to promote “widening participation” and the development of skilled as well as knowledgeable graduates mean that universities need to take a step back to re-examine and challenge traditional curriculum design and delivery models. This paper explores the current landscape and requirements such as modularisation, credits, learning outcomes and levels and how those support and/or constrain curriculum design and delivery that engages and develops students who come from backgrounds that are mixed in terms of previous educational experience, socio-economic class and cultural heritage. The central theme is to challenge perceptions that certain approaches to curriculum design and delivery are not possible because of these requirements.

Institution wide assessment practice-connections2014conferencepresentation

Linda Townsend

This document discusses strategies for developing a culture of assessment at the institution-wide level. It outlines challenges faced and 15 elements of successful assessment programs. The author then details Longwood University's strategies, including faculty ownership of assessment, comprehensive program review, ongoing professional development, information sharing, celebrating successes, and continuous improvement. Other strategies are discussed for meeting challenges and further fostering an assessment culture.

Pam Muth and Lisa Bolton: Optimising QILT to improve the student experience

Studiosity.com

The document discusses optimizing the Quality Indicators for Learning and Teaching (QILT) program to improve the student experience. QILT administers the Student Experience Survey (SES) and Graduate Outcomes Survey (GOS) to measure student engagement, teaching quality, and graduate employment outcomes. The SES collects data from current students on their educational experience. Results are available on the QILT website to allow students to compare institutions. Institutions receive detailed SES results and can integrate QILT data into strategic planning to monitor performance indicators over time. Customizing QILT surveys allows institutions to address questions not covered and better evaluate specific strategic initiatives.

This document discusses improvements made to module evaluations at the university. It provides insights from students on the changes, which included improved communication about the purpose and importance of evaluations, dedicated class time to complete them, and ensuring students saw the impact of their feedback. The students union felt partnership was key in effectively delivering the evaluations. Overall, the changes led to higher student engagement and participation in the surveys.

Developing Data Systems - Portugal

EduSkills OECD

2008 regional educational laboratory board of directors (rel midwest)

Christopher Thorn

This document summarizes Dr. Christopher Thorn's work assisting multiple projects using value-added analysis to link state data systems to local education reforms. It discusses technical challenges around integrating human resources, student, and financial data systems to support performance evaluation and decision making. Core issues identified include weaknesses in linking student and teacher data, a lack of diagnostic assessments, and analytical gaps that limit the usefulness of accountability systems. The document argues for more formal approaches to problem solving these complex problems through interdisciplinary collaboration.

Role of school manager

International advisers

The document outlines the key roles and responsibilities of a school manager according to Hallinger (2005), which include creating a shared vision and high expectations, guiding continuous improvement, monitoring curriculum and learning outcomes, and being a visible presence. It also discusses three major indicators for evaluating schools: performance based on dropout rates, passing rates and attendance; potential based on school climate, experienced faculty and resources; and achievement based on exam scores and proficiency standards.

Institution wide assessment practice-sacscoc2013conferencepresentation

Linda Townsend

This document discusses strategies for developing a culture of assessment at institutions of higher education. It outlines challenges such as gaining faculty ownership of assessment and providing ongoing professional development. Some of the stepping stones Longwood University has taken include establishing faculty committees focused on assessment, conducting comprehensive program reviews, and offering workshops and coaching to support assessment. The university also celebrates assessment successes through an annual conference and provides mini-grants to encourage assessment research. Developing a culture of assessment is an ongoing journey that requires continuous improvement efforts.

Midlands Conference 2015: Strategic responses presentation

Association of University Administrators

The document discusses strategic challenges facing universities in a competitive environment. It outlines two main challenges: 1) managing supply and demand of students in a capped market with increasing alternative providers, and 2) addressing the perception of students as consumers who evaluate the value of tuition. Universities are responding strategically with investments in student experience, campus redevelopments, and improving employability and satisfaction metrics. The higher education sector is predicted to become more market-driven, competitive, and unpredictable for universities.

Blackboard Analytics for Learn: A recipe for success

Richard Stals

So much of the current discussion around Learning Analytics seems to be caught up in the realm of Big Data that informs the top executives and decision makers who are shaping institution-wide strategies. While these kinds of topics need to be explored, truly significant and transformative uses of learning analytics can be had at the grassroots level of the teacher and student. This session will look at how Edith Cowan University is using Blackboard Analytics for Learn to empower staff and students with their own data, allowing them to make informed and timely decisions in their own teaching and learning journeys. We will explore how learning analytics data enables staff to do things like identify and support students at risk of disengaging from the course early, monitor how students are actually engaging in their course and collect real evidence on student interactions that informs a continual process of improvement in learning design and resources.

Ian Scott and Roger Grew - Module Evaluation at Oxford Brookes

Association of University Administrators

This document discusses module evaluation at Oxford Brookes University. It describes the process of module evaluation and some common issues, such as lack of demonstrated validity, biased samples, and patchy use of evaluation data. The university implemented an electronic module evaluation system integrated with Moodle and a business intelligence tool. This allows evaluation data to be automatically included in program review dashboards. While module evaluation is intended to enhance quality and provide student feedback, there is no clear evidence it improves overall student satisfaction at the institutional level.

LTAG Horizon Scanning

Anne-Marie Scott

The document discusses learning analytics, which involves measuring, collecting, analyzing, and reporting data about learners and learning environments to understand and optimize the learning process. Learning analytics can be used to monitor learner activity and progress, predict outcomes, and enable interventions. For the University of Edinburgh, learning analytics could increase student satisfaction and overall survey ratings. However, opportunities are balanced with barriers like determining what data is most useful, defining student success, ensuring ethical practices and data literacy, and understanding impacts on students. Usage of the university's virtual learning environments has increased but opportunities also exist in analyzing library, computing, and wireless network usage data.

Education Data in England

EduSkills OECD

The role of assessment data in Austria

EduSkills OECD

Challenges of Quality in Teaching and Learning - Gwen van der velden

Association of University Administrators

Embedding into the Nursing Community on an Academic Health Center Campus - MA...

Virginia Commonwealth University

Objective: To describe and evaluate the process by which a librarian became embedded into the nursing community on an academic health sciences center campus. These efforts will be evaluated through qualitative and quantitative measures to determine the success of serving the nurses on campus and to address potential improvements. Setting/Participants/Resources: An urban academic health sciences library serving the schools of allied health professions, dentistry, medicine, nursing, pharmacy as well as a magnet designated health system. The nursing population on campus is made up 900 students and 142 faculty/staff in the school of nursing, and approximately 1900 nurses in the health system. Method: The library’s goal is to transition from library-based liaisons to an embedded model of librarianship. Initial steps included engaging faculty, staff and students in discussions about their research and clinical practice needs. A model for becoming embedded was developed from these discussions. New services include customized bibliographic instruction sessions, onsite office hours in the school of nursing, virtual and face-to-face consultations, individualized journal citation alerts and subject alerts, and creating a social media presence on twitter and a blog. Service on School of Nursing and Health System committees provides greater insight and further contributions. Being proactive in addressing the research needs of the faculty and staff has led to assisting with literature reviews, grant proposals, quality improvement projects, magnet reacreditation in the hospital, and other endeavors.

Development Conference 2014, The Professional Behaviours Framework,Ian Darker...

Association of University Administrators

This document discusses the role of professional behaviors and leadership at the Royal Veterinary College (RVC). It provides background on the historical origins of the RVC and describes its current facilities and student/staff composition. The document advocates for using a Professional Behaviors Framework to promote reflection, resilience, and flexible thinking among teams in order to help the RVC adapt to a changing environment. It also discusses the importance of leadership and motivation among RVC staff and students.

Thinking about Governance: A Story and some Dimension Lines

EduSkills OECD

Meeting the professional development and social connectedness needs of a disp...

todocontexto

The document discusses the challenges of supporting a dispersed online faculty and the initiatives taken by Western Governors University to address these challenges. It outlines tools like a learning community platform and in-service sessions used to foster connection and professional development among remote faculty. Effectiveness measures showed that these efforts improved faculty participation, growth, and student service metrics over a 12 month period.

EVAورشة / آجي كسونكا رئيسة وحدة

IEFE

This document discusses using evaluation as a tool for quality improvement in education. It defines evaluation and quality assurance, and presents three types of evaluation: (1) Accreditation, which assesses whether institutions meet predefined criteria through a review process; (2) Multi-method program evaluations, which broadly evaluate implementation, challenges, and best practices; (3) School-level evaluation of teaching practices through self-evaluation and analysis of student outcomes and achievements. While external reviews provide accountability and identification of strengths and weaknesses, they may be costly and perceived as controlling. Program evaluations disseminate knowledge but do not enhance accountability or provide school-specific guidance. A sound evaluation system balances accountability and improvement through various appropriate evaluation models.

Va 101 ppt

Akron Public Schools

The document provides an overview of Ohio's value-added assessment and accountability system, including: 1) It describes the development of Ohio's two value-added models (URM and MRM) and the unification into a single system to expand teacher-level reporting statewide. 2) It summarizes recent changes and enhancements to Ohio's value-added reports, including the addition of subjects, inclusion of student historical data across districts, and expanded report views. 3) It discusses the role of value-added analysis in Ohio's accountability system and teacher evaluations under new laws.

Awsp Seattle Hilton May 2013

Glenn E. Malone, EdD

The document discusses principal evaluation, including: 1) The history and current status of principal evaluation in Washington state, with multiple new laws and guidelines passed since 2010. 2) Recommendations and challenges around calibrating evaluation elements to determine summative scores and collecting evidence for student growth areas. 3) An overview of the AWSP Leadership Framework that is intended to support principal development through criteria, rubrics, and research.

What's hot

Using Ping SSO to Boost Student Engagement

Dave Shields

Developing Data Systems- Slovenia

EduSkills OECD

Financial Aid Policy Changes Mean Early Career Development

Elizabeth Cole, M.A., LPCC, BCC

Helen Carmichael, Matt Grange and Matt Linguard - Working with Students

Association of University Administrators

Developing Data Systems - Portugal

EduSkills OECD

2008 regional educational laboratory board of directors (rel midwest)

Christopher Thorn

Role of school manager

International advisers

Institution wide assessment practice-sacscoc2013conferencepresentation

Linda Townsend

Midlands Conference 2015: Strategic responses presentation

Association of University Administrators

Blackboard Analytics for Learn: A recipe for success

Richard Stals

Ian Scott and Roger Grew - Module Evaluation at Oxford Brookes

Association of University Administrators

LTAG Horizon Scanning

Anne-Marie Scott

Education Data in England

EduSkills OECD

The role of assessment data in Austria

EduSkills OECD

Challenges of Quality in Teaching and Learning - Gwen van der velden

Association of University Administrators

Embedding into the Nursing Community on an Academic Health Center Campus - MA...

Virginia Commonwealth University

Development Conference 2014, The Professional Behaviours Framework,Ian Darker...

Association of University Administrators

Thinking about Governance: A Story and some Dimension Lines

EduSkills OECD

Meeting the professional development and social connectedness needs of a disp...

todocontexto

EVAورشة / آجي كسونكا رئيسة وحدة

IEFE

What's hot (20)

Using Ping SSO to Boost Student Engagement

Developing Data Systems- Slovenia

Financial Aid Policy Changes Mean Early Career Development

Helen Carmichael, Matt Grange and Matt Linguard - Working with Students

Developing Data Systems - Portugal

2008 regional educational laboratory board of directors (rel midwest)

Role of school manager

Institution wide assessment practice-sacscoc2013conferencepresentation

Midlands Conference 2015: Strategic responses presentation

Blackboard Analytics for Learn: A recipe for success

Ian Scott and Roger Grew - Module Evaluation at Oxford Brookes

LTAG Horizon Scanning

Education Data in England

The role of assessment data in Austria

Challenges of Quality in Teaching and Learning - Gwen van der velden

Embedding into the Nursing Community on an Academic Health Center Campus - MA...

Development Conference 2014, The Professional Behaviours Framework,Ian Darker...

Thinking about Governance: A Story and some Dimension Lines

Meeting the professional development and social connectedness needs of a disp...

EVAورشة / آجي كسونكا رئيسة وحدة

Similar to An Empirical Investigation of Ohio’s Educator Evaluation System (MWERA)

Va 101 ppt

Akron Public Schools

Awsp Seattle Hilton May 2013

Glenn E. Malone, EdD

Educator Effectiveness in Wisconsin (DPI)

Wisconsin Education Association Council

The document summarizes Wisconsin's Educator Effectiveness system which was designed to evaluate teachers and principals using multiple measures of educator practice and student outcomes. It describes the purpose of developing the system to identify and support educator effectiveness. It outlines the key parts of the system including the standards and rubrics used, how student and educator outcomes are measured, how the evaluation process is managed through technology, and the timeline for implementing the system statewide.

Karthik Muralidharan on research on achieving universal quality primary educa...

Twaweza

Reviewing the Research and PEAC Recommendations around Principal Evaluation

Richard Voltz

Educator evaluation policy overview-final

Research in Action, Inc.

Euclid City Schools DLT Presentation Feb 9 2009

tlysiak

This presentation was conducted by the Euclid City Schools' District Leadership team. Speakers were: Superintendent, Dr. Joffrey Jones, Principal, Dr. Charlie Smialek, Teachers Airel Townes and Margo Smolic, and State Support Team members Paula Woods and Ross May. The purpose of this presentation is to provide an overview of the actions the District Leadership Team has taken while examining data and forming goals.

Principal and-teacher-evaluation-key-ideas-your-role-and-your-school's-leadin...

Winnie de Leon

This document provides an overview of principal and teacher evaluation processes in Mississippi, including: - The Mississippi Principal Evaluation System (MPES) uses student achievement goals, organizational goals, and staff surveys to evaluate principals. - Teacher evaluations incorporate classroom observations, student growth objectives, and professional growth goals. Non-tested teachers use student learning objectives to measure student growth. - Principal and teacher evaluations are interconnected, as principals and teachers must work together to achieve schoolwide student growth. - "Leading indicators" and "lagging indicators" help define goals and measure progress, with leading indicators more easily influenced during the school year. - Counselors can support the new evaluation processes by fostering

Documents oerc_160913_va_symp_lindsey_lewis-final

An Empirical Investigation of Ohio’s Educator Evaluation System (MWERA)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to An Empirical Investigation of Ohio’s Educator Evaluation System (MWERA)

Similar to An Empirical Investigation of Ohio’s Educator Evaluation System (MWERA) (20)

More from Ohio Education Research Center

More from Ohio Education Research Center (20)

Recently uploaded

Recently uploaded (20)

An Empirical Investigation of Ohio’s Educator Evaluation System (MWERA)

Editor's Notes