iNACOL Webinar: Blended Learning Program Evaluation
Upcoming SlideShare
Loading in...5
×
 

iNACOL Webinar: Blended Learning Program Evaluation

on

  • 530 views

This webinar focused on current practices for evaluating program effectiveness, the evaluation tools in use and how blended learning schools analyze multiple sources of data to understand program ...

This webinar focused on current practices for evaluating program effectiveness, the evaluation tools in use and how blended learning schools analyze multiple sources of data to understand program success. Presenters will lead a discussion of important considerations around the ongoing formative data collected to inform teachers and administrators about what contributes to student success in online courses. The panelists will explore how their programs approach collection of data and what methodology they use to organize and present data for school or district leaders.

Speakers:
Ernie Silva, Director of External Affairs, SIATech
Elizabeth Hessom, Director of Education Services, SIATech
Mary Esselman, Deputy Chancellor, Instructional Support & Educational Accountability Education Achievement Authority of Michigan

Statistics

Views

Total Views
530
Views on SlideShare
529
Embed Views
1

Actions

Likes
1
Downloads
13
Comments
0

1 Embed 1

https://columbia.blackboard.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Rob <br />
  • Rob <br />
  • KRIS <br />
  • KRIS <br />
  • KRIS <br />
  • KRIS <br />
  • KRIS <br />
  • JOHN <br />
  • JOHN <br />
  • KRIS <br />
  • KRIS <br />
  • KRIS <br />
  • KRIS <br />
  • KRIS <br />
  • KRIS <br /> Top ??% of all participating high schools <br />
  • KRIS <br />
  • KRIS <br /> Talent development philosophy <br /> High expectations for student effort / performance <br /> Clear pathways to student success <br /> Provide frequent, meaningful feedback <br /> Enhance advising <br /> Human scale settings <br />
  • KRIS <br />
  • KRIS <br />
  • KRIS <br />

iNACOL Webinar: Blended Learning Program Evaluation iNACOL Webinar: Blended Learning Program Evaluation Presentation Transcript

  • Approaches to Evaluating Blended Learning Programs • Mary Esselman, Deputy Chancellor, EAA, Michigan • Ernie Silva, Director of External Affairs, SIATech, California • Elizabeth Hessom, Director of Education Services, SIATech January 2014
  • Webinar Format • • • • Overview Introductions Presentation Type questions in the chat window
  • Blended Learning • “a formal education program in which a student learns at least in part through online delivery of content and instruction with some element of student control over time, place, path, and/or pace and at least in part at a supervised brickand-mortar location away from home…” - (Horn and Staker, 2013)
  • Tech-rich = blended
  • Teaching and Learning • What the student is doing and where the student is.  What the teacher is doing and where the teacher is.  What and where the content is.
  • Webinar Focus • Blended Learning Program Evaluation – What ongoing measures are used to measure program effectiveness? – What evaluation tools/processes are used by teachers to determine student and program success? – What evaluation tools/processes are used by administrators and leadership to determine student and program success?
  • Today’s Presenters • Ernie Silva, Director of External Affairs, SIATech, California • Mary Esselman, Deputy Chancellor, EAA, Michigan
  • SIATech Overview • Appropriate Accountability for Drop Out Recovery
  • Job training, mentors & internshi ps + Blended Learning & Competency Based + Adaptive Assessments = Educated Productive Graduates
  • Our programs are recovering drop-outs. It is that simple. 13,186 standardsaligned diplomas earned since 1998. Key lessons have been Key lessons have been learned in the past 15 years learned in the past 15 years serving aa100% drop-out serving 100% drop-out population. population.
  •   Florida California Arizona [N=1241] [N=2110] [N=640] New  Mexico Arkansas [N=261] [N=302] Average Age on  Entry STAR Math  Entry STAR Reading Entry Socioeconomically  Disadvantaged 19.31 G.E.   6.4   [N=1128] 19.44 19.81 19.06 18.77 G.E.  6.4   G.E.   6.0  G.E.   6.1  [N=1805] [N=430] [N=270] G.E.  6.0  [N=282] G.E.   6.9    G.E.  7.2   G.E.   6.7  G.E.   7.4  [N=1133] [N=1806] [N=431] [N=272] G.E.  6.6  [N=304] 88% 100% 100% 98% 100%
  • Educational Achievement Authority (EAA) Overview • Detroit, Michigan
  • dd Hi Hi gg hh EE xx pp ee t ct t at o io nn ss SS l el -fPP aa i ci nn gg aa nn dd M M aa t st r er yy BB aa ss . e. dd l Bl ee nn dd ee dd n “The mission of the EAA, as a catalyst for change, is to disrupt traditional public schooling and provide a prototype for next generation learning.”    Year 1 Year 2 Year 3
  • dd Hi Hi gg hh EE xx pp ee t ct t at o io nn ss SS l el -fPP aa i ci nn gg aa nn dd M M aa t st r er yy BB aa ss . e. dd l Bl ee nn dd ee dd n EAA – Year 1 – Five Pillars 1.Students are grouped by readiness, not by age or grade. 2.Students create and assume ownership for their respective personalized learning paths and are able to communicate their progress relative to their individualized learning goals. 3.Students are allowed to work at their own pace using a blended delivery system to master rigorous standards aligned to next generation readiness. 4.Students provide evidence of mastery through relevant performance tasks and common assessments. 5.Continuous feedback is provided to students, teachers, administrators and parents.
  • dd Hi Hi gg hh EE xx pp ee t ct t at o io nn ss SS l el -fPP aa i ci nn gg aa nn dd M M aa t st r er yy BB aa ss . e. dd l Bl ee nn dd ee dd n EAA – Year 2 • Professional Development was redesigned to follow a blended, personalized delivery model • Schools began to look at new ways to use time, space, talent and resources. • PASE, the SCL Village, the Open Math Lab, and the Learning Pods were all created within EAA schools to allow students to move completely at their own pace and to start leveraging time, space, and talent more efficiently and effectively. • Students began transitioning freely between learning spaces and time. Students are no longer tethered to one classroom and one subject.
  • dd Hi Hi gg hh EE xx pp ee t ct t at o io nn ss SS l el -fPP aa i ci nn gg aa nn dd M M aa t st r er yy BB aa ss . e. dd l Bl ee nn dd ee dd n EAA – Year 3: Next Generation High School • Staffing structures will be redesigned to truly leverage time, talent, and space with a distributed leadership model rather than a traditional principal led school. • Students will demonstrate mastery of Next Generation Competencies that truly prove they are Next Generation Ready. • Staffing structures will allow for multiple adults, including full time and part-time teachers, tutors, paras, instructional assistants, mentors and interventions
  • dd Hi Hi gg hh EE xx pp ee t ct t at o io nn ss SS l el -fPP aa i ci nn gg aa nn dd M M aa t st r er yy BB aa ss . e. dd l Bl ee nn dd ee dd n “The mission of the EAA, as a catalyst for change, is to disrupt traditional public schooling and provide a prototype for next generation learning.” .. EAA Year One EAA Year One Five Pillars: Five Pillars: 1.Students are grouped by 1.Students are grouped by readiness, not by age or grade. readiness, not by age or grade. 2.Students create and assume 2.Students create and assume ownership for their respective ownership for their respective personalized learning paths and personalized learning paths and are able to communicate their are able to communicate their progress relative to their progress relative to their individualized learning goals. individualized learning goals. 3.Students are allowed to work at 3.Students are allowed to work at their own pace using aa blended their own pace using blended delivery system to master rigorous delivery system to master rigorous standards aligned to next generation standards aligned to next generation readiness. readiness. 4.Students provide evidence of 4.Students provide evidence of mastery through relevant mastery through relevant performance tasks and common performance tasks and common assessments. assessments. 5.Continuous feedback isis 5.Continuous feedback provided to students, teachers, provided to students, teachers, administrators and parents. administrators and parents. EAA Year Two EAA Year Two •Professional Development was •Professional Development was redesigned to follow aablended, redesigned to follow blended, personalized delivery model personalized delivery model •Schools began to look at new ways •Schools began to look at new ways to use time, space, talent and to use time, space, talent and resources. resources. •PASE, the SCL Village, the Open •PASE, the SCL Village, the Open Math Lab, and the Learning Pods Math Lab, and the Learning Pods were all created within EAA schools were all created within EAA schools to allow students to move completely to allow students to move completely at their own pace and to start at their own pace and to start leveraging time, space, and talent leveraging time, space, and talent more efficiently and effectively. more efficiently and effectively. •Students began transitioning freely •Students began transitioning freely between learning spaces and time. between learning spaces and time. Students are no longer tethered to Students are no longer tethered to one classroom and one subject. one classroom and one subject. EAA Year Three EAA Year Three Next Generation High School Next Generation High School •Staffing structures will be redesigned to truly •Staffing structures will be redesigned to truly leverage time, talent, and space with aa leverage time, talent, and space with distributed leadership model rather than aa distributed leadership model rather than traditional principal led school. traditional principal led school. •Students will demonstrate mastery of Next •Students will demonstrate mastery of Next Generation Competencies that truly prove Generation Competencies that truly prove they are Next Generation Ready. they are Next Generation Ready. •Staffing structures will allow for multiple •Staffing structures will allow for multiple adult including full time and part-time adult including full time and part-time teachers, tutors, paras, instructional teachers, tutors, paras, instructional assistants, mentors and interventions assistants, mentors and interventions
  • SIA Tech Assessment
  • •• Achievement scores on state Achievement scores on state mandated exams penalize mandated exams penalize SIATech students, because the SIATech students, because the majority enter our schools majority enter our schools achieving at the 6th grade level achieving at the 6th grade level in reading & math. in reading & math. •• State tests do not work well State tests do not work well for us for us •• Prescribed time table Prescribed time table •• Limited administrations Limited administrations •• Too hard/too long for too Too hard/too long for too many many SIATech schools needed a different way to demonstrate accountable for student learning!
  • Advocating for Alternative Accountability Reaching At Promise Students Association http://www.rapsa.org Save the date: November 14-15 San Diego 3rd Annual Alternative Accountability Policy Forum
  • •• Adaptive Adaptive •• Multiple administration times Multiple administration times •• Short Short •• Correlated to state standards Correlated to state standards •• Reliable Reliable •• Measure & report growth and achievement Measure & report growth and achievement •• Can use results to improve learning and Can use results to improve learning and teaching teaching Renaissance STAR Math and Reading*
  • What is learning growth? • Student learning growth is a statistical measure that estimates how much each student progresses from one test to the next compared to similar students CST ELA SCALE SCORE 600 400 Advanced 349 Proficient 15 25 299 Below Basic 150 330 Basic 270 345 Far Below Basic 305 Grade 6 Grade 7 Grade 8
  • Benefits of measuring learning growth Benefit #1 – Holds schools accountable for the things they have control over Benefit #2 – Measuring learning growth fosters student hope and engagement because gains are based on individual progress – not unrealistic targets Benefit #3 – Measuring student learning growth sets high expectations for every student, because every student (high, middle, and low achieving) must increase his or her learning every year
  • Students take ownership of personal learning as schools make efficient use of core processes •• Interpretation of Individual Interpretation of Individual Student Growth Data Student Growth Data •• Individual Student Goal Setting Individual Student Goal Setting (weekly, biweekly, monthly) (weekly, biweekly, monthly) •• Student Accountability for Student Accountability for Tracking and Adjusting Tracking and Adjusting Academic and Learning Academic and Learning Progress Progress •• Evaluation of Course Evaluation of Course Completion and Academic Completion and Academic Growth Growth
  • •• •• •• •• Annual Development of SITE Professional Development Plans Interpretation of Learning Interpretation of Learning Data to determine general Data to determine general success and gaps in school success and gaps in school outcomes. outcomes. Site Goal Setting Site Goal Setting Determination of a Site Determination of a Site Professional Development Professional Development Focus Focus Evaluation of Plan Evaluation of Plan Implementation and Results Implementation and Results (EOY) (EOY)
  • Annual Development of Individual Staff Member Professional Development Plans •• Interpretation of Learning Data as Interpretation of Learning Data as related to specific content area(s). related to specific content area(s). •• Individual Staff Goal Setting Individual Staff Goal Setting •• Planning Pertinent Training Planning Pertinent Training (examples) (examples) •• Lesson Plan Development Lesson Plan Development •• Differentiation Differentiation •• Knowledge Acquisition Knowledge Acquisition •• Teaching Strategies Teaching Strategies •• Data Driven Instruction Data Driven Instruction •• Evaluation of Plan Implementation Evaluation of Plan Implementation and Outcomes (EOY) and Outcomes (EOY)
  • • Monthly Content Teacher Conversations Wednesdays 3:30-4:30 pm Math, ELA, Science, Social Studies, CTSP (4/year) Sharing of best content practice with a focus on: - literacy (reading and writing across content areas) - small group lessons - opening and closing whole group lessons/activities
  • • Monthly distribution of active student learning data • Research into a computer based, individual student reading program (included into 2013-14 budget for all sites)
  • Multiple Measures Include Gallup Poll of Hope, Engagement and Well Being
  • Hope the ideas and energy we have for the future. Hope drives: attendance, credits earned, and high school GPA Hope predicts: college GPA and retention Hope scores are better predictors of college success than high school GPA, SAT, and ACT scores.
  • HOPE Ways around a problem Response Average Average student response on a five point scale, where 1=strongly disagree and 5=strongly agree. Question Text: Question Text: I Ican find lots of ways can find lots of ways around any problem. around any problem. (Five point scale) (Five point scale) Possible Student Activities to Impact this Area: During direct instruction problem-solving, ask students for other ways they would solve the problem or issue. Remind students of the variety of ways they can get help and support for problem-solving Help students learn that there are many ways to success and learning (not just innate talent-there are MANY talented but unsuccessful folks) teach multiple problem solving techniques encourage students to learn how they ‘best learn of learn best’ provide students with specific strategies to deal with their ‘blocks’ (e.g. memorizing, planning, organizing, classifying, preparing, …) Your Reflections / Actions for this Area:
  • Engagement the involvement and enthusiasm for school. Engagement scores separate high-performing from low-performing schools
  • ENGAGEMENT Schoolwork important Response Average Response Average Average student response on a five point scale, where 1=strongly disagree and 5=strongly agree. Possible Student Activities to Impact this Area: Question Text: Question Text: My teachers make me My teachers make me feel my schoolwork is feel my schoolwork is important. important. (Five point scale) (Five point scale) Teachers provide frequent, sincere, specific feedback on schoolwork in a variety of ways Teachers make relevant connections between schoolwork and goal attainment; Students are provided with opportunities to discuss with teachers and/or mentors how their schoolwork is helping them… Your Reflections / Actions for this Area:
  • Well-Being how we think about and experience our lives. Well-being is an indicator of how students are currently doing. Well-being also predicts future success. A thriving student earns 10% more credits and a 2.9 GPA compared to a struggling or suffering student who completes fewer credits and earns a 2.4 GPA.
  • WELL-BEING Treated with respect Percent who said Yes Possible Student Activities to Impact this Area: Question Text: Question Text: • • • • Were you treated with Were you treated with respect all day respect all day yesterday? yesterday? (Yes or no) (Yes or no) • • ALL staff knows and USES student names Be and Do what you want to see from students Explain the reasons/why or why not for critiques of student work, not just X Reward and/or acknowledge acts of respect to build a culture Demonstrate positive approach to classroom management and student discipline (preserve dignity) Find ways to help students answer their own questions and then honor their success Your Reflections / Actions for this Area:
  • EAA Assessment
  • Program Evaluation Questions Question 1: To what extent is the studentcentered system being executed as planned? Question 2: To what extent is there sufficient support available to staff, parents, and students who are participating in a student-centered system of education? Question 3: To what extent is implementation of the student-centered system achieving desired changes in student performance outcomes, socioemotional well-being and teacher practice? Question 4: To what extent is the program being scaled in terms of depth, sustainability, spread, shift, and evolution?
  • 4-pronged Assessment System   Individual Growth (Performance Series) Performance Tasks to assess Deeper Learning (3 pieces of evidence for each Learning Target) Learn (reflection)  Practice (Understanding)  Apply (real-world connections) Common Assessments    Achievement (State Assessment) Next Generation Ready—Graduation by Defense
  • Integrated Teaching & Learning Platform . Real-time Analytics • Supports a robust collection of open sourced, licensed, and teacher created resources. • Learning objects parsed in bite sized chunks. • Digital assets mapped to standards--local, state, common core. • Curriculum maps directly linked to the digital assets • Students have access to choicebased learning pathways. • Dashboards provide data for students, parents and teachers. • Data is provided as a prescriptive newsfeed for immediate application. • Integrated planner, profiles, personalized learning plans with student, teacher, parent, communication tools. • Peer to Peer support system.
  • Part 2: Access to Information— monthly, weekly, real-time      Time & Topic Pace/Progress Badges for leadership, scholarship, character, service personal milestones Student Profile Mastery:    School/Class/Individual by Level/Unit/Standard Mastered/In progress/Not Yet Attempted Digital Portfolio of Evidence
  • EDUCATION ACHIEVEMENT AUTHORITY OF MICHIGAN EDUCATION ACHIEVEMENT AUTHORITY OF MICHIGAN The teaching and learning platform: The student experience…
  • Student Progress Page
  • Creating a personalized learning path
  • The Teacher Experience: 3-D Classroom Radar Report of Student Productivity—who’s progressing and who is not vs. traditional 2-D, linear gradebook. Color-coding on students represents Color-coding on students represents self-assessment information self-assessment information Click a student Click a student to drill down to to drill down to view real-time view real-time personal personal performance performance
  • Classroom Summary Report Click a student to drill down to view Click a student to drill down to view real-time personal performance real-time personal performance Pie charts indicate mastery of Pie charts indicate mastery of individual learning targets. individual learning targets. Self-assessment results Self-assessment results Teachers can click to select Teachers can click to select individual or groups of individual or groups of students to assign students to assign supplemental resources supplemental resources and/or send notes. and/or send notes.
  • Standards Report Teachers, students and Teachers, students and parents can review parents can review progress against progress against individual standards as individual standards as well as access well as access supplemental study supplemental study resources. resources.
  • Teacher Grading and Messaging Center Item for grading are collected in Item for grading are collected in the to-do list making it easy for the to-do list making it easy for teachers to provide grades and teachers to provide grades and feedback. feedback.
  • Teacher Personal Progress Just as teachers can monitor real-time Just as teachers can monitor real-time student progress they also can monitor their student progress they also can monitor their own progress. own progress.
  • Lessons Learned - EAA
  • Part 3: Other Assessments of Progress/Implementation     End of Course for non-tested Subjects GRIT Value-added Growth Model for Pay for Performance S ystem Instructional Practice—scaffolded walkthroughs     Classroom Set-up/Visual Cues Classroom Structures Rituals/Routines Student Ownership/Engagement Note: We triangulate data across reports/surveys/observations/focus groups
  • Lessons Learned – SIA Tech
  • 1. A Learning Growth model is imperative for School Evaluation/Monitoring of reengaged students. 2. Graduation Rate Calculations must be appropriate in terms of cohort. 3. Strong advocacy work at the local, state and federal level needs to be a priority. 4. Competency-based education needs engaging, specialized and customized curriculum to truly accelerate learning 5. Staff development and training is crucial to success with a dropout population 6. A robust data system needs to be in place in order to show success in multiple, meaningful ways.
  • Contact Information • Mary Esselman, EAA. – MEsselman@eaaofmichigan.org • Ernie Silva, SIATech – Ernie.Silva@siatech.org • Elizabeth Hessom, SIATech – Elizabeth.Hessom@siatech.org