On an ending note, I want to share a short video from a new course we are offering at St. Gregory, Design Build Te ch Innovation.
It lies instead in the “adjacent possible.” SJ
Anecdote of Maybeck Teacher
Acis assessment presentation for posting
Learning Assessment for the 21st Century Jonathan E. Martin www.21k12blog.net firstname.lastname@example.org ACIS Heads, January 2013
“How do we measure what wevalue rather than value what wemeasure?”
My interest because:• I want to experiment and innovate more, and evaluate progress on those experiments.• I want to reassure anxious parents and trustees that we are not jeopardizing academic excellence as we reinvent 21st century learning.• I want to be really sure we are teaching the skills kids need, 21st century capacities and digital literacies.• I loved my independent school education and yet I see so many ways it could have been so much more.• I ran under-enrolled schools and had to show competitive advantage.
Kevin Mattingly, dean of faculty at LawrencevilleSchool (NJ):“We talk about 21st century skills andconceptual frameworks, and we say, this is whatour kids need!How, then, do we find out and evaluate whatthey know, probing empathetically and fair-mindedly?”
Less Actually, More Uncertainty Measurement isn’t going away: it isn’t likely to diminish in importance in the coming years. Governments, Funders, Foundations, Parents/Consumers, Boards
Our Goal Today:Considering the purposes served and valueprovided by a variety of (mostly) newassessment tools and techniques which mightbetter evaluate and advance the learning goalsof your mission, and in particular 21st centurycapacities and differentiated instruction.
I Intro and OverviewII External Measurements of Mission Short Activity 3 Tools I’ve used 4 Tests on the HorizonBreakIII Developing 21st c. Internal Assessments of Learning 3 Approaches Short ActivityIV Exploring Non-Cog Assessment Short ActivityV Digital Portfolios and Demonstrations of Learning Short ActivityVI Q & A
Commission on Accreditation Criterion 13 The standards require a school to provide evidence of a thoughtful process, respectful of its mission, for the collection and use in school decision-making of data (bothexternal and internal) about student learning
ACIS Standard D9:The school shall have a curriculum that isarticulated in detailed written form, and a corresponding process to assess individualstudent growth, development and achievementthat reflects the schools mission.
• Does the school have a process for assessing student growth, development, and achievement that is consistent with the mission? (See related standards A7, B1, D1.)• Do the assessment methods correspond with the curriculum so that teachers have a means for tracking student progress in all areas of the educational program?• Does the assessment process support effective communication with parents about their children’s learning? (See related standard C5, and D1.)• Does the assessment process give teachers the information needed to help them provide appropriate instruction that meets the capacities, styles, and developmental needs of the students enrolled in the school? (See related standard D1, D4).
Twenty-first century readers and writers need to• Develop proficiency with the tools of technology• Build relationships with others to pose and solve problems collaboratively and cross-culturally• Design and share information for global communities to meet a variety of purposes• Manage, analyze and synthesize multiple streams of simultaneous information• Create, critique, analyze, and evaluate multi-media texts• Attend to the ethical responsibilities required by these complex environments
NETP Assessment: Measure What Matters: – Technology-based assessments that combine cognitive research and theory about how students think with multimedia, interactivity, and connectivity make it possible to directly assess these types of skills. – This can be done within the context of relevant societal issues and problems that people care about in everyday life. – Systems can be designed to capture students’ inputs and collect evidence of their knowledge and problem- solving abilities.
Often an important decision requires betterknowledge of the alleged intangible, but whenan executive believes something to beimmeasurable, attempts to measure it will noteven be considered. As a result, decisions are less informed thanthey could be. The chance of error increases.Resources are misallocated, good ideas arerejected, and bad ideas are accepted.measurement is a quantitativelyexpressed reduction ofuncertainty based on one ormore observations.
Longitudinal Data• The National Clearinghouse has 93% of all US colleges collecting and providing longitudinal data• Two independent school associations require student tracking for accreditation (freshman GPA):• CAIS Canada email@example.com• ISASW www.isasw.org Lyons/Niebold
New and Next Gen. External Measurements of learning
Questions• Assessment of learning Does the assessment respect our mission and measure what we value? Does the assessment measure 21st century skills and/or digital literacies? Does it provide data we can use for accountability, communications?• Assessment as learning Are our students meaningfully engaged in the experience of the assessment? Is taking the test itself a learning experience.• Assessment for learning Does the assessment effectively provide data and resources we can use for continuous improvement of learning?
Sample CWRA and HSSSE resultsHow could each of these assessment reports be used to support and strengthen the program at your school? KentGuest
Measuring Key Thinking and Communicating Skills
• “CAE is doing truly groundbreaking work in developing assessments of the skills that matter most in the 21st century.” – Tony Wagner
Three Metrics Internal growth at a schoolComparison to other participating schools College readiness
CWRA/CLA median percentile, College Freshman Normed 97 67 50All College Freshmen St. Gregory Freshmen St. Gregory Seniors
Measuring Student Perspective on Learning Key 21st c. Skills
Engagement Matters- and there is excellent engagement data Not just about Engagement
student, teacher and parent surveyscollect data and generate reportsfocused on school climate, classroomconditions, and student engagement 34
What Works in the Classroom? Ask the Students Teachers whose students described them as skillful atmaintaining classroom order, at focusing their instruction and at helping their charges learn from their mistakes are often the same teachers whose students learn the most in the course of a year, as measured by gains onstandardized test scores, according to a progress report on the research.
II. How do the results inform practice? (cont.)• HSSSE complements assessment tests and performance data by identifying school features that affect outcomes• HSSSE stimulates discussion on effective teaching and learning – Many schools use HSSSE data during PLCs or other PD activities to identify areas of improvement to include in School Improvement Plan – Can help in accreditation process which typically requires self-reflection• Can be used in the fall and spring to investigate trends and changes in student engagement over the course of a school year• Many schools use it from one year to the next to track outcomes over time• Educators report “aha moments” and “chances to celebrate”
I am engaged in School 90 88 76All HSSSE 2009 STG 2009 STG 2010
I have opportunities to be creative in the classroom Percentage Strongly Agreeing 90 86 75STG 2009 STG 2010 All Schools (Averaged)
We regularly discuss questions with no clear answers 92 82 72 All Schools St. Gregory 09 St. Gregory 10
School emphasizes exploring new ideas 92 90 72 All HSSSE 2009 STG 2009 STG 2010
What Excites & Engages Me?All HSSSE students 65 60 55 28 Lecture Discussion and Group Projects Projects Involving Debate Technology
MAP Testing is a tool that becomes anextension of our mission, which is to maximize individual growth.Pam Shaw, Canton Country Day School (OH)
• Traditional/Conventional Standardized Testing Updated• Computer Adaptive Assessment• More frequent (3x) than once a year• Reporting on not just achievement but growth• More rapid feedback to teachers• More detailed analysis of student proficiency• Curricular resources aligned with achievement
“Formative Assessment works…when FA isused, students learn better—lots better.”W. James Popham, “Formative Assessment’sAdvocatable Moment.” Edweek, 1.8.13
Sample Math Question• Notice the calculator.• To use it, click the numbers with your mouse.
Sample Reading Question• Read the passage and select the best answer• Click the <Go on> button
DesCartes ExampleStudent Scores a RIT range of 171-180 on the Goal Strand of Estimation and AccurateComputations
Teachers using online portal, grabbing studentreports and class reports– and using curriculumalignment resources.Parents and tutors getting these same reportsfor supplemental instruction.Reports correlated with state standardizedtesting.
generating student achievement international comparison data
OECD TEST FOR SCHOOSL BASED ON PISA Have you ever wanted to compare your school’s effectiveness with the world’s best national educational systems?
BLOOD DONATION NOTICE Blood donation is essential. There is no product that can fully substitute for human blood. Blood donation is thus irreplaceable and essential to save lives. In France, each year, 500,000 patients benefit from a blood transfusion. The instruments for taking the blood are sterile and single-use (syringe, tubes, bags). There is no risk in giving your blood.Blood donation:It is the best-known kind of donation, and takes from 45 minutes to 1hour.A 450-ml bag is taken as well as some small samples on which tests and checks will bedone. - A man can give his blood five times a year, a woman three times. - Donors can be from 18 to 65 years old.An 8-week interval is compulsory between each donation.
Question 1Level 2 item – 81.2% of students across OECD can perform tasks at least atthis level
Section II. Your School’s Results in anEXAMPLE: International comparisons of international context your school’s performance: PISA 2009 School ABCD Countries and Economies Reading: 512 Mathematics: 513 Science: 507 Your school’s results are statistically significantly above Your school’s average score is not significantly different Compared to Percentiles (to be developed) Your school’s results are statistically significantly below Country or Reading Mathematical ScientificCompared to Averages Economy literacy literacy literacy Country or Reading Mathematical Scientific Shanghai- Economy literacy literacy literacy ChinaShanghai- KoreaChina FinlandKorea CanadaFinland United StatesCanada UnitedUnited States KingdomUnited GermanyKingdom TurkeyGermany OECD AverageTurkeyOECD Average
Section II. Your school’s results in an EXAMPLE: Your school’s performance in international context the context of PISA results for Canada School ABCD (with socio-economic background) Reading: 512 School performance and schools’ socio-economic background Student performance and students’ socio-economic background within schools Score 700 Private school Public school in rural area Public school in urban area Your SchoolStudent performance 493 493 Index calculated from student questionnaires 300 200 -2 -1 0 1 2 Disadvantage PISA Index of socio-economic background Advantage
Proposed Structure of School Reports (Preliminary)Contents of print-ready reports will include: I. Understanding your school’s results from the PISA- Based Test for Schools assessment II. Your school’s results in an international context III. Additional insights from international PISA results Annexes A. Summary of PISA-Based Test for Schools assessment questions (items, units and response-types) B. Summary Description of PISA Assessment Frameworks Will review option for schools to electronically explore their confidential results online
OECD- PISA Based Testing for Schools The good and the bad: • Measures valuable application of knowledge. • Don‘t think it constrains. • Extraordinarily comprehensive reporting. • Administration not too onerous. • Price uncertain.
CAIS score reports for TIMMS replica test ABC Country Day School TIMMS “released item” test results95% of students scored in the top 1/2 of I.A.92% of students scored in the top 1/3 of I.A.90% of students scored in the top 10% of I.A.
PISA and TIMMS Assessment ofAssessment Assessment As For
sustaining competitive position with whatand how the public education “competition” or alternative will be assessing soon
―Today is the day that marks thebeginning of the development of a newand much-improved generation ofassessments for America’sschoolchildren. Today marks that startof Assessment 2.0‖ Beyond the Bubble Tests: The NextGeneration of Assessments – Secretary Arne Duncan’s Remarks to State Leaders September 2, 2010
―For the first time, many teachers willhave the state assessments they havelonged for—tests of critical thinkingskills and complex student learningthat are not just fill-in-the-bubble testsof basic skills but support goodteaching in the classroom.‖ September 2, 2010
―In performance-based tasks, whichare increasingly common in testsadministered by the military and inother fields, students are given aproblem—they could be told, forexample, to pretend they are a mayorwho needs to reduce a city’spollution—and must sift through aportfolio of tools and write analyticallyabout how they would use them tosolve the problem.‖ September 2, 2010 New York Times
PARCC and Smarter Balanced 2014-15 KEY SIMILARITIESSummative Assessments: Other Assessments, Resources, and Tools:•Online assessments for grades 3–8 •Optional interim assessments.and high school in English languagearts and literacy and in mathematics. •Professional development modules.•Use of a mix of item types, including •Formative items and tasks for classroom use.selected-response, constructed-response, technology-enhanced, and •Model curricular and instructional units.complex performance tasks. •Online reporting suite.•Two required components, bothgiven during the final weeks of the •Digital library for sharing vetted resources andschool year. tools.•Use of both electronic and humanscoring, with results expected within 2weeks.
Type I: Tasks assessing concepts, skills and procedures
TYPE III: TASKSASSESSING/ MODELING / APPLICATIONS
Narrative Task (The Narrative Task broadens the way in which students may use thistype of writing. Narrative writing can be used to convey experiences or events, real or imaginary. In this task, students may be asked to write a story, detail a scientific process, write a historical account of important figures, or to describe an account of events, scenes or objects, for example).
As the public schools move their educational assessment“beyond the bubble” – more challenging, – more authentic/situated performance tasks, – more evaluation of 21st c. skills and digital literacies, – more formative assessment – more integrated teacher PD and collaboration toolsWhere do we stand? What must we do to offer value?
Paralleling some of the tools and resources of PARCC Expanding 21st c. Capacities and Literacies to K-8
CBAL• Extended, constructed-response tasks that are delivered by computer and automatically scored.• Pilot testing occurred in 2010 and 2011, spring of 2012.• Tests should be available for use in 2012.• Sample tests available online
Project Based LearningRe-conceptualize– not project oriented learning.When re-conceptualized, rethink assessment:PBL offers opportunity for a wide array ofassessment for a single, extended project.
Peer Assessment Teamwork/ Self-Collaboration Assessment Final Product: Creativity and Craft Content Expert Knowledge AssessmentDemonstrated Project Organization
OCT: Open Computer Testing What matters is no longer what one knows, but whatone can do with what one knows and with information one can access, evaluate, and apply. Tony Wagner
From BBC News, about DenmarkAt five to nine, the room falls silent. One of the teachersstands in front of the class and explains the rules. She tellsthe candidates they can use the internet to answer any ofthe four questions. They can access any site they like,even Facebook, but they cannot message each other oremail anyone outside the classroom.Minister for education in Denmark, Bertel Haarder, says:“Our exams have to reflect daily life in the classroom anddaily life in the classroom has to reflect life in society. Theinternet is indispensable, including in the exam situation.I’m sure that is would be a matter of very few years whenmost European countries will be on the same line.”
What do we want for our students?• 21st c. skills and mindsets• Authentic, real-world situated problem solving skills• Lasting understanding of concepts• Stronger Skills in Using tools they are already most familiar and comfortable with.• Preparation for professional careers
The Takeaways• For Schools, OCT provides a vehicle to prioritize and drive instruction toward 21st century skills, applied thinking, and better preparation for future careers.• For Teachers, OCT entails rethinking assessment and designing “google-proof” questions which demand higher order thinking and information literacy and assess more lasting understanding.• For Students, OCT when well designed is rigorous, and more authentic, more motivating, and requires greater information access, analysis, and application skills, better organization, and stronger real-world problemsolving skills.
Strengthening Assessment of 21st century and higher order thinking skills
Read and review the documentDiscuss at your table:1. What would be elements of a quality answer?2. What would be aspects of a poor answer?3. What skills would a student need to have to answer well?
Exploring tools and methods for assessing essential 21st century non-cog capacities
Wagner’s 7 Survival Skills• critical thinking and problem solving• effective oral and written communication• accessing and analyzing information -• curiosity and imagination• collaboration across networks and leading by influence -• agility and adaptability• initiative and entrepreneurship
In Eastern cultures, Stiglersays, its just assumed thatstruggle is a predictable partof the learning process.Everyone is expected tostruggle in the process oflearning, and so strugglingbecomes a chance to showthat you, the student, havewhat it takes emotionally toresolve the problem bypersisting through thatstruggle.
Measure your grit score– – http://www.sas.upenn.edu/~duckwort/ » Or – http://goo.gl/5RRjVFind, get your grit score, complete, and discuss howyou might use with students.
Character and 21st c. skills report cards www.21k12blog.net
St. Gregory’s “egg”• Character – Integrity – Resilience• Leadership – Positive role model – Facilitating Collaboration• Scholarship – Inquisitiveness – Analytic thinking – Synthetic thinking – Critical thinking• Innovation – Adaptability – Initiative – Experimentation www.21k12blog.net
Work ethic and perseverance – Works hard – Persists in the face of difficulty – Takes responsibility for his/her own learning – Accepts challenges as growth opportunities – Is dedicated and determined to succeed
Robert J. Sternberg Theory of Successful Intelligence• Individuals are successfully intelligent to the extent they display – Creative skills to generate novel ideas – Analytical skills to ascertain whether the ideas are good ones – Practical skills in order to implement their ideas and persuade others of their value – Wisdom-based/ethical skills in order to ensure the ideas help to achieve a common good based upon positive ethical principles 123
Source for Details Sternberg, R. J. (2010). College Admissions for the 21st Century. Cambridge, MA: Harvard University Press. 124
The Rainbow Project• This project was conducted at Yale University in the early 2000s on roughly 1000 students with freshmen from 13 colleges and seniors from 2 high schools. The institutions were geographically very dispersed and were of greatly varying levels of selectivity, from community colleges to highly prestigious ones. There was a high level of ethnic diversity. Assessments were proctored in the students’ own institutions but scored at Yale. Robert Sternberg125
Predicting GPA: SAT + Analytic, Creative, Practical 19.9Step 1: 20SAT-Verbal, SAT-Math 15Step 2: R squared (%)All Rainbow Project Items 9.8(STAT Analytic, Practical, 10Creative,Practical performance,Creative performance) 5 0 Step 1 Step 2 126 Robert J. Sternberg
Predicting GPA: All measures (practical before creative)*Step 1: SAT-M 30 24.8 SAT-V 25 HSGPA 20 R squared 15.6 15.2 15.9 15Step 2: + Analytic 10Step 3: + PracticalStep 4: + Creative 5 0*Controlling for school quality in Step Step Step Step 1 2 3 4dependent variable 127 Robert J. Sternberg
In sum• In the Rainbow sample, – Adding Rainbow measures over SAT roughly doubles prediction of college success – Adding Rainbow measures over SAT + High School GPA increases prediction by roughly half Robert J. Sternberg 128
Choate Rosemary Hall ProjectThis battery included a variety of measures toenhance prediction of academic success in theenvironment of Choate Rosemary Hall Robert J. Sternberg 129
• A good internal locus of control—the ability to shoulder the blame when things weren’t working, and also to take credit for one’s own successes.• Sensible self-confidence. Students who were overly confident or lacked confidence did not adjust as well.• A tacit knowledge about how the independent school environment worked. Students with such knowledge did a better job of mastering the environment over time.”
Students respond to about 40 statementsdescribing their own views of themselves andthe world, rating them on a strongly agree-strongly disagree scale.
Statements like:I like to learn for the sake of learning.Many people fail to get the recognition theydeserve no matter how hard they try.
• The survey is then generated into a report on each student, explaining where that applicant lies on a scale normed to their ages of self-efficacy, locus of control, and intrinsic motivation.• even brief self-reports forming the PACE SRL assessments contribute approximately 10% of unique predictive variance…• Especially useful for outlier candidates.
• High Tech High Digital Portfolios• New Tech Network Digital Portfolios Digital Portfolios
Demonstrations of Learning: “What you do, not what you know, is the ultimate test of education.” ~PFB TweetConduct a fluent conversation in a foreign language aboutof piece of writing in that language.Declaim with passion and from memory a passage that ismeaningful, of one’s own or from the culture’s literature orhistory.Invent a machine or program a robot capable of performinga difficult physical task.
Demonstrations of Learning for 21st. C. SchoolBy these demonstrations, schools…• Reunite content and action.• Backward-design curriculum from desired outcomes.• Demonstrate student outcomes recorded in electronic portfolios.• Facilitate student-led teacher/parent conferences.• Conduct action research and lesson study to grow professionally.
History at Lakeland Prep:In the four year History sequence at Lakeland Prep, all students will complete the following Demonstrations of Learning:• 24 research based position papers (4 to 7 pages) in which an analysis, synthesis and/ or evaluation of both original and modern sources is offered in answer to a provocative question in history.• 6 research based position papers (10 to 15 pages) focused on a students original response to one of identified Essential Questions in American History.• 12 oral presentations• 8 collaborative projects,• 3 projects completed in collaboration with students in other schools and/ or countries• 4 interviews with elected officials• 6 Letters to the Editor written on a current topic in local and/or state government
Discussion• What would you highlight in your students portfolios as demonstrations of learning? – Things your students already do, representative of mission? – What might you add?• How would you highlight, display and measure them?