Measuring what we value - lyons and niblock presentation
Measuring what we value:21st Century Assessments for Independent Schools Douglas Lyons, Executive Director, CAIS-CT Andrew Niblock, Lower School Head, Hamden Hall (CT)
Good to Great Jim Collins• Determine what you value most, then find a way to measure it.• Success can be a powerful disincentive: it may be hard to become a great school if you are a very good school
Goals of this workshop1. Identify measures of school quality that have historically been valued by educators and/or the public.2. Suggest new ways to report achievement in those measures. Why?• To better tell our story (the Value Proposition)• To make certain that these measures have stature appropriate to their significance
Goals, cont. 3. Provide a quick preview of new andemerging assessment tools –instrumentsdesigned to measure skills that are increasingin demand in the new century4. Describe the assessment practices in aselect group of schools that define themselves– and are recognized by others – as “Schools ofthe Future”
Criterion 13:The Standards require a school to provide evidence ofa thoughtful process, respectful of its mission, for thecollection and use in school decision making of data(both external and internal) about student learning.
“Not everything that can be counted,counts, and not everything that counts can be counted”
“It would be easier to change the course of history…”
Powerpoint presentation is available at www.caisct.orglook for ADVIS presentation on home page
Jim Collins – “whenever possible, use thelanguage of metrics to define what you value” What is the language of metrics?
4 “ways of knowing”• Data: raw input, no context, facts, figures, symbols• Information: organized, processed, analyzed data• Knowledge: information with higher context - accurate, relevant, current• Wisdom: evaluated knowledge; merged with life experience
• Heads letter in viewbook; wisdom• Description of school history and mission in viewbook: knowledge• Course catalogue: information• SAT scores, college placement stats: data
What are people most interested in ? Data!• conveys a lot of information - quickly• Is viewed as objective, “no spin”• Can be benchmarked, used for comparison Data is the language of metrics
Risks / Misuses of Data• Garbage in; garbage out• Data is easily manipulated, corrupted: Harvard Business School caution If you torture data long enough, it will admit to anything
Data Management / Data Creation in the independent school communityThe Challenge:• To frame the data that define us, or have defined us in the past, in ways that do not elevate modestly valuable information• To gather and/or to present new data that is beneficial to educators in our planning for the future and is data that measures performance in relation to the achievement of our highest goals.
The S.A.T.• Decreasing in stature, but still powerful• Has poor validity statistics• Does not measure 21st century Skills• Historically, did not provide faculty with instructionally useful information• Consider giving the School Day SAT with Enhanced Scoring
Standardized Achievement Tests These test are increasing in statureWhat happened to elevate these tests above all other forms of data, in public education and to a lesser degree, in independent schools?
“Effects” of standardized tests today: from educating the whole child to educating the whole test-taker• Hyper-focus on scores, minor fluctuations• Unprecedented “score chasing”• M.D.I “measurement-driven instruction”• Mind numbing drill and practice
Most popular form of data presentation: percentiles Math Math Reading Reading concepts applications comprehensio Skill n1 91 87 83 832 88 84 85 833 92 90 88 884 84 89 80 905 84 86 78 826 89 80 81 797 90 89 90 878 87 87 81 83
Second most popular form of data presentation: Grade Equivalence Math Math Reading Reading concepts application comprehensio skill s n 1 1.8 2.1 2.3 2.5 2 2.6 2.8 3.3 3.6 3 3.5 4.0 3.9 4.0 4 4.5 4.9 4.8 4.7 5 5.8 5.2 6.3 6.0 6 6.6 7.3 7.0 7.5 7 8.2 9.0 8.8 8.8 8 9.0 9.6 9.9 10.4
A better way to present achievement data• Determine and defend your norming group• Determine a worthy and realistic goal within the norming group• Publish data relative to the goal
Goal: to score within the top third of norming group on all subtests Reading Reading Math Math skill comprehensio skills applications n1 √ √ √ √2 √ √ -4 √3 √ √ √ √4 √ √ √ √5 √ √ √ √6 √ √ √ √7 -2 √ √ √8 √ √ √ √
The International Database• Most schools administer one or more normed tests that compare American student achievement with American peers• Is there a way to assess our international competitiveness?• Would this data be valuable to us?
Benefits of the New York State “Truth in Testing” Law• Thousands of released items available to educators• Released items available for NAEP tests• Released items available for TIMMS tests• Released items available for PISA tests• Construct your own “replica test” or form a research partnership to develop replica tests
CAIS score reports for TIMMS replica test ABC Country Day School TIMMS “released item” test results95% of students scored in the top 1/2 of I.A.92% of students scored in the top 1/3 of I.A.90% of students scored in the top 10% of I.A.
Course of study guides: how we describe our program US HISTORY HONORS GREENWICH HIGH SCHOOL This course addresses the events and experiencesthat comprise American history from the period ofEuropean colonial settlement through the Civil War(1st semester) And from the period of Reconstructionthrough the advent of the Second World War. Thegoal of the course is to provide for our studentssubstantial opportunity to develop the ability to makeinformed and reasoned decisions as citizensconcerned with the public good. (More text follows)
US HISTORY HONORSA study of the events and critical changesthat took place from the first Americansettlement to the present day.We will focuson these events in the context of largerthemes; including the shift from anagricultural to an industrial society, therecognition and cultural identification ofdifferent groups of people, the transition to astronger national government, immenseterritorial expansion, technological changeand globalization.
History at Lakeland Prep:In the four year History sequence at Lakeland Prep, all students will complete the following Demonstrations of Learning:• 24 research based position papers (4 to 7 pages) in which an analysis, synthesis and/ or evaluation of both original and modern sources is offered in answer to a provocative question in history.• 6 research based position papers (10 to 15 pages) focused on a students original response to one of identified Essential Questions in American History.• 12 oral presentations• 8 collaborative projects,• 3 projects completed in collaboration with students in other schools and/ or countries• 4 interviews with elected officials• 6 Letters to the Editor written on a current topic in local and/or state government
HSSSE - what do you believe your high school emphasizes most?• 21% memorizing facts and figures• 32% understanding information and ideas• 22% analyzing ideas in depth• 68% my school challenges me academically
HSSSE: what instructional methods do you find exciting or engaging?• 60% discussion and debate• 60% group projects• 44% student presentations• 24% teacher lectures
Using an Engagement Survey• Use the HSSSE and contribute your school data for national benchmarking;• Or, create your own survey, then compare your results to the national HSSSE 2009 data, where applicable .
The Emergence of Longitudenal Data• The National Clearinghouse has 93% of all US colleges collecting and providing longitudenal data• Two independent school associations require student tracking for accreditation (freshman GPA):• CAIS Canada firstname.lastname@example.org• ISASW www.isasw.org
Measuring teacher engagement, professionalism, attachment to schoolCan that be quantified?• Longevity statistics• % faculty with advanced degrees• % faculty participating in Annual Giving
Occasional Teacher Absenteeism• Reported in “school district report cards”• Measures only consecutive days absent, less than 5, 7 or 10 days. After the threshold, absence considered “long term illness”, removed from calculation• O.C.A considered an indicator of faculty commitment, professionalism, attachment
Measure your O.T.A.• Compare it to your local or state public school average• Inform your Board of Trustees• Inform your Parents Association
Measuring teacher effectiveness: in the best public schools, teacher evaluation is an informed, professional process How can independent school leadersbecome more skilled in the clinicalobservation and evaluation of teaching? How can we measure what we value inteaching practice? The TIMSS video study – teaching practices in 7 select countries
New and emerging assessment instruments – workshop goal 3• We have just reviewed ideas for improving the use of existing data• What about new tools and techniques?• Are the assessment instruments and practices of earlier generations obsolete or incomplete?• If so, why is that true?
Did something occur on January 1, 2001 that changed everything?
Turn of the millenium eventsFirst successful HTTP communication (modern WEB)1990Netscape, easy to use browser 1995Google, as a research project 1996LiveJournal, Blogger, hosting sites 1999Ericsson smartphone 2000Wikipedia 2001Facebook 2004MIT Open Courseware 2004YouTube 2005Skype 2005The World is Flat first edition 2005Internet/Multimedia Smart phone (iphone) 2007Global financial crisis 2008Khan Academy (2600 videos and growing) 2009
100 most influential people of the 2nd Millenium• Jonas Salk 97• Steven Spielberg 91• Elvis Presley 57• Gregor Mendel 42• Martin Luther King 33• Henry Ford 29• Michelangelo 19• Galileo 10• Columbus 6• Charles Darwin 4• Who is Number 1?
Johann GutenbergThe printing press was information technologyWhat about modern day visionariesin information technology?What number is Bill Gates? Steve Jobs?
What’s Past is Prologuevs The Future is Not What it Used to Be • “What does an educated person need to know?” • Education is defined by a remembered body of knowledge, the “canon” • Critical, sarcastic view of the canon: Education as inoculation:” American history? I had that, Tetanus shot? I had that…)
20th century technology:• Radio, television and film had great promise, but no demonstrable effect on schooling What 20th century technology had a revolutionary effect on teaching and learning?
SCANTRON! Bubble answer sheets!• 1948 – Scantron Corporation revolutionized the speed and efficiency of data collection and advanced the notion that student proficiency and school quality can be determined through mass-produced, multiple choice metrics• Scantron, to this day, has had a greater impact on k-12 curricular design than any other technology in history.
in post scantron decades: “What gets measured is what gets taught”.• Tests “drive” instruction in ways that mimic both the content and format of the test.• What gets measured is almost exclusively content• In the Information Age, we measured recall of information
In 2012, in The Conceptual Age There are no books, conferences, op-ed pieces on “21st Century Content”. The canon has been buried under the information explosionHowever, There is near universal agreement on a short list of 21st century skills. There is near universal agreement on the need toemploy technology in a thoughtful but robust manner
Nicholas Negroponte on applying technology in a robust manner:• “When you drop a penny into a glass of clear water, you get a glass of clear water with a penny in it; the change is additive.”• “When you place a drop of red dye in a glass of clear water, you get a glass of pink water. The change is ecological.”• Technology in education needs to be ecological; pink water
The i generation• Defined mostly by their use of technology• Accustomed to learning things on their own and learning from peers• Expect technology to be interactive and customizable• Non-linear thinkers; web thinkers, scanners, multi- taskers
Clay Shirke, futurist describing the i generation:“A father sets up a new television in the livingroom. His 4 year old daughter is seenrummaging through the box. What is shelooking for?”Passive media experienceswill hold less appeal for thisgeneration
The 21st century skills movement,the Schools of the Future movement, focus on the development of these skills: Communication Collaboration Critical/Analytical Thinking Creativity Problem-solving Content is still important; but content in these areas will need to be acquired through active exploration as well as through instruction.
Non–Routine Tasks defined in the Journal of Economics, volume 118• Gathering, synthesizing, and analyzing information.• Working autonomously to a high standard with minimal supervision.• Leading other autonomous workers through influence.• Being creative and turning that creativity into action.• Thinking critically and asking the right questions.• Striving to understand others’ perspectives and to understand the entirety of an issue. Communicating effectively, often using technology.
Current assessment tools do not measure these skills.You cannot have 21st century schools using 20th century assessments.
Ideal Assessment:Provides accurate demonstration of studentproficiencyYields information for faculty planningIs valid as a learning experience in and of itselfAn assessment of, for and as learning
What is a performance task?Students assume roles in ascenario that is based in the"real world" and contains thetypes of problems they mightneed to solve in the future. Thetask requires critical thinking,analytical reasoning andproblem solving.Communication skills are usedin describing the solution.
Ohio Mastery Test, Grade 9• Ms. Johnson installs new insulation to save money on heating costs, but then learns that her bills have not declined by much from the previous year. Her contractor points out that heating costs have risen and weather has been colder. Ms. Johnson wants to find out how much she has actually saved due to the insulation she installed. On the basis of the situation painted above, details about Ms. Johnson’s heating bills (rates, units of heat used), temperature changes, and some initial information to help them begin to research “heating degree days” on the internet, students are given two tasks:• (1) Assess the cost-effectiveness of Ms. Johnson’s new insulation and window sealing.• (2) Create a short pamphlet for gas company customers to guide them in making decisions about increasing the energy efficiency of their homes.
Students get answers to questions THEY ask• For example – Family history – Is this breast cancer possibly caused by abnormal oncogene expression? If so, certain types of hormonal therapy or receptor antagonists are more effective treatments. – What level of stage III cancer, A, B or C?
CBAL• Extended, constructed-response tasks that are delivered by computer and automatically scored.• Pilot testing occurred in 2010 and 2011, spring of 2012.• Tests should be available for use in 2012.• Sample tests available online Website information is in your folders.
features real-time, scenario-based tasks thatmeasure an individuals ability to navigate,critically evaluate and understand the wealthof information available through digitaltechnology
Ken Robinson Age and education:• Increase routines of behavior and habits of thought (left brain logical thinking )• Decrease divergent thinking (free association of ideas. Right brain, creative thinking)
Creativity Index: the new state mandate?• Gov. Deval Patrick has made Massachusetts the first state in the country to call for the formation of a creativity index aimed at rating public schools statewide based on their ability to teach, encourage and foster creativity in students.• Similar legislation is pending in California and Oklahoma
Torrence Test of Creative Thinking Verbal Activity 4: ProductImprovement Look at the stuffed toy elephant inthe drawing. It is six inchestall and weighs a half pound. In thespace provided, list the cleverest, mostinteresting and unusual ways you canthink of for changing this toy so that children will havemore fun playing with it. Do not worry about howmuch the change would cost.Think only about whatwould make it more fun to play with. Activity 2 and 3: Guessing Causesand Guessing Consequences measures“idea fluency”
What do you get if you solve this problem and visit the website?
The 4th goal of this workshop: Examples of assessment practices in a select groupof schools that define themselves – and are recognized by others as “Schools of the Future”
Schools that define themselves as Schools of the future• Who are the pioneers?• What do these schools have in common?
Science Leadership Academy Philadelphia, Pennsylvania
All of these schools have 2 common characteristics• urban public charter schools.• experimenting with a dramatically different view of teaching and learning; A Collaborative, Conceptual Model
21st Century education: from coverage model to conceptual model• Recall of information (content) is still important• Skill in accessing and selecting information (internet searches now deliver 2000 hits) vitally important• Ability to use or apply information in new and/or novel settings most important (Its not what you know, but what you can do with what you know)• Expanded role of the teacher: guide, coach, facilitator
Schools of the Future: characteristics• Performance tasks• Project-based learning, individually and in groups• Capstone projects, individually and in groups• Independent study• Online learning, online tools (courseware, Skype, You Tube, Ning, Moodle, Web 2.0 etc…)• Students given choices in assignments and in demonstrations of mastery
Schools of the future, cont.• Extensive use of essential questions relating to content area why, how and what if questions• Computer-adaptive learning (program adjusts to student skill level)• E-portfolios, published within the community or on the web – seeking Facebook-type conversations in the academic community, on academic topics• Flipped classroom strategies – routinely or occasionally• Partnerships, learning experiences beyond the school campus• RUBRICS used to assess performance
A New Definition of School “we need to invert the conventional classroom dynamic: instead of teaching information and content first, and then asking students to answer questions about it second, we should put the question/problem first, and then facilitate students with information and guidance as they seek the answer and hold them accountable for the excellence of their solutions and of their presentation of their results”.-Ted Mccain Teaching for Tomorrow
Independent schools• lead the nation in communication skills; writing, speaking, the performing arts• Engagement has been supported by very strong student-faculty relationships• An incremental approach to the challenges of the future; preserving strong, successful, traditional programs while expanding collaborative learning, online learning, project-based assessments, exhibitions of learning and use of digital portfolios• Growing interest in “Essential Questions” theory of learning
Lessons from ourresearch:Schools in the 21st century willdefine success in much broadertermsGreat Schools in the 21st centurywill include some that have far fewerresources than independentschools. What they have is thefreedom to take big risks indesigning innovative cultures
“Measuring What We Value” Sites Referenced in PresentationHechinger Article containing multiple links of sample questions on new 2012assessmentshttp://hechingered.org/content/are-new-online-standardized-tests-revolutionary-decide-for-yourself_5655/Information on Torrance Testhttp://www.ststesting.com/High School Survey of Student Engagementwww.indiana.edu/~ceep/hssse/College Student Experiences Questionnairehttp://cseq.iub.edu/cseq_generalinfo.cfmNational Student Clearinghousewww.studentclearinghouse.org/The Self-Regulation Questionnairewww.psych.rochester.edu/SDT/measures/SRQ_text.phpMHISC: Mental Health in Independent Schoolshttp://www.harthosp.org/InstituteOfLiving/OtherServices/MHISC/default.aspx
C-bal Cognitively Based Assessment for Learninghttp://www.ets.org/research/topics/cbal/initiative/The CWRA: College to Work Readiness Assessmentwww.cae.org/cwra/Science Leadership Academywww.scienceleadership.org/High Tech Highhttp://www.hightechhigh.org/New Tech Highhttp://newtechhigh.org/Big Picture Learninghttp://www.bigpicture.org/NYC i schoolhttp://www.nycischool.org/Microsoft School of the Futurehttp://www.microsoft.com/education/en-us/leadership/partners_in_learning/Pages/School-of-the-Future.aspx
Performance Assessment Group of NYC Schools (check out the rubics!)http://performanceassessment.org/Rubics – Association of American Colleges and Universities (rubics on criticalthinking, creative thinking, problem-solving and others!)www.aacu.org/value/rubicsAvenueshttp://www.avenues.org/Haverford School (Decision Education)http://www.haverford.org/Decision Education (critical thinking/character education program)http://www.decisioneducation.org/Independent Curriculum Grouphttp://www.independentcurriculum.org/Greens Farms Academyhttp://www.gfacademy.org/RelId/607374/ISvars/default/Capstone.htmHotchkiss/Loomis Collaborative Learning Projecthttp://tinyurl.com/3kq8v8f
Project-based Learning (450 sample projects – all subjects and grade levels –templates for organizing projects)http://pbl-online.org/Siemens Challenge (sample award-winning projects)http://www.wecanchange.com/American Society for Engineering Education (ASEE)http://teachers.egfi-k12.org/Exploravision (sample award-winning student projects)http://www.exploravision.org/Toyota Tapestry Grants for Science Teachers (sample grant-winning ($10,000) projects)http://www.nsta.org/pd/tapestry/