Global Cities Education Network Brian Stecher Assessing 21st Century Skills
•
1 like•258 views
Presentation by Brian Stecher of RAND on the assessment of 21st century skills. Given at the January 2013 Global Cities Education Network Symposium in Seattle, WA.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Global Cities Education Network Brian Stecher Assessing 21st Century Skills
Similar to Global Cities Education Network Brian Stecher Assessing 21st Century Skills (20)
Recently uploaded
Recently uploaded (20)
Global Cities Education Network Brian Stecher Assessing 21st Century Skills
- 1. Practical Concerns for Educators in Assessing 21st Century Skills Brian Stecher, RAND
- 2. Definition of 21st Century Skills • Cognitive – Non-routine problem solving, critical thinking, systems thinking • Interpersonal skills – Complex communication, social skills, teamwork, cultural sensitivity, dealing with diversity • Intrapersonal skills – Self-management, time management, self- development, self-regulation, adaptability, executive functioning National Research Council, 2011
- 3. the constructs we chose to focus on. Many Conceptions of 21st Century Skills Table 1. Mapping of 21 st Century Skills Frameworks Research-based P21 Framework NRC Framework ATC 21 construct terminology terminology Framework terminology Critical thinking Learning and Cognitive – critical Ways of thinking – innovation – critical thinking critical thinking, thinking problem-solving, and decision- making Collaboration Learning and Interpersonal – Ways of working – innovation – complex communication and communication and communication, collaboration collaboration social skills, teamwork Creativity Learning and Cognitive – non- Ways of thinking – innovation – routine problem creativity and creativity and solving innovation innovation Motivation Life and career Intrapersonal – self- Living in the world skills –initiative, development, –adaptability, flexibility adaptability flexibility, self- direction Metacognition Life and career Intrapersonal - self- Ways of thinking – skills – self- management, self- metacognition or direction, regulation learning to learn productivity Lai & Viering, 2012
- 4. Systems are Adding 21st Century Skills Hong Japan China Finland Singapore United Kong States Learning to Zest for Living Greater New focus on New Common Core Learn reform education reform emphasis on ―citizen skills‖: Framework State addresses stresses the students’ (1) thinking for 21st Standards applied importance of ability to skills, including Century Initiative learning and experimentation, communicate problem Competencies redefines ―other‖ problem finding, and work in solving and and Student standards to learning and problem teams, pose creative Outcomes is make them experiences, solving instead of and solve thinking; intended to ―inclusive of including rote memorization problems, and (2) ways of better position rigorous service and learn to learn working and students to content and workplace interacting; take advantage applications of learning (3) crafts and of global knowledge expressive opportunities through skills; higher-order (4) participatio skills, so that n and initiative; all students are and (5) self- prepared for awareness and the 21st personal century‖ responsibility 2000 2006 2010 2010 2010 2010 Saavedra & Opfer 2011
- 5. Many Ways to Assess Such Skills • Self-Reports • Ratings by Others • Observational Scales • Biographical Data • Portfolio • Performance Tests • Selected Response Tests Adapted from Kyllonen (2012)
- 6. Many Ways to Assess Such Skills • Self-Reports • Ratings by Others Each approach raises different • Observational Scales practical concerns • Biographical Data • Portfolio • Performance Tests • Selected Response Tests Adapted from Kyllonen (2012)
- 8. Example: Self-Report 14 Item Resilience Scale™
- 9. Example: Ratings by Others Knowledge and Creativity Resilience • Has a broad perspective on the field • Accepts feedback without getting defensive • Is among the brightest persons I know • Works well under stress • Produces novel ideas • Can overcome challenges and setbacks • Is intensely curious about the field • Works extremely hard Communication Skills Planning and Organization • Speaks in a clear, organized and logical • Sets realistic goals manner • Organizes work and time effectively • Writes with precision and style • Meets deadlines • Speaks in a way that is interesting • Makes plans and sticks to them • Organizes writing well Ethics and Integrity Teamwork • Is among the most honest people I know • Supports the efforts of others • Maintains high ethical standards • Behaves in an open and friendly manner • Is worthy of trust from others • Works well in group settings • Demonstrates sincerity • Gives criticism/feedback to others in a helpful way Educational Testing Service (2012)
- 10. Example: Biographical Data I stood on the sidelines, watching my teammates receive the first place trophy at the Ranger Challenge competition. I could not help but feel joy for their success. I knew they had earned it. Or, I should say, “we” had earned it. Even though I did not get to compete, I considered myself part of the team. I am a cadet in the Army’s Reserve Officers Training Corps (ROTC) at Wheaton College. Ranger Challenge is the “competitive sport” of ROTC, culminating in a tough regional competition. Wheaton College sends two teams, the A team (varsity) and the B team (junior varsity) to compete against more than twenty colleges and universities in the Great Lakes region. The competition consists of a physical fitness test, a demanding land navigation course, various tests of military skills, and a six-mile run in full gear with a forty-pound backpack. It is quite demanding, both physically and mentally. The competition requires weeks of grueling preparation. Wheaton College, 2012
- 12. June 20, 2011 Standard Indicators Level I Level II Level III Level IV Aspire Public Schools, 2012 The teacher uses student data Example: Observational Scales The teacher draws a specific sources to inform content The teacher does not use student The teacher uses student data connection between the student organization, instructional 1.3 data sources to guide how sources to inform his/her data sources used, the content strategies, and grouping in order A) Lesson design Use student data guided by data content is organized or planning for whole groups of organization, instructional to meet individual student needs. to guide planning instructional strategies are students. strategies and grouping in order The teacher includes specific selected. to meet specific student needs. opportunities for students to use self‐assessment data to guide their instructional choices. The teacher identifies the The teacher identifies the The teacher identifies the necessary The teacher does not identify the A) Knowledge of necessary prerequisite content necessary prerequisite content prerequisite content knowledge 1.4 subject matter to necessary prerequisite content knowledge and skills for the knowledge and skills for and skills for students, and includes Use knowledge identify pre‐ knowledge and skil s for class as a whole but does not students, and includes specific specific learning activities to students, and/or does not of subject matter requisite address known gaps in students’ have a clear plan for addressing learning activities to address address known gaps in that content/skills knowledge learning. known gaps in knowledge or known gaps in that knowledge prerequisite knowledge for and learning skil s. for the class as a whole. individual students. processes to plan The teacher identifies common for student The teacher identifies common content misconceptions and The teacher identifies common learning B) Addresses The teacher does not accurately content misconceptions but content misconceptions and includes multiple strategies for common content identify common content includes strategies for avoiding explaining, avoiding, and correcting misconceptions misconceptions. has limited strategies for and correcting these cognitive these cognitive errors as well as for avoiding these cognitive errors. errors. uncovering additional misconceptions. 1.5 Assessments measure al aspects Design Assessments measure all of the learning objectives and have assessments to A) Selection of Assessments do not measure Assessments measure some, aspects of the learning been adapted to meet student but not all aspects of the objectives and include varied needs. There are planned ensure student assessments the learning objectives. learning objectives. methods so that students have opportunities for students to use mastery multiple ways to show mastery. assessments to reflect on their progress towards mastery.
- 13. Example: Portfolio Asia Society, 2012
- 14. Cognitive Skills
- 15. Depth of Knowledge • Depth of Knowledge (DOK) 1: Recall of a fact, term, concept or procedure, basic comprehension • DOK 2: Application of concepts or procedures involving some mental processing • DOK 3: Applications requiring abstract thinking, reasoning, or more complex inferences • DOK 4: Extended analysis or investigation that requires synthesis and analysis across multiple contexts and non-routine applications
- 16. Example: Selected-Response Test Smarter Balanced Assessment Consortiuim, 2012
- 18. Example: Animated Performance Test
- 19. Example: Performance Test • Skateboarding • Two-part, evidence-based question • Familiarize teachers with PARCC-like assessments • Questions can be asked at different depths of knowledge Arizona Department of Education, 2012
- 21. Alan Gelfand Biography Alan Gelfand - Growing Up Alan Gelfand was born in New York but moved to Florida when he was just four. He started to skateboard as an 11 year-old after his father bought him his first skateboard. The sport was beginning to gain popularity and in 1977, a skate park was built right in Alan's hometown of Hollywood, Florida. It was there that young Alan and his skating buds would invent and perfect new tricks. Alan Gelfand - The Ollie The origin of the ollie was the result of a creative skateboarder and an imperfect skate park. In the summer of 1978, Alan made the best of the poorly built park. He credits the less-than-perfect walls and the over-vertical sections of the bowl as reasons why he even attempted to come up with the now famous stunt in the first place. An ollie is a no-hands aerial trick, where it looks like the skateboard is stuck to your feet in the air, but is not attached by anything at all. Alan's friends named the trick after Alan's nickname "Ollie", which was given to him just months earlier. The ollie serves as the basis for many of the basic tricks in skateboarding today.
- 22. Questions Can Be Asked at Different Depths of Knowledge Depth of Knowledge DOK1 DOK2 DOK3 DOK4 In what year was What might happen if you Explain how to do one of the Research Gelfand born? came down with just one other tricks listed on the skateboarding videos. foot on the skateboard while “more complicated moves” Present the history of trying to perform an ollie? website. a move, who invented it, and how it can be Compare/contrast how the mastered. trick was explained through the article vs. the Website.
- 23. Issues in Moving Assessment from the Laboratory to the School • Quality – Reliability (of scoring and of student scores) – Validity • Practicality – Administrative time, complexity – Demands for equipment and materials – Response time and format – Cost • Usability – Formative uses: to improve instruction – Summative uses: confirm college- and career-readiness
- 24. Plan: Investigate and Report on the Current State of the Art • Assessing 21st Century skills in schools • From the perspective of educators and educational systems
Editor's Notes
- Tony Jackson asked RAND to help the Global Cities Education Network think about assessment of 21st Century skills from the perspective of educators and education system leaders. A preview of a paper we would like to prepare to offer some practical insights into the challenge of measuring 21st Century skills.
- I’m most familiar with the discussion of 21st Century skills in a recent report from the US National Research Council. They include academic skills at higher levels of cognitive demand, interpersonal skills, and self-regulatory within-the-person skills.
- Acknowledge that there is not a consensus about defining 21st Century skills, although difference conceptions have similarities. This slides shows how one might map different frameworks onto a common set of constructs. At some point it will be important to agree on naming conventions, but for now it seems OK that different perspectives seem to overlap to a large extent.
- The question of assessment is relevant because the members of the Global Cities Education Network are taking steps to incorporate 21st Century skills into their curricula.
- At present most academic outcomes are measured through highly structured, standardized formats, like multiple-choice tests. Those are not adequate for measuring the range of 21st Century skills. We need to broaden the methods we use to assess student skills.Self-Ratings: examples:cooperation (“I like to work with people”), leadership (“I like leading groups”), on time (“I complete my language arts homework on time”), planning (“I like to make lists of things to do for school”) Ratings by Others: examples include Likert scales, behaviorally-anchored scales, anchoring vignettes, forced choiceObservational scales: a type of rating by othersBiographical Data: activities engaged in (think college application, portfolio)Performance Tests: collaborative problem solving, creativity, quantitative literacySelected response tests: with higher levels of cognitive demand. A MC test can still be challenging.
- Educational systems will have to adapt to incorporate these different kinds of measurement. Teachers will need to be trained. More time will have to be allocated to assessment. Fixed schedules may have to be made more flexible. Computers and high-speed access will have to be provided. Students will have to be prepared for different activities and performances. Other adults will have to give input into educational space. Etc.
- Examples to illustrate the different kinds of assessments.
- These items come from a resilience scale. It has some validity if people are honest when they rate themselves, but that may not be the case with students whose answers are important for their future. And it we don’t know how teachers will interpret the results and how it will affect their intertactions with students.
- ETS Personal Potential Index. Student provides contact information for the evaluators you would like to complete an ETS PPI evaluation. Each evaluator logs in to the ETS PPI system to rate you on six personal attributes and provide an overall evaluation. Evaluators also may provide optional comments for each attribute as well as for the overall rating.How do students choose evaluators? Will evaluators have access to technology? Will they have similar standards of comparison and similar willingness to follow behavioral rubrics?
- Students might be asked to assemble evidence to show they have overcome adversity, show fortitude, pressed on in the face of disappointment, etc. Sample essay from college senior, but you can imagine the complications of student self affirmation of experiences.
- Turn to a different skill area.
- This is part of a rubric for rating teaching practice. It takes a lot of time and effort to train observers to understand the rubric and use it accurately. If we are rating the behaviors of students in a group interaction, for example, how much effort will be required to get reliable ratings?
- Asia Society Graduation Portfolio System measures, among other things, the development of global competence.The GPS performance outcomes and rubrics (first green box) are the foundation, providing a definition of “what success looks like” by delineating the knowledge, skills, and dispositions students need to be college ready and globally competent. The GPS performance cycle (the four blue boxes in the middle) is implemented through a series of authentic learning experiences with embedded formative and summative tasks. The performance cycle is a series of continuous processes that drive improvement across the curriculum, both in school and in out-of-school time.The GPS portfolio (third purple box) is made up of the student work that results from multiple performance cycles across all courses and learning experiences, which certifies that the student is both college ready and globally competent by graduation (final red box).Using this approach involves changes to normal pattern of classroom activity, training of teachers to understand outcomes and rubrics, etc.
- Schools are more familiar measuring academic skills, although 21st C skills are more cognitively complex
- Norman Webb devised four-level taxonomy of the demands of an assessment. One of the emphases in 21st C skills is higher-order thinking, like levels 3 and 4.
- From SBAC. Illustrates that simple choice question can be cognitively demanding.
- From PISA, a multi-step test, that presumes some familiarity with computer interface, and an ability to work independently.
- From SBAC. The worker and the ladder are animated. So students must understand that they must wait for the explanation to be given before they undertake the task. Will schools want to incorporate such illustrations into instruction to prepare students for the assessments?
- From the Arizona Department of Education. This illustrates the multiple ways a prompt can be used to assess at different levels.
- Examples designed to illustrate some of the challenges in moving 21st Century skills assessment from a research topic to an operational education tool.
- The study would involve communicating with educators and schools that are implementing these kinds of assessments and uncovering what the real implementation challenges are for system and school level educators.