Your SlideShare is downloading. ×
0
How Does the World Assess Student Learning? Carol Campbell, Stanford Center for Opportunity Policy in Education (SCOPE) As...
A New Day in Assessment <ul><li>“ I am calling on our nation’s Governors and state education chiefs to develop standards a...
Reform of Standards, Curriculum, and Assessment is Underway World Wide <ul><li>… The goal is less dependence on rote learn...
U.S. Scores on PISA and PIRLS Have Dropped Since 2000
International Outcomes (8 th  Grade PISA Results in OECD Nations, 2006) <ul><li>Science </li></ul><ul><li>Finland </li></u...
Differences Among Assessments <ul><li>Most U.S. standardized tests are designed to assess if students learned what they we...
Two Kinds of Tests <ul><li>After a person learns to drive, he or she goes to the DMV to be tested.  There, the person take...
How the demand for skills has changed Economy-wide measures of routine and non-routine task input (U.S.) (Levy and Murnane...
Expectations for Learning are Changing <ul><li>The new context means new expectations. Most studies include: </li></ul><ul...
High-Achieving Countries Rely Increasingly on Performance Assessments <ul><li>While multiple choice testing predominates i...
Assessment Design Principles Effective Assessment Systems Designed around a clear vision of purpose and learning goals: 21...
1. Clear Vision of Purpose and Learning Goals - Specifying 21 st  century skills, college and career-readiness <ul><li>Hig...
Purpose and learning goals - 21 st  Century Skills:  Ontario - Achievement Chart – English, Grades 9-12 – Thinking Skills ...
2. A Teaching and Learning System: Align with and advance common standards, curriculum expectations, learning objectives a...
3. Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning...
Singapore GCE A-Level Examinations
GRADE 12 GRADING (Diploma Subject),  Alberta - Canada Classroom 50% Examination 50% <ul><li>Unit and Final Tests,  </li></...
4. Embody high expectations to engage, motivate, support and stretch all students through assessing knowledge, skills and ...
S INGAPORE  E XAMINATIONS AND  A SSESSMENT  B OARD <ul><li>To Assess Experimental Skills and Investigations, Students…  </...
GCSE ICT Task (England) <ul><li>Litchfield Promotions works with over 40 bands and artists to promote their music and put ...
Alberta Social Studies 30 Diploma Examination Questions <ul><li>The Nazis’ systematic genocide of ethnic and religious gro...
Example from Biology 30 Bulletin, Alberta <ul><li>January 2008 Biology 30 Diploma Examination Open-Response Question  </li...
Time-Line of the Effects of Thalidomide on Embryonic Development <ul><li>Scientists have genetically engineered mice that ...
<ul><li>Identify  two areas of the brain that can be affected in an individual with autism. </li></ul><ul><li>Explain  the...
<ul><li>Are students required/expected to: </li></ul>5. Assess Range of Knowledge in Authentic, Applied Ways: English Lang...
Applying Knowledge: A Literacy Task  (Grade 10, Ontario – Canada) <ul><li>Task:  Write a  minimum  of  three paragraphs  e...
Applying Knowledge:  A Mathematics Task (Ontario)
A Rich Task: Science and Ethics Confer  (Queenland, Australia) <ul><li>Students must identify, explore, and make judgments...
6. Are part of a clear and flexible learning trajectory  National Qualifications Framework, England: Learner Pathways
7. Build and Develop Professional Capacity – Teachers are integrally involved <ul><li>Across higher performing countries, ...
Professional Capacity: Test Design  (Alberta, Canada) <ul><li>Example: Teacher collaboration on assessment design and mark...
Teacher Collaboration – Reviewing Tests, Confirming Standards (Alberta) <ul><ul><li>Reviewing Tests: </li></ul></ul><ul><u...
Teacher Collaboration –  Marking and Advising (Alberta) <ul><ul><li>The Marking Process: </li></ul></ul><ul><ul><li>Each J...
Scoring of Assessments - Ontario <ul><li>EQAO is an independent agency that coordinates all assessments and administers an...
Teachers who engaged consistently in moderation processes were able to: <ul><li>Assess student performance more consistent...
Key Roles in the Teacher Moderation Process <ul><li>Principal’s Role </li></ul><ul><li>Schedules regular moderation sessio...
Using Assessments to Inform Improvements: Example: Ontario Teaching-Learning Critical Pathway . Gathering Evidence Area of...
<ul><li>New technologies may support assessment quality and timeliness, as well as information systems that support report...
Technology Uses <ul><li>To distribute and administer assessments  </li></ul><ul><li>To enable simulations, research tasks,...
Students School Boards/Authorities Teachers, Principals & Administrators <ul><li>Funding </li></ul><ul><li>Policy/Programs...
9. Place quality of assessment over quantity <ul><li>Examples: </li></ul><ul><li>Australia – National standards (underway)...
10. Encompass and combine Assessment  of, for   and  as  learning  <ul><li>Assessment… </li></ul><ul><li>Of   learning - s...
Hong Kong: Learning to Learn and Use of School-Based Assessments   <ul><li>“ The primary rationale for School Based Assess...
<ul><li>“ If you want to appear accountable, test your students.  </li></ul><ul><li>If you want to improve schools, teach ...
Assessment Design Principles Effective Assessment Systems Designed around a clear vision of purpose and learning goals: 21...
Race To The Top Assessment Grants <ul><li>$350 million  for state consortia to develop new assessments “that are valid, su...
How Might U.S. Assessment Systems Become Internationally Comparable?   <ul><li>Develop a  coherent ,  aligned and integrat...
SCOPE: Commissioned Papers http://edpolicy.stanford.edu <ul><li>Suzanne Lane,  Performance Assessment: The State of the Ar...
Thank You! [email_address] http://edpolicy.stanford.edu
Upcoming SlideShare
Loading in...5
×

measure student learning

1,400

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,400
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
56
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Discuss fundamental difference in focus of tests.
  • Notes:
  • Notes for the facilitator The commonly agreed list of capacities expected of individuals in the modern workplace that is outlined on this and the following slide (from Professor Cheng’s paper provided as a pre-reading in the participants’ handbook) constitutes a key reason for the adoption of the 334 senior secondary reform programme. The new curriculum envisaged in 334 is designed not only to build Hong Kong’s knowledge base by ensuring all students receive a full six years of secondary education, but also developing the broader range of knowledge based skills required for success in today’s highly competitive global marketplace. (The list continues on the next slide)
  • Students must construct an answer, produce a product, or perform an activity. PA may range from a short response or problem solution, to writing an essay, to designing, conducting, and analyzing a laboratory experiment. PA measures students’ reasoning skills and their ability to apply knowledge to frame and solve meaningful problems.
  • Assessment criteria for Functional Skills – “Assessment must, according to the skill areas: Provide realistic contexts, scenarios and problems Specify tasks that are relevant to the context Require application of knowledge, skills and understanding for a purpose Require problem solving Assess process skills and the outcome of their application in different contexts Generally assessments externally defined (exam boards) and internally assessed (teachers), including controlled assessments in classroom and projects. Combinations of open and fixed response items varies by subject area and level. Performance assessments increasingly used in range of qualifications – coursework, projects, extended writing Opportunities for accredited work experience in some qualifications, e.g. Diploma composite qualification includes 10 days work experience Singapore – Knowledge and Inquiry Essay: Why ask questions? What is knowledge? How is knowledge constructed? What makes knowledge valid? How is knowledge affected by society? How should knowledge be used?
  • Time-based written examinations For the time-based written examinations, say at the H2 Level, students offer between 2 to 4 papers and each paper is about 3 hours in duration. The format of the assessment items in A-Level papers would typically be open-ended essays/free response questions, structured questions, case studies and source-based questions. The design of the assessment specifications and assessment criteria, item setting, marking/scoring are done externally by the Board in collaboration with CIE. School-based coursework Assessment We have coursework examinations for a number of subjects which includes Project Work and Knowledge and Inquiry. The assessment tasks are typically for a duration of about 6 months and may involve producing a piece of product, oral presentation or an independent study/extended essay (H2: 2,000-2,500 words; H3: 3,000 – 3,500 words). Our School-based Assessment of coursework is tightly defined in that the assessment tasks are externally set by the Board/CIE but they are internally marked by the school teachers and externally moderated by the Board/CIE.
  • For the sciences, we have introduced a component called “School-based Science Practical Assessment” (SPA) (20% weighting). SPA, which takes place over an appropriate period, seeks to develop our students the ability to: 1. follow a detailed set or sequence of instructions and use techniques, apparatus and materials safely and effectively; 2. Making and recording observations, measurements, methods and techniques with precision and accuracy; 3. interpret and evaluate observations and experimental data; 4. identify a problem, design and plan investigations, evaluate methods and techniques, and suggest possible improvement. SPA assesses the following key areas: Manipulation, measurement and observation (MMO) Presentation of data and observations (PDO) Analysis, conclusions and evaluation (ACE)
  • Example: General Certificate of Secondary Education (GCSE) (age 15-16), England GCSEs are available in over 50 subjects: students choose. GCSEs normally take 2 years to complete. Many courses offer assessment in each unit rather than only at the end. GCSEs use constructed response items, and usually include structured classroom-based tasks that are more extended. Increased emphasis on extended writing and course work. Controlled assessments are taken under supervised conditions and are either set by the awarding body and marked by teachers or set by teachers and marked by the awarding body. Teachers determine the timing of controlled assessments. Controlled assessments comprise up to 60% of a grade, depending on the subject. Introduction of assessment of functional skills and personal learning and thinking skills across subjects Cost of assessing one unit = $10 approx. Average costs of a GCSE course assessment = $40 approx Controlled assessments: 25% of the assessment in: business studies, classical civilisation, English literature, geography, history, humanities, statistics, and 60% of the assessment in: applied business, art and design, citizenship studies, construction and the built environment, dance, design and technology, drama, engineering, English, English Language, expressive arts, health and social care, home economics, hospitality and catering, ICT, leisure and tourism, manufacturing, media studies, modern foreign languages, music, physical education
  • Key Stage 4 onwards – 14-19 curriculum – increasing range of options and pathways through academic and vocational routes into training, college, university, employment. Example: Exam boards in England ensure consistency of standards within subjects while allowing flexibility of approach across subjects. Five exam boards - governed by specifications, criteria, frameworks and oversight of Office of the Qualifications and Examinations (OFQUAL) Regulator Exam boards: set out the course syllabus &amp; specifications to meet criteria set by the regulator oversee writing of exam questions and how exam papers should be marked check coursework/controlled assessments provide training and quality control for examiners, including use of ‘standardization meetings’ to compare marking across examiners set grade boundaries to align marks and standards
  • Example: Teacher collaboration on assessment design and marking, Alberta, Canada Test design: Identify student characteristics Assist in exam blueprint document Ensure curricular “fit” of the exam Pilot prototype multiple-choice and written response forms Help develop writing assignments and their scoring Item building: Item development sessions held throughout the province involve teachers in the creation of new multiple-choice and written responses . Reviewing Tests: Each new examination form is reviewed by a committee that includes classroom teachers. The committee examines both the written response and multiple choice sections to ensure that the examination is fair, and demonstrates fidelity to the curriculum. Confirming Standards: Before written responses are marked a committee of teachers meets with examination branch staff to select student work for use in marker training (example papers, training papers and reliability review papers). The Marking Process: Each January and June approximately 1300 teachers meet to mark the written responses in a process with tight calibration to produce consistency in scoring. Examination Advisory Committee: Once yearly representatives of various stakeholder groups, including the Alberta Teachers’ Association meet with branch staff to review examination results from the previous school year, and to offer suggestions for the improvement of the examination program.
  • Notes:
  • Notes:
  • Seven Centres have been established across Ontario: Six regional, English-language Centres; One province-wide, French-language Centre PNCs function as linked professional learning communities PNCs complement the efforts of individual boards and schools as they build capacity to work with data in support of evidence-informed decision-making Ontario School Information System (OnSIS) receives information from each school and board (web based system) OnSIS collects data on courses, classes, students and educators Data Collection occurs three times a year (October, March and June) Benefits More efficient Better data integrity Student level data Allows for greater data correlation Elementary/Secondary Data Warehouse (ESDW) holds depersonalized data Enables analysis and reporting of student achievement over time Supports policies and programs (e.g., Student Success, etc.)
  • Research indicates that when consistently carried out as a matter of routine within and across classrooms, assessment practices that focus on promoting learning in the classroom have been linked to achievement gains of one-half to two standard deviations on. Black and William (1998), after a systematic review of several studies, concluded that assessment practices that support instruction can produce gains with typical effect sizes of between 0.4 and 0.7. This gain is large enough to raise a country’s position in the Third International Mathematics and Science Study (TIMMS) from the middle of the 41 countries involved to being one of the top five. The largest gains are made by those students usually found at the lower one-third of the achievement hierarchy. “ If you want to appear accountable, test your students. If you want to improve schools, teach teachers to assess their students. If you want to maximize learning, teach students to assess themselves.”
  • “ Learning to Learn” (2001) – goal is to shape curriculum and instruction around: Critical thinking Problem solving Self-management skills Collaboration Students’ meta-cognitive thinking skills “ SBA, which typically involves students in activities such as : making oral presentations, developing a portfolio of work, undertaking fieldwork, carrying out an investigation, doing practical laboratory work or completing a design project, help students to acquire important skills, knowledge and work habits that cannot readily be assessed or promoted through paper-and-pencil testing. Not only are they outcomes that are essential to learning within the disciplines, they are also outcomes that are valued by tertiary institutions and by employers.”
  • Discuss that regulations have been released. How they are based on Innovation. What does it mean for teachers? How can this competition promote new assessments and standards in the United States that are more in line with those abroad?
  • Transcript of "measure student learning"

    1. 1. How Does the World Assess Student Learning? Carol Campbell, Stanford Center for Opportunity Policy in Education (SCOPE) Asia Society Partnership for Global Learning 3 rd Annual Conference July 9, 2010
    2. 2. A New Day in Assessment <ul><li>“ I am calling on our nation’s Governors and state education chiefs to develop standards and assessments that don’t simply measure whether students can fill in a bubble on a test, but whether they possess 21st century skills like problem-solving and critical thinking, entrepreneurship and creativity.” </li></ul><ul><li>-- President Barack Obama </li></ul><ul><li> March 10, 2009 </li></ul>
    3. 3. Reform of Standards, Curriculum, and Assessment is Underway World Wide <ul><li>… The goal is less dependence on rote learning, repetitive tests and a ‘one size fits all’ type of instruction, and more on engaged learning, discovery through experiences, differentiated teaching, the learning of life-long skills, and the building of character through innovative and effective teaching approaches and strategies… </li></ul><ul><li>- Singapore Education Minister Tharman Shanmugaratnam, 2005 </li></ul>
    4. 4. U.S. Scores on PISA and PIRLS Have Dropped Since 2000
    5. 5. International Outcomes (8 th Grade PISA Results in OECD Nations, 2006) <ul><li>Science </li></ul><ul><li>Finland </li></ul><ul><li>Canada </li></ul><ul><li>Japan </li></ul><ul><li>New Zealand </li></ul><ul><li>Australia </li></ul><ul><li>Netherlands </li></ul><ul><li>Korea </li></ul><ul><li>Germany </li></ul><ul><li>United Kingdom </li></ul><ul><li>U.S. is # 29 / 40 </li></ul><ul><li>top nations </li></ul>Math Finland Korea Netherlands Switzerland Canada Japan New Zealand Belgium Australia U.S. is #35 / 40 top nations
    6. 6. Differences Among Assessments <ul><li>Most U.S. standardized tests are designed to assess if students learned what they were taught in school, focusing on recall and recognition of facts. </li></ul><ul><li>PISA is a set of international tests designed to assess if students can apply what they’ve learned to new problems and situations, focusing on inquiry and explanations of ideas. </li></ul><ul><li>Shift from ‘did students learn what we taught them?’ to ‘what can students do with what they have learned?’ </li></ul><ul><li>Assessments in high-achieving nations increasingly emphasize demonstrations of learning authentic contexts. </li></ul>
    7. 7. Two Kinds of Tests <ul><li>After a person learns to drive, he or she goes to the DMV to be tested. There, the person takes two different test to demonstrate his or her capability to be a licensed driver. </li></ul><ul><li>One test is a multiple-choice test. The person must answer correctly questions about the laws related to driving. </li></ul><ul><li>The other test is a performance test. The person must skillfully drive a car in a variety of road situations. </li></ul><ul><li>Which test is most important? </li></ul><ul><li>Would you feel confident in knowing that a new driver had only been tested in one of these ways? </li></ul>
    8. 8. How the demand for skills has changed Economy-wide measures of routine and non-routine task input (U.S.) (Levy and Murnane) Mean task input as percentiles of the 1960 task distribution
    9. 9. Expectations for Learning are Changing <ul><li>The new context means new expectations. Most studies include: </li></ul><ul><li>Ability to communicate </li></ul><ul><li>Adaptability to change </li></ul><ul><li>Ability to work in teams </li></ul><ul><li>Preparedness to solve problems </li></ul><ul><li>Ability to analyse and conceptualise </li></ul><ul><li>Ability to reflect on and improve performance </li></ul><ul><li>Ability to manage oneself </li></ul><ul><li>Ability to create, innovate and criticise </li></ul><ul><li>Ability to engage in learning new things at all times </li></ul><ul><li>Ability to cross specialist borders </li></ul>Chris Wardlaw, &quot;Mathematics in Hong Kong/China – Improving on Being First in PISA&quot;
    10. 10. High-Achieving Countries Rely Increasingly on Performance Assessments <ul><li>While multiple choice testing predominates in the U.S., most high- achieving countries largely use written & oral examinations, plus samples of student work, such as research projects and exhibitions, to evaluate what students have learned. </li></ul>To “B” or Not to “B” ……
    11. 11. Assessment Design Principles Effective Assessment Systems Designed around a clear vision of purpose and learning goals: 21 st Century Skills A Teaching and Learning System: Align with and advance common standards, curriculum expectations, learning objectives and instructional strategies Encompass and combine assessment of, for and as learning Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning Embody high expectations to engage, motivate, support and stretch all students to progress and achieve Build and develop professional capacity Assess a range of content, knowledge, skills and performance in authentic, applied and appropriate ways Are part of a clear and flexible learning trajectory across subjects and over education/career pathways Support data- informed decision-making for local and system improvement Place quality of assessment over quantity
    12. 12. 1. Clear Vision of Purpose and Learning Goals - Specifying 21 st century skills, college and career-readiness <ul><li>High school examinations in England combine subject/curricular knowledge with: </li></ul><ul><li>Functional skills – practical skills in English, mathematics and information and communication technology” (ICT). </li></ul><ul><li>Personal learning and thinking skills (PLTS) – “require learners to be: Independent enquirers; Creative thinkers; Reflective learners; Team workers; Self-managers; Effective participants. </li></ul><ul><li>Singapore: Life Skills and Knowledge Skills </li></ul><ul><li>Example of Knowledge and Inquiry as a Humanities subject: </li></ul><ul><li>Essay on theory and application of knowledge </li></ul><ul><li>Critical thinking paper to analyze different information and arguments </li></ul><ul><li>Independent study resulting in extended essay of 2,500 – 3,000 words </li></ul>
    13. 13. Purpose and learning goals - 21 st Century Skills: Ontario - Achievement Chart – English, Grades 9-12 – Thinking Skills <ul><li>Thinking – The use of critical and creative thinking skills and/or processes </li></ul>Categories 50-59% (L1) 60-69% (L 2) 70-79% (L 3) 80-100% (L4) The student: Use of planning skills (e.g., generating ideas, gathering information, focusing research, organizing information) uses planning skills with limited effectiveness … some effectiveness … considerable effectiveness … high degrees of effectiveness Use of processing skills (e.g. drawing inferences, interpreting, analysing, synthesizing, evaluating) uses processing skills with limited effectiveness … some effectiveness … considerable effectiveness … high degrees of effectiveness Use of critical/creative thinking processes (e.g. oral discourse, research, critical analysis, critical literacy, metacognition, creative process) uses critical/creative thinking processes with limited effectiveness … some effectiveness … considerable effectiveness … high degrees of effectiveness
    14. 14. 2. A Teaching and Learning System: Align with and advance common standards, curriculum expectations, learning objectives and instructional strategies <ul><li>Example: Ontario, Canada </li></ul><ul><li>Content Standards </li></ul><ul><ul><li>Curriculum for Grades 1-12 for all subjects and courses </li></ul></ul><ul><ul><ul><li>Developed through extensive research, benchmarking, and consultation with all stakeholders </li></ul></ul></ul><ul><ul><ul><li>All teachers are required to teach the curriculum expectations </li></ul></ul></ul><ul><li>Performance Standards (criterion-referenced) </li></ul><ul><ul><li>Criteria are described for student achievement of the curriculum expectations for Grades 1-12 </li></ul></ul><ul><ul><ul><li>Four levels of achievement (provincial standard is Level 3, equivalent to ‘B’ or 70-79% </li></ul></ul></ul><ul><ul><ul><li>Four categories of knowledge and skills: Knowledge and Understanding; Thinking; Communication; Application </li></ul></ul></ul><ul><li>Consistency: </li></ul><ul><ul><li>Exemplars of student work for all curriculum </li></ul></ul><ul><ul><li>Standard provincial report cards - aligned with content and performance standards </li></ul></ul>
    15. 15. 3. Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning <ul><li>Multi-faceted assessments used internationally include: </li></ul><ul><li>Assessments of knowledge (recall & analysis) and assessments of performance (demonstration of ability to apply knowledge in practice) </li></ul><ul><li>Multiple-choice, constructed-response, extended tasks and projects </li></ul><ul><li>On-demand and curriculum-embedded elements </li></ul><ul><li>Externally-developed (by teachers and developers) and classroom-developed/managed </li></ul><ul><li>Scoring made consistent through training, moderation, calibration, and auditing </li></ul>
    16. 16. Singapore GCE A-Level Examinations
    17. 17. GRADE 12 GRADING (Diploma Subject), Alberta - Canada Classroom 50% Examination 50% <ul><li>Unit and Final Tests, </li></ul><ul><li>Quizzes </li></ul><ul><li>Projects (group/individual) </li></ul><ul><li>Labs </li></ul><ul><li>Portfolios </li></ul><ul><li>Oral Presentations </li></ul><ul><li>Exhibitions </li></ul>Machine Scoreable 50% - 70% of items 50% of score weight Open – Ended 30% - 50% of items 50% of score weight
    18. 18. 4. Embody high expectations to engage, motivate, support and stretch all students through assessing knowledge, skills and performance <ul><li>For Example: Project Work in Singapore, England, and International Baccalaureate </li></ul><ul><li>Interdisciplinary coursework </li></ul><ul><li>Extensive research (4000 word essay) </li></ul><ul><li>Oral presentation </li></ul><ul><li>Both product and process are assessed </li></ul><ul><li>In Singapore, collaborative learning through group work is required and assessed </li></ul>
    19. 19. S INGAPORE E XAMINATIONS AND A SSESSMENT B OARD <ul><li>To Assess Experimental Skills and Investigations, Students… </li></ul><ul><li>Identify a problem, design and plan an investigation, evaluate their methods and techniques </li></ul><ul><li>Follow instructions and use techniques, apparatus and materials safely and effectively </li></ul><ul><li>Make and record observations, measurements, methods, and techniques with precision and accuracy </li></ul><ul><li>Interpret and evaluate observations and experimental data </li></ul>School-based science practical assessment, Singapore
    20. 20. GCSE ICT Task (England) <ul><li>Litchfield Promotions works with over 40 bands and artists to promote their music and put on performances in England... Litchfield Promotions needs to create an ICT solution to ensure that they have all necessary information and that it is kept up to date… </li></ul><ul><li>Candidates will need to: </li></ul><ul><li>Work with others to plan and carry out research to investigate how similar companies have produced a solution…. </li></ul><ul><li>2) Clearly record and display your findings. </li></ul><ul><li>3) Recommend a solution… </li></ul><ul><li>4 ) Produce a design brief… </li></ul><ul><li>Produce a solution , ensuring that the following are addressed: </li></ul><ul><li>It can be modified to be used in a variety of situations. </li></ul><ul><li>2) It has a friendly user interface. </li></ul><ul><li>3) It is suitable for the target audience. </li></ul><ul><li>4) It has been fully tested. </li></ul><ul><li>You will need to: 1) incorporate a range of: software features, macros, modeling, and validation checks - used appropriately. 2) Obtain user feedback. 3) Identify areas that require improvement, recommending improvement, with justification. 4) Present information as an integrated document. 5) Evaluate your own and others’ work. </li></ul>
    21. 21. Alberta Social Studies 30 Diploma Examination Questions <ul><li>The Nazis’ systematic genocide of ethnic and religious groups during the Second World War provided impetus for the: </li></ul><ul><li>a) creation of a new autonomous state in Central Europe </li></ul><ul><li>b) establishment of a war crimes tribunal at Nuremberg </li></ul><ul><li>c) exclusion of Germany from the original membership of NATO </li></ul><ul><li>d) establishment of agencies fostering European economic and political cooperation </li></ul>
    22. 22. Example from Biology 30 Bulletin, Alberta <ul><li>January 2008 Biology 30 Diploma Examination Open-Response Question </li></ul><ul><li>(Written Response 2), Sample Responses and Scoring Guide </li></ul><ul><li>Use the following information to answer the next question. </li></ul><ul><li>Autism is a complex behavioural disorder. The symptoms of autism vary greatly and occur in different combinations. Symptoms include a reduced ability to communicate, a reduced ability to develop relationships, difficulty coordinating facial muscles, and difficulty interpreting social cues. </li></ul><ul><li>In the late 1950s and early 1960s, the drug thalidomide was prescribed to pregnant women to combat morning sickness. Thalidomide was found to cause birth defects, such as stunted growth of the arms and legs. Some children also developed autism as a result of being exposed to thalidomide in the utero. In comparison with the general population, the frequency of autism is many times higher in people with birth defects caused by thalidomide, which suggests that autism may originate early in embryonic development. </li></ul>
    23. 23. Time-Line of the Effects of Thalidomide on Embryonic Development <ul><li>Scientists have genetically engineered mice that have symptoms similar to those of autism. These mice have a defective copy of the Hoxal gene, which is also present in humans, is normally active only during very early embryonic development. </li></ul><ul><li>Although people affected with autism are more likely to have the defective Hoxal gene than people without the disorder, the presence of the effective gene does not ensure the development of autism. Further investigation is required to determine whether enviromental factors work in conjunction with genes to produce autism. </li></ul><ul><li>Rodier, Patricia M. 2000. The early origins of Autism. </li></ul><ul><li>Scientific American , February, 56 -63 </li></ul>Age of embryo 20 21 22 23 24 25 26 27 28 29 30 31 32 33 Nursing Mothers Small ears and other malformations Stunted arms Stunted legs
    24. 24. <ul><li>Identify two areas of the brain that can be affected in an individual with autism. </li></ul><ul><li>Explain the relationship between the areas of the brain identified and the symptoms of autism. </li></ul><ul><li>Identify one germ layer in which development could be disrupted by thalidomide and identify one structure that develops from this germ layer. Hypothesize how a person who has autism as a result of in utero exposure to thalidomide can have abnormal ear development but no malformations of the arms or legs. </li></ul><ul><li>Describe how the defective Hoxal protein is synthesized in the cytoplasm of a cell. Explain how the defective Hoxal protein influences brain development and can lead to autism. </li></ul><ul><li>Identify and describe two technologies that can be used by scientists to replace an active Hoxal gene with a defective copy of the gene. </li></ul><ul><li>Describe three difficulties that researchers could encounter when they attempt to determine the cause of autism in humans. </li></ul>Time-Line of the Effects of Thalidomide on Embryonic Development - Continued
    25. 25. <ul><li>Are students required/expected to: </li></ul>5. Assess Range of Knowledge in Authentic, Applied Ways: English Language Arts -Constructing Scoring Rubrics: Identifying the Scoring Categories (Alberta) Discuss the relationship among sources? Demonstrate an understanding of multiple perspectives? Apply learned concepts to new situations? Demonstrate control of language conventions? Interpret a source or sources? Recall specific or factual knowledge? Take and defend a position? Demonstrate understandings of broad concepts?
    26. 26. Applying Knowledge: A Literacy Task (Grade 10, Ontario – Canada) <ul><li>Task: Write a minimum of three paragraphs expressing an opinion on the topic below. Develop your main idea with supporting details (proof, facts, examples, etc.). </li></ul><ul><li>Purpose and Audience: an adult who is interested in your opinion </li></ul><ul><li>Length: The lined space provided for your written work indicates the approximate length of the writing expected. </li></ul><ul><li>Topic: Are today’s famous people good role models for young people? </li></ul>
    27. 27. Applying Knowledge: A Mathematics Task (Ontario)
    28. 28. A Rich Task: Science and Ethics Confer (Queenland, Australia) <ul><li>Students must identify, explore, and make judgments on a biotechnological process to which there are ethical dimensions . Students identify scientific techniques used as well as significant recent contributions to the field. They will also research frameworks of ethical principles for coming to terms with an identified ethical issue or question. Using this information, they prepare pre-conference materials for an international conference that will feature selected speakers who are leading lights in their respective fields . </li></ul><ul><li>In order to do this, students must choose and explore an area of biotechnology where there are ethical issues under consideration and undertake laboratory activities that help them understand some of the laboratory practices . This enables them to: </li></ul><ul><li>a) Provide a written explanation of the fundamental technological differences in some of the techniques used, or of potential use, in this area (included in the pre-conference package for delegates who are not necessarily experts in this area). </li></ul><ul><li>b) Consider the range of ethical issues raised in regard to this area’s purposes and actions, and scientific techniques and principles, and present a deep analysis of an ethical issue about which there is a debate in terms of an ethical framework. </li></ul><ul><li>c) Select six real-life people who have made relevant contributions to this area and write a 150-200 word précis about each one indicating his/her contribution, as well as a letter of invitation to one of them. </li></ul>
    29. 29. 6. Are part of a clear and flexible learning trajectory National Qualifications Framework, England: Learner Pathways
    30. 30. 7. Build and Develop Professional Capacity – Teachers are integrally involved <ul><li>Across higher performing countries, teachers are integrally involved in assessment design, validation, administration, moderation, and application for teaching and learning </li></ul><ul><li>Finland – teacher quality: </li></ul><ul><ul><li>teacher preparation includes strong focus on how to use formative performance assessment in service of student learning; </li></ul></ul><ul><ul><li>teachers use and adapt national curriculum and assessment criteria to develop more detailed curriculum, learning outcomes and assessment benchmarks </li></ul></ul><ul><li>Australia (Queensland) - school-based assessment: </li></ul><ul><ul><li>Teachers develop, administer and score school-based assessments in relation to the national curriculum and state syllabi, and panels involving teachers and college faculty moderate assessments </li></ul></ul><ul><li>Canada (Alberta, Ontario) – provincial/state assessments: </li></ul><ul><ul><li>Teachers integrally involved in: test design; item building; reviewing tests; confirming standards; marking process; examination advisory committee. </li></ul></ul>
    31. 31. Professional Capacity: Test Design (Alberta, Canada) <ul><li>Example: Teacher collaboration on assessment design and marking, Alberta, Canada </li></ul><ul><li>Test design: </li></ul><ul><ul><li>Identify student characteristics </li></ul></ul><ul><ul><li>Assist in exam blueprint document </li></ul></ul><ul><ul><li>Ensure curricular “fit” of the exam </li></ul></ul><ul><ul><li>Pilot prototype multiple-choice and written response forms </li></ul></ul><ul><ul><li>Help develop writing assignments and their scoring </li></ul></ul><ul><ul><li>Item building: </li></ul></ul><ul><ul><li>Item development sessions held throughout the province involve teachers in the creation of new multiple-choicce and written responses. </li></ul></ul>
    32. 32. Teacher Collaboration – Reviewing Tests, Confirming Standards (Alberta) <ul><ul><li>Reviewing Tests: </li></ul></ul><ul><ul><li>Each new examination form is reviewed by a committee that includes classroom teachers. </li></ul></ul><ul><ul><li>The committee examines both the written response and multiple choice sections to ensure that the examination is fair, and demonstrates fidelity to the curriculum. </li></ul></ul><ul><ul><li>Confirming Standards: </li></ul></ul><ul><ul><li>Before written responses are marked a committee of teachers meets with examination branch staff to select student work for use in marker training (example papers, training papers and reliability review papers). </li></ul></ul>
    33. 33. Teacher Collaboration – Marking and Advising (Alberta) <ul><ul><li>The Marking Process: </li></ul></ul><ul><ul><li>Each January and June approximately 1300 teachers meet to mark the written responses in a process with tight calibration to produce consistency in scoring. </li></ul></ul><ul><ul><li>Examination Advisory Committee: </li></ul></ul><ul><ul><li>Once yearly representatives of various stakeholder groups, including the Alberta Teachers’ Association meet with branch staff to review examination results from the previous school year, and to offer suggestions for the improvement of the examination program. </li></ul></ul>
    34. 34. Scoring of Assessments - Ontario <ul><li>EQAO is an independent agency that coordinates all assessments and administers and reports on provincial large-scale assessments </li></ul><ul><li>EQAO engages teachers in every step of the cycle of provincial assessments, from development and review to scoring (Approx. 2500 teachers per year). </li></ul><ul><li>Grade 9 Mathematics – raw score points = 46% multiple-choice, 54% open-response </li></ul><ul><li>Ontario Secondary School Literacy Test – raw score points = 48% multiple-choice, 52% open-response </li></ul><ul><li>Multiple choice items are machine scored. Written open-ended responses systematically scored by qualified scorers (mostly Ontario educators) </li></ul><ul><li>Use of generic rubric to describe levels of performance, item-specific rubric for each item, and anchors to illustrate descriptors for the score points to the rubric </li></ul><ul><li>Committee of educators define range of performance within each code of scoring rubric – ‘range finding’ development </li></ul><ul><li>Every scorer participates in extensive training. System of checks and monitoring applied throughout process </li></ul><ul><li>Quality assurance and statistical techniques to validate interrater reliability and scoring validity. Use of equating process to place student scores on common scale over two adjacent years and to determine cut scores </li></ul>
    35. 35. Teachers who engaged consistently in moderation processes were able to: <ul><li>Assess student performance more consistently, effectively, confidently, and fairly; </li></ul><ul><li>Build common knowledge about curriculum expectations and levels of achievement; </li></ul><ul><li>Identify strengths and areas for growth based on evidence of student learning; </li></ul><ul><li>Adjust and acquire new learning by comparing one’s thinking to that of another student or teacher; </li></ul><ul><li>Share effective practices to meet the needs of all students, monitor progress, and celebrate growth. </li></ul>Little, J.W. et al (2003). Looking at student work for teacher learning, teacher community and school reform. Phi Delta Kappan, 85(3), 185-192.
    36. 36. Key Roles in the Teacher Moderation Process <ul><li>Principal’s Role </li></ul><ul><li>Schedules regular moderation sessions; </li></ul><ul><li>Ensures resources are available for to interpret assessment results; </li></ul><ul><li>Participates in moderation sessions, learning side-by-side with teachers and increasing shared knowledge about students’ performance; </li></ul><ul><li>Supports opportunities for distributed leadership. </li></ul><ul><li>Teacher’s Role </li></ul><ul><li>Collectively discusses results to plan and refine instruction </li></ul><ul><li>Gives students feedback in a timely fashion to help improve performance; </li></ul><ul><li>Actively participates in sessions through asking effective questions; </li></ul><ul><li>Shares successful instructional strategies and resources with team, improving opportunities to support students’ individual needs </li></ul><ul><li>Student’s Role </li></ul><ul><li>Reflects on strengths and weaknesses in their learning; </li></ul><ul><li>Incorporates and applies feedback to continuously improve work. </li></ul>Adapted from Coalition of Essential Schools (CES) http://www.essentialschools.org/resources/60
    37. 37. Using Assessments to Inform Improvements: Example: Ontario Teaching-Learning Critical Pathway . Gathering Evidence Area of Greatest Need Current Practice Rubric, Data Wall, Culminating Task Professional Learning Community Action Moderated Marking Share Findings, Communication
    38. 38. <ul><li>New technologies may support assessment quality and timeliness, as well as information systems that support reporting and accountability. </li></ul>8. Support data-informed decision-making for local and system improvement
    39. 39. Technology Uses <ul><li>To distribute and administer assessments </li></ul><ul><li>To enable simulations, research tasks, and other means for evaluating applied learning </li></ul><ul><li>To support both human scoring and machine scoring of open-ended items </li></ul><ul><li>As a measure of the potential for technology to streamline performance testing, the National Assessment of Educational Progress has found that human and computer scoring of a set of physics simulations matches 96 percent of the time. </li></ul>
    40. 40. Students School Boards/Authorities Teachers, Principals & Administrators <ul><li>Funding </li></ul><ul><li>Policy/Programs </li></ul><ul><li>Public Reporting </li></ul>Ontario School Information System (OnSIS) Staging Area Board Validation & Sign-off by Director Ministry Validation Encryption & Depersonalization Elementary Secondary Data Warehouse (ESDW) Statistics Canada EQAO Other Sources Data Data Data Data repository or data warehouse Student Management Systems (SMS) Decision Support System EQAO Stats Can Other Board Administrators Province-Wide Information Data Data Data M i SA G i ARE Info Info Info Ministry of Education
    41. 41. 9. Place quality of assessment over quantity <ul><li>Examples: </li></ul><ul><li>Australia – National standards (underway) </li></ul><ul><li>State Syllabi, State and Local Assessments </li></ul><ul><li>(3 grades plus high school) </li></ul><ul><li>Canada – Provincial standards, syllabi & assessments (2 grades plus high school) </li></ul><ul><li>England – National standards & curriculum School-based assessments + national tasks </li></ul><ul><li>(1 grade plus high school) </li></ul><ul><li>Five secondary examination boards </li></ul><ul><li>Singapore – “Teach Less, Learn More” </li></ul><ul><li>National standards and curriculum </li></ul><ul><li>National exams + school-based tasks </li></ul><ul><li>(1 grade plus high school) </li></ul>
    42. 42. 10. Encompass and combine Assessment of, for and as learning <ul><li>Assessment… </li></ul><ul><li>Of learning - summative assessments to confirm student knowledge, skills, understanding of curriculum, proficiency </li></ul><ul><li>For learning – formative assessments using information throughout the learning process to investigate student learning, inform instruction and provide feedback to support student progress </li></ul><ul><li>As learning – assessment as a process of meta-cognition involving student self-assessment and understanding of own learning processes and knowledge </li></ul>
    43. 43. Hong Kong: Learning to Learn and Use of School-Based Assessments <ul><li>“ The primary rationale for School Based Assessments (SBA) is to enhance the validity of the assessment, by including the assessment of outcomes that cannot be readily assessed within the context of a one-off public examination. SBA can also reduce dependence on the result of public examinations, which may not always provide the most reliable indication of the actual abilities of candidates. Obtaining assessments based on student performance over an extended period of time and developed by those who know the students best – their subject teachers – provides more reliable assessment of each student… SBA is to promote a positive impact on teaching and learning . It can serve to motivate students by engaging them in meaningful activities; and for teachers, it can reinforce curriculum aims and good teaching practices…” </li></ul><ul><li>(Hong Kong Education Examinations Authority, 2009). </li></ul>
    44. 44. <ul><li>“ If you want to appear accountable, test your students. </li></ul><ul><li>If you want to improve schools, teach teachers to assess their students. </li></ul><ul><li>If you want to maximize learning, teach students to assess themselves.” </li></ul>
    45. 45. Assessment Design Principles Effective Assessment Systems Designed around a clear vision of purpose and learning goals: 21 st Century Skills A Teaching and Learning System: Align with and advance common standards, curriculum expectations, learning objectives and instructional strategies Encompass and combine assessment of, for and as learning Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning Embody high expectations to engage, motivate, support and stretch all students to progress and achieve Build and develop professional capacity Assess a range of content, knowledge, skills and performance in authentic, applied and appropriate ways Are part of a clear and flexible learning trajectory across subjects and over education/career pathways Support data- informed decision-making for local and system improvement Place quality of assessment over quantity
    46. 46. Race To The Top Assessment Grants <ul><li>$350 million for state consortia to develop new assessments “that are valid, support and inform instruction, provide accurate information about what students know and can do, and measure student achievement against standards designed to ensure that all students gain the knowledge and skills needed to succeed in college and the workplace. These assessments are intended to play a critical role in educational systems: provide administrators, educators, parents, and students with the data and information needed to continuously improve teaching and learning; and help meet the President’s goal of restoring, by 2020, the nation’s position as the world leader in college graduates” (Federal Register/Vol.75, No.68/ April, 9, 2010) </li></ul><ul><li>2 categories: </li></ul><ul><li>Comprehensive Assessments Systems grants </li></ul><ul><li>High School Course Assessment Programs grants </li></ul>
    47. 47. How Might U.S. Assessment Systems Become Internationally Comparable? <ul><li>Develop a coherent , aligned and integrated system that links curriculum, assessments, instruction, and student opportunities-to-learn </li></ul><ul><li>Create lean, but clear curriculum guidance (frameworks or syllabi) around the new Common Core standards that can guide: </li></ul><ul><ul><li>End-of-year summative assessments built on well-articulated content and performance standards </li></ul></ul><ul><ul><li>During-the-year performance tasks embedded in curriculum and scored as part of the overall assessment system </li></ul></ul><ul><ul><li>Formative assessment supports that provide rich information </li></ul></ul><ul><ul><li>Focused preparation and professional development linked to teachers’ roles in developing and scoring assessments, as well as instruction </li></ul></ul>
    48. 48. SCOPE: Commissioned Papers http://edpolicy.stanford.edu <ul><li>Suzanne Lane, Performance Assessment: The State of the Art. </li></ul><ul><li>Raymond Pecheone and Stuart Kahl, Developing Performance Assessments: Lessons from the United States. </li></ul><ul><li>Brian Stecher, Performance Assessment in an Era of Standards-Based Educational Accountability. </li></ul><ul><li>Jamal Abedi, Performance Assessments for English Language Learners. </li></ul><ul><li>Linda Darling-Hammond, with Laura Wentworth, Benchmarking Learning Systems: Student Performance Assessment in International Context. </li></ul><ul><li>Lawrence Picus, Will Montague, Frank Adamson, and Maggie Owens, A New Conceptual Framework for Analyzing the Costs of Performance Assessment. </li></ul><ul><li>Barry Topol, John Olson, and Edward Roeber, The Cost of New Higher Quality Assessments: A Comprehensive Analysis of the Potential Costs for Future State Assessments. </li></ul><ul><li>FUNDED BY FORD AND NELLIE MAE EDUCATION FOUNDATIONS </li></ul>
    49. 49. Thank You! [email_address] http://edpolicy.stanford.edu
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×