MILEXAssessmentRubrics2014

411 views

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
411
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • This is to cue raters to the process.
  • Panel 2
  • MILEXAssessmentRubrics2014

    1. 1. Welcome! MILEX: March 2014 TU’s Excellent Rubric Assessment Adventures: On the RAILS Shana Gass Claire Holmes Lisa Sweeney sgass@towson.edu cholmes@towson.edu sweeney@towson.edu
    2. 2. Agenda for Today : • Background on Assessment, RAILS & Rubrics • Norming & Rating Sessions • Working Lunch: Create Draft Rubrics • Reflections & Questions
    3. 3. Assessment… • Knowing what you are doing • Knowing why you are doing it • Knowing what students are learning as a result • Changing because of the information (~Debra Gilchrist, Dean of Libraries and Institutional Effectiveness, Pierce College)
    4. 4. Identify learning outcomes Create and enact learning activities Gather data to check for learning Interpret data Enact decisions to increase learning Information Literacy Instruction Assessment Cycle (ILIAC) Oakleaf, Megan. "The Information Literacy Instruction Assessment Cycle: A Guide for Increasing Student Learning and Improving Librarian Instructional Skills." Journal of Documentation. 65.4. 2009.
    5. 5. The Institute of Museum and Library Services is the primary source of federal support for the nation’s 123,000 libraries and 17,500 museums. The Institute's mission is to create strong libraries and museums that connect people to information and ideas. Megan Oakleaf, founder of all things RAILS.
    6. 6. RAILS Project Purpose • Investigated an analytic rubric approach to IL assessment in higher education • Developed a suite of IL rubrics • Investigated rubric reliability & validity • Developed training materials for training/ norming/ scoring • Explored indicators of rater expertise
    7. 7. Cook’s RAILS Purpose • Gain rubric experience: creating/norming/rating • Identify assessment opportunities within TU’s Core Curriculum • Develop a rubric for use on campus • Assess students’ information literacy skills • Examine instructional practices
    8. 8. Cook Library’s Priorities:
    9. 9. • Begin cycle of tracking student learning. • Begin cycle of tracking instruction practices. • Begin cycle of collecting aggregated & anonymous data. • Reinforce regular opportunities for reflection & discussion among library instruction colleagues. (facilitate development of a Community of Reflective Practice) (Image: AP Images) Our assessment adventure…
    10. 10. Understanding by Design 1. What do you want students to learn? (outcome) 2. How will you know that they have learned it? (assessment) 3. What activities will help them learn, and at the same time, provide assessment data? (teaching method & assessment) (Wiggins & McTighe, 2006)
    11. 11. Performance/Integrated Assessment Students reveal their learning when they are provided with: complex, authentic LEARNING ACTIVITIES to explain, interpret, apply, shift perspective, empathize and self-assess. What we assess.What they learn. (Megan Oakleaf, Assessment: Demonstrating the Educational Value of the Academic Library, ACRL Assessment Immersion, 2011.)
    12. 12. 5 Questions for Assessment Design: 1. Outcome What do you want the student to be able to do? 2. IL Curriculum What does the student need to know in order to do this well? 3. Pedagogy What type of instruction will best enable the learning? 4. Assessment How will the student demonstrate the learning? 5. Criteria for evaluation How will you know the student has done well? (Lisa Hinchcliffe, Student Learning Assessment Cycle. ACRL Assessment Immersion, 2011)
    13. 13. Evidence of “authentic” student learning: For instance, the research worksheet in your packet that asks students to break down and practice sequential steps in the search process. Brainstorm… What other possible examples of evidence of student learning do we collect? What could we collect?
    14. 14. Brainstorm ideas…
    15. 15. Evidence: Possible examples of authentic student learning… • Research journal • Reflective writing • “think aloud” • Self or peer evaluation • Works cited page • Annotated bibliography • Posters • Multimedia presentations • Speeches • Open-ended question responses • Group projects • Performances • Portfolios • Library assignments • Worksheets • Concept maps • Citation maps • Tutorial responses • Blogs • Wikis • Lab reports
    16. 16. • 2 dimensions 1. criteria 2. levels of performance • grid or table format • judges quality • translates unwieldy data into accessible information (Image: thefirstgradediaries.blogspot.com)
    17. 17. Criteria 1.“the conditions a [student] must meet to be successful” (Wiggins) 2.“the set of indicators, markers, guides, or a list of measures or qualities that will help [a scorer] know when a [student] has met an outcome” (Bresciani, Zelna & Anderson) 3.what to look for in [student] performance “to determine progress…or determine when mastery has occurred” (Arter)
    18. 18. Performance Levels mastery, progressing, emerging, satisfactory, marginal, proficient, high, middle, beginning, advanced, novice, intermediate, sophisticate d, competent, professional, exemplary, need s work, adequate, developing, accomplished, d istinguished (or numerical…)
    19. 19. SAMPLE RAILS RUBRIC (green handout in your packet) Performance Level 3 Student: Performance Level 2 Student: Performance Level 1 Student: Performance Level 0 Student: 1. Determines Key Concepts Determines multiple key concepts that reflect the research topic/thesis statement accurately. Determines some concepts that reflect the research topic/thesis statement, but concept breakdown is incomplete or repetitive. Determines concepts that reflect the research topic/thesis statement inaccurately. Does not determine any concepts that reflect the research question/thesis statement. 2. Identifies synonyms and related terms Identifies relevant synonyms and/or related terms that match key concepts. Attempts synonym (or related term) use, but synonym list is incomplete or not fully relevant to key concepts. Identifies synonyms that inaccurately reflect the key concepts. Does not identify synonyms. 3. Constructs a search strategy using relevant operators Constructs a search strategy using an appropriate combination of relevant operators (for example: and, or, not) correctly. Constructs a search strategy using operator(s), but uses operators in an incomplete or limited way. Constructs a search strategy using operators incorrectly. Does not use operators. 4. Uses evaluative criteria to select source(s) Uses evaluative criteria to provide in- depth explanation of rationale for source selected. Uses evaluative criteria to provide a limited/superficial explanation of rationale for source selected. Attempts to use evaluative criteria, but does so inaccurately or incorrectly. Does not use evaluative criteria. 5. Uses Citations Uses an appropriate standard citation style consistently and correctly. Uses an appropriate standard citation style consistently (bibliographic elements intact), but with minimal format and/or punctuation errors. Attempts an appropriate standard citation style, but does not include all bibliographic elements consistently or correctly. Does not include common citation elements or does not include citations.
    20. 20. Workshop Norming Practice Round 1 • For first student work sample, Claire will “norm aloud.” • Participants will rate 2 work samples individually. • Group discussion: Can we reach consensus for what constitutes evidence for each performance level?
    21. 21. Norming: Round 2 • Participants will rate 2 more work samples individually. • Group discussion: Are we closer to consensus? • Do we establish rating ground rules? • Does the rubric need to be modified?
    22. 22. Keep in mind… • An info lit skills rubric does not score discipline content; it scores information literacy skills. • You can only score what you can see.
    23. 23. Norming/Rating Discussion • How do we achieve consensus? • What was challenging?
    24. 24. Rubrics – Benefits Learning • Articulate and communicate agreed upon learning goals • Provide direct feedback to learners • Facilitate self-evaluation • Focus on learning standards
    25. 25. 1. What are our expectations of students completing this assignment? 2. What does a successful learning of this type look like? 3. What specific learning outcomes do we want to see in the completed assignment? 4. What evidence can we find that will demonstrate learning success? Creating a rubric:
    26. 26. More benefits of a (normed) rubric… Data • Facilitate consistent, accurate, unbiased scoring • Deliver data that is easy to understand, defend, and convey • Offer detailed descriptions necessary for informed decision-making • Can be used over time or across multiple programs Other • Are inexpensive ($) to design & implement
    27. 27. Rubrics – Limitations • Possible design flaws that impact data quality • Require significant time for development • Sometimes fail to balance between holistic and analytic focus • May fail to balance between generalized wording and detailed description • Can lack differentiation between performance levels
    28. 28. RAILS Lessons • Explicit, detailed performance descriptions are crucial to achieve inter-rater reliability. • Raters appear to be more confident about their ratings when student artifacts under analysis are concrete, focused, and shorter in length. • The best raters “believe in” outcomes, value constructed consensus (or “disagree and commit”), negotiate meaning across disciplines, develop shared vocabulary, etc.
    29. 29. Identify learning outcomes Create and enact learning activities Gather data to check for learning Interpret data Enact decisions to increase learning Information Literacy Instruction Assessment Cycle (ILIAC) Oakleaf, Megan. "The Information Literacy Instruction Assessment Cycle: A Guide for Increasing Student Learning and Improving Librarian Instructional Skills." Journal of Documentation. 65.4. 2009.
    30. 30. • Internally – Instruction improvement – Assessment improvement • Professionally – Conferences – Publications Using Assessment Results…
    31. 31. References Arter, J. (2000). Rubrics, scoring guides, and performance criteria: Classroom tools for assessing and improving student learning. Retrieved from http://eric.ed.gov/?id=ED446100 Bresciani, M., Zelna, C. & Anderson, J. (2004). Assessing student learning and development: A handbook for practitioners. Washington, DC: NASPA-Student Affairs Administrators in Higher Education. Wiggins, G. P., & McTighe, J. (2006). Understanding by design. Upper Saddle River, NJ: Pearson Education, Inc., 2006. Wiggins, G. P. (1998). Educative assessment: Designing assessments to inform and improve student performance. San Francisco, CA: Jossey-Bass.
    32. 32. Selected Readings: Diller, K. R., & Phelps, S. F. (2008). Learning outcomes, portfolios, and rubrics, oh my! Authentic assessment of an information literacy program. Portal: Libraries and the Academy, 8 (1), 75-89. Fagerheim, B. A., & Shrode, F. G. (2009). Information literacy rubrics within the disciplines. Communications in Information Literacy, 3(2), 158-170. Holmes, C. & Oakleaf, M. (2013). The Official (and Unofficial) Rules for Norming Rubrics Successfully. Journal of Academic Librarianship, 39(6), 599-602. Knight, L. A. (2006). Using rubrics to assess information literacy. Reference Services Review, 34(1), 43-55. Oakleaf, M. (2007). Using rubrics to collect evidence for decision-making: What do librarians need to learn? Evidence Based Library and Information Practice, 2(3), 27-42. Oakleaf, M. (2009). The information literacy instruction assessment cycle: A guide for increasing student learning and improving librarian instructional skills. Journal of Documentation, 65(4), 539-560. Oakleaf, M., Millet, M., & Kraus, L. (2011). All together now: getting faculty, administrators, and staff engaged in information literacy assessment. Portal: Libraries and the Academy, 11(3), 831- 852. Stevens, D. D., & Levi, A. (2005). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Sterling, VA: Stylus Publishing.
    33. 33. MILEX: March 2014 TU’s Excellent Rubric Assessment Adventures: On the RAILS Shana Gass Claire Holmes Lisa Sweeney sgass@towson.edu cholmes@towson.edu sweeney@towson.edu SlideShare URL: http://www.slideshare.net/claireholmes/milex-assess-norm2014 Thank you!

    ×