Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Clear expectations and habits of mind: a self-evaluation checklist for student writing - Marines & Zachmeier

272 views

Published on

Presented at LILAC 2017

Published in: Education
  • Be the first to comment

Clear expectations and habits of mind: a self-evaluation checklist for student writing - Marines & Zachmeier

  1. 1. Clear expectations and habits of mind: A self-evaluation checklist for student writing Annette Marines, Research Support Librarian Aaron Zachmeier, Instructional Designer University of California, Santa Cruz LILAC 2017 Swansea University
  2. 2. Project Context ● Campus-wide assessment ○ Active program outcomes assessment ○ Focus: Written communications and information literacy ● Teaching support efforts ○ Liaison Librarian: Collections and Information Literacy Instruction ○ Instructional Designer: Online Education ○ Assessment Assistant Director: Assessment Support ● This Research Team ○ Liaison Librarian ○ Instructional Designer ○ Institutional Research, Assessment, and Policy Studies (IRAPS)
  3. 3. Introduction ● What we were hearing and seeing ○ Students were not citing; were citing incorrectly; were not evaluating sources ○ Assignment handouts avoided guidance ● The intervention ○ Opportunity for Librarian and Instructional Designer to collaborate ○ Developed a checklist to use as a research paper cover sheet ● The research project ○ Teamed up with IRAPS and a Psychology professor ○ Assessed the checklist
  4. 4. Why it’s a problem? ● Expert versus novice ● Information deluge ● Communication ● Time management
  5. 5. Some Literature ● Assigning Inquiry: How Handouts for Research Assignments Guide Today's College Students, Alison J. Head & Michael B. Eisenberg, Project Information Literacy ● They can find it but they don't know what to do with it: describing the use of scholarly literature by undergraduate students, Stephanie Rosenblatt ● Checklist manifesto: How to get things right, Atul Gawande ● Teaching Research Processes, William B. Badke
  6. 6. Hypothesis Students’ proficiencies in citation, attribution, and source evaluation can improve with a checklist
  7. 7. Research Strategy With full cooperation with the course instructor, the checklist intervention was assigned in a senior seminar course. Students were required to turn in the checklist as a cover sheet to the draft of a research paper A rubric addressing learning outcomes of the checklist was used to evaluate the draft research papers of two classes: one using no checklist and one using the checklist
  8. 8. Method Participants ● Upper division, undergraduate seniors ● Psychology majors ● Senior seminar course enrollees
  9. 9. Intervention : The Checklist ● We provided the course instructor with a PDF copy of the checklist ● The instructor: ○ Assigned the checklist along with the draft research paper ○ And made it available in the LMS ○ No additional instruction was provided ● Students handed in the checklist with the paper ● The intervention was handled exclusively by the course instructor
  10. 10. Sample ● No checklist n = 24 ; Checklist n = 27 ● Unit of analysis: Draft Research Paper ● 100% of the research papers from each class were included ○ Started by reading 50% (n=12 and n=14). IRAPS asked us to read all the papers to increase significance ● Classes were taught three year apart by the same instructor
  11. 11. Measurement Instrument : Rubric ● Informed by AACU Value Rubrics and others ● 9 criteria ○ Each checklist item is matched to a rubric criterion, Ex. Evaluation separated into 3 domains to capture ■ Makes rubric more precise in evaluating information literacy and critical thinking ● 4 levels ○ Unacceptable (no signs), Beginning, Developing, Proficient (mastered)
  12. 12. Data Collection ● 2 rubric norming sessions ○ Discussed discrepancies until we came to an agreement ● 2 readers, read through and scored papers ● Readers evaluated: ○ Citation style, formatting, consistency ○ Attribution ○ Evaluation of sources ● Blind review
  13. 13. Results IRAPS ran the data analysis. The questions covered: ● Setting a baseline ● Comparing no checklist and checklist classes ● Areas that remained low ● Effect of checklist on specific populations
  14. 14. Key Decision ● The Developing & Proficient levels were combined ● The counts for Proficient were too low to analyze
  15. 15. In what areas do most students have developing or proficient skills and in what areas do they need improvement? (the baseline) MEDIUM 60-79% C HIGH 80-100% D2, A [ F, D3 ] LOW 0-59 % B1, D1, B2, E A - Source Types B&C - Bibliography D - Attribution E - Evaluation F - Quoted Text
  16. 16. LOW 0-59 % D1*, B2**, E In what areas did the checklist lead to improvement? MEDIUM 60-79% B1***, C HIGH 80-100% D2, A [ F, D3 ] * Significance at p <.05 ** at p <.07 *** Not significant A - Source Types B&C - Bibliography D - Attribution E - Evaluation F - Quoted Text
  17. 17. LOW 0-59 % D1, B2, E In what areas do students still lack skills? MEDIUM 60-79% B1, C A - Source Types B&C - Bibliography D - Attribution E - Evaluation F - Quoted Text
  18. 18. LOW 0-59 % D1 Study 1 - SOC & Bilingual D1 Study 2* - SOC Were there any differences related to background characteristics (ethnicity, language, and transfer status)? MEDIUM 60-79% D2 Study 1 - SOC D1 Study 2* - Bilingual HIGH 80-100% D2 Study 2* - SOC * Significance differences Fisher’s exact test p < .08 Study 1 findings in blue Study 2 findings in purple SOC - Students of color D1 - attribution of facts D2 - attribution of ideas
  19. 19. Results Sample Overview ● Both classes were comparable in terms of ethnicity and language status ○ Bilingual students included those who at the age of 5 spoke English and another language or learned English later in school. ● Class 2 (checklist) had fewer transfer students than Class 1 (no checklist) ● There were more first generation students in Class 2 than in Class 1 ● There were greater number of women in both classes, with fewer men in Class 2
  20. 20. Discussion ● The results support our hypothesis: Students proficiencies can and did improve with the checklist ● Students improved in every area, with significant differences in 3 of these ● Clear guidelines and steps help students
  21. 21. Evaluating Sources MED-HIGH LOW 21-59% BOTTOM LOW 0-20 % E A - Source Types B&C - Bibliography D - Attribution E - Evaluation F - Quoted Text
  22. 22. New text ❏ Have you included an evaluation of your sources? Pay particular attention to the (1) the authoritativeness, perspective, and potential bias of the author; (2) credibility or validity of the findings; (3) relevance and purpose of the source itself. Evaluating Sources Old text ❏ Have you evaluated your sources? Pay particular attention to the (1) the authoritativeness, perspective, and potential bias of the author; (2) credibility or validity of the findings; (3) relevance and purpose of the source itself.
  23. 23. Why it’s a problem, again? Expert versus novice, Information deluge, Communication, Time management Limitations ● The checklist is not a teaching tool ● No evidence yet that repeated use will develop habits of mind Unexpected findings ● Closed gaps seen in ethnic and language populations ○ The checklist helped everyone, but helped these groups more
  24. 24. Extending the Project ● We’re working with the same instructor with their lower-division, introduction to the major course. ● We’re currently reading papers for a Senior Seminar course in the History Department Teaching Support ● We promote the checklist in our teaching support consultations ● Our new rubric experience has opened doors with assessment conversations, where I wouldn’t have brought it up before
  25. 25. Photo Credits ● The University of California Santa Cruz, Doc Searls, (CC BY-SA 2.0), https://flic.kr/p/jmLrq ● Understand context, Paul Downey, (CC BY 2.0), https://flic.kr/p/eTB94c ● Ginger man... NAILED IT!, Sharon Vicino, (CC BY-SA 2.0), https://flic.kr/p/quA8aW ● Rubik's Cubes, CC0 Public Domain, http://maxpixel.freegreatpicture.com/photo-1243091 ● College Students, CollegeDegrees360, (CC BY-SA 2.0), https://flic.kr/p/cEJAgJ ● cmyk color management linen tester, Markus Spiske, (CC0 1.0), https://flic.kr/p/RS7kbT ● Crunches, Everkinetic (http://everkinetic.com/), (CC BY-SA 3.0), https://commons.wikimedia.org/wiki/File%3ADecline-crunch-2.png & https://commons.wikimedia.org/wiki/File%3ADecline-crunch-2.png ● Crystal Wynwood Mural, Elvert Barnes, (CC BY-SA 2.0), https://flic.kr/p/pic2z7 ● Hooray Penguin, David Goehring, (CC BY 2.0), https://flic.kr/p/JZ4Dmn ● Students in Class, Tulane Public Relations, (CC BY 2.0), https://commons.wikimedia.org/wiki/File%3AStudent_in_Class_(3618969705).jpg ● The Questioning Roboto, Matthew Hutchinson, (CC BY 2.0), https://flic.kr/p/7VhPft
  26. 26. Annette Marines amarines@ucsc.edu Aaron Zachmeier azachmei@ucsc.edu More information (checklists) http://guides.library.ucsc.edu/checklists

×