The document discusses the importance of learning outcomes assessment in libraries, outlining processes for developing and assessing instructional activities. It emphasizes the need for clear, measurable outcomes that focus on what students should achieve, and provides guidelines for creating effective learning outcomes. Additionally, it offers various assessment methods and highlights the role of assessment in improving instructional efficacy and library contributions to student learning.

1. Assessing for Improvement Diane Harvey I&O Brown Bag November 11, 2009

2. Learning outcomes for today At the end of this session, participants will be able to: 1. Describe the learning outcomes assessment process in libraries. 2. Write learning outcomes for an information literacy instruction session. 3. Choose appropriate assessment methods. 4. Define ways to work with assessment results.

3. Assessment climate in higher education Spellings Commission http:// www.ed.gov/about/bdscomm/list/hiedfuture/index.html Voluntary System of Accountability (VSA) http:// www.voluntarysystem.org /

4. Accrediting Agencies http://www.sacs.org/ The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: 3.3.1.1 educational programs, to include student learning outcomes

5. Assessment at Duke SACS re-accreditation Trinity Office of Assessment Libraries Strategic Plan “ Implement a student learning outcomes assessment program for the Libraries’instructional activities.” (4.2)

6. Assessing for improvement We don’t assess to prove… … but to improve.

7. Assessment in libraries Focus on collections and services (e.g. ARL LibQUAL) Focus on standards (e.g. ACRL Info Lit Competency Standards)

8. What is LOA: learning outcomes assessment? Systematic look at what students are learning Moving from “What am I going to teach today?” to “What do I want students to learn today?”

9. What LOA is NOT NOT evaluation of teaching NOT evaluation of program

10. Why assess student learning in libraries? Affirm commitment to student learning Improve effectiveness of instructional programs Align with campus, disciplinary, and higher ed efforts Be recognized on campus for contributions to student learning

11. Assessment loop 2.Set criteria 3.Devise assessment measures 1.Develop student learning outcomes 4.Do assessment 5. Work with results

12. Levels of assessment Institutional (University wide) Program (e.g. Department) Course Session (e.g. IL instruction)

13. LOA in libraries: constraints Lack of consistent, reliable access to students IL instruction: is it a “program”? Program assessment vs. session assessment Difficult to isolate effects of library instruction

14. Mapping an IL program First Year Writing and Freshman Seminars Subject specific instruction Honors capstone seminars Research methods/R &W

15. Where can student learning be assessed? On the instruction session level? Across a series of library instruction sessions? Broadly across the student population?

16. It’s important to remember that… …library instruction doesn’t work the same way as instruction in an academic department. …we need to be able to explain how and why we assess student learning.

17. Some simple approaches Develop learning outcomes across instruction program, and look at where they are taught – “snapshots” Focus on large programmatic efforts e.g. first year writing Highlight specific academic departments Assess individual library instruction sessions – develop shared learning outcomes, compare/contrast

18. What is a learning outcome? A learning outcome is one sentence that indicates what students should represent, demonstrate or produce as a result of what they learn. - source: Peggy Maki

19. Good learning outcomes: Focus on what students will learn/know/be able to do Describe actions or behaviors Are results oriented Are observable and measurable Include a time frame

20. ACRL Standards 3 levels: standards, performance indicators, outcomes Can be used for LOA Examples of ACRL outcomes: (1)Identifies keywords, synonyms and related terms for the information needed [Standard Two] (2) Selects an appropriate documentation style and uses it consistently to cite sources [Standard Five]

21. Writing outcomes Today, we’ll write learning outcomes for a library instruction session.

22. Exercise: What do you cover? Think of an information literacy instruction session that you have taught or are going to teach. List what you would cover in that session.

23. Bloom’s Taxonomy Classification of educational objectives Published in 1956, revised in 2001 (changes: noun to verb, synthesis/creating becomes highest level) Taxonomy = classification Cognitive levels (lower  higher) Provides way to express outcomes

24. Bloom’s levels (lowest  highest) Knowledge/Remembering Comprehension/Understanding Application/Applying Analysis/Analyzing Evaluation/Evaluating Synthesis/Creating

25. Verbs for information literacy Knowledge/Remembering : define, list, recognize Comprehension/Understanding : characterize, describe, explain, identify, locate, recognize, sort Application/Applying : choose, demonstrate, implement, perform Analysis/Analyzing : analyze, categorize, compare, differentiate Evaluation/Evaluating : assess, critique, evaluate, rank, rate Synthesis/Creating : construct, design, formulate, organize, synthesize

26. Verbs to avoid Understand Appreciate Know about Become familiar with Learn about Become aware of

27. Learning outcomes formula 1. Time frame 2. Student focus 3. Action verb 4. Product/process/outcome

28. Using the formula Time frame: “After attending this instruction session…” Student focus: “…students will be able to…” Action verb: “…identify…” Product/process: “…a relevant database for their term paper research.”

29. Another example: Time frame: “After completing the online tutorial…” Student focus: “…students will be able to…” Action verb: “…differentiate between…” Product/process: “…scholarly journals and popular magazines.”

30. One more example: Time frame: “By the time they complete their undergraduate program…” Student focus: “…students will be able to…” Action verb: “…construct…” Product/process: “…a research question that can be investigated using primary archival resources.”

31. Examples: bad and better Bad: Students will understand how to use social science databases. Better: Students will perform a search in Social Sciences Abstracts that retrieves relevant items. Bad: Students will be able to search the catalog. Better: Students will construct a catalog search to identify a relevant book on their topic. Bad: Students will appreciate the importance of correct citation. Better: Students will produce citations in correct APA format.

32. Now, write your own! Choose two items from the list of what you cover in an information literacy session. Transform them into learning outcomes, using the formula. Find a partner and critique your outcomes using the following checklist.

33. Checklist for learning outcomes Includes a time frame? Focuses on students? Uses action verbs? Names a product or process? Is measurable/observable? Prompts a measure/method? Will be useful for you to assess?

34. Sample size Use any convenient sample that will give you useable information. Make sure that groups of interest are represented. Sample size is important only if you plan to publish your results. Do you want to generalize to the entire student population?

35. Direct and Indirect Measures Direct: Students demonstrate an expected learning outcome Indirect: Students report their perception of how well a given learning outcome has been achieved Direct is always preferable.

36. Easy assessment methods Knowledge test One Minute paper & variations Bibliography analysis Concept Inventory Standardized test

37. Knowledge tests Tests knowledge/skills before and/or after library instruction session. Can be given at end of library session or later in semester. Can use clickers to gather data during library session. Sample questions : 1. What is the difference between a library catalog and a database? 2. The Boolean operator “or” narrows a search statement (true/false).

38. One minute paper & variations Sample questions: 1. What is the most important thing about library research you learned today? 2. 3-2-1 (three things you learned, two things you’re still confused about, one thing you’d change about session) 3. What is one question you still have? 4. In your research, what will you do differently after today’s session?

39. Using Blackboard Tests: are graded, can look at individual results Surveys: not graded, results are aggregated Assignments: allow students to upload documents

40. Using Viewsflash Create a survey/questionnaire Students can login with net id or the survey can be open Results can be emailed to you, and/or collected and exported to Excel

41. Bibliography analysis Look for citations from scholarly/peer reviewed journals Look for citations for books and journals owned by Duke libraries Look for articles retrieved from Duke databases

42. Concept inventory Make a checklist of 3-12 important concepts students need to master. Ask students to explain each concept in a sentence or two. If a concept is unfamiliar, they should leave blank. Count good responses for each concept, then plan future instruction.

43. Standardized tests I-Skills ILT (James Madison University) SAILS

44. Using assessment information Rewrite learning outcomes Change what you do/how you teach Work collaboratively with colleagues (librarians and faculty) Revise assessment measures Share the news in the library and on campus

45. Learning outcomes are: Fundamental components of a learning assessment program Focused on student learning Indicate what students will know/be able to do Measurable, observable, overt

46. Learning Outcomes Assessment A tool to help librarians improve student learning

47. LOA resources Angelo, Thomas (1993). Classroom assessment techniques. Maki, Peggy (2004). Assessing for learning : building a sustainable commitment across the institution. Neely, Teresa Y(2006). Information literacy assessment : standards-based tools and assignments. Radcliff, Carolyn et.al (2007). A practical guide to information literacy assessment for academic librarians. Student Learning Outcomes Assessment (University of Virginia) http://www.web.virginia.edu/iaas/assessment/outcomes.htm