Assessing for Improvement Diane Harvey I&O Brown Bag November 11, 2009
Learning outcomes for today <ul><li>At the end of this session, participants will  </li></ul><ul><li>be able to: </li></ul...
Assessment climate in  higher  education <ul><li>Spellings Commission  http:// www.ed.gov/about/bdscomm/list/hiedfuture/in...
Accrediting Agencies <ul><li>http://www.sacs.org/ </li></ul><ul><li>The institution identifies expected outcomes, assesses...
Assessment at Duke <ul><li>SACS re-accreditation </li></ul><ul><li>Trinity Office of Assessment </li></ul><ul><li>Librarie...
Assessing for improvement We don’t assess to prove… … but to improve.
Assessment in libraries <ul><li>Focus on collections and services (e.g. ARL LibQUAL) </li></ul><ul><li>Focus on standards ...
What is LOA: learning outcomes assessment? <ul><li>Systematic look at what students are learning  </li></ul><ul><li>Moving...
What LOA is NOT <ul><li>NOT evaluation of teaching </li></ul><ul><li>NOT evaluation of program  </li></ul>
Why assess student learning in libraries? <ul><li>Affirm commitment to student learning </li></ul><ul><li>Improve effectiv...
Assessment loop 2.Set criteria 3.Devise assessment  measures 1.Develop student  learning outcomes 4.Do assessment 5. Work ...
Levels of assessment <ul><li>Institutional (University wide) </li></ul><ul><li>Program (e.g. Department) </li></ul><ul><li...
LOA in libraries: constraints <ul><li>Lack of consistent, reliable access to students  </li></ul><ul><li>IL instruction: i...
Mapping an IL program First Year Writing and Freshman Seminars Subject specific instruction Honors capstone seminars Resea...
Where can student learning  be assessed? <ul><li>On the instruction session level? </li></ul><ul><li>Across a series of li...
It’s important to remember that… <ul><li>…library instruction doesn’t work the same way as instruction in an academic depa...
Some simple approaches <ul><li>Develop learning outcomes across instruction program, and look at where they are taught – “...
What is a learning outcome? <ul><li>A learning outcome is one sentence that indicates what students should represent, demo...
Good learning outcomes: <ul><li>Focus on what students will learn/know/be able to do </li></ul><ul><li>Describe actions or...
ACRL Standards  <ul><li>3 levels: standards, performance indicators, outcomes </li></ul><ul><li>Can be used for LOA  </li>...
Writing outcomes <ul><li>Today, we’ll write learning outcomes for a library instruction session. </li></ul>
Exercise: What do you cover? <ul><li>Think of an information literacy instruction session that you have taught or are goin...
Bloom’s Taxonomy <ul><li>Classification of educational objectives </li></ul><ul><li>Published in 1956, revised in 2001  (c...
Bloom’s levels  (lowest   highest) <ul><li>Knowledge/Remembering </li></ul><ul><li>Comprehension/Understanding </li></ul>...
Verbs for information literacy <ul><li>Knowledge/Remembering : define, list, recognize </li></ul><ul><li>Comprehension/Und...
Verbs to avoid <ul><li>Understand </li></ul><ul><li>Appreciate </li></ul><ul><li>Know about </li></ul><ul><li>Become famil...
Learning outcomes formula <ul><li>1. Time frame </li></ul><ul><li>2. Student focus </li></ul><ul><li>3. Action verb </li><...
Using the formula <ul><li>Time frame:  “After attending this instruction session…” </li></ul><ul><li>Student focus: “…stud...
Another example: <ul><li>Time frame:  “After completing the online tutorial…” </li></ul><ul><li>Student focus: “…students ...
One more example: <ul><li>Time frame:  “By the time they complete their undergraduate program…” </li></ul><ul><li>Student ...
Examples: bad and better <ul><li>Bad:  Students will  understand  how to use social science databases. </li></ul><ul><li>B...
Now, write your own! <ul><li>Choose two items from the list of what you cover in an information literacy session. </li></u...
Checklist for learning outcomes <ul><li>Includes a time frame? </li></ul><ul><li>Focuses on students? </li></ul><ul><li>Us...
Sample size <ul><li>Use any convenient sample that will give you useable information. </li></ul><ul><li>Make sure that gro...
Direct and Indirect Measures <ul><li>Direct: Students demonstrate an expected learning outcome  </li></ul><ul><li>Indirect...
Easy assessment methods <ul><li>Knowledge test </li></ul><ul><li>One Minute paper & variations </li></ul><ul><li>Bibliogra...
Knowledge tests <ul><li>Tests knowledge/skills before and/or after library instruction session. </li></ul><ul><li>Can be g...
One minute paper & variations <ul><li>Sample questions: </li></ul><ul><li>1.  What is the most important thing about libra...
Using Blackboard <ul><li>Tests:  are graded, can look at individual results </li></ul><ul><li>Surveys: not graded, results...
Using Viewsflash <ul><li>Create a survey/questionnaire </li></ul><ul><li>Students can login with net id or the survey can ...
Bibliography analysis <ul><li>Look for citations from scholarly/peer reviewed journals </li></ul><ul><li>Look for citation...
Concept inventory <ul><li>Make a checklist of 3-12 important concepts students need to master. </li></ul><ul><li>Ask stude...
Standardized tests <ul><li>I-Skills </li></ul><ul><li>ILT (James Madison University) </li></ul><ul><li>SAILS </li></ul>
Using assessment information <ul><li>Rewrite learning outcomes </li></ul><ul><li>Change what you do/how you teach </li></u...
Learning outcomes are: <ul><li>Fundamental components of a learning assessment program </li></ul><ul><li>Focused on studen...
Learning Outcomes Assessment  <ul><li>A tool to help librarians  </li></ul><ul><li>improve student learning </li></ul>
LOA resources <ul><li>Angelo, Thomas (1993). Classroom assessment techniques. </li></ul><ul><li>Maki, Peggy (2004). Assess...
Upcoming SlideShare
Loading in …5
×

Assessing for Improvement: learning outcomes assessment for library instruction

3,916 views
3,722 views

Published on

Presentation on learning outcomes assessment for library instruction; Duke University Libraries, November 2009.

Published in: Education, Technology
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,916
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide

Assessing for Improvement: learning outcomes assessment for library instruction

  1. 1. Assessing for Improvement Diane Harvey I&O Brown Bag November 11, 2009
  2. 2. Learning outcomes for today <ul><li>At the end of this session, participants will </li></ul><ul><li>be able to: </li></ul><ul><li>1. Describe the learning outcomes assessment process in libraries. </li></ul><ul><li>2. Write learning outcomes for an information literacy instruction session. </li></ul><ul><li>3. Choose appropriate assessment methods. </li></ul><ul><li>4. Define ways to work with assessment results. </li></ul>
  3. 3. Assessment climate in higher education <ul><li>Spellings Commission http:// www.ed.gov/about/bdscomm/list/hiedfuture/index.html </li></ul><ul><li>Voluntary System of Accountability (VSA) http:// www.voluntarysystem.org / </li></ul>
  4. 4. Accrediting Agencies <ul><li>http://www.sacs.org/ </li></ul><ul><li>The institution identifies expected outcomes, assesses the extent to </li></ul><ul><li>which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: </li></ul><ul><li>3.3.1.1 educational programs, to include student learning outcomes </li></ul>
  5. 5. Assessment at Duke <ul><li>SACS re-accreditation </li></ul><ul><li>Trinity Office of Assessment </li></ul><ul><li>Libraries Strategic Plan </li></ul><ul><li>“ Implement a student learning outcomes assessment program for the Libraries’instructional activities.” (4.2) </li></ul>
  6. 6. Assessing for improvement We don’t assess to prove… … but to improve.
  7. 7. Assessment in libraries <ul><li>Focus on collections and services (e.g. ARL LibQUAL) </li></ul><ul><li>Focus on standards (e.g. ACRL Info Lit Competency Standards) </li></ul>
  8. 8. What is LOA: learning outcomes assessment? <ul><li>Systematic look at what students are learning </li></ul><ul><li>Moving from “What am I going to teach today?” to “What do I want students to learn today?” </li></ul>
  9. 9. What LOA is NOT <ul><li>NOT evaluation of teaching </li></ul><ul><li>NOT evaluation of program </li></ul>
  10. 10. Why assess student learning in libraries? <ul><li>Affirm commitment to student learning </li></ul><ul><li>Improve effectiveness of instructional programs </li></ul><ul><li>Align with campus, disciplinary, and higher ed efforts </li></ul><ul><li>Be recognized on campus for contributions to student learning </li></ul>
  11. 11. Assessment loop 2.Set criteria 3.Devise assessment measures 1.Develop student learning outcomes 4.Do assessment 5. Work with results
  12. 12. Levels of assessment <ul><li>Institutional (University wide) </li></ul><ul><li>Program (e.g. Department) </li></ul><ul><li>Course </li></ul><ul><li>Session (e.g. IL instruction) </li></ul>
  13. 13. LOA in libraries: constraints <ul><li>Lack of consistent, reliable access to students </li></ul><ul><li>IL instruction: is it a “program”? </li></ul><ul><li>Program assessment vs. session assessment </li></ul><ul><li>Difficult to isolate effects of library instruction </li></ul>
  14. 14. Mapping an IL program First Year Writing and Freshman Seminars Subject specific instruction Honors capstone seminars Research methods/R &W
  15. 15. Where can student learning be assessed? <ul><li>On the instruction session level? </li></ul><ul><li>Across a series of library instruction sessions? </li></ul><ul><li>Broadly across the student population? </li></ul>
  16. 16. It’s important to remember that… <ul><li>…library instruction doesn’t work the same way as instruction in an academic department. </li></ul><ul><li>…we need to be able to explain how and why we assess student learning. </li></ul>
  17. 17. Some simple approaches <ul><li>Develop learning outcomes across instruction program, and look at where they are taught – “snapshots” </li></ul><ul><li>Focus on large programmatic efforts e.g. first year writing </li></ul><ul><li>Highlight specific academic departments </li></ul><ul><li>Assess individual library instruction sessions – develop shared learning outcomes, compare/contrast </li></ul>
  18. 18. What is a learning outcome? <ul><li>A learning outcome is one sentence that indicates what students should represent, demonstrate or produce as a result of what they learn. </li></ul><ul><li>- source: Peggy Maki </li></ul>
  19. 19. Good learning outcomes: <ul><li>Focus on what students will learn/know/be able to do </li></ul><ul><li>Describe actions or behaviors </li></ul><ul><li>Are results oriented </li></ul><ul><li>Are observable and measurable </li></ul><ul><li>Include a time frame </li></ul>
  20. 20. ACRL Standards <ul><li>3 levels: standards, performance indicators, outcomes </li></ul><ul><li>Can be used for LOA </li></ul><ul><li>Examples of ACRL outcomes: (1)Identifies keywords, synonyms and related terms for the information needed [Standard Two] (2) Selects an appropriate documentation style and uses it consistently to cite sources [Standard Five] </li></ul>
  21. 21. Writing outcomes <ul><li>Today, we’ll write learning outcomes for a library instruction session. </li></ul>
  22. 22. Exercise: What do you cover? <ul><li>Think of an information literacy instruction session that you have taught or are going to teach. </li></ul><ul><li>List what you would cover in that session. </li></ul>
  23. 23. Bloom’s Taxonomy <ul><li>Classification of educational objectives </li></ul><ul><li>Published in 1956, revised in 2001 (changes: noun to verb, synthesis/creating becomes highest level) </li></ul><ul><li>Taxonomy = classification </li></ul><ul><li>Cognitive levels (lower  higher) </li></ul><ul><li>Provides way to express outcomes </li></ul>
  24. 24. Bloom’s levels (lowest  highest) <ul><li>Knowledge/Remembering </li></ul><ul><li>Comprehension/Understanding </li></ul><ul><li>Application/Applying </li></ul><ul><li>Analysis/Analyzing </li></ul><ul><li>Evaluation/Evaluating </li></ul><ul><li>Synthesis/Creating </li></ul>
  25. 25. Verbs for information literacy <ul><li>Knowledge/Remembering : define, list, recognize </li></ul><ul><li>Comprehension/Understanding : characterize, describe, explain, identify, locate, recognize, sort </li></ul><ul><li>Application/Applying : choose, demonstrate, implement, perform </li></ul><ul><li>Analysis/Analyzing : analyze, categorize, compare, differentiate </li></ul><ul><li>Evaluation/Evaluating : assess, critique, evaluate, rank, rate </li></ul><ul><li>Synthesis/Creating : construct, design, formulate, organize, synthesize </li></ul>
  26. 26. Verbs to avoid <ul><li>Understand </li></ul><ul><li>Appreciate </li></ul><ul><li>Know about </li></ul><ul><li>Become familiar with </li></ul><ul><li>Learn about </li></ul><ul><li>Become aware of </li></ul>
  27. 27. Learning outcomes formula <ul><li>1. Time frame </li></ul><ul><li>2. Student focus </li></ul><ul><li>3. Action verb </li></ul><ul><li>4. Product/process/outcome </li></ul>
  28. 28. Using the formula <ul><li>Time frame: “After attending this instruction session…” </li></ul><ul><li>Student focus: “…students will be able to…” </li></ul><ul><li>Action verb: “…identify…” </li></ul><ul><li>Product/process: “…a relevant database for their term paper research.” </li></ul>
  29. 29. Another example: <ul><li>Time frame: “After completing the online tutorial…” </li></ul><ul><li>Student focus: “…students will be able to…” </li></ul><ul><li>Action verb: “…differentiate between…” </li></ul><ul><li>Product/process: “…scholarly journals and popular magazines.” </li></ul>
  30. 30. One more example: <ul><li>Time frame: “By the time they complete their undergraduate program…” </li></ul><ul><li>Student focus: “…students will be able to…” </li></ul><ul><li>Action verb: “…construct…” </li></ul><ul><li>Product/process: “…a research question that can be investigated using primary archival resources.” </li></ul>
  31. 31. Examples: bad and better <ul><li>Bad: Students will understand how to use social science databases. </li></ul><ul><li>Better: Students will perform a search in Social Sciences Abstracts that retrieves relevant items. </li></ul><ul><li>Bad: Students will be able to search the catalog. </li></ul><ul><li>Better: Students will construct a catalog search to identify a relevant book on their topic. </li></ul><ul><li>Bad: Students will appreciate the importance of correct citation. </li></ul><ul><li>Better: Students will produce citations in correct APA format. </li></ul>
  32. 32. Now, write your own! <ul><li>Choose two items from the list of what you cover in an information literacy session. </li></ul><ul><li>Transform them into learning outcomes, using the formula. </li></ul><ul><li>Find a partner and critique your outcomes using the following checklist. </li></ul>
  33. 33. Checklist for learning outcomes <ul><li>Includes a time frame? </li></ul><ul><li>Focuses on students? </li></ul><ul><li>Uses action verbs? </li></ul><ul><li>Names a product or process? </li></ul><ul><li>Is measurable/observable? </li></ul><ul><li>Prompts a measure/method? </li></ul><ul><li>Will be useful for you to assess? </li></ul>
  34. 34. Sample size <ul><li>Use any convenient sample that will give you useable information. </li></ul><ul><li>Make sure that groups of interest are represented. </li></ul><ul><li>Sample size is important only if you plan to publish your results. </li></ul><ul><li>Do you want to generalize to the entire student population? </li></ul>
  35. 35. Direct and Indirect Measures <ul><li>Direct: Students demonstrate an expected learning outcome </li></ul><ul><li>Indirect: Students report their perception of how well a given learning outcome has been achieved </li></ul><ul><li>Direct is always preferable. </li></ul>
  36. 36. Easy assessment methods <ul><li>Knowledge test </li></ul><ul><li>One Minute paper & variations </li></ul><ul><li>Bibliography analysis </li></ul><ul><li>Concept Inventory </li></ul><ul><li>Standardized test </li></ul>
  37. 37. Knowledge tests <ul><li>Tests knowledge/skills before and/or after library instruction session. </li></ul><ul><li>Can be given at end of library session or later in semester. </li></ul><ul><li>Can use clickers to gather data during library session. </li></ul><ul><li>Sample questions : </li></ul><ul><li>1. What is the difference between a library catalog and a database? </li></ul><ul><li>2. The Boolean operator “or” narrows a search </li></ul><ul><li>statement (true/false). </li></ul>
  38. 38. One minute paper & variations <ul><li>Sample questions: </li></ul><ul><li>1. What is the most important thing about library research you learned today? </li></ul><ul><li>2. 3-2-1 (three things you learned, two things you’re still confused about, one thing you’d change about session) </li></ul><ul><li>3. What is one question you still have? </li></ul><ul><li>4. In your research, what will you do differently after today’s session? </li></ul>
  39. 39. Using Blackboard <ul><li>Tests: are graded, can look at individual results </li></ul><ul><li>Surveys: not graded, results are aggregated </li></ul><ul><li>Assignments: allow students to upload documents </li></ul>
  40. 40. Using Viewsflash <ul><li>Create a survey/questionnaire </li></ul><ul><li>Students can login with net id or the survey can be open </li></ul><ul><li>Results can be emailed to you, and/or collected and exported to Excel </li></ul>
  41. 41. Bibliography analysis <ul><li>Look for citations from scholarly/peer reviewed journals </li></ul><ul><li>Look for citations for books and journals owned by Duke libraries </li></ul><ul><li>Look for articles retrieved from Duke databases </li></ul>
  42. 42. Concept inventory <ul><li>Make a checklist of 3-12 important concepts students need to master. </li></ul><ul><li>Ask students to explain each concept in a sentence or two. If a concept is unfamiliar, they should leave blank. </li></ul><ul><li>Count good responses for each concept, then plan future instruction. </li></ul>
  43. 43. Standardized tests <ul><li>I-Skills </li></ul><ul><li>ILT (James Madison University) </li></ul><ul><li>SAILS </li></ul>
  44. 44. Using assessment information <ul><li>Rewrite learning outcomes </li></ul><ul><li>Change what you do/how you teach </li></ul><ul><li>Work collaboratively with colleagues (librarians and faculty) </li></ul><ul><li>Revise assessment measures </li></ul><ul><li>Share the news in the library and on campus </li></ul>
  45. 45. Learning outcomes are: <ul><li>Fundamental components of a learning assessment program </li></ul><ul><li>Focused on student learning </li></ul><ul><li>Indicate what students will know/be able to do </li></ul><ul><li>Measurable, observable, overt </li></ul>
  46. 46. Learning Outcomes Assessment <ul><li>A tool to help librarians </li></ul><ul><li>improve student learning </li></ul>
  47. 47. LOA resources <ul><li>Angelo, Thomas (1993). Classroom assessment techniques. </li></ul><ul><li>Maki, Peggy (2004). Assessing for learning : building a sustainable commitment across the institution. </li></ul><ul><li>Neely, Teresa Y(2006). Information literacy assessment : standards-based tools and assignments. </li></ul><ul><li>Radcliff, Carolyn et.al (2007). A practical guide to information literacy assessment for academic librarians. </li></ul><ul><li>Student Learning Outcomes Assessment (University of Virginia) http://www.web.virginia.edu/iaas/assessment/outcomes.htm </li></ul>

×