We will briefly look a the big picture of strategic
Introduction, challenges of assess one-shot instruction (time and staff limitations, unaccomdating instructors). Brief discussion on nature of UCR’s workshops, how long, how many students, challengesNo opportunityfor the kind of grading rubrics used in full-length courses
Focus on how well we’re achieving learning goals makes us better teachers and. Assessment becomes part of our workflowSpecifically for assessing LI sessions; Are they grasping certain concepts and not others? Do we need to spend more time on certain concepts? Are we trying to teach too much?formative: see where students are now and adjust teaching accordingly in sessions
What are we going to teach in this 50-min session?How are we going to teach it (lecture, exercises)3. Use our chosen assessment methods. Have objectives been met for this session?4. Results: What did they learn? Did we teach well? Did we assess well? – students also use results to improve their learning 5. (1) Reflect and make improvements to learning goals, teaching and assessment methods?
Program evaluation: Are we measuring what’s important and are we measuring them accurately? Are all of methods of instruction working well? We can also perform program evaluation as it relates specifically to LI sessions. Are LI sessions a good use of student and library time and resources? Student Outcomes? For this presentation I’ll be focusing primarily on student outcomes as they relate to LI session
Shouldn’t wait until the program is perfect, get started with assessment immediately, assess, adjusts and improve
Learning outcomes for each course can be determined by librarians and appropriate faculty. We can “turn” this table and see how we can then start to develop learning goals and assessments for specific LI sessions.Map allows us to see the bigger picture beyond the individual workshop.
Which ones are chosen for a specific workshop will also depend on where students are within an assignment. This must be done collaboratively within the library and also, if possible, key faculty (better goals, more buy-in).Some classes targeted for more intense instruction and assessment, such as college englishThere is a real balancing act in working so many learning demands into a session.
Articulate specific outcomes:Show what it look like, acceptable levels (specific score on test, citation format accuracy %, etc.)Assessment plan review: What don’t we know about student learning? Are we measuring what is important? Are we asking the right questions? (phrase it like a research question) What isn’t measured well? What do we already know and can stop measuring? Focus on new information.Are goals realistic and scaffolded/targeted well? What outcomes are best taught and assessed in IL sessions? What should be taught through a different vehicle?
There should be one direct and indirect measure when possible. E.g. a post-test survey asking what they feel they learned or didn’t learn.
no one best method formative informs instruction: pretesting students’ existing knowledge and adjust content of session accordingly so it’s more appropriate and engaging (clicker technology or assessment done before session through a learning activity, follow up with pretest (could be clicker at beginning of session to see if students mastered skills measured in assignment)summative or “outcomes” assessment: for improving instruction, measuring effectiveness of sessions
multiple choice pre- and post test. More quantitative, easily linked to specific learning outcomes, validated questions available that are commercially and institutionally developed and administered products that are used among librarian peers across the country and throughout the world.summative Ask librarians what they have used, what they liked or disliked about eachMultiple choice post-test questions mapped to specific ACRL learning outcomes. could use already validated questions from survey instruments developed by other institutions, can always modify for UCR and specific programs. Develop test bank. Each year questions can be chosen from bank for specific classes so comparisons and aggregate data can be collected.Easy to administer, takes burden off teaching librarians, easy to administer (online so results are recorded are automatically tallied, focus on learning goalsIn general grading rubrics are too complex for one-shot library instruction
interview/survey instructor: was session helpful to student research? Did learning happen? indirect, summativeformative: Survey Monkey (attitude, pre/post test, Google docs survey (can see results immediately and address in class). make image
Example exercise involving: (frame question, research answers, convey results)
There is often a trade-off between the quality of the assessment data in measuring learning outcomes and how easy it is to score and tabulate the data!This form is especially valuable to the librarian teaching this specific session.Can also send an electronic follow-up survey to instructor a few weeks later. If their learning goals were met, effect on long-term student performance, good use of class time, etc.
Target specific classes (where instructors are especially supportive) for longer sessions or collaborations outside library sessions. For example, assignments requiring interaction with librarians, researching in library, instructor shares assessments with librarians. Collaboration on e-portfolios. This would show a greater impact of library (with instructor help) .Much or all outside assessment done by class instructor. Assessment is part of course grade so students are motivatedInteractive tutorial and quiz developed in Captivate finished before workshop (accessed by student through LMS)
For example, if we’re assessing if students use a wide variety of sources, is our measurement reliably predict if they actually do use a wide variety of sources in their actual assignments? One problem is short-term vs. long-term learning.Each repetition of the assessment Going through the assessment cycle after workshop allows librarians to adjust learning goals and outcomes, vary instructional strategies, experiment with different assessment methods, and improve over time.
First example is active collaborative learning with assessment embedded in assignment.Second is quantitative and easily measured and tabulated. Good questions can get students to use critical thinking skills, can be tied to and follow up on active learning activities
New social, information and communication technologies, immediate feedback to students and librarian, no need to tabulate from paper form (but can have print version if students need it).Performance based exercise (exploring general info standard) They find a appropriate encyclopedia article given a topic and find something that is bacgroundrathan than spcific and give citation.Could be emailed to librarian stright from database and checked in session possibly, or cut and paste into Google spreadsheet or fill out Google Docs worksheet. Also place on form for students to provide further questions (they can include name and email for librarian followup)
What kind of impact can we make in 50 minutes? Ways to get more bang for our buck.
Assessments are more than evaluation, they are part of the learning process – students can learn by completing a well designed assessment.Assessment improves our own teaching skills because it makes us focus on learning goals and outcomes, gives us feedback so we can implement change.It becomes a cyclical learning process for both students and librarians.
Information LiteracyAssessmentChallenges & Opportunities forOne-shot Library Instruction Presented by Diana Myers Hyatt October 23, 2012
IntroductionExamine assessment process andbest practices for assessing LI sessionsPlanning for learning and assessment,from the program to the session levelSpecific assessment tools suitable forsessions, their strengths and weaknessesSpecific examples of incorporatingassessmentHow can we leverage limited resources formaximum learning impact?
ChallengesChallenges of assessing in one-shot instructionsessions Limited class time. How do you effectively measure learning in such a short time? Librarian staffing and workload Limited librarian training in teaching and assessment Limited support from faculty/college administration UCR trends? Budget cuts?
Why assess? Improve instructional programs and teaching methods to better meet learning goals. Improve student learning. Gives them a chance to review learning goals, see their progress and what they still need to learn. Part of accountability, meeting department and institutional learning goals. Demonstrate value of library with evidence.
What is assessment? Assessment to improve learning: How well are students learning what we intend them to learn? Measurement of student learning based on established learning outcomes. Continuous, cyclical process of incremental improvement and change Assessment Primer: How to do assessment, University of Connecticut www.assessment.uconn.edu/primer/how1.html
Assessment Best Practices (ACRL) Program evaluation: Develops a process for program planning, evaluation, and revision. Measures the progress of meeting the program’s goals and objectives Integrates with course and curriculum assessment, institutional evaluations and regional/professional accreditation initiatives. Uses appropriate assessment/evaluation method for relevant purposes, for example formative and summative and/or short-term and longitudinal. Student outcomes: Acknowledge differences in learning and teaching styles in the outcome measures. Multiple methods: needs assessment, pre-tests, post-tests, portfolio assessment, oral defense, quizzes, direct observation, peer and self review, and experience. Focus on learner performance, knowledge acquisition, and attitude appraisal. Assess the learners’ process and product. Include learner-, peer-, and self-evaluation.Characteristics of Programs of Information Literacy that Illustrate Best Practices: A Guidelinehttp://www.ala.org/acrl/standards/characteristics
Assessment Best PracticesAssessment program should: be continuous: drive change and innovation in an ongoing, iterative process, pinpointing areas for improvement and development be learner-centered: focus on student learning rather than on us measure what students are learning as accurately and directly as possible learning goals should be flexible enough to be adaptable to specific courses, integrating current program goals (implicit and explicit) with ACRL standards When possible, allow for immediate teaching improvement by informing teaching librarian of current student skill levels through assessment opportunities prior to the workshop
Strategic plan for learning andassessment Moving from the general level to the specific (session level) Library develops main learning goals integrating university and library missions, ACRL standards and input from librarians and faculty From these goals, articulate specific learning outcomes which can be assessed and measured ◦ target groups, scaffolding (e.g. first-year English, graduate art students) ◦ Delivery vehicles (LI sessions, online tutorials, embedded teaching) Determine which specific learning outcomes are best addressed in LI sessions.
Mapping learning outcomes tospecific courses Oakleaf, M. (2009). WRITING INFORMATION LITERACY ASSESSMENT PLANS: A guide to best practice. Communications in Information Literacy, 3(2), 80-90.
Learning plan for LI sessionsLearning outcomes for specific LI sessionsbased on: Library-level goals for information literacy Learning goals of the specific course and professor. Instructor collaboration, course syllabus and library curriculum map Students’ current course assignments and IL mastery level. Assignment description, pre- session assessment (assignment or pre-test)
Assessment component oflearning plan A good plan gives teaching librarians time to focus on teaching and engaging students, because learning objectives and assessment tools for each workshop are clear. Articulate what specific results will show that students have mastered a learning outcome. How will assessment measures be gathered and reported? How will the gathered data be reported and shared? Assessment plan is reviewed on a regular basis. New assessment methods should be tested before using on a widespread basis.
Assessment planning for LI sessions Rubric-based assessment plan combining learning outcomes paired with appropriate assessment tools and scoring. Done collaboratively within the library. Learning outcome Assessment Acceptable instrument performance criteria Explores general Learning activity: Find Appropriate source, information sources to and cite one reason given, proper increase familiarity with appropriate article from citation format the topic online subject encyclopedia Assessment resources developed (test banks, learning activities with worksheets, etc.) How to measure success. What scores will be considered adequate? Teaching librarian now has appropriate learning outcomes and ready-to-use assessment tools that can be easily adapted for specific LI sessions.
Summary of assessment techniquesTechnique Potential Strength Potential WeaknessLocally developed multiple Can provide direct evidence of Reliability and validity generallychoice tests student mastery of learning are unknown objectives Creating effective exams requires Appropriate mixes of items allow time and skill faculty to address various types Traditional testing methods may of learning objectives not provide authentic Can be centrally developed and measurement then adapted for specific workshops Easily scored and tabulatedEmbedded assignment and Can provide direct evidence of Requires time to develop andactivities student mastery of learning coordinate and score, need objectives simple grading rubrics Involves active learning, possibly Requires faculty trust that the more retention program will be assessed, not Can provide authentic individual teachers assessment of learning Reliability and validity generally objectives are unknown Data collection is unobtrusive to Norms generally are not available studentsUniversity of Connecticut Assessment Primer, Assessment Primer, http://assessment.uconn.edu/primer/how1.html
Summary of assessment techniquesTechnique Potential Strength Potential WeaknessSurveys, one-minute Are flexible in format and can include Provide indirect evidence aboutpapers, 1-2-3 mode, questions about many issues student learningSurvey monkey, clicker Can be administered to large groups of Validity depends on the quality oftechnology, Poll respondents the questions and responseAnywhere options (avoid guessing) Can easily assess the views of various stakeholders What people say they do or know Questions generally have a clear relationship may be inconsistent with what to the objectives being assessed they actually do or know Can be conducted relatively quickly and Open-ended responses can be easily difficult and time-consuming to analyze Responses to closed-ended questions are easy to tabulate and to report in tables or graphs Open-ended questions allow faculty to uncover unanticipated results Can be used to track opinions across time to explore trends Polls can be used for quick pre-tests with instant feedback, encourage active participation
Embedded performance-basedassessment Our assessment methods should support problem- based and other active learning scenarios rather than straight lectures. Combining active learning activities with assessment Could include collaborative learning exercises, creating 3-item bibliography (encyclopedia article, book, journal article), other focused library activities with worksheets, concept maps or other concrete product that can be assessed -- better learning using constructivist methods. Rubric for quantitative measurements. Or multiple choice post-test afterwards for easier assessment. Trade-off: more complex learning, but harder to measure in class.
LI session evaluation by students 3-2-1 model using Google forms. More structured than one-minute paper. Indirect measure, but can reveal important information. How well are responses aligned with our teaching goals? Gives students a chance to reflect on and articulate what they’ve learned for better retention Although open-ended questions are harder to compile for statistical purposes, they give better feedback to librarians to improve instruction. Provides an opportunity for personal follow-up with students who want it by providing space for questions and email info.Nardine, J. and Meier, C. (2010), "Assessing the one-shot instruction session: Leveraging technology for optimum results", VirginiaLibraries, Vol. 56 No. 3, pp. 25-8
Assessment outside workshops Library/Faculty collaboration on class assignments integrating IL learning outcomes. Target key classes and assignments such as English composition, introductory classes. ◦ bibliographies, annotated bibliographies ◦ research blogs/journals on their research experience - very qualitative but could be eye- opening. ◦ shared assignments with linked learning outcomes ◦ students complete online IL module with tutorials, activities and assessments.
Evaluating our assessment methods Is the method accurately measuring the learning outcomes it’s supposed to? Is it valuable? Does it improve our teaching? Does it put reasonable demands on our staff and resources for the value it brings? Comparison of multiple forms of evidence for the same learning outcome (including class assignments)
Assessment method examplesACRL standard 1.1: The information literate student defines and articulates the need for information. Small group activity: hypothetical information need situation (buying a car, taking a trip), articulate the information need and specifics of what information is needed, also brainstorm possible sources (friends, consumer reports, review or travel websites, travel guides). Assessment: state general information need, 3 questions that need to be answered, 3 possible sources in Google Docs Multiple choice post-test using Google form: what is the most important first step in library research? ◦ What is the most important first step in library research? a) identify key authors on research problem. b) know research problem c) Locate key articles on research problem d) Locate key books on research problem
Give them a broad topic such as “commuters and Internet crime”. Have them come up with a good research question and identify keywords for first stage of research. Could use Google spreadsheet or Google form. Could use concept map or Wordle for keyword brainstorming.
Sample use of Google Docs for learning activity Standard 1.1c Explores general information sources to increase familiarity with the topic. Performance based exercise: Given a topic by librarian, find an appropriate encyclopedia article that gives background information on aspect of the topic and create citation. Answers cut and pasted into Google spreadsheet or fill out Google Docs form.Shannon R. Simpson, Google Spreadsheets and real-time assessment: Instant feedback for library instruction, Coll. res. libr. news October 2012 73:528-549.
Reliability and Validity ofAssessment MethodsReliability Consistent responses over time. Variation in scores is actual difference in what is being measured, such as knowledge Students have adequate time to take assessment, guessing is minimized Consistent methods for all sessions (e.g. same post-test questions). Importance of collaboration and buy-in in developing assessment methods. Must have consistent scoring or rating scale, so easy to score and guidelines for scoring are clearValidity The assessment tool measures what we want to measure? Do our results correlate with other proven measures that assess the same learning outcome? Does our method accurately predict future student performance?
Conclusion No one best way to assess student learning. Use a variety of methods, direct and indirect. Trade-offs between more thorough assessments and time needed to do them. Have to prioritize learning outcomes and make sure they are realistic given time and staff constraints. Developing a “learning bank” of active learning activities and specific assessment tools, such as a test bank, can give teaching librarians more time to focus on improving their teaching methods. Program-level learning plan helps us see the big picture. Learning objectives in one LI session are scaffolded with other sessions. Complement other learning vehicles such as class assignments, tours. Use assessment methods that involve a learning activity. Learning and assessment together.
“Good teaching is inseparable from good assessing. The question, therefore, is not whether to evaluate students, but how to measure performance in ways that will enrich learning, rather than restrict it.”Ernest L. Boyer, The Basic School: A Community for Learning (New York: The Carnegie Foundation for theAdvancement of Teaching, 1995), 104.