PPT, PDF427 views

Gs Os El Os For Ilp 10 30 09

The document discusses program planning steps for instruction, including identifying needs, setting goals, objectives, and expected learning outcomes (ELOs). It provides examples of goals, objectives, and ELOs, explaining the elements that make up a good ELO. Finally, it discusses using assessment instruments like surveys and checklists to measure whether objectives and ELOs were achieved.

Education◦Technology◦

Gs Os El Os For Ilp 10 30 09

1.
ELOs Esther Grassian17 Nov 2009 (10/30/09)
2.
Outline User SatisfactionForms Program Planning Steps ELOs = Expected Learning Outcomes
3.
User Satisfaction FormsSelf-Reports Opinions Affective Value Examples: 1-Minute Paper What Will You Do Differently? Transfer & Apply
4.
Adapted from… Angelo,Thomas A. and K. Patricia Cross. 1993. Classroom Assessment Techniques . 2d ed. San Francisco: Jossey-Bass Publishers.
5.
Questions?
6.
Measuring Learning ProgramPlanning Steps G’s & O’s ELOs What, Why, How & When
7.
6 Program PlanningSteps ID/Recognize Need Describe & Analyze Develop Gs, Os & ELOs Select & Describe Appropriate Instructional Methods/Materials Implement Instruction Evaluate & Revise
8.
9.
What are Gs,Os, & ELOs? Goals : Overall Intent Objectives : More Concrete Expected Learning Outcomes : Demonstration & Measurement of Learning
10.
11.
Example: GoalTo help students learn how to identify useful information sources.
12.
Example: ObjectiveFollowing instruction, learners will distinguish correctly between journals and magazines.
13.
Example: ExpectedLearning Outcome Following a 50-minute one-shot session, 80% of students will identify correctly 3 journal article citations from a bibliography of 10 assorted publications.
14.
Common Elements ofELOs? Learners (audience) Task (behavior) Conditions Standards or Criteria (degree)
15.
Example: ExpectedLearning Outcome Following a 50-minute one-shot ILI session, 80% of students will identify correctly 3 journal article citations in a bibliography of 10 assorted publications.
16.
Example: Expected LearningOutcome Following a 50-minute one-shot ILI session , 80% of students will identify correctly 3 journal article citations in a bibliography of 10 assorted publications .
17.
Types of Os& ELOs Cognitive Affective
18.
Example: Affective O& ELO O: Following instruction, learners will feel more comfortable using chat reference. ELO: Following a 50-minute one-shot ILI session, when offered the choice of email or chat reference, 50% of those who have not used chat before will try it
19.
Questions?
20.
Creating Assessment Instruments
21.
22.
Assessment – How?2008 CL Ref Survey Top Teaching/Learning Choices Critical Thinking Effective Use of Catalog & Databases Awareness of Library Resources Plagiarism [Faculty’s concern] Draft ELOs for CL Instruction
23.
ELO Elements (Mager)Learner(s) Task Conditions Criteria
24.
25.
CL Assessment StepsWrite ELOs ID Questions to Measure Learning Prepare Checklist for Spring
26.
Sample ELO Followingan instruction session, 80% or more of learners will correctly identify 3 criteria for selecting useful databases to search.
27.
Sample Q Whatare the most important criteria for selecting an article index to search? (select all that apply) if it’s freely available what time period it covers how easily you can email citations. what types of materials it includes which topics it covers
28.
Sample ELO Followingan instruction session, 80% or more of learners will correctly list 3 reasons for citing materials.
29.
Sample Q The3 most important reasons for citing are to… show how much you have read on a topic allow others to locate your sources give credit for others' ideas and words make a paper look more scholarly provide evidence to support your argument
30.
31.
32.
Questions?
33.
Review User SatisfactionForms Gs, Os & ELOs What, Why, How & When
34.
Discussion…

Editor's Notes

#2 Hi. This is a workshop on using Expected Learning Outcomes or ELOs for assessment of our instruction I’m going to be combining 2 sessions that Diane Mizrachi & I presented to CL reference this past summer In the 1 st session, Diane talked about the types of user satisfaction forms that many of us have used Then I covered Goals, Objectives & Expected Learning Outcomes in the 2d session
#3 So today, I’m going to start with User Satisfaction Forms Then I’ll go on to talk about: Program planning, Goals, Objectives & ELOs & what all of these are… Why, how & when we hope to use them
#4 User Satisfaction forms ask students to provide their opinions about our instruction They’re self-reports They’re valuable from an affective standpoint—they can tell us how our users feel & think about our instruction Let me show you some examples from the CL web site 1-Min Paper asks students to answer 2 Q’s anonymously… What will you do differently also tries to pinpoint what students think they’ve learned from our instruction Transfer & Apply takes a different tack, as it tries to get people to think about how they’ll use what they’ve learned
#5 [BRING ANGELO & CROSS TO PASS AROUND…] We used to have many more of these kinds of forms on our web site before the redesign, mostly adapted from this 1993 Angelo & Cross book, CLASSROOM ASSESSMENT TECHNIQUES But they don’t measure whether or not students have actually learned from our instruction, & what they’ve learned
#6 22 Any questions so far?
#7 Ok, now we’re going to talk about measuring learning We’re going to do this by talking briefly about program planning steps & Gs and Os, Goals and Objectives And then ELOs, or Expected Learning Outcomes What they all are, why we should be using them, & how and when we’ll be doing this Let’s start with Program Planning Steps
#8 These are 6 basic planning steps for instruction, and I know that many of you are probably already doing a lot of this subconsciously already, though you may not have thought of it this way 1 st , ID/recognize a need—an instructor may ask you to teach a class, or someone may post a list of classes that have been requested Or, you may get a lot of questions at the ref desk about using a particular db, like Lexis-Nexis, and may think a handout, or a workshop, or a guide would be helpful 2d, describe & analyze an instructional need… you probably do this in your own head—oh, someone wants a session for her/his class—what kind of class is it, which dept., what’s the research assignment, what kinds of materials are required or prohibited, how long is the class, how much time will you have with the group, how many students, & so on.
#9 3d, when you discuss with instructors what they want for this session, you’re actually developing goals, objectives & ELOs, though you may not have called them that One additional & important way to figure out what to teach would be to administer a pre-test, so that both you and the instructor could see what the students already know and are able to do & what the problem areas are Ok, 4 th , once you’ve worked with an instructor to figure out what the focus of the session will be, then, again, you probably figure out how you’ll be helping students learn—hands-on demo, in-class exercises, handouts, PPT show, videos, IL tutorials, etc. 5 th –you teach a session or create a course web page, a LibGuide, a video, a handout 6 th , you would evaluate in some way to measure learning, comparing post-test results to the pre-tests, to see what the students learned, and then revise your instruction for next time
#10 I mentioned Gs and Os WHAT’S A GOAL SUPPOSED TO DESCRIBE? The overall, broad intent of the program, the workshop, the 1-shot session, the LibGuide, the handout—whatever it is you’ll be preparing WHAT ARE OBJECTIVES? More concrete—what you want learners to know & be able to do once they’ve completed instruction Objectives describe observable behavior; learning outcomes tell how learner will demonstrate the behavior. Both require active verbs (see ACRL IS Objectives)
#11 WHAT ARE EXPECTED LEARNING OUTCOMES (ELO’S)? How students demonstrate what they’ve learned + measurement that indicates their level of success. Also called Performance Indicators HANDOUT: USING BLOOM’S TAXONOMY IN WRITING OUTCOMES--VERBS—1 PAGE
#12 Here’s an example of a goal: To help students learn how to identify useful information sources. It’s pretty general… It tells you what is the overall intent
#13 Here’s an example of a more concrete objective: Following instruction, learners will distinguish correctly between journals and magazines. KEY QUESTION: How will you know that the learners have achieved this objective? By observing & measuring learning— Let’s look at an example of an ELO for this objective…
#14 ELOs tell you how you can find out that students have learned what you wanted them to: Following a 50-minute one-shot session, 80% of learners will identify correctly 3 journal article citations from a bibliography of 10 assorted publications. Very specific & measurable! And just 1 aspect of the differences between mags and j’s—there are many others, of course, so you could use others if you wanted to QUESTIONS?
#15 Mager, Robert. 1984. Preparing Instructional Objectives. 2d rev. ed. Atlanta, GA: Center for Effective Performance. Learners (audience) Task (behavior): observable action Conditions: situations within which the task is carried out Standards or Criteria (degree): measurable criteria, such as speed and accuracy or # of correct responses, etc., that need to be met if instruction is to be considered successful WHY OBSERVABLE & MEASURABLE? So you can evaluate a program's effectiveness by comparing learner behavior prior to instruction with behavior following instruction. Let’s take another look at our ELO example…
#16 DO YOU SEE THE 4 ELEMENTS HERE? Learners? Task? Conditions? Standards or Criteria?
#17 Learners… (red) Task… (green) Conditions… (reddish/brown) Standards or Criteria… (blue) QUESTIONS?
#18 Ok, not to confuse you, but there are 2 types of Os & ELOs COGNITIVE—WHAT DOES THAT TRY TO MEASURE? Knowledge, use & understanding—e.g., can a learner select a database or a specialized print encyclopedia that will meet her/his info needs? Can a learner use a database effectively? Make ethical use of information? AFFECTIVE? Emotions & feelings—does learner feel comfortable using the Web? Often self-reported, but highly subjective
#19 CAN YOU MEASURE OR OBSERVE CHANGES IN BEHAVIOR THAT WOULD INDICATE THAT AFFECTIVE OBJECTIVES HAVE BEEN ACHIEVED? Yes! Here’s an example…
#20 22 QUESTIONS? Ok, let’s go on to creating assessment instruments
#21 You may find people jumping on the results of instruments that are simply self-reports or “happiness scales.” This just happened recently – In March 2009, Primary Research Group published: The Survey of American College Students: Student Evaluation of Information Literacy Instruction, ISBN 1-57440-116-5. This report presented 125 tables of data exploring how full time college students in the United States view, use and evaluate their college library's information literacy training All of the data in this very lengthy report was gathered through self-reports—It was all about what students thought about ILI, rather than what they learned
#22 That’s important, but it doesn’t really tell you anything about the effectiveness of instruction We need to measure learning through observation or testing to find out what people have really learned But many people find it difficult to construct instruments that measure learning. Why? Often it's because they haven't written down detailed goals, objectives & measurable learning outcomes for their instruction, so they don’t have a firm grasp on what to measure or how to measure it. If we have all of this written down in measurable terms, assessing learning will be much easier. So how are we going to create our own assessments to measure student learning? Diane & I have already gotten a big head start—we’ve developed draft ELOs we could all draw on w/o much time or effort
#23 We started w/ the results of last year’s CL survey where we each listed the top 3 things we wanted undergrads to learn from our instruction Our top choices were: ability to think critically, ability to use the Library Catalog & dbs effectively, & awareness of library resources, including the librarians We added plagiarism to the list, as that’s one of the faculty’s top concerns Then we came up with draft ELOs for each of those topics
#24 Each one includes 4 elements identified by Mager as essential in writing learning outcomes (Mager, Robert Frank. Preparing Instructional Objectives, 2d rev ed., 1984) : Learner or learners; Task; Conditions; Criteria Let’s take a look at them—HAND OUT COPY Notice that up at the top of the page it says, “Following an instruction session, 80% or more of learners will correctly…” So we’ve got the learners in there, we’ve got the condition, and a criteria Look below each category and you’ll see more… Task, using active verbs & additional criteria—e.g., “…list 3 differences between visible and invisible web research tools.” DOES THIS MAKE SENSE SO FAR?
#25 Ok, what I’d like you to do now is to take a closer look at these ELOs I’m going to give each of you a category to examine ASSIGN CATEGORIES—THE SAME CATEGORY TO 2 PEOPLE SITTING NEXT TO EACH OTHER First, take 2 minutes to pick out 2 of the ELOs in your category that you would probably address in a typical instruction session 2 MINUTES Now, I’d like you to take 2 minutes to work with the person next to you. Share what your ELOs are & discuss why you picked those Then we’ll have everyone report back 2 MINUTES REPORT BACK So, now you can see how you can pick ELOs for a session—it doesn’t take that long
#26 So that’s the 1st step—writing & selecting ELOs—we’ve done that for CL instruction, though we could change, add or delete ELOs in this draft The 2d step will be to identify, adapt or create questions that measure learning for each of these ELOs Let’s take a look at some examples of ELOs and corresponding questions that would measure achievement of those ELOs
#27 ELO: Following an instruction session, 80% or more of learners will correctly identify 3 criteria for selecting useful databases to search.
#28 QUESTION: What are the most important criteria for selecting an article index to search? (select all that apply) if it’s freely available what time period it covers how easily you can email citations. what types of materials it includes which topics it covers
#29 Following an instruction session, 80% or more of learners will correctly… list 3 reasons for citing materials
#30 2. The 3 most important reasons for citing are… A. show how much you have read on a topic B. allow others to locate your sources C. give credit for others' ideas and words. D. make a paper look more scholarly. E. provide evidence to support your argument QUESTIONS?
#31 I’m working with Adele Dobry, our Reference/Instruction Intern, in Fall Q to identify, adapt or create q’s to match our ELOs Some may be drawn from existing questions in RTR, while we may come up with others Once we have the questions, I hope that they would be entered into a database along with the ELOs, possibly in Winter Q Then the 3d step… For Spring, we’ll planning to prepare a checklist of these ELOs As CL librarians arrange for an instruction session, they’ll check off which ELOs apply to that session Then they’ll take maybe 2 questions for each ELO & use the ones they select as a pretest and a post-test
#32 Ideally, we would have all of this set up online w/a checklist form that a librarian fill out that would automatically generate test questions as web page forms You could send the url in advance as a pretest & then use it as a posttest in 5 minutes at the end of class Once it’s established, this system could be used by anyone—you could develop ELOs that apply to what you’re teaching and questions that measure whether or not your students have learned what you expected them to All could be input into a database & then any of us could use the checklist form to create our own pre-/post-tests to measure learning Until it’s set up online, we would have do this manually, and our Intern will help us do that in CL Librarians will fill out the checklist & Adele will pull the questions & create the pre-test/post-test as a Word doc or as a Zoomerang survey, for distribution to the instructor & for use at the end of the class
#33 Questions? <a title=&quot;Questions Answered...&quot; href=&quot;http://flickr.com/photos/travelinlibrarian/223839049/&quot;><img src=&quot;http://farm1.static.flickr.com/93/223839049_12f0df6d95.jpg&quot; /></a><br /><small><a title=&quot;Questions Answered...&quot; href=&quot;http://flickr.com/photos/travelinlibrarian/223839049/&quot;>cc licensed flickr photo</a> shared by <a href=&quot;http://flickr.com/people/travelinlibrarian/&quot;>Travelin' Librarian</a></small>
#34 Ok, we talked about user satisfaction forms, what they are, why we use them, how and when I showed you some examples & referred you to the Angelo & Cross book for more Then we talked about Goals, Objectives & Expected Learning Outcomes, again… What they are, why we use them, how and when We looked at some examples & did an ELO exercise Then I explained that the database, checklist and forms I described could be utilized by anyone, once created Now…

Gs Os El Os For Ilp 10 30 09

More Related Content

What's hot

Similar to Gs Os El Os For Ilp 10 30 09

More from Esther Grassian

Recently uploaded

Gs Os El Os For Ilp 10 30 09

Editor's Notes