Question we think we’re all struggling with — how to teach evaluation, and in particular - how to teach evaluation in a way that transcends the specific requirements of a particular assignment. I’m going to spend a little time framing the next hour, and I’ll ground that in a few concrete things we’ve done at OSU. And then Meredith and Sara will both talk about some specific things they’ve done — and the data they’ve gathered — to push the conversation about sources and evaluation further.
So this is something that I brought back from the Annual Conference on the FYE last year —it’s just a quick trick to help with reflection that I think also provides a really good jumping off point to talk about what pulls our thinking together.
Three short questions that you can use to structure all kinds of reflections. When I saw them, it was to structure reflective writing about learning experiences — what (what did you do). So what? (What did you learn) Now what? (What are you going to do with it).
They captured my imagination in part because they get at an issue that frequently see in student evaluation — a lot of my students want to stop at the “what” stage. At best, they focus on accurately describing their sources, usually in the context of “does this meet X requirement” and they don’t push on to the
so what – what does this mean for their learning and reflecting Or Now what – what does it mean for their research process, what will they do next.
And I think some of this comes from how many of them are thinking about research – and I won’t spend a ton of time on this. I think we’re all familiar with students who approach research as a process of finding sources to support what they want to say, or of finding sources that will define what they want to say – and I think we know there are many good reasons– ranging from cognitive development to prior experience to the feedback and rewards they get in their college classes -- driving this way of thinking about research
And if this is how you think about research – finding sources to support stuff – then relevance is a super helpful evaluation tool. Using “is it likely to say what I want it to say” as a filter that helps you manage or make sense out of all of that information you’re pulling out of a discovery layer, or google scholar – seems pretty rational to me.
Now, what this slide is meant to show is that we have a good amount of research looking at how students choose sources
One thing we’ve fairly consistently found that they do prioritize relevance and convenience/accessibility when they choose sources.
This is sometimes framed millennial behavior, or lack of engagement or lack of skills - and I don’t find any of those frames that useful. I do think of it as sensemaking, or organizing – mental tools they’re using to filter or scan through a lot of stuff.
And some of these studies go further, comparing how they talk about research, or do on an evaluation lesson to what they choose - and they find that they can talk about the things you talk about (currency, authority, accuracy) pretty well — but that in the throes of an actual research process, faced with actual results, they default to relevance and convenience.
And we do try and give them other concepts, other filters they can use to make sense of sources. But I’m not sure we have figured out how to do that – especially in a one-shot – in a way that pushes them beyond what they can see by looking at the source itself.
I purposefully organized this slide so that you couldn’t read these individual frameworks and checklists – because the point here isn’t a critique of specific criteria -- whether or not you use a checklist or a framework or a set of criteria to talk about evaluation or not – these criteria are out there. These concepts are in the air and they come up when evaluation does – they’re an attempt to make it possible for novices to mimic the cognitive processes experts use to filter through information.
But they fail – because experts don’t stop at “what”
What do I mean by that. Let’s take a few common examples –
Authorship – what is pretty easy, but to get to So What almost anyone is going to have to do some additional research. So what will tell the students that Professor Behrenfeld is on the faculty of the Department of Botany and Plant Pathology at OSU, that he’s an expert on marine algae & so on. Now what pushes further — so what are you going to do with this source and this information? How are you going you use it? What’s your next step, connected to this one.
http://staffieclub.co.za/ Evaluation, as it exists in our culture, is an inherently social act. We don’t have a platonic ideal of sources. Sources don’t have inherent value, that can be ascertained by how well they match up to a set of predetermined standards or criteria. Sources are as valuable as we think they are.
And their value changes — with the community, with the rhetorical situation — this value is not a static thing. And if what we’re doing to help students with evaluation encourages them to think of evaluation as static, as a static set of standards, then we’re not helping them to take what they learn with us out beyond the classroom.
When we think of where we want students to get with evaluation, I think for all of us it’s fair to say that we want them to start thinking about their sources rhetorically — that they consider these contexts and conversations to make sense of the sources they find and want to use.
So at OSU, this is how this played out in our comp classroom.
The students were required to use research and peer reviewed sources, so we started them off here - browsing popular sources about research
Taught them how to search our discovery layer by author before we searched by keyword. This had extra value - pulled back a variety of resources on the same topic, and could also show them when their scholar is also treated like an expert in the popular media
They checked out faculty pages.
And if possible, the lab, research group, etc.
Now, our FYC comp curriculum changed dramatically after the year we piloted this - so we don’t have great assessment about student learning. But it dramatically simplified the teaching process.
This simplified the teaching dramatically: In terms of what they can expect to find in scholarly sources (research - b/c starting from the research and researcher)
In terms of building on existing knowledge — they know what an author is and some methods for looking up a person.
In terms of integrating their evaluation into their paper. MLA works for this.
Working with Deb Gilcrest, librarians identified a guiding inquiry question to inspire our assessment of student learning - To what degree do students CARE where their information comes from?
With that question in mind, and growing from conversations between writing faculty and librarians, we became interested in doing an exploratory study of student language around source selection.
We were curious to look at the words students used to describe why they did or didn’t choose a particular source in order to get insight into what considerations they make (or don’t make) while selecting a source.
We also chose to start with their language because we were curious whether students would articulate values we weren’t considering.
After a fall pilot we emailed individual WR instructors at all campuses and online courses and asked them to give students the online survey (using a Google site and Google forms) during the 10th week of the term, post instruction.
The survey presented students with a claim and a list of citations and direct links to four sources- -and the asks students to select one source to support the claim
Students who were able to articulate more than 2 different kinds of reasons why they selected a particular source - for example, the author and references, or publisher and currency - were significantly more likely to select the best one
as educators, we aim for complexity and nuance of thought - ah ha! just like how we like to approach beer in Portland! Deschutes Brewery tasting lineup
the more complex their thinking about sources, the more nimble they will be while navigating the information landscape
this is the engaging and exciting aspects of thinking critically about sources that librarians are passionate about - and we don’t get to teach enough -
that the value of a source is situated and contextual - and that there are power structures in place that determine authority and the publication process - the SO WHAT of how information is made and accessed (or not) -
good and bad buckets don’t help our students, and this kind of thinking is what we’re faced with
realized that a checklist approach (which I’ve never *liked* but used begrudgingly) was doing a disservice to my students by reinforcing the notion that sources could be plunked in a good or bad bucket
i dreaded that inevitable last-minute request from teaching faculty to “ also cover how to evaluate sources” - the kind of situation where a checklist was most efficient, it was getting to the point where I felt I was truly misleading my students by diluting it for them -
and when I did get the chance to unpack and have engaging discussions about sources in ways that make them complex - i had nothing to point to that the students could refer back to - or to use as a conversation starter to jog their thinking
while we face many barriers to teaching students how to evaluate information sources, one problem is that of a gap in understanding - the first year students i work with typically have significant leaps to make in their conceptual understanding of how sources are made and how to differentiate sources, and what to consider when deciding if an information source meets their particular needs
so while the checklist approach aims to prompt critical thinking about sources, the conceptual understanding of how information is produced and accessed isn’t there to latch on to.
without a nuanced and contextualized understanding of concepts such as authorship, authority, editorial process, audience - we can’t expect students to apply such a checklist when picking out sources - and we know they don’t do it outside of when we force them to, anyway
[better image ideas...skateboard, making the leap from one thing to another?]
i looked to infographics as a teaching tool to introduce considerations to make when navigating information sources
motivated to show what goes into making sources instead of tell, expose visually some of the weight of different considerations one can make when evaluating sources
avoid “better than” paradigm
limitations: significantly limited by the data we could find medium - fixed images, scrolling environment
“classroom enhancement” grant from student government ($200)
work driven by student interns: Ashley and Andrew
embarked on a process of asking ourselves both what metrics we use to evaluate information sources and what data sources might provide insight into the why some criteria have weight or are worth consideration
Ashley did research and writing Andrew did the design
after many iterations and input from colleagues we developed this infographic
it starts by describing sources and introducing icons for each
add: list of source types
volume of sources in the world, why we come across some more than others, might need to go out of our way to find others
editorial process, time and number of reviewers levels of scrutiny BEFORE source is published introduces “comments” in social media as a form of crowdsourcing editorials post-publication
education as ONE factor and number of authors this is a good one to talk about what’s missing from establishing expertise, authority
why do we include other people’s ideas in our writing? why do we expect that from things we read? does it matter to know where the author got his information from?
one of the more successful graphics, originally from the author’s perspective, the amount of jargon or specialized vocabulary used by the writer
Andrew did a graphic from a readers perspective, much more elegant representation of the experience of encountering sources and an opportunity to talk about the value of building understanding in order to bridge to more complex and specialized information sources
we can all relate to the advanced reader being puzzled, normalizes the experiencer of encountering difficult texts
Good for What? Teaching Sources for Sustainable Lifelong Information Literacy
Good for What?
Teaching Sources for Sustainable
Lifelong Information Literacy
Meredith Farkas, Portland Community College
Sara Seely, Portland Community College
Anne-Marie Deitering, Oregon State University
26 March 2015ACRL 2015 Portland, Oregon
BEAM me up!
Portland Community College
Freshman Inquiry Assessment
FRINQ = a year long GenEd class focused on writing,
critical thinking, quantitative literacy, diversity, and ethics
and social responsibility
The library has a close relationship with the program
Students create ePortfolios
Assessed for critical thinking and writing already
How clearly do students define their research question or
How well do students integrate outside information into
their paper and attribute it?
Do students use relevant sources?
Do students use authoritative sources?
Students didn’t seem to understand the purpose of
sources in research
Many summarized sources and didn’t use to bolster
argument or illustrate point
Sources often not relevant to the topic
Seemed forced in to meet a requirement
We felt that “sources” wasn’t meaningful to students
Do students really know what
to do with the sources they’ve
Do they know what they’re
looking for in the first place?
“Students… think of research as
going to the library or the Web to
find articles to support a pre-
Bean, John. “Backward Design: Towards and Effective
Model of Staff Development in Writing in the
Disciplines.” In Writing in the Disciplines. New York:
Palgrave Macmillan, 2011.
Choose a topic
Search for Sources
What we see
What we’d like to see
How do librarians typically classify
sources when explaining them to
We tend to focus on how they’re
made rather than how they
can be used
Bizup, Joseph. "BEAM: A rhetorical vocabulary for teaching
research-based writing." Rhetoric Review 27.1 (2008): 72-
“If we want students to adopt a rhetorical
perspective towards research-based writing
then we should use language that focuses
their attention not on what their sources and
other material are… but on what they as
writers might do with them.”
-Joseph Bizup, “BEAM: A rhetorical vocabulary…”
B = Background
E = Exhibit
A = Argument
M = Method
“Writers rely on background sources, interpret or analyze
exhibits, engage arguments, and follow methods.”
B = Background
Provides you with
context and big
E = Exhibit
Used to explain,
illustrate, analyze or
Usually are primary
sources, primary data,
A = Argument
Used to strengthen, refine,
or complicate an argument
Written by experts (books
M = Method
An idea, framework, or
lens that informs your
Can be an
approach, a theoretical
And when students are thinking
about their sources
Activities for the Classroom
Have students list the sorts of evidence they think would be
useful for their research using at least 2 or 3 of the
categories in BEAM
Students who already have found sources: consider how
they plan to use each one using the rhetorical vocabulary of
Reading an article to see how the author uses sources
based on BEAM
Have students evaluate sources for authority based on how
their intended use in the BEAM model
Know Your Sources
using infographics to inspire complex thinking
Sara Robertson Seely
Portland Community College
WR 121 & WR 122
“Locate, evaluate and use
information effectively and
“Evaluate source materials for
authority, currency, reliability,
sound reasoning and validity
What considerations do
students make when
describing why they
select one source over
What do students value
in a source?
source selection assessment
over 200 student responses
at least 38 WR121 & WR 122 courses
~20% response rate
Students who made 2 or more different types of considerations
were significantly more likely to select the best source.
Ashley Downs, MS
Graduate Student Intern
School of Information
College’s Graphic Design