We are presently working in an environment where one of the predominant narratives about universities and higher education is one of crisis, arguing that our institutions are failing to deliver in their educational mission and our students are not gaining the skills they need. Whether or not we agree with this narrative—and there is much to critique in these studies—universities are being asked more often, and with greater intensity, to justify themselves by measuring their educational outcomes.
ERIAL Project Information. . . .
The ERIAL project was motivated by the question: What do students actually do when they are assigned research projects for class? We know about the assignment, and we know about the results, but what happens in between is a “black box.”
We asked members of each group about expectations that they had for one another and about experiences that they had working with each other.
But, for my purposes here I’d like to like to focus in on just one part of this larger project, which was an evaluation of first-year information literacy at Illinois Wesleyan university.  that was designed to evaluate the effectiveness of library instruction in first-year seminars [READ SLIDE]. We were hoping to show that library instruction improved information literacy—but unfortunately it didn’t
ERIAL was built on a unique ethnographic methodology, which employed close observation of students’ research habits. In order to construct a holistic portrait of students’ research practices, the ERIAL project used a mixed-methods approach that integrated nine qualitative research techniques that were was designed to generate verbal, textual, and visual data. In total, the project involved more than 700 research contacts with over 600 participants. This presentation will focus on two of these methods: semi-structured ethnographic interviews and research process interviews. Largest ethnographic study of libraries to date.
The starting point for this research was two liberal arts universities that I worked at had recently implemented different discovery tools– Illinois Wesleyan University and Bucknell University. I was not involved in the decision to purchase the discovery tool at either university, but as an anthropologist I was very interested both in how students search for information generally, and how they use discovery tools that are meant to break down the silos of information that are often typical in academic libraries. IWU and BU have demographically similar student populations, as well as similar collections & electronic subscriptions, so this provided an opportunity to compare these tools “in the field”
We wanted to understand both how students used discovery tools, but also how this usage compared to other “discovery” type tools like Google scholar, as well as the conventional library catalog and subject databases set up. And we wanted to collect both quantitative measures of effectiveness and qualitative measures of students’ experience. We recruited 86 students to participate, [Sirsi at Bucknell and the Voyager VuFind interface at IWU.]
And divided them into 5 test Groups. We gave each student a set of four research tasks, which were similar to questions that they be given for an academic assignment. [Civil Rights Act, Women’s Baseball, Wealth and Happiness, and Climate Change]. They were asked to find 2 resources they would use for this assignment (recorded on a web form). These were then scored for quality against a rubric of 0-3. We recorded the students using screen capture software, which we played back the students in a debriefing interview. This interview focused on the thought processes the students went through to construct the search and to evaluate the resources they found
When scored with the rubric, students using Google Scholar receive the lowest scores, while EDS was the highest. Summon was in the middle at 1.92. This difference between Summon and EDS is usually the finding most interesting for librarians, and explaining why this difference occurred is one of most interesting outcomes of this research. EDS significant compared to all others (ANOVA, .05)
As we watch the ways students searched for information we found definite patterns:
[Explain] Every search is Google like
Every search found on 1st page
In particular, students were having tremendous difficulties evaluating their materials—the score on the evaluation section of the test was relatively high-- but it appeared that students knew the right answer, but couldn’t actually perform the task [read slide]
Back to evaluation– Together these equal == determination --Explain slide This explained a good share of variation in scores—especially between EDS and Summon
Backed up by Bucknell’s usage stats--
Returning to the issue of evaluation, these processes result in the defacto outsourcing of the evaluation step to the algorithms of the search system. This potentially introduces several biases into students search practices. [read slide & explain]
What, in general terms, does an ethnographic project look like? I see ethnography as an iterative process that begins and ends with research question development. It then moves through a data collection phase, a period of data processing, data analysis, developing and evaluating recommendations, and finally creating new research questions and assessment strategies. I’ve developed a ethnographic toolkit for use in libraries that describes in detail the steps of conducting an ethnographic project form beginning to end. I will briefly talk about each of these steps:
The Changing Role of LibrariesEthnographic Approaches to Understanding Students Andrew Asher Lynda Duke Assessment Librarian Academic Outreach Librarian Indiana University Bloomington Illinois Wesleyan University
How do students use these tools? 86 students participating 41 IWU 46 Bucknell Qualitative and quantitative measures of search practices
Methods 5 Test Groups Summon EDS Google Scholar “Conventional” Library Catalog No tool 4 Research Tasks Find 2 sources per task Evaluated using a 0-3 scoring rubric by instructional librarians Debriefing Interview Open-ended questions on search practices and evaluation processes
Google structures expectations Single search box Simple keyword search
(Almost) Every search is a Google search:Overall, simple search was used 82% of the time.
“I basically throw whatever I want into thesearch box and hope it comes up. . . .But it’s like Google and I use it likeGoogle. I don’t know how to use it anyother way.” --Junior inNursing
Constructing a Search “Too much information”Simple Search “Not enough information” Students iterate search rather than refine “Magic” Search Terms Poorer quality search terms
92% of the resources utilized werefound on the first page of search results.
Search Evaluation Cursory evaluation of sources Eclectic, and sometimes inaccurate, methods of evaluation Assumption that if information is not easily found then it must not exist “Apparently you don’t have much on Rock and Roll” --First Year in French
What a tool searches determines what students use:
Usage of selected newspaper databases at Bucknell, 2009-2011. 2009 Click- 2010 Click- Usage increase 2011 Click- Usage increase throughs throughs compared to 2009 throughs compared to 2009 ProQuest National 131 1,475 1026% 918 601% Newspapers Premier Ethnic NewsWatch 60 562 837% 481 702%ABI/INFORM Trade & 28 220 686% 107 282% Industry Americas Historical 15 101 573% 24 60%Newspapers, 1690-1922LexisNexis Academic 1,280 6,977 445% 5,233 309% Total, All Databases 49,886 90,854 82% 89,116 79%
Search EpistemologyDe facto outsourcing of evaluation to the search algorithm itself. Brand Bias Default Bias Trust Bias
An Ethnographic Project Assessme nt Data Evaluation/ ResearchData Collection Processing Data Analysis Recommendatio Question n Research Question
For more information:E-mail:firstname.lastname@example.org@iwu.eduTwitter: @aasherWebsite: www.erialproject.orgToolkit: www.erialproject.org/publications/toolkit/Discovery Paper: http://crl.acrl.org/content/early/2012/05/07/crl-374.full.pdf+htmlBook: College Libraries and Student Culture (ALA Editions, 2012)