Download to read offline
Knowing Me, Knowing You: Assessment and the Archival Metrics Toolkit at the RBML
- 1. KNOWING ME, KNOWINGYOU: Assessment & the Archival Metrics Toolkit at the RBML COSA – NAGARA – SAA Joint Annual Meeting Friday August 15, 2014
- 2.
- 3.
- 4.
- 5. Goals • Tounderstand users’ needs and experiences using the RBML • To evaluate service at the RBML • To collect data in a more strategic way • To establish a long term, iterative, sustainable assessment strategy for the RBML
- 6. Lots of Data-No Big Picture LibQual In-house Database Reading Room Stats
- 7.
- 8. Archival Metrics ToolkitSurveys •Onsite Users • Students who have attended an orientation session • Instructors who use the archives for teaching • Online users of the website • Online users of finding aids
- 9.
- 10. HOW DID WEDO IT?
- 11. Survey Form andCollection Box
- 12. WHAT DID WELEARN?
- 13. Are you affiliatedwith Columbia University? Yes 37% No 63%
- 14. What best describesyour position? Other 23% Faculty member or post-doc 24% Undergrad. 15% Master's 16% Doctoral 22% Students 53%
- 15. Which best characterizesthe project that motivated this visit to the Rare Book and Manuscript Library? 198 (35%) 134 (24%) 117 (21%) 49 (9%) 42 (7%) 29 (5%) 10 (2%) 9 (2%) 7 (1%) Publication (e.g. article, book) Dissertation or thesis Class assignment Gathering information, but don't have a final Administrative or work-related product Family history project Curriculum development/teaching preparation Film or video project in mind Other RESPONSE COUNT (RESPONSE %) TYPE OF PROJECT
- 16. How many timeshave you used the Rare Book and Manuscript Library? 74% 16% 4% 7% First time 2-5 times 6-10 times > 10 times
- 17. How long haveyou been using archival materials? 31% 11% 26% 32% First time < one year 1-5 years > 5 years
- 18. Please indicate yourlevel of satisfaction with the facilities, services and staff of the RBML. Questionnaire items 1 (completely dissatisfied) 5 (completely satisfied) No opinion Response count Study areas 0% 0% 4% 16% 72% 8% 495 Noise level 0% 2% 6% 21% 71% 0% 502 Furniture 1% 1% 7% 18% 71% 2% 498 Physical Access to the building 0% 1% 4% 18% 70% 7% 489 Lighting 1% 3% 8% 21% 67% 0% 504 Informational/ navigational signs 0% 2% 8% 20% 60% 9% 491 Temperature 2% 5% 12% 20% 60% 0% 502 Internet access 0% 1% 1% 7% 57% 34% 479 Hours of service 1% 5% 16% 26% 51% 1% 499 Website 0% 1% 6% 21% 50% 22% 477 Catalogs/ indexes/ findings aids 0% 2% 5% 20% 43% 30% 467 Exhibits 0% 0% 1% 10% 41% 47% 466 Reference books 0% 0% 2% 5% 28% 65% 465 Photocopying / duplication services 1% 2% 2% 5% 24% 67% 455 Microfilm and fiche viewing facilities 0% 1% 1% 3% 19% 76% 466
- 19. What else didwe learn? • Planning is key • Get “buy-in” from your entire staff • Paper and immediacy • Be prepared to make change and build on what you’ve learned • A culture of assessment
- 20. Questions? Carrie Hintz ceh2148@columbia.edu Jocelyn Wilk jkw19@columbia.edu
Editor's Notes
- #11 So, how did we implement this assessment project? The nuts and bolts were actually fairly simple – our slightly adjusted Archival Metrics Researcher Questionnaire, a photocopier, luridly colored paper, a collection box, and a Survey Monkey account. But it also required an attention to detail and “buy-in” from the entire RBML staff.
- #12 As you might know, the Archival Metrics Researcher Questionnaire is divided into six distinct sections: Use of the repository (RBML), Staff, Services and Facilities (the longest section), Feedback on Your Visit, Background Information, and General Feedback. There are a total of 22 questions asked, the majority of which are just check boxes, though there are also a few text boxes throughout the form to allow for more specific feedback in all but one of the sections. Printed out it takes up six pages (3 pages double sided) and we were instructed by our assessment librarian to print it out on extremely bright paper to make sure it got noticed by patrons and so we could quickly identify it when sitting on research desks. We also consecutively numbered the forms we printed up, so we could easily track the rate of return. A deposit box was set up at the front of our reading room for completed forms and was emptied regularly with completed surveys stored securely, in number order, until the info could be entered into electronic version of the form in Survey Monkey. Our survey period was for one year (September 1, 2011 to August 30, 2012). There are so many different cycles of researchers throughout the year at the RBML that we figured a full year of feedback would give us a sufficient baseline for any future assessment efforts. We anticipate that any future assessment projects will not be conducted over such long a period of time. Our procedure was to hand out surveys to any new researcher who came to our reading room or to any returning researchers who had not been here since we started handing out the surveys that September. The surveys were given out, with an explanation, when patrons were given their first box or volume of material. Researchers only filled out the survey once, even if they were repeat visitors. It was up to them if they completed it on their first day or at the end of their research period. If a survey was distributed to a patron but left incomplete on a vacated reading room desk, it was discarded. Finally, only those surveys of people 18 years or older were counted to ensure IRB standards were met . Most were enthusiastic about filling it out and returning it to the collection box at the front of the room; a few even mailed the form back to us! One of our biggest challenges was making sure surveys were actually distributed as necessary. This was especially difficult on very busy days and there were definitely inconsistencies in the distribution of the forms depending on who was staffing the Reading Room. But our procedure was effective enough. Handing out these paper surveys yielded an incredibly high response rate. The survey was distributed to a sample of 910 researchers (out of 1688 unique visitors that year) and completed by 566, an overall response rate of 62%. Once a sufficient number of surveys were collected, staff and student workers began entering the data into the corresponding electronic form in Survey Monkey so, in the end, statistics and results could be more easily interpreted. Poor handwriting made some comments unreadable, but on the whole we managed to transfer the info and text without too much difficulty. It was just a time consuming process.
- #14 The RBML collections are used by a broad range of patrons from across the university and the general public. As you can see on this slide, approximately 1/3 of patrons are affiliated with Columbia, while 2/3 are not.
- #15 The majority of patrons are undergraduate and graduate students (53%)
- #16 Of these students, about 45% were working on dissertations, theses, and class assignments .
- #17 Patrons are newer than commonly thought: 74% were first-time users of the RBML and 68% have been using archival materials for five years or less.
- #18 Overall, a substantial number of researchers (98%) indicated that they were highly satisfied with the facilities, discovery tools, services and staff of the RBML.
- #19 Key areas for improvement are increasing hours and upgrading the overall environment in the reading rooms.
- #20 This assessment was a good, first step in implementing an iterative assessment plan that can help us better understand RBML patrons, their needs and gauge the effectiveness of our collections and services. The survey findings verified anecdotal evidence about the shortcomings of the reading areas, as well as the highly favorable way in which patrons view the staff that works with them at all levels. The high number of non-Columbia researchers communicates the value of archives as a community outreach mechanism for the university, just as the high number of researchers using the archives for the first time speaks to the learning mission of the university. So, besides these useful statistics, what else did we learn from this process? First of all, Planning is key. A seemingly simple idea but really important to remember. You need to know what you are trying to accomplish and how you want to accomplish it before embarking on a large scale endeavor of any kind, but especially something complex like a department wide assessment project. The background reading and the information needs brainstorming we did as a group were vital to coming up with an effective strategy. We also learned that getting “Buy-In” from your entire staff is important. Whether big or small, you need to make sure your staff is all on board with all aspects of your project. Everyone needs to be invested to make it a successful venture. This idea was especially apparent when we encountered strong resistance from staff about implementing the Toolkit’s student questionnaire we anticipated using for class sessions. Although we made efforts to reach out to this staff prior to implementation, concerns did not surface until the eleventh hour, effectively forcing us to abandon this part of the project at the last minute. Thirdly, Paper and Immediacy Although online surveys are friendly to trees, the fact is people are not as likely to respond to an email link as they are to a piece of brightly colored paper sitting on their desk. Having an online version of the form was helpful for data entry and generating statistics and results more easily, but I don’t think we would have gotten as high a rate of return if we just provided each patron with a link to the survey via email. Also, if a patron did not have an email account (which does sometimes still happen) we would not have been able to get their feedback. Fourth, be prepared to make change and build on what you’ve learned Our findings and implementation of the project were well received by both our director and the library’s upper level administration, but once it was over and the report created, there was not much done with the information gathered. We’ve been able to change some patterns of staff allotment and have increased outreach to undergraduate students through class offerings by curatorial staff, but, on the whole, the needs clearly articulated by our patrons (e.g., longer hours, environmental issues) have not really changed. Some, such as environmental and furniture issues, are just not easily remedied, but it’s become obvious that if the department wants to act as fully as possible on the results of this survey, our administrators and staff have to be more than just open to change, they have to be willing to accept some inconvenient truths and be prepared to act on them. Lastly, and probably most importantly, this project has led to a formal culture of assessment finally being instituted at the RBML which continues to influence how we go about our work. This culture of assessment has undoubtedly played a role in our staff’s acceptance and use of standardized tools such as Aeon (providing us us with far more accurate data concerning reading room visitors, visitor demographics and collection use) and Desk Tracker (a system used throughout the Columbia University Libraries which we use to track non-reading room reference interactions and class sessions). These tools, while imperfect, are allowing our statistics to become more and more standardized, giving us the numbers the Library administration wants to see and allowing us to methodically change the way we think about and serve our patron base. Whether these new tools obviate the need for formal assessment surveys like the one we used in 2011-2012 still is to be decided. But they will most certainly influence what kind of assessment will be necessary as we move forward. In conclusion, an extraordinary amount of work and support, both from within our unit and from others within the Library, was required to see this assessment project through to its successful conclusion. But in spite of the challenges of tackling such a large project, it was ultimately a very worthwhile exercise and has given us a much clearer sense of who our patrons are and how best we can meet their needs.
- #21 Thank you