• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Getting the Right Fit: Tailoring Assessment Strategies for your Library: A Collective Insight Event
 

Getting the Right Fit: Tailoring Assessment Strategies for your Library: A Collective Insight Event

on

  • 37 views

Webinar keynote and workshop at Brandeis University, April 22, 2014, Waltham, MA.

Webinar keynote and workshop at Brandeis University, April 22, 2014, Waltham, MA.

Statistics

Views

Total Views
37
Views on SlideShare
37
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Getting the Right Fit: Tailoring Assessment Strategies for your Library: A Collective Insight Event Getting the Right Fit: Tailoring Assessment Strategies for your Library: A Collective Insight Event Presentation Transcript

    • Getting the Right Fit: Tailoring Assessment Strategies for Your Library April 22nd, 2014 in partnership withPresented by
    • Welcome virtual participants! World map by www.freeworldmaps.net
    • Getting the Right Fit: Tailoring Assessment Strategies for Your Library April 22nd, 2014 Presented by in partnership with Welcome and Introductions Rob Favini Member Relations Liaison, OCLC Susan Stearns Executive Director, Boston Library Consortium
    • Getting the Right Fit: Tailoring Assessment Strategies for Your Library April 22nd, 2014 Presented by in partnership with Opening Keynote Lynn Silipigni Connaway, Ph.D. Senior Research Scientist, OCLC Chair of Excellence, Departmento de Biblioteconomía y Documentación, Universidad Carlos III de Madrid @LynnConnaway connawal@oclc.org Marie L. Radford, Ph.D. Chair, Department of Library & Information Science, Rutgers University, NJ @MarieLRadford mradford@rutgers.edu
    • Some Initial Questions • What is your definition of assessment? • What comes to mind when you hear the term “assessment”? • What benefits do you see for assessment? • What are your concerns? 5
    • Interpreting Analyzing Collecting 6 Assessment Defined Process of… • Defining • Selecting • Designing • Collecting • Analyzing • Interpreting • Using information to increase service/program effectiveness
    • Why Assessment? • Answers questions: • What do users/stakeholders want & need? • How can services/programs better meet needs? • Is what we do working? • Could we do better? • What are problem areas? • Traditional stats don’t tell whole story 7
    • “Librarians are increasingly called upon to document and articulate the value of academic and research libraries and their contribution to institutional mission and goals.” (ACRL, 2010, p. 6)
    • 9 Formal vs. Informal Assessment • Formal Assessment • Data driven • Evidence-based • Accepted methods • Recognized as rigorous • Informal Assessment • Anecdotes & casual observation • Used to be norm • No longer acceptable
    • 10 Outcomes Assessment Basics • Outcomes: “The ways in which library users are changed as a result of their contact with the library’s resources and programs” (ALA, 1998). • “Libraries cannot demonstrate institutional value to maximum effect until they define outcomes of institutional relevance and then measure the degree to which they attain them” (Kaufman & Watstein, 2008, p. 227).
    • 11 Outputs & Inputs • Outputs • Quantify the work done • Don’t relate factors to overall effectiveness • Inputs • Raw materials • Measured against standards • Insufficient for overall assessment
    • 12 Principles for Applying Outcomes Assessment • Center on users • Assess changes in service/resources use • Relate to inputs - identify “best practices” • Use variety of methods to corroborate conclusions • Choose small number of outcomes • Need not address every aspect of service • Adopt continuous process
    • 13 Examples of Outcomes • User matches information need to information resources • User can organize an effective search strategy • User effectively searches online catalog & retrieves relevant resources • User can find appropriate resources
    • 14 Steps in Assessment Process • Why? Identify purpose • Who? Identify team • How? Choose model/approach/method • Commit • Training/planning
    • 15 Assessment Tools • Survey Research • Interviews • Focus Group Interviews • Structured Observations • Ethnographic Research • Analytics
    • Analysis • Collection of data affects analysis of data • Ongoing process • Feeds back into research design • Theory, model, or hypothesis must grow from data analysis 16
    • Finding the Right Fit “One size fits none!” (Lynn’s Mom) 17
    • Finding the Right Fit – What to Consider? • What do you already know? (Previous assessments) • What have others done? (Literature review) • What you want to know? (Problems/questions)
    • Finding the Right Fit - Local Considerations • Available resources • Budget • Human resources • In house? • Should you hire a consultant? • Faculty resources? (university/college) • Other institutional resources • Parent institution needs/demands 19
    • 20 What We Know About Assessment • Ongoing process to understand & improve service • Librarians are busy with day-to-day work & assessment can become another burden • Assessment is a reality, a challenge we must meet!
    • Get the Right Fit! 21
    • Lynn Silipigni Connaway, Ph.D. connawal@oclc.org Marie L. Radford, Ph.D. mradford@rutgers.edu Discussion and Questions 22
    • References ACRL. (2010). Value of academic libraries: A comprehensive research review and report. Chicago: Association of College and Research Libraries. ALA/ACRL. (1998). Task force on academic library outcomes assessment report. Retrieved from http://www.ala.org/acrl/publications/whitepapers/taskforceacademic Kaufman, P., & Watstein, S. B. (2008). Library value (Return on Investment, ROI) and the challenge of placing a value on public services. Reference Services Review, 36(3), 226- 231. 23
    • Recommended Readings Besara, R. (2013, June 4). See results: Tips and tools for visualizing library data. Presented at the Qualitative and Quantitative Methods in Libraries International Conference (QQML 2013). Connaway, L. S. (1996). Focus group interviews: A data collection methodology. Library Administration & Management, 10(4), 231-239. Connaway, L. S., Johnson, D. W., & Searing, S. (1997). Online catalogs from the users’ perspective: The use of focus group interviews. College and Research Libraries, 58(5), 403-420. Connaway, L. S., Lanclos, D., White, D. S., Le Cornu, A., & Hood, E. M. (2012). User-centered decision making: A new model for developing academic library services and systems. IFLA 2012 Conference Proceedings, August 11-17, Helsinki, Finland. Connaway, L. S., & Powell, R. R. (2010). Basic research methods for librarians (5th ed.). Westport, CN: Libraries Unlimited. Connaway, L. S., & Radford, M. L. (2011). Seeking Synchronicity: Revelations and recommendations for virtual reference. Dublin, OH: OCLC Research. Retrieved from http://www.oclc.org/reports/synchronicity/full.pdf Connaway, L. S., & Snyder, C. (2005). Transaction log analyses of electronic book (eBook) usage. Against the Grain, 17(1), 85-89. Retrieved from http://www.oclc.org/research/publications/archive/2005/connaway-snyder-atg.pdf Connaway, L. S., & Wakeling, S. (2012). To use or not to use Worldcat.org: An international perspective from different user groups. OCLC Internal Report. 24
    • Recommended Readings Consiglio, D., Furlong, K., & Holbert, G. (2012, February). Setting the state for success: A discussion of insights from the MISO survey. Presentation at the ELI Annual Meeting, Austin, TX. Consiglio, D. M., Furlong, K., & Holbert, G. (2013, June 6). Assessing the quality & effectiveness of library services using the MISO survey. Presented at the Qualitative and Quantitative Methods in Libraries International Conference (QQML 2013). Dervin, B., Connaway, L. S., & Prabha, C. (2003-2006). Sense-making the information confluence: The whys and hows of college and university user satisficing of information needs. Funded by the Institute of Museum and Library Services (IMLS). Retrieved from http://www.oclc.org/research/activities/past/orprojects/imls/default.htm The ERIAL Project. (2011). So you want to do anthropology in your library? Or: A practical guide to ethnographic research in academic libraries. Chicago: Andrew Asher and Susan Miller. Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51(4), 327-358. Foster, N. F., & Gibbons, S. (2007). Studying students: The undergraduate research project at the University of Rochester. Chicago: Association of College and Research Libraries. Geertz, C. (1973). The interpretation of cultures: Selected essays. New York: Basic Books. Harvard University Library. (2010). Library analytics toolkit. Retrieved from https://osc.hul.harvard.edu/liblab/proj/library-analytics-toolkit Hernon, P., & Altman, E. (1998). Assessing service quality: Satisfying the expectations of library customers. Chicago: ALA. 25
    • Recommended Readings Janes, J. (1999). On research survey construction. Library Hi Tech, 17(3), 321-325. Khoo, M., Rozaklis, L., & Hall, C. (2012). A survey of the use of ethnographic methods in the study of libraries and library users. Library and Information Science Research, 34(2), 82-91. Maciel, M. L. (2013, June 4). Identifying our customers’ needs and expectations: A data driven approach and analysis through the use of LibQUAL+TM (using five years’ data from LibQUAL+TM in Europe). Presented at the Qualitative and Quantitative Methods in Libraries International Conference (QQML 2013). QSR International. (2011). NVivo 9: Getting started. Retrieved from http://download.qsrinternational.com/Document/NVivo9/NVivo9-Getting-Started-Guide.pdf Radford, M. L. (2006). The critical incident technique and the qualitative evaluation of the Connecting Libraries and Schools Project. Library Trends, 54(1), 46-64. Radford, M. L. (1999). The reference encounter: Interpersonal communication in the academic library. Chicago: Association of College & Research Libraries. Radford, M. L. (1996). Communication theory applied to the reference encounter: An analysis of critical incidents. The Library Quarterly, 66(2), 123-137. Radford, M. L., Radford, G. P., Connaway, L. S., & DeAngelis, J. A.. (2011). On virtual face-work: An ethnography of communication approach to a live chat reference interaction. The Library Quarterly, 81(4), 431-453. Radford, M. L. (2008). How to conduct a focus group. Marketing Library Services, 22(1), 1-3. 26
    • Recommended Readings Rodriguez, D. A. (2011, August 23). The “Understanding Library Impacts” protocol: Demonstrating academic library contributions to student learning outcomes in the age of accountability. Paper presented at the 9th Northumbria International Conference on Performance Measurement in Libraries and Information Services. Retrieved from http://academia.edu/1041005/The_Understanding_Library_Impacts_protocol_demonstrating_acad emic_library_contributions_to_student_learning_outcomes_in_the_age_of_accountability Rodriguez, D. A., & Norberg, L. R. (2013, June 7). Assessing library impact on student learning outcomes: A case study using the Understanding Library Impacts protocol. Presented at the Qualitative and Quantitative Methods in Libraries International Conference (QQML 2013). Taylor, M. (2013, June 4). Academic library use and undergraduate engagement and persistence at a large public research university. Presented at the Qualitative and Quantitative Methods in Libraries International Conference (QQML 2013). White, D. S., & Connaway, L. S. (2011-2012). Visitors and residents: What motivates engagement with the digital information environment. Funded by JISC, OCLC, and Oxford University. Retrieved from http://www.oclc.org/research/activities/vandr/ Zweizig, D., Johnson, D. W., Robbins, J., & Bewsant, M. (1996). The tell it! manual. Chicago: ALA. 27
    • Getting the Right Fit: Tailoring Assessment Strategies for Your Library April 22nd, 2014 Presented by in partnership with Assessment for Context: Using Qualitative Methods Lisa Horowitz, Assessment Librarian Massachusetts Institute of Technology Libraries
    • Qualitative Assessment • How are new technologies and formats changing how MIT scholars find and use information? • What are the three most important things that people expect to be able to do when they come to the MIT Libraries home page? 7/9/2014
    • Photo Diary Study( & & ( ( ! ! ! # ! ' ! ! ! " && " " 6 ( ! # ! & # & " ( ( # ! ( # ' "7 ' & ! ( ( ' " ( & ' ! ! ' ' & ! ! " 8 # ' ! & ( ' * & +" "' ' " ( " ' & ! ! & ' ( ( ' ! ' 7/9/2014
    • Photo Diary Results 7/9/2014
    • Home Page Redesign Focus Groups 7/9/2014
    • Home Page Redesign Results Most important to users 7/9/2014
    • Getting the Right Fit: Tailoring Assessment Strategies for Your Library April 22nd, 2014 Presented by in partnership with User Surveys Linda Plunket, AUL, Graduate & Research Services Library Assessment Committee, Chair Boston University
    • BU’s Library Assessment Committee Dan Benedetti Brendan DeRoo David Fristrom Lisa Philpotts Linda Plunket Steve Smith Russell Sweet Sarah Struble Konstantin Starikov
    • Why Assess?
    • Some Survey Options
    • BU’s User Surveys
    • What is your purpose in using this electronic resource?
    • Impact & Changes
    • www.bu.edu/library/about/library-assessment/
    • Getting the Right Fit: Tailoring Assessment Strategies for Your Library April 22nd, 2014 Presented by in partnership with Assessing Building Use and Renovation Success with Suma G. Karen Merguerian, User Engagement Librarian Northeastern University
    • Assessing our renovations Spring 2012 • Building use survey Early fall 2012 • Renovation of level 2 into Digital Media Commons (DMC) Spring 2013 • Suma pilot, one-week snapshot of DMC Fall 2013 • Opening of renovated main floor Spring 2014 • Suma occupancy counts, building- wide
    • Level 2 was seen as experimental…
    • Assumptions: 1. That students needed high end computers with dual monitors for effective collaboration 2. That students like sitting in groups at tables in order to talk and collaborate 3. That students would appreciate cozy, casual furniture 4. That it would be easy to communicate what students wanted to architects
    • Of course, our assumptions turned out to be false
    • Why we chose Suma as a tool • To track actual, not self-reported behavior • Because card swipes measure entrances, but not occupancy • We needed more accuracy • To capture more than just numbers: – Activities – Time and date stamps – Locations http://www.lib.ncsu.edu/dli/projects/spaceassesstool
    • 3 Components of Suma 1. Data management server 2. Web application (right and below) 3. Data analysis dashboard
    • Level 2 activity snapshot • Most students were using their own devices, sometimes alone, sometimes along with library technology. • Very few students were using dual monitor workstations. • A third were studying with printed material, whiteboards, etc… Numbers do not add up to 100% since each student could have multiple simultaneous “activities”
    • More findings with Suma: Locations 0 10 20 30 40 50 60 Atrium area Window area 1 Window area2 Window area3 End of stakes by back elivator Within stack area Group study 401 Individual study Avg Occupancy Level-4
    • Large Tables
    • Preference ratio of tables over carrels at different levels of occupancy
    • We learned that… • In the DMC on level 2, most students were bringing their own computers. • On quiet floors 3 and 4 (left), tables are valued to spread out and “study along”, “alone together” not just to collaborate* *See O’Connor, Seeing duPont within Sewanee and Student Life, 2006 http://www.sewanee.edu/teaching/dupont
    • We also learned 1. Group studies and high- density labs are preferred over casual seating. 2. It’s not easy to convince architects of anything!
    • Free as in…. • Flexible • Feature rich • NCSU-supported (thank you, Jason Casden) • Requires server admin to install and maintain • Software bugs • Webkit browsers only Source: wikimedia commons, Nejmlez http://commons.wikimedia.org/wiki/File:Dutch_b eers.jpg
    • Challenges What is the seating capacity of a couch?
    • Getting the Right Fit: Tailoring Assessment Strategies for Your Library April 22nd, 2014 Presented by in partnership with Using Tableau for Data Visualization at the UMass Amherst Libraries Rachel Lewellen, Assessment Librarian University of Massachusetts Amherst
    • So Much Data! • Multiple data formats, sources, locations – Door counts (Microsoft SQL Server Database) – Aleph data including circulation, purchasing, duplication, and user demographics (Oracle) – Service desk transactions (LibAnalytics exported to Excel) – MINES proxy intercept survey (MySQL Database) – Carrels, headcounts, vendors, ARL, and more
    • Tableau Software • Tableau Software is an American computer software company headquartered in Seattle, WA, USA. It produces a family of interactive data visualization products focused on business intelligence. (Wikipedia)
    • UMass Amherst Learning Commons