Judith Schwartz/QuestionPoint Use Study at Hunter College


Published on

This presentation is a "QuestionPoint Use Study" for Hunter college conducted during the spring of 2011. My research includes a statistical analysis of 500 virtual reference transcripts during a three month period of online transactions at Hunter College's Wexler Library, New York.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Judith Schwartz/QuestionPoint Use Study at Hunter College

  1. 1. An Explorative Evaluationof Chat Reference ServicesJudith SchwartzSummer 2011
  2. 2. IntroductionThe focus of this research is mainly onchat-based reference in academic libraries. • Online chat reference is also referred to as: “virtual, real-time, synchronous, and live chat” (Johnson, 2004).
  3. 3. IntroductionGoalsThe purpose is to give a more accurate accounting ofthe effectiveness of chat reference at Hunter. • Evaluating 3 months of Ask-a-Librarian transcripts at Hunter: • will identify types of questions students are asking • will lead to additional subject specific library instruction • Find ways to increase students awareness of the service • Evaluate areas of the Hunter library website to be improved
  4. 4. Motivational BackgroundHunter College Results/May 2011 CUNY Cooperative Answering percentage increased to 76% in 2010 2008 • 16% 2009 • 19% 2010 • 76%
  5. 5. Literature ReviewThe literature focuses on similar themes evolvedfrom library research studies including: • Awareness and access • Barriers • Types of reference questions • Privacy concerns
  6. 6. Literature ReviewAwareness and access: exploration of ways toincrease chat patron usage in academic libraries.• Marketing, outreach, and chat widgets • Findings revealed that the majority of participants were uninformed about chat, telephone, and email (Naylor et al., 2008) • Strategic placement of chat widgets is crucial (Bedwell, 2008) • Contact pages • OPACs • LibGuides • Databases
  7. 7. Literature ReviewBarriers: Exploration of issues. • why the numbers are down • questions whether it is a viable reference tool “Reference librarians have responded to chat reference in various ways, from feeling it is the most exciting development in reference work, to being overrated” (Lee, 2004).
  8. 8. Literature ReviewBarriers • Virtual interaction has been described as a “fleeting encounter.” • No patron/reference librarian relationship development (Radford, 2006). • Patrons are unaware of available chat services (Johnson, 2004). • Patrons terminate sessions due to impatience (Lee, 2004). • Poor technology to accomplish the reference interview (Lee, 2004). • The same question takes about half as long to answer over the phone as it does in chat (Coffman & Arret, 2004).
  9. 9. Literature ReviewTypes of reference questions • Auburn University, AL. reports one-third of its questions have been “specific search questions,” a little less than a quarter are ready-reference, and one was a “research question.” All the rest are directional, policy, instructional or technical (Janes, 2002). • Chat reference is best suited for directional or policy questions that can be answered relatively quickly and with few text characters” (Naylor et al., 2008).
  10. 10. Literature ReviewUse of closed and clarifying questions• Students are seeking information related to imposed queriesassigned by their instructor (Gross, 1999). • Librarians should ask closed or clarifying questions when appropriate. • Librarians should type twice as many characters as the patron. • At the end of session, ask: “does this completely answer your question?”(Radford et al., 2011).
  11. 11. Literature ReviewMotivation • Librarians believe that users with detailed research questions are poorly served by digital reference (Johnson, 2004). • Studies have concluded that users use chat because it’s convenient, not that they are in a hurry (Radford et al., 2011).
  12. 12. Literature ReviewDuring chat transactions, recommendations for recreating a moreinterpersonal experience would be: (Radford 2006) • going off of scripts • mirroring of emoticons • using informal languageReasons to stay with virtual reference are: (Lee, 2004)• keep current with innovative technology• keep up with the needs of students• remain current with other academic institutions
  13. 13. Literature ReviewPrivacy concernsA portion of chat sessions need to be followed upby email. Anonymous users = no follow-up• Reasons for anonymity are: • In an MIT study, they revealed that students felt chat reference was invasive, and some were concerned about asking “stupid questions” (Naylor et al., 2008). • “Patrons are afraid they will be negatively evaluated by a librarian, appearing unprofessional, or having reference transcripts revealed to professors” (Connaway et al., 2007). • Librarians are nervous that transcripts will be used unfairly for performance evaluation (Johnson, 2004).
  14. 14. Literature ReviewSeeking Synchronicity results can help an academic institutionpromote their Virtual Reference Services by: • Increase marketing to make it more visible • Highlight the convenience of the service (available anywhere with web accessibility 24/7) • Increase the availability of instruction and tutorials • Integrate VRS into library use instruction programs and courseware • Improve librarian’s skills in fostering virtual relationships • Involve teachers in promoting and recommending the service to students • Publicize VRS as a private and confidential serviceOCLC and Rutgers, conducted studies of virtual reference services (VRS)through “Seeking Synchronicity,” a two-year project funded by an IMLSgrant.(Connaway et al., 2007)
  15. 15. Data Collection TechniquesQuantitative research • Log Data Analysis drawing a sample of the MONTHS March = Midterms chat reference logs from a three-month period May = Widget June = Finals March, May, and June 2011 • Online survey 10-15 questions (see appendix) • Demographic info gender, education level, and language ability • Awareness of chat reference services • Location of usage and time of day • Preferable methods would patrons consider SMS (text) over chat?
  16. 16. Method/MeasureThe evaluation of services includes operational definitionsfor example: Was the question answered or did it need a follow up or referral? Categories include: • Efficiency • Privacy issues • Access • Types of questions • Completion of service • Subject specificity
  17. 17. Method/Measure• Efficiency is measured by session waiting and completion time: • waiting time starts when the query is submitted • ends when the librarian responds to it• Privacy issues are categorized as: • patron gave their email address • logged on anonymously• Access is categorized as how the patron accessed chat reference to ask their question: • widget • no widget
  18. 18. Method/Measure• Type of questions are categorized into three types: • reference, ready-reference, and directional • complexity of question usually defines the need for a reference interview• Completion of service refers to questions answered: • session ended abruptly • any follow-up required • referral necessary• Subject specific data was collected in the following areas: • Social work, and Health Sciences • Style guides: students asked for help with their citations or Refworks questions.
  19. 19. Procedure for Gathering Data
  20. 20. Procedure for Gathering Data
  21. 21. Results: Hunter College May 2011 • Reference - 55% • Ready Reference - 20% • Directional - 25% Completion of service • Answered - 57% • Ended - 18% • Follow-up - 19% • Referred - 6%
  22. 22. Results: Hunter College 2011- 3 months Depth of the question is an interpretation by the researcher as to whether the question is simple or complex. • A more complex question usually leads to a reference interview.
  23. 23. Results: Hunter College March 2011
  24. 24. Limitation and Future WorkMore in-depth analysisA complete year should be analyzed for a more accurate reading. • The widget was first introduced on May 15, 2011. Additional months of study would give a better indication of whether there was an increase in users. • Reviewing Directional Questions and answers to identify where improvements are necessary. A pattern of similar questions with weak answers are: - CLICs - Interlibrary loans - Metro passes - Reserves - Policies page procedures
  25. 25. Limitation and Future WorkThoughts on the future of reference services• Explore additional areas of VRS such as SMS, video chat, and co-browsing. “New media—chat, texting, mobile devices, Twitter, etc., may not be ideal in answering complex questions, but they will often be the media that our patrons prefer”(Frantz & Westra, 2010)
  26. 26. ConclusionExpectations from results specific to Hunter College VRS • Marketing: promote chat reference services at Hunter more aggressively • Accessibility: Strategic placement of widgets to additional pages • Identify subject areas where additional library skills training is needed • Promote LibGuides • Make them more prominent • Add chat widgets to LibGuides • Chat Privacy Policy Page • Enhancing web wayfinding systems and policy page • This may reduce directional questions and enable librarians to help patrons more efficiently • identify barriers and explore ways to improve services
  27. 27. Appendix - SurveyOnline Chat Reference Usage Survey QuestionsPlease fill out this survey about Reference Services at Hunter College library and we will enter your name in a drawing to win an iPod touch!1. Which one of the following describes you?___Undergraduate Student___Graduate Student___University Faculty___Alumni___Other2. Gender___Male___Female3. Is English your first language?___Yes___NoIf not, what is it? ______________4. Which of the following library reference services have you used at Hunter College?___Asked a question in person at the reference desk___Face to Face Consultation, (made an appointment with a reference librarian)___Phoned the reference desk and spoke with a librarian.___Emailed a question to the reference librarian and waited for a reply.___Online Chat Reference 24/7 (chatted with a reference librarian online about your question)5. Are you aware that your campus library offers online Chat Reference?___Yes___No
  28. 28. Survey6. If you are aware of Online Chat reference at Hunter College, where did you learn about it?___Library website___From a classmate, friend or relative___Through an electronic discussion list/email announcement___A library instruction classOther (please specify) ______________7. What location would you be using Online Chat Reference__While in the Hunter Library__From Home8. During which one of the time slots would you most likely use Online Chat reference__midnight – 8am__8am-noon__5pm-9pm__9pm-midnight__never9. How do you rate online chat? 1 is worst and 5 is best.1 2 3 4 510. Besides Hunter, what other city college libraries have you accessed online?__Baruch__CUNY__Queens__BMCC__Brooklyn College__NoneOther (please specify) ______________
  29. 29. Survey11. What would your first choice of reference be at Hunter College Library?__Online chat reference__Face to Face__Phone__email12. Please rate your reference experience at Hunter College. Do you feel your reference questions have been answered effectively? 1 isworst and 5 is best.1 2 3 4 5 n/aOnline chat referenceFace to FacePhoneemail13. Do you think Online Chat Reference can answer your reference questions equally as well as Face-to -Face reference? Please explain.____________________________________________________________________________________________________________________________________________14. Would you consider text messages over chat reference?___Yes___No15. In your opinion, as technology makes more information accessible, students will need LESS human help in doing research for theirprojects and papers.___True___False
  30. 30. ContactInfo@judithstudio.comJudith Schwartz, New Yorkhttp://www.linkedin.com/in/judithschwartz