2007 Ala


Published on

Presentation given for RSS RUSA at the 2007 ALA Annual Conference on the transcript analysis project at Binghamton University Libraries.

Published in: Education
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • 2007 Ala

    1. 1. Evaluating the Quality of Instant Messaging Reference Service through Transcript Analysis 13th Annual Reference Research Forum Sunday, June 24th Sarah Maximiek [email_address] Erin Rushton [email_address] . edu Binghamton University
    2. 2. Binghamton University <ul><li>FTE 14,000 + 769 faculty </li></ul><ul><li>Two libraries: Bartle Library (Humanities, Social Sciences); Science Library (Sciences, Psychology, Nursing) </li></ul><ul><li>Undergraduate through Doctorate </li></ul>
    3. 3. Background <ul><li>Fall 2003 – Launched virtual reference service using QuestionPoint </li></ul><ul><li>Fall 2004 – Migrated to Docutek VRLPlus </li></ul><ul><li>March 2005 – Digital Reference Committee (DRC) formed </li></ul><ul><li>June 2005 – IM pilot service launched using Trillian </li></ul>
    4. 4. Background <ul><li>6 IM accounts: 2 usernames across 3 services (Yahoo, MSN, AOL) </li></ul><ul><ul><li>BuMainLib: Bartle Library </li></ul></ul><ul><ul><li>BuSciLib: Science Library </li></ul></ul><ul><li>Service mostly staffed at Reference service point </li></ul><ul><li>Service staffed either at 1 or 2 libraries </li></ul><ul><li>Staffed by librarians, staff, and grad students </li></ul>
    5. 5. Transcript Analysis Objectives <ul><li>Evaluate quality of the service and recommend improvements </li></ul><ul><li>Produce data that describes the use and users of the service </li></ul><ul><li>Recommend changes for library services, web design and collections based on needs of virtual users </li></ul>
    6. 6. Methodology <ul><li>Reviewed literature to determine what qualitative and quantitative factors to measure and evaluate </li></ul><ul><li>Printed 284 transcripts from June 2005 to June 2006 </li></ul><ul><li>Data from transcripts was collected on an Access form </li></ul>
    7. 7. Methodology <ul><li>Transcripts except simple catalogue look-ups and directional questions were double analyzed </li></ul><ul><li>Coding key help avoid ambiguity and data was cross-checked by the DRC </li></ul><ul><li>DRC analyzed data </li></ul>
    8. 8. Demographics <ul><li>31% of users were identified as students </li></ul>
    9. 9. Session Length
    10. 10. Sessions by Day
    11. 11. Sessions by Day and Time <ul><li>A.M. P.M. </li></ul>
    12. 12. Sessions by Month
    13. 13. Types of Reference Questions *More than one question type could be selected per transcript
    14. 14. Instructional Questions <ul><li>23% of questions demonstrated instruction </li></ul><ul><li>Instruction challenges: </li></ul><ul><ul><li>No co-browsing </li></ul></ul><ul><ul><li>Multitasking multiple reference services (in-person, phone, email, and IM) </li></ul></ul><ul><ul><li>Sometimes easier to just give the patron the answer </li></ul></ul><ul><li>Tutorials and guides may help when its not possible to provide instruction </li></ul>For further discussion see: Stephanie Graves and Christina M. Desai. Instruction via chat reference: does co-browse help? Reference Services Review, Vol. 34 No. 3, 2006, pp. 340-357
    15. 15. Research/Quick Reference <ul><li>22% research/quick reference </li></ul><ul><li>Patrons feel comfortable asking in-depth research questions </li></ul><ul><li>Need time, staffing, resources and expertise to answer research questions through IM </li></ul>
    16. 16. Resources Used to Answer Question *More than one question type could be selected per transcript
    17. 17. RUSA Guidelines for Behavioral Performance <ul><li>friendly greeting </li></ul><ul><li>cordial, and encouraging </li></ul><ul><li>uses a tone of voice and/or written language appropriate </li></ul><ul><li>maintains or re-establishes &quot;word contact&quot; </li></ul><ul><li>allows the patrons to state fully their information need </li></ul><ul><li>rephrases the question or request and asks for confirmation </li></ul><ul><li>uses open-ended questioning techniques </li></ul>finds out what patrons have already tried, and encourages patrons to contribute ideas explains the search strategy and sequence to the user
    18. 18. Correctness and Completeness Not complete : incomplete reference interview OR no referral/follow-up Not correct : used for incorrect answers Model adapted from “Evaluating the Quality of a Chat Service. Julie Arnold. Neal Kaske. portal: Libraries and the Academy 5.2 (2005) 177-193
    19. 19. Next Steps <ul><li>Further analysis to determine: </li></ul><ul><ul><li>Staffing levels </li></ul></ul><ul><ul><li>FAQ’s </li></ul></ul><ul><ul><li>Quality of service especially when provided by grad students </li></ul></ul><ul><ul><li>Referrals </li></ul></ul><ul><ul><li>Improvements to RUSA Guidelines for Behavioral Performance </li></ul></ul><ul><li>Recommendations to Head of Research, Information & Instructional Services </li></ul><ul><li>User satisfaction </li></ul><ul><li>Solicit feedback from colleagues </li></ul><ul><li>Provide additional/ongoing training </li></ul><ul><li>Revise transcript analysis strategy and continue analyzing transcripts </li></ul>
    20. 20. Findings <ul><li>76% of sessions answered in under 1 minute </li></ul><ul><li>Mean session length was 1h and 9 minutes, mode was 2 minutes </li></ul><ul><li>Over 85% of questions were answered correctly </li></ul><ul><li>IM traffic patterns mirrors in-person reference traffic </li></ul><ul><li>Web navigation questions are most common </li></ul><ul><li>Variety of questions from basic reference to in-depth research questions </li></ul><ul><li>RUSA Behaviors were demonstrated </li></ul>
    21. 21. <ul><li>Bibliography </li></ul><ul><li>Arnold, Julie, and Neal K. Kaske. “Evaluating the Quality of a Chat Service.” portal: Libraries and the Academy . 5.2 (2005): 177-193. 22 November 2006 <http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v005/5.2arnold.html> </li></ul><ul><li>Diamond, and Pease. “Digital reference: a case study of question types in an academic library.” Reference Services Review . 29.3 (2001): 210-219. 22 November 2006 <http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/EmeraldFullTextArticle/Articles/2400290305.html>. </li></ul><ul><li>Graves, Stephanie J., and Christina M. Desai. “Instruction Via Chat Reference: Does Co-browse Help?” Reference Services Review . 34.3 (2006): 340 -357. 1 June 2007 <http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/EmeraldFullTextArticle/Articles/2400340303.html>. </li></ul><ul><li>Houlson, Van, Kate McCready, and Carla Stienberg Pfahl. “A Window into Our Patron's Needs: Analyzing Data from Chat Transcripts.” Internet Reference Services Quarterly . 11.4 (2006): 19 - 39. </li></ul><ul><li>Luo, Lili. “Chat Reference Competencies: Identification from a Literature Review and Librarian Interviews.” Reference Services Review . 35.2 (2007): 195-209. </li></ul><ul><li>McClure, Charles, Lankes, R. David, Gross, Melissa, and Choltco-Devlin, Beverly. Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures. . 22 November 2006 <http://eric.ed.gov/ERICWebPortal/Home.portal?_nfpb=true&_pageLabel=RecordDetails&ERICExtSearch_SearchValue_0=ED472588&ERICExtSearch_SearchType_0=eric_accno&objectId=0900000b8017a71e>. </li></ul><ul><li>Paster, amy, and Kathy Fescemyer. “Assessing Reference: Using the Wisconsin-Ohio Reference Evaluation Program in an Academic Science Library.” Issues in Science and Technology Librarianship . (2006). </li></ul><ul><li>Sears, JoAnn. “Chat Reference Service: An Analysis of one Semester's Data.” Issues in Science and Technology Librarianship . (2001). 22 November 2006 <http://scholar.google.com/scholar?hl=en&lr=&q=%22chat+reference+service+an+analysis%22&btnG=Search>. </li></ul><ul><li>Smyth, Joanne. “Virtual Reference Transcript Analysis: A Few Models.” Searcher . 11.3 (2003). 22 November 2006 <http://referenc.lib.binghamton.edu:2510/ehost/detail?vid=2&hid=1&sid=a85713a4-9ac9-4d51-84cf-9f6420ea2d7a%40sessionmgr2>. </li></ul><ul><li>Ward, David. “Measuring the Completeness of Reference Transactions in Online Chats: Results of an Unobtrusive Study.” Reference and User Services Quarterly . 44.1 (2004): 46-56. </li></ul>
    22. 22. Sample Transcript <ul><li>[09:59] hello </li></ul><ul><li>[09:59] i have a quick question </li></ul><ul><li>[09:59] would like to know where i can find the JSTOR database </li></ul><ul><li>[09:59] BUmainlib: ok what can i hlep you with </li></ul><ul><li>[09:59] BUmainlib: oh ok </li></ul><ul><li>[10:00] thx alot </li></ul><ul><li>[10:00] BUmainlib: go to METALINK on the library homepage </li></ul><ul><li>[10:00] BUmainlib: and in the find database search box type in jstor </li></ul><ul><li>[10:00] BUmainlib: the link will appear </li></ul><ul><li>[10:01] oooo </li></ul><ul><li>[10:01] BUmainlib: let me know if you find it </li></ul><ul><li>[10:01] ooo lol that was easy </li></ul><ul><li>[10:01] i found it!!:) </li></ul><ul><li>[10:01] BUmainlib: oh good! </li></ul><ul><li>[10:01] BUmainlib: any other questions for me? </li></ul><ul><li>[10:01] no thats it </li></ul><ul><li>[10:01] thx so much </li></ul><ul><li>[10:01] BUmainlib: ok have a good morning then! you are welcome! </li></ul><ul><li>[10:01] BUmainlib: bye </li></ul><ul><li>[10:02] u 2 thanx again </li></ul>
    23. 23. Sample Transcript <ul><li>[15:11] hello </li></ul><ul><li>[15:11] hello </li></ul><ul><li>[15:12] can you help me i am trying to locate information on &quot; normalizing constant factors&quot; in Bio Stats? </li></ul><ul><li>[15:13] What type of information do you need? Is Bio Stats a textbook? </li></ul><ul><li>[15:13] its a class biostatistics , i need to see how it is expressed mathamaticly </li></ul><ul><li>[15:15] So you just need the formula? Let me look a few places </li></ul><ul><li>[15:15] outstanding :-) </li></ul><ul><li>[15:17] I found this article in Wikipedia. Wikipedia is always 100 percent accurate, but on something like this, you can probably trust it. (Link: http://en.wikipedia.org/wiki/Normalization_factor)http://en.wikipedia.org/wiki/Normalization_factor </li></ul><ul><li>[15:20] Is that what you were looking for? </li></ul><ul><li>[15:20] yes the formula </li></ul><ul><li>[15:20] Any other questions? </li></ul><ul><li>[15:20] :-) thats it </li></ul><ul><li>[15:20] why that wasnt hard enough lol </li></ul><ul><li>[15:21] have a good day. bye </li></ul>
    24. 24. <ul><li>[12:29] hello, is this the right screen name for help with psychology research? </li></ul><ul><li>[12:30] yes - what are you working on? </li></ul><ul><li>[12:30] i am working on a research paper focusing on the effects noise has on memory </li></ul><ul><li>[12:31] where have you started your research? </li></ul><ul><li>[12:32] i looked online with psycArticles and psycINFO </li></ul><ul><li>[12:32] do you need articles, books, either, both? </li></ul><ul><li>[12:32] im mainly looking for articles </li></ul><ul><li>[12:32] ok </li></ul><ul><li>[12:33] I think you were on the right track with PsycINFO </li></ul><ul><li>[12:33] I'm giving it a try... </li></ul><ul><li>[12:33] ok...thank you </li></ul><ul><li>[12:34] i found that a lot of the articles i found were very loosely related to the topic, they didnt have enough information to be one of my actual references </li></ul><ul><li>[12:34] are you looking for studies on humans or animals? </li></ul><ul><li>[12:34] humans </li></ul><ul><li>[12:34] ok </li></ul><ul><li>[12:38] I'm trying a search of the descriptors of (auditory stimulation or distraction) and memory </li></ul><ul><li>[12:39] I'd like to email you a couple of results to know if this is on the right track </li></ul><ul><li>[12:39] ok my email is XXXXX </li></ul><ul><li>[12:39] ok, hang on </li></ul><ul><li>[12:39] unfortunately i have to go to a class but i will check the articles </li></ul><ul><li>[12:39] ok </li></ul><ul><li>[12:40] thank you very much </li></ul>