Two Programs, One Service: Recursive Assessment in the Gelman Library System's IM Reference Service American Library Association Annual Conference 2008 MERS Virtual Poster Session Sarah Palacios-Wilhelm Reference & Instruction Librarian Eckles Library Elizabeth Edwards Reference/Technology Librarian Gelman Library

Two Programs, One Service: 2008 MERS Virtual Poster Session Abstract This virtual poster session presents the recursive implementation process utilized by the George Washington University’s Gelman Library System in the context of its successful IM reference service. In this presentation, librarians from two campus libraries discuss the advantages of using pilot testing in a recursive cycle to inform testing, implementation, and evaluation of services, software, training, and assessment models. This process has resulted in a climate that encourages innovation and decreases anxiety amongst service providers. Specific examples of recursive implementation are explored, as well as challenges faced throughout the process. The presentation closes with a list of lessons learned that will inform institutions that wish to follow a similar model.

This virtual poster session presents the recursive implementation process utilized by the George Washington University’s Gelman Library System in the context of its successful IM reference service. In this presentation, librarians from two campus libraries discuss the advantages of using pilot testing in a recursive cycle to inform testing, implementation, and evaluation of services, software, training, and assessment models. This process has resulted in a climate that encourages innovation and decreases anxiety amongst service providers. Specific examples of recursive implementation are explored, as well as challenges faced throughout the process. The presentation closes with a list of lessons learned that will inform institutions that wish to follow a similar model.

Two Programs, One Service: 2008 MERS Virtual Poster Session Background In 2005, the Gelman Library System pilot tested the delivery of reference services via IM. This pilot test began at Eckles Library, which serves a primarily residential and freshman population. Since its inception, this service has expanded to include Gelman Library, which serves the entire University population. During the 2006-2007 school year the IM reference service fielded 1,458 questions, and was nominated for a campus-wide Service Excellence Award. The success of this service is in no small part a result of the innovative testing and evaluation model that integrates the two campuses in a recursive cycle.

In 2005, the Gelman Library System pilot tested the delivery of reference services via IM. This pilot test began at Eckles Library, which serves a primarily residential and freshman population. Since its inception, this service has expanded to include Gelman Library, which serves the entire University population. During the 2006-2007 school year the IM reference service fielded 1,458 questions, and was nominated for a campus-wide Service Excellence Award. The success of this service is in no small part a result of the innovative testing and evaluation model that integrates the two campuses in a recursive cycle.

Two Programs, One Service: 2008 MERS Virtual Poster Session The George Washington University Located in the heart of the Foggy Bottom neighborhood of Washington, D.C. Comprised of nine schools: Columbian College of Arts & Sciences School of Medicine & Health Sciences Law School School of Engineering & Applied Science Graduate School of Education & Human Development School of Business Elliott School of International Affairs School of Public Health & Health Services College of Professional Studies High research activity according to Carnegie ratings. 17,494 students enrolled full-time in Fall 2007. 7,092 degrees conferred in 2007, including 264 Doctorates.

Located in the heart of the Foggy Bottom neighborhood of Washington, D.C.

Comprised of nine schools:

Columbian College of Arts & Sciences

School of Medicine & Health Sciences

Law School

School of Engineering & Applied Science

Graduate School of Education & Human Development

School of Business

Elliott School of International Affairs

School of Public Health & Health Services

College of Professional Studies

High research activity according to Carnegie ratings.

17,494 students enrolled full-time in Fall 2007.

7,092 degrees conferred in 2007, including 264 Doctorates.

Two Programs, One Service: 2008 MERS Virtual Poster Session The Gelman Library System Gelman Library System Gelman Library Eckles Library Virginia Campus Library

Two Programs, One Service: 2008 MERS Virtual Poster Session Gelman Library Main library for urban campus 30,000 visitors per week Reference desk staffed by 17 librarians with assistance from student workers IM service established in Fall 2006 Eckles Library Small library serving residential campus 7,000 visitors per week Single service point staffed by 2 librarians, 3 staff, and 7 students IM service piloted in 2005, established in 2005

Gelman Library

Main library for urban campus

30,000 visitors per week

Reference desk staffed by 17 librarians with assistance from student workers

IM service established in Fall 2006

Eckles Library

Small library serving residential campus

7,000 visitors per week

Single service point staffed by 2 librarians, 3 staff, and 7 students

IM service piloted in 2005, established in 2005

Two Programs, One Service: 2008 MERS Virtual Poster Session Models of Implementation The “Normal” Implementation Process The Recursive Implementation Process

The “Normal” Implementation Process

The Recursive Implementation Process

Two Programs, One Service: 2008 MERS Virtual Poster Session The "Normal" Implementation Process

Two Programs, One Service: 2008 MERS Virtual Poster Session The "Normal" Implementation Process Each phase has a defined beginning and end. Each phase occurs in order. Pilot programs lead directly to implementation. Test groups are organized as necessary, but do not persist after the implementation process is complete.

Each phase has a defined beginning and end.

Each phase occurs in order.

Pilot programs lead directly to implementation.

Test groups are organized as necessary, but do not persist after the implementation process is complete.

Two Programs, One Service: 2008 MERS Virtual Poster Session The "Normal" Implementation Process: An Example A problem is identified: patrons are eating in the library, which is in violation of library policy. A solution is proposed: create a space where patrons can eat without violating library policy. Pilot testing: a student lounge is opened, and patrons attempting to bring food into the library are directed to this new space. Implementation: the student lounge is used for a special event, drawing attention to the new space. Evaluation: the problem of patrons eating in the library has not decreased. A new solution must be devised.

A problem is identified: patrons are eating in the library, which is in violation of library policy.

A solution is proposed: create a space where patrons can eat without violating library policy.

Pilot testing: a student lounge is opened, and patrons attempting to bring food into the library are directed to this new space.

Implementation: the student lounge is used for a special event, drawing attention to the new space.

Evaluation: the problem of patrons eating in the library has not decreased. A new solution must be devised.

Two Programs, One Service: 2008 MERS Virtual Poster Session The Recursive Implementation Process

Two Programs, One Service: 2008 MERS Virtual Poster Session The Recursive Implementation Process In the recursive implementation process, testing, implementation, evaluation, and reflection are cyclical, with each step returning to the pilot setting. Problems can be identified at any location, and solutions are proposed based on previous experience, understanding of patron needs, and possible resources. Pilot testing occurs with a selected population, informed by feedback from ongoing brainstorming and evaluation from both locations. Based on success in the pilot phase, implementation expands to include both locations, with the next phase of brainstorming or pilot testing beginning at the smaller location. Evaluation is an ongoing process, incorporating both formal and informal methods of assessment.

In the recursive implementation process, testing, implementation, evaluation, and reflection are cyclical, with each step returning to the pilot setting.

Problems can be identified at any location, and solutions are proposed based on previous experience, understanding of patron needs, and possible resources.

Pilot testing occurs with a selected population, informed by feedback from ongoing brainstorming and evaluation from both locations.

Based on success in the pilot phase, implementation expands to include both locations, with the next phase of brainstorming or pilot testing beginning at the smaller location.

Evaluation is an ongoing process, incorporating both formal and informal methods of assessment.

Two Programs, One Service: 2008 MERS Virtual Poster Session The Recursive Model in Practice Pilot Testing Service Hours Staffing Models Training Assessment Platform Testing

Pilot Testing

Service Hours

Staffing Models

Training

Assessment

Platform Testing

Two Programs, One Service: 2008 MERS Virtual Poster Session The Recursive Model in Practice: Pilot Testing Experience with virtual reference collaboratively staffed with other local academic libraries resulted in hesitation about new forms of electronic reference. Librarians at Eckles, in an attempt to provide enhanced service for their small, tech-savvy population, recognized the potential of IM for service delivery and experimented with IM reference with limited hours. Following the positive reception at Eckles, the IM service expanded to Gelman with limited hours, with ongoing evaluation occurring at both locations.

Experience with virtual reference collaboratively staffed with other local academic libraries resulted in hesitation about new forms of electronic reference.

Librarians at Eckles, in an attempt to provide enhanced service for their small, tech-savvy population, recognized the potential of IM for service delivery and experimented with IM reference with limited hours.

Following the positive reception at Eckles, the IM service expanded to Gelman with limited hours, with ongoing evaluation occurring at both locations.

Two Programs, One Service: 2008 MERS Virtual Poster Session The Recursive Model in Practice: Service Hours After the initial pilot, Eckles expanded service hours. Eckles' statistics revealed peak hours, resulting in an expansion of Gelman's service hours. Demand for Eckles’ service decreased as a result of Gelman's expansion. Statistics allow both campuses to anticipate service needs and inform discussions of staffing models.

After the initial pilot, Eckles expanded service hours.

Eckles' statistics revealed peak hours, resulting in an expansion of Gelman's service hours.

Demand for Eckles’ service decreased as a result of Gelman's expansion.

Statistics allow both campuses to anticipate service needs and inform discussions of staffing models.

Two Programs, One Service: 2008 MERS Virtual Poster Session The Recursive Model in Practice: Staffing Models Eckles' service was initially staffed by one librarian, with additional support from student workers. Gelman's service was initially staffed by librarians from individual desks. Gelman’s distributed model proved problematic, so service moved to librarian staffing at the reference desk, with assistance from student workers. Eckles' service moved to triage staffing model using student workers, with librarians available for questions. Gelman is exploring new staffing models in the context of moving to a single service point and incorporating new departments into the IM reference service.

Eckles' service was initially staffed by one librarian, with additional support from student workers.

Gelman's service was initially staffed by librarians from individual desks.

Gelman’s distributed model proved problematic, so service moved to librarian staffing at the reference desk, with assistance from student workers.

Eckles' service moved to triage staffing model using student workers, with librarians available for questions.

Gelman is exploring new staffing models in the context of moving to a single service point and incorporating new departments into the IM reference service.

Two Programs, One Service: 2008 MERS Virtual Poster Session The Recursive Model in Practice: Training Technology-adept student workers at Eckles complete reference service training modules developed for new Gelman librarians. Gelman librarians receive training on new technologies first piloted at Eckles. The exchange of training modules draws on the expertise and improves the skills of service providers at both campuses.

Technology-adept student workers at Eckles complete reference service training modules developed for new Gelman librarians.

Gelman librarians receive training on new technologies first piloted at Eckles.

The exchange of training modules draws on the expertise and improves the skills of service providers at both campuses.

Two Programs, One Service: 2008 MERS Virtual Poster Session The Recursive Model in Practice: Assessment Usage statistics from both locations are used to inform how the services are tested, implemented, and expanded. Question-type statistics from both locations are used to inform staffing models and training. Ongoing qualitative transcript analysis is used to inform service definition, training, and staffing models. "Secret Shopper" evaluation used in student worker training at Eckles identified larger training needs, which will inform future training strategies that will be used at Gelman.

Usage statistics from both locations are used to inform how the services are tested, implemented, and expanded.

Question-type statistics from both locations are used to inform staffing models and training.

Ongoing qualitative transcript analysis is used to inform service definition, training, and staffing models.

"Secret Shopper" evaluation used in student worker training at Eckles identified larger training needs, which will inform future training strategies that will be used at Gelman.

Two Programs, One Service: 2008 MERS Virtual Poster Session The Recursive Model in Practice: Platform Testing Eckles adopted Trillian and Pidgin/GAIM. Gelman adopted Trillian. Eckles abandoned Pidgin/GAIM for Trillian. Eckles tested and adopted Trillian Pro. Gelman adopted Trillian Pro. Platform testing remains an ongoing project, with Meebo and other widgets under consideration for implementation.

Eckles adopted Trillian and Pidgin/GAIM.

Gelman adopted Trillian.

Eckles abandoned Pidgin/GAIM for Trillian.

Eckles tested and adopted Trillian Pro.

Gelman adopted Trillian Pro.

Platform testing remains an ongoing project, with Meebo and other widgets under consideration for implementation.

Two Programs, One Service: 2008 MERS Virtual Poster Session Challenges of the Recursive Model Scalability Service Providers Patrons Trying new things is always hard

Scalability

Service Providers

Patrons

Trying new things is always hard

Two Programs, One Service: 2008 MERS Virtual Poster Session Challenges of the Recursive Model: Scalability Organizational challenges differ between locations. Recursive cycle does not always flow evenly. The larger the organization or the greater the bureaucracy, the more staff time and energy will be required to encourage and implement innovation.

Organizational challenges differ between locations.

Recursive cycle does not always flow evenly.

The larger the organization or the greater the bureaucracy, the more staff time and energy will be required to encourage and implement innovation.

Two Programs, One Service: 2008 MERS Virtual Poster Session Challenges of the Recursive Model: Service Providers Service providers vary in technology skills, attitudes towards service, and interpretations of service requirements. Training models vary based on staff needs, which will affect the service provided at each location. Staffing models directly affect the quality of service provided. Technology in different contexts may affect providers’ ease of use. Familiarity with appropriate resources will affect the service provided.

Service providers vary in technology skills, attitudes towards service, and interpretations of service requirements.

Training models vary based on staff needs, which will affect the service provided at each location.

Staffing models directly affect the quality of service provided.

Technology in different contexts may affect providers’ ease of use.

Familiarity with appropriate resources will affect the service provided.

Two Programs, One Service: 2008 MERS Virtual Poster Session Challenges of the Recursive Model: Patrons Even the most representative pilot group will be different than the full population of both sites. Patrons may be less patient with the new service or changes in service. The service may not be able to meet the expectations of patrons used to more personalized service in the pilot location. Patron usage may shift away from the pilot location as services expand, which may result in a smaller population for future testing.

Even the most representative pilot group will be different than the full population of both sites.

Patrons may be less patient with the new service or changes in service.

The service may not be able to meet the expectations of patrons used to more personalized service in the pilot location.

Patron usage may shift away from the pilot location as services expand, which may result in a smaller population for future testing.

Two Programs, One Service: 2008 MERS Virtual Poster Session Challenges of the Recursive Model Trying new things is always hard.

Trying new things is always hard.

Two Programs, One Service: 2008 MERS Virtual Poster Session Lessons Learned from the Recursive Model Effective Pilot Groups Identifying Pilot Groups What Not to Do Reflections

Effective Pilot Groups

Identifying Pilot Groups

What Not to Do

Reflections

Two Programs, One Service: 2008 MERS Virtual Poster Session Lessons Learned from the Recursive Model Effective Pilot Groups Create a safer environment for testing, which inspires innovation. Include both providers and patrons. Facilitate a two-way conversation. Provide both positive and negative feedback. Allow your audience to engage in the testing process. Work collaboratively: with you rather than for you .

Effective Pilot Groups

Create a safer environment for testing, which inspires innovation.

Include both providers and patrons.

Facilitate a two-way conversation.

Provide both positive and negative feedback.

Allow your audience to engage in the testing process.

Work collaboratively: with you rather than for you .

Two Programs, One Service: 2008 MERS Virtual Poster Session Lessons Learned from the Recursive Model Identifying Pilot Groups Look for groups with a population that changes regularly: Freshmen or graduating seniors Student workers Groups with a small population: Special or departmental libraries Branch libraries Cohort-based programs

Identifying Pilot Groups

Look for groups with a population that changes regularly:

Freshmen or graduating seniors

Student workers

Groups with a small population:

Special or departmental libraries

Branch libraries

Cohort-based programs

Two Programs, One Service: 2008 MERS Virtual Poster Session Lessons Learned from the Recursive Model What Not to Do Don’t use your staff as a pilot group. Needs are not the same as those of patrons. Expectations are not the same as those of patrons. Feedback will reflect service provider concerns and biases rather than genuine patron experience. Don’t forget about staff input. While not representative of patron experience, the service provider’s perspective is critical to any new service.

What Not to Do

Don’t use your staff as a pilot group.

Needs are not the same as those of patrons.

Expectations are not the same as those of patrons.

Feedback will reflect service provider concerns and biases rather than genuine patron experience.

Don’t forget about staff input.

While not representative of patron experience, the service provider’s perspective is critical to any new service.

Two Programs, One Service: 2008 MERS Virtual Poster Session Lessons Learned from the Recursive Model Reflections Innovation encourages cycles of evaluation and improvement in other service areas. Challenges or failures will not derail other library services when done in parallel. Established framework for comparison and improvement builds an expectation of ongoing evaluation. Stagnation is prevented because something interesting is always happening.

Reflections

Innovation encourages cycles of evaluation and improvement in other service areas.

Challenges or failures will not derail other library services when done in parallel.

Established framework for comparison and improvement builds an expectation of ongoing evaluation.

Stagnation is prevented because something interesting is always happening.

Two Programs, One Service: 2008 MERS Virtual Poster Session Lessons Learned from the Recursive Model You Can Do It! The process has been challenging, but has proved to be both successful and rewarding. Creating an environment that facilitates innovation also sparks excitement among service providers, ensuring that library service remains dynamic and relevant for the patron.

You Can Do It!

The process has been challenging, but has proved to be both successful and rewarding.

Creating an environment that facilitates innovation also sparks excitement among service providers, ensuring that library service remains dynamic and relevant for the patron.

Two Programs, One Service: 2008 MERS Virtual Poster Session Contact Information Sarah Palacios-Wilhelm Reference & Instruction Librarian Eckles Library [email_address] AIM/Google Talk/ Yahoo!: sarahpwilhelm Elizabeth Edwards Reference/Technology Librarian Gelman Library [email_address] AIM: leepforward AIM/Google Talk/Yahoo!: GelmanInfo and EcklesInfo

Sarah Palacios-Wilhelm Reference & Instruction Librarian Eckles Library [email_address] AIM/Google Talk/ Yahoo!: sarahpwilhelm Elizabeth Edwards Reference/Technology Librarian Gelman Library [email_address] AIM: leepforward AIM/Google Talk/Yahoo!: GelmanInfo and EcklesInfo

Two Programs, One Service: 2008 MERS Virtual Poster Session Presentation prepared for: 2008 American Library Association Annual Conference MARS Management of Electronic Resources and Services (MERS) Committee Virtual Poster Session: Evaluation Virtual Reference Services

Presentation prepared for:

2008 American Library Association Annual Conference

MARS Management of Electronic Resources and Services (MERS) Committee

Virtual Poster Session: Evaluation Virtual Reference Services