Evaluate: Evaluating School Media Services

Evaluate ! Evaluating School Media Services LIB 620 Library Management Fall 2008

Preemptive Action : The busy librarian’s guide to program evaluation Based in part on: A workshop developed by the Pennsylvania School Librarians Association Professional Development Committee ( No longer available online)

Evaluate ?

What do we mean by evaluation?

TRANSITIVE VERB:

1. To ascertain or fix the value or worth of.

2. To examine and judge carefully; appraise. See synonyms at estimate .

3. Mathematics To calculate the numerical value of; express numerically.

American Heritage® Dictionary of the English Language , Fourth Edition. 2000 http://www.bartleby.com/61/

What is evaluation?

Evaluation aims to:

find out how far aims and objectives have been met

examine the outputs, outcomes and impacts that have been produced

explore effectiveness of procedures and delivery

assess cost-effectiveness and value-for-money

Step-by-Step Guide to Evaluation

What about school library media center evaluation?

Evaluation of school library media centers:

When you examine something or someone, you examine the subject and make a judgment about the quality, significance, or condition of whatever is to be evaluated.

Emanuel T. Prostano and Joyce S. Prostano, The School Library Media Center 5 th ed. (Englewood, CO: Libraries Unlimited, 1999), 44.

Purposes of evaluation

To find out what is right

To find out what is wrong

Blanche Woolls, The School Library Media Manager 2 nd ed. (Westport, CT: Libraries Unlimited, 1999).

What can you evaluate?

Everything!

Collection

Personnel

Library Layout

Programs

Curriculum collaboration

Reference service/collection:

Interview skills

Quality of the answers provided

Satisfaction of your patrons

Students or teachers—or other school staff

What should you evaluate?

That depends . . .

On your priorities

On your economic needs and situation

On the policies and procedures established by your school, school district and/or state

On how much time you have left over from your other duties to devote to planning and executing an evaluation project

Why evaluate?

Because you have to

Continuous assessment of services can be an ongoing requirement for accreditation/funding, etc.

Because you want to

You want to find out how well you’re doing

You want to find out how well others think you’re doing

You want to find ways to improve your service

Why Evaluate?

Schools are being evaluated by student academic achievement in reading, writing and math.

Recent studies show student achievement correlates positively with effective school library programs.

Why Evaluate?

Studies show the positive correlation with student achievement occurs when there is an effective school library program:

School librarians are full time and

Librarians collaborate with teachers on instructional planning.

Why Evaluate?

Studies also show an effective school library program includes the following attributes:

Quality collections;

Increased hours of access beyond school day;

Professional development for teachers and librarians;

Student access to technology; and

Collaboration with other types of libraries.

Why Evaluate? Because. . .

To improve, the librarian must have baseline data about the program.

To be relevant, the librarian must know if the library services and resources are aligned with the school’s goals for student achievement.

To build a case for better support, the librarian needs data that demonstrates the value of the program.

What Can an Evaluation Do?

Nancy Everhart, Evaluating the School Library Media Center , 1998)

An evaluation enables you to:

Determine success in attaining program goals.

Determine students’ and teachers’ needs so they can be incorporated into the program.

Provide a basis for resource allocation.

Recognize strengths and accomplishments.

Examine the impact of the program on student learning.

It makes sense—

Focus data collecting and evaluation on those components of a school library program that the research shows correlate with student achievement.

Types of Evaluation

Formal

Informal

External

Internal

Formative

During a program

Summative

After the program is completed

Evaluation requires standards

How do you know if the media center is “good”/ “not good” or even “good enough”?

“ Good enough” for what? For whom?

Guidelines or rubrics or objectives for collections, performance

Beyond Proficiency: Achieving a Distinguished Library Media Program . Kentucky Department of Education. August 2001

Administering the Library Media Program

The library media specialist:

Evaluates the Library Media Program through regular surveys for the purpose of enhancing services

Types of Data

Quantitative

Qualitative

Number of OPAC searches Size of collection Number of lessons planned with teachers Students’ success rate in locating appropriate resources in OPAC searches Collection supports the curriculum Comprehensive and collaborative planning is in place

Standards require interpretation

Quantitative: relatively easy

Library Media Staffing ( Beyond Proficiency )

ENROLLMENT PROFICIENT DISTINGUISHED * ** * ** Under 200 1 0 1 .5 200 - 500 1 .5 1 1 501 - 800 1 1 1 1.5 801 - 1200 1 1.5 1.5 1.5 1201 - 1600 1.5 1.5 2 2 1601 - 2000 2 2 2 2 2001 and Up 2.5 2 3 2 * Certified Library Media Specialist **Library Clerk - Classified

Standards require interpretation

Qualitative: more difficult

Program Evaluation Rubric, “ Beyond Proficiency ,” pp. 20-26.

Standard 2:

“ The Library Media Program promotes and supports student learning and achievement through its policies, programs and collection.”

Distinguished:

“ Students are empowered to use the media center to access information and reading for pleasure.”

Interpretation means operationalization

Operationalize:

To define a concept in a way that can be measured. In evaluation research, to translate program inputs, outputs, objectives, and goals into specific measurable variables.

Program Evaluation Glossary http://www.epa.gov/evaluate/glossary/o-esd.htm

Interpretation means creating goals and objectives

The purpose and study goals should determine the types of methods and measures you use to conduct the evaluation

Objectives will define your standard of excellence--the minimum level of appropriate service for your particular clientele

Jo Bell Whitlach, Evaluating Reference Services

Methods of Evaluation

Obtrusive

People are aware of the evaluation

Self-evaluation

Surveys

Observation

Unobtrusive

People are unaware of the evaluation

Unobtrusive measures of physical facilities

Use of proxies—“mystery patrons”

Obtrusive reference evaluations in a school library context

Questionnaires or interviews of students or teachers

Numbers gathering:

Reference question counts

Numbers/types of reference books used

Circulation statistics

Observation

By external observer

Self-observation: Journal

For example. . .

Research finding:

Students whose school librarian plays an instructional role tend to achieve higher than average test scores. This is also dependent on collaboration between school librarians and teachers and the inclusion of the library materials in the curriculum.

Evaluation Model

Define the question.

Collect data--determine needed data and method of collection.

Analyze the data.

Formulate recommendations.

Develop an action plan.

Research Finding

Students whose school librarian plays an instructional role tend to achieve higher than average scores.

What data are needed to determine whether or not the school librarian plays an instructional role in the school?

The Question Evaluation Model Step 1. Define the Question

Look at the rubric

Beyond Proficiency :

3. The Library Media program supports collaborative planning with the staff for the enhancement of instruction and support of student achievement. Distinguished: The LMS teams with teachers in the formal planning of student-centered authentic learning and project-based teaching and is a teaching partner.

According to Information Power--

Instructional role is determined by:

Collaboration between school librarians and teachers that results in the inclusion of library materials in the curriculum.

According to Information Power--

Information literacy standards

Collaborative planning

Effective teaching

Differentiated learning options

Inquiry

Assessment

Student Engagement

Instructional role is defined by:

Evaluation Model Step 2. Collect data

Determine the type needed.

Quantitative/Qualitative

Determine the method of collection.

Existing statistic

Survey, questionnaire, focus group, observation

One method to collect data--

Use Information Power’s Teaching and Learning Rubric as a questionnaire.

Administer it to

School Library staff,

Administrative staff,

Teaching staff.

Summarize each group; determine where perceptions differ and agree.

Data collected from the Teaching and Learning Rubric are qualitative--

When completing the questionnaire--

School librarian & principal:  EXEMPLARY!

Math teacher: 0 NOTHING!

Example: Principal puts librarian on weekly faculty meeting agenda to encourage library use and promote curriculum-oriented library materials. Math teacher does not see an easy connect and tunes out.

Evaluation Model Step 3. Analyze the data

Identify areas of agreement and disagreement among groups.

Identify areas of strength and areas that need improvement.

Identify areas where more data is needed.

How can the data be analyzed to determine whether or not the librarian plays an effective instructional role in the school?

Agreement Example--

Target Indicator: Information Literacy Standards are integrated into content learning .

Library Staff Response

____Basic 100% Proficient ____ Exemplary

Administrative Response

Teaching Staff Response

12% Basic 88% Proficient ____ Exemplary

Disagreement Example--

Target Indicator: Curriculum development is modeled and promoted.

100% Basic _____ Proficient ____ Exemplary

Strength Area Example--

Target Indicator: Collaborative planning is modeled and promoted.

Weakness Area Example--

Target Indicator: Students are engaged in reading, writing, speaking, viewing & listening for enjoyment, enrichment, & understanding.

100% Basic ____ Proficient ____ Exemplary

Contradictory Data Example—

Target Indicator:Effective teaching modeled & promoted.

Library Basic ____ Prof. 100% Exemplary

Admin. ____ Basic ____ Prof. 100% Exemplary

Teacher Basic 60% Prof. 40% Exemplary

Target Indicator: Student achievement is assessed.

Library 100% Basic Prof. Exemplary

Admin. 100% Basic ____ Prof. Exemplary

Teacher 90% Basic 10% Prof. Exemplary

Back to Step 2 Collecting More Data – Focus Groups

How are student products and performances assessed in units involving use of the library?

How do you determine the products and performances to demonstrate mastery of content in units involving use of the library?

What is the role of reflection in student work involving use of the library?

Evaluation Model Step 4. Formulate recommendations

Review the data, formulate recommendations.

For example:

Because of the strong correlation between the school librarian’s instructional role and student achievement, the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric

Role of an Advisory Committee of stakeholders—

Example: Recommendations for First Year

Review the Information Literacy Standards to ensure they align with content standards and set student expectations for analysis, evaluation and inquiry.

Meet with teaching teams to solicit recommended revisions.

Convene a workshop for new teachers to review the standards.

Information Literacy Standards

Collaborative Planning

Establish regular common planning time for teachers with the library media staff.

Curriculum Development

Review school policies to remove any barriers that prevent librarians from participating in building and district curriculum sessions.

Reading, Writing, Speaking, Viewing

Provide a series of author workshops and develop plans with teachers for student reading, writing, speaking, and viewing responses to the author workshops.

Involve Parents.

Effective Teaching

Clarify conflicting feedback through focus groups targeting use of assessment, differentiation, & inquiry in instruction.

Develop recommendations based on new data.

Evaluation Model Step 5 . Develop an action plan

Library staff develop an action plan for each recommendation.

The Advisory Committee of Stakeholders reviews and approves the action plans.

Action Plan Example: Target Indicator: Curriculum development is modeled and promoted Objective Activity Documentation Participants New Resources Completion Date Review barriers preventing librarians from participating in curriculum sessions. Review policies School Council minutes School Council & admin. None projected Sept. 1 Secure schedule Post schedule in lib. & faculty rm. Principal None Sept. 10 Participate in mtgs. Meeting minutes Librarians None projected June 1 Report back Faculty mtg. minutes Librarians None Monthly Written material In professional library Librarians None Within 5 days of mtg

This example of an evaluation was…

Type

Formal

Internal

Formative

Data collection methods:

Research

Rubric-based questionnaire

Focus Group

Data type:

Quantitative

Qualitative

Unobtrusive evaluation in schools 1

Unobtrusive measures of physical facilities

“ The basic premise . . . is that you can learn a great deal . . . by looking at how things wear (‘erosion’), how things are left in the building (‘traces’) and how things are rearranged (‘adaptations for use’).”

Nancy Everhart, Evaluating the School Library Media Center : Analysis Techniques and Research Practices. Libraries Unlimited, 1998 .

Unobtrusive evaluation in schools 2

Use of proxies—“mystery patrons”

Technique used more often in academic libraries and government documents reference services

Half-right reference: the 55% rule

5-minute rule

Peter Hernon and Charles R. McClure, “ Unobtrusive Reference Testing: The 55 Percent Rule ” Library Journal April 15, 1986, 37-41.

“ It’s not true, and now we know why . . . the so-called “55% rule” has never been tested against a truly representative field sample.”

John V. Richardson, Jr., “ Reference Is Better Than We Thought ,” Library Journal April 15, 2002, 41-42.

Importance of a sophisticated model

Richardson:

“ The reference service performance model [that led to the 55% rule] was overly simplistic, samples were way too small, and the test questions were not representative of real-world reference questions.”

“ Students of reference service should learn about the existence of multiple performance outcomes (i.e., accuracy, utility, and satisfaction) and to recognize that each outcome is driven by different factors.”