This document discusses evaluating school library media programs. It provides definitions of evaluation, purposes of evaluation such as improving services and demonstrating value, and types of data that can be collected including quantitative and qualitative data. The document also presents an evaluation model including defining questions, collecting and analyzing data, formulating recommendations, and developing an action plan. Methods of evaluation discussed include questionnaires, observation, and focus groups. Challenges of evaluation in a school context and potential solutions are also outlined.

1. LIB 620 Library ManagementFall 2009Evaluate!Evaluating School Media Services

2. Based in part on:Preemptive Action:The busy librarian’s guide to program evaluationA workshop developed by thePennsylvania School Librarians Association Professional Development Committee(No longer available online)indicates the slide was originally from the above presentation

3. Evaluate?What do we mean by evaluate?verb (used with object), -at⋅ed, -at⋅ing. to determine or set the value or amount of; appraise: to evaluate property. to judge or determine the significance, worth, or quality of; assess: to evaluate the results of an experiment. Mathematics. to ascertain the numerical value of (a function, relation, etc.).The American Heritage® Dictionary of the English Language, Fourth Edition © 2006 by Houghton Mifflin Company.

4. What is evaluation?[1]As defined by the American Evaluation Association: evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness. Evaluation is the systematic collection and analysis of data needed to make decisionsEvaluation Definition: What is Evaluation?

5. What about school library media center evaluation?Evaluation of school library media centers:When you examine something or someone, you examine the subject and make a judgment about the quality, significance, or condition of whatever is to be evaluated.Emanuel T. Prostano and Joyce S. Prostano, The School Library Media Center5th ed. (Englewood, CO: Libraries Unlimited, 1999), 44.

6. Purposes of evaluationTo find out what is rightTo find out what is wrongSummary of definition in Blanche Woolls, The School Library Media Manager 2nd ed. (Westport, CT: Libraries Unlimited, 1999).

7. What can you evaluate?Everything!CollectionPersonnelLibrary LayoutProgramsCurriculum collaborationReference service/collection:Interview skillsQuality of the answers providedSatisfaction of your patronsStudents or teachers—or other school staff

8. What should you evaluate?That depends . . . On your prioritiesOn your economic needs and situationOn the policies and procedures established by your school, school district and/or stateOn how much time you have left over from your other duties to devote to planning and executing an evaluation project

9. Why evaluate?Because you have toContinuous assessment of services can be an ongoing requirement for accreditation/funding, etc.Because you want toYou want to find out how well you’re doingYou want to find out how well others think you’re doingYou want to find ways to improve your service

10. Why Evaluate?Schools are being evaluated by student academic achievement in reading, writing and math.Recent studies show student achievement correlates positively with effective school library programs.

11. Why Evaluate?Studies show the positive correlation with student achievement occurs when there is an effective school library program [and]:School librarians are full time andLibrarians collaborate with teachers on instructional planning.

12. Why Evaluate? Studies also show an effective school library program includes the following attributes: Quality collections;Increased hours of access beyond school day;Professional development for teachers and librarians;Student access to technology; andCollaboration with other types of libraries.

13. Why Evaluate? Because. . .To improve, the librarian must have baseline data about the program. To be relevant, the librarian must know if the library services and resources are aligned with the school’s goals for student achievement.To build a case for better support, the librarian needs data that demonstrates the value of the program.

14. What Can an Evaluation Do?An evaluation enables you to:Determine success in attaining program goals.Determine students’ and teachers’ needs so they can be incorporated into the program.Provide a basis for resource allocation.Recognize strengths and accomplishments.Examine the impact of the program on student learning.Nancy Everhart,Evaluating the School Library Media Center, 1998)It makes sense—Focus data collecting and evaluation on those components of a school library program that the research shows correlate with student achievement.Types of EvaluationFormal

15. Informal

16. External

17. Internal

18. Formative

19. During a program

20. Summative

21. After the program is completedEvaluation requires standardsHow do you know if the media center is “good”/ “not good” or even “good enough”?“Good enough” for what? For whom?Guidelines or rubrics or objectives for collections, performance can be found in:Beyond Proficiency: Achieving a Distinguished Library Media Program. Kentucky Department of Education. August 2001Administering the Library Media ProgramThe library media specialist:Evaluates the Library Media Program through regular surveys for the purpose of enhancing services

22. Types of DataQuantitativeQualitativeStudents’ success rate in locating appropriate resources in OPAC searchesNumber of OPAC searches Size of collectionCollection supports the curriculumNumber of lessons planned with teachersComprehensive and collaborative planning is in place

23. Standards require interpretationQuantitative: relatively easyLibrary Media Staffing (Beyond Proficiency)ENROLLMENT PROFICIENT DISTINGUISHED * ** * **Under 200 1 0 1 .5200 - 500 1 .5 1 1501 - 800 1 1 1 1.5801 - 1200 1 1.5 1.5 1.51201 - 1600 1.5 1.5 2 21601 - 2000 2 2 2 22001 and Up 2.5 2 3 2* Certified Library Media Specialist**Library Clerk - Classified

24. Standards require interpretationQualitative: more difficultProgram Evaluation Rubric, “Beyond Proficiency,” pp. 20-26.Standard 2: “The Library Media Program promotes and supports student learning and achievement through its policies, programs and collection.”Distinguished:“Students are empowered to use the media center to access information and reading for pleasure.”

25. Interpretation means operationalizationOperationalize:To define a concept in a way that can be measured. In evaluation research, to translate program inputs, outputs, objectives, and goals into specific measurable variables. Program Evaluation Glossary http://www.epa.gov/evaluate/glossary/o-esd.htm

26. Interpretation means creating goals and objectivesThe purpose and study goals should determine the types of methods and measures you use to conduct the evaluationObjectives will define your standard of excellence--the minimum level of appropriate service for your particular clientele Jo Bell Whitlach, Evaluating Reference Services

27. Methods of EvaluationObtrusivePeople are aware of the evaluationSelf-evaluationSurveysObservationUnobtrusivePeople are unaware of the evaluationUnobtrusive measures of physical facilitiesUse of proxies—“mystery patrons”

28. Obtrusive reference evaluations in a school library contextQuestionnaires or interviews of students or teachersNumbers gathering:Reference question countsNumbers/types of reference books usedCirculation statisticsObservationBy external observerSelf-observation: Journal

29. For example. . .Research finding: Students whose school librarian plays an instructional role tend to achieve higher than average test scores. This is also dependent on collaboration between school librarians and teachers and the inclusion of the library materials in the curriculum.

30. Evaluation ModelDefine the question.Collect data--determine needed data and method of collection.Analyze the data.Formulate recommendations.Develop an action plan.

31. Evaluation Model Step 1.Define the QuestionThe QuestionResearch Finding Students whose school librarian plays an instructional role tend to achieve higher than average scores. What data are needed to determine whether or not the school librarian plays an instructional role in the school?

32. Look at the rubricBeyond Proficiency:3. The Library Media program supportscollaborative planning with the staff forthe enhancement of instruction and support of student achievement.Distinguished:The LMS teams with teachers in the formal planning of student-centered authentic learning and project-based teaching and is a teaching partner.

33. According toInformation Power--Instructional role is determined by:Collaboration between school librarians and teachers that results in the inclusion of library materials in the curriculum.

34. According toInformation Power--Instructional role is defined by:Information literacy standardsCollaborative planningEffective teachingDifferentiated learning optionsInquiryAssessmentStudent Engagement

35. According to Empowering LearnersGuideline 1:The school library media program promotes collaboration among members of the learning community and encourages learners to be independent, lifelong users and producers of information.

36. Guideline 3:The school library media program provides instruction that addresses multiple literacies, including information literacy, media literacy, visual literacy, and technology literacy. According to Empowering Learners

37. Evaluation Model Step 2.Collect dataDetermine the type needed.Quantitative/QualitativeDetermine the method of collection.Existing statisticSurvey, questionnaire, focus group, observation

38. One method to collect data--Use Information Power’s Teaching and Learning Rubric as a questionnaire.Administer it toSchool Library staff,Administrative staff,Teaching staff.Summarize each group; determine where perceptions differ and agree.

39. Data collected from the Teaching and Learning Rubric are qualitative--Example:Principal puts librarian on weekly faculty meeting agenda to encourage library use and promote curriculum-oriented library materials. Math teacher does not see an easy connect and tunes out.When completing the questionnaire--School librarian & principal:  EXEMPLARY!Math teacher: 0 NOTHING!

40. Evaluation Model Step 3.Analyze the dataHow can the data be analyzed to determine whether or not the librarian plays an effective instructional role in the school?Identify areas of agreement and disagreement among groups.Identify areas of strength and areas that need improvement.Identify areas where more data is needed.

41. Agreement Example--Target Indicator: Information Literacy Standards are integrated into content learning. Library Staff Response ____Basic 100% Proficient ____ Exemplary Administrative Response ____Basic 100% Proficient ____ Exemplary Teaching Staff Response 12% Basic 88% Proficient ____ Exemplary

42. Disagreement Example--Target Indicator: Curriculum development is modeled and promoted.Library Staff Response 100% Basic _____ Proficient ____ ExemplaryAdministrative Response ____Basic 100% Proficient ____ ExemplaryTeaching Staff Response 82%Basic 18% Proficient ____ Exemplary

43. Strength Area Example--Target Indicator: Collaborative planning is modeled and promoted.Library Staff Response____Basic 100% Proficient ____ ExemplaryAdministrative Response____Basic 100% Proficient ____ ExemplaryTeaching Staff Response____Basic 100% Proficient ____ Exemplary

44. Weakness Area Example--Target Indicator: Students are engaged in reading, writing, speaking, viewing & listening for enjoyment, enrichment, & understanding. Library Staff Response ____Basic 100% Proficient ____ Exemplary Administrative Response100% Basic ____ Proficient ____ Exemplary Teaching Staff Response 82% Basic 18% Proficient ____ Exemplary

45. Contradictory Data Example—Target Indicator:Effective teaching modeled & promoted.Library Basic ____ Prof. 100% Exemplary Admin. ____ Basic ____ Prof. 100% Exemplary Teacher Basic 60% Prof. 40% ExemplaryTarget Indicator: Student achievement is assessed.Library 100% Basic Prof. Exemplary Admin. 100% Basic ____ Prof. Exemplary Teacher 90% Basic 10% Prof. Exemplary

46. Back to Step 2Collecting More Data – Focus GroupsHow are student products and performances assessed in units involving use of the library?How do you determine the products and performances to demonstrate mastery of content in units involving use of the library?What is the role of reflection in student work involving use of the library?

47. Evaluation Model Step 4.Formulate recommendationsRole of an Advisory Committee of stakeholders—Review the data, formulate recommendations.

48. For example:

49. Because of the strong correlation between the school librarian’s instructional role and student achievement, the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning RubricExample:Recommendations for First YearStandards for the 21st Century LearnerReview the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis, evaluation and inquiry.Meet with teaching teams to solicit recommended revisions.Convene a workshop for new teachers to review the standards.Adapted from

50. Example:Recommendations for First YearCollaborative PlanningEstablish regular common planning time for teachers with the library media staff.Curriculum DevelopmentReview school policies to remove any barriers that prevent librarians from participating in building and district curriculum sessions.

51. Example:Recommendations for First YearReading, Writing, Speaking, ViewingProvide a series of author workshops and develop plans with teachers for student reading, writing, speaking, and viewing responses to the author workshops.Involve Parents.Effective TeachingClarify conflicting feedback through focus groups targeting use of assessment, differentiation, & inquiry in instruction.Develop recommendations based on new data.

52. Evaluation Model Step 5.Develop an action planLibrary staff develop an action plan for each recommendation.The Advisory Committee of Stakeholders reviews and approves the action plans.

53. Action Plan Example:Target Indicator: Curriculum development is modeled and promoted

54. This example of an evaluation was…Data collection methods:ResearchRubric-based questionnaireFocus GroupData type:QuantitativeQualitativeType (which one?)FormalInternalFormative

55. Unobtrusive evaluation in schools 1Unobtrusive measures of physical facilities“The basic premise . . . is that you can learn a great deal . . . by looking at how things wear (‘erosion’), how things are left in the building (‘traces’) and how things are rearranged (‘adaptations for use’).”Nancy Everhart, Evaluating the School Library Media Center: Analysis Techniques and Research Practices. Libraries Unlimited, 1998.

56. Unobtrusive evaluation in schools 2Use of proxies—“mystery patrons”Technique used more often in academic libraries and government documents reference servicesHalf-right reference: the 55% rule5-minute rulePeter Hernon and Charles R. McClure, “Unobtrusive Reference Testing: The 55 Percent Rule” Library Journal April 15, 1986, 37-41.“It’s not true, and now we know why . . . the so-called “55% rule” has never been tested against a truly representative field sample.”John V. Richardson, Jr., “Reference Is Better Than We Thought,” Library Journal April 15, 2002, 41-42.

57. Importance of a sophisticated modelRichardson:“The reference service performance model [that led to the 55% rule] was overly simplistic, samples were way too small, and the test questions were not representative of real-world reference questions.” “Students of reference service should learn about the existence of multiple performance outcomes (i.e., accuracy, utility, and satisfaction) and to recognize that each outcome is driven by different factors.”“Reference Is Better Than We Thought.”

58. Characteristics of good modelsMeasures of reference service must beValidThey “accurately reflect the concept being studied.”ReliableThey “are stable and dependable, and provide consistent results with each repeated use.”PracticalThey “require that data be relatively easy to collect.”UsefulThey “provide information that can be used to improve reference services.”Whitlach, Evaluating Reference Services.

59. Problems of Evaluation in School ContextOften the school library media specialist is the only one working in the libraryAn evaluation project can be time-consuming to plan and to put into action

60. Solutions to evaluation problemsDepends on your local situationYour resources:Time, money, available warm bodiesYou and your creativityMake evaluation part of your routineAs you plan/prepare your program(s), include an evaluation componentAppropriate, affordable, accessible, accountable