1. LIB 620 Library Management 1 Fall 2011Evaluating School Media Services
2. 2Based in part on: Preemptive Action: The busy librarian’s guide to program evaluation A workshop developed by the Pennsylvania School Librarians Association Professional Development Committee (No longer available online) indicates the slide was originally from the above presentation
5. 5 What about school library evaluation? Evaluation of school library mediacenters:– When you examine something or someone, you examine the subject and make a judgment about the quality, significance, or condition of whatever is to be evaluated. • Emanuel T. Prostano and Joyce S. Prostano, The School Library Media Center 5th ed. (Englewood, CO: Libraries Unlimited, 1999), 44.
6. 6Purposes of evaluation To find out what is right To find out what is wrong • Summary of definition in Blanche Woolls, The School Library Media Manager 2nd ed. (Westport, CT: Libraries Unlimited, 1999).
7. 7What can you evaluate? Everything!– Collection – Reference– Personnel service/collection:– Library Layout • Interview skills– Programs • Quality of the answers provided– Curriculum collaboration • Satisfaction of your patrons – Students or teachers—or other school staff
8. 8What should you evaluate? That depends . . .– On your priorities– On your economic needs and situation– On the policies and procedures established by your school, school district and/or state– On how much time you have left over from your other duties to devote to planning and executing an evaluation project
9. 9 Why evaluate? Because you have to– Continuous assessment of services can be an ongoing requirement for accreditation/funding, etc. Because you want to– You want to find out how well you‟re doing– You want to find out how well others think you‟re doing– You want to find ways to improve your service
10. 10Why Evaluate? • Schools are being evaluated by student academic achievement in reading, writing and math. • Recent studies show student achievement correlates positively with effective school library programs.
11. 11Why Evaluate? Studies show the positive correlationwith student achievement occurs whenthere is an effective school libraryprogram [and]: – School librarians are full time and – Librarians collaborate with teachers on instructional planning.
12. 12Why Evaluate? Studies also show an effectiveschool library program includesthe following attributes:– Quality collections;– Increased hours of access beyond school day;– Professional development for teachers and librarians;– Student access to technology; and– Collaboration with other types of libraries.
13. 13Why Evaluate? Because. . .To improve, the librarian must have baselinedata about the program.To be relevant, the librarian must know if thelibrary services and resources are aligned withthe school‟s goals for student achievement.To build a case for better support, thelibrarian needs data that demonstrates thevalue of the program.
14. 14What Can an Evaluation Do? An evaluation enables you to:– Determine success in attaining program goals.– Determine students‟ and teachers‟ needs so they can be incorporated into the program.– Provide a basis for resource allocation.– Recognize strengths and accomplishments.– Examine the impact of the program on student learning. • Nancy Everhart, Evaluating the School Library Media Center, 1998)
15. 15 It makes sense— Focus data collecting andevaluation on those components of aschool library program that theresearch shows correlate with studentachievement.
16. 16 Types of Evaluation Formal Informal External Internal Formative During a program Summative After the program is completed
17. 17 Evaluation requires standards How do you know if the media center is ―good‖/ ―notgood‖ or even ―good enough‖? – “Good enough” for what? For whom? Guidelines or rubrics or objectives forcollections, performance can be found in:
18. 18 Types of DataQuantitative Qualitative Number of OPAC Students‟ success rate in locating appropriate searches resources in OPAC searches Size of Collection supports the collection curriculum Number of Comprehensive and lessons planned collaborative planning is in with teachers place
19. 19 Standards require interpretation Quantitative: relatively easy– Library Media Staffing– The school library media program is staffed as follows: – Under 200: 1 full-time school media librarian (SML) – 200-500: 1 full-time SML; .5 clerk – 501-800: 1 full-time SML; 1 clerk – 801-1200: 1 full-time SML; 1.5 clerk – 1201-1600: 1.5 SML; 1.5 clerk – 1601-2000: 2 full-time SML; 2 clerks – 2001 and Up: 2.5 SML; 2 clerks » Library Media Program Rubric August 2010
21. 21Interpretation means operationalization Operationalize:– To define a concept in a way that can be measured. In evaluation research, to translate program inputs, outputs, objectives, and goals into specific measurable variables. • Program Evaluation Glossary http://www.epa.gov/evaluate/glossary/o-esd.htm
22. 22Interpretation means creating goals and objectives The purpose and study goals should determine the types of methods and measures you use to conduct the evaluation Objectives will define your standard of excellence--the minimum level of appropriate service for your particular clientele – Jo Bell Whitlach, Evaluating Reference Services
23. 23Methods of Evaluation Obtrusive– People are aware of the evaluation • Self-evaluation • Surveys • Observation Unobtrusive– People are unaware of the evaluation • Unobtrusive measures of physical facilities • Use of proxies—“mystery patrons”
24. 24Obtrusive reference evaluations in a schoollibrary context Questionnaires or interviews of students orteachers Numbers gathering: – Reference question counts – Numbers/types of reference books used – Circulation statistics Observation – By external observer – Self-observation: Journal
25. 25For example. . .Research finding:Students whose school librarian plays aninstructional role tend to achieve higher thanaverage test scores. This is also dependent oncollaboration between school librarians andteachers and the inclusion of the librarymaterials in the curriculum.
26. 26 Evaluation Model1. Define the question.2. Collect data--determine needed data and method of collection.3. Analyze the data.4. Formulate recommendations.5. Develop an action plan.
27. 27Evaluation Model Step 1.Define the QuestionResearch Finding The QuestionStudents whose school What data are neededlibrarian plays an to determine whetherinstructional role or not the schooltend to achieve higher librarian plays anthan average scores. instructional role in the school?
28. 28Look at the rubric
29. According to 29Information Power (1998)-- Instructional role is determined by:Collaboration between school librariansand teachers that results in the inclusion oflibrary materials in the curriculum.
30. According to 30Information Power-- Instructional role is defined by: – Information literacy standards – Collaborative planning – Effective teaching • Differentiated learning options • Inquiry • Assessment – Student Engagement
31. 31According to Empowering Learners (2009) Guideline 1: – The school library media program promotes collaboration among members of the learning community and encourages learners to be independent, lifelong users and producers of information.
32. 32According to Empowering Learners Guideline 3:– The school library media program provides instruction that addresses multiple literacies, including information literacy, media literacy, visual literacy, and technology literacy.
33. 33Evaluation Model Step 2.Collect data Determine the type needed.– Quantitative/Qualitative Determine the method of collection.– Existing statistic– Survey, questionnaire, focus group, observation
34. 34One method to collect data-- Use Information Power‟s Teaching andLearning Rubric as a questionnaire. Administer it to – School Library staff, – Administrative staff, – Teaching staff. Summarize each group; determine whereperceptions differ and agree.
35. 35Data collected from the Teaching and Learning Rubricare qualitative-- Example: Principal puts librarian on weekly faculty meeting agenda to encourage library use and promote curriculum-oriented library materials. Math teacher does not see an easy connect and tunes out. When completing the questionnaire-- School librarian & principal: EXEMPLARY! Math teacher: 0 NOTHING!
36. 36 Evaluation Model Step 3. Analyze the data How can the data be analyzed to determinewhether or not the librarian plays an effectiveinstructional role in the school? – Identify areas of agreement and disagreement among groups. – Identify areas of strength and areas that need improvement. – Identify areas where more data is needed.
40. 40Weakness Area Example--Target Indicator: Students are engaged in reading, writing, speaking, viewing & listening for enjoyment, enrichment, & understanding.Library Staff Response ____Basic 100% Proficient ____ ExemplaryAdministrative Response100% Basic ____ Proficient ____ ExemplaryTeaching Staff Response82% Basic 18% Proficient ____ Exemplary
41. 41Contradictory Data Example—Target Indicator:Effective teaching modeled & promoted. Library Basic ____ Prof. 100% Exemplary Admin. ____ Basic ____ Prof. 100% Exemplary Teacher Basic 60% Prof. 40% ExemplaryTarget Indicator: Student achievement is assessed. Library 100% Basic Prof. Exemplary Admin. 100% Basic ____ Prof. Exemplary Teacher 90% Basic 10% Prof. Exemplary
42. 42 Back to Step 2 Collecting More Data – Focus Groups1. How are student products and performances assessed in units involving use of the library?2. How do you determine the products and performances to demonstrate mastery of content in units involving use of the library?3. What is the role of reflection in student work involving use of the library?
43. 43 Evaluation Model Step 4. Formulate recommendations Role of an Advisory Committee of stakeholders—• Review the data, formulate recommendations.• For example: – Because of the strong correlation between the school librarian‟s instructional role and student achievement, the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
44. Example: 44Recommendations for First Year Standards for the 21st Century Learner– Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis, evaluation and inquiry.– Meet with teaching teams to solicit recommended revisions.– Convene a workshop for new teachers to review the standards. Adapted from
45. Example: 45 Recommendations for First Year Collaborative Planning– Establish regular common planning time for teachers with the library media staff. Curriculum Development– Review school policies to remove any barriers that prevent librarians from participating in building and district curriculum sessions.
46. Example: 46 Recommendations for First Year Reading, Writing, Speaking, Viewing– Provide a series of author workshops and develop plans with teachers for student reading, writing, speaking, and viewing responses to the author workshops.– Involve Parents. Effective Teaching– Clarify conflicting feedback through focus groups targeting use of assessment, differentiation, & inquiry in instruction.– Develop recommendations based on new data.
47. 47 Evaluation Model Step 5. Develop an action plan Library staff develop an action plan foreach recommendation. The Advisory Committee ofStakeholders reviews and approves theaction plans.
48. 48Action Plan Example:Target Indicator: Curriculum development is modeled and promotedObjective Activity Documentation Participants New Completion Resources DateReview barriers Review School Council School None Sept. 1preventing policies minutes Council & projectedlibrarians from admin.participating in Secure Post schedule in Principal None Sept. 10curriculum schedule lib. & faculty rm.sessions. Participate Meeting minutes Librarians None June 1 in mtgs. projected Report Faculty mtg. Librarians None Monthly back minutes Written In professional Librarians None Within 5 material library days of mtg
49. 49This example of an evaluation was…• Type (which one?) • Data collection methods: – Formal – Research – Internal – Rubric-based – Formative questionnaire – Focus Group • Data type: – Quantitative – Qualitative
50. 50Unobtrusive evaluation in schools 1 Unobtrusive measures of physical facilities– “The basic premise . . . is that you can learn a great deal . . . by looking at how things wear („erosion‟), how things are left in the building („traces‟) and how things are rearranged („adaptations for use‟).” • Nancy Everhart, Evaluating the School Library Media Center: Analysis Techniques and Research Practices. Libraries Unlimited, 1998.
51. 51Unobtrusive evaluation in schools 2 Use of proxies—―mystery patrons‖– Technique used more often in academic libraries and government documents reference services • Half-right reference: the 55% rule • 5-minute rule – Peter Hernon and Charles R. McClure, “Unobtrusive Reference Testing: The 55 Percent Rule” Library Journal April 15, 1986, 37-41. • “It‟s not true, and now we know why . . . the so-called “55% rule” has never been tested against a truly representative field sample.” – John V. Richardson, Jr., “Reference Is Better Than We Thought,” Library Journal April 15, 2002, 41-42.
52. 52 Importance of a sophisticated model Richardson:– “The reference service performance model [that led to the 55% rule] was overly simplistic, samples were way too small, and the test questions were not representative of real-world reference questions.”– “Students of reference service should learn about the existence of multiple performance outcomes (i.e., accuracy, utility, and satisfaction) and to recognize that each outcome is driven by different factors.” • “Reference Is Better Than We Thought.”
53. 53 Characteristics of good models Measures of reference service must be– Valid • They “accurately reflect the concept being studied.”– Reliable • They “are stable and dependable, and provide consistent results with each repeated use.”– Practical • They “require that data be relatively easy to collect.”– Useful • They “provide information that can be used to improve reference services.” – Whitlach, Evaluating Reference Services.
54. 54Problems of Evaluation in School Context Often the school librarian is the onlyone working in the library An evaluation project can be time-consuming to plan and to put into action
55. 55 Solutions to evaluation problems Depends on your local situation– Your resources: • Time, money, available warm bodies– You and your creativity Make evaluation part of your routine– As you plan/prepare your program(s), include an evaluation component • Appropriate, affordable, accessible, accountable