Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building a Culture of Assessment: What I've Learned from Research, Trial, and (a lot of) Error

903 views

Published on

Presentation for Council of Chief Librarians, CA Community Colleges

Published in: Education
  • Be the first to comment

Building a Culture of Assessment: What I've Learned from Research, Trial, and (a lot of) Error

  1. 1. BUILDING A CULTURE OF ASSESSMENT WHAT I'VE LEARNED FROM RESEARCH, TRIAL, AND (A LOT OF) ERROR Meredith Farkas Portland Community College
  2. 2. ABOUT ME ➤ Faculty Librarian at Portland Community College (OR) ➤ Previously: ➤ Head of Instructional Services, Portland State University ➤ Head of Instructional Initiatives, Norwich University ➤ Significant research and publishing on what it takes to build an assessment culture ➤ Former psychotherapist interested in what it takes to really change culture
  3. 3. “A culture of assessment is an organizational environment in which decisions are based on facts, research, and analysis, and where services are planned and delivered in ways that maximize positive outcomes and impacts for customers and stakeholders. -Amos Lakos, Shelley Phipps, and Betsy Wilson Lakos, A. & Phipps, S. E. (2004). "Creating a culture of assessment: A catalyst for organizational change." portal: Libraries and the Academy, 4(3), 345-361.
  4. 4. MY DEFINITION ➤ Assessment is the norm - a regular part of practice ➤ User-focused culture ➤ Learning and curiosity driven ➤ Assessment is done for improvement ➤ Changes and decisions are made based on what is learned ➤ Planning for new services always includes planning for assessment
  5. 5. ASSESSMENT AT MANY INSTITUTIONS ➤ Focused on doing what is needed to satisfy accreditors or administrators ➤ Coercive and top-down ➤ Little support ➤ Coalition of the willing ➤ Data is gathered, but rarely used ➤ Not seen as an integral part of teaching and learning ➤ Often a lot of cynicism around assessment
  6. 6. MY EXPERIENCE: ADVENTURES WITH COORDINATOR SYNDROME
  7. 7. “ The word ‘Coordinator’ in a position title. Beware it, especially in combination with a new or non-traditional job niche… The problem with these jobs is that as often as not, there’s nothing actually to coordinate. No budget. No dedicated staff. No established service. ‘Coordinate’ all too often means ‘try to establish a beachhead by begging your new colleagues to vouchsafe you a few minutes of their time now and then, knowing that their supervisors won’t tell them to and you have no authority whatever to demand anything of them.’ Oh, but the buck will stop with you, Coordinator, on all matters regarding your service. Be aware, also, that because what you’re doing is new, you must hit a home run with it, quickly; slow and steady gains will not do. If you don’t, two damaging assumptions arise: that the service is a waste of effort, and that you’re a lousy librarian. You’re a coordinator now; congratulations! -Gavia Libraria, “The C-word"
  8. 8. YOU SAY YOU NEED SUPPORT? I’VE GOT YOUR SUPPORT RIGHT HERE.
  9. 9. ➤ Farkas, Meredith G. "Building and sustaining a culture of assessment: best practices for change leadership." Reference services review 41.1 (2013): 13-31. ➤ Farkas, Meredith G., and Lisa J. Hinchliffe. "Library faculty and instructional assessment: Creating a culture of assessment through the high performance programming model of organizational transformation." Collaborative Librarianship 5.3 (2013): 177-188. ➤ Farkas, Meredith Gorran, Lisa Janicke Hinchliffe, and Amy Harris Houk. "Bridges and barriers: factors influencing a culture of assessment in academic libraries." College & Research Libraries (2014). ➤ Farkas, Meredith. "Accountability vs. Improvement: Seeking Balance in the Value of Academic Libraries Initiative." OLA Quarterly 19.1 (2014): 4-7.
  10. 10. “For a scholarship of assessment to thrive, we must align faculty culture, institutional structures, and leadership for change. The importance of this point cannot be overstated. A meaningful assessment program is more than just a new activity to be undertaken, it is a change in how we think about what we do in higher education. -Don Haviland “Leading Assessment, from Faculty Resistance to Faculty Engagement.”
  11. 11. THESE STUDIES SUGGEST SPECIFIC FACILITATING/HINDERING FACTORS 1. Oakleaf, M. & Hinchliffe, L. J. (2008). Assessment cycle or circular file: Do academic librarians use information literacy assessment data? In Proceedings of the 2008 Library Assessment Conference: Building Effective, Sustainable, Practical Assessment. 2. Ndoye, A. & Parker, M. A. (2010). "Creating and sustaining a culture of assessment." Planning for Higher Education, 38(2), 28-39. 3. Hiller, S., Kyrillidou, M., & Self, J. (2008). "When the evidence is not enough: Organizational factors that influence effective and successful library assessment." Performance Measures and Metrics, 9(3), 223-230. 4. Wright, S. & White, L. (2007). SPEC Kit 303, Library Assessment. Washington, D.C.: Association of Research Libraries.
  12. 12. FACILITATING FACTORS SUGGESTED IN THE LITERATURE ➤ Assessment is a library priority (4) ➤ Commitment from library leadership (1, 3, 4) ➤ Commitment from faculty/staff (2, 3, 4) ➤ Organizational culture (3, 4) ➤ Staff with needed expertise and time (1, 3, 4) ➤ Staff development (4) ➤ Evidence-based decision-making (1, 2, 3, 4)
  13. 13. DEMONSTRATING THAT ASSESSMENT IS A LIBRARY PRIORITY ➤ Shared understanding of assessment exists within the organization ➤ Clear expectations or concrete goals for assessment ➤ Building an assessment plan
  14. 14. COMMITMENT FROM ADMINISTRATION ➤ Articulates the value of assessment ➤ Assessment is built into ALL planning ➤ Assessment is built into reward structures ➤ Supports library faculty/staff engaging in assessment ➤ Uses assessment data to inform decisions
  15. 15. EXPERTISE AND TIME ➤ Staff need dedicated time to learn, try, fail, and learn more ➤ Find ways to capitalize on expertise outside of the library
  16. 16. STAFF DEVELOPMENT ➤ Don’t just send one person to a conference ➤ Opportunities for staff to learn together ➤ Build a culture of reflective practice and learning communities
  17. 17. WHY DON’T LIBRARIANS USE ASSESSMENT RESULTS? ➤ Too busy with other work ➤ Won’t be rewarded for assessment work ➤ Lack of expertise ➤ Lack of centralized support within the library ➤ Not valued by their library Oakleaf and Hinchliffe. (2008). “Assessment Cycle or Circular File: Do Academic Librarians Use Information Literacy Assessment Data?” Proceedings of the 2008 Library Assessment Conference.
  18. 18. FACILITATING FACTORS - EXPANDED 1. Clear expectations for assessment in the library 2. Assessment is a priority of administration 3. Library has adopted learning outcomes 4. Library has an assessment plan 5. Library has a shared understanding of the purpose of assessment 6. Library faculty/staff are supported in doing assessment 7. Education and training about assessment are supported 8. Library has assessment expertise in-house 9. Library has systems/technologies to support assessment 10. Library leadership offers explicit support to get staff doing assessment 11. Data is available to interested parties 12. Leadership uses assessment data in decision-making 13. Librarians use results to improve practice 14. Library culture is user-focused
  19. 19. ULTIMATELY, THIS IS ABOUT CHANGING CULTURE
  20. 20. JOHN KOTTER’S LEADING CHANGE: THE ORGANIZATIONAL CHANGE BIBLE
  21. 21. CREATE A SENSE OF URGENCY ➤ Urgency must be driven by opportunities, not by fear ➤ Appeal to the things that your audience cares about (student learning, being better instructors, etc.) ➤ Sometimes urgency comes from outside (mandates from administrators, outside agencies, etc.)
  22. 22. FORM A GUIDING COALITION (NOT JUST A COORDINATOR) ➤ Form a group to lead and support assessment efforts ➤ Membership based on skills, interest, and politics ➤ They will need admin support ➤ Examples: ➤ Northwestern University libraryassessment.org/ bm~doc/ 4macayeallightingtalk.pdf ➤ Portland Community College www.pcc.edu/edserv/learning- assessment/LACAboutUs.html
  23. 23. MacAyeal & Ryan. (2014). Support System: Establishing, Sustaining,and Growing a Frameworkfor Assessment. Library Assessment Conference.
  24. 24. MacAyeal & Ryan. (2014). Support System: Establishing, Sustaining,and Growing a Frameworkfor Assessment. Library Assessment Conference.
  25. 25. PCC LEARNING ASSESSMENT COUNCIL
  26. 26. CREATE AND COMMUNICATE A VISION ➤ Common understanding of the purpose of our assessment program ➤ Develop an assessment plan OR ➤ Develop concrete expectations for what progress in assessment looks like
  27. 27. EMPOWER OTHERS TO ACT ON THE VISION ➤ Remove barriers ➤ Time ➤ Lack of knowledge/education on assessment ➤ Fear of failure ➤ Incentivize assessment ➤ Grants, rewards, recognition ➤ Give staff/faculty the freedom to determine what and how they will assess ➤ Demonstrate that assessment results will be used
  28. 28. CREATE SHORT-TERM WINS (OR “THE PERFECT IS THE ENEMY OF THE GOOD”) ➤ Start small and simple ➤ Assessments that are easy to do but yield valuable learning ➤ What are we curious about? ➤ Debrief together and learn from each experience ➤ Talk about changes made based on assessment results
  29. 29. CONSOLIDATE IMPROVEMENTS AND INSTITUTIONALIZE NEW APPROACHES ➤ Develop formal mechanisms for reporting and sharing assessment results ➤ Build mechanisms for discussing assessment results within the library ➤ Demonstrate evidence-based decision-making ➤ Build assessment into planning ➤ Build assessment into reward structures (tenure?)
  30. 30. THERE ARE LIMITATIONS WITH THE LITERATURE WE RELY ON TO MAKE CLAIMS ABOUT BUILDING AN ASSESSMENT CULTURE
  31. 31. 1. Oakleaf, M. & Hinchliffe, L. J. (2008). Assessment cycle or circular file: Do academic librarians use information literacy assessment data? In Proceedings of the 2008 Library Assessment Conference: Building Effective, Sustainable, Practical Assessment. 2. Ndoye, A. & Parker, M. A. (2010). "Creating and sustaining a culture of assessment." Planning for Higher Education, 38(2), 28-39. 3. Hiller, S., Kyrillidou, M., & Self, J. (2008). "When the evidence is not enough: Organizational factors that influence effective and successful library assessment." Performance Measures and Metrics, 9(3), 223-230. 4. Wright, S. & White, L. (2007). SPEC Kit 303, Library Assessment. Washington, D.C.: Association of Research Libraries. Survey sent to an infolit listserv Based on their experience with research libraries that want to build an assessment culture Sent to an assessment listserv Survey of ARLs only
  32. 32. Our goal: A systematic survey of culture of assessment and investigation of factors that facilitate or hinder libraries. A survey of libraries, not librarians
  33. 33. THE SURVEY ➤ 60 Question Survey ➤ 10 descriptive questions about the institution or the individual responding for the institution ➤ 47 yes/no questions ➤inquiry about presence/absence of 16 facilitating factors named in other publications ➤does the presence/absence facilitate or hinder culture of assessment ➤ 3 open-ended questions about what most facilitates or hinders creation of a culture of assessment ➤ Respondents presented with different questions depending on their answers to presence/absence of facilitating factors (total questions for each respondent = 41 or 42)
  34. 34. FACILITATING FACTORS 1. Campus-wide assessment initiative 2. Library involvement in campus- wide assessment initiative 3. Clear expectations for assessment in the library 4. Assessment is a priority of administration 5. Library has adopted learning outcomes 6. Library has an assessment plan 7. Library has a shared understanding of the purpose of assessment 8. Library faculty/staff are supported in doing assessment 9. Education and training about assessment are supported 10. Library has assessment expertise in- house 11. Library has systems/technologies to support assessment 12. Library leadership offers explicit support to get staff doing assessment 13. Data is available to interested parties 14. Leadership uses assessment data in decision-making 15. Librarians use results to improve practice 16. Library culture is user-focused
  35. 35. Stage 1: Winter 2013 Libraries at all BA/MA/PhD-granting institutions Received a 42% response rate
  36. 36. Results presented at ACRL 2013 and published in College & Research Libraries Farkas, Meredith Gorran, Lisa Janicke Hinchliffe, and Amy Harris Houk. "Bridges and Barriers: Factors Influencing a Culture of Assessment in Academic Libraries." College & Research Libraries 76.2 (2015): 150-169.
  37. 37. BUT COMMUNITY COLLEGES ARE DIFFERENT
  38. 38. Stage 2: Spring 2014 Libraries at all U.S. public community colleges Received a 32% response rate (n=292)
  39. 39. WOULD YOU DESCRIBE YOUR LIBRARY AS HAVING A CULTURE OF ASSESSMENT, WHERE ASSESSMENT IS A REGULAR PART OF YOUR INSTITUTIONAL PRACTICE?
  40. 40. THIS IS HIGHER THAN AT FOUR YEAR AND ABOVE INSTITUTIONS?
  41. 41. IS THERE A CAMPUS-WIDE ASSESSMENT INITIATIVE AT YOUR COLLEGE? Also higher than at 4 year and above institutions (84.9%) Only 80% of these libraries are involved in the campus-wide assessment initiative
  42. 42. IS ASSESSMENT A PRIORITY OF LIBRARY ADMINISTRATION? Higher than at 4 year and above institutions (77.8%)
  43. 43. IS THERE A SHARED UNDERSTANDING OF THE PURPOSE OF ASSESSMENT IN YOUR LIBRARY? Higher than at 4 year and above institutions (54.3%)
  44. 44. ARE THERE CLEAR EXPECTATIONS FOR ASSESSMENT IN YOUR LIBRARY? Higher than at 4 year and above institutions (45.7%)
  45. 45. DOES YOUR LIBRARY HAVE AN ASSESSMENT PLAN? Higher than at 4 year and above institutions (40.6%)
  46. 46. ARE LIBRARY FACULTY/STAFF ADEQUATELY SUPPORTED IN THEIR ASSESSMENT WORK? Higher than at 4 year and above institutions (55.8%)
  47. 47. DOES YOUR LIBRARY HAVE THE NECESSARY SKILLS IN-HOUSE TO DEVELOP AND CONDUCT MEANINGFUL ASSESSMENTS AND ANALYZE THE RESULTS? Higher than at 4 year and above institutions (56%)
  48. 48. DOES THE LIBRARY LEADERSHIP USE ASSESSMENT DATA SYSTEMATICALLY IN DECISION-MAKING AND PLANNING? Higher than at 4 year and above institutions (58.5%)
  49. 49. DOES THE LIBRARY STAFF USE ASSESSMENT DATA SYSTEMATICALLY IN DECISION-MAKING AND PLANNING? Higher than at 4 year and above institutions (71%)
  50. 50. ON WHICH FACILITATING FACTORS DID COMMUNITY COLLEGES SCORE SIGNIFICANTLY (P<0.05) HIGHER? ➤Is there a shared understanding of the purpose of assessment in your library? ➤Are library faculty/staff adequately supported in their assessment work? ➤Does the library leadership use assessment data systematically in decision-making and planning.
  51. 51. WHAT FACTORS ARE ASSOCIATED WITH HAVING A CULTURE OF ASSESSMENT?
  52. 52. REGIONAL ACCREDITING ASSOCIATION IS ASSOCIATED WITH A CULTURE OF ASSESSMENT
  53. 53. TENURE STATUS AND FACULTY STATUS IS ASSOCIATED WITH A CULTURE OF ASSESSMENT p=0.02 This was not the case in the study of four year and above institutions
  54. 54. HAVING A CAMPUS-WIDE ASSESSMENT INITIATIVE IS ASSOCIATED WITH A CULTURE OF ASSESSMENT P<0.001
  55. 55. ALL OF THE FACILITATING FACTORS ARE SIGNIFICANTLY ASSOCIATED WITH A CULTURE OF ASSESSMENT
  56. 56. FACTORS WHOSE PRESENCE IS MOST STRONGLY ASSOCIATED WITH A CULTURE OF ASSESSMENT ➤Clear expectations for assessment in the library (90% who have it have an assessment culture) ➤Assessment plan (82%) ➤Shared understanding of the purpose of assessment in the library (81%) ➤Library leadership uses assessment data (78%) ➤Library leadership offers explicit support (77%)
  57. 57. FACTORS MOST STRONGLY ASSOCIATED WITH NOT HAVING A CULTURE OF ASSESSMENT ➤Assessment is not a priority of administration (77% who report this do not have an assessment culture) ➤Librarians do not use assessment data (72%) ➤Assessment data is not available to interested parties (74%) ➤Library leadership does not use assessment data (70%) ➤No shared understanding of the purpose of assessment (69%)
  58. 58. WHAT FACTORS ARE MOST OFTEN PERCEIVED AS FACILITATING? ➤ Clear expectations for assessment in the library (94% who perceive that this facilitates their assessment culture have an assessment culture) ➤ Library has an assessment plan ➤ Librarians use assessment data to improve practice ➤ Library leadership uses assessment data ➤ Existence of learning outcomes
  59. 59. OPEN-ENDED RESPONSES: WHAT ARE THE MOST IMPORTANT FACILITATING FACTORS? ➤Institution-wide emphasis on assessment ➤Administrative priority and/or support ➤Accreditation ➤Organizational culture ➤Information literacy as a campus-wide outcome
  60. 60. OPEN-ENDED RESPONSES: WHAT MOST HINDERS CREATING AN ASSESSMENT CULTURE? ➤Lack of sufficient staffing ➤Lack of time ➤Faculty or staff have not made assessment a priority – either due to lack of interest or resistance to assessment ➤Lack of expertise/access to training ➤Institution doesn’t prioritize assessment
  61. 61. REFLECTIONS
  62. 62. “CULTURE OF ASSESSMENT” IS IN THE EYE OF THE BEHOLDER http://www.flickr.com/photos/kylemay/1573158163/
  63. 63. WITHOUT AN INSTITUTIONAL COMMITMENT TO ASSESSMENT, A CULTURE OF ASSESSMENT IS FAR LESS LIKELY TO EXIST Institutions with a campus-wide assessment initiative 66% of libraries report having an assessment culture Institutions with no campus-wide assessment initiative 36% of libraries report having an assessment culture
  64. 64. THERE ARE SIGNIFICANT ASSOCIATIONS WITH TENURE AND FACULTY STATUS THAT ARE NOT PRESENT AT FOUR YEAR AND ABOVE INSTITUTIONS
  65. 65. HAVING A PLAN IS CRITICAL Fewer than half of libraries... ➤ Have clear expectations for assessment (52%) ➤ Have an assessment plan (45%) Of libraries reporting clear expectations and an assessment plan, 92% report having a culture of assessment (versus 32% of those who report the absence of both)
  66. 66. HOW ADMINISTRATION PROMOTES ASSESSMENT MAKES A DIFFERENCE
  67. 67. Of those indicating assessment is a priority of administration 74% report having a culture of assessment 
 Of those indicating administration uses assessment data in decision- making 
 78% report having a culture of assessment 
 Of those indicating administration adequately supports faculty/staff in assessment work 
 69% report having a culture of assessment 
 Of those indicating a library has all three factors 
 88% report having a culture of assessment
  68. 68. HAVING INSTRUCTION AND ASSESSMENT LEADERS ALSO HELPS
  69. 69. Of those academic libraries with an instruction coordinator 74% report having a culture of assessment 
 Of those with an assessment coordinator 
 87% report having a culture of assessment 
 Of those with an instruction committee 
 71% report having a culture of assessment 
 Of those with an assessment committee 
 87% report having a culture of assessment Of those with none of these positions 
 57% report having a culture of assessment
  70. 70. MOST COMMUNITY COLLEGE LIBRARIES ARE ASSESSMENT LEADERS
  71. 71. CONSIDER ➤ What has helped your library move toward (or achieve!) a culture of assessment? ➤ What has hindered your library in moving toward a culture of assessment? ➤ Which of these facilitating factors seem most important for you to work toward at your institution?
  72. 72. FUTURE RESEARCH?
  73. 73. Questions? Comments? Challenges? Meredith Farkas Portland Community College Email: meredith.farkas at pcc.edu Twitter: librarianmer

×