The document outlines a mixed-model approach to assessment in academic libraries, emphasizing the need to create a culture of assessment to enhance service quality and meet customer expectations. It discusses statistical analysis methods used to evaluate library service quality, including the impact of reporting structures and the number of constituents served on satisfaction ratings. The findings from various surveys, including LibQUAL+, are used to support the hypotheses related to quality of service and institutional characteristics.

1. E (p) owering Your Institution A Mixed-Model Approach to Assessment © 2004 Douglas Joubert Douglas James Joubert Greenblatt Library

2. Objectives Creating a culture of assessment H 1 and H 3 Data Considerations Sample Characteristics Statistical Analysis Collaboration & Innovation

3. Illusions of Certainty 1 Retaining and growing their customer base, and focusing more energy on meeting their customers’ expectations is the only way for academic libraries to survive in this volatile environment --Rowena Cullen, 2001

4. Creating a Culture of Assessment Culture of Assessment Benchmarking ARL Service Academy AAHSL Task Force Questions LibQUAL+ “ Aspirational” “ Comparator”

5. Speaking a common language Q ua n t i t a t i v e Input Output Outcomes Qualitative Performance Indicators Assessment

6. Influential Players Input Output Outcomes Qualitative Performance Indicators Assessment

7. Mixed Model Approaches

8. Linear Model of Research Adapted from Flick, 2002 Theory H 1 Sampling Analysis Validation

9. Circular Model of Research Adapted from Glaser & Strauss, in Flick, 2002 Comparing Comparing Comparing Sampling Sampling Case Textual analysis Case Textual analysis Case Textual analysis PA T

10. Quantitative Research Quick! get to the cheese To Maze

11. Qualitative Research I really don’t like cheese I need a vacation Do these frames make me look smart? To Maze Why am I here?

12. Qualitative Research “ Triangulation ” “ Grounded Theory” “ Bricoleur ”

13. Statistical Inference Flowchart Adapted from Rosner, 2000 Two Variables? Both continuous? Prediction? r/s between 2 variables? Both variables normal? Pearson correlation Rank correlation Linear regression Y Y N Y Y N Y

14. Overall Data Considerations 24 th AAHSL 25 th AAHSL Local SPSS Tables Local SPSS Tables Local SPSS Tables 2002 LibQUAL+

15. Overall Data Considerations N > 13,000 24 th AAHSL 25 th AAHSL Local SPSS Tables Local SPSS Tables Local SPSS Tables 2002 LibQUAL+

16. Overall Data Considerations Descriptive 24 th AAHSL 25 th AAHSL Local SPSS Tables Local SPSS Tables Local SPSS Tables 2002 LibQUAL+

17. Overall Data Considerations Around 120 cases Formatted for reading, not for statistical analysis Formatting must be monitored Are missing scores “systematic” or “random”? AAHSL Data

18. Overall Data Considerations Over 13,000 cases In SPSS, built for statistical analysis Formatting and Missing scores built into the design of the variables LibQUAL+ Data

19. Overall Data Considerations Use independent reviewer before data migration Determine pattern of missing data before migration How are you going to deal with missing scores Easier to spot errors with 120 cases than with 13,000 AAHSL Data LibQUAL Data

20. The First AAHSL Group How do satisfaction ratings differ for various reporting structures? How does the size of library staff affect satisfaction ratings? How does the number of constituents affect satisfaction ratings? How does the ratio of staff to constituents affect satisfaction ratings? Source: AAHSL Task Force on Quality Assessment

21. Hypothesis 1 Hypothesis I (H 1 ) was operationalized as, “What is the affect of library reporting structure on the mean measure of overall quality of service?”

22. H 1 Data Considerations Not originally included in the study design Included to rectify a priori considerations for the remaining three hypotheses “ person-level subscales” “ institution-level subscales” means by dimension for each person means by dimension for each institution

23. H 1 Data Considerations Medical School Other Health Science School University Library Health Science Center Administration University Administration Other Question 3 (Section 5.6) of the 2002 LibQUAL Library Reporting Structure Overall Quality of Service + Defined by the 2001-2002 AAHSL Library Statistics Survey

24. H 1 Sample Characteristics 35 AAHSL institutions contributed data for the 24th Annual Survey and participated in 2002 LibQUAL+. Missing (2) University Library (3) Other (2) Medical School (12) University Administration (2) Health Science (14)

25. H 1 Unit of Analysis This analysis needed to demonstrate if OVERSAT 1 differed by reporting structure. H 01 required an analytical method for dealing with the institutional differences Variable reserved from LibQUAL data Library reporting structure Differences not explained by the grouping Differences explained by the grouping +

26. H 1 Unit of Analysis A Linear Mixed Model (LMM) allowed me to control for fixed and random effects OVERSAT [DV] = LIBSTRUC [IV1] + INSTID(LIBSTRUC) [IV2] Fixed-effects Random-effects

27. Descriptive Statistics for Overall Quality of Service 19.2% 1.408 7.35 13127 Total 19.4% 1.393 7.19 759 University Library 20.2% 1.457 7.22 649 Other 17.6% 1.323 7.50 5972 Health Science Center 22.0% 1.583 7.18 879 University Administration 20.1% 1.451 7.23 4868 Medical School CV (%) SD Mean Count Library Reports To How would you rate overall quality of the service provided by the library?

28. Type III Tests of Fixed Effects for Overall Quality of Service a a. Dependent Variable: How rate overall quality of the service? .464 .924 27.038 4 LIBSTRUC .000 6640.536 26.927 1 Intercept Sig. F Den df Num df Source

29. Hypothesis 3 Hypothesis 3 (H 03 ) was operationalized as, “How does the number of constituents served affect satisfaction ratings?”

30. H 3 Data Considerations Academic and Primary Hospital Staff (STAFFTOT) = NO Secondary Clientele Served Table Q4 Primary Clientele Served Table Q3A-Q3C Total constituents served (CONSTOT) Faculty (FACTOT), Interns, residents, and fellows (IRF) Students (STUTOT)

31. H 3 Data Considerations From Question 3 (Section 5.6) of the 2002 LibQUAL computed mean overall quality of service by institution “ person-level subscales” “ institution-level subscales” Mean OQS for each person Mean OQS for each institution

32. H 3 Unit of Analysis Because of the findings from H 1 , H 3 could be analyzed with correlation and simple linear regression models CONSTOT M_OQS by Inst IV DV

33. Descriptive Statistics for H 3

34. Dealing with Outliers Distinct score relative to the bulk of the distribution Not all outliers are influential Cook’s D is an index based on the F-statistic Cook’s D > 1 deemed an outlier Cook’s D = 3.10679 X-Direction IV

35. Dealing with Outliers Studentized (SRESID) residual provides information about outliers in the “Y” direction Framed within the context of the SND, so scores > ± 3 Sign is irrelevant because we are only interested in how far the point is in the tail of the distribution SDRESID =1.75842 Y D i r e c t i o n DV

36. Dealing with Outliers Removing outliers in an objective method is ethical Filter variable ($_filter) created to exclude score 4 from the analysis Correlation and linear regression was performed with (non-transformed) and without (transformed) the data from case 4 DV IV

37. Transformed Correlations 32 N .027 Sig. (2-tailed) -.391* Pearson Correlation Total Constituents Served LibQUAL+ Mean Overall Quality of Service by Institution * Correlation is significant at the 0.05 level (2-tailed).

38. Total Constituents as a Predictor of Measures of Overall Quality of Service .153 .391 1,30 5.423 -0.391 0.355 -8.10 R 2 R df F  Std. Err B Predictors: (Constant), Total Constituents Served Dependent Variables: Mean Overall Quality by Institution

39. Total Constituents as a Predictor of Measures of Overall Quality of Service Total Constituents accounts for 15.3% of the variance of M_OQS X Y .153 .391 R 2 R

40. Further Investigation Controversy on Significance Testing Evaluating Results Replicability Confidence Intervals for Effect Size Emphasize Effect-size Interpretation β versus Structured Coefficient exploration for effect size Thompson, 2002, 1996, and 1992

41. Collaboration & Innovation Building Connections Biostatistics AAHSL Peer Networks New Directions Qualitative 2005 LibQUAL Local Projects