E(p)owering Your Institution


Published on

Presentation describes creating a culture of assessment at your institution. And outlines a study of four analytical questions developed by the AAHSL Task Force On Qualitative Assessment.

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Coefficient of variation is useful for comparing the variability of several different samples: (s/mean) x 100
  • E(p)owering Your Institution

    1. 1. E (p) owering Your Institution A Mixed-Model Approach to Assessment © 2004 Douglas Joubert Douglas James Joubert Greenblatt Library
    2. 2. Objectives <ul><li>Creating a culture of assessment </li></ul><ul><li>H 1 and H 3 </li></ul><ul><ul><li>Data Considerations </li></ul></ul><ul><ul><li>Sample Characteristics </li></ul></ul><ul><ul><li>Statistical Analysis </li></ul></ul><ul><li>Collaboration & Innovation </li></ul>
    3. 3. Illusions of Certainty 1 <ul><li>Retaining and growing their customer base, and focusing more energy on meeting their customers’ expectations is the only way for academic libraries to survive in this volatile environment </li></ul><ul><li>--Rowena Cullen, 2001 </li></ul>
    4. 4. Creating a Culture of Assessment Culture of Assessment Benchmarking ARL Service Academy AAHSL Task Force Questions LibQUAL+ “ Aspirational” “ Comparator”
    5. 5. Speaking a common language Q ua n t i t a t i v e Input Output Outcomes Qualitative Performance Indicators Assessment
    6. 6. Influential Players Input Output Outcomes Qualitative Performance Indicators Assessment
    7. 7. Mixed Model Approaches
    8. 8. Linear Model of Research Adapted from Flick, 2002 Theory H 1 Sampling Analysis Validation
    9. 9. Circular Model of Research Adapted from Glaser & Strauss, in Flick, 2002 Comparing Comparing Comparing Sampling Sampling Case Textual analysis Case Textual analysis Case Textual analysis PA T
    10. 10. Quantitative Research Quick! get to the cheese To Maze
    11. 11. Qualitative Research I really don’t like cheese I need a vacation Do these frames make me look smart? To Maze Why am I here?
    12. 12. Qualitative Research “ Triangulation ” “ Grounded Theory” “ Bricoleur ”
    13. 13. Statistical Inference Flowchart Adapted from Rosner, 2000 Two Variables? Both continuous? Prediction? r/s between 2 variables? Both variables normal? Pearson correlation Rank correlation Linear regression Y Y N Y Y N Y
    14. 14. Overall Data Considerations 24 th AAHSL 25 th AAHSL Local SPSS Tables Local SPSS Tables Local SPSS Tables 2002 LibQUAL+
    15. 15. Overall Data Considerations N > 13,000 24 th AAHSL 25 th AAHSL Local SPSS Tables Local SPSS Tables Local SPSS Tables 2002 LibQUAL+
    16. 16. Overall Data Considerations Descriptive 24 th AAHSL 25 th AAHSL Local SPSS Tables Local SPSS Tables Local SPSS Tables 2002 LibQUAL+
    17. 17. Overall Data Considerations <ul><li>Around 120 cases </li></ul><ul><li>Formatted for reading, not for statistical analysis </li></ul><ul><li>Formatting must be monitored </li></ul><ul><li>Are missing scores “systematic” or “random”? </li></ul>AAHSL Data
    18. 18. Overall Data Considerations <ul><li>Over 13,000 cases </li></ul><ul><li>In SPSS, built for statistical analysis </li></ul><ul><li>Formatting and Missing scores built into the design of the variables </li></ul>LibQUAL+ Data
    19. 19. Overall Data Considerations <ul><li>Use independent reviewer before data migration </li></ul><ul><li>Determine pattern of missing data before migration </li></ul><ul><li>How are you going to deal with missing scores </li></ul><ul><li>Easier to spot errors with 120 cases than with 13,000 </li></ul>AAHSL Data LibQUAL Data
    20. 20. The First AAHSL Group <ul><li>How do satisfaction ratings differ for various reporting structures? </li></ul><ul><li>How does the size of library staff affect satisfaction ratings? </li></ul><ul><li>How does the number of constituents affect satisfaction ratings? </li></ul><ul><li>How does the ratio of staff to constituents affect satisfaction ratings? </li></ul>Source: AAHSL Task Force on Quality Assessment
    21. 21. Hypothesis 1 Hypothesis I (H 1 ) was operationalized as, “What is the affect of library reporting structure on the mean measure of overall quality of service?”
    22. 22. H 1 Data Considerations <ul><li>Not originally included in the study design </li></ul><ul><li>Included to rectify a priori considerations for the remaining three hypotheses </li></ul>“ person-level subscales” “ institution-level subscales” means by dimension for each person means by dimension for each institution
    23. 23. H 1 Data Considerations <ul><li>Medical School </li></ul><ul><li>Other Health Science School </li></ul><ul><li>University Library </li></ul><ul><li>Health Science Center Administration </li></ul><ul><li>University Administration </li></ul><ul><li>Other </li></ul><ul><li>Question 3 (Section 5.6) of the 2002 LibQUAL </li></ul>Library Reporting Structure Overall Quality of Service + Defined by the 2001-2002 AAHSL Library Statistics Survey
    24. 24. H 1 Sample Characteristics <ul><li>35 AAHSL institutions contributed data for the 24th Annual Survey and participated in 2002 LibQUAL+. </li></ul>Missing (2) University Library (3) Other (2) Medical School (12) University Administration (2) Health Science (14)
    25. 25. H 1 Unit of Analysis <ul><li>This analysis needed to demonstrate if OVERSAT 1 differed by reporting structure. </li></ul><ul><li>H 01 required an analytical method for dealing with the institutional differences </li></ul>Variable reserved from LibQUAL data Library reporting structure Differences not explained by the grouping Differences explained by the grouping +
    26. 26. H 1 Unit of Analysis <ul><li>A Linear Mixed Model (LMM) allowed me to control for fixed and random effects </li></ul>OVERSAT [DV] = LIBSTRUC [IV1] + INSTID(LIBSTRUC) [IV2] Fixed-effects Random-effects
    27. 27. Descriptive Statistics for Overall Quality of Service 19.2% 1.408 7.35 13127 Total 19.4% 1.393 7.19 759 University Library 20.2% 1.457 7.22 649 Other 17.6% 1.323 7.50 5972 Health Science Center 22.0% 1.583 7.18 879 University Administration 20.1% 1.451 7.23 4868 Medical School CV (%) SD Mean Count Library Reports To How would you rate overall quality of the service provided by the library?
    28. 28. Type III Tests of Fixed Effects for Overall Quality of Service a a. Dependent Variable: How rate overall quality of the service? .464 .924 27.038 4 LIBSTRUC .000 6640.536 26.927 1 Intercept Sig. F Den df Num df Source
    29. 29. Hypothesis 3 Hypothesis 3 (H 03 ) was operationalized as, “How does the number of constituents served affect satisfaction ratings?”
    30. 30. H 3 Data Considerations Academic and Primary Hospital Staff (STAFFTOT) = NO Secondary Clientele Served Table Q4 Primary Clientele Served Table Q3A-Q3C Total constituents served (CONSTOT) <ul><li>Faculty (FACTOT), </li></ul><ul><li>Interns, residents, and fellows (IRF) </li></ul><ul><li>Students (STUTOT) </li></ul>
    31. 31. H 3 Data Considerations <ul><li>From Question 3 (Section 5.6) of the 2002 LibQUAL computed mean overall quality of service by institution </li></ul>“ person-level subscales” “ institution-level subscales” Mean OQS for each person Mean OQS for each institution
    32. 32. H 3 Unit of Analysis <ul><li>Because of the findings from H 1 , H 3 could be analyzed with correlation and simple linear regression models </li></ul>CONSTOT M_OQS by Inst IV DV
    33. 33. Descriptive Statistics for H 3
    34. 34. Dealing with Outliers <ul><li>Distinct score relative to the bulk of the distribution </li></ul><ul><li>Not all outliers are influential </li></ul><ul><li>Cook’s D is an index based on the F-statistic </li></ul><ul><li>Cook’s D > 1 deemed an outlier </li></ul><ul><li>Cook’s D = 3.10679 </li></ul>X-Direction IV
    35. 35. Dealing with Outliers <ul><li>Studentized (SRESID) residual provides information about outliers in the “Y” direction </li></ul><ul><li>Framed within the context of the SND, so scores > ± 3 </li></ul><ul><li>Sign is irrelevant because we are only interested in how far the point is in the tail of the distribution </li></ul><ul><li>SDRESID =1.75842 </li></ul>Y D i r e c t i o n DV
    36. 36. Dealing with Outliers <ul><li>Removing outliers in an objective method is ethical </li></ul><ul><li>Filter variable ($_filter) created to exclude score 4 from the analysis </li></ul><ul><li>Correlation and linear regression was performed with (non-transformed) and without (transformed) the data from case 4 </li></ul>DV IV
    37. 37. Transformed Correlations 32 N .027 Sig. (2-tailed) -.391* Pearson Correlation Total Constituents Served LibQUAL+ Mean Overall Quality of Service by Institution * Correlation is significant at the 0.05 level (2-tailed).
    38. 38. Total Constituents as a Predictor of Measures of Overall Quality of Service .153 .391 1,30 5.423 -0.391 0.355 -8.10 R 2 R df F  Std. Err B <ul><li>Predictors: (Constant), Total Constituents Served </li></ul><ul><li>Dependent Variables: Mean Overall Quality by Institution </li></ul>
    39. 39. Total Constituents as a Predictor of Measures of Overall Quality of Service Total Constituents accounts for 15.3% of the variance of M_OQS X Y .153 .391 R 2 R
    40. 40. Further Investigation <ul><li>Controversy on Significance Testing </li></ul><ul><ul><li>Evaluating Results Replicability </li></ul></ul><ul><ul><li>Confidence Intervals for Effect Size </li></ul></ul><ul><ul><li>Emphasize Effect-size Interpretation </li></ul></ul><ul><ul><li>β versus Structured Coefficient exploration for effect size </li></ul></ul>Thompson, 2002, 1996, and 1992
    41. 41. Collaboration & Innovation Building Connections Biostatistics AAHSL Peer Networks New Directions Qualitative 2005 LibQUAL Local Projects
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.