Aea Conference Presentation November 2007 J Sheldon


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Aea Conference Presentation November 2007 J Sheldon

  1. 1. Contextual Variables In Elementary Schools Influencing Organizational Learning & Predicting Evaluative Inquiry Jeffrey Sheldon, M. A, Ed. M. School of Behavioral & Organizational Sciences Claremont Graduate University The Claremont Colleges 9 November 2007
  2. 2. Overview1) Specific Aims 1) Recruitment2) Background & Significance 2) Data Collection3) Gaps in the Research 3) Coverage4) Research Questions 4) Participating Schools5) Hypotheses 5) Individual Respondents6) Research Design 6) Results – Research Question 17) Instrumentation 7) Results – Research Question 28) Analysis Strategy: Phase I 8) Results – Research Question 39) Analysis Strategy: Phase II 9) Findings10) Missing Values 10) Discussion11) Sampling 11) Outcomes & ImplicationsThis research was made possible through the generosity of Dr. Barbara DeHart,Professor and Director of the Urban Leadership Program, School of EducationalStudies, Claremont Graduate University, Claremont, CA 2
  3. 3. Specific Aims Exploring elementary schools for contextual variables known to influence organizational learning and evaluative inquiry. Predicting from contextual variables present whether evaluative inquiry for organizational learning is likely to take place and which variable is the best predictor. Using a relatively new quantitative instrument to refute criticism about using quantitative instruments to measure organizational learning and providing evidence of its utility in school-based organizational learning/evaluation research. Illuminating for each participating school the contextual variables necessary for organizational learning and evaluation they have in place, and those that either could be put in place or strengthened. Contributing to a better understanding of the linkage between evaluative inquiry and organizational learning. 3
  4. 4. Background & Significance Keefe & Howard (1997); Reed, Kinzie, & Ross (2001); Watkins & Marsick (1999): School reform efforts most successful when schools are learning organizations and school personnel are change agents. Lashway (1998): Schools dedicating themselves to systematic, collaborative problem-solving can continually develop and implement new ideas, thereby not just improving, but transforming themselves. Mancini & Marek (2004): An organizational learning approach is necessary in schools to promote sustainability of any changes shown to be effective in improving academic achievement. Trubowitz (2005): Schools should resist external pressures that compel immediate and precipitous action unsupported by prior thinking. Bowen, Rose, & Ware (2006): Organizational learning is the best way to address critical issues facing schools today because they must be ready to receive and act upon new information for change to occur. 4
  5. 5. Background & Significance Fiol & Lyles (1985); Senge (1990): When a context supportive of organizational learning is in place an organization is said to be a learning organization. Cousins & Leithwood (1986); Cousins & Shulha (2002): Context factors influence how and to what extent learning will occur in an organization. Shulha & Cousins (1997): Culture, leadership, communication, teamwork, and systems/structures are considered to be the elements of organizational context necessary for evaluative inquiry and organizational learning. Lam & Pang (2003): The prevailing influence of organizational learning process comes primarily from internal school conditions - transformational leadership, supportive structure and culture are key factors promoting the learning process. Silins & Mulford (2004): Four components of organizational learning in schools - collaborative climate, taking initiatives and risks, improving school practices, and professional development. Preskill & Torres (1999); Russ-Eft, Atwood, & Egherman (2002): Evaluative inquiry contributes to individual, group, and organizational learning. Preskill & Torres (2000): An organization’s culture and context influence the extent to which evaluative inquiry occurs in support of learning and decision making. 5
  6. 6. Gaps in the Research Botcheva, et al. (2002); Goh et al. (2005); Lee (1999); Scribner et al. (1999): Schools as learning organizations studied at middle and secondary levels, but little evidence that elementary schools are learning organizations. Goh et al. (2005): Few published empirical studies showing views, perspectives and importance teachers and administrators attach to organizational learning practices and activities. Alkin & Taut (2003): It would be insightful to investigate how contextual factors within organizations influence evaluative inquiry. Cousins et al. (2004): Knowledge base linking evaluative inquiry to organizational learning relatively immature. Bryk, Camburn, & Louis (1999); Mebane & Galassi (2003): Quantitative tools addressing organizational learning suffer from abstract definitions of constructs, lack clearly articulated conceptual frameworks reflecting multiple dimensions of learning organizations, and are adapted from business and industry. 6
  7. 7. Research Questions1) Which, if any, contextual variables necessary for organizational learning are present in elementary schools:  Culture (collaboration and problem solving; risk taking; participatory decision making)?  Leadership?  Systems and structures (open and accessible work environment; rewards and recognition system and practice; relationship of work to organizational goals)?  communication (availability of information; dissemination of information)?  Teams?4) Can evaluative inquiry be predicted from those contextual variables present in elementary schools?6) If evaluative inquiry can be predicted, what is the best predictor? 7
  8. 8. Hypotheses• A majority of sampled elementary schools do not have a sufficient number of contextual variables to support organizational learning and therefore are not learning organizations.• Evaluative inquiry can be predicted from each or some combination of organizational learning contextual variables present.• In elementary schools that are characterized as learning organizations, the best predictor of evaluative inquiry is leadership. 8
  9. 9. Research Design Quantitative. Exploratory. Two-phases: 1) Used a questionnaire to determine: • Which, if any, contextual variables necessary for organizational learning were present in a small sample of elementary schools. • Whether schools could be considered to be learning organizations. • Whether schools were either receptive to or engaged in evaluative inquiry. 1) Used multiple regression to predict whether evaluative inquiry was likely in a school that had contextual variables for organizational learning in place and determined which one was the best predictor. 9
  10. 10. Instrumentation Preskill & Torres’ (2000) highly reliable (Cronbach’s Alpha = .97) Readiness for Organizational Learning and Evaluation Instrument (ROLE). Operationalizes constructs of organizational learning - culture, leadership, systems and structures, communication, teams – and evaluation. Unlike other quantitative instruments ROLE: • Measures organizational learning constructs that are concrete and well- established in the extant literature. • Reflects the multiple dimensions of organizational learning. • Can be used across a wide spectrum of organizational types and sectors. Two other instruments measure organizational learning, but not evaluation: • Bowen & Powers’ (2003) School Success Profile-Learning Organizations. • Ellis, Globerson, & Parson’s (1997) Organizational Learning Mechanisms Questionnaire. 10
  11. 11. Analysis Strategy: Phase I Unit of analysis: The school. Based, in-part, on Silins & Mulford’s (2004) study focusing on school-level variables associated with leadership and organizational learning. Descriptive statistics used to determine which, if any, contextual variables necessary for organizational learning were present in sample and to what extent (and by whom) evaluative inquiry was taking place. Results of each ROLE aggregated across individual respondents by school to come up with each school’s composite score. Independent sample t - tests used to determine if certain attributes (e.g., staff gender, grade level configuration, or school district) covaried with organizational learning and evaluation contextual variables present in sample. 11
  12. 12. Analysis Strategy: Phase II Based, in part, on Lam & Pang’s (2003) study of organizational learning in schools. Multiple regression used to determine 1) whether engagement in evaluation activities (i.e., evaluative inquiry) could be predicted from contextual variables discovered in Phase I, and 2) the best predictor. • The 5 constructs of organizational learning = independent variables. • Evaluation = dependent variable. To increase precision in making predictions, constructs weighted and used as a conservative adjustment for the unequal number of items within each construct. After weighting, independent variables entered, in no predetermined order, as a block into the model. A Postieri analyses. 12
  13. 13. Missing Values A priori decisions for making missing value corrections to avoid the elimination of data: • For constructs comprised of subconstructs with 5 items, 3 of 5 had to be answered. • For constructs comprised of subconstructs with 4 items, 3 of 4 had to be answered. • For constructs comprised of subconstructs with 3 items, 2 of 3 had to be answered. • A majority of items had to be answered within whole constructs. Each respondent’s scores for whole construct added together and divided by the number of items excluding zero scores. That score then entered into the unweighted construct score prior to weighting. 13
  14. 14. Sampling Public elementary schools with attendant administrators, faculty, and staff comprised sample population. Schools were to be drawn from a sampling frame of 778 elementary schools within 80 districts across 4 counties. Sampling was to have been based on probability. Did not realize: • Multiple permissions needed before study could be conducted in schools • The difficulty of gaining permission to conduct school-based research Sampling strategy became one of convenience. 12 administrators, 156 classified and support staff, and 290 teachers totaling 458 school personnel could have responded. 14
  15. 15. Recruitment Three-tiered approach to introduce study, gain endorsement and support, and obtain necessary authorizations/permissions. 1) Letter sent to Board of Education Superintendent in each of four counties. 3) Letters sent to 80 local school district superintendents in three counties. • 17 Affirmative responses • 10 Negative responses • 2 Non-committal responses • 51 Non-responses 2) Selected 11 local school districts randomly using the Research Randomizer web site and subsequently sent letters of invitation to 110 principals. 15
  16. 16. Data CollectionFollowed Dillman’s (2000) best-practice in survey research. Box of ROLE packets sent within a week of receiving written permission and confirmation a principal wanted school to participate. Follow-up “thank-you/reminder to fill out the survey if you haven’t already” post-cards and emails sent out five days after return deadline to prompt principals and in turn, their staff. In lieu of re-sending ROLE packets to schools as suggested by Dillman email sent to principals letting them know that new ROLE packets would be sent immediately if necessary; PI did not have the name of individual respondents so all communication went through principals. 16
  17. 17. Coverage Budget constraints necessitated trade-offs regarding number of ROLE packets that could be sent to each school. ROLE packets sent covered approximately 60 percent of the potential respondents at each school. Two schools were small enough in staff size for 100 percent coverage. Countervailing argument for covering fewer schools completely dismissed in favor of gaining a broader sample of schools by type, district, and county for greater generalizability. But, no guarantee complete coverage in fewer schools would have improved the response rate. 17
  18. 18. Participating Schools Schools represented 5 of 80 districts (6.3 %) and three of four southern California counties (75 %) initially identified and targeted for inclusion. • 3 schools from 2 districts in County A. • 5 schools from 2 districts in County B. • 1 school in County C. All 9 schools met the definition of an elementary school as determined prior to the study. • 2 schools configured K – 6. • 7 schools configured K – 5. All schools met the criteria of a small organization (< 1,000 students; < 100 staff). • Student enrollment ranged from 218 to 871, averaging 666. • Staff member size (including administrators, classified and certified staff, and teachers) ranged from 15 to 72, averaging 52. 18
  19. 19. Individual Respondents N = 114: • Principals = 7. • Vice – principals = 2. • Teachers = 82. • Classified staff (e.g., school psychologist) = 7. • Support staff (e.g., teacher aids) = 9. • Other (unspecified) = 2. • Missing = 5. The Majority: • Female Teachers. • > 7 years in current title. • > 7 years in elementary education. • From one district and one county. • In K – 5 schools. • From schools above average in API statewide and similar school rankings. • From schools with enrollments between 501 and 1,000 students. 19
  20. 20. Results – Research Question 1 For a school to have contextual variables supportive of organizational learning and evaluation average respondent scores across each section of the ROLE had to average 3.5 or higher. • 3 of 9 schools had a culture supportive of organizational learning. • 3 of 9 schools had leadership supportive of organizational learning. • 3 of 9 schools had systems & structures supportive of organizational learning. • 3 of 9 schools had communication supportive of organizational learning. • 3 of 9 schools worked in teams. • 3 of 9 schools engaged in evaluative inquiry. For a school to be considered a learning organization the overall average score on the ROLE across all constructs and respondents had to be 3.5 or higher. • 3 of 9 schools were considered to be a learning organization. 20
  21. 21. Results – Research Question 1 The 3 learning organization schools found to have an internal context in which: • Staff members collaborate and solve problems. • Staff members take risks. • Staff members participate in decision – making. • Leaders support staff member learning throughout the organization. • The work environment is open and accessible. • Reward and recognition system and practices are in place. • There is a recognized relationship between work done and school goals. • Information is readily available to staff members. • Information is disseminated to staff members. • Staff members work in teams. • Evaluation is conducted by external agents or a combination of internal and external agents. 21
  22. 22. Results – Research Question 2 Weak relationship between each predictor variable and evaluation. Taken together, the five independent variables appear to be a significant predictor (F = 3.944; p < .05) of evaluation. Cautiously answers the second evaluation question whether evaluation can be predicted from the presence of organizational learning characteristics. Does not imply evaluative inquiry is occurring or will occur. Indicates that if contextual variables are present it is more likely evaluative inquiry can take place. 22
  23. 23. Results – Research Question 2 Model Sum of df Mean Square F Sig. Squares 1 Regression 6.479 5 1.296 3.944 .003 (a) Residual 28.910 88 .329 Total 35.389 93a Predictors: (Constant), School Teams -weighted, School Leadership - weighted, School Communication - weighted, SchoolSystems and Structures - weighted, School Culture - weightedb Dependent Variable: Evaluation - weighted 23
  24. 24. Results – Research Question 3 School systems and structures was the only significant predictor (t = 2.420; p < .05) and answers, albeit tentatively, the third research question. It appears evaluative inquiry likely occurs when: • A school has a work environment that is open and accessible. • There is a rewards and recognition system and practices in place. • The work staff members do is related to their school’s goals. 24
  25. 25. Results – Research Question 3Model Unstandardized Coefficients Standardized t Sig. Coefficients B Std. Error Beta1 (Constant) 1.969 .547 3.600 .001 School Culture – -.060 .154 -.059 -.390 .697 weighted School Leadership – .166 .115 .193 1.447 .152 weighted School Systems & .437 .181 .346 2.420 .018 Structures – weighted School -.042 .152 -.033 -.279 .781 Communication – weighted School Teams – -.068 .123 -.074 -.554 .581 weighteda Dependent Variable: Evaluation - weighted 25
  26. 26. Findings A majority of sampled elementary schools did not have a sufficient number of contextual variables to support organizational learning and therefore were not learning organizations. In this sample, evaluative inquiry was predicted from the presence the contextual variables necessary for organizational learning. The best predictor of evaluative inquiry in sampled elementary schools characterized as learning organizations was school systems and structures. In this sample, school size, grade level configuration, school quality, staff roles, experience, and gender didn’t appear to covary with the contextual variables necessary for organizational learning. In this sample, evaluative inquiry was either being conducted by external evaluators or some combination of internal and external evaluators. 26
  27. 27. Discussion  Isaacson & Bamburg (1992) concluded that schools rarely fit the description of a learning organization. • In this study only a small sub-sample of elementary schools were considered to be learning organizations.  Leithwood, Leonard, & Sharrat (1998) and Brandt (2003) put forth characteristics a school possesses if it is a learning organization, a number of which were confirmed: • Collaboration is supported. • Mutual support is a norm. • Honest feedback can be provided to colleagues. • There is respect for colleagues’ ideas. • Risk taking, encouragement for open discussion of difficulties, and sharing of success is supported. • There are open and inclusive decision-making processes. • Staff members (inclusive) are available for meetings. • In some schools school staff work in teams. • To an extent there is shared problem-solving. 27
  28. 28. Discussion Lam & Pang (2003) and Leithwood et al. (1998) affirm leadership, culture, and structures are critical elements in promoting evaluative inquiry and organizational learning. • Respondents in this study were inclined to agree leadership, culture, and systems and structures were part of the internal context of their schools. 28
  29. 29. Plausible Limitations1) Threats to internal validity  Selection bias - principals self-selecting their school’s participation  Mortality – respondents quitting the survey before completion  Response bias – selecting socially desirable responses3) Threats to external validity  Findings generalize only to schools configured K – 5 and K – 6 across five districts and three southern California counties5) Survey Error  Coverage – convenience sampling  Sampling – small subset of the sampling frame  Non-response - differences between responders and non-responders; principals determined survey distribution method; time of year  Measurement – challenges in cognitive processing of items; extreme response set 29
  30. 30. Plausible Limitations1) Small sample size and response rate  9 schools / 114 total respondents  37 % response rate4) Overrepresentation of respondents:  By school (23 % of known respondents from one school)  By district (44 % of known respondents from one district)  By county (49 % of known respondents from one county)  By type (75 % of known respondents teachers)  By time in current title (55 % of known respondents < 10 years)  By time in elementary education (70 % of known respondents > 10 years)  By gender (86 % of known respondents female)5) Small, non-representative sample of respondents within most schools. 30
  31. 31. Outcomes & Implications Offered insights into the extent to which elementary schools function as learning organizations and are willing to become, or are engaged in evaluative inquiry. Contextual variables identified in each school that can be developed, refined, or improved in support of organizational learning and evaluative inquiry to improve student outcomes. Positioned principals to promote organizational learning and evaluative inquiry as mechanisms leading to school adaptation and change. Provided potential ROLE users with information about the efficacy of its use in elementary school settings. Used ROLE in an empirical study as the sole quantitative measure of organizational learning and evaluation in elementary schools. 31