This document summarizes research on contextual variables that influence organizational learning and evaluative inquiry in elementary schools. The research had three aims: 1) identify which contextual variables support organizational learning in schools, 2) determine if evaluative inquiry can be predicted from these variables, and 3) identify the best predictor of evaluative inquiry. The results found that a majority of schools lacked most contextual variables, but in schools that had them, evaluative inquiry could be predicted and was best predicted by supportive school systems and structures.

1. Contextual Variables In Elementary Schools
Influencing Organizational Learning
&
Predicting Evaluative Inquiry
Jeffrey Sheldon, M. A, Ed. M.
School of Behavioral & Organizational Sciences
Claremont Graduate University
The Claremont Colleges
9 November 2007

2. 2
Overview
1) Specific Aims
2) Background & Significance
3) Gaps in the Research
4) Research Questions
5) Hypotheses
6) Research Design
7) Instrumentation
8) Analysis Strategy: Phase I
9) Analysis Strategy: Phase II
10) Missing Values
11) Sampling
12) Recruitment
13) Data Collection
14) Coverage
15) Participating Schools
16) Individual Respondents
17) Results – Research Question 1
18) Results – Research Question 2
19) Results – Research Question 3
20) Findings
21) Discussion
22) Outcomes & Implications
This research was made possible through the generosity of Dr. Barbara DeHart,
Professor and Director of the Urban Leadership Program, School of Educational
Studies, Claremont Graduate University, Claremont, CA

3. 3
Specific Aims
 Exploring elementary schools for contextual variables known to influence
organizational learning and evaluative inquiry.
 Predicting from contextual variables present whether evaluative inquiry for
organizational learning is likely to take place and which variable is the best
predictor.
 Using a relatively new quantitative instrument to refute criticism about using
quantitative instruments to measure organizational learning and providing
evidence of its utility in school-based organizational learning/evaluation
research.
 Illuminating for each participating school the contextual variables necessary for
organizational learning and evaluation they have in place, and those that either
could be put in place or strengthened.
 Contributing to a better understanding of the linkage between evaluative inquiry
and organizational learning.

4. 4
Background & Significance
 Keefe & Howard (1997); Reed, Kinzie, & Ross (2001); Watkins & Marsick (1999):
School reform efforts most successful when schools are learning organizations and
school personnel are change agents.
 Lashway (1998): Schools dedicating themselves to systematic, collaborative
problem-solving can continually develop and implement new ideas, thereby not just
improving, but transforming themselves.
 Mancini & Marek (2004): An organizational learning approach is necessary in
schools to promote sustainability of any changes shown to be effective in
improving academic achievement.
 Trubowitz (2005): Schools should resist external pressures that compel immediate
and precipitous action unsupported by prior thinking.
 Bowen, Rose, & Ware (2006): Organizational learning is the best way to address
critical issues facing schools today because they must be ready to receive and act
upon new information for change to occur.

5. 5
Background & Significance
 Fiol & Lyles (1985); Senge (1990): When a context supportive of organizational learning
is in place an organization is said to be a learning organization.
 Cousins & Leithwood (1986); Cousins & Shulha (2002): Context factors influence how
and to what extent learning will occur in an organization.
 Shulha & Cousins (1997): Culture, leadership, communication, teamwork, and
systems/structures are considered to be the elements of organizational context necessary
for evaluative inquiry and organizational learning.
 Lam & Pang (2003): The prevailing influence of organizational learning process comes
primarily from internal school conditions - transformational leadership, supportive
structure and culture are key factors promoting the learning process.
 Silins & Mulford (2004): Four components of organizational learning in schools -
collaborative climate, taking initiatives and risks, improving school practices, and
professional development.
 Preskill & Torres (1999); Russ-Eft, Atwood, & Egherman (2002): Evaluative inquiry
contributes to individual, group, and organizational learning.
 Preskill & Torres (2000): An organization’s culture and context influence the extent to
which evaluative inquiry occurs in support of learning and decision making.

6. 6
Gaps in the Research
 Botcheva, et al. (2002); Goh et al. (2005); Lee (1999); Scribner et al. (1999):
Schools as learning organizations studied at middle and secondary levels, but
little evidence that elementary schools are learning organizations.
 Goh et al. (2005): Few published empirical studies showing views, perspectives
and importance teachers and administrators attach to organizational learning
practices and activities.
 Alkin & Taut (2003): It would be insightful to investigate how contextual factors
within organizations influence evaluative inquiry.
 Cousins et al. (2004): Knowledge base linking evaluative inquiry to
organizational learning relatively immature.
 Bryk, Camburn, & Louis (1999); Mebane & Galassi (2003): Quantitative tools
addressing organizational learning suffer from abstract definitions of constructs,
lack clearly articulated conceptual frameworks reflecting multiple dimensions of
learning organizations, and are adapted from business and industry.

7. 7
Research Questions
1) Which, if any, contextual variables necessary for organizational learning are present
in elementary schools:
 Culture (collaboration and problem solving; risk taking; participatory decision
making)?
 Leadership?
 Systems and structures (open and accessible work environment; rewards and
recognition system and practice; relationship of work to organizational goals)?
 communication (availability of information; dissemination of information)?
 Teams?
2) Can evaluative inquiry be predicted from those contextual variables present in
elementary schools?
3) If evaluative inquiry can be predicted, what is the best predictor?

8. 8
Hypotheses
1) A majority of sampled elementary schools do not have a sufficient number of
contextual variables to support organizational learning and therefore are not
learning organizations.
2) Evaluative inquiry can be predicted from each or some combination of
organizational learning contextual variables present.
3) In elementary schools that are characterized as learning organizations, the best
predictor of evaluative inquiry is leadership.

9. 9
Research Design
 Quantitative.
 Exploratory.
 Two-phases:
1) Used a questionnaire to determine:
• Which, if any, contextual variables necessary for organizational learning
were present in a small sample of elementary schools.
• Whether schools could be considered to be learning organizations.
• Whether schools were either receptive to or engaged in evaluative
inquiry.
2) Used multiple regression to predict whether evaluative inquiry was likely in
a school that had contextual variables for organizational learning in place
and determined which one was the best predictor.

10. 10
Instrumentation
 Preskill & Torres’ (2000) highly reliable (Cronbach’s Alpha = .97) Readiness for
Organizational Learning and Evaluation Instrument (ROLE).
 Operationalizes constructs of organizational learning - culture, leadership, systems
and structures, communication, teams – and evaluation.
 Unlike other quantitative instruments ROLE:
• Measures organizational learning constructs that are concrete and well-
established in the extant literature.
• Reflects the multiple dimensions of organizational learning.
• Can be used across a wide spectrum of organizational types and sectors.
 Two other instruments measure organizational learning, but not evaluation:
• Bowen & Powers’ (2003) School Success Profile-Learning Organizations.
• Ellis, Globerson, & Parson’s (1997) Organizational Learning Mechanisms
Questionnaire.

11. 11
Analysis Strategy: Phase I
 Unit of analysis: The school.
 Based, in-part, on Silins & Mulford’s (2004) study focusing on school-level
variables associated with leadership and organizational learning.
 Descriptive statistics used to determine which, if any, contextual variables
necessary for organizational learning were present in sample and to what extent
(and by whom) evaluative inquiry was taking place.
 Results of each ROLE aggregated across individual respondents by school to come
up with each school’s composite score.
 Independent sample t - tests used to determine if certain attributes (e.g., staff
gender, grade level configuration, or school district) covaried with organizational
learning and evaluation contextual variables present in sample.

12. 12
Analysis Strategy: Phase II
 Based, in part, on Lam & Pang’s (2003) study of organizational learning in
schools.
 Multiple regression used to determine 1) whether engagement in evaluation
activities (i.e., evaluative inquiry) could be predicted from contextual variables
discovered in Phase I, and 2) the best predictor.
• The 5 constructs of organizational learning = independent variables.
• Evaluation = dependent variable.
 To increase precision in making predictions, constructs weighted and used as a
conservative adjustment for the unequal number of items within each construct.
 After weighting, independent variables entered, in no predetermined order, as a
block into the model.
 A Postieri analyses.

13. 13
Missing Values
 A priori decisions for making missing value corrections to avoid the elimination of
data:
• For constructs comprised of subconstructs with 5 items, 3 of 5 had to be
answered.
• For constructs comprised of subconstructs with 4 items, 3 of 4 had to be
answered.
• For constructs comprised of subconstructs with 3 items, 2 of 3 had to be
answered.
• A majority of items had to be answered within whole constructs.
 Each respondent’s scores for whole construct added together and divided by the
number of items excluding zero scores.
 That score then entered into the unweighted construct score prior to weighting.

14. 14
Sampling
 Public elementary schools with attendant administrators, faculty, and staff
comprised sample population.
 Schools were to be drawn from a sampling frame of 778 elementary schools within
80 districts across 4 counties.
 Sampling was to have been based on probability.
 Did not realize:
• Multiple permissions needed before study could be conducted in schools
• The difficulty of gaining permission to conduct school-based research
 Sampling strategy became one of convenience.
 12 administrators, 156 classified and support staff, and 290 teachers totaling 458
school personnel could have responded.

15. 15
Recruitment
 Three-tiered approach to introduce study, gain endorsement and support, and
obtain necessary authorizations/permissions.
1) Letter sent to Board of Education Superintendent in each of four counties.
2) Letters sent to 80 local school district superintendents in three counties.
• 17 Affirmative responses
• 10 Negative responses
• 2 Non-committal responses
• 51 Non-responses
3) Selected 11 local school districts randomly using the Research Randomizer
web site and subsequently sent letters of invitation to 110 principals.

16. 16
Data Collection
Followed Dillman’s (2000) best-practice in survey research.
 Box of ROLE packets sent within a week of receiving written permission and
confirmation a principal wanted school to participate.
 Follow-up “thank-you/reminder to fill out the survey if you haven’t already”
post-cards and emails sent out five days after return deadline to prompt
principals and in turn, their staff.
 In lieu of re-sending ROLE packets to schools as suggested by Dillman email
sent to principals letting them know that new ROLE packets would be sent
immediately if necessary; PI did not have the name of individual respondents so
all communication went through principals.

17. 17
Coverage
 Budget constraints necessitated trade-offs regarding number of ROLE packets
that could be sent to each school.
 ROLE packets sent covered approximately 60 percent of the potential
respondents at each school.
 Two schools were small enough in staff size for 100 percent coverage.
 Countervailing argument for covering fewer schools completely dismissed in
favor of gaining a broader sample of schools by type, district, and county for
greater generalizability.
 But, no guarantee complete coverage in fewer schools would have improved the
response rate.

18. 18
Participating Schools
 Schools represented 5 of 80 districts (6.3 %) and three of four southern California
counties (75 %) initially identified and targeted for inclusion.
• 3 schools from 2 districts in County A.
• 5 schools from 2 districts in County B.
• 1 school in County C.
 All 9 schools met the definition of an elementary school as determined prior to the
study.
• 2 schools configured K – 6.
• 7 schools configured K – 5.
 All schools met the criteria of a small organization (< 1,000 students; < 100 staff).
• Student enrollment ranged from 218 to 871, averaging 666.
• Staff member size (including administrators, classified and certified staff, and
teachers) ranged from 15 to 72, averaging 52.

19. 19
Individual Respondents
 N = 114:
• Principals = 7.
• Vice – principals = 2.
• Teachers = 82.
• Classified staff (e.g., school psychologist) = 7.
• Support staff (e.g., teacher aids) = 9.
• Other (unspecified) = 2.
• Missing = 5.
 The Majority:
• Female Teachers.
• > 7 years in current title.
• > 7 years in elementary education.
• From one district and one county.
• In K – 5 schools.
• From schools above average in API statewide and similar school rankings.
• From schools with enrollments between 501 and 1,000 students.

20. 20
Results – Research Question 1
 For a school to have contextual variables supportive of organizational learning and
evaluation average respondent scores across each section of the ROLE had to
average 3.5 or higher.
• 3 of 9 schools had a culture supportive of organizational learning.
• 3 of 9 schools had leadership supportive of organizational learning.
• 3 of 9 schools had systems & structures supportive of organizational learning.
• 3 of 9 schools had communication supportive of organizational learning.
• 3 of 9 schools worked in teams.
• 3 of 9 schools engaged in evaluative inquiry.
 For a school to be considered a learning organization the overall average score on
the ROLE across all constructs and respondents had to be 3.5 or higher.
• 3 of 9 schools were considered to be a learning organization.

21. 21
Results – Research Question 1
 The 3 learning organization schools found to have an internal context in which:
• Staff members collaborate and solve problems.
• Staff members take risks.
• Staff members participate in decision – making.
• Leaders support staff member learning throughout the organization.
• The work environment is open and accessible.
• Reward and recognition system and practices are in place.
• There is a recognized relationship between work done and school goals.
• Information is readily available to staff members.
• Information is disseminated to staff members.
• Staff members work in teams.
• Evaluation is conducted by external agents or a combination of internal and
external agents.

22. 22
Results – Research Question 2
 Weak relationship between each predictor variable and evaluation.
 Taken together, the five independent variables appear to be a significant predictor (F =
3.944; p < .05) of evaluation.
 Cautiously answers the second evaluation question whether evaluation can be
predicted from the presence of organizational learning characteristics.
 Does not imply evaluative inquiry is occurring or will occur.
 Indicates that if contextual variables are present it is more likely evaluative inquiry can
take place.

23. 23
Results – Research Question 2
Model Sum of
Squares
df Mean Square F Sig.
1 Regression 6.479 5 1.296 3.944 .003 (a)
Residual 28.910 88 .329
Total 35.389 93
a Predictors: (Constant), School Teams -weighted, School Leadership - weighted, School Communication - weighted, School
Systems and Structures - weighted, School Culture - weighted
b Dependent Variable: Evaluation - weighted

24. 24
Results – Research Question 3
 School systems and structures was the only significant predictor (t = 2.420; p < .05)
and answers, albeit tentatively, the third research question.
 It appears evaluative inquiry likely occurs when:
• A school has a work environment that is open and accessible.
• There is a rewards and recognition system and practices in place.
• The work staff members do is related to their school’s goals.

25. 25
Results – Research Question 3
Model Unstandardized Coefficients Standardized
Coefficients
t Sig.
B Std. Error Beta
1
(Constant) 1.969 .547 3.600 .001
School Culture –
weighted
-.060 .154 -.059 -.390 .697
School Leadership –
weighted
.166 .115 .193 1.447 .152
School Systems &
Structures –
weighted
.437 .181 .346 2.420 .018
School
Communication –
weighted
-.042 .152 -.033 -.279 .781
School Teams –
weighted
-.068 .123 -.074 -.554 .581
a Dependent Variable: Evaluation - weighted

26. 26
Findings
 A majority of sampled elementary schools did not have a sufficient number of
contextual variables to support organizational learning and therefore were not
learning organizations.
 In this sample, evaluative inquiry was predicted from the presence the contextual
variables necessary for organizational learning.
 The best predictor of evaluative inquiry in sampled elementary schools
characterized as learning organizations was school systems and structures.
 In this sample, school size, grade level configuration, school quality, staff roles,
experience, and gender didn’t appear to covary with the contextual variables
necessary for organizational learning.
 In this sample, evaluative inquiry was either being conducted by external
evaluators or some combination of internal and external evaluators.

27. 27
Discussion
 Isaacson & Bamburg (1992) concluded that schools rarely fit the description
of a learning organization.
• In this study only a small sub-sample of elementary schools were
considered to be learning organizations.
 Leithwood, Leonard, & Sharrat (1998) and Brandt (2003) put forth
characteristics a school possesses if it is a learning organization, a number of
which were confirmed:
• Collaboration is supported.
• Mutual support is a norm.
• Honest feedback can be provided to colleagues.
• There is respect for colleagues’ ideas.
• Risk taking, encouragement for open discussion of difficulties, and
sharing of success is supported.
• There are open and inclusive decision-making processes.
• Staff members (inclusive) are available for meetings.
• In some schools school staff work in teams.
• To an extent there is shared problem-solving.

28. 28
Discussion
 Lam & Pang (2003) and Leithwood et al. (1998) affirm leadership, culture,
and structures are critical elements in promoting evaluative inquiry and
organizational learning.
• Respondents in this study were inclined to agree leadership, culture, and
systems and structures were part of the internal context of their schools.

29. 29
Plausible Limitations
1) Threats to internal validity
 Selection bias - principals self-selecting their school’s participation
 Mortality – respondents quitting the survey before completion
 Response bias – selecting socially desirable responses
2) Threats to external validity
 Findings generalize only to schools configured K – 5 and K – 6 across
five districts and three southern California counties
3) Survey Error
 Coverage – convenience sampling
 Sampling – small subset of the sampling frame
 Non-response - differences between responders and non-responders;
principals determined survey distribution method; time of year
 Measurement – challenges in cognitive processing of items; extreme
response set

30. 30
Plausible Limitations
4) Small sample size and response rate
 9 schools / 114 total respondents
 37 % response rate
5) Overrepresentation of respondents:
 By school (23 % of known respondents from one school)
 By district (44 % of known respondents from one district)
 By county (49 % of known respondents from one county)
 By type (75 % of known respondents teachers)
 By time in current title (55 % of known respondents < 10 years)
 By time in elementary education (70 % of known respondents > 10
years)
 By gender (86 % of known respondents female)
6) Small, non-representative sample of respondents within most schools.

31. 31
Outcomes & Implications
 Offered insights into the extent to which elementary schools function as learning
organizations and are willing to become, or are engaged in evaluative inquiry.
 Contextual variables identified in each school that can be developed, refined, or
improved in support of organizational learning and evaluative inquiry to improve
student outcomes.
 Positioned principals to promote organizational learning and evaluative inquiry
as mechanisms leading to school adaptation and change.
 Provided potential ROLE users with information about the efficacy of its use in
elementary school settings.
 Used ROLE in an empirical study as the sole quantitative measure of
organizational learning and evaluation in elementary schools.