Predictors of Success: Student Achievement in Schools


Published on

Predictors of Success: Linking Student Achievement to School and Educator Successes through Professional Learning
This study show how some schools have seen a dramatic increase in student achievement after developing a strong, online professional learning program.

1 Comment
1 Like
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Predictors of Success: Student Achievement in Schools

  1. 1. Predictors of Success: Linking Student Achievement to School and EducatorSuccesses through On-Demand, Computer-Based Professional LearningSteven H. Shaha, PhD, DBA Professor, Center for Public Policy & AdministrationIndependent Program EvaluatorAbstract Early research has begun to investigate the potentialYear-over-year changes in student achievement were benefits of computer-based and online professionalanalyzed for 734 schools selected due to utilization learning (cf. Farnsworth et al. 2002; Lewis et al. 2003;history for online (i.e. on-demand) and computer-based Magidin et al. 2012; Rienties et al. 2013). However, atprofessional learning applications. Results showed that least one recent study cited a “dearth of scientificschools with higher engagement in on-demand profes- research … on whether changes in teachers’ knowl-sional learning by educators significantly outperformed edge and instructional practices resulting from onlinetheir lower engagement counterparts in measures of professional learning are linked to changes in students’quantity and quality of utilization, participation, and knowledge and practices” (Masters et al. 2012). Weengagement. Higher engagement schools also had sig- also found that very few studies have investigated thenificantly greater gains in student achievement as mea- importance of leadership’s engagement in ensuringsured by percentages of students performing at profi- the efficacy of professional learning, regardless of thecient or advanced levels. Higher engagement schools mode of delivery (cf. Sebastian and Allensworth, 2012).also outperformed their lower engagement counter-parts for gains in four key school- and educator-related Finally, there are education-related metrics that havemeasures: teacher retention, dropout rates, student dis- societal implications—metrics reflecting factors that arecipline issues, and rates of students with college-related critical in assessing the success of educators, schools,goals. Conclusions were that higher levels of utilization, and education as a societal institution. Despite theengagement, and active use are correlated with higher importance of these assessments, we found virtuallystudent achievement and successes for both educators no connection established in research between teacherand the schools in which they operate. participation in professional learning and improvements in student-related measures of non-test performanceI. Introduction and Overview beyond at-risk preschool children, such as dropout rates or disciplinary rates (Wasik and Hindman 2011).Educators need high-impact help to keep their skills Even such seemingly simple correlations as improve-well honed and to maintain their educational effec- ments in teacher retention (cf. Lathan and Vogt 2007)tiveness. Yet the body of literature linking professional or teacher attitudes and perceptions (cf. Guskey 2002)learning and development to gains in student perfor- resulting from professional learning are unaccountablymance and teacher-related outcomes arguably remains minimal in the research literature.inadequate (cf. Shaha et al. 2004). Some studies haveshown that professional learning can lead to improved Taken as a whole, research indicates that providing edu-student performance (cf. Garet et al. 2001; Desimone et cators with readily accessible learning opportunities hasal. 2002; Shaha et al. 2004; Meiers and Ingvarson 2005; a substantive and favorable impact. We relabeled thisBuczynski and Hansen 2010; Avalos 2011). Yet, it seems approach as “on-demand learning” to accentuate whyclear from research that the more active an educator’s it is effective instead of how it is delivered. One reasonparticipation is beyond traditional, passive profession- for the effectiveness of the on-demand approach is thatal learning—such as sitting in a workshop or passively educators learn about what they are most interestedwatching a video alone—the greater the impact of par- in, or most in need of, at the time of interest or need,ticipation (cf. Garet et al. 2002; Desimone et al. 2002; rather than when it fits sequentially into any prescrip-King 2002; Darling-Hammond 2004; Santagata, 2009). tive curriculum.In addition, roadblocks to teacher participation in pro-fessional learning and implementation of skills learned Thus timeliness of learning, synchronized with inter-have been cited in recent research and remain import- est and need, mean that educators benefit from whatant barriers to impact (cf. Buczynski and Hansen 2010). could be labeled “just-in-time learning.” This concept1 Steven H. Shaha, PhD, DBA, Professor and/or Lecturer at University of Utah, Zayed University (UAE) and Harvard University
  2. 2. mirrors the business successes achieved in other in- exclusion criteria were implemented reflecting state ordustries through just-in-time approaches—or JIT—as a district, rural or urban areas, school size or any otherscience for maximizing efficiency and profitability while variable associated with school or student demograph-minimizing costs associated with doing business (cf. ics. Year-over-year improvement was computed as theBongiorni 2004; Hirano and Makota 2006; Ohno 1988; percentage change (i.e. gain or loss) for each metricRuffa 2008). Education benefits from sciences proven (i.e. [2011-2010]/2010]).in industry further refine educator efficacy and its im-pact on students. In this case, the near immediate and Educator Engagement: Levels of educator utilization,personally customized benefits of online accessibility participation, and engagement were loaded directlyprovide for JIT educator learning: on-demand profes- from the on-demand applications as captured automat-sional learning. ically and transparently to users, thus ensuring objec- tivity and accuracy, representing 27 metrics (furtherWe undertook the designing and execution of an evalu- explained in Results).ation study of on-demand professional learning in orderto answer a crucial set of inquiries regarding its impact Student Performance: Performance data were gath-on students, educators, and schools. The driving re- ered from publically available sources. In order tosearch questions therefore were whether schools with enable analyses across states with varying testing andhigher utilization or engagement experienced greater scoring approaches, data analyzed reflected the per-impact than those of lower utilization or engagement centage of students classified as either proficient orfor the following: advanced on whatever approach applied within any state or appropriate governing body. Data were limited• ducator engagement in other metrics or areas of E to reading and math only (2 metrics), as these were the utilization, participation, and engagement only two areas of measurement consistent across all• Student performance states. Sixteen schools were excluded from analysis for• Other measures of school- and educator-related success inadequate data regarding student performance.Additional questions to be addressed: School-and Educator-Related Measures: A set of four• s viewing professional learning alone as strong a I metrics were gathered by structured phone interviews predictor of success and impact as other metrics of with each school, including rates for teacher retention, educator utilization, participation, and engagement? dropouts, student discipline, and the number of stu-• s there a model or framework for predicting maxi- I dents reported as being college bound. Year-over-year mum impact from educator utilization, participation, improvement was computed as the percentage change and engagement in on-demand or computer-based (i.e. gain or loss) in the rate or percentage for each metric. teacher development applications? The final study included 734 schools in 211 districts within 39 states. Schools were next classified intoII. Methods quartiles reflecting their average minutes of use by educator as a proxy for relative utilization or engage-A retrospective study was undertaken leveraging a ment rates. To make analyses and conclusions moresample of 750 schools reflecting high engagement in straightforward for execution and interpretation,on-demand professional learning (i.e. PD 360® and analyses contrasted only the top and bottom quar-Observation 360®, School Improvement Network, tiles: the highest quartile of schools (higher engage-Salt Lake City, UT). Data included the 2009-2010 and ment schools) versus the lowest quartile schools2010-2011 school years, categorized during analyses (lower engagement schools).as pre versus post. Schools were selected for inclusionfrom the universe of on-demand users based upon All analyses were conducted by an independent,their active use as measured by minutes of viewing doctoral prepared, internationally recognized stat-professional learning videos, and minimum criterion for istician and program evaluator, using SPSS versioninclusion was set at a minimum average of 90 minutes 17.0 or higher, and SAS for confirmatory purposes asper educator within any school, and all schools meeting needed or appropriate.those minimum criteria were included. No inclusion or 2
  3. 3. higher engagement schools (p.001, see Figure 2),III. Results Initial Interpretations illustrating that comparative gains were great against lower engagement schools for measures reflectingViewed collectively, results showed that higher en- more active participation and engagement by educa-gagement schools outperformed their lower engage- tors. Similarly, the magnitude of difference in teacherment counterparts in every area of measurement: observations performed by leadership was 63.8% higher for higher engagement schools (p.001, seeEducator Engagement: Higher engagement schools Figure 3), illustrating that active engagement byoutperformed their lower engagement counterparts leadership was greater versus in lower engagementin 15 of the 27 metrics of utilization, participation, and schools, as well.engagement, and performed equally well or better inthe remaining 12 metrics, although none significantly(p0.05). Higher engagement schools were significant-ly higher in measures of implementation, accountability,and oversight, or those metrics most appropriately as-cribed to leaders and their role in successful executionof the on-demand or computer-based, educator-learn-ing program.Metrics reflecting greater gains for higher engagementschools included, for example, number of focus objec-tives set up, observations performed, percent of regis-tered users, and percent of users in communities. High- Figure 1. Comparative difference average minutes viewed per educatorer engagement schools performed significantly higherin utilization metrics and measures of more passiveparticipation, including minutes viewed, forums viewed, Forums Posted 23.1programs viewed, segments viewed, and links viewed. 25.0In metrics classified as measures of engagement, higher 20.0 15.0 13.7engagement schools outperformed the lower engage- 10.0ment counterparts in metrics reflecting more active 5.0engagement, including follow-up questions answered, 0.0reflection questions answered, focus objectives set up, Lower Engagement Higherer Engagementforums posted, downloaded files, uploaded files, and Schools Schoolsparticipation in communities. Figure 2. Comparative difference in forums posted per educatorRegarding the degree of comparative impact ofminutes viewed versus the other engagement met-rics, higher engagement schools had 4.3% greater Teacher Observations Performedminutes viewed (p.01), a significant and important 40.0 33.7utilization-related advantage (see Figure 1). This was 35.0 30.0expected, since the assignment of schools to higher 25.0 20.6and lower engagement categories was based upon 20.0their comparative measures of viewing. 15.0 10.0 5.0However, more revealing were the comparative 0.0gains for higher versus lower engagement schools, Lower Engagement Higher Engagement Schools Schoolswhich were substantially and significantly greater forutilization-related metrics, reflecting great levels ofactive participation and engagement beyond simple Figure 3. Comparative difference in observations performed per educatorviewing. For example, we noted the magnitude ofdifference in forums posted was 68.6% higher for 3
  4. 4. A complete view of the 15 measures for which high- While the lower engagement schools improved by aner engagement schools outperformed their lower impressive 4.9% year over year (p.001), the higher en-engagement counterparts is found in Table 1. For gagement schools improved by 18.0% (p.001), nearlyconvenience in interpretation of the results, the 15 four times the rate of improvement comparatively.metrics were categorized into logical groupings re-flecting the apparent nature of the underlying con- In math, higher engagement schools not only closedstructs being measured. The grouping labeled Lead- the pre-existing performance gap, but significantlyership, Implementation, and Accountability included surpassed the lower engagement schools year overmetrics reflecting program setup and active leader year (p.001, see Figure 5). Lower engagement schoolsengagement. The Educator Utilization grouping in- did experience improvement from on-demand profes-cluded metrics reflecting the more passive measures sional learning at 0.5% year over year (p.05). However,of participation as contrasted with educator engage- the higher engagement schools improved by 18.9%ment, for which the metrics reflected more active (p.001), over 30 times the rate of improvement com-and productive participation, for example, beyond paratively. Interestingly, this rate of improvement veryviewing alone. nearly equaled the rate achieved in reading.Table 1. omparative performance in measures of educator C participation as categorized Percent of Students Proficient or Advanced: Reading Higher Lower 68.0 66.6 Engagement Engagement Percent 66.0 63.5 Schools Schools Difference Difference 64.0 67.2Leadership, Implementation, Accountability 62.0 Lower Engagement 60.0Focus Objectives Set Up 130.1 29.2 100.9 345.5% 58.0 SchoolsObservations Performed 3120.7 2149.0 971.7 45.2% 56.0 Higher EngagementPercent Registered Users 87.8% 83.2% 0.0 5.5% 54.0 56.9 52.0 SchoolsPercent of Users in Communities 43.0% 36.5% 6.5% 17.8% 50.0Educator Utilization Pre PostMinutes Viewed 359.9 80.6 279.3 346.5%Forums Viewed 138.7 87.0 51.7 59.4%Programs Viewed 588.7 223.5 365.2 163.4% Figure 4. Comparative gains in reading performanceSegments Viewed 2298.0 528.9 1769.1 334.5%Links Viewed 12.2 10.6 1.6 15.1%Educator Engagement Percent of Students Proficient or Advanced:Follow-up Questions Answered 359.6 167.0 192.6 115.3%Reflection Questions Answered 588.3 340.4 247.9 72.8% MathFocus Objectives Set Up 3120.7 2149.0 971.7 45.2% 72.0 69.5Forums Posted 28.5 23.4 5.1 21.8% 70.0 68.0Downloaded Files 45.2 35.4 9.8 27.7% 66.0 Lower EngagementUploaded Files 46.3 29.3 17.0 58.0% 64.0 62.7 63.0 SchoolsParticipation in Communities 43.0% 36.5% 6.5% 17.8% 62.0 60.0 58.4 Higher Engagement 58.0 Schools 56.0 Pre PostThe implication was that video viewing alone, or othermore passive metrics (e.g. % users registered), werenot as great of predictors or discriminators of educator Figure 5. Comparative gains in math performanceengagement as were measures of utilization reflectingmore active engagement. School-Related Engagements: Results revealed statis- tically significant relationships between key metrics ofStudent Performance: Higher engagement schools educator/school-related success and higher and morecollectively began at a significant performance disad- active utilization of the on-demand professional learn-vantage in both reading (p.001) and math (p.001) in ing. Both higher and lower engagement school cohortsterms of the percentage of students classified as either saw statistically significant gains in school-relatedproficient or advanced. However, in reading, higher en- metrics. However, higher engagement schools, whichgagement schools successfully closed the performance were consistently those with higher utilization rates forgap with the lower engagement schools (see Figure 4). on-demand professional learning, also achieved better improvement year over year versus lower engagement 4
  5. 5. schools in every measure of educator- and school-relat- success, including teacher retention, student discipline,ed success available, including the following: dropout rates, and the number of students reported as college bound.• 0.0% lower dropout rates (p.001) versus 4.9% 2 lower dropout rates for the lower engagement Additionally, video viewing alone was not as great an schools (p.01), representing 4-times greater in gains indicator of student and educator- and school-relat- (see Figure 6) ed gains as the other host of utilization, participation,• .6% gain in rate of students with goals to attend col- 9 and engagement metrics. Performance on the other lege (p.001) versus flat gains for lower engagement metrics of engagement far exceeded those found for schools (p=ns), or 12-times the gains video viewing alone, generally by magnitudes of 10 to• 3.2% lower rate for student discipline occurrences 3 20 times. Several interpretations might fit to explain (p.001) versus 7.4% lower for the lower engagement the finding, perhaps best expressed as questions: Does schools (p.01), greater than 4-times the gains video viewing alone possibly include multi-tasking by• .8% higher teacher retention rates (p.001) versus 2 the participants who may be reading, updating grade 1.7% lower for the lower engagement schools (p.01), books, or emailing while videos are streaming? Do nearly 2/3 greater gains metrics of more active participation reflect higher levels of personal engagement, and therefore more active learning and focus and higher likelihood of application Dropout Rate of things learned? While other explanations may apply, 5.5 5.3 5.0 these data support conclusions that the gains achieved 5.0 for students, educators, and school-related impacts 5.1 4.5 Lower Engagement support leveraging professional learning programs that Schools 4.0 go far beyond video watching alone. 4.1 Higher Engagement 3.5 Schools Taken as a whole, results reveal significant predictive 3.0 Pre Post correlations between the quantity of educator utiliza- tion, participation, and engagement with better studentFigure 6. Comparative gains/reductions in dropout rates results and school-related outcomes. While correlation cannot prove causation, the systematic and consistent findings within these data clearly support a conclu-IV. Discussion Conclusions sion that participation and engagement in this form of teacher development resulted in the advantages andResults substantiated significant and substantive gains found. It is intuitive that highly active and moreadvantages to the use of on-demand and comput- frequent participation in professional learning shoulder-based professional learning. Further, results clearly lead to educators more focused on critical behaviorsindicated that the more engaged the user is beyond and techniques that would help them teach better, andvideo participation alone, the greater the impact of the help their students achieve more proficiency.professional learning. All five research objectives wereachieved. Higher, more frequent on-demand activity, combined with the perception of improved student successHigher engagement schools—those with higher uti- before quantified, also resulted in the higher teacherlization and participation—outperformed their lower retention rates observed, another indicator of educatorengagement counterparts in the majority of the metrics satisfaction and greater enthusiasm for teaching. Whileanalyzed, and never underperformed on any other met- it could be argued that retention is a prescient indicatorrics. Higher engagement schools experienced signifi- of improved success—better teachers stay—the oppo-cantly greater gains for students in math and reading, site causal perspective is at least equally supported inequaling or exceeding 18% gains over prior-year per- this study. In each measurement area, higher engage-formance levels. For school-related measures, higher ment schools outperformed the corresponding lowerengagement schools experienced significantly great- engagement cohort, even when student performanceer gains in critical measures of educator and school began at comparatively lower levels. Thus, the data support the conclusion that higher engagement in 5
  6. 6. on-demand learning results in higher teacher percep- Gus ey, Thomas R. “Professional Development and Teacher k Change.” Teachers and Teaching 8, no. 3 (08/01; 2012/11,tions of their impact on students, contributing to higher 2002): 381-91.satisfaction, and ultimately higher retention. Hir no, Hiroyuki, and Furuya Makota, eds. JIT is flow: Practice and a Principles of Lean Manufacturing. ISBN 0-9712436-1-1 ed.: PCSThe next phase of research should include a repeat of Press, Inc., 2006.these analyses for data reflecting the 2011-2012 school Kin , Kathleen P. “Identifying Success in Online Teacher Education gyear. That may include a contrast of schools for which and Professional Development.” The Internet and Higher Edu- cation 5, no. 3 (0, 2002): 231-46.the 2011-2012 was the first year of participation versus Lat am, N., and W. P. Vogt. “Do Professional Development Schools hthe gains found in these year-to-year analyses for 2010 Reduce Teacher Attrition? Evidence from a Longitudinal Studyversus 2011. Additionally, new analyses might contrast of 1,000 Graduates.” Journal of Teacher Education 58, no. 2the newly implemented schools for 2012 with a two- (2003): 153-67.year history of gains for those included in these anal- Lews, V. K., S. H. Shaha, B. Farnsworth, L. Benson, and D. Bahr. iyses, thus adding the insight of multiple year-on-year “The Use of Assessment in Improving Technology-Based Instruction Programs.” Journal of Instructional Psychology 30,participation from 2010 through 2012 versus a first year no. 2 (2003).alone. Magdin de Kramer, Raquel, Jessica Masters, Laura M. O’Dwyer, i Sheralyn Dash, and Michael Russell. “Relationship of OnlineFuture research might also include contrasts of gains Teacher Professional Development to Seventh-Grade Teachers’with changes experienced in equivalent non-participat- and Students’ Knowledge and Practices in English Languageing schools to ensure that the accomplishments found Arts.” The Teacher Educator 47, no. 3 (06/27; 2012/11, 2012): reflect achievements above and beyond chance Mas ers, Jessica, Raquel Magidin de Kramer, Laura O’Dwyer, Sher- talone. This would be ideally executed with a study that alyn Dash, and Michael Russell. “The Effects of Online Teacherincludes analyses of changes contrasting schools with Professional Development on Fourth Grade Students’ Knowl-and without the on-demand or computer-based teach- edge and Practices in English Language Arts.” Journal of Tech-er learning, thus controlling for important underlying nology and Teacher Education 20, no. 1 (January, 2012): 21-46.differences such as socioeconomic mixes. Additional Mei rs, M. and L. Ingvarson. “Investigating the Links Between e Teacher Professional Development and Student Learning Out-research might also include the contrasting metrics comes.” Australian Council for Educational Research 1, (2005):from higher utilization schools with non-participating 1-93.schools from their respective districts, thus controlling Ohn , Taiichi, ed. Just-in-time for today and tomorrow. ISBN ofor differences in demographics and socioeconomics. 0-915299-20-8 ed.: Productivity Press, 1988. Rie ties, Bart, Natasa Brouwer, and Simon Lygo-Baker. “The Ef- n fects of Online Professional Development on Higher Education Teachers’ Beliefs and Intentions Towards Learning FacilitationReferences and Technology.” Teaching and Teacher Education 29, no. 0 (1, 2013): 122-31.Avaos, Beatrice. “Teacher Professional Development in Teaching l Ruf a, Stephen A. “Going Lean: How the Best Companies Apply f and Teacher Education Over Ten Years.” Teaching and Teacher Lean Manufacturing Principles to Shatter Uncertainty, Drive In- Education 27, no. 1 (1, 2011): 10-20. novation, and Maximize Profits.” AMACOM (American Manage-Bon iorni, Sara. “All in the Timing.” The Greater Baton Rouge Busi- g ment Association) (2008). ness Report (July 19, 2004). San agata, R. “Designing Video-Based Professional Development tBuc ynski, Sandy, and C. Bobbi Hansen. “Impact of Professional z for Mathematics Teachers in Low-Performing Schools.” Journal Development on Teacher Practice: Uncovering Connections.” of Teacher Education 60, no. 1 (2009): 38-51. Teaching and Teacher Education 26, no. 3 (2010): 599-607. Seb stian, J. and E. Allensworth. “The Influence of Principal Lead- aDaring-Hammond, L. “Standards, Accountability, and School Re- l ership on Classroom Instruction and Student Learning: A Study form.” Teachers College Record 106, no. 6 (2004): 1047-85. of Mediated Pathways to Learning.” University Council forDesmone, L. M., A. C. Porter, M. S. Garet, K. S. Yoon, and B. F. i Educational Administration 10, (2012): 626-63. Birman. “Effects of Professional Development on Teacher’s Sha a, S. H., V. K. Lewis, T. J. O’Donnell, and D. H. Brown. “Evaluat- h Instruction: Results from a Three-Year Longitudinal Study.” Ed- ing Professional Development: An Approach in Verifying Pro- ucational Evaluation and Policy Analysis 24, no. 2(2002): 81-112. gram Impact on Teachers and Students.” Journal of ResearchFar sworth, B. and et al. “Preparing Tomorrow’s Teachers to use n in Professional Learning 1, no. 1 (2004): 1. Technology: Learning and Attitudinal Impacts on Elementary Students.” Journal of Instructional Psychology 29, no. 3(2002).Gar t, Michael S., Andrew C. Porter, Laura Desimone, Beatrice e F. Birman, and Kwang Suk Yoon. “What Makes Professional Development Effective? Results from a National Sample of Teachers.” American Educational Research Journal 38, no. 4 (Winter 2001, 2001): 915. 6