Data-Driven Decision-Making in K-12
Education: Using the Learning Delta to
Manage for Results
A High Delta Learning  Whit...
Page 1

        quot;…When you can measure what you are speaking about, and express it in numbers, you
        know somet...
Page 2

The TVAAS has been in use since          progress being made in the class-       material satisfactorily by the en...
Page 3

• As interim Learning Deltas are         Thought
Page 4

Page 5

Page 6

                                          model), which need to be improved,                     3. Do our measure...
Page 7

performance for the future. In other
words, we can improve the state of
education if we move beyond the
Page 8

                                                as teamwork, culture, and be-
But, as with company earnings, we   ...
Page 9

The point I wish to make here about
quot;digitizingquot; the classroom is differ-       Figure 7
ent from the quot...
Page 10

These components comprise a                                                       the data fairly or constructive...
Page 11

                                         PRESS RELEASE (Excerpt)
Page 11

                                                                 Appendix A

Learning Delta White Paper  Greg Luther 2003
Learning Delta White Paper  Greg Luther 2003
Learning Delta White Paper  Greg Luther 2003
Learning Delta White Paper  Greg Luther 2003
Learning Delta White Paper  Greg Luther 2003
Upcoming SlideShare
Loading in …5

Learning Delta White Paper Greg Luther 2003


Published on

White paper -- Using the Learning Delta to Manage Teaching and Learning Performance
May, 2003

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Learning Delta White Paper Greg Luther 2003

  1. 1. Data-Driven Decision-Making in K-12 Education: Using the Learning Delta to Manage for Results A High Delta Learning  White Paper Gregory D. Luther Founder and Principal High Delta Learning, LLC May, 2003
  2. 2. Page 1 quot;…When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of Science, whatever the matter may be.quot;  Lord Kelvin, 1884 sector. Typically, instead of em- In this paper I make a three-part Introduction ploying a quot;generally accepted argument: The emerging market for K-12 accounting principlequot; for learning education services is placing cities, measurement, there is a call for 1. More can be done to improve school districts, and parents into the outside research to assess or validate education by managing the ef- position of education consumers a program's salutary effects, to be fectiveness of teaching than by who have a growing need for performed using multivariate statis- any other single factor. accurate, understandable, and timely tical analysis and experimental 2. Using the Learning Delta to information on which to base their designs taken from social science measure learning progress is choices. Similarly, education phi- and clinical drug trials. In short, fundamental to managing and lanthropies have choices to make because schools do not measure improving the effectiveness of with regard to which reforms or their own performance, somebody education reform programs, innovations to back with their else has tousually at considerable schools, and teaching generally. financial resources. In the words of expense and with substantial delays the Thomas B. Fordham Founda- between the point of data capture 3. In order to implement Learning tion: and the reporting of results. Delta Management, classroom workflows must be digitized in In this paper I propose the Learn- Private individuals and organizations… order to enable the instrumen- ing Delta as a top-level perform- can spend their dollars exactly where tation of teaching and learning ance metric for schools. Learning they seek to do the most good. If they in schools. Delta is a measure of learning direct their money, energy, and influ- progress that captures each stu- ence toward the right targets. . . their dent's change in mastery for a given leverage will help move the system Part 1: Value-Added Testing period of time. In addition to itself. The philanthropic sector-America's and Teacher Effectiveness improving the manageability of unique blend of private organizations individual student progress, Learning There is already a substantial body with public-minded goals-has the Delta data can be aggregated to of empirical work from the state of freedom of action to push on the right arrive at progress measures for Tennessee that substantiates the pressure points, and it has clout that classrooms, entire schools, or school i importance and utility of collecting most parents lack . systems. Learning Deltas can be data on students' learning progress. But a critical difficulty facing educa- used to minimize the time students Together with the Tennessee Value- tion philanthropiesas well as cities, spend waiting for learning to hap- Added Assessment System school districts, and parentsis that pen. Perhaps most importantly, (TVAAS), Dr. William Sanders, at schools do not adequately account continuously accumulating and the University of Tennessee Value- for their primary product: learning. analyzing longitudinal Learning Added Research and Assessment Measuring and analyzing the impact Delta data for all students represents Center, found that of all the factors of reforms and innovations on an embedded research process that influencing the observed change in students' learning progress is a can inform and help to improve student test scores from year to year, problem because schools do not instruction in every classroom, teacher effectiveness was by far the employ performance measures and school, and school system that uses most powerful. management processes like enter- it. prises outside of the education
  3. 3. Page 2 The TVAAS has been in use since progress being made in the class- material satisfactorily by the end of 1991. Every student in grades 2-8 is room. the term. tested each year in math, reading, language arts, science, and social Part 2: Using Learning Delta Data Discipline studies. Testing of high school (and An Illustration) students began in 1995. Each The quot;data disciplinequot; associated with student’s test results are compared Learning Delta is quite simple. Learning Delta is given as the to his or her scores from the previ- change in a student's mastery of an Step 1 - A quot;pre-testquot; is administered ous year, and the average score academic subject divided by the on or near the first day of class to increases of all the students taught change in time. Where Sanders' establish the baseline learning level by each teacher are reported to of each student. quot;value-addedquot; is a measure of administrators for use in evaluations academic distance, Learning Delta is • Pre-test questions are drawn and personnel decisions. a measure of speed, i.e. distance from the same set of test ques- tions that would appear on a final The headlines from Sanders' find- exam. A pre-test may also in- ings in Tennessee include the Figure 2 clude questions that represent followingii: prerequisite knowledge and knowledge that is advanced be- 1. The top quintile of teachers-- Learning Delta: yond the scope of the course. ranked according to their stu- the Rate of dents' average increase in test • A student's pre-test score is a Learning scores--raised their students’ measure of the subject knowl- Progress ∆ Mastery achievement test scores 39 per- edge they brought into the class Success centile points more than teachers with them. Rate from the bottom quintile. • Pre-test resultstagged to each ∆ Time 2. Teachers performing in the two student, to the specific subject units and skills being examined, lowest quintiles in Tennessee failed to produce any achieve- and to the teacher and classare divided by time, as shown in Figure recorded and/or stored. ment gains with most of their 2. The quot;distancequot; term can be students. thought of as a scale score that is • The difference between a stu- used in the same way as Sanders dent's pre-test score and the 3. The performance difference uses it. Time is measured in school score required to demonstrate between students having three terms. So if a course lasting one mastery in a subject. is their ini- consecutive teachers in the top tial Learning Gap, which also term is given a value of 100 points, one-fifth of the ranking and serves as the Planned Learning and a student masters half of the three in the bottom one-fifth was Delta for that student. units making up that course in one 50 percentile points(!!). half of the academic term, then the Step 2 - As instruction progresses, 4. The teacher quality effect on student is proceeding at a pace of additional tests are administered on student performance was found 100 points per term and has a a weekly, bi-weekly, or monthly to be highly significant and larger Learning Delta of 100 (i.e. 50 points basis. Once again, each test is in size than any other factor-- ÷ 0.5 terms = 100 points/term). If effectively a final exam, and stu- including class size and quot;hetero- dents' scores reflect their new level on the other hand, the student has geneity.quot; of subject knowledge attained since only mastered curriculum units the beginning of the term. worth 30 points by the mid-term, In Dr. Sanders view, then, quot;…more that student has a Learning Delta of Step 3 - At the time of each assess- can be done to improve education 60 (i.e. 30 points ÷ 0.5 terms = 60 ment Learning Delta figures are by improving the effectiveness of points/term), and unless she/he calculated for each student and the teachers than by any other single increase their Learning Delta to 140, average is calculated for the class as factor.quot; And the key to that effort she/he will not complete the course a whole. is collecting data on the learning
  4. 4. Page 3 • As interim Learning Deltas are Thought Figure 3 measured, the students, teachers, Experiment and parents can see if adequate % of Number of Students Out of 7,000 Students in Population Students Below Grade progress is being made to the Reform Program What follows Level goal of mastery, and if not, they is a quot;thought Group 1 - Reading at Grade can adjust course while there is Level (at 50th %ile or above) experimentquot; 78.9% 5,523 still time. and illustra- Group 2 - About One Year Behind (assume they score tion of why • Students who are progressing too around 20%ile) 11.7% 819 819 data on slowly will have low Learning Group 3 - About Two Years students' Deltas that identify them for Behind (assume they score special assistance or motivating. learning around 10%ile) 9.4% 658 658 progress • Students who are moving much Total 7,000 1,577 should be faster than the rest of the group used to drive will have high Learning Deltas decision-making in education that identify them for additional The experiment involves applying philanthropy. The example I will use quot;enrichmentquot; work or accelera- some of the findings from the work is based on an actual philanthropy- tion into a more advanced class. of Dr. William Sanders and TVAAS funded reading reform program to this sample education philan- A key difference between the (with some of the details disguised) thropy program and posing the Learning Delta and quot;value-addedquot; in a state where roughly one out of following questions: measures is that Learning Delta is every five students in the target designed to be a frequent measure population fail to achieve quot;profi- that enables the teacher and school • What if we measured the Learning cientquot; status on the state's reading leaders to make immediate-term Deltas being produced by teachers tests, and about one in ten demon- course corrections in every class- participating in a philanthropy-funded strate only quot;minimalquot; reading ability. room. The quot;value-addedquot; measure education reform program by captur- For the purposes of this example, as used by TVAAS is a once-a-year ing data from the students on a the program covers 7,000 students measure that, as a result, is retro- monthly or quarterly basis? across 20 schools, and it costs about spective. Although it can give a $2 million per year to administer. reading on how a school performed • If Sanders' findings about teacher in the prior year, as any particular Figure 1 shows a simple cost-per- performance in Tennessee were school year progresses it provides success calculation for this pro- duplicated elsewhere, how would this less and less help with how to gramwhich may involve expendi- information enable us to increase the improve learning performance in tures on professional development, specific classrooms next week or quot;returnquot; of an education reform pro- specialized curricular materials, next month. gram or other innovation that is un- technology innovations, or all these derway? elements. Appendix A shows what Learning Delta data would look like over a term and how that allows classrooms Having made this calculation, the The Base Case and teachers to be grouped into question remaining for the donor to performance quintiles. Measures of learning progress are answer is, quot;Do you think $4000 per important because, as Sanders has student is a satisfactory num- shown, highly effective teachers ber?quot; In the event that the Figure 1 - Program Cost Per Success move their students along at a much donor is a venture-philanthropist Number of Students in Program 7,000 greater pace than average or below- with an especially keen interest Total Program Cost for One Year $2,000,000 average teachers. And the situation in results, the further question in any particular school is further may be, quot;Shall we continue Number of Students Meeting 500 complicated because the curricular Reading Standards Who Would- with this program or do n't Have Otherwise quot;distancequot; that must be covered in something else where the order to meet standards of compe- Program Cost per Student- $4,000 impact may be greater?quot; Success tency can vary greatly from one student to another.
  5. 5. Page 4 school year. Figure 4 Quintile 3 teachers accomplish some Base Case improvement, but Q 1+2 Q3 Q4 Q5 Total not enough to bring $3,000,000 1500 their students up to standard (see table in Appendix B for $2,000,000 1000 detail). As a result of these combina- Write-off Yield = 4.1 Yield = 7.4 Yield = 2.3 tions of teacher $1,000,000 500 Number of Successes 132 effectiveness and Program Cost 132 328 student deficit, 164 0 164 $0 0 then, only 40% of 0 0 Q 1+2 Q3 Q4 Q5 Total Group 2 students ($400,000) ($400,000) ($400,000) will be brought up ($1,000,000) ($800,000) -500 to grade level, and only 20% of the Group 3 students. ($2,000,000) -1000 ($2,000,000) Overall, based on Sanders' findings in ($3,000,000) -1500 Tennessee, only Teacher Performance Quintiles 7% of the 7,000 Program Cost Group 2 Students Group 3 Students students in this program are likely to be counted as Case graphic in Figure 4 shows the successes at the end of the year As shown in Figure 3, because 20% effect of the differing levels of (i.e. students who met the statewide of students in the reform program teacher effectiveness predicted from reading standard who would not we are looking at score below Sanders' findings for students who have otherwise). The cost per quot;proficientquot; status on the state's are one and two years behind in student-success will, therefore, come reading tests and about 10% demon- their reading skills. Specifically, in at about $4,400 and the return can strate only quot;minimalquot; reading ability, teachers in the bottom two quintiles be thought of as 2.3 successes-per- we get roughly 1,500 students out of do not produce any improvement, $10,000 quot;investedquot; by the philan- the 7,000 who are functioning below and there is a 39 percentile point thropy. grade-level. difference in results between quin- tiles 3 through 5, which is assumed For the purposes of this illustration However, as shown in the figure, to be distributed evenly across these we then assume that the 800+ analyzing results according to quintiles. So Quintile 5 teachers Group 2 students that scored performance quintile makes certain produce gains of 39%ile points in quot;BASICquot; on the statewide compe- program improvement actions their students; Quintile 4 teachers tency test are one year behind, and immediately obvious. For example, produce gains of 26%ile points; and the remaining Group 3 students who the yield achieved by Quintile 5 Quintile 3 teachers produce gains of scored quot;MINIMALquot; are two years teachers is 7.4 successes- per- behind. To represent a typical 13%ile points in their students. $10,000, and the Quintile 4 yield is heterogeneous classroom these 4.1. But over half of the $2 million student groups are distributed Under these conditions, when program cost was spent on class- uniformly across all the teacher combined with the children's varying rooms that produced no suc- quintiles. levels of skill deficit, only Quintile 4 cesses. What if actions were taken and 5 teachers are likely to suc- The implications of the Sanders to either a) increase the effective- ceed in bringing students up to findings for these 1,500 students is ness of lower quintile teachers, or b) grade level by the end of the what is most interesting. The Base move more Group 2 and 3 students
  6. 6. Page 5 the program's impact and yield. In into quintile 4 and 5 classrooms? In program's reading interventions into the absence of such data, though, addition, there are borderline cases the next grade in order to finish the proactive management of this type is (the yellow cells in the Base Case job that was started with the border- much harder because decisions table in Appendix B) where some line students. This timeassuming depend on a chain of personal improvement was seen, but not that spending increases proportion- observations, opinions, and judge- enough to meet the proficiency ally by 4% to $2.08 millionthe ments, which without hard data to standard. What if additional funds program's success rate increases by back them up, more often get were spent to extend the program 64%; the cost per student-success translated into politics rather than and put those kids across the finish drops by 37%; and the program's action. line? yield has increased from 2.3 to 3.6 successes-per-$10,000. (See table in Managing with the Learning Delta Appendix D for detail.) What-If Scenarios The point of management informa- As shown in Figure 5 (and in Ap- tion, like the Learning Delta, is to The second scenario in this experi- pendix E), if scenarios two and three identify and target the areas that ment moves Group 2 and 3 students are both put into effect, the pro- need to be better understood and out of the bottom quintile classes gram's success rate would increase that are candidates for scrutiny and and into the top two quintiles. Even by 193%; the cost per student- change. Once exceptions have been assuming that program money success would drop by 64%; and identified, though, the purpose of follows the kids and overall spending the program's yield would increase management information is not to remains at $2 million the pro- from 2.3 to 6.3 successes- per- answer the question quot;what really gram's success rate almost doubles $10,000, a jump of 176%! happened?quot;but rather, quot;who am I to 13%; the cost per student-success going to call?quot; As was illustrated in If, then, we had in our possession has dropped by 50%; and the the above thought experiment, the Learning Delta data that allowed us program's yield has doubled from critical first step is to identify to rank classrooms/teachers in order 2.3 to 4.6 student-successes per exceptions either based upon of average actual Learning Delta per $10,000. (See table in Appendix C specific criteria or using a normative class, we could modify the allocation for detail.) approachas in the examples of students to teachers and/or above. enhance the skills of the low per- In the third scenario we extend the forming teachers in order to increase Unlike impact studies, the purpose of performance management information is not to produce Figure 5 scientifically precise analyses of Summary of Scenarios efficient causes. Instead, perfor- 4500 70.00 mance measures and manage- ment information, to be of value, 4000 60.00 must support real-time navigation Number of Successes / $ Cost per Success in the sense of providing near 3500 immediate feedback on and 50.00 3000 control over speed, course, and Successes per $10,000 resource consumption. So 40.00 2500 because performance measure- ment is very much like a steering 2000 30.00 wheel in a car, it needs to be attached to and a part of a 1500 20.00 school's everyday processes. 1000 Otherwise, teachers and school 10.00 leaders cannot steer around 500 obstacles; they can only drive into them, and then file an 0 0.00 Base Case Scenario 2 Scenario 3 Scenarios 2 + 3 accident report later. Student Successes Students Successes per $10,000 Cost per Student Success
  7. 7. Page 6 model), which need to be improved, 3. Do our measures lead us to Learning Delta Supports a Manage- and which represent performance actions that improve perform- rial Approach to Education Im- beyond what we thought we could ance? provement produce? 4. What is our track record vis a vis A managerial approach to education these standards? At a minimum, a managerial ap- improvement asks different ques- proach to education improvement 5. Are we improving? tions about schools than a curricu- requires answers to five key ques- lum-based reform or one based tions: The key elements of a managerial upon community outreach, etc. But model for education improvement is the answers to the managerial 1. What are the standards that the subject of another paper, but questions can be critical to the represent how we expect educa- some of the important considera- success of these other reforms. For tional processes to perform? tions are listed in Figure 6. example, which attributes of a classroom's, a school's, or a school 2. How do we measure actual Learning Delta, as one top-level system's performance are what we performance against these stan- performance measure, is a first-level expect (under a particular reform dards? response to these questions no matter what other type of reform is being pursued. We know, for example, that a rising fifth grader Figure 6 who reads at a third grade level Key Performance must progress more quickly Management Questions Selected Specifics than other fifth graders in order to meet the standard for reading • Processes include student grouping, curriculum 1. What are the standards competence by the end of fifth planning, lesson planning, instruction delivery, class- that represent how we ex- work/homework assignments, tutoring, testing, stu- grade. We can express that dent performance planning, parent communication, pect educational proc- requirement as a Learning Delta etc. esses to perform? like quot;150 points per term.quot; At • What are the reform-related or other standards that an interim point in the school correspond to each of the educational processes above? Are they widely publicized and/or formally year we can identify students or adopted? classrooms who are progressing too slowly, and we may be able • Are there data capture mechanisms in place that 2. How do we measure ac- to apply additional interventions support performance measures? tual performance against to accelerate them to a success- • Are they fact-based, objective, and timely? these standards? ful conclusion. Or we may find • Are they efficient to use and non-disruptive to teaching? that the original interventions prescribed by the reform model • in use were never implemented Are measures captured and reported frequently 3. Do our measures allow us enough and with adequate turn-around to enable cor- in certain classrooms. In either to improve performance? rective action? case, by capturing and analyzing • Are measures specific enough to enable positive Learning Delta data to identify identification of the people and/or processes needing improvement? exceptions, school leaders can • Are measures specific enough to enable positive do two things: a) reduce the identification of people and/or processes demonstrat- cost of observing (i.e. inspecting) ing excellence? classes that are already meeting performance standards, and b) • Is there a store of historical measurement data 4. What is our track record build their fact-base regarding allowing for trend analysis? vis a vis these standards? • what works and what doesn't in What are the performance trends versus key educational processes and standards? practice. Using this fact-base, schools will be able to make • Are there clearly articulated improvement goals and sustainable, repeatable im- 5. Are we improving? objectives? provements to classroom and • Are they being or have they been met? school-wide processes that raise
  8. 8. Page 7 performance for the future. In other words, we can improve the state of education if we move beyond the quot;random acts of progressquot; too often seen in K-12 schools to data-driven decision-making that focuses on actual learning progress and its underlying drivers. Maximizing Learning Progress by Reducing Student Wait-States Another important reason to adopt the Learning Delta as an interim performance measure is that it provides a better means than annual standardized tests for ensuring that all students will make adequate learning progress.iii Imagine a school that has as its standard that every student will achieve a Learning Delta of 60 points per term. Such a school would be different from it is almost useless if used in isola- These examples show how the conventional schools in several ways: tion. Publicly traded companies get problem of student quot;wait-statesquot; in compared on ratios like Return On • If an advanced student were to conventional classrooms retard Equity (ROE) and Price-Earnings score 80 out of 100 on a pre-test, learning progress. Students who are ratio (P/E), both of which require that studentand that student's substantially above or below the an earnings number. But the key class average in attainment are either teacher would not be able to drivers of ROE for a nationwide waiting for the class to catch up to meet the school's standard of 60 clothing retailer which are them- them, or waiting for someone to points per term of learning prog- selves performance measuresare help them understand what is going ress. Because the student's very different from those for finan- on. As shown in Figure 7, student maximum gain in that class is cial services companies or car wait-states are the mirror image of only 20 points, it is up to the manufacturers. If as investors all we the distribution of attainments in a teacher and the school to make a had to compare the performance of classroom that was grouped by age. change to that student's place- these disparate companies were raw Simply by using pre-tests and ment or curriculum, or adequate earnings numbers, we'd be in trouble Learning Delta measures to proac- learning progress will not be (…more trouble). However, we've tively manage learning progress, achieved. still got to have accurate earnings student wait-states will be reduced • Similarly, if a student scored 10 numbers! They are critical, but they and learning progress improved are also just the beginning. out of 100 on a pre-test, it is un- during a school year rather than likely that student would be able one or two years later. The case for Learning Delta is the to achieve learning progress of same: don’t mistake Learning Delta 90 points in one term. In fact, for the only measure of a school's Limitations that student's low score shows instructional success. Schools, like that she/he is not in the right Learning Delta, though, is not and companies, are more complicated classroom to begin with. Some- cannot be the single measure of than that, and there are underlying thing needs to change if she/he success of a classroom, school, or drivers of Learning Delta that is to meet the school's learning education reform program. Learning should be measured and managed. progress standard. And it is up to Delta as a measure behaves a lot like And there are also other important the teacher and the school to an earnings number for a commer- top-level measures, like graduation change it. cial company. Earnings is a key rates, that should be attended to. indicator of a company's health, but
  9. 9. Page 8 as teamwork, culture, and be- But, as with company earnings, we Conventional schools and class- havior. need Learning Delta as a measure of rooms deliver instruction, post the learning progress produced by assignments, collect and correct • They create new information teachers and schools because homework, and administer tests in flows that provide same-day an entirely manual fashion. Manag- learning progress is the very thing that feedback on key performance ing the effectiveness of large num- schools are supposed to produce. Without attributes. These feedback loops bers of such classrooms eventually such a measure, it is hard to imagine support teachers, students, and becomes infeasible as a simple result how teachers and schools can be parents in their roles by quickly of increasing numbers of students systematically managed and im- identifying exception condi- and classes. proved. On the other hand, perhaps tions and bringing responses to it is because we have been muddling By failing to separate the negative bear so that the exceptions are through without such a measure that effects of size from the positive brought into compliance with schools have remained essentially effects of good management, the the school's predefined, pub- unimproved over the last forty critics miss the opportunity we are lished, and acknowledged stan- yearsin spite of the tripling of per presented with: high-performance dards. pupil expendituresiv. school management can be scaled- But high performanceas re- up if schools would only do what searched in Samuel Casey Carter's the rest of the U.S. economy did the book No Excusesvis based starting 20 years ago as a response to Part 3: Putting the Learning largely on quot;management-by-walking- growth: get digitized. Delta into Practice aroundquot;. And it is as a result that high performance schooling is We can identify high performance Digitize what? negatively correlated with school schools because they are the ones size. that produce greater than average Four or five years ago the excite- Learning Deltas. And as with other ment about quot;cyber-schoolsquot; and the Critics of the quot;No Excusesquot; research human enterprises, high perfor- quot;virtual classroomquot; was mostly often claim that the schools studied mance schooling is largely a matter focused on distance learning and are statistical outliers that are irrele- of good leaders providing sound how it enabled quot;anywhere, anytime vant to the problems of large management using effective tech- learning,quot; i.e. the elimination of time schools because of the special niques and systems. For example, and space as barriers to taking conditions involved, small size KIPP Academy in Houston enrolls courses or earning degrees from off- among them. They ask, quot;What is the fifth graders who for the most part campus locations. Subsequently, lesson of high performance school- function at a third grade level. In the educational institutions began to ing for the management of tens of space of four years KIPP turns out discover the benefits to on-campus thousands of students rather than ninth graders operating at a ninth students of on-line courses, and so hundreds?quot; And they conclude that grade level--as defined by exclusive the quot;Web-enhanced campusquot; there is none. preparatory schools. On average, became an object of enthusiasm in There are distinct diseconomies of KIPP's students make 6 year's higher education. Accordingly, the scale at work and classic quot;locus of academic progress in 4 year's time. major distributed-learning platforms controlquot; problems that stand in the like Blackboard, WebCT, and Lotus KIPPand other high performance way of scaling up quot;No Excusesquot; LearningSpace tried to provide on- schools like it employ methods, practices. Simply put, attempting to line analogues to the blackboard, the practices, tools and techniques that duplicate quot;No Excusesquot; results using lectern, the classroom discussion, are different from those found in quot;No Excusesquot; techniques in large the student's raised hand, and so ordinary schools. And they are well school settings would require many forth. As a result, the surrounding managed by leaders who are em- more principals and instructional debate about quot;e-Learningquot; has powered to do so. For example: leaders quot;managing-by-walking- mostly focused on whether an on- aroundquot; than school systems are line class discussion is as good as the • They explicitly define common able or willing to pay for. face-to-face variety, or whether roles and standards for teachers, listening to a lecture is as rich an students, and parents to play But a well understood lesson from experience on-line as it would be within well-defined processes outside the education sector is that actually sitting in a lecture hall. covering attendance, classwork, even very effective manual processes homework, and testing, as well break down under high volumes.
  10. 10. Page 9 The point I wish to make here about quot;digitizingquot; the classroom is differ- Figure 7 ent from the quot;virtual classroomquot; Key Attributes of a Learning Delta Management System discussion described above. Class- rooms, schools, and school systems • Curriculum-Neutral  Accommodates whatever curriculum is 1. Least already in use by a school. Does not impose a new one. are enterprises, which–like others– intrusive • Pedagogy-Neutral  Does not require that classroom-based are comprised of methods, practices, teaching be replaced with computer-based instruction. tools, techniques, processes, knowl- • Ease-of-Use  Fits well with and supports classroom edge and people. What needs to be routines. Data capture and feedback mechanisms are user- digitized in K-12 education are the friendly. classroom workflowswhich • connect the teacher to the students, Allows use of hardware with lowest total cost of ownership: 2. Least handheld computers like Palm OS or Pocket PC devices. parents, and the schoolso that costly • Minimizes requirement for network and/or centralized applica- basic data on what is getting done tion support though use of application-service-provider (ASP) and what isn't can be used to man- model. age classroom performance and • improve results. Capturing data like Should automate the testing process from start-to-finish in 3. Most order to relieve teachers of manual tasks and free up new time Learning Gap, planned vs. actual efficient for teaching. Should not simply impose a new clerical process. Learning Delta, and the like is both • Should provide teachers with detailed, useful performance necessary and desirable because it analyses on demand in order to increase teacher and class- enables instrumentation of the room effectiveness. classroom rather than virtualiza- tion. first step to making that type of when high-stakes tests are adminis- Using digitized performance infor- analysis-driven facilitation possible. tered, but by then it is too late to mation, school and district leaders improve outcomes for the students can exercise anytime, anywhere, being tested. Digitize How? quot;line-of-sightquot; management of bricks-and-mortar classrooms in With digital mechanisms for meas- To put into practice data-driven much the same way as with virtual uring the actual learning progress of performance management based on classrooms, but without the disrup- each student, schools could begin to the Learning Delta alone, three key tion of trying to deliver instruction correlate learning results to teaching components are required: in a new way. Digitizing the class- performance, rank classrooms room to accomplish instrumenta- accordingly, and then scrutinize • A testing process producing tion would enable broad-based classroom methods, tools and scales (scores) that have a de- performance improvement through techniques to identify improvement the regular use of on-line reporting fined (i.e. mapped) relationship actions. The current state in our facilities and quot;drill-downquot; analysis to: to the curriculum, and which schools, though, is that very little produces measurements that a) identify the Master Teachers information comes out of class- extend above and below grade who are the exemplars of high rooms once the teachers and stu- level; performance that should be dents have gone in. As a result, emulated, • A database and data capture instructional leaders, administrators, tools that track individual stu- b) identify the classrooms and and parents cannot tell on a daily, dent scores and maintain links teachers needing help to im- weekly, or monthly basis which prove their performance, and to classes, teachers and schools; classrooms are on schedule and and which are not, which curriculum c) bring the two groups together elements--new or old--work very so that expertise and experi- • A reporting and analysis facility well or very poorly, or which stu- ence can be shared. that supports the type of data- dents are functioning at grade-level, driven decision-making illus- That is a fitting role for school who have pulled ahead, and who trated above. leaders and managers, and digitiz- have fallen behind. Eventually, ing the classroom is the critical these questions may get answered
  11. 11. Page 10 These components comprise a the data fairly or constructively, the Change Management and People minimum infrastructure for data- environment needs to change to Factors driven performance improvement in address those fears and establish schools that would relatively simple common goals. Most importantly, though, a to develop and implement. And for Learning Delta Management In the case of schools, control of the many schools having a system with System can only enable perform- environment is shared mostly just these components would ance improvement if the real solu- between two groups: the teachers' represent a logical, high-impact tion is already in place: a concerted unions and school administration. If starting point for quot;getting digitalquot; and serious demand for meaningful either one of these groups is op- and starting to practice data-driven performance data from schools. That posed to adopting a new system, it decision-making. But for a school to will most probably fail. Similarly, if has to come first, or anyone at- obtain performance tracking and it's not in the interest of these tempting to implement such a management functionality in the groups for learning performance system will likely fail. educational software market often management practices to take root requires that an entire instructional As experienced system implementers and spread, then they will not. So management system be pur- can all attest, quot;people factorsquot; are unlike smaller class size and in- often more decisive than a system's chasedalong with substantial creased professional develop- design or features in determining the amounts of quot;course-warequot; and mentwhich probably benefit kids success or failure of a project. K-12 embedded curriculum that may not in some ways and certainly benefit schools are no exception. Three be appropriate or useable by class- the adults performance measure- dimensions always need to be room teachers. In addition, such ment as a means of managing addressed: systems require amounts of com- schools is problematic because it has puter hardware that many schools • Ability - Do the intended users the clear potential to create winners can ill afford. Taken together, these of a system have the training, and losers. It increases the riskiness attributes of vendor offerings time, and resources required to of being a teacher or an administra- represent substantial barriers to use it successfully? tor in a school that is not perform- adoption by schools that compound ing well, and so it is harder to win the usual issues of cost, staff • Willingness - Are the users of a votes on the quot;willingnessquot; dimen- training, and curriculum manage- system willing to change how sion, both among a school's teaching ment. they go about their jobs in order staff and the groups in charge of the to use the system successfully? The broad-based application of environment. Learning Delta measures in schools • Environment - Do system users Of course, parents and communities requires that a more focused and believe that doing what it takes have long wanted tools to measure easier-to-implement Learning to make the system successful our children's absolute and relative Delta Management System be will be recognized and rewarded educational progress, but they are made available. To be broadly as a positive achievement by the not the ones being asked to change. applicable in divers types of schools groups that control the environ- And they do not control the envi- and communities it should: ment? ronment. • Accommodate rather than It turns out that the first two dimen- disrupt the established instruc- sions, ability and willingness, are tional methods and materials, heavily conditioned by the third: environment. For example, if a • Be designed for very low total school's teaching staff are unable to cost of ownership, and implement a new learning manage- • Provide teachers, students, and ment system because they don't have the training or the release time to get parents with the usual and cus- the training, the environment needs tomary benefits of automation to change to accommodate those rather than creating additional needs. If a school's teaching staff work (see Figure 7). are opposed to a learning manage- ment system because they don't trust administrators/management to use
  12. 12. Page 11 PRESS RELEASE (Excerpt) Conclusion PALO ALTO, Calif., May 12 /PRNewswire/ -- NewSchools Venture Fund announced the launch of its new Performance Accelerator Fund, targeted at $20 million and designed to invest in ven- There are signs of positive change, tures that enhance the capacity of school systems to produce high levels of student achieve- however. Because of the No Child ment. . . Left Behind Act and other factors, quot;As districts work to increase student achievement, we must ensure that they have the tools they the secular climate for learning need to be successful,quot; said Kim Smith, co-founder and CEO of NewSchools Venture Fund. . . performance measurement in State and federal education policies are increasingly pushing for school systems to shift from classrooms is probably better than it compliance-driven organizations -- governed by rules, regulations and court orders -- to per- formance-based cultures focused on high student achievement. . . has ever been. But ideology and labor relations remain very high . . .The ventures in the Performance Accelerator Fund will focus on developing people who know how to lead and teach in a performance-based environment; designing tools that give those barriers to the success of data-driven people the information they need to make good instructional decisions; and supporting practices learning management within K-12 that promote and reinforce success. The Fund will identify, fund and build ventures that address education. human capital -- including the recruitment, preparation and support of high-quality teachers and leaders -- and performance tools -- including information systems and assessment tools that enable teachers and leaders to make data-based decisions about their students' in- One bellwether of this positive struction. (emphasis added) change may be the New Schools Venture Funda leading venture philanthropywho in their recent systemic improvement in K-12 One particularly good chance to press release point to a shift away schools is to transform the class- demonstrate the value of Learning from schools being compliance- room into the atomic unit of data- Delta management is, then, is for the driven organizations to perform- driven performance management. quot;managerial modelquot; of school ance-based cultures focused on The first step is to digitize the improvement to be incubated inside achievement. classroom and to start using meas- a substantial education reform ures like the Learning Delta to Venture philanthropy-based educa- program or EMO while the market manage and improve school per- tion reforms are in an excellent for Learning Delta management formance as a regular part of a position to demonstrate the power software and services develops. To school's weekly routine. for positive change that performance be convincing, a demonstration of measures like Learning Delta repre- the managerial model needs to span sent. Education management multiple classrooms, schools, and organizations (EMOs) like Edison even school systems. It needs to Schools and Chancellor Beacon have shown its effectiveness on a Academies are also in a position to substantial scale in order to be a make a positive impact. In fact, candidate solution for the systemic because they are for-profit and so improvement of education in this get criticized for diverting money country (something that No Excuses quot;away from the children,quot; they schools cannot do). should be strongly motivated to use Learning Delta measures to en- sure—and then later prove—that * * * they are doing a better job than traditional schools. And so paying Because improving the effectiveness profits to shareholders is in some of teachers can improve education sense justified if it makes possible a more than by any other single factor, demonstrably better education for our greatest opportunity to achieve our kids. For more information contact: Gregory D. Luther, Founder and Principal ! High Delta Learning, LLC ! P.O. Box 398 ! Litchfield, CT 06759 tel: 860.567.8242 ! e-mail:
  13. 13. Page 11 Appendix A EXAMPLE OF LEARNING DELTA DATA - Classroom A Mr. Smith Teacher Name: 80 Mastery Score: 40 points per term Learning Delta Standard: Plan vs. Actual Timeframe: Aug Sep Oct Dec Learning Delta Learning Gap / Interim Interim Final Planned Learning Learning Learning Learning Delta Delta Delta Delta Aug - Dec Student Pre-Test Test 2 Test 3 Final Test Student 100 39 41 47 24 61 34 82 43 2 Student 101 21 59 28 21 33 19 55 34 -25 Student 102 24 56 36 37 55 47 80 56 0 Student 103 55 25 57 6 72 25 95 40 15 Student 104 35 45 53 56 58 35 67 33 -13 Class Average Test Score 35 44 56 76 Class Average Learning Delta 45 29 64 41 Finished a Class needs to Going too Speeding little under achieve this slowly! to catch plan average quot;speedquot; up 41 Teacher's Learning Delta Result for Term 40% % of Students Meeting Learning Delta Standard Teacher Inter-ranking Average Class Name Rank Learning Delta Quintile Hume 1 72 Q1 Hobbes 2 68 James 3 66 Locke 4 60 Q2 Descarte 5 58 Smith 6 41 Weber 7 40 Q3 Berkeley 8 39 Russell 9 38 Machievelli 10 32 Q4 Mosca 11 22 Michels 12 18 Rousseau 13 16 Q5 Nozick 14 15 Rawls 15 15