1) The document discusses ways to identify and measure qualities that are valued by educators and the public in schools, and suggests new ways to report achievement of these measures. 2) It provides a quick preview of new assessment tools designed to measure 21st century skills and describes the assessment practices of schools recognized as "Schools of the Future." 3) The document discusses challenges in using standardized test data and presents alternatives like developing "replica tests" based on released test items to allow international benchmarking and comparison.

1. Measuring what we value:
21st Century Assessments for Independent
Schools
Douglas Lyons, Executive Director, CAIS-CT
Andrew Niblock, Lower School Head, Hamden Hall
(CT)

2. Good to Great
Jim Collins
• Determine what you value most, then find a
way to measure it.
• Success can be a powerful disincentive: it
may be hard to become a great school if you
are a very good school

3. Goals of this workshop
1. Identify measures of school quality that have
historically been valued by educators and/or
the public.
2. Suggest new ways to report achievement in
those measures.
Why?
• To better tell our story (the Value Proposition)
• To make certain that these measures have
stature appropriate to their significance

4. Goals, cont.
3. Provide a quick preview of new and
emerging assessment tools –instruments
designed to measure skills that are increasing
in demand in the new century
4. Describe the assessment practices in a
select group of schools that define themselves
– and are recognized by others – as “Schools of
the Future”

5. Criterion 13:
The Standards require a school to provide evidence of
a thoughtful process, respectful of its mission, for the
collection and use in school decision making of data
(both external and internal) about student learning.

6. “Not everything that can be counted,
counts, and not everything that counts
can be counted”

7. “It would be easier to change the course of
history…”

8. Powerpoint presentation is available at
www.caisct.org
look for ADVIS presentation on home page

9. Jim Collins – “whenever possible, use the
language of metrics to define what you value”
What is the language of metrics?

10. 4 “ways of knowing”
• Data: raw input, no context, facts, figures,
symbols
• Information: organized, processed, analyzed
data
• Knowledge: information with higher context -
accurate, relevant, current
• Wisdom: evaluated knowledge; merged with
life experience

11. • Heads letter in viewbook; wisdom
• Description of school history and mission in
viewbook: knowledge
• Course catalogue: information
• SAT scores, college placement stats: data

12. What are people most interested
in ?
Data!
• conveys a lot of information - quickly
• Is viewed as objective, “no spin”
• Can be benchmarked, used for
comparison
Data is the language of metrics

13. Risks / Misuses of Data
• Garbage in; garbage out
• Data is easily manipulated, corrupted:
Harvard Business School caution
If you torture data long enough, it will
admit to anything

16. Data Management / Data Creation in
the independent school community
The Challenge:
• To frame the data that define us, or have
defined us in the past, in ways that do not
elevate modestly valuable information
• To gather and/or to present new data that is
beneficial to educators in our planning for the
future and is data that measures performance
in relation to the achievement of our highest
goals.

17. The S.A.T.
• Decreasing in stature, but still powerful
• Has poor validity statistics
• Does not measure 21st century Skills
• Historically, did not provide faculty with
instructionally useful information
• Consider giving the School Day SAT with
Enhanced Scoring

19. Standardized Achievement Tests
These test are increasing in stature
What happened to elevate these tests above all other
forms of data, in public education and to a lesser
degree, in independent schools?

21. “Effects” of standardized tests
today:
from educating the whole child to
educating the whole test-taker
• Hyper-focus on scores, minor fluctuations
• Unprecedented “score chasing”
• M.D.I “measurement-driven instruction”
• Mind numbing drill and practice

22. Most popular form of data presentation:
percentiles
Math Math Reading Reading
concepts applications comprehensio Skill
n
1 91 87 83 83
2 88 84 85 83
3 92 90 88 88
4 84 89 80 90
5 84 86 78 82
6 89 80 81 79
7 90 89 90 87
8 87 87 81 83

23. Second most popular form of data presentation:
Grade Equivalence
Math Math Reading Reading
concepts application comprehensio skill
s n
1 1.8 2.1 2.3 2.5
2 2.6 2.8 3.3 3.6
3 3.5 4.0 3.9 4.0
4 4.5 4.9 4.8 4.7
5 5.8 5.2 6.3 6.0
6 6.6 7.3 7.0 7.5
7 8.2 9.0 8.8 8.8
8 9.0 9.6 9.9 10.4

24. A better way to present achievement
data
• Determine and defend your norming
group
• Determine a worthy and realistic goal
within the norming group
• Publish data relative to the goal

25. Goal: to score within the top third of norming
group on all subtests
Reading Reading Math Math skill
comprehensio skills applications
n
1 √ √ √ √
2 √ √ -4 √
3 √ √ √ √
4 √ √ √ √
5 √ √ √ √
6 √ √ √ √
7 -2 √ √ √
8 √ √ √ √

26. The International Database
• Most schools administer one or more normed
tests that compare American student
achievement with American peers
• Is there a way to assess our international
competitiveness?
• Would this data be valuable to us?

27. Benefits of the New York State
“Truth in Testing” Law
• Thousands of released items available to
educators
• Released items available for NAEP tests
• Released items available for TIMMS tests
• Released items available for PISA tests
• Construct your own “replica test” or form a
research partnership to develop replica tests

30. CAIS score reports for TIMMS replica test
ABC Country Day School
TIMMS “released item” test results
95% of students scored in the top 1/2 of I.A.
92% of students scored in the top 1/3 of I.A.
90% of students scored in the top 10% of I.A.

32. Course of study guides: how we describe
our program
US HISTORY HONORS
GREENWICH HIGH SCHOOL
This course addresses the events and experiences
that comprise American history from the period of
European colonial settlement through the Civil War
(1st semester) And from the period of Reconstruction
through the advent of the Second World War. The
goal of the course is to provide for our students
substantial opportunity to develop the ability to make
informed and reasoned decisions as citizens
concerned with the public good. (More text follows)

33. US HISTORY HONORS
A study of the events and critical changes
that took place from the first American
settlement to the present day.We will focus
on these events in the context of larger
themes; including the shift from an
agricultural to an industrial society, the
recognition and cultural identification of
different groups of people, the transition to a
stronger national government, immense
territorial expansion, technological change
and globalization.

34. History at Lakeland Prep:
In the four year History sequence at Lakeland Prep, all students will
complete the following Demonstrations of Learning:
• 24 research based position papers (4 to 7 pages) in which an
analysis, synthesis and/ or evaluation of both original and
modern sources is offered in answer to a provocative question
in history.
• 6 research based position papers (10 to 15 pages) focused on a
students original response to one of identified Essential
Questions in American History.
• 12 oral presentations
• 8 collaborative projects,
• 3 projects completed in collaboration with students in other
schools and/ or countries
• 4 interviews with elected officials
• 6 Letters to the Editor written on a current topic in local and/or
state government

35. HSSSE -
what do you believe your high
school emphasizes most?
• 21% memorizing facts and figures
• 32% understanding information and
ideas
• 22% analyzing ideas in depth
• 68% my school challenges me
academically

36. HSSSE:
what instructional methods do you
find exciting or engaging?
• 60% discussion and debate
• 60% group projects
• 44% student presentations
• 24% teacher lectures

37. Using an Engagement
Survey
• Use the HSSSE and contribute your
school data for national benchmarking;
• Or, create your own survey, then
compare your results to the national
HSSSE 2009 data, where applicable .

42. The Emergence of Longitudenal
Data
• The National Clearinghouse has 93% of all US
colleges collecting and providing longitudenal
data
• Two independent school associations require
student tracking for accreditation (freshman
GPA):
• CAIS Canada ntp@cais.ca
• ISASW www.isasw.org

43. Measuring teacher engagement,
professionalism, attachment to school
Can that be quantified?
• Longevity statistics
• % faculty with advanced
degrees
• % faculty participating
in Annual Giving

44. Occasional Teacher Absenteeism
• Reported in “school district report cards”
• Measures only consecutive days absent, less
than 5, 7 or 10 days. After the threshold,
absence considered “long term illness”,
removed from calculation
• O.C.A considered an indicator of faculty
commitment, professionalism, attachment

45. Greenwich CT High School: State Report Card

46. Measure your O.T.A.
• Compare it to your local or state public school
average
• Inform your Board of Trustees
• Inform your Parents Association

47. Measuring teacher effectiveness: in the
best public schools, teacher evaluation is
an informed, professional process
How can independent school leaders
become more skilled in the clinical
observation and evaluation of teaching?
How can we measure what we value in
teaching practice?
The TIMSS video study – teaching
practices in 7 select countries

52. New and emerging assessment
instruments – workshop goal 3
• We have just reviewed ideas for improving
the use of existing data
• What about new tools and techniques?
• Are the assessment instruments and
practices of earlier generations obsolete or
incomplete?
• If so, why is that true?

53. Did something occur on January 1, 2001 that
changed everything?

54. Turn of the millenium events
First successful HTTP communication (modern WEB)1990
Netscape, easy to use browser 1995
Google, as a research project 1996
LiveJournal, Blogger, hosting sites 1999
Ericsson smartphone 2000
Wikipedia 2001
Facebook 2004
MIT Open Courseware 2004
YouTube 2005
Skype 2005
The World is Flat first edition 2005
Internet/Multimedia Smart phone (iphone) 2007
Global financial crisis 2008
Khan Academy (2600 videos and growing) 2009

55. 100 most influential people of the 2nd
Millenium
• Jonas Salk 97
• Steven Spielberg 91
• Elvis Presley 57
• Gregor Mendel 42
• Martin Luther King 33
• Henry Ford 29
• Michelangelo 19
• Galileo 10
• Columbus 6
• Charles Darwin 4
• Who is Number 1?

56. Johann Gutenberg
The printing press was information technology
What about modern day visionaries
in information technology?
What number is Bill Gates?
Steve Jobs?

57. What’s Past is Prologue
vs The Future is Not What it Used
to Be
• “What does an educated person need to know?”
• Education is defined by a remembered body of knowledge,
the “canon”
• Critical, sarcastic view of the canon: Education as
inoculation:” American history? I had that, Tetanus shot?
I had that…)

58. 20th century technology:
• Radio, television and film had great promise,
but no demonstrable effect on schooling
What 20th century technology had a
revolutionary effect on teaching and learning?

59. SCANTRON!
Bubble answer sheets!
• 1948 – Scantron Corporation revolutionized the
speed and efficiency of data collection and advanced the
notion that student proficiency and school quality can be
determined through mass-produced, multiple choice
metrics
• Scantron, to this day, has had a greater impact on k-12
curricular design than any other technology in history.

60. in post scantron decades:
“What gets measured is what
gets taught”.
• Tests “drive” instruction in ways that mimic
both the content and format of the test.
• What gets measured is almost exclusively
content
• In the Information Age, we measured recall of
information

61. In 2012, in The Conceptual
Age
There are no books, conferences, op-ed pieces on “21st
Century Content”.
The canon has been buried under the information
explosion
However,
There is near universal agreement on a short list of 21st
century skills.
There is near universal agreement on the need to
employ technology in a thoughtful but robust manner

62. Nicholas Negroponte
on applying technology in a robust
manner:
• “When you drop a penny into a glass of clear water, you
get a glass of clear water with a penny in it; the change is
additive.”
• “When you place a drop of red dye in a glass of clear
water, you get a glass of pink water. The change is
ecological.”
• Technology in education needs to be ecological;
pink water

63. The i generation
• Defined mostly by their use of technology
• Accustomed to learning things on their own and
learning from peers
• Expect technology to be interactive and customizable
• Non-linear thinkers; web thinkers, scanners, multi-
taskers

64. Clay Shirke, futurist describing the i generation:
“A father sets up a new television in the living
room. His 4 year old daughter is seen
rummaging through the box. What is she
looking for?”
Passive media experiences
will hold less appeal for this
generation

65. The 21st century skills movement,
the Schools of the Future movement, focus on
the development of these skills:
Communication
Collaboration
Critical/Analytical Thinking
Creativity
Problem-solving
Content is still important; but content in these areas will
need to be acquired through active exploration as well as
through instruction.

66. American work in the 21st century

67. Non–Routine Tasks
defined in the Journal of Economics, volume
118
• Gathering, synthesizing, and analyzing information.
• Working autonomously to a high standard with
minimal supervision.
• Leading other autonomous workers through
influence.
• Being creative and turning that creativity into action.
• Thinking critically and asking the right questions.
• Striving to understand others’ perspectives and to
understand the entirety of an issue. Communicating
effectively, often using technology.

69. Current assessment tools do not
measure these skills.
You cannot have 21st century schools
using 20th century assessments.

70. Ideal Assessment:
Provides accurate demonstration of student
proficiency
Yields information for faculty planning
Is valid as a learning experience in and of itself
An assessment of, for and as learning

71. What is a performance task?
Students assume roles in a
scenario that is based in the
"real world" and contains the
types of problems they might
need to solve in the future. The
task requires critical thinking,
analytical reasoning and
problem solving.
Communication skills are used
in describing the solution.

72. Ohio Mastery Test, Grade 9
• Ms. Johnson installs new insulation to save money on heating costs,
but then learns that her bills have not declined by much from the
previous year. Her contractor points out that heating costs have risen
and weather has been colder. Ms. Johnson wants to find out how much
she has actually saved due to the insulation she installed. On the basis
of the situation painted above, details about Ms. Johnson’s heating bills
(rates, units of heat used), temperature changes, and some initial
information to help them begin to research “heating degree days” on
the internet, students are given two tasks:
• (1) Assess the cost-effectiveness of Ms. Johnson’s new insulation and
window sealing.
• (2) Create a short pamphlet for gas company customers to guide them
in making decisions about increasing the energy efficiency of their
homes.

87. Students get answers to
questions THEY ask
• For example
– Family history – Is this breast cancer
possibly caused by abnormal oncogene
expression? If so, certain types of
hormonal therapy or receptor antagonists
are more effective treatments.
– What level of stage III cancer, A, B or C?

90. CBAL
• Extended, constructed-response tasks that are
delivered by computer and automatically scored.
• Pilot testing occurred in 2010 and 2011, spring of
2012.
• Tests should be available for use in 2012.
• Sample tests available online
Website information is in your folders.

93. features real-time, scenario-based tasks that
measure an individual's ability to navigate,
critically evaluate and understand the wealth
of information available through digital
technology

95. Ken Robinson
Age and education:
• Increase routines of behavior and habits of
thought (left brain logical thinking )
• Decrease divergent thinking (free association
of ideas. Right brain, creative thinking)

98. Creativity Index: the new
state mandate?
• Gov. Deval Patrick has made Massachusetts
the first state in the country to call for the
formation of a creativity index aimed at rating
public schools statewide based on their ability
to teach, encourage and foster creativity in
students.
• Similar legislation is pending in California and
Oklahoma

99. Torrence Test of Creative Thinking
Verbal Activity 4: Product
Improvement
Look at the stuffed toy elephant in
the drawing. It is six inches
tall and weighs a half pound. In the
space provided, list the cleverest, most
interesting and unusual ways you can
think of for changing this toy so that children will have
more fun playing with it. Do not worry about how
much the change would cost.Think only about what
would make it more fun to play with.
Activity 2 and 3: Guessing Causes
and Guessing Consequences measures
“idea fluency”

100. What do you get if you solve this problem and
visit the website?

101. The 4th goal of this workshop:
Examples of assessment practices in a select group
of schools that define themselves – and are recognized
by others as “Schools of the Future”

102. Schools that define themselves as
Schools of the future
• Who are the pioneers?
• What do these schools have in
common?

109. Science Leadership Academy
Philadelphia, Pennsylvania

110. All of these schools have 2 common
characteristics
• urban public charter schools.
• experimenting with a dramatically different
view of teaching and learning; A
Collaborative, Conceptual Model

111. 21st Century
education: from
coverage model to
conceptual model
• Recall of information (content) is still important
• Skill in accessing and selecting information (internet
searches now deliver 2000 hits) vitally important
• Ability to use or apply information in new and/or novel
settings most important (Its not what you know, but
what you can do with what you know)
• Expanded role of the teacher: guide, coach, facilitator

112. Schools of the Future:
characteristics
• Performance tasks
• Project-based learning, individually and in groups
• Capstone projects, individually and in groups
• Independent study
• Online learning, online tools (courseware, Skype,
You Tube, Ning, Moodle, Web 2.0 etc…)
• Students given choices in assignments and in
demonstrations of mastery

113. Schools of the future, cont.
• Extensive use of essential questions relating to
content area
why, how and what if questions
• Computer-adaptive learning (program adjusts to
student skill level)
• E-portfolios, published within the community or on the
web – seeking Facebook-type conversations in the
academic community, on academic topics
• Flipped classroom strategies – routinely or
occasionally
• Partnerships, learning experiences beyond the
school campus
• RUBRICS used to assess performance

115. RubricBuilderV2.xhtml

116. A New Definition of School
“we need to invert the conventional
classroom dynamic: instead of teaching
information and content first, and then asking
students to answer questions about it second,
we should put the question/problem first, and
then facilitate students with information and
guidance as they seek the answer and hold
them accountable for the excellence of their
solutions and of their presentation of their
results”.
-Ted Mccain Teaching for Tomorrow

119. What about independent
schools?

120. Independent schools
• lead the nation in communication skills; writing,
speaking, the performing arts
• Engagement has been supported by very strong
student-faculty relationships
• An incremental approach to the challenges of the
future; preserving strong, successful, traditional
programs while expanding collaborative learning,
online learning, project-based assessments,
exhibitions of learning and use of digital portfolios
• Growing interest in “Essential Questions” theory of
learning

125. http://stolaf.edu/depts/cis/wp/ebouvier/ind
ex.html

126. Lessons from our
research:
Schools in the 21st century will
define success in much broader
terms
Great Schools in the 21st century
will include some that have far fewer
resources than independent
schools. What they have is the
freedom to take big risks in
designing innovative cultures

128. “Measuring What We Value”
Sites Referenced in Presentation
Hechinger Article containing multiple links of sample questions on new 2012
assessments
http://hechingered.org/content/are-new-online-standardized-tests-
revolutionary-decide-for-yourself_5655/
Information on Torrance Test
http://www.ststesting.com/
High School Survey of Student Engagement
www.indiana.edu/~ceep/hssse/
College Student Experiences Questionnaire
http://cseq.iub.edu/cseq_generalinfo.cfm
National Student Clearinghouse
www.studentclearinghouse.org/
The Self-Regulation Questionnaire
www.psych.rochester.edu/SDT/measures/SRQ_text.php
MHISC: Mental Health in Independent Schools
http://www.harthosp.org/InstituteOfLiving/OtherServices/MHISC/default.aspx

129. C-bal Cognitively Based Assessment for Learning
http://www.ets.org/research/topics/cbal/initiative/
The CWRA: College to Work Readiness Assessment
www.cae.org/cwra/
Science Leadership Academy
www.scienceleadership.org/
High Tech High
http://www.hightechhigh.org/
New Tech High
http://newtechhigh.org/
Big Picture Learning
http://www.bigpicture.org/
NYC i school
http://www.nycischool.org/
Microsoft School of the Future
http://www.microsoft.com/education/en-
us/leadership/partners_in_learning/Pages/School-of-the-Future.aspx

130. Performance Assessment Group of NYC Schools (check out the rubics!)
http://performanceassessment.org/
Rubics – Association of American Colleges and Universities (rubics on critical
thinking, creative thinking, problem-solving and others!)
www.aacu.org/value/rubics
Avenues
http://www.avenues.org/
Haverford School (Decision Education)
http://www.haverford.org/
Decision Education (critical thinking/character education program)
http://www.decisioneducation.org/
Independent Curriculum Group
http://www.independentcurriculum.org/
Greens Farms Academy
http://www.gfacademy.org/RelId/607374/ISvars/default/Capstone.htm
Hotchkiss/Loomis Collaborative Learning Project
http://tinyurl.com/3kq8v8f

131. Project-based Learning (450 sample projects – all subjects and grade levels –
templates for organizing projects)
http://pbl-online.org/
Siemens Challenge (sample award-winning projects)
http://www.wecanchange.com/
American Society for Engineering Education (ASEE)
http://teachers.egfi-k12.org/
Exploravision (sample award-winning student projects)
http://www.exploravision.org/
Toyota Tapestry Grants for Science Teachers (sample grant-winning ($10,000) projects)
http://www.nsta.org/pd/tapestry/

132. aniblock@hamdenhall.org

133. lyons@caisct.org