Tools and resources for continuous
improvement of technology in
schools
Kevin Oliver, LaTricia Townsend, Rodolfo Argueta,
Daniel Stanhope
Institute on Leading Innovation: Implementing Effective 1:1 Learning
Technology Programs
July 8-10, 2009. The Friday Institute, NCSU, Raleigh, NC

Description
This interactive session will highlight resources
school leaders can utilize to facilitate data-
driven decisions to improve the effectiveness of
a technology project. Participants will learn
about a variety of data collection tools freely
available to them, practice using these tools,
and discuss ways in which the results may
shape practice in their schools.

Defining Evaluation
• two common evaluation modes--formative and
summative
• formative evaluation is concerned with collecting
data to help revise an innovation of interest (e.g.,
one-to-one computing)
• summative evaluation is concerned with
collecting data to help judge the worth of an
innovation and whether to adopt/reject
• 1:1 leaders may be involved in both modes

Evaluation vs. Assessment
• evaluation is concerned with improving or
judging the worth of some innovation
• assessment is concerned with how much a
student knows
• evaluation is the more inclusive term, often
making use of assessment data as one data
source, in addition to surveys, observations,
interviews, and more

Evaluation vs. Assessment
Evaluation
Assessment
questionnaires
pre-test/post-test surveys
observations interviews
artifacts

What is Commonly Evaluated?
• instructional materials (e.g., a problem-based
learning lesson and resources)
• projects (e.g., school professional development or
technology plans)
• programs (e.g., one-to-one computing in a school
district or state)
• in project or program evaluation, you may study
instructional materials purchased or created, but you
will also look at leadership, professional
development, collaboration, sustainability, etc.

Where to Begin?
• evaluation questions are difficult to write for
project or program evaluation with so many
factors to consider
• evaluation models can help you develop
evaluation questions to study the most
important elements
• for example, Stufflebeam's CIPP Model
encourages a close look at Context, Inputs,
Processes, and Products

CIPP Model
• context questions seek to identify needs of target
pop., opportunities to address them, determine how
well goals address needs
• input questions seek to define capabilities, project
strategies and designs, types of support useful in
reaching goals, what was required to achieve goals
• process questions seek to define deficiencies in the
process or implementation, how resources were
allocated, barriers to success
• product questions seek to define outcomes, describe
lessons learned

Flashlight Model
• the Flashlight program is another evaluation
model developed by AAHE for technology-
based evaluations
• Flashlight is based on the concept of "triads,"
consisting of the relationship between
outcomes (goals/objectives), strategies, and
technologies, tools, or supporting materials

Flashlight Model

Logic Models
• logic models can also help you identify
questions for your evaluation; they contain
four elements:
1. inputs or the resources that go into the project
2. activities that take place as part of the project
3. short term objectives (e.g., "students will
use...")
4. long term goals (e.g., "increase student
scores...")

After Generating Evaluation
Questions
• helpful to generate a table that matches
evaluation questions with indicators and
benchmarks of success (expectations), and
data sources that you will use to check on
that success (today's session topic!)
Questions Indicators Benchmarks Data Results,
Sources, Changes,
Instruments Outcomes

Indicators
• indicators are a continuous factor used to
describe a construct of interest (e.g.,
unemployment percentage, lots per acre,
violent crime per 1000 citizens, etc.)
• have some historical/past value, present
value, and future value
• value is in constant flux and can be changed
by projects we are evaluating

Benchmarks
• if an indicator states: number of students who
use the Internet for research
• a benchmark would specify the time interval
by which you would expect to see changes
• if we know 40% used Internet last year before
our project was implemented, we might
expect 60% to use it this year, and 80% next
year

Examples
Indicators Benchmarks
number of wireless 1 in 2006, 2 in 2007,
access points and 3 in 2008
mobile labs in the 5 in 2006, 10 in 2007,
district 15 in 2008
teacher access to 80% in 2006, 90% in
e-mail 2007, 100% in 2008

A More Detailed Planning Grid
• Irving, TX long-range technology plan
• different look, similar planning items: goals-questions,
objectives-benchmarks, evaluation-data sources

Remainder of Session
• focus on tools and resources you can use to
collect data in evaluating a 1:1 computing
program
Questions Indicators Benchmarks Data Results,
Sources, Changes,
Instruments Outcomes

Data source instruments

School Technology Readiness
Measures
CDW-G One-to-One Readiness Assessment
Online access: http://tinyurl.com/35v305

School Technology Readiness
Measures (continued)
ISTE/CEO Forum: STaR chart (CEO Forum’s
Interactive School Technology and Readiness
(STaR) Chart )
Online access:
http://www.iste.org/inhouse/starchart/index.cfm?Section=ST

School Technology Readiness Measures
(continued)
exas Teacher STaR Chart http://starchart.esc12.net/docs/TxTSC.pdf
exas Campus STaR Chart http://starchart.esc12.net/docs/TxCSC.pdf
ichigan 2003-04 Freedom to Learn School Readiness Rubric
http://school.discoveryeducation.com/schrockguide/pdf/schooltechrubric.pdf
tate of Washington Technology Essential Conditions Rubric
http://www.k12.wa.us/edtech/TechEssCondRubric.aspx or
http://www.k12.wa.us/edtech/pubdocs/TECR-WA.doc

School Technology Readiness
Measures (continued)
SEIR*TEC School Technology Needs
Assessment (STNA)
Sample of the online survey:
http://www.keysurvey.com/survey/134127/1516/
Download Paper-Pencil Versions:
http://www.serve.org/evaluation/capacity/EvalFramework/resou

STNA 4.0 - School Technology
Needs Assessment
• What is STNA?
• What data does STNA collect?
• What does this data tell us?
• How can findings be used?
• What is the STNA process?
• What data is reported?
• How is STNA data interpreted?

What is STNA?
• STNA is a valid and reliable instrument allowing effective assessment
of educational technology needs to better design and evaluate projects
and initiatives
• STNA provides a free, user-friendly online tool that allows for planning
and formative evaluation of technology projects in educational settings
• Helps planners collect and analyze needs data related to
implementation of technological innovation aimed at examining
technology use in teaching and learning
• Guides school- and district-level decisions about professional
development for educators

What data does STNA collect?
• STNA reports at the school level
• Documents respondents’ perceptions and attitudes
• Broad areas of school technology
– Supportive Environment for Technology Use
• Vision, Planning and Budget, Communication, Infrastructure and Staff
Support
– Professional Development
• Professional Development Needs, Professional Development Quality
– Teaching and Learning
• Teacher Technology Use, Student Technology Use
– Impact of Technology
• Teacher Impact, Student Impact

What does this data tell us?
• STNA reports descriptive data
– Item and response set
– Frequencies
– Percentages
• Scales
– 1 (Strongly Agree) to 5 (Strongly Disagree) and 6 (Do Not Know)
– 1 (Daily) to 5 (Never) and 6 (Do Not Know)
• Look at the profile:
– Generally positive or negative?
– Number of highs and lows?
– Height of highs and lows?
– Very different than other items?
• You might get more questions instead of answers

How is STNA data interpreted?
• Each construct examined is theoretically beneficial to successful
implementation of technology in teaching and learning settings
– Strongly agreeing with an item or indications of daily use is inherently
“positive”
– “Do not know” is neither positive nor negative
• All respondents SA or A: “Needs are being met”
• Mostly SA or A: “Not as emphatically positive, perhaps there is room for
improvement”
• Mostly neutral, D, or SD: “Area for concern”
• Large number of DNK: “Lack of awareness in this item”
• Mixed responses: “Lack of strong feelings about this item… why?”
• Split between D/SD and A/SA: “Much disagreement between staff, and
an area of concern”

How can findings be used?
• Plan your technology program implementation
– Incorporate into your school improvement plan or technology plan
– Define your priorities
– Plan professional development
– Allocate resources – funding, staffing, infrastructure
• Clarify your technology program implementation steps
– Provide rationale for your goals and objectives
– Connect to your strategies
– Make your case with needs data
• Repeated uses track changes in the school STNA “profile” over time
• Ultimately, uses are driven by the questions you are looking to answer

What is the STNA process?
• Currently free, thanks to Friday Institute support
• Convene your team, decide if you need STNA data and how it will be
used
• Communicate with staff for buy-in
• Email Danny at daniel.s.stanhope@gmail.com with:
– Names of each school participating
– Opening and closing dates
– An accurate count of expected respondents
• Have all staff working with students complete STNA – goal is 100% response
rate
• Coordinate STNA completion and track who has completed the survey
• Reports are distributed shortly after STNA closes
• Reconvene team to make decisions based on STNA results

Technology Integration Measures
ISTE Classroom Observation Tool (ICOT®)
http://icot.craftyspace.com/
- Register (create an account and confirm it).
- Install tool (Adobe AIR Runtime needed for installation).
- Record observations (online or pencil-and-paper)
- Export and analyze data
- Help (pdf file)
Online tutorial:
http://iste.acrobat.com/p92443979/

Technology Integration Measures
(continued)
SEIR*TEC Technology Integration Progress
Gauge (2000)
Download PDF version:
http://www.serve.org/Evaluation/Capacity/EvalFramework/reso

Technology Integration Measures
(continued)
North Central Regional Technology in Education
Consortium Scoring Guide for Student Products
– create scoring guides to evaluation a variety of student
products
– export scoring guides to pdf
Description and directions:
http://www.ncrtec.org/tl/sgsp/teachers.htm
Tool page: http://goal.learningpt.org/spsg/GetProd.asp

Technology Integration Measures
(continued)

Why use the LoFTI Instrument?
• Determine how technology is being used
school-wide.
• Record instances of particular uses of
technology, not “how well” it is used.
• This is not a teacher evaluation tool.

Who Can Use the LoFTI Instrument?
• The observer can be any member of the
school staff.
• Observers should be trained before
conducting observations using the LoFTI
instrument.

Using the Protocol
• The observer’s presence will undoubtedly
influence what they observe.
• Be mindful of the observer’s placement and
interactions with students and teachers.
• Observers should record only what is
observed during the actual visit.

Using the Protocol
• Visits are not scheduled.
• Keep a separate record of visits.
• Avoid typical non-instructional times –
beginnings, endings, or transitions.
• Visit a given room at different times and days
of the week.

Using the Protocol
• The observer’s presence will undoubtedly
influence what they observe.
• Be mindful of the observer’s placement and
interactions with students and teachers.
• Observers should record only what is
observed during the actual visit.

Practice Using the LoFTI
Online LoFTI
Sample Lesson - Artic Ice

Additional Data Sources
• Teacher Lesson Plans
• Teacher Reflection Logs
• Technology Use Logs
• Rubrics

Using the Data to Effectively Plan
Professional Development
As a whole group, identify professional
development opportunities that seem to be a
good match for the needs identified using the
data from the STNA results.

Effective Professional Development
• Fosters a deepening of subject-matter
knowledge, understanding of learning, and
appreciation of students’ needs
• Centers around the critical activities of
teaching and learning
• Engages educators in professional learning
communities
• Is sustained, intensive, and woven into the
everyday fabric of the teaching profession

Professional Development
Questionnaire (PDQ)

Data Source Instruments
(Summary)

Ning
Join the discussion online
Technology Evaluation in K-12 schools
http://nc1to1.ning.com/group/technologyevaluationink12schools