The document provides guidance on summarizing formative evaluation data from instructional materials in 3 steps: 1) Describe learners and performance on entry tests, 2) Compile comments on instruction, 3) Obtain individual and objective performance scores. There are several methods to summarize including analyzing individual item performance, attitude questionnaires, time data, and comments. Summaries should identify weaknesses in materials or instruction based on learner performance and suggestions to guide revisions. The goal is to determine if tests or rubrics need changes by examining learner performance, or if instruction requires changes.

1.  
LaSheena Nation

2.  Describe various methods for summarizing data
obtained from formative evaluation
 Summarize date obtained from evaluation studies
 Given summarized formative evaluation data, identify
weaknesses in instructional materials and instructor-0led
instruction
 Given formative evaluation data for a set of instructional
materials, identify problems in the materials, and suggest
revisions for the materials

3.  During the revision process, designers must keep a
system perspective on their work and remain open to
possibilities that revisions may be warranted in any stage
of the design process.
 Two basic types of revision should be considered with
your materials:
o Changes made to the content or substance of the materials to
make them more accurate or effective as a learning tool
o Changes related to the procedures employed in using your
materials

4.  There are many different ways in which data collected in
a formative evaluation may be summarized to point
areas of learner difficulties and possible revisions. The
methods that will be described are merely suggestions,
you may find other techniques that may work better for
your data.

5.  The designer has five types of basic information
available
1. Learner characteristics and entry level behavior
2. Direct response to instruction
3. Learning time
4. Posttest performance
5. Response to an attitude questionnaire

6.  There is a three step process in collecting data
1. Step 1: Describe the learners and indicate their performance
based off of entry level measurements.
2. Step 2: Compile all comments and suggestions about the
instructor that resulted from going through with each individual
learner.
3. Step 3: Obtain the individuals item performance and them
combine it with item scores for each objective and for a total
score.

7.  Revising the instruction with information collected from
what's available and through the 3 step process.
 There are three main sources of information/evaluation
for changes to be made
1. Learner suggestions
2. Performance of the learner
3. Your own insights

8.  The goal is to determine if the rubric or test items are
fault by basing them on the learners performance. If a
flaw is found to exists then changes must be made to
make them clear and consistent with the overall
objective and the intent of the instruction. If there is no
flaw found, but the learners sis not perform well, then
consideration of changing the instructor will be reviewed.

9.  The small group formative evaluation provides the
designer with a somewhat different data summary
situation. The data from eight to twenty learners are of
greater collective interest than individual interest;
therefore showing the overall problems and reactions
this representative group of learners have.

10.  The available data usually includes the following:
1. Item performance on the pretest
2. Posttest
3. Responses to an attitude questionnaire
4. Learning and test time
5. Comments made directly in materials

11.  The fundamental unit of analysis for all the assessments
is the individual assessment item. Performance on each
item must be scored either correct or incorrect. If an item
has multiple parts, then each part should be scored and
reported separately so that the information is not lost.
The individual item information is required for three
reasons.

12.  What are the three reason individual item information is
required?
1. To determine weather there are particular problems with the
item or weather it is effectively measuring the performance
described in the objective.
2. To identify the nature of difficulties learners are having with the
instruction.
3. To indicate learner performance on an objective, and eventually
on an entire test.

13.  Graphing Learners Performance can be done In two
ways
1. Display data through carious graphing techniques
2. Summarizing formative evaluations data involves the
instructional an analysis chart

14.  Ways data can be used:
o For attitude questionnaire
o For obtaining comments /reacting to materials
o To discuss alternative approaches during wither small-group or
field-trials evaluations

15.  The suggested sequence to use data are as follows:
o Instructional analysis and entry level skills
o Objectives, pretest, and posttest
o Instructional strategy
o Learning time
o Instructional procedures

16.  As you begin the revision process it is suggested that
you summarize you data based upon the following:
o In the left column, list the components of the instructional
strategy.
o Add additional columns for summarizing
• Problems identifies
• Proposed changes
• Evidence gathered for identifying problem

17.  Working with selected materials avails little opportunity
to revise materials directly, especially if they are
copyrighted or commercially produced.
o Consider the following adaptions for the future
1. Omit potions of the instruction
2. Include other available materials
3. Develop supplementary instruction

20. Lasheena Nation
lasheenanation32hhs@yahoo.com