This document provides an overview of formative evaluation and its importance in improving project design. Formative evaluation involves testing a project before or during implementation to ensure needs are being met, feedback is collected, and the design is finalized. It describes different evaluation methods like one-on-one interviews and small group testing that provide early feedback to improve the design. The document emphasizes evaluating in a real-world context and being prepared to identify and address problems to strengthen instruction.

1. Designing and
Conducting Formative
Evaluations
c. Loder

2. Objective:Topresentanoverviewondesigning
andconductingformativeevaluations.
Outcomes:Forviewerstohaveanunderstanding
ofthestructureofthisstageanditsimportanceof
conductingformativeevaluations.

3. Formative Evaluation
This is a form of evaluation that is used to improve
program design by testing before or during a projects
implementation. This evaluation ensures that the
desired needs are being met, relevant feedback is being
collected, and helps with the finalization of the chosen
design.

4. Proactive Clarificative Interactive Monitoring
When
Pre-project
Project
development
Project
implementation
Project
implementation
Why
To understand or
clarify the need
for the project
To make clear
the theory of
change that the
project is based
on
To improve the
project’s design
(continual
improvement) as
it is rolled out
To ensure that
the project
activities are
being delivered
efficiently and
effectively
Examples
Literature
Review
Stakeholder
Analysis
Problem /
Solution tree
analysis
Logframe Matrix
Program Logic
Semi-structured
Interview
Focus Group /
ORID
Project Diary
Budget tracking
Time tracking
Questionnaire
Dartboard
Observation
Categories of formative evaluationhttp://evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=24&Itemid=125

5. Role of Subject-Matter, Learning, and Learner
Specialists in Formative Evaluation
In order to conduct a Design Review, you must have:
Subject-Matter or the information or ideas that are
discussed or dealt with in a book, movie, etc.: what
something is about

6. Role of Subject-Matter, Learning, and Learner
Specialists in Formative Evaluation
The purpose of the review or evaluation is to ensure
students are
Learning: the activity or process of gaining knowledge or
skill by studying, practicing, being taught, or
experiencing something: the activity of someone who
learns
http://www.merriam-webster.com/

7. Role of Subject-Matter, Learning, and Learner
Specialists in Formative Evaluation
The content and activities that take place during the
evaluation are reviewed by a specialist, a person who
has special knowledge and skill relating to a particular
job, area of study that ensures that the design supports
its objectives.

8. One on One Learning
One-to-one evaluation can be used to evaluate the
intrinsic qualities of the design (clarity, usability,
sequencing, completeness) and the learning effects
(performance, practice, time, transfer).

9. One on One Learning
A one-to-one evaluation can give you feedback from a
learner that will help you improve your design. In on-to-
one evaluation, you sit with learner as learner reviews
instruction, observing how the learner uses it, noting
learner's comments, probing about comments, and
questioning the learner during/after instruction.

10. One on One Learning
You should do one-to-one evaluation as early as
possible, usually with roughs. Often you will do a
sequence of 2-4 evaluations with the first focusing on
obvious errors, later on intrinsic quality and later
performance. One-to-one evaluations should be done
before small group and field tests and
before/during/after expert review.

11. One on One Learning
Pros
• you get the learners' point of view
• the interactive evaluation gives you process data
• it's easy, cheap, and quick
• it's productive (Wagner, 1983; Robeck 1965; Lowe,
Thurston, Brown 1983)
• it can be used to rapidly test multiple versions of the
design

12. One on One Learning
Cons
• the small number of users prevents you fro
seeing patterns
• it has low environmental validity because it's out of
context, involves an strange 1-1 interaction; and
usually involves a rough version of the design.
•
https://loft.io/guide/eeole/one-to-one-evaluation/

13. Small Group Evaluationhttps://loft.io/guide/eeole/small-group/
A small group evaluation can be used to quickly get you
information about the learning impact and likability of your
design. The whole point of your learning environment is to
promote learning -- small group analysis is often the method in
the design process that can provide the earliest evidence that
the design is having the desired effect.

14. Small Group Evaluationhttps://loft.io/guide/eeole/small-group/
In small group evaluation, you test an early version of the
learning environment with a representative sample of learners.
Usually, the instruction is taught by a teacher while the design
team records observations and gathers data. Small group
evaluations typically include tests and questionnaires.

15. Small Group Evaluationhttps://loft.io/guide/eeole/small-group/
Advantages
• they can be cheap
• you can get lots of data quickly
• they can require less interaction than one-to-one
evaluation because the tester may not necessarily need to
constantly question the participants

16. Small Group Evaluationhttps://loft.io/guide/eeole/small-group/
Disadvantages
• may give less process/interaction data than one-to-one
evaluation
• may not catch a problem as quickly as in one-to-one
evaluation in which you are constantly, directly monitoring
a single learner
• may not allow you to ask why something is not working
• can be expensive if you must pay learners
• may provide more data than you have time to analyze.

17. According to
Educscapes
(www.eduscapes.com),

18. During the field trial, the
materials are tested in a "real-
world situation".

19. The purpose of the field trial is to ensure
that all of the problems from earlier
evaluations have been corrected and to
see whether the instruction works in the
intended context.

20. The instructional materials should be
tested in a "real world" situation with a
typical group of learners. The
procedures followed should be those
outlined in the instructional materials..

21. Finally, any final revisions can be
made.
http://eduscapes.com/instruction/13.htm

22. Formative Evaluation of Selected
MaterialsDoes not always follow the traditional
evaluation process.
Usually results in the creation of supplemental
materials such as study guides, workbooks, and
syllabi.

23. Formative Evaluation of Selected
MaterialsEvaluating selected materials includes Expert
Judgment Studies. These studies help to
determine if the current course of study is
meeting requirements and if other courses or
additional supplemental also meet
requirements.

24. Formative Evaluation of Instructor-Led
Instruction
These evaluations are focused on the
effectiveness of current instruction and how it
can be improved.

25. Formative Evaluation of Instructor-Led
Instruction
Field trial instructors pay attention to these
following learner knowledge: entry skills, prior
topic knowledge, posttest scores, and learner
attitude.

26. Formative Evaluation of Instructor-Led
Instruction
These field trials are helpful as they provide
instructors with learner feedback and provide a
practice medium for instructional procedures.

27. Concerns Influencing Formative
Evaluation

28. Context Concerns
• Instructors should ensure equipment is running
and stored properly for next instructor
• Students should be encouraged to work within a
quiet and education focused environment

29. Concerns about Learners
Before instruction, administer entry level testing in
order to gauge learner knowledge. Results should be
focused on those who have mastered the skills

30. Concerns about Learners
Those who have mastered the skills should continue
as learners but the focus should be on obtaining a
100 percent performance ranking.

31. Concerns about Learners
These learners can be selected as “subject matter
sophisticates “who can help others obtain 100
percent performance.

32. Concerns about Learners
Those who do not pass the entry level testing should
be included in instruction in order to help identify
what skills are required for learning and what skills
need to be included as entry level skills.

33. Concerns about Formative
Evaluation Outcomes
Be prepared to find problems and concerns with
your instructional design.

34. Concerns about Formative
Evaluation Outcomes
Understand that the positive feedback is limited and
can only be related to a limited group of
learners.

35. Concerns about Formative
Evaluation Outcomes
Be objective about the instruction design in order to
spot areas that need improvement

36. Problem Solving during Instructional
DesignWhen a problem arrives dealing with your
Instructional Design, use learner information to
determine the answer. Information derived
from formative evaluation and other methods
provides data that can be used to improve
instruction.