1. Instructional Design and
Content Development
Workshop

2. Topics
Define a program of evaluation
Evaluation in learning
Outline major reasons to conduct a program
evaluation
Why conduct an evaluation?
General evaluation models.
Who is Kirkpatrick?
Kirkpatrick’s four level learning evaluation
model
Write questions for each level to assess and

3. What is an evaluation?
Evaluation is a systematic method for collecting, analyzing, and
using information to answer basic questions about a program.
Evaluation is valuable for strengthen the quality of the
program and improve outcomes for the learners.
Evaluation answers basic questions about a program’s
effectiveness, and evaluation data can be used to improve
program services.

4. Evaluation In Learning
- Evaluation means assessing the effectiveness and
possible improvement of a course.
- Evaluation is part of any instructional design model.
- Evaluation provides review checkpoints for each
phase of ADDIE that allow the instructor to evaluate
- Without completing this produced.
the work that has been portion
of ADDIE model, the e-course
is incomplete because the
course cannot be redesigned
and improved without

5. MAJOR REASONS TO CONDUCT AN EVALUATION
Evaluation finds out “what works” and “what does not work.”
Evaluation showcases the effectiveness of a program to the community.
Evaluation improves staff’s practice with participants.
Evaluation increases transfer of learning to behavior/performance in
order to maximize program results.
Evaluation increases a program’s capacity to conduct a critical self
assessment and plan for the future.
Evaluation builds knowledge for the out-of-campus time field.

6. Why conduct an evaluation?
The ADDIE model stresses the concept that
good training programs require
planning, review, and revision.
The evaluation phase focuses on gabs to mend
them.
Increase transfer of learning in order to maximize
learning / e-content results.
The evaluation phase measures the course's

7. Benefits of Evaluation
Effectiveness: It shows the progress made toward program goals and objectives.
Best Practices: It provides the ability to determine what program approaches are most
effective.
Improvement: It provides ongoing assessment of program design and implementation
to identify areas of improvement.
Impact: It demonstrates economic or human impact.
Accountability: It provides the base for interpreting an organization or programs’ worth
to its stakeholders.
Promotion & Advocacy: It informs policymakers about programs successes.
Appraisal & Coordination: It gives managers the performance information to make
better operational decisions.

8. General Evaluation Models
There are a lot of models clarifying the evaluation process and determining the
information needs of the intended audiences of the evaluation, these models
such as:
Tyler’s early conception of assessing attainment of program objectives.
Decision-Making Evaluation Approaches.
Naturalistic Evaluation Approaches
Kirkpatrick’s Four Levels for evaluating program effectiveness; which suggests
the most appropriate evaluation methodology to be used.

9. Who is Kirkpatrick?
Donald Kirkpartick is Professor of the
University of Wisconsin in the USA and a
past president of the American Society for
Training and Development.
He has developed a very popular evaluation
model that has been used by the
training/learning community.
He focused on measuring four kinds of

10. Kirkpatrick’s four levels of
evaluating learning
Kirkpatrick’s model includes four levels or steps of
outcome evaluation:

11. Level one - Evaluate Learner Reaction
Level One: Evaluate Learner Reaction
How well did the learners like the learning process?
The goal is find out the reaction of the learners to the
instructor, course and learning environment.
The purpose is not to measure what the learners has learned, but
whether the delivery method was effective and appreciated.
This type of evaluation can be incorporated at the end of the
instruction and can be delivered online.

12. Level one - Evaluate Learner Reaction
General questions can include the following:
• Did the instructor attend the sessions on time?
• Did the instructor respond to learners’ comments and questions?
• Did the instructor deliver the information clearly and smoothly?
• Were there distraction?
• Did the learners feel comfortable in the surroundings?
E-content questions can include the following:
• Level of appeal of the instruction.
• The ease of navigation and use of tools.
• The ability of the course to motivate and retain interest.
• The amount of interactive exercises.
• The relevance of the objectives.
• Quality and relevance of multimedia.
• The ease of navigation and use of tools.

13. Level two - Evaluate Learning
Level two: Evaluate Learning
What did learners learn? (the extent to which
they gain knowledge, skills and attitude)
The goal is measuring learning results.
In other words: did the learners learn what
intended to be taught?

14. Level two - Evaluate Learning
Measurement methods of level two include:
Formal and informal testing.
Self assessments at the beginning (pre-test) and end (post-test) for
learners.
http://www.reap.ac.uk/reap/public/papers//DN_SHE_Final.pdf
Interviews, observation and feedback.
Product Assessment. Creation of a project as an authentic assessment to
showcase the knowledge and talents.
This form of assessment evaluates whether the learner can apply the learned
skills or concepts in a concrete fashion.

15. Level three - Evaluate Performance
Level three: Evaluate Performance
What changes in performance resulted from the learning process?
To what degree participants apply what they learned during training when they are back on the job.
The goal is measuring “what happens when learners leave the classroom and return
to their daily lives/job.
Did the learners put their learning into effect when back on the job/live?
Would the learners be able to transfer their learning to another person?
How much transfer of learning occurs?
It is the most obvious sign to evaluate the training program’s effectiveness.

16. Level three - Evaluate Performance
Measurement methods of level two include:
Allow time for a change in behavior to take place
Testing
Observation, interview and survey.
The learner interactions on the job.
Evaluate both before and after the program if that is
practical

17. Level four - Evaluate Results
Level four: Evaluate Results
What are the tangible results of the learning process in terms improved
quality, increased production, efficiency, etc.?
To what degree targeted outcomes occur as a result of learning event(s) and subsequent
reinforcement
The goal is to find out if the training program led to final
results, especially business or environment results
Level four outcomes include the major results that contribute to the
well functioning of an organization.
Level four outcomes Improve quality of work, increase productivity and
profits.
Across an entire organization it becomes more challenging and affect
organizational and business performance, which cloud the true cause
of good or poor results.

18. Level four - Evaluate Results
Measurement methods of level four include:
Methods of Measuring Learning Outcomes Grid
How colleges and universities can measure and report on the knowledge and
abilities their students have acquired during their college years is an issue of
growing interest.
http://web.mit.edu/tll/assessment-evaluation/methods-of-measuring-learning-outcomes-
grid.doc
http://www.elcamino.edu/academics/slo/docs/SLOFocusOnResults.doc
Course-embedded assessment/Assurance of Learning Standards.
http://www2.cortland.edu/dotAsset/f2e9f1ee-0100-40ab-b78d-48ead9c047d3.pdf

19. To Sum Up

20. Checklist for effective questions
Do Do Not
Give clear instructions Use complex phrases
Keep questions structure simple Frame questions in the negative
Ask one question at a time Use abbreviations, contractions or symbols
Maintain a parallel structure for all questions Mix different words for the same concept
Define terms before asking the question Use “loaded” words or phrases
Be explicit about the period of time being referenced by the Combine multiple response dimensions in the same question
question
Provide a list of acceptable responses to closed questions Give the impression that you are expecting a certain response
Ensure that response categories are both exhaustive and mutually Bounce around between topics or time periods
exclusive
Label response categories with words rather than numbers Insert unnecessary graphics or mix many font styles and sizes
Ask for number of occurrences, rather than providing response Forget to provide instructions for returning the completed survey
Save personal and demographic questions for the end of the
survey

21. Activity
Based on your study of
Kirkpatrick's model of learning
/ training programs
evaluation, apply this model to
evaluate your course using the
4 levels of Kirkpatrick's model.
o Write questions that explores
the items within each level

22. ONLINE DETAILED COURSE MANUAL
http://www.kirkpatrickpartners.com/Portals/0/Training-
Events/Program%20Prework/Kirkpatrick%20Four%20Levels(t
m)%20Evaluation%20Online%20Certificate%20Manual.pdf