a brief paper written to explain how we have created measurable and effective outcomes at KU.

1. Developing measurable and effective outcomes for online courses
Amy J. Hilbelink, PhD
Kaplan University
Introduction
When creating a course, whether it is to be conducted online or in a face-to-face environment, one question that
often arises is, “how do we know that the outcomes for the course are being effectively delivered and that we are
actually measuring student learning?” In 2008, Kaplan University created an innovative approach to effectively and
consistently measure course outcomes. The process was introduced in the summer of 2008, and has been
systematically incorporated into Kaplan University courses across all programs.
An initial step to ensuring measurable course outcomes was to first determine that each course had measurable and
appropriate outcomes to begin with; therefore, each course at Kaplan University had to be systematically reviewed
to ensure that course outcomes were aligned with program outcomes. An initial review of programs and their
courses determined that often course assessments were not linking to course outcomes and that oftentimes course
outcomes were not aligning with program outcomes, resulting in unnecessary and inappropriate assessments. In
order to ensure that the new initiatives were successful, we needed to devise a better way to develop measurable and
effective course outcomes.
Outcomes
What does a measurable outcome look like and why is it important? Upon an initial review of the School of Nursing
course outcomes we determined that the majority of courses had more than eight outcomes and many of them were
being poorly assessed. In addition, it seemed that courses were commonly designed with no focus on the original
outcomes, which resulted in unnecessary or unfocused assignments. To the student, this often appeared as busywork.
After this initial assessment, we devised a number of traits for an effective and measurable outcome. An appropriate
outcome should be measurable, it should generally contain only one verb, and an appropriate outcome should be at a
Bloom’s Taxonomy Level (1956) suitable for the level of the course and the assessment.
In order for an outcome to be measurable, it must be written in such a way that there is little doubt as to what is
being measured. Oftentimes, an outcome will have two or more verbs or requirements. Refer to Table 1 for an
example. In this outcome, it would be exceedingly difficult to accurately measure the student on each requirement
of the stated outcome. It is then left to interpretation as to how the outcome should be assessed.
Table 1. Example of poorly written outcome
Original Outcome
Incorporate nutritional implications, cultural observations, discharge planning, and
learning needs into care delivery for children.
Revised Outcome
Develop plans of care that address the needs of children and families with common
health concerns and complex health problems.
Amy J. Hilbelink
Kaplan Higher Education Page 1 10/26/2009

2. Within a Master’s level courses we discovered that assessments were often written at a different Bloom’s level than
the original outcome and were therefore considered misaligned. Refer to Table 2 for an example.
Table 2. Misalignment of outcomes and assessments
Outcome
Identify individual learning needs for the selected advanced nursing role
(Bloom’s Taxonomy level 1)
Assessment
Student is to prepare step One of their Personal Career Development Plan using
provided template
(Bloom’s Taxonomy level 3)
In order to ensure that all Kaplan University courses had measurable outcomes, a systematic evaluation of all
program outcomes, course descriptions and outcomes was undertaken. Within the School of Nursing, we had
faculty work in pairs on all courses to evaluate outcomes and to ensure that there were no more than eight outcomes
per course. Some faculty had taught the course, while others had indirect content knowledge. We found that one can
create excellent outcomes and assessments without specific content knowledge as long as they adhere to the traits of
a good outcome. Once outcomes were created, they were then presented to the Kaplan University Assessment
Committee for approval. Once approval was gained, course design could begin. In addition, a common rubric style
was established to measure all course outcomes.
A backward design method was implemented based on the review of outcomes. This required all subject matter
experts (SMEs) to create a course outcome table (Table 3) prior to beginning any actual course development.
Table 3. Example of the course outcome table
Course Course Outcome Bloom’s Assessment Program Outcome
Outcome Taxonomy
Number Level
1 Demonstrate the appropriate use 2 Creation of a MSN.NE-1
of an instructional technology in narrated
an educational setting PowerPoint
2 Evaluate technologies for 6 Final written MSN.NE-2
meeting educational objectives. project
3 Report on uses of technology in 5 Creation of blog MSN.NE-1
various educational settings.
4 Evaluate the legal implications 6 Discussion MSN-7
of instructional technology. question
Amy J. Hilbelink
Kaplan Higher Education Page 2 10/26/2009

3. Issues
Some issues confronted included a short time frame in which to revise all Kaplan courses with appropriate and
measurable outcomes. The process continues through 2009, as new courses are created and as courses are revised.
Subject matter experts and program directors need to be trained on how to create an effective outcome, and the
reasons behind it. Many faculty originally felt that the planned evaluation of outcomes was difficult and too time
consuming and therefore needed to have a better understanding of the need for better outcomes and assessments.
Results
As a result of this exercise with course outcomes, we have changed our course development process to a backward
design process. There are a number of advantages to utilizing the method of backward design. It requires SMEs to
give serious thought to the flow of the course they are designing prior to the creation of any content because each
outcome can only be assessed once. It assists the SME in creating assignments that can be scaffolded if necessary or
applicable throughout the scope of the course. It also helps eliminate extraneous assignments, resulting in less busy
work for the student.
References
Bloom B.S. (1956). Taxonomy of Educational Objectives, Handbook I; The Cognitive Domain. New York: David
McKay Co Inc.
Amy J. Hilbelink, PhD; Assistant Dean of Curriculum, Kaplan University, 605 Crystal Grove Blvd., Lutz, Fl.
33548 ahilbelink@kaplan.edu
Amy J. Hilbelink
Kaplan Higher Education Page 3 10/26/2009