Faculty Focus Special Report: Assessing Online Learning: Strategies, Challenges and Opportunities

Assessing Online Learning:
Strategies, Challenges and
Opportunities

Featuring content from

A MAGNA PUBLICATION

Effective Group Work Strategies for the College Classroom. • www.FacultyFocus.com

Assessing Online Learning:
Strategies, Challenges and Opportunities
As online education moves from the fringes to the mainstream, one question still persists:
“How do I know what my online students have learned?” There are no simple answers,
just as there aren’t in face-to-face courses, but with a little creativity and flexibility, you
soon discover that the online learning environment opens up a host of new student as-
sessment possibilities. And, just as with traditional courses, the trick is finding the right
combination that works best for your particular course.

This special report features 12 articles from Online Classroom that will cause you to
examine your current methods of online assessment, and perhaps add something new to
your assessment toolbox. It even talks about some of the common assessment mistakes
you’ll want to avoid.

Take a look at some of the articles you will find in Assessing Online Learning: Strategies,
Challenges and Opportunities:
• Authentic Experiences, Assessment Develop Online Students’ Marketable Skills
• Four Typical Online Learning Assessment Mistakes
• Assessing Whether Online Learners Can DO: Aligning Learning Objectives with
Real-world Applications
• Strategies for Creating Better Multiple-Choice Tests
• Assessing Student Learning Online: It’s More Than Multiple Choice
• Using Self-Check Exercises to Assess Online Learning
• Measuring the Effectiveness of an Online Learning Community
• Ongoing Student Evaluation Essential to Course Improvement

Online courses enable a strong student-centered approach to learning and, as a result,
assessment. We hope this report helps you design and develop online assessment strate-
gies that take full advantage of the many formal and informal assessment tools now at
your fingertips.

Rob Kelly
Editor
Online Classroom

Assessing Online Learning: Strategies, Challenges and Opportunities • www.FacultyFocus.com
2

Table of Contents
Four Typical Online Learning Assessment Mistakes ..................................................................................................4

Authentic Experiences, Assessment Develop Online Students’ Marketable Skills ........................................................6

Assessing Whether Online Learners Can DO: Aligning Learning Objectives with Real-world Applications ................8

Strategies for Creating Better Multiple-Choice Tests ................................................................................................10

Assessing Student Learning Online: It’s More Than Multiple Choice ........................................................................13

To Plan Good Online Instruction, Teach to the Test ..................................................................................................14

Using Self-Check Exercises to Assess Online Learning ..............................................................................................16

Assessment for the Millennial Generation ................................................................................................................17

Self-Assessment in Online Writing Course Focuses Students on the Learning Process ..............................................19

Using Online Discussion Forums for Minute Papers ..................................................................................................20

3

• Determine the best tools to use

Four Typical Online Learning for defining, modeling, and
rendering of 2-D and 3-D objects.

Assessment Mistakes Mercer graded students based on
weekly homework assignments (10
percent of the grade), two projects (20
percent of the grade), and a final test
By Patti Shank, PhD, CPT (70 percent of the grade). More than a
third of the students got a C or lower
on the final and as a result, because
the final was such a large percentage
he goal of learning assessments Here’s a scenario for us to consider.

T should be to measure whether
actual learning outcomes match
desired learning outcomes. Here’s an
Lana Mercer, a new instructor, has
just finished teaching her online
of the final grade, received low grades
for the course. Lana didn’t understand
why students were upset, because
final grade scores reflected a bell
analogy. Your freezer stops keeping
People who build instruction curve, so the range of grades was as
foods frozen, so you call the
she expected. See any assessment
appliance repair folks. They show up
on schedule and charge you exactly make some typical but problems? (Yep, you should.)
what they estimated on the phone. Is
that enough information for you to
unfortunate mistakes when Four typical mistakes
People who build instruction make
know if the desired outcome (frozen designing learning some typical but unfortunate mistakes
food) has been achieved? No, of
when designing learning assessments,
course not. assessments, and these
and these mistakes compromise both
We use freezers to achieve specific
outcomes. We build instruction to mistakes compromise both their competence as designers of in-
struction and the quality of the in-
achieve specific outcomes as well.
Well-written instructional objectives
their competence as designers struction they build. These mistakes
include:
describe the desired outcomes of in- of instruction and the quality of 1.Expecting a bell curve
struction and are critical to designing
2.The wrong type of assessment
good courses and assessments. the instruction they build. 3.Not valid (enough) assessments
A freezer that works means the
4.Poorly written multiple-choice
food stays frozen as expected.
tests
Instruction that works means people course, Introduction to Computer
learn as expected. Adequate learning Graphics. Here are the three most
assessments tell us whether the in- critical terminal objectives for this
Expecting a bell curve
Benjamin Bloom (1968), a distin-
struction we built works and provides course (these are reasonably well-
guished educational psychologist,
us with data to adjust our efforts. written, unlike most of the objectives
proposed that a bell curve model,
We measure whether instruction I see, which makes it far easier to
with most students performing in the
“works” by seeing if the instruction determine what assessments are
middle and a small percentage per-
we build actually helps people needed):
forming very well and very poorly
achieve the learning objectives. I’d • Analyze common uses for these
(e.g., a normal or bell curve) is the
even argue that we cannot be consid- computer graphics methods: 2-D
wrong model of expected outcomes
ered competent builders of instruction representation and manipulation,
from most instruction. The bell curve
if we can’t show that what we built 3-D representation and manipula-
model is what might be expected
helps learners learn. Some might say tion, animation, and image pro-
without instruction. Instruction
that’s a big “duh,” but I’m guessing a cessing and manipulation.
should be specifically designed to
fair number of people who build in- • Describe methods for defining,
provide the instruction, practice,
struction haven’t really thought about modeling, and rendering of 2-D
it. and 3-D objects.
PAGE 5

4

FROM PAGE 4 how to troubleshoot the cardiac if another assessment is written at too
monitor (procedural objective) high a reading level. What is it
feedback, and remediation needed to
involves far deeper skills. So asking measuring? For one thing, reading
bring about achievement of the
biomedical technicians to name parts skills. Both tests are likely less valid
desired outcomes. His “mastery”
or even list the troubleshooting steps than is necessary. The best way to
model assumes that most students
on a final test is an inadequate assess- establish validity is to carefully match
will be high achievers and that the in-
ment of their troubleshooting skills. objectives and assessments, as
struction needs to be fixed if this does
The bottom line is whether they can, explained in the last mistake.
not occur.
in fact, troubleshoot, and that requires Lack of validity impacts course
Mercer should not have expected
a performance assessment. quality and fairness. And if the results
her students’ final grades to fall on a
When it comes to designing of assessments impacts passing the
bell curve. A mastery model assumes
adequate assessments, it’s inadequate course (as they usually do), invalid
that most students will achieve the
to determine only whether learners assessments are not only unfair but
desired outcomes, and therefore, most
know about if you need to determine potentially illegal.
will achieve higher grades.
whether they actually can perform in Objectives and assessments in
the real world. Many higher education Mercer’s class didn’t match up.
The wrong type of assessment instructors don’t adequately infuse Because students in Mercer’s class
There are two primary learning as-
their courses with real-world implica- needed a passing grade in order to
sessment formats: performance as-
tions and skills, and I believe this is a take higher-level graphics courses, she
sessments and “test” assessments.
mistake. needs to rethink the validity of her as-
The former involves assessing per-
Mercer’s objectives are a mix of de- sessments, starting with mapping as-
formance in a more realistic way (in
clarative and procedural, but her assessment types to objective types.
situations), and the second involves
sessment scheme is heavily weighted
paper or computer-based forms with
toward achievement of declarative ob- Poorly written multiple-choice
multiple choice, matching, fill-in-the-
jectives (and the tests used to assess tests
blank, and short- and long-answer-
them). That made her grading scheme Many assessments, even if they are
type (i.e., essay) questions. Test
unbalanced and inappropriate. A the right kind, are poorly written. Two
assessments are by their nature a less
more balanced and appropriate of the most common mistakes for
authentic way of assessing learning
grading scheme would have given multiple-choice questions are
but are very practical and are
more weight to projects that show confusing or ambiguous language and
therefore commonly used.
achievement of procedural objectives. implausible distractors (wrong alter-
The optimal assessment type
natives from which the learner selects
depends primarily on whether the
Not valid (enough) the correct answer[s]). A poorly
objective is declarative (facts: name,
assessments written multiple-choice question auto-
list, state, match, describe, explain…)
The gold standard for assessment matically lowers the validity of the as-
or procedural (task: calculate,
quality is validity. A valid assessment sessment.
formulate, build, drive, assemble,
measures what it claims to measure.
determine…). Research shows that
For example, a biomedical equipment Final thoughts
there is a big difference between these
troubleshooting assessment should Inadequate learning assessments
two types—the difference between
measure the skills of the person doing are at best frustrating. At worst, they
knowing about and knowing how
actual or simulated troubleshooting. can damage students and institutions.
(practical application to real-world
It’s easier than you might think to Adequate learning assessments are
tasks).
design assessments that measure one of the hallmarks of competence in
Let’s take, for example, a biomed-
something other than what is building good instruction and
ical technology course. A biomedical
intended. markedly improve the quality of in-
technician needs to know the names
Let’s say the biomedical equipment struction.
of a cardiac monitor’s parts (declara-
troubleshooting assessment primarily The final assessment for the
tive objective) in order to find appli-
asks students to match parts, Introduction to Computer Graphics
cable information in the
functions, and typical problems. Is course suffered from all the mistakes
troubleshooting manual. But knowing
this a valid assessment of trou-
part names only goes so far. Knowing PAGE 6
bleshooting skills? Unlikely. And what

5

FROM PAGE 5 on Measurement in Education Workshop Materials. Denver, CO:
(NCME). (1999). Standards for educa- Learning Peaks.
listed, even though the instructor was tional and psychological testing. Shrock, S., & Coscareli, W. (2000).
well-intentioned. In an online course, Bloom, B. (1968). Learning for Criterion-referenced test development.
where students often require extra mastery. Evaluation Comment, 1(2), Silver Spring, MD: International
feedback and motivation, unintended 1–5. Society for Performance Improvement.
frustrations and unfairness can cause Bond, L. A. (1995, August). Norm-
many problems including complaints, referenced testing and criterion-refer- Patti Shank, PhD, CPT, is a widely
reduced enrollments, and lack of per- enced testing: The difference in recognized instructional designer and
sistence. purpose, content, and interpretation of technologist, writer, and author who
Writing good performance assess- results. Oak Brook, IL: North Central teaches and helps others teach online.
ments and test questions is a skill that Regional Educational Laboratory. She can be reached through her
takes training, time, and feedback. (ERIC Document Reproduction Service website: http://www.
No. ED402327). learningpeaks.com/. @
References/Resources Shank, P. (2005). Developing
American Educational Research learning assessments for classroom,
Association, American Psychological online, and blended learning.
Association (APA), & National Council

Authentic Experiences, Assessment Develop
Online Students’ Marketable Skills
By Rob Kelly

aureen Colenso, an instruc- assesses learning outcomes. “I teach prospective employers. “It gives

M tor at Northeast Wisconsin
Technical College, sums up
her assessment philosophy for her
at a community college, and we fre-
quently have returning adult students
who need to be trained for a new
students something real. It makes a
world of difference in a competitive
job market if the applicant, instead of
online courses as follows: “Whatever career. They need marketable skills, just saying, ‘Yes, I can write a
it is students are going to need to be and we need to know that they really program that will suit your needs,’
doing on the job, then that’s how have learned these skills,” Colenso can say, ‘Oh yes, here’s an example,’”
they need to be assessed for the says. Colenso says. “For example, I
classroom. I start with the assump- The courses Colenso teaches online encourage students to build a custom
tion that the competencies for the lend themselves to authentic assess- database in the database class and
class represent marketable skills, and ment. Colenso teaches documenta- then use the documentation class to
I find that there’s pretty much always tion, training, database theory, provide good documentation for that
a way to figure out how to have database applications, and several system and use the training class to
students do a project that will be rep- entry-level software skills courses. In develop training materials to teach
resentative of the type of work they all these courses, she finds ways to somebody how to use that system,
would do for an employer.” have students produce authentic and then they have a whole package
Colenso didn’t come to teaching products for local businesses. they can show a prospective
from a traditional path and says she Students produce electronic employer.”
has a pragmatic approach in her products that are easily delivered PAGE 7
online courses and the ways she online and that can be shown to

6

FROM PAGE 6 lenges, but all of these software would somehow be watered-down
packages that are used for online versions of what students did in the
Working with real clients adds to delivery have things like virtual class- classroom. It turned out that that was
the authenticity of the learning expe- rooms.” a common assumption. I said, ‘I’ll
rience. “It inspires much better work. Even courses that are more theory- show you. These aren’t going to be
To me that’s one of the real benefits. based include some form of authentic even remotely watered-down.’”
For one thing, it’s hard for a student assessment. For example, in a Because they do not represent
to be really motivated when he or recently created database theory anything that students will be asked
she knows that all that’s going to course, Colenso includes an assign- to do on the job, Colenso does not
happen is a certain amount of points ment that has the students work with use objective, multiple-choice tests in
being awarded and nobody’s ever a local business to analyze its needs her courses, except as a way to
going to think about it again. If and to design an information-level encourage students to complete the
something is real, if a local business database and make a recommenda- readings and review basic facts.
is going to be using [the product] and One of the selling points of the
if they’re going to be sharing the college’s online computer science
results with other members of the program is the reputation the
class, it just spurs them to make a Because they do not represent program has among the local IT
much greater effort,” Colenso says. community. “This is a small
The experience of working online anything that students will be community. Everybody in the IT
adds another element of authenticity community kind of knows everybody
because in many work situations asked to do on the job, Colenso else. If word gets out that some of
workers are called on to collaborate our students can get all the way
online with their coworkers. When
does not use objective,
through and graduate our program
Colenso began teaching online six multiple-choice tests in her and not be able to perform on the
years ago, she found that the differ- job, that would be bad. Part of how
ence distinguishing online from face- courses, except as a way to we sell our program is the good em-
to-face learning is that students have ployment results at the end,”
the additional challenge of negotiat- encourage students to Colenso says.
ing distance communication Business clients get free services
methods. “But since that’s very much complete the readings and from the students, but they also help
a part of what happens in the
review basic facts. by providing mentoring for the
working world today, I think that that students. Before working with
provides a better experience,” business clients, it’s important that
Colenso says. everybody knows the nature of this
The benefits of this experience are tion as to which platform would be relationship, how students will be
so strong that Colenso has begun in- best for that customer to implement evaluated, and the client’s role in that
corporating online collaboration in this database. The students then evaluation. @
her face-to-face courses. “Once I write a proposal, which may or may
taught online, I thought the tradi- not lead to that database being
tional students weren’t really getting created, “but it still provides that in-
their full measure, because they teraction,” Colenso says. “It’s still
should be having these experiences real work. When it comes to lower-
too. level classes, I’m more inclined to put
“Online students might be more together lab practicals that represent
likely to pick a client for their project what businesses need.”
where they’re going to be communi- Course quality also figures into
cating more by e-mail and attach- Colenso’s use of authentic assess-
ments and things like that. I do find ment. “We received a grant to make
that where I want to have students all of our courses available online,
work in collaborative groups, there and I was somewhat horrified when a
are some additional logistical chal- few people implied that the courses

7

To illustrate the match between

Assessing Whether Online objective type and assessment type,
let’s analyze three objectives in a

Learners Can DO: Aligning personal wellness course.
As a result of course activities,
learners will
Learning Objectives with 1. explain how the seven dimen-
sions of wellness impact daily

Real-world Applications living,
2. analyze how deficits in nutrition
impact human performance and
longevity, and
3. plan a personal diet and fitness
By Patti Shank, PhD, CPT program based upon a personal
diet and fitness assessment and
personal goals.
any of the criticisms of verbs such as name, list, state,

M education come from the
fact that much instruction
isn’t designed so learners can DO. For
match, describe, and explain.
Procedural objectives ask learners to
apply what they know in realistic or
Objective #1 is declarative, asking
the learner to recall facts and
example, it’s not good enough to real situations. They ask if learners concepts about the seven dimensions
design math instruction so learners of wellness. Test questions are an ap-
can do the problems at the back of propriate and efficient way to assess
each chapter. They need to be able to if the learner can recall how physical,
calculate discounts, determine the How can you assess if intellectual, emotional, social, occu-
implications of different interest pational, environmental, and spiritual
rates, and so on, in the real world. learners can develop an wellness impact daily living.
And instructors need to assess if
learners can DO, which commonly appropriate diet and fitness Objective #2 is a fairly uncompli-
involves designing real or realistic cated procedural objective that asks
performance assessments. plan without having them learners to be able to reach conclu-
In this article, I’ll first explain how sions based on a limited amount of
the type of assessment should match
develop the plan to see how
information. Although a performance
the level of learning objectives and well it was done? You can’t. A assessment might be optimal, simpler
then describe how checklists and procedural objectives that don’t
rating scales can be built to assess test isn’t adequate. involve much complexity can be
learning objectives that require assessed quite nicely (and efficiently)
demonstration of more complex using well-written case or scenario-
skills. A foundational skill for devel- based multiple-choice test questions.
oping good performance assessments can DO. They use task-oriented verbs
is writing good learning objectives. such as calculate, formulate, build, Objective #3 is a more complicated
drive, assemble, and determine. procedural objective that requires
Two levels of objectives, two Most, if not all, terminal learning learners to deal with myriad factors
types of assessments objectives in our courses should be in a realistic way. How can you
There are two levels of objectives, procedural because we need learners assess if learners can develop an ap-
and each level is ideally assessed to be able to DO real tasks in the real propriate diet and fitness plan
using a different type of assessment. world. Knowing is foundational to without having them develop the
Declarative objectives ask learners to doing, but it doesn’t go far enough. plan to see how well it was done?
remember or recall facts and Courses with mostly declarative You can’t. A test isn’t adequate.
concepts. In other words, they ask if terminal objectives probably don’t go
PAGE 9
learners know. These objectives use far enough either.

8

FROM PAGE 8 formance. There is no specific tional designers for ideas). For
number of levels to use in a rating example, in an online presentation
scale. It is better to start with fewer skills course I taught, learners peer
Performance assessments levels and increase them if needed, reviewed each others’ presentation
Performance assessments assess because with more levels judgments plans and slides using a checklist I
actual or simulated performance. For become finer and finer, and the likeli- devised. Learners gave their presenta-
example, if we want to see if a hood of rater error increases. Rating tion to a real audience and were
computer science student can write a scales with descriptors are best when assessed by that audience using the
real program, we provide the student there are clearly more than two levels rating scale I devised. The rating
with a programming challenge and of performance. scales were faxed back to me directly.
then assess his or her performance. Providing checklists and rating
In order to reduce subjectivity and scales for download is the same as
improve validity when assessing per- providing other types of print-based
formance, we can build a checklist or Providing checklists and rating materials. The real challenge is
rating scale. A performance checklist designing real or realistic activities
or rating scale lists specific behaviors
scales for download is the that allow learners to DO, and
or results to be demonstrated. building clear and specific enough
same as providing other types checklists and rating scales to assess
Checklists and rating scales will often
include a mechanism for tallying a of print-based materials. The them. Think out of the box.
final “score.” Many instructors share Remember that not all activities have
these in advance with students so real challenge is designing real to occur online (and many
they know how they will be assessed shouldn’t).
(good idea!). These checklists and or realistic activities that allow
rating scales are also commonly Patti Shank, PhD, CPT, is a widely
learners to DO, and building recognized instructional designer and
called rubrics.
Many instructors use authentic ac- clear and specific enough technologist, writer and author who
tivities but haven’t considered adding builds and helps others build good
checklists or rating scales to assess checklists and rating scales to online courses and facilitate learning.
them. Checklists and rating scales She can be reached through her
reduce subjectivity and increase relia- assess them. website, http://www.
bility and validity. They help learners learningpeaks.com/. @
achieve the desired success.
Let’s look at examples of portions Rating Scale with Descriptors and
of common types of performance as- Weighted Score
sessment checklists and rating scales. When building a rating scale, some
behaviors or results may need more
Checklist weight than others. In that case, a
A checklist lists the criteria and weighted rating scale may be
specifies the presence or absence of appropriate.
the behavior or results. A checklist is
often the best verification instrument Performance assessment
to use when unqualified presence is online
important (and it’s also easy to Some online instructors dumb
assess) because it doesn’t require the down their activities and assessments
rater to make in-between judgments. because online course systems make
grading tests easy. This shortchanges
Rating Scale with Descriptors learners and learning.
A rating scale with descriptors Consider what’s needed to make
provides a list of criteria with a de- real or realistic performance a possi-
scription of what the behavior or bility and how to assess it (ask other
results look like for each level of per- online instructors and the instruc-

9

answers from a list of alternatives.
Because they do not ask learners to
construct an answer or actually
Strategies for Creating Better perform, they tend to measure
knowing about rather than knowing
Multiple-Choice Tests how.
Multiple-choice items cannot assess
learners’ ability to construct, build, or
perform. They are best used for ob-
By Patti Shank, PhD, CPT jectives that can be assessed by
selecting the correct answer from a
list of choices rather than supplying
the answer or performing a task.
ultiple-choice tests are easy to score and less susceptible to Think for a moment about how

M commonly used to assess
achievement of learning ob-
jectives because they can be efficient.
scoring subjectivity than short-
answer or essay-type items. They
don’t measure writing ability (which
different selecting is from construct-
ing and performing and you’ll
recognize the limitations of multiple-
Despite their widespread use, they’re can be a plus or minus) and often do choice testing.
often poorly designed. Poorly written assess reading ability (another Consider the following learning ob-
multiple-choice tests are equally potential plus or minus, but in reality jectives (Table 1) and decide if you
damaging in classroom-based and often a minus). They are more think a multiple-choice test is a good
online courses, but in online courses subject to guessing than many other assessment option. My answers
learners often have to contend with types of learning assessments. follow, so you may wish to cover
more challenges, and poor assess- Multiple-choice tests are often these up before proceeding.
ments can add insult to injury. In promoted as “objective.” Although
addition, poorly written tests can be scoring them doesn’t involve subjec- My answers: Learning objectives
more visible to the world when they tivity, humans do judge what one and two can be assessed with
are placed online. questions to ask and how to ask multiple-choice items because the
In this article, I’ll look at the them. These are very subjective response can be selected effectively.
plusses and minuses of multiple- decisions! Multiple choice is not the best way to
choice tests, when they are appropri- assess learning objectives three and
ate and less appropriate, typical When multiple-choice is four because they require the learner
mistakes writers of multiple-choice appropriate to either construct a response or
questions make, and how to avoid Multiple-choice test items call for perform a task.
these mistakes. learners to select an answer or
PAGE 11

Some plusses and minuses Table 1
Multiple-choice tests can be
developed for many different types of Learning objective Is multiple choice a good
content and, if the test items are well assessment option?
written, can measure achievement of
multiple levels of learning objectives, 1. Identify risk factors… yes no
from simple recall and comprehen- reason:
sion to more complex levels, such as
ability to analyze a situation, apply 2. Detect errors… yes no
reason:
principles, discriminate, interpret,
judge relevance, select best solutions,
3. Explain the purpose… yes no
and so on.
reason:
Multiple-choice tests are easy to ad-
minister and can be improved using 4. Create a plan… yes no
item analysis in order to eliminate or reason:
correct poorly written items. They are

10

FROM PAGE 10
Table 2
Item parts and problems
It’s important to use multiple- Part Description Common Problems
choice tests wisely and to write good
test items when you use them. Let’s 1. Stem Describes the • Confusing or
look at the parts of a multiple-choice • question to be answered, ambiguous language
item and common problems with • incomplete statement to • Inadequate instruc-
those parts. (Table 2) be completed, tions
A multiple-choice question test • decision to be made, or
item has two parts: a stem and • problem or situation to
be resolved
multiple alternatives. The stem
initiates the item with a question, in-
complete statement, or situation. The 2. Alternatives The alternatives from • Confusing or ambiguous
alternatives are a list of possible which the learner language
answers or solutions. They include selects the correct • No clear right answer
the right answer(s), also known as answer(s) • Poorly written or im-
the key, and inferior or wrong plausible distractors
answers, known as distractors.
Let’s look at a few examples to see - Key The correct answer(s)
these problems in action. - Distractors The incorrect answer(s)

Example 1
Plants that are bona fide annuals,
The writing is clearer, and a and c A better item:
rather than half-hardy annuals,
are unambiguously correct. Under what circumstances can a
biennials, or perennials:
Distractors b and d are plausible to staff member accept a gift from a
a. Live for one growing season
those who don’t know the content. vendor, based on the vendor gift
b. Bloom during their growing
policy? Select the best answer.
season
Example 2 a. A gift can be accepted if the gift
c. Live and bloom for multiple
A vendor offers a staff member two is not considered valuable.
years
tickets to the World Series. Based on b. A gift can be accepted if the
d. Are more attractive than other
the rules listed in the vendor gift gift’s actual value is under $25.
flowers
policy, he or she cannot accept them: c. A staff member may never
a. Unless the tickets have no street accept a gift from a vendor.
The stem is potentially confusing
value
and the directions do not tell us to
b. If the vendor is expecting “quid The writing is clearer and the dis-
select all that apply. Both a and b are
pro quo” tractors are plausible to those who
correct, and b is also true for the
c. If the gift is worth more than $25 don’t know the content.
other types of plants described. And
or is considered to be an induce-
d is clearly implausible.
ment Writing better multiple-choice
A better item: d. Unless the vendor is also a items
Which of the following statements friend Confusing and ambiguous language
are true about annuals? Select all that e. None of the above and poorly written or implausible dis-
apply. tractors are very common errors
a. They grow and bloom during The language requires understand- when writing multiple-choice test
one season. ing of “quid pro quo” and “induce- items. Here’s a to-do list to help you
b. Many bloom throughout their ment” and includes confusing avoid these mistakes and write better
growing season. negatives. Distractor c has two multiple-choice test items.
c. Many bloom in the early spring answers in one distractor; d is
only. obviously implausible; and e, “None
PAGE 12
d. They commonly have shorter of the above,” is not recommended as
growing seasons than annuals. a distractor choice.

11

FROM PAGE 11 less knowledge or skill) 6.Make sure to use different place-
b) Statements that are true, but ments of the correct answer. (The
Use clear, precise language do not answer this question most common placement of the
1.Provide clear directions. Group c) Content that is paraphrased correct answer is c, and test-wise
questions with the same direc- incorrectly learners know this.)
tions together.
2.Include as much of the question Your turn
as possible in the stem, and Get together with other multiple-
reduce wordiness of alternatives. choice item writers (faculty, instruc-
Include words in the stem that Poorly written multiple-
tional designers, etc.) and use this
would otherwise be repeated in
each of the alternatives.
choice test items are almost guidance to critique existing test
items and suggest improvements.
3.Make sure language is precise, as ubiquitous as multiple- Poorly written multiple-choice test
clear, and unambiguous. Include items are almost as ubiquitous as
qualifiers as needed, but don’t choice tests, but it doesn’t multiple-choice tests, but it doesn’t
add unnecessary information or have to be this way. There are many
irrelevant sources of difficulty. have to be this way. There ways to frustrate learners in online
4.Avoid highly technical language courses, but this shouldn’t be one of
or jargon unless technical
are many ways to frustrate
them. Although it takes time and
knowledge and jargon are part of learners in online courses, practice to write good items, this time
the assessment. and effort is certainly well spent.
5.Avoid negatives and these words: but this shouldn’t be one of
always, often, frequently, never,
none, rarely, and infrequently. them. References/resources
When a negative is used, it Shank, P. (2006) Developing
should be CAPITALIZED, under- Learning Assessments for Classroom,
lined, or bolded to call attention Online, and Blended Learning.
to it. 4.Avoid distractors that combine Workshop Materials. Centennial, CO:
6.Don’t use double negatives or distractors (“b and c”), “all of the Learning Peaks.
double-barreled questions above,” and “none of the above.”
(asking two things in one 5.Avoid giving away the correct Patti Shank, PhD, CPT, is a widely
question). answer in the stem or alterna- recognized instructional designer and
tives. Common problems: technologist, writer, and author who
Write good alternatives/distractors
a) Alternatives that do not follow builds and helps others build good
1.If alternatives include best and
grammatically from the stem online courses and facilitate learning.
not-as-good alternatives (“Select
indicate an incorrect response. She can be reached through her
the best answer…”), provide
b) An alternative that uses the website: http://www.
enough detail to differentiate best
exact same terminology as the learningpeaks.com/. @
from not-as-good.
stem indicates a correct
2.Make sure that each item has an
response.
unambiguously correct answer or
c) Two options that are synony-
answers.
mous or almost identical
3.Make sure distractors are
indicate two incorrect or two
plausible, especially to those with
correct responses.
lower skills or lesser knowledge.
d) An alternative that is much
These are the best types of
longer than the others
plausible-but-incorrect distrac-
indicates a correct response.
tors:
e) Don’t give away the answer in
a) Common errors and
the wording of another
commonly held myths or mis-
question!
conceptions (for those with

12

How can you actively engage
Assessing Student Learning students in the assessment
process?
Online: It’s More Than Multiple Assessment strategies are most suc-
cessful if they replicate something
that the student will do in his or her
Choice profession, that is clearly relevant to
the course, and that is useful in
demonstrating his or her knowledge
and abilities. This type of assessment
By Elizabeth Reed Osika, PhD strategy, known as authentic assess-
ment, actively engages students and
demonstrates to the instructor that
they not only understand the
s more and more instructors Typical assessment strategies of

A move their courses into the
online environment, one con-
sistent question that arises is, “How
objective tests are easily administered
through the various course manage-
ment systems. Once online, these
concepts but can also apply them in
real-life scenarios (Mueller, 2006).
In addition to using appropriate
and, when possible, authentic assess-
do I know what the students have objective tests can now incorporate ments in an online environment, it is
learned?” The answer is not simple, video, audio, and other media. Other important to keep students actively
but it can be effectively addressed assessment strategies commonly used engaged in the “classroom.” This is
with some common sense and a little in the traditional classroom can also best accomplished by requiring
bit of creativity. be easily moved into the online envi- frequent, small assessments that will
ronment, such as discussions and require the student to access the
How do I know what the submission of written papers, essays, course two or three times a week. For
students have learned? or reports. example, Wednesday requires a short,
The best place to start to resolve With a bit of creativity, effort, and objective quiz on the assigned
this question is your course syllabus. basic technical skills you can add 1) reading material; Friday, a submis-
Aligning assessment strategies with simulations and activities where sion of a written summary of this
your learning objectives allows you students can virtually arrange week’s activity and the posting of a
to check whether or not students elements, manipulate variables in ex- reflection of what was learned
have met the objectives of the course. periments and equations, and even through the assignment on the dis-
Any course, online or face-to-face, diagnose virtual patients; 2) group cussion board; and finally, Sunday, a
should have clear, measurable objec- projects where students can address reply to a fellow student’s reflection.
tives. An instructor need only pick case studies, collaborate on presenta- Frequent, small assessments lets
the assessment technique that best tions or reports, and develop original them know how they are doing in the
matches each learning objective. programs, lesson plans, or other course and provides ample room for
For example, if the objective states materials; and 3) role plays where improvement or adjustment in study
that students will be able to describe each person has a role, position, or habits, if necessary.
a concept, ask them to write a paper, character to represent in course dis-
post information on a discussion cussions. How do you know it’s the
board, or create a flowchart. If they Allow the students the opportunity student’s work?
need to be able to identify or locate to practice with the technology prior How can I be sure that this is really
particular elements, they can to a formal assessment. Practice the student’s work? Unfortunately,
complete an objective quiz, post allows the students, and you, to there is no guarantee; though if you
relevant URLs, or submit digital become comfortable with the honestly think about a traditional
pictures. If you have taken the time required technology; thus, results are classroom, you typically can’t be sure
up front to identify clear, measurable less likely to be skewed by students’ there either. While this may seem like
objectives, determining assessment technical skills. a dire situation, there are ways in-
strategies becomes much simpler
(Florida State University, 2006). PAGE 14

13

FROM PAGE 13 to use all relevant resources to
complete a test, as this best reflects References
structors can limit the amount of dis- what occurs in real life. If we are Carneson, J., Delpierre, G., &
honesty in their courses. faced with a question we don’t know Masters, K. (1996). Designing and
One of the easiest ways is to assess the answer to, we Google the topic, Managing Multiple Choice Questions.
higher-order and critical-thinking seek knowledge from others, and, if Available online at
skills and limit the use of objective necessary, pull out texts to determine http://web.uct.ac.za/projects/cbe/
assessments such as multiple-choice the answer. Why not encourage this mcqman/mcqman01.html.
tests. Objective assessments are the problem-solving strategy in your Florida State University (2006).
easiest for students to negatively ma- students by posing complex Information about behavioral objec-
nipulate. Use multiple-choice or other questions that aim at higher-level tives and how to write them.
objective questions to assess informa- thinking skills (Carneson, Delpierre, Available online at
tion that provides a foundation for & Masters, 1996)? www.med.fsu.edu/education/
the course, the information they will Use a variety of assessments. If you FacultyDevelopment/objectives.asp.
have to use later to demonstrate their have only one or two large assess- Mueller, J. (2006). Authentic
understanding through an authentic ments in the course, the pressure on Assessment Toolbox. Available online
assessment. This allows the instruc- students to succeed is enormous, and at http://jonathan.mueller.
tor the opportunity to make sure that many may revert to less-than-honest faculty.noctrl.edu/toolbox/.
each student is familiar with the most activities. However, if you provide a
important aspects in the text. variety of assessments that build on Elizabeth Reed Osika, Ph.D. is an
If you use objective assessments as each other, the pressure is reduced assistant professor at Chicago State
a primary assessment strategy, drastically. Using frequent assessment University. @
assume your students will use their points also allows the instructor to
books and discuss the concepts with get to know students’ work and per-
friends and write your questions ac- sonalities, making the identification
cordingly. In fact, encourage students of dishonest work easier.

To Plan Good Online Instruction, Teach to the Test
By Patti Shank, PhD, CPT

uilding effective instruction test. You likely have heard the words typically have specialized training in

B involves multiple tasks, but
planning is one of the most
critical. For online courses, planning is
“teaching to the test” uttered contemp-
tuously. But it can be a very good thing
indeed. I’m going to take a bit of a cir-
using cognitive and other principles to
design effective instruction) call the
practice of systematically planning in-
especially important because even cuitous route in explaining why so you struction “instructional design.” There
under the best of circumstances, online can understand my logic. are numerous philosophies of instruc-
learners often struggle with under- Objectives are the cornerstone for tional design but all have certain things
standing what’s expected of them. At a planning effective instruction, and in common, including following a list
distance, they can get unbelievably good assessments determine if the ob- of tasks that ensure better end results.
frustrated (or worse) and stop trying. jectives have been met. You might Here is a list of typical instructional
That’s one of the best reasons for using consider these the “bookends” of planning tasks, in order:
a systematic approach to planning your planning effective instruction. 1.Identify learning objectives
instruction. 2.Design assessments
One of the best planning strategies ADDIE who?
for good instruction is teaching to the Instructional designers (people who PAGE 15

14

TABLE 1
FROM PAGE 14
If the learning objective is… A matching assessment could be…
3.Design content and activities
1. Learners will label the parts of Illustration of the human respira-
4.Select media and delivery options
the human respiratory system, tory system prompting learners to
5.Develop the course materials
including the trachea, bronchi, lungs, label the trachea, bronchi, lungs,
6.Implement the course
thoracic cavity, and diaphragm. thoracic cavity, and diaphragm
7.Evaluate and revise

If you have worked with instruc- 2. Learners will demonstrate three Demonstration(s) of the three
tional designers or read articles or elements of a proper phone greeting. elements of a proper phone greeting
books on instructional design, you
may be familiar with the ADDIE 3. Learners will use the three- Verbal or written application of the
model, one of the most common criteria model to evaluate a play. three-criteria model to evaluate a play
models for the systematic design of
instruction. ADDIE is an acronym for
Analysis, Design, Development,
TABLE 2
Implementation, and Evaluation.
Following a systematic process such If the learning A matching assess- Matching content
as ADDIE can help prevent some of objective is… ment could be… and activities
the typical problems that happen 1. Learners will label the An image of the human • Content: Images of the
when instruction isn’t well planned, parts of the human respira- respiratory system different parts, sepa-
including instruction that doesn’t tory system, including the prompting learners to label rately and together
seem to have a clear goal; quirky (not trachea, bronchi, lungs, the trachea, bronchi, lungs, • Activity: Practice
in a good way) or deficient course thoracic cavity, and thoracic cavity, and labeling the parts
content, activities, and assessments; diaphragm. diaphragm
and poor evaluations for the course
and instructor. 2. Learners will demon- Demonstration(s) of the • Content: Discussion of
Notice that identifying learning ob- strate three elements of a three elements of a proper the three elements
jectives is first on the list of tasks. proper phone greeting. phone greeting • Content: Examples of
And designing assessments is next, the three elements in
for good reason. use
• Activity: Practice using
Design assessments after the three elements
identifying learning objectives 1. Learners will demon- Verbal or written appli- • Content: Discussion of
Designing assessments should strate three elements of a cation of the three-criteria the three-criteria model
optimally occur right after identifying proper phone greeting. model to evaluate a play • Content: Examples of
2. Learners will use the the three-criteria model
learning objectives. That’s because
three-criteria model to applied to a variety of
assessments should measure if the
evaluate a play. plays
objectives were met. If the learning
• Activity: Practice
objectives are well written, appropri- applying the three-
ate methods of assessment are criteria model to a
generally quite clear. variety of plays
See TABLE 1 on how the appropri-
ate assessment matches the learning
objective? If you design assessments words, designing the assessment I’ve finally made my way to telling
as an afterthought at the end of (test) right after identifying the you to design to the test. First
designing the instruction (a common learning objectives 1) makes the identify the learning objectives and
but unfortunate mistake), you are needed assessment very obvious and matching assessment (test). The
likely to design the wrong content 2) provides clear cues about what learning objectives should clearly
and the course activities and the as- content and activities are needed. state what the learner should be able
sessments are likely to be far less Design content and activities after
meaningful or appropriate. In other PAGE 16
designing assessments

15

FROM PAGE 15 Aside from the fact that this would be with someone else (another faculty
unengaging for learners, this member or maybe even an instruc-
to do, and the assessment (test) approach is not instruction. Activities tional designer) to get feedback.
should measure if they can, in fact, and feedback are needed for instruc- Some people think it’s hard or even
do that. The content and activities tion. In fact, I’d go so far as to say impossible to create meaningful
should then be designed specifically that the purpose of instructional online activities, but that’s not so. In
so that the learner can pass the test, content is to support instructional ac- fact, an asynchronous online learning
because that means they have met tivities. Activities allow learners to environment provides opportunities
the learning objectives. And that’s reflect on and apply the content and for activities that would be hard to do
the goal of effective instruction. make it personally meaningful. When in person.
Let’s look at TABLE 2 once again at we don’t do this, we’re likely
the three objectives and matching as- teaching at only a surface level, References
sessments to see what content and preparing learners to do nothing with Shank, P. (2006.) Developing
activities make sense. the content other than forget about it Learning Assessments for Classroom,
As you can see, a well-written once the test is over. Online, and Blended Learning.
objective and matching assessment Workshop Materials. Centennial, CO:
provide pretty clear cues about what Your turn Learning Peaks.
content and activities are needed. It If activities are the opportunities Smith, P.L., & Ragan, T.J. (2005).
makes the instruction not only more for learners reflect on and apply the Instructional Design, 3e. San
effective, but also easier to design. content so that it becomes meaning- Francisco: John Wiley & Sons, Inc.
Better instruction and less work. ful to them, now would be a good
Terrific! time for you to do that with the Patti Shank, PhD, CPT, is a widely
content in this article! Using my recognized instructional designer and
A few more words about previous articles on writing objectives instructional technologist, and writer
activities and developing assessments, if who builds and helps others build
Some people ask me whether good online and blended courses and
needed, see if you can write two
content plus assessments is enough facilitate learning. She can be reached
good learning objectives and then
for a good online course—for through her website: www.
match assessments and content and learningpeaks.com/. @
example, PowerPoint slides and tests. activities. Try swapping your work

Using Self-Check Exercises to
Assess Online Learning
By Patti Shank, PhD, CPT

he intermediate statistics class One especially valuable technique folks who did easily gained confi-

T I took quite a number of years
ago had two types of learners
at the outset—those who were
was consistent use of self-check
exercises. These were handed out at
the end of each class along with an
dence and passed the course.
What are self-check exercises,
exactly? They are problems (with
worried about passing the course and answer sheet. Class started each time answers) given to learners that allow
those who were sure they couldn’t with a question-and-answer period them to assess how they are doing on
pass it. The professor clearly under- about the self-check exercises from an ongoing basis. Doing them online
stood the “fear-of-stats” phenomenon the previous session. Doing the with self-grading provides immediate
and used a number of instructional exercises was optional and they feedback. Links to additional
techniques to help learners gain con- weren’t handed in or graded, but
fidence and skills. nearly everyone did them, and the PAGE 17

16

FROM PAGE 16 • Helping learners determine what Getting Started
they do and do not understand so Consider how self-check exercises
materials can be provided to help they can target where extra study can be used in the courses you teach.
anyone who is having difficulties. is needed; Are there concepts that learners con-
Online learners can do these • Providing immediate feedback to sistently have problems understand-
exercises and submit questions they learners and an option to link to ing? Are there terms that learners
have, which the instructor can additional materials (which may need to memorize or concepts that
aggregate and respond to for the reduce the number of unfocused they need to understand? These
benefit of all learners. questions sent to the instructor); might be the best places to start.
Studies show that these types of ac- • Providing feedback to the instruc-
tivities help learners keep tabs on tor about where learners are Patti Shank, PhD, CPT, is a widely
their progress and adjust their efforts, having difficulties so immediate recognized instructional designer and
know when to seek help, and stay on interventions can be imple- instructional technologist, and writer
track. These outcomes are especially mented; and who builds and helps others build
important in online courses. • Increasing learner satisfaction good online and blended courses and
Some of the most important with the instructor and the facilitates learning. She can be
benefits of self-check exercises for course. reached through her website:
online learning include www.learningpeaks.com/. @

Assessment for the Millennial Generation
By Christopher Hill

resh assessment techniques are She believes that the new wave of with social networking sites like

F needed to gauge student
learning in online classes, Julie
Giuliani says. Giuliani is the executive
so-called Millennial students calls for
what she refers to as “new millen-
nium assessment approaches.” What
YouTube and MySpace, with podcasts
and blogs, and with devices like cell
phones and iPods. She came to
dean of the Virtual College at Florida she means by this is the employment believe that to engage these students
Community College- of assessment techniques that are in the process of assessment your best
Jacksonville(FCCJ), which serves specifically designed for online media bet was to use the same kinds of
some 39,000 distance students. that are part of the structure of the media. FCCJ had been forward
Giuliani has what she refers to as “a course, and where students perform looking in the emphasis it gave to
passion for assessment.” She has self assessment as they participate in multimedia in its online program. The
been working with assessment for course activities. school had created a faculty resource
years, since she was a graduate One of the first things Giuliani did center where any faculty member—
student researching classroom assess- when she came into her position was full-time or adjunct—could produce
ment techniques. to familiarize herself with the quality their own multimedia tools.
“There’s been volumes of research of teaching and learning in the online Giuliani soon began to look for an
that says there’s no significant differ- courses. As she went through the opportunity to try out her ideas. The
ence in terms of grades between process, it became evident to her that, school’s interim associate dean
students on land and online,” Giuliani while they were doing many things happened to also be an online adjunct
says. “But there isn’t any formative right, they still weren’t where they history professor. Giuliani proposed to
kind of mechanism to gauge online needed to be in terms of creating a her that they use her American
student learning progress. We really culture of systematic observation of history course as a laboratory to ex-
can’t validate how well our students student learning.
are learning online.” The students were already familiar PAGE 18

17

Faculty Focus Special Report: Assessing Online Learning: Strategies, Challenges and Opportunities

Faculty Focus Special Report: Assessing Online Learning: Strategies, Challenges and Opportunities

Recommended

Recommended

More Related Content

What's hot

What's hot (8)

Viewers also liked

Viewers also liked (6)

Similar to Faculty Focus Special Report: Assessing Online Learning: Strategies, Challenges and Opportunities

Similar to Faculty Focus Special Report: Assessing Online Learning: Strategies, Challenges and Opportunities (20)

More from Dillard University Library

More from Dillard University Library (20)

Recently uploaded

Recently uploaded (20)

Faculty Focus Special Report: Assessing Online Learning: Strategies, Challenges and Opportunities