Pedagogical and Assessment Design for Online Learning

Pedagogical and Assessment
Designs for Online Learning

Barbara Means
Center for Technology in Learning
SRI International

Keynote presentation for the CITE Research Symposium
E-Learning Designs & Designs for Learning
March 6, 2010
Hong Kong
© 2007 SRI International

Outline of Talk

 Emergence of online learning at secondary level in
U.S.

 Variations in online learning designs

 Research synthesis on the effectiveness of online
learning

 Role of formative assessment in enhancing
learning outcomes for blended instruction

2

Growing Importance of Online Learning

 Long a part of corporate and military training

 Online learning has become a big part of higher
education and of self-directed adult learning

 Now it is becoming part of K-12 education (mostly
at the secondary level)


K-12 adoption of online learning rising in U.S.
Enrollments up from 45,000 in 2000 to 1,000,000 in 2007

Online Learning Enrollment
0.08
0.07
0.06
0.05
0.04 Online Learning
0.03 Enrollment
0.02
0.01
0
1998 2000 2002 2004 2006 2008

Source: Estimates of secondary school online enrollments from Picciano & Seaman, 2009

Copyright Clayton M. Christensen

Enrollments up from 45,000 in 2000 to 1,000,000 in 2007

0.10

0.09

0.08

0.07

0.06

0.05

0.04

0.03

0.02

0.01
Online Learning
Enrollment
0
1998 2000 2002 2004 2006 2008 2010 2012 2014

Source: Estimates of secondary school online enrollments from Picciano & Seaman, 2009



0.10

0.09

0.08

0.07

0.06

0.05

0.04

0.03

0.02

0.01
Online Learning
Enrollment
0
1998 2000 2002 2004 2006 2008 2010 2012 2014

Clayton Christensen predicts that by 2019 half of all
high school courses will be taken online.


U.S. districts and states are starting to
encourage online learning
 Most school districts (75% in 2007) provide online
learning options for at least some of their students
 Several state-funded programs offer online high school
courses (Florida, Illinois, Michigan, Kentucky)
 24 of 50 states now have statewide full-time online
schools (“cyberschools”)
 Two states (Michigan and Alabama) require students to
take an online course in order to earn a high school
diploma
 One state (Florida) requires all of its districts to make
online learning options available to their students

7

Typical Reasons for Offering Online Course
Options

 Provide a course not otherwise available
 Provide more college-level (Advanced Placement)
course options
 Give students a second chance through “credit
recovery” programs

8

Different Forms of Online Learning

 Fully online v. hybrid or “blended”
 Module within a course, supplemental formal course,
degree program
 Scheduled course (“class-linked”) v. self-paced
 Instructor-developed v. professionally developed


Instructional Design Elements in Online
Learning

 Communication type
- Synchronous
- Asynchronous

 Learner control
 Instructional approach
- Expository
- Active
- Interactive


Expository Learning Environments

Present content to the learner

11

Active Learning Designs

Support the practice of
skills and elicit student
responses

Carnegie Learning’s Algebra I Tutor

12

Interactive Learning Designs

Involve collaboration with
other learners or an
instructor with the focus
of engagement emerging
over time

Political Science 105 from University of North Carolina Greensboro

13

SRI’s Meta-Analysis of Online Learning
Research
 Systematic search for studies
from 1996 through July 2008 on the
effectiveness of online learning
 Limited to studies that
- compared an e-learning or blended
condition to face-to-face instruction
-measured learning objectively with the
same measure for treatment and control
groups
-used an experimental or quasi-
experimental design with control for any
pre-existing differences between groups
-provided the statistical information
needed to compute an effect size

 Out of 1,132 articles reviewed, 51
study effects met the requirements
for meta-analysis


Meta-Analysis Findings

 Average effect size of +0.24
favoring the online condition
 Advantage over face-to-face
instruction was larger for studies
using blended approaches (E.S. =
+0.35) than for studies using pure
e-learning (E.S. = +0.14)
 Advantage over face-to-face
instruction was larger for studies
where
- online students spent more time
learning than did those in the face-to-
face class
- the online and face-to-face conditions
varied in terms of content and
instructional approach

Taken as a whole, the findings suggest that the observed advantage of online learning is a
product of redesigning the learning experience, not of the medium per se.

SRI’s Review of Studies Comparing Different
Online Learning Designs
 Online learning usually more
effective when
- it stimulates more active engagement
with the content

- it includes prompts for learner
reflection
-learners have an element of control
over their interactions with the software

 Some common practices usually
had no effect
-adding additional media not related to
the content to be learned
- adding multiple-choice quizzes

Results were inconclusive with
respect to different instructional
approaches
www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf


Conjectures on the Blended Advantage

 More learning time
 More interactivity
 Selective re-teaching of difficult concepts and
skills


The Concept of Formative Assessment

 Assessment for learning rather than
assessment of learning
 Occurs during the course of learning while there
is still time to improve outcomes
 A quality of the way in which assessment
information is used, not inherent in the test itself
 One of the most powerful levers for improving
learning outcomes (Black & Wiliam, 1998)


e-Learning Offers New Options for Formative
Assessment

 We can do much more than just interspersing sets of
multiple-choice questions into instructional materials
 Technology enables automated capture of complex
learner behaviors
 Assessment can be embedded in the learning in ways
that feel natural rather than forced

19

Learning in a Multi-User Virtual Environment
Chris Dede’s River City MUVE (Science, 2009)

Enlargement of Microscope

20

River City

 Teaches concepts from biology, ecology and
epidemiology
 Students work in teams of three, moving through the
city to run tests in response to the mayors’ challenge
 Teams keep online journals, analyze data, form
hypotheses, and write up their research in a report for
the mayor
 Researchers developed measures of science concept
knowledge, science inquiry skills, and sense of
efficacy as a scientist based on student actions within
the River City environment

21

Such online learning environments can be
designed to capture . . .

 Where a student went
 With whom the student communicated and what was said
 The artifacts the student activated
 Databases the student used
 Data that the student gathered using virtual scientific
instruments
 Screenshots and notations the student entered in a virtual
notebook
 Hints that the learner requested

All of this information can be used for assessment.
22

Examples from Carnegie-Mellon’s Online
Learning Initiative (OLI)

12 web-based colleges courses that can be taken
entirely online


 Some of the systems contain virtual laboratories

24


 Some of the systems contain virtual laboratories

25


 Others incorporate intelligent tutoring and carefully
designed exercises

26


 Others incorporate intelligent tutoring and carefully
designed exercises

27

The Boxplot

The boxplot graphically represents the distribution of a quantitative variable by visually displaying
The five number summary and any observation that was classified as a suspected outlier using the
15(IQR) criterion.

Here is how a boxplot is constructed (this is for the “Best Actress” dataset – to see the dataset click here.)

To see a static version of this movie, click here. Use the pull-down menu to label the various points on the boxplot.

X That is not quite right. I think you might be
Source: Carnegie-Mellon University, OLI confusing Q1 and Q3.
Recall that Q1, the first quartile of the distribution,
Course in Statistics is represented
By the bottom edge of the box. Q3, the third
quartile of the 28

Multiple Feedback Loops

Source: Thille, 2009

29

Formative Assessment Is Woven into the
Online Learning
 Interactive simulations that students can manipulate are
woven into the courseware. Students interact with the
simulation and then respond to probes that get at their
understanding of how the simulation works.

 “Did I Get This?” quizzes follow the presentation of new
material so that students can check their understanding
without fear of hurting their course grade.

 Short essay questions that call on students to make
connections between different concepts are embedded
throughout the online course material.

 “Muddiest Point” requests ask students what they thought
was confusing.

30

In Blended OLI Courses Instructors Take
Advantage of eLearning Assessments

Each student works with the
online course materials
outside of class.

31


Source: Heffernan & Thille, Carnegie Mellon Online Learning Initiative web presentation

The e-learning system aggregates information from all the students who used it.

32


The instructor sees what concepts
students had trouble with.

33


In class, the instructor focuses on concepts that students
struggled with when they worked on line.

34

The OLI Statistics Course Study
 Students were assigned at random to take college statistics
on line or in a conventional class.
 Students in the online version of the class finished the
material in half a semester or half the time taken by
students in the conventional class.
 The two classes took the same final examination, and the
online students’ scores were significantly better than those
of the students in the conventional class.
 No difference between the two groups on a long-term
retention test.
 Study has been replicated in a community college and a
large public university.

Source: Lovett, Meyer, & Thille, 2008
35

Principles for e-Learning Design
 Incorporate a range of online activities that engage learners
with content.
 Give learners options for different ways to learn content and
with options for review and support (scaffolding).
 Embed assessment within the learning activities and use
multiple assessment approaches.
 Make assessment feedback available to the student as well
as to the instructor.
 Use information from students’ online activity to shape the
content of face-to-face instruction.

36

Thank you!

NIH Media and Children
May 15,

Pedagogical and Assessment Design for Online Learning

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Similar to Pedagogical and Assessment Design for Online Learning

Similar to Pedagogical and Assessment Design for Online Learning (20)

More from CITE

More from CITE (20)

Recently uploaded

Recently uploaded (20)

Pedagogical and Assessment Design for Online Learning

Editor's Notes