This document summarizes lessons learned from 10 formative assessments of educational initiatives at MIT from 2000-2003. It finds that educational technologies are most effective when they meet an unmet educational need, are well integrated into the curriculum, and their use is connected to learning objectives, pedagogies and assessment. Technologies are less effective when they require students to learn the technology before course content or when there is too much technology. The relationships between technologies and the learning environment they are used in must be carefully considered.

1. Lessons Learned
Findings from Ten Formative Assessments
of Educational Initiatives at MIT
(2000-2003)
Lori Breslow, Ph.D.
and
The Staff of the
Teaching and Learning Laboratory

2. The Context
From 1999 to the present, MIT has undertaken approximately 40
experiments in educational innovation. TLL has assessed:
Subjects
1.00,* 6.001*, 6.002x,* 8.02
(TEAL),* 8.224, HST582J,
18.03, Mission 200X*
Programs
The Undergraduate Exchange
with University of Cambridge,*
Residence-Based Advising*
Technologies
PIVoT, Cross Media Annotation
System (XMAS),* MetaMedia,*
PRS
Space
The d’Arbeloff/TEAL
classroom (26-152)
*Denotes multi-year or multi-semester assessment

3. Agenda
• Four lessons about the use of educational
technology
• Observations on best practices in design,
implementation, and assessment
• Where do we go from here?

4. Four Lessons about
Educational Technology
• Educational technologies have contributed to gains in
learning
• Educational technologies that have met a specific
educational need that has been unmet or poorly met
by traditional media have been more successful
• Too much new technology, too many technologies,
technologies that are poorly integrated into the
curriculum have been less successful
• Studying the relationship between educational
technologies and the contexts in which they are
embedded has yielded important understandings

5. Technology-Enabled Active Learning
Increased Learning Gains
Lesson #1
Learning gains <g> = %Correct post-test - % Correct pre-test
100% - %Correct pre-test
0
0.1
0.2
0.3
0.4
0.5
0.6
Low Intermediate High
Learning
ga
ins
<g>.
Lecture Students 2002 TEAL Students 2003

6. A Problem-Based Module Increased
Learning Gains
Source: Greenberg, J. Smith, N. & Newman, J. (2003) “Instructional Module in Fourier Spectral Analysis,
Based on Principles of How People Learn,” Journal of Engineering Education
Lesson #1

7. What Has Worked
• Online lectures to teach students basic concepts
• Visualizations to help students see what cannot
be seen
• Archives of visual images to help students
strengthen different kinds of literacies
• Wireless laptops to learn programming
• Remote online laboratories
• Technologies that provide instantaneous, more
individualized feedback
Lesson #2

8. Online Lectures Motivate Students
Lesson #2

9. Online Lectures Were More
Effective in Intro Computer Science
Lesson #2
Source: Newman, J. (2002) “6.001 Report on the Outcomes of Online Learning,”
unpublished assessment report

10. Visualizations Help Students See
What Cannot Be Seen
An animation used in Physics II, a course in electromagnetism” (8.02T). Click to see how field lines move.
For more information on the Technology-Enabled Active Learning project, go to
http://web.mit.edu/8.02t/www.

11. MetaMedia Creates Visual
Archives that Expand Literacy
MetaMedia homepage. For more information on MetaMedia, go to
http://metaphor.mit.edu/.

12. iLab Makes Laboratory Facilities
Available Remotely
The Flagpole Project allows students to take readings remotely. For more information on iLab, go to
http://icampus.mit.edu/projects/iLab.shtml.

13. Two Other Effective Applications of
Educational Technology
• Using wireless laptops to learn programming in
“Introduction to Computers and Engineering
Problem Solving” (1.00)
• Improving feedback
– To the students through hint and check buttons
used to answer problems embedded in online
lectures (6.001)
– To the faculty through a personal response
system that records student answers to
conceptual questions asked during class (8.02T)
Lesson #2

14. What Hasn’t Worked
• Technologies designed for interaction (e.g.,
discussion boards) have been less successful at
MIT
• When students need to learn the technology before
they can learn the material, they have two things
to master
• Too much technology can be detrimental
• When technology is poorly integrated into the
curriculum, it is either not used at all or not used
effectively by the students
Lesson #3

15. Applications That Needed
Some Work
• Functionality of
– MetaMedia
– XMAS
– Simulation in “Biomedical Signal Image and Processing”
• Discussion boards in “Exploring Black Holes”
• Unchat in Mission 200X
• Technology in 8.02T
– PRS
– PowerPoint
• Laptops when used in lectures in 1.00
Lesson #3

16. Edtech and the
Learning Environment
• Edtech has been most successful when there
are strong connections between it and
– Learning objectives
– Pedagogies
– Assessment
• The same technology will have different
effects in different environments
• Edtech exerts its impact by changing the
nature of information in the system
Lesson #4

17. There Are Strong Connections
in TEAL
• Objective: to increase students’ conceptual
and analytical understanding about the
nature of electromagnetic fields
• Technology’s contribution
– Visualizations
– Desktop experiments
– PRS
Lesson #4

18. But the Same Technology May
Have Different Effects . . .
• In different learning environments
– PIVoT as used at MIT, RPI, and Wellesley
• In different situations
– Laptops in 1.00 during active learning and during lecture
• For different students
– For example, less prepared MIT students were
significantly more likely than better prepared MIT
students to think PIVoT helped their conceptual
understanding
– But at RPI, students with better high school preparation
derived greater benefit than those with weaker preparation
Lesson #4

19. Edtech Changes the Nature of
Information in the System
Information Has Five Properties
Form Magnitude Velocity Direction Access
Source: Nystrom, C. (1973) “Towards a Science of Media Ecology: The Formulation of Integrated Conceptual
Paradigms for the Study of Human Communication Systems,” unpublished doctoral dissertation
Lesson #4

20. How Edtech Changes Information
in the System and the Impact
• If form changes student literacies
expand
• If magnitude changes students can be
overwhelmed by information
• If accessibility and direction changes
relationship between students and instructor
shifts
• If velocity changes feedback can be
more instantaneous
Lesson #4

21. Best Practices in Design,
Implementation, and . . .
• Design
– Begin with learning objectives
– Research what is already known or has been
done related to the innovation
• Implementation
– Estimate the amount of resources (time,
funding, space, etc.) needed; then increase it
– Assume mid-course corrections will be needed

22. ... Assessment
Assessment at MIT works best when:
• It is formative
• It is collaborative, which means
– Demands on faculty time must be honored
– Differences in research in the “hard” and
“soft” sciences must be made explicit
• Quantitative and qualitative methodologies are
used

23. Next Steps
• The longer term impact of the earliest projects are
being studied
– TEAL
– Mission 200X
– 6.002x
• The “lessons learned” will be disseminated
through a variety of venues at the Institute and in
the wider academic community
• The next set of research questions of interest to the
faculty and the TLL staff will be prioritized