This document discusses the design of online assessments to measure metacognition. It aims to compare online assessments, like games measuring strategy and reflection, to traditional offline self-assessments. Two online games discussed are Oregon Trail, which assesses qualitative thinking aloud, and Frog Escape, which quantitatively measures performance. The goals are to align the assessments with an established model of metacognition and address limitations of offline measures like recall failure or inability to directly observe metacognitive skills. Challenges include implementing the assessments at scale, the labor required, technological issues, and whether metacognition can be directly measured during cognitive tasks.

1. RESEARCH POSTER PRESENTATION DESIGN © 2011
www.PosterPresentations.com
Oregon Trail
• A history based, online strategy game aimed at creating a
fun and engaging learning experience about Westward
Expansion through the journeying from Independence,
Missouri to the Willamette River and valley in Oregon.
• As students come across various obstacles along the way,
we will assess through the qualitative Think-Aloud their
degree of metacognitive application.
Example Prompt and Student Response:
o Prompt: Whenever a student changes pace or
food rations/stops to rest, trade, hunt, etc., ask
student why they made the decision that they did
immediately after.
o Sample response for stopping to rest:
“My people were in poor health and by
stopping, I hoped they would be able to get
better and not get sick again.”
o Potential feedback for coding: Student
demonstrates both ‘Applying strategy’ and
‘Reflect and Adjust’ steps of metacognition
through her decision of and reasoning for
stopping to rest.
Frog Escape
• An online strategy game with the goal of preventing a frog
from escaping out of the pond by removing all possible paths
to the perimeter of the pond.
• Through a student’s final score at the end of several levels of
the game, as well as the amount of moves taken and the
time it took the student to complete the level, we will
quantitatively measure the amount of metacognition used by
that student.
• Metacognition is understanding and controlling one’s
own cognitive processes, and is helpful for students to
learn more effectively.
• LEARN 2 LEARN was an intervention developed by
former Trinity students that sought to enhance the
learning and academic performance of 8th grade social
studies students by incorporating the instruction of
metacognitive strategies into their curriculum.
PROJECT EVOLUTION: Because this project has been
part of an ongoing partnership between Trinity College
and HMTCA, this year’s intervention will differ mainly
through:
• The inclusion of two online based assessments of
metacognition, one incorporating a Think-Aloud
component, in addition to the past offline assessments
(Veenman et al., 2014).
o Online assessment = A method of obtaining
data during task performance
o Offline assessment = A method presented
either before or after the task performance
o Think-Aloud = A strategy that has students say
aloud and/or answer questions about what they
are thinking as they go through the activity itself
• Assessments are done both in order to measure the
change in metacognition usage for each student as the
intervention progresses and how each student is
applying metacognition at the end of the intervention.
OVERVIEW
PAST MEASURES
RESEARCH GOALS
NEW MEASURES
REFERENCES
DESIGNING ONLINE ASSESSMENTS
FOR METACOGNITION
Advisors: Prof. Dina Anselmi & David Reuman
Bettina Gonzalez (‘16), Lauren Thomann (‘16), Evan Scollard (‘17)
ACKNOWLEDGEMENTS
We would like to thank Professor Reuman and Professor
Anselmi for their assistance and guidance throughout the
creation of our newest assessments, as well as the coding
process. We’d also like to thank Ms. Avery for her ongoing
support of our intervention in her classroom. Additionally,
we’d like to acknowledge Melody Fulton and Jennifer
Schackner for their guidance with the MC5 coding system,
of which we have begun to use as a basis for establishing
our own coding schemes for the Oregon Trail and Frog
Escape. Lastly, we’d like to thank the Faculty Research
Committee for supporting our summer work.
Benefits:
• Allows one to generalize outcomes across different
content areas
• Provides immediate response while ideas are still fresh
• Can track progress throughout the entirety of an activity
• Some online measures (e.g., Frog Escape) do not require
self-reports
Qualitative MC5: An offline assessment that evaluates
students’ metacognitive abilities at an eighth grade level
and consists of 8 open ended, self-report questions based
on the Ambrose et al. 5-step model of metacognition.
• Example: “What did you think were the two most
important purposes of the Civil War project? Did you
understand the directions your teacher gave you? If not,
what did you do?” (Assessing the Task, Applying
Strategy)
Quantitative MC5: An offline assessment linked to the
Ambrose et al. 5-step model of metacognition that consists
of 35 self-report items that individually target the 5 different
steps at an eighth grade level.
• Example: “After I get an assignment back, I try to figure
out how I could improve my work for next time.” (Reflect
and Adjust)
PROBLEMS WITH OFFLINE SELF-ASSESSMENTS
• Questions may be too global
• Questions may be too removed in time from the
original metacognitive activity, leading to recall failure
• Given the automatic nature of metacognitive skills,
students may not realize they are using them
• Students may not be able to apply metacognitive skills
or describe them in our questions
• Students tend to under- or over-estimate their
metacognitive abilities
ONLINE MEASURES
DISCUSSION
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M.
C., & Norman, M. K. (2010). How Learning Works: Seven
Research-based Principles for Smart Teaching. John
Wiley & Sons.
Veenman, M. V., Bavelaar, L., De Wolf, L., & Van Haaren,
M. G. (2014). The on-line assessment of metacognitive
skills in a computerized learning environment. Learning
and Individual Differences, 29, 123-130.
Ambrose et al.’s Model of
Metacognition
Overall Purposes:
• To design online assessments of metacognition.
• To compare online assessments with offline self-assessments.
• To create a rubric for quantifying behavioral and verbal indicators of
metacognition.
• Our aim is to have our online assessments align with the Ambrose et al.
Model of Metacognition shown to the left.
• After running pilot tests with two early
adolescents, we have determined that
the greatest challenge to these new
assessments will be creating the
coding rubrics and the overall
accessibility of these metacognitive
innovations to teachers in various
classroom settings. We plan to run a
larger pilot test this Fall semester with
an entire class of eighth grade students
at another magnet school to determine
how to help with these problems.
• For our 2015-2016 metacognition
intervention we plan to implement this
new set of mainly online based games
and assessments. Using the Oregon
Trail, which we will use to perform both
a Think-Aloud assessment for a subset
of students, as well as a qualitative
self-assessment for all students, and
Frog Escape, which we will use as a
quantitative measure, we hope to foster
innovative and more successful ways
of measuring metacognitive
achievement and improvement.
Challenges:
• Hard to implement with large groups
• Labor-intensive for researchers
• Time-intensive for students
• Technological obstacles
• Can you measure metacognitive skills directly from a
cognitive task?