The document discusses different types of questions that instructors can ask to evaluate the effectiveness and impact of teaching projects, including questions about whether learning outcomes were achieved, how student learning can be better understood, and how students are experiencing the course. It provides examples of specific questions and suggests collecting a variety of evidence, such as student work and surveys, to help answer evaluation questions. The goal of evaluation is to determine what is working and how teaching can be improved.

1. Evaluating Your Students’
Learning & Whether Your Project
Is Succeeding
Innovators’ Toolkit 4
Julie Sievers, Center for
Teaching Excellence

2. Why Evaluate Your Project?
To see if it was worth your
time and effort
To decide whether your
colleagues / department /
field should adopt similar
strategies in other courses.
To enable you to base your
arguments about these
issues on evidence, not just
anecdote and impression.
Scholarship of
Teaching &
Learning:
To enable you to
present at a
conference, or
publish in a
journal

3. A Taxonomy of Questions
1. Assessing Effectiveness of
Course, Project, Method
• Did I accomplish my goals?
• Does this strategy work?
• Does it work better than
other strategies?
“What Works?” Q
– in Pat Hutchings
taxonomy of questions,
Opening Lines (2000)
Assessment Q:
– Similar to program
assessment for SACS
• “Are you accomplishing
what you set out to
do?”
• “How do you know?”

4. Effectiveness Questions
Success or failure question:
Did students meet the SLOs? Yes or no.
• Variation: Did students’ meet this or
that SLO?
• Collect evidence similar to that used
for program assessment.
Comparative question:
Did the students meet the SLOs BETTER
in this version of the class than in
another?
• Requires comparison data
• May compare to your previous
version, or a standard version you are
also offering.
• May compare to a colleague’s course.
Improvement question:
Did the students IMPROVE in their
ability to meet an SLO over the course
of the semester?
• Requires baseline data – where
were students on this SLO in week
1? Where are they in week 15?
• Proves success even if SLO is still
not met
Yardstick:
learning outcomes

5. More effectiveness questions
Questions about non-SLO
goals
For example: Did this project . . .
• Increase student
engagement?
• Increase student motivation?
• Facilitate deeper learning?
• Improve classroom climate?
• Create a classroom that
supports productive failure
Yardstick: Your goals
(need to be articulated,
but does not have to be
an official SLO)

6. A Taxonomy of Questions
2. Inquiry Into Student Learning
Goal: To better understand
• learning processes,
• classroom social dynamics,
• student errors or misconceptions,
• the roles played by various factors in
student learning, etc.
• Descriptive / documentarian
• Less oriented around practical questions
of a method, more about fundamental
learning issues and processes at work in
the course / experience.
“What Is
(Happening)?” Q –
in Pat Hutchings
taxonomy of questions,
Opening Lines (2000)
SoTL Q
• Common in Scholarship
of Teaching & Learning
inquiry
• Less similar to
assessment questions.

7. Inquiry into Student Learning
Questions
• What thinking processes do
students engage in when
learning about X?
• What are students’ beliefs about
how they best learn Discipline X?
• How do practices of self-
reflection change their learning
about X?
• What do students find most
difficult about learning in Course
X?
• How do students prior
understanding of Discipline X
affect their ability to acquire
new understanding in that field?
• How does their attitude towards
Y affect their ability to do work
on issue X?
“Here the effort is aimed not so
much at proving (or disproving) the
effectiveness of a particular
approach or intervention but at
describing what it [learning] looks
like, what its constituent features
might be.” – Pat Hutchings

8. A Taxonomy of Questions
3. How Are Things Going?
Goal: To better understand
• How students are experiencing the
course
• How students are doing – before they
submit a major assignment
• Student preconceptions, misconceptsion
• Student study and work habits /
processes
May be:
• Anonymous
• Ungraded
• Diagnostic
Designed to enable
you to intervene
quickly before
semester / unit /
project – even
class! -- is over

9. How Are Things Going Questions
Simple / quick tools:
• Misconception / Preconception
check
• minute paper
• Muddiest point (minute) paper
• Classroom opinion polls
• Course-related self-confidence
surveys
• Productive study-time logs
• Annotated portfolios
• Documented problem solutions
• mid-semester evaluation
(These can also be evidence
towards answering your other
questions)
See Angelo & Cross, Classroom
Assessment Techniques

10. What’s Your Problem, Anyway?
“One telling measure of how differently teaching is regarded from traditional
scholarship or research within the academy is what a difference it makes to have a
“problem” in one versus the other.
In scholarship and research, having a “problem” is at the heart of the investigation
process; it is the compound of the generative questions around which all creative
and productive activity revolves.
But in one’s teaching, a “problem” is something you don’t want to have, and if you
have one, you probably want to fix it. Asking a colleague about a problem in his or
her research is an invitation; asking about a problem in one’s teaching would
probably seem like an accusation.
Changing the status of the problem in teaching from terminal remediation to
ongoing investigation is precisely what the movement for a scholarship of
teaching is all about.”
Randy Bass, “The Scholarship of Teaching: What’s the Problem?” Inventio:
Creative Thinking about Learning and Teaching. (1999)

11. Activity: Draft Your Question(s)
By Yourself -- 10 minutes
1. Brainstorm a list of
questions you wish to
answer at end of project
2. Narrow down list to the top
1 or 2
3. Revise the questions to
1. make them more specific
2. Make them measurable
With your peer -- 5 min each
1. Explain your top 1 or 2
questions.
2. Peer’s job: Help peer revise
questions.
Pairs:
• Alex & Yuliya
• Gary & Jason
• Richard & Mary
• Kate & Jimmy
• Chris & Rachael

12. Answering Your Question with Evidence
Evidence . . .
AKA
• Data
• Artifacts
• Student
work
Some things to consider:
• Direct vs. indirect evidence
• Qualitative, quantitative, or both?
• Embedded (“normal educational
practice”) vs. add-on
• Product vs. process

13. Two Examples
From the “How to Start”
worksheet in the
Vanderbilt SoTL Guide
For literature class, in which a
“difficulty log” has been
introduced, to be completed as
students read a text.
• Analyze the logs, looking for
themes or patterns in
responses. From the logs,
document specific types of
difficulty.
From calculus class, in which
students have been asked to
document their problem solving
steps as a way of helping them
develop metacognitive skills.
• Looking at scores on test
prior to and after the new
activity.

14. Activity: What’s Your Evidence?
By Yourself -- 5 min
Brainstorm the types of
evidence you will need to
collect to answer your
question?
Evidence should fit:
• Your questions
• Your discipline
• Your course
• Your timeline
With your peer -- 5 min each
1. Explain your evidence
choices.
2. Peer’s job: Help peer refine
plan.
Pairs:
• Alex & Yuliya
• Gary & Jason
• Richard & Mary
• Kate & Jimmy
• Chris & Rachael

15. Blog Post 5
Post to your blog your question and the
evidence you plan to collect to answer that
question. Discuss your choices.
Due: Fri, May 29