The document summarizes a faculty assessment workshop that provides guidance on conducting course assessments using backwards design principles. It recommends that faculty [1] list all course learning outcomes and assessments, [2] rank assessments by perceived and actual difficulty based on grade data, [3] examine differences between perceptions and evidence, and [4] use assessment results to strengthen areas where students struggle most. The workshop also covers formative and summative assessment, psychometrics, using multiple measures, and aligning outcomes, content and activities through backwards design mapping.

1. Faculty Assessment Workshop
David Wiley
Kim Thanos
Ronda Neugebauer

2. Course Assessment Listing
• Create a new Google Doc/Word Doc
• List all course learning outcomes
• List all course assessments - anything graded
• Be as brief as you like - this working document
is only for you
2

3. Estimated Difficulty of Assessments
• Copy assessment listing and paste above
original list
• Without looking at the grade book, rank
assessments:
– Top = assessments you think students struggle
with the most
– Bottom = assessments you think students struggle
with the least
3

4. Grade Book Results
• Copy assessment listing and paste above
estimated difficulty of assessments list
• Access grade book
• Rank assessments:
– Top = assessments with grades indicating students
struggled with the most
– Bottom = assessments with grades students
struggled with the least
4

5. Assessment List Comparison
• Compare the two lists you ranked
• Are there any differences in what you
perceived versus what the grades suggest?
• Does anything surprise you about the
differences?
5

6. Perception vs. Evidence
• How does our perception of what was most
difficult compare to the grades indicating
what was most difficult?
• Attending to the data helps us to develop
meaningful feedback to improve our courses
• The evidence is often quite different from
what we perceive
6

7. Psychometrics
• We can’t open our students’ brains to see
what they know, but we can give them
opportunities to tell us what they know
• Assessment results yield both signal and noise
• Rather than give one big assessment, use
multiple measures to tease out the signal
from the noise
7

8. The Pixel Analogy
• Why are multiple measures of assessment
important?
• http://www.pcworld.com/article/2027906/chrom
ebook-pixel-the-google-pc-weve-been-waiting-
for-.html
• Think of multiple measures of assessment like
the pixels that make up the entire digital picture.
Aim to see the whole picture of the student,
instead of just a few pixels.
8

9. Multiple Measures of Assessment
Formative Examples
Assists in modifying teaching and • Muddiest Point
learning activities for student
success • Minute Paper
• Concept Test
Typically involves qualitative • Directed Paraphrase
feedback (rather than scores) for
both students and teacher • RSQC2 – Recall, Summarize,
focusing on details of content and Question, Comment, Connect
performance • Transfer and apply
• Anecdotal Records
Assessment FOR learning http://www.venturacollege.edu/assets/pdf/president_office/types_of_me
http://en.wikipedia.org/wiki/Formative_assessment asurements.pdf
9

10. Multiple Measures of Assessment
Summative Examples
Refers to evaluation of learning • Unit Exams
and summarizes development of • Portfolios
learners after a period of time. • Discussions
• Case Studies
Seeks to monitor educational • Reflective Essay
outcomes often for purposes of
accountability
• Standardize Test
• Exit Interview
Assessment OF learning
• Capstone Project
http://www.venturacollege.edu/assets/pdf/president_office/types
http://en.wikipedia.org/wiki/Summative_assessment _of_measurements.pdf
10

11. Backwards Design Application
Start with Outcomes
– Identify Course Learning Outcomes (≈3-5)
– Identify Module Learning Outcomes (≈15-40)
Connect content & activities
– Identify student learning activities (≈1-3 per MLO)
11

12. Backwards Design Application
Course Learning Outcomes
CLO
what students know by the end of the course
MLO
Module Learning Outcomes
what students know by the end of the module
LA Learning Activities
opportunities for students to demonstrate knowledge
12

13. Backwards Design Application
Align Course Learning Outcomes, Module
Learning Outcomes, and Learning Activities
MLO LA MLO LA
MLO LA CLO
CLO
LA
LA LA LA MLO MLO
MLO
LA CLO MLO
MLO MLO LA
13

14. Backwards Design Application
Mapping is a quality issue & assists in peer review. If we
cannot connect, there is no way to improve the course.
MLO LA LA LA
CLO MLO LA This area needs work.
MLO LA LA LA
14

15. Backwards Design Application
• Back in doc, copy and paste outcomes to assessments
• Look for imbalances with outcomes that aren’t met
based on grade evidence
• Choose 4-5 outcomes that students struggle with most
• Strengthen those areas of the course instead of
improving the whole course
• Use Advanced Google Search to identify OER content
15

16. Backwards Design Application
• Which areas of the course need improving?
• How do you know?
• What might you do to achieve this?
Using data to drive decision-making helps improve
the course overtime
Empirically, you know where students are struggling
and how best to improve the course
16

17. Instructure’s
Demo
– Speedgrader
– Communication Settings
– Orientation Module with Canvas Videos
– Analytics: Class, Individual Student
– Import Content
Access your course invitation via your email
David’s Building Open Course Frameworks
https://lumen.instructure.com/courses/33165
17