1. QUALITY MATTERS
JOSEPH WONG | Brad Hughes
Designing Quality Online Professional Development:
A Mixed Methods Review and Analysis of Efforts and Outcomes
DDDD May 21, 2020ESCAPE PROJECT | NSF GRANT#
1321343
3. About ESCAPE
ESCAPE | QUALITY MATTERS
3page
Learn Now
EQUITABLE
SCIENCE
CURRICULUM
INTEGRATING
ARTS IN
PUBLIC
EDUCATION
ESCAPE is a professional development program funded by the
National Science Foundation and developed in collaboration by
UC Irvine, OCDE, and the Segerstrom Center for the Arts.
4. Large-scale 5 year study funded by NSF Exploring the
potential impact of arts-integrated science instruction on
the science content knowledge and English language
development of English language learners.
ESCAPE
GOALS
Employing inquiry, creative movement, and visual
arts, the goal of ESCAPE is to help children:
• better understand scientific concepts
• reverse scientific misconceptions
• reduce cognitive load through the educational
techniques of embodied cognition
www.escape.uci.edu
4page
5. HOW HAS YOUR ONLINE
LEARNING EXPERIENCE BEEN
THIS QUARTER?
ESCAPE | QUALITY MATTERS
Let’s Talk
6. BIG IDEAS
ESCAPE | QUALITY MATTERS
6page
Extensive. Effective. Scalable.
SUSTAINABLE PD DESIGN
Self-paced. Metacognition. Transfer.
RESEARCH ON ONLINE
LEARNING
Situation Cognition Theory. Instructional Design.
E-LEARNING COURSE DESIGN
7. SUSTAINABLE PD
DESIGN.
ESCAPE | QUALITY MATTERS
7page
• PD activities that engage teachers in learning
over extended periods of time can be effective
in improving instruction and student
achievement
• Teachers are less likely to enroll in such long-
term intensive PD programs citing issues
related to balancing the demands of work and
family, and travel time
• In-person PD require large amounts of
funding, time away from school, and human
resources, which lead to issues of scalability
and sustainability
• Most PDs have an end date with no ongoing
support or feasibility to practice
Teachers participating in a 40 hour intensive PD
training prior to fall school launch.
(Boyle, While, & Boyle, 2004; Goldenberg & Gallimore, 1991; Guskey 2002),
(Zhang et al., 2008), (Alqarni, 2015; Marrongelle, Sztajn, & Smith, 2013).
8. ONLINE LEARNING
ESCAPE | QUALITY MATTERS
8page
Take the course on any connected
device, at any location, and on your
own schedule.
CONVENIENC
E.
Access lesson plans, PD videos,
and interactive practice simulations
at any time.
ACCESSIBILI
TY.
SCALIBILITY.
Ability to reach larger populations of
teachers and students across the
U.S.
“I dozed off from exhaustion
midway through my module,
rewinded, woke up at 3 AM local
Hawaii time and was able to
complete the module by breakfast
to catch a morning wave and surf
it.”
~Online Elementary
Teacher
9. 9page
RESEARCH ON ONLINE
LEARNING.
ESCAPE | QUALITY MATTERS
TRANSFER.METACOGNITION.
• Self-paced courses, also called learner-
paced courses, enable students to start
any time and work through the course
at their own speed
• When looking specifically at adults, the
benefits of self-paced courses include:
• learner independence
• individualized instruction
• practice
• increased test preparation
(Morris, Surber, & Bijou, 1978; Holmberg, 1989)
SELF-PACED.
• When learners practice pacing and
actively engage in their own productive
metacognitive monitoring, students are
more likely to succeed in self-paced
online learning environments
• Metacognitive monitoring through:
• control of study-allocation time
• personal responsibility
• self-agency
• ability to restudy
(Benjamin and Bird, 2006, Toppino et al., 2009).
• Transfer is the ability for teachers and students to use
prior knowledge to participate in an activity in one
situation and have that positively influence one’s
ability to participate in an activity in a different situation
fostering learning in varying contexts
• Transfer through expansive framing:
• setting expectations
• creating explicit links
• learner accountability
• prior knowledge
• promoting authorship
• Online courses develop practical skills that lead to the
application of classroom knowledge to real-world
settings.
(Greeno, Smith, & Moore, 1993; Engle, et al 2012; Hansen, D. E. (2008).
10. INSTRUCTIONAL
MODEL.
ESCAPE | QUALITY MATTERS
is centered around the idea that knowing is
“inseparable” from actually doing and highlights
the importance of learning within context.
SITUATED COGNITION
THEORY
10page
(Brown, Collins, & Duguid, 1989)
Learners must grasp the
concepts and skills that are
being taught in the context in
which they will eventually be
utilized.
Context.
Interactive branching
scenarios and simulations of
real life settings.
Immersion.
11. Cognitive Apprenticeship Model.
ESCAPE | QUALITY MATTERS
Modeling involves an expert
demonstrating a concept or task so
that learners are able to gain in
depth understanding.
MODELING
Coaching occurs when a subject
matter expert allows learners to
perform the task in question and
then offers them feedback on their
performance.
COACHING
Scaffolding refers to the act of
implementing strategies and
methods that serve to enhance
learners’ educational experience.
SCAFFOLDIN
G
1 2 3
(Collins, Brown, & Holum, 1991)
12. 12page
Articulation occurs when the
instructor encourages learners to
articulate the problem solving
process of the eLearning content or
concepts.
ARTICULATIO
N Reflection enables learners to
compare their problem solving skills
to those of an expert or peer.
REFLECTION
Exploration takes place when
learners are encouraged to solve
problems on their own.
EXPLORATIO
N
4 5 6
Cognitive Apprenticeship Model.
ESCAPE | QUALITY MATTERS
(Collins, Brown, & Holum, 1991)
13. INSTRUCTIONAL DESIGN.
ESCAPE | QUALITY MATTERS
13page
has made it possible to design online PD courses that
promote rich learning opportunities for participating
teachers (Vrasidas & Zembylas, 2004).
TECHNOLOGY
DESIGNERS
must ensure that the resulting courses consider barriers to
learning, such as the quality of the learning experiences
and the accessibility of the technology (Meyen, & Yang,
2005).
ONLINE LEARNING ENVIRONMENTS
must be accessible, easy to navigate, and actually
promote mastery of the learning goals for the PD program
(Marrongelle et al., 2013).
14. MIXED
METHOD
APPROACH.
ESCAPE | QUALITY MATTERS
Did the online course achieve Quality Matters Certification as
perceived by teachers?
RQ 1
What parts of the course were well received and which aspects need
future attention for ongoing improvement?
RQ 2
Did the instructional design frameworks adequately support teachers’
online learning experience?
RQ 3
14page
This study explores the efforts required to convert
a large-scale PD program into a high
quality, scalable and sustainable online course,
utilizing the Quality Matters CPE Rubric, 2nd
Edition, to guide the design efforts.
OBJECTIVE.
Let’s find out!
15. PARTICIPANTS.
ESCAPE | QUALITY MATTERS
15page
Title I Elementary schools in the Los
Angeles Unified School District
55 Teachers. 26 Schools.
African American/Black 4%
Asian/Pacific Islander 11%
Caucasian/White 30%
Hispanic 50%
Other 6%
1-2 Days 19%
3-4 Days 17%
5-6 Days 31%
7 days or more 15%
Formal Science Training (Last 3 Years)
17. QUALITY MATTERS.
Instrument.
ESCAPE | QUALITY MATTERS
18page
The overall design of the
course is made clear to the
learner at the beginning of the
course.
STANDARD 1
Learning objectives or
competencies describe what
learners will be able to do upon
completion of the course.
STANDARD 2
Assessment strategies are integral to the
learning process and are designed to
evaluate learner progress in
achieving the stated learning objectives
or mastering the competencies.
STANDARD 3
Instructional materials enable
learners to achieve stated
learning objectives or
competencies.
STANDARD 4
QM Continuing and Professional Education Rubric, Second Edition, 2015
43 item. 3 point scale. 100 points total.
QM scores of 85% or greater qualify for QM Certification
18. QUALITY MATTERS.
Instrument.
ESCAPE | QUALITY MATTERS
19page
Course activities facilitate and
support learner interaction and
engagement.
STANDARD 5
Course technologies support
learners’ achievement of
course objectives or
competencies.
STANDARD 6
The course facilitates learner
access to support services
essential to learner success.
STANDARD 7
The course design reflects a
commitment to accessibility
and usability for all learners.
STANDARD 8
QM Continuing and Professional Education Rubric, Second Edition, 2015
QM scores of 85% or greater qualify for QM Certification
43 item. 3 point scale. 100 points total.
19. RESULTS.
ESCAPE | QUALITY MATTERS
20page
Overall, the average total score of 90.1% equated to
the course reaching Quality Matters Certification during
the first semester of implementation.
Did we meet compliance?
MEANS ANALYSIS
Results suggest there was adequate variance from the teachers
responses showing differing views of each standard being met, the
means for the standards were high, and standard deviations were
respectable.
N
Alpha
Reliability Min Max Mean SD
Total Points
Possible
Percent
Score
Standard 1 55 .82 6.00 16.00 14.8000 2.05 16 92.5%
Standard 2 56 .94 5.00 15.00 14.0714 2.02 15 93.8%
Standard 3 56 .85 5.00 13.00 12.1250 1.84 13 93.3%
Standard 4 56 .78 5.00 14.00 13.1964 1.60 14 94.3%
Standard 5 56 .65 6.00 11.00 10.4286 1.08 11 94.8%
Standard 6 56 .78 6.00 9.00 8.4821 .83 10 84.8%
Standard 7 56 .76 3.00 9.00 7.9464 1.61 9 88.3%
Standard 8 56 .85 2.00 12.00 9.9107 2.64 12 82.6%
QM
Standards
Combined
Total
55 .91 41.00 99.00 90.8909 11.33 100 90.9%*
Quality Matters Means Analysis
Note: QM scores 85% or higher qualify the course for QM Certification Level. Standards from
the Quality Matters Continuing and Professional Education Rubric, 2nd Edition.
https://www.qualitymatters.org/sites/default/files/PDFs/StandardsfromtheQMContinuingandProfessionalEducationRubri
c.pdf
20. QM SCORES. (> 90%).
ESCAPE | QUALITY MATTERS
STANDARD 1
STANDARD 4
STANDARD 5
94.3%GOALS &
COMPETENCIES.
94.8%INTERACTIVITY.92.5%FINDABILITY.
21page
The overall design of the
course is made clear to the
learner at the beginning of the
course.
Participants believed the instructional materials
enabled learners to achieve the stated learning
objectives or competencies.
The course activities facilitated and
supported learner interaction and
engagement.
21. QM SCORES. (<85%).
ESCAPE | QUALITY MATTERS
The course technologies’ ability to
support the learners’ achievement
of course objectives
STANDARD 6
The course design reflecting a
commitment to accessibility and
usability for all learners
STANDARD 8
22page
84.8% 82.6%
“In addition, because of the laptop that I
was using, I could not save Experience
Log notes, and come back to them.”
“More direct links to individual module
content on the homepage, maybe a
separate homepage for each grade? I
would love to be able to track my
progress during the student perspective
storylines, similar to the teacher
perspectives courses. ”
“The only part that was not perfect was
the connection between the completion of
work and the grades. I'm sure that had
something to do with some technology
aspect of the program.”
ROOM FOR IMPROVEMENT.
23. CODEBOOK SNAPSHOT.
ESCAPE | QUALITY MATTERS
Subcode: Simulation. Design. Communication.
• “The strengths are the videos and the simulated experiments.”
• “Interacting with other teachers learning the lessons and
discussing difficulties and challenges online.”
INTERACTIVITY
SELF-PACED
Subcode: Speed. Flexibility. Schedule.
• “Being home and having the choice of the free time available to
work on any assignments”
• “The ability to work at your own pace and not be restricted to a
specific class schedule.”
MODALITY
Subcode: Translate. Repeat.
• “I like how they approach the activities from different perspectives.
The hands-on activities will translate well in the classroom.”
• “The online course allows for the learners to acquire the lessons
when they are available and the multiple avenues to share the
materials were easily accessible.”
24. SUMMARY.
ESCAPE | QUALITY MATTERS
Did the online course achieve Quality Matters Certification as
perceived by teachers?
RQ 1
Overall, the average total score of 90.1% equated to the course reaching Quality Matters Certification during the first
semester of implementation.
YES.
25. SUMMARY.
ESCAPE | QUALITY MATTERS
What parts of the course were well received and which aspects need
future attention for ongoing improvement?
• Flexibility to participate in the course anytime or anywhere as a major strength of the course
• Teachers expressed their appreciation for the sense of community that emerged, including the numerous
opportunities to interact with their peers
• If teachers re-opened an item that they had previously completed, their scores would revert to zero
• Teachers expressed frustration that the automatic grading of in-module content completion was not working properly
THE GOOD & BAD.
RQ 2
26. SUMMARY.
ESCAPE | QUALITY MATTERS
Did the instructional design frameworks adequately support teachers’
online learning experience?
• Teachers expressed their appreciation for the sense of community that emerged, including the numerous
opportunities to interact with their peers
• Teachers referencing transferable skills from the online professional development to their classroom
• Teachers referencing how they enjoy watching videos of actual classrooms and how the students react in a real
classroom setting
• Experiencing how the kids go through the activity first gave teachers better understanding on how to teach it
YES.
RQ 3
27. IMPLICATIONS.
ESCAPE | QUALITY MATTERS
28page
• Designers of online PD must ensure that the
resulting courses consider barriers to online
learning such as accessibility of the technology
• Teachers referencing modality may be an indicator
that transfer is taking place using the the situated
cognition learning model
• Recognizing transfer as a part of the design
process may help ensure that online PD programs
are a rigorous and effective platform for preparing
teachers
• Designing online courses grounded in effective
instructional design frameworks may further the
promise of online courses that are more flexible,
self-paced, scalable, and wide-reaching
(Meyen, & Yang, 2005; Runyon, 2006; Swan, Matthews, & Bogle, 2010).
28. FUTURE WORK.
ESCAPE | QUALITY MATTERS
29page
Based on this feedback, the research team designed a
second iteration of the course that did not utilize
automatic grading, thus using both the QM scores in
conjunction with qualitative feedback to inform future
course design.
ITERATIVE DESIGN
PROCESS.
RESEARCH.
Students using gestures and embodied
representation to learn science vocabulary.
• Whether QM rating on online design predict teacher
self efficacy and perceptions on VAPA & Inquiry
Methods
• Whether QM rating affects teacher implementation
fidelity in the classroom
using arts strategies to teach science to third for and 5th grade students in title 1 schools, predomininantly high LatinX areas with many ELLs/ bilingual learners
and the curriculum integrates guided inquiry, such as the 5E instructional model, creative movement such as dance and kinesthetic movement (axial vs locomoter) creating skits, and visuals arts such as coloring, painting
ask audience for their opinions.
if no one responds, talk about yourself.
well i appreciate everyone sharing,
and i hope that this presentation offers a different perspective or point of view of online learning
Similar to the face-to-face PD, teachers first experienced each lesson from the students’ perspective before
exploring pedagogical strategies from the teacher perspective, which research shows supports
teachers in feeling more comfortable and prepared to implement curriculum in their own
classrooms (Abd-El- Khalick, 2013; Gillies and Nichols, 2014; Yoon, Joung, and Kim, 2012).
In order to operationalize situation cognition theory, we use the cognitive apprenticeship model. There are other models such as successive approximation model (SAM) MODEL which first collects data on students and branches them to different course based on prior knowledge. There is also the ADDIE MODEL: analyze design, develop, implement, evaluate which is more project based. But in this case, we sued the Cognitive apprecticeship model
So how do we do we put this model into practice and create an effective online course? CAREFUL INSTRUCTIONAL DESIGN : visual hierarchy, gestalt design guidelines (have distinct lines and putting the most important information below it) , hicks law of usability to give distinct chocies to navigate the user to the target area.
1) We wanted to determine if the course we designed met compliance
2) We waned to see what the teachers thought about this course?
3) We wanted to engage qualitatively to see if our we designed the online instructional frameworks with a certain level of fidelity.
Similar to the face-to-face PD, teachers first experienced each lesson from the students’ perspective before
exploring pedagogical strategies from the teacher perspective, which research shows supports
teachers in feeling more comfortable and prepared to implement curriculum in their own
classrooms (Abd-El- Khalick, 2013; Gillies and Nichols, 2014; Yoon, Joung, and Kim, 2012).
In order for use to measure if our course met quality certification, we used a research-based standards to evaluate and inform the quality of online course design
The QM Higher Education Rubric was stemmed from the Improvement of Postsecondary Education (FIPSE) grant from the U.S. Department of Education as a tool for designing online courses and to promote continuous improvement in online course development.
The goal was to score 85% or higher in the survey that was distributed to all the the teachers. It contained 43 items within 8 constructs, or in this case standards. These 8 standards are….
The standards themselves have been based on best practices promulgated by accrediting
bodies (Legon, 2006) and on research literature identifying factors that improve student learning
and retention rates, as well as activities that increase learning and engagemen.
It is grounded in research practice and it allows for designers to deliver on our promise of effective online instruction.
But what does meeting compliance actually tell us? Let’s find out.
DESIGN. LEARNING OBJECTIVES. INTERACTIVITY.
THE THREE STANDARDS FOCUSED ON DESIGN, LEARNING OBJECTIVES, AND INTERACTIVITY SORED THE HIGHEST IN THE QM RUBRIC.
But while we did score high on those standard, we still needed improvement on the course technology, accessibility, and usability. Some excerpts were taken from the online survey….THESE TWO WERE THE LOWEST SCORES
When looking at the survey data from the online course, the three themes/ analytical categories that emerged were on the topics of self-paced learning, course design, and transfer. As you can see in the chart, after running the codes in dedoose, we can see the approximate the frequency of the subcodes and anchor codes categorized into these major themes.
to look a little further into the qualitative process, a word cloud is shown using the teacher feedback about the online course. Also here is a snapshot of the codebook and three examples of the anchor codes with their associated subcodes.
This tells us a lot, that the course that we designed effectively does what it is supposed to do in an aesthethetic, theory driven and designed way.
We also find through the teacher feedback that having the course be self-paced and flexible on teacher schedules is a major strength. Teachers also expressed the interaction and presence such that a community emerged.
But not everything was perfect, the automatic grading became a huge issue. Teachers really care about grades and grading and tracking their progress.
One of the hardest things, but also one of the most important things is to develop presence in an online course. Presence can be in the form of clear goals and competencies, but also in instructor and student social presence. Feeling like they can rely on each other and practice together. We did accomplish this through digital media interactivity as well as fostering discussions, video camera uploads of their work, and online group projects. Additionally, teachers mentioning how they enjoyed going through the student perspective first to see what it is like to learn using dance and arts to teach science. Then learning how to facilitate lessons and carry out the different scenarios.
The vice provest of education at uci was recommending teachers film weekly checkins
Online course offerings continue to surge, yet design issues often short-circuit the goals of students and instructors alike
Good online course design may lead to successful online instruction.
Poorly designed online instruction
Prototyping, testing, analyzing, and refining the product
s
Previous research int he literature has found the navigability and findability, meandering through the course has been the single predictor of teacher self-efficacy and motivation
