Clark, T., Barbour, M. K., & Debruler, K. (2021, June). Using formative evaluation to keep your program on track [Panel]. Digital Learning Annual Conference, Austin, TX.

1. Using Formative Evaluation to
Keep Your Program On Track
Tom Clark, Clark Consulting
Kristen DeBruler, Michigan Virtual
Michael Barbour, Touro University California
DLAC [Online] 6/16/2021

2. Part of DLAC’s
Session Learning Objectives:
● Better understand how formative evaluation is
commonly used in school improvement
● Better understand how it can help identify best practices
and areas of concern and keep your program on track
● Learn how leaders are using it to improve programs
Using Formative Evaluation to Keep Your Program on Track

3. Why Use Formative Evaluation?
Tom Clark
Using Formative Evaluation to Keep Your Program on Track

4. Using Formative Evaluation to Keep Your Program on Track
Why use formative evaluation?
● Formative evaluation is commonly used in online
school improvement. For example:
○ Tracking student progress in the LMS
○ Applying quality indicators to course reviews
● But it can do more - finding best practices, addressing
problem areas, tracking program growth over time.

5. Using Formative Evaluation Tools to
Identify Best Practices
Kristen DeBruler
Using Formative Evaluation to Keep Your Program on Track

6. LEAD. COLLABORATE. BUILD. | MICHIGANVIRTUAL.ORG
Key Strategies for Engaging Students
in Virtual Learning Environments
Report published on April 6, 2021
Harrington, C. & DeBruler, K. (2021). Key strategies for engaging students in virtual learning environments. Michigan Virtual
University. https://michiganvirtual.org/research/publications/key-strategies-for-engaging-students-in-virtual-
learning-environments/

7. MICHIGANVIRTUAL.ORG | LEAD. COLLABORATE. BUILD.
Research Questions:
1. What strategies do teachers of virtual
courses employ to engage students?
2. What student engagement strategies do
teachers of virtual courses believe to be
most effective?
3. How do teachers of virtual courses
develop relationships that nurture
student engagement?
Key Strategies for
Engaging Students
in Virtual Learning
Environments
(National)

8. MICHIGANVIRTUAL.ORG | LEAD. COLLABORATE. BUILD.
Methodology:
● Survey of 1,809 educators
(1,721 teachers; 88 supervising administrators)
● 17 statewide virtual schools/programs
○ Well-established programs
○ Dedicated focus on virtual learning
Key Strategies for
Engaging Students
in Virtual Learning
Environments
(National)

9. LEAD. COLLABORATE. BUILD. | MICHIGANVIRTUAL.ORG
Key Strategies for Engaging Students
in Virtual Learning Environments
Findings:
● Experienced teachers of statewide virtual schools or programs use a variety of
specific strategies to engage students learning in a virtual environment.

10. MICHIGANVIRTUAL.ORG | LEAD. COLLABORATE. BUILD.
Full-time Teacher Part-time Teacher
Total Count 393.0 1,328.0
I use multiple formats of content such as text-based articles, video, audio, etc. 79.4% 73.2%
I make myself available to students through scheduled office hours so they can connect or
“drop in” as needed
77.1% 74.0%
I post motivational/relational announcements in my course(s) 75.3% 77.1%
I include video/audio recordings of myself in my course(s) 71.2% 60.6%
I include activities in my course(s) such as discussion forums, journal entries, and/or
reflections
70.5% 70.2%
I connect one-on-one with students by telephone or video conferencing 66.4% 52.9%
I provide personalized remediation options for students in need 66.2% 61.9%
I provide frequent opportunities for formative assessments 65.9% 53.3%
I include interactive activities in my course(s) such as polls, quizzes, and/or games 62.1% 46.7%
I interact synchronously (in “real time”) with groups of students through audio or video
conferencing
56.2% 28.0%
I provide personalized enrichment options for students in need 46.8% 34.6%
I have students collaborate or work together on projects or activities 29.0% 25.3%
Student Engagement Strategies Used by Full-time and Part-time Virtual Teachers

11. LEAD. COLLABORATE. BUILD. | MICHIGANVIRTUAL.ORG
Key Strategies for Engaging Students
in Virtual Learning Environments
Findings:
● Experienced teachers of statewide virtual schools or programs use a variety of
specific strategies to engage students learning in a virtual environment.
● Teachers of synchronous virtual courses (48 teachers) were more likely to use
four specific engagement strategies:
○ providing frequent opportunities for formative assessments,
○ including interactive activities in courses such as polls, quizzes, and/or games,
○ interacting synchronously (in “real time”) with groups of students through audio
or video conferencing, and
○ having students collaborate or work together on projects or activities.

12. LEAD. COLLABORATE. BUILD. | MICHIGANVIRTUAL.ORG
Key Strategies for Engaging Students
in Virtual Learning Environments
Findings:
● Experienced teachers of statewide virtual schools or programs use a variety of
specific strategies to engage students learning in a virtual environment.
● Teachers of synchronous virtual courses (48 teachers) were more likely to use
four specific engagement strategies
○ providing frequent opportunities for formative assessments,
○ including interactive activities in courses such as polls, quizzes, and/or games,
○ interacting synchronously (in “real time”) with groups of students through audio
or video conferencing,
○ having students collaborate or work together on projects or activities).
● Specific teacher actions to connect and develop relationships with students help
to increase student engagement in virtual learning environments.

13. LEAD. COLLABORATE. BUILD. | MICHIGANVIRTUAL.ORG
Key Strategies for Engaging Students
in Virtual Learning Environments
Findings:
Relationships
● Communication
“I find that when I have been able to connect with a student through various forms of
communication (email, text, one-on-one conferencing), our relationship becomes more
personal, and students are more successful.”

14. LEAD. COLLABORATE. BUILD. | MICHIGANVIRTUAL.ORG
Key Strategies for Engaging Students
in Virtual Learning Environments
Findings:
Relationships
● Communication
● Feedback
“I address them by name in feedback; I remember things about them (such as their
nickname, favorite activities, etc.) and maintain positivity during live sessions and calls.
I include relevant details about myself and my family in discussions so they also feel
like they can get to know me as well.”

15. LEAD. COLLABORATE. BUILD. | MICHIGANVIRTUAL.ORG
Key Strategies for Engaging Students
in Virtual Learning Environments
Findings:
Relationships
● Communication
● Feedback
● Appealing to students’ interests
“I speak to students on the phone and I try to find a common ground to relate to them
other than the course such as their sport, activity, or interest. I email students and
encourage them. I use the reminder app to motivate students. I post congrats on
announcements with first names of students who completed a specific assignment or
who have worked to complete assignments.”

16. LEAD. COLLABORATE. BUILD. | MICHIGANVIRTUAL.ORG
Key Strategies for Engaging Students
in Virtual Learning Environments
Findings:
Relationships
● Humanizing
“Having actual face-time with my students. Cameras on when we are one-to-one at
least for a while. Being real and human in our live sessions and showing them that I am
a person, too. Specific feedback given on assignments.”

17. LEAD. COLLABORATE. BUILD. | MICHIGANVIRTUAL.ORG
Key Strategies for Engaging Students
in Virtual Learning Environments
Findings:
Relationships
● Humanizing
● Synchronous meetings
“Teaching kindergarten online is a unique challenge, and we take advantage of as
much real-time interaction as we can with our students.”

18. LEAD. COLLABORATE. BUILD. | MICHIGANVIRTUAL.ORG
Key Strategies for Engaging Students
in Virtual Learning Environments
Findings:
Relationships
● Communication
● Feedback
● Appealing to students’ interests
● Humanizing
● Synchronous meetings

19. LEAD. COLLABORATE. BUILD. | MICHIGANVIRTUAL.ORG
Key Strategies for Engaging Students
in Virtual Learning Environments
Implications:
● Committing the time and energy to learn how to implement specific strategies
would enable teachers to develop a strong foundation for ongoing student
engagement in virtual learning environments.
● For educators new to teaching in a virtual environment, some student
engagement strategies may require developing additional skills to implement
effectively.
● Regardless of whether the learning environment is asynchronous or synchronous,
all virtual teachers should strongly consider the implementation of specific
strategies ranked as the most effective in engaging virtual students.
● When students and teachers get to know each other at a deeper level, strong
relationships develop, and students are far more likely to be engaged in their
learning.

20. Using Formative Evaluation to Keep
Your Program on Track
Tom Clark
Using Formative Evaluation to Keep Your Program on Track

21. “You've got to be ready to prepare and
respond quickly. Support teachers in terms
of changing their pedagogy and their
practices. Allocate resources and personnel.”
“I really have an exception to the term "pivot
to be online." Because online is based on a
whole set of experiences and it takes time to
develop that practice. Schools were expected
to just change everything right now. ”
Randy LaBonte [with Michael Barbour]
Insert audio/video link
or play separately
Canadian School Leaders: Developing New Programs During COVID

22. “Schools will emerge with an opportunity to
evaluate how well they were able to implement
emergency remote teaching to maintain continuity
of instruction.”.
It is important to avoid the temptation to equate
emergency remote teaching with online learning
during these evaluations.”
-- Michael K. Barbour, et al (2020)
Understanding Pandemic Pedagogy
Canadian School Leaders: Developing New Programs During COVID

23. Using Formative Evaluation to Keep Your Program on Track
Formative evaluation begins during program planning
● Launching a new online program often requires
extensive planning of the fly
● It’s also the best time to begin formative evaluation
● What data do you need to gather from the beginning to
know if your program is on track?
● How will you gather it and when? How will you use it?
●
Martin (2015). Plan for program evaluation from the start

24. Using Formative Evaluation to Keep Your Program on Track
Monitoring & Evaluation can help
programs define and measure quality
process indicators and measures, gauge
progress toward desired outcomes,
increase stakeholder participation, and
empower school leaders and teachers.
-Tom Clark, 2014 (Microsoft in Ed)
tinyurl.com/QAinformMandE

25. Using Formative Evaluation to Keep Your Program on Track
Outcomes-based planning and evaluation. Planning for
program services/activities that lead towards observable,
intended changes in individuals
● Example: Students actively engage in online learning
○ Teachers use interactive activities
○ Students actively participate in these activities
○ Student work products reflect learning
https://www.imls.gov/grants/outcome-based-evaluations
MVLRI research in review: K-12 online program evaluation, quality, and policy

26. Using Formative Evaluation to Keep Your Program on Track
Design-Based / Developmental Evaluation
Evaluation researchers work with school leaders to develop
and test educational innovations over time that address
problems of interest to both parties
● Example: Riverview School District & local university
create an institutionalized collaborative research space
within a professional development school
https://files.eric.ed.gov/fulltext/EJ1147232.pdf

27. Using Formative Evaluation to Keep Your Program on
Track
Thanks for attending!
Tom Clark tom@tomclarkconsulting.net
Kristen DeBruler kdebruler@michiganvirtual.org
Michael Barbour mkbarbour@gmail.com