EDTECH 505 Final Project Evaluation Report and Analysis

1. FINAL EVALUATION
REPORT
Effectiveness of Virtual Direct Instruction of Unit 6, MJUSH
Submitted to:
Esther Webb, Instructional Leader for Liberal Arts, Young Virtual School
Submitted by:
Kathryn Booth, Graduate Student, Boise State University

2. P a g e 1 | 15
Summary
The MJUSH team at Young Virtual School has been providing supplemental direct instruction
sessions to their students for a unit on Jacksonian Democracy over the past six weeks. The
program consists of three sessions for a total of three hours of supplemental, direct instruction.
The team created three new the sessions due to their general consensus that this unit was creating
an above average challenge to their students resulting in low unit exam scores. The team
implemented virtual, direct instructional session for students to complete as a supplemental
resource to the curriculum. Each session was conducted by a team instructor two times per week
in a live virtual classroom. The goal of the project was to provide a greater understanding of the
unit concepts to students which would then result in improved unit exam scores. The ultimate
goal was to increase student exam scores by five percentage points.
An evaluation of the program was requested by the Instructional Leader to determine if the
program objectives were being met. In addition to gathering data on oral and unit exam scores
prior to the program evaluation start date, pre and post program surveys with participating
instructors regarding program effectiveness were collected and analyzed. Pretest, oral exam, and
unit exam scores for participating students were gathered for students who completed the
program during the evaluation period. Additionally, post-program student surveys were also
collected in order to obtain a snapshot of the student perspective on program effectiveness.
Results of the evaluation reflect agreement amongst instructors the need for supplemental
resources in reaching the objectives. Instructors found the programs live sessions to meet the
current need. Suggestions for further development of resources is described in more detail later
in this report. Evaluation of student surveys found the program, from the student perspective, to
be beneficial. Qualitative data reflects the program objectives were met.
Description of the program evaluated
The supplemental direct instruction sessions on Jacksonian Democracy were created by the
Young Virtual School MJUSH team based on the unit objectives and curriculum standards. The
current curriculum and resources had been underway since July 2013. The Instructional Leader,
Team Lead, and team instructors were concerned with the low unit exam scores for Jacksonian
Democracy. All parties cited are individually evaluated on end-of-course exams which include
content on the program unit. As a team, they self-assessed the need for improved unit
six/Jacksonian Democracy unit scores in order to achieve higher end-of-course exam scores.
With the approval of the Instructional Leader, sessions were created by the team and then
conducted by a team instructor in a live, virtual classroom two days per week from June 1, 2014
through July 11, 2014.
Program Objectives
 Increase student understanding of the unit concepts
 Increase student unit exam scores by five percentage points

3. P a g e 2 | 15
Program Components
Students that were enrolled in the MJUSH course and had completed course content through unit
five were eligible to participate in the program. Three MJUHS Instructors conducted live, direct
instruction session two times per week in the MJUSH virtual classroom via Blackboard
Elluminate. Students joined one of the three program sessions at a time to complete a total of
three hours of supplemental direct instruction on Jacksonian Democracy.
Prior to the program sessions, all students completed a unit pretest in their online course
assessment area. Students were invited to live sessions via the Course Announcement page live
session schedule and an invitation emailed to students and parents on Sunday of each week.
Students selected from the session schedule the day(s) and time they would attend. Each session
included a five question post-quiz which attending students were required to complete. Prior to
exiting the final session, students were required to complete a student survey (Appendix C).
After completing all three sessions, students completed a one-on-one discussion of unit content
with their instruction via the telephone. After the discussion based assessment, students were
assigned the unit exam. Students who participated in all three sessions and completed the unit
exam were included in the program evaluation.
Evaluation Method
Participants
Primary data for the evaluation came from MJUHS active students who had taken the unit six
pretest, completed all three live, direct instruction sessions, a one-on-one content discussion with
their instructor, and completed their unit exam between June 1, 2014 and July 11, 2014. A total
of ninety-three students completed all components of the program during the evaluation period.
No sample was purposely used. Due to the self-paced nature of the course, only data gathered for
those students who completed all components of the program was considered.
Additional data was compiled from student and instructor surveys. All ten instructors for the
MJUSH course participated in the evaluation. Instructors completed pre and post surveys using a
Likert Scale (Appendices A and D, respectively). Three of the ten instructors delivered the live
instructional session during the evaluation period (Appendix B). Students completed surveys at
the end of the third synchronous session (Appendix D).
Variables include but may not be limited to varying student learning levels (i.e. gifted, gen-ed,
ESE), time spent on course content outside of program, time lapse between session completion,
and different instructor instructional delivery techniques.

4. P a g e 3 | 15
Procedures
Surveys were given to all ten instructors prior to beginning the program. Instructors responded to
questions regarding the strength of the curriculum resources and the need for supplemental
resources to meet the program objectives. Instructors also reported in the survey the previous
school year (July 1, 2013 - May 31, 2014) data for their classroom mean unit exam scores
(Appendix A).
Students who entered the unit during the evaluation period completed a pretest before
completing unit assignments and participating in the program sessions. The three live
instructional sessions were offered two times each week during the six week evaluation period.
Students were invited via email and a posted calendar of sessions on their course announcement
page. Students joined the session that best fit their schedule. Each live session included five post
session comprehension questions. The three instructors responsible for delivering the sessions
reported after each session via a Google Spreadsheet the mean of each sessions post quiz scores
(Appendix B).
At the end of the third live session, students completed a Likert scale survey to determine, from
the student perspective, the effectiveness of the live session components and the time lapse
between live session attendances (Appendix C). Following student completion of the pretest, unit
assignments, and all three direct instruction sessions, a one-on-one discussion took place
between each student and their instructor of record. Discussion were focused on the unit content
and instructors used the curriculum grading rubric to determine the student level of content
knowledge comprehension.
A post-program Google form survey using the Likert scale was completed by instructors in
which they reported to the evaluator their classroom mean unit pretest score, discussion-based-
assessment score, and unit exam score. Averages for only those students who completed all
components of the program between June 1, 2014 and July 11, 2014 were reported (Appendix
D). The survey included instructor attitudes regarding the effectiveness of the program.
Data Sources
Both qualitative and quantitative data were used in evaluating the effectiveness of the program.
A Google form survey, using the Likert scale, was sent through email to the ten participating
team instructors prior to program implementation (Appendix A). The surveys collected instructor
feedback on strengths of the curriculum resources and the need for supplemental resources to
reach program objectives. Quantitative data, in the form of the previous school year’s (July 1,
2013 – May 2014) mean unit exam scores, were also reported on the survey. Three instructors
assigned to delivering the synchronous sessions provided mean post-quiz scores for each session
in a Google spreadsheet (Appendix B). Students who completed all three of the programs
synchronous sessions completed a Google form survey (Appendix C) at the end of session three
and before exiting the sessions. The student surveys collected qualitative data regarding the
student perspective on program session effectiveness. The post-program survey completed by

5. P a g e 4 | 15
instructors provided to the evaluator their classroom quantitative data on pretest, discussion
based assessment, and unit exam mean scores (Appendix D).
Results
The program evaluation results are summarized and related to the program objectives. Survey
data compiled from students and their attitude toward the effectiveness of the program
synchronous sessions reflect positive results. All students who completed the program’s three
synchronous sessions completed an attitude survey.
Over half of the participants reported knowing very little about the unit content prior to
beginning the program unit with 76% knowing little to nothing about the unit.
Effectiveness of the program from the student standpoint indicates success with over 90 % of
participating students finding the sessions very beneficial to their understanding of the unit
content. Details of the student survey are elaborated in Appendix C.

6. P a g e 5 | 15
Similarly, 80% of the instructors
reported the program sessions were
beneficial to their students
understanding of the lesson content
and a mean discussion-based
assessment score of 88.6%. This is
an improvement from the pre-
program mean score of 84.2%. This
data indicates the first objective was
met from both the instructor and
student standpoint.
In evaluating the quantitative results collected, both the discussion-based-assessment scores and
the unit exam scores increased. The seven percentage point increase in unit exam scores suggests
objective number two, to “increase student unit exam scores by five percentage points,” has been
met. Appendices A and D illustrate additional data regarding quantitative data.
Discussion
The results of this study are used to address the program objectives of increasing learner
understanding of unit content and increasing student achievement scores on unit exams by five
percentage points. Post survey results and instructor reported discussion-based-assessment scores
were used to evaluate the level of program effectiveness for the former. Student unit exam scores
reported during the evaluation period were used to evaluate the program effectiveness for the
latter.

7. P a g e 6 | 15
Qualitative data collected from students and instructors displays a correlation between student
understanding of unit content and the program synchronous sessions. Quantitative results show
an overall increase of five percentage points for discussion-based-assessment scores with sixty
percent of instructors having improved scores, twenty percent having no change, and twenty
percent having decreased scores. Variables include a different sample group, subjectivity of
instructor when grading assessment with the rubric, and the time lapse between completion of
session and the discussion-based-assessment. Therefore, it can be suggested that program
sessions increased student understanding of the unit content but cannot absolutely attribute the
increase to the sessions entirely.
Data gathered on pre and post exam scores were compared to determine the programs impact on
the second objective – to increase student unit exam scores by five percentage points. Based on
the difference between pre-program unit exam scores and post-program unit exam scores, the
second objective was achieved. Records indicate a seven percentage increase which exceeds the
goal of a five percentage point increase. However, the data employed in the evaluation came
from different sample populations between pre and post program scores. It must be considered
that the seven percentage point increase in unit exam scores was possibly related to variables
outside of the program sessions. Variables to consider but not limited to include mixed student
learning levels (i.e. gifted, gen-ed, ESE), mixed learning styles, time spent on course content
outside of program, time lapse between session completion, and different instructional delivery
techniques among instructors.
Project Costs

8. P a g e 7 | 15
The project cost for the program evaluation on the effectiveness of virtual direct instruction of
Unit 6, MJUSH for Young Virtual School is outlined above. One principal evaluator was used at
the request of the Young Virtual School Liberal Arts Instructional Leader, Esther Webb. The
evaluator twice met with principal stakeholders at Young Virtual School’s administrative offices,
collected, analyzed, and compiled data to produce a formal written report. All documents were
delivered electronically to Ester Webb.

9. P a g e 8 | 15
Appendix A
How would you rate the curriculum resources for Unit 6 in helping your
students comprehend the content?
1 0 0%
2 0 0%
3 1 10%
4 5 50%
5 4 40%
How would you rate the need for supplemental resources for Unit 6 in
helping your students comprehend the content?
1 7 70%
2 3 30%
3 0 0%
4 0 0%
5 0 0%

10. P a g e 9 | 15
What was your classroom Discussion Based Assessment average score
between July 1, 2013 and May 31, 2014?
92 90 82 80 86 84 76
What letter grade was your classroom average on the Unit 6 Discussion
Based Assessment between July 1, 2013 and May 31, 2014?
A+ 0 0%
A 0 0%
A- 3 30%
B+ 1 10%
B 1 10%
B- 4 40%
C+ 1 10%
C 0 0%
C- 0 0%
D+ 0 0%
D 0 0%
D- 0 0%
F(50% or below) 0 0%
What was your classroom unit 6 exam score between July 1, 2013 and
May 31, 2014?
71.2 71.25 67 69 71 73 62 74.1 74.6 50

11. P a g e 10 | 15
What letter grade was your classroom average on the Unit 6 Exam
between July 1, 2013 and May 31, 2014?
A+ 0 0%
A 0 0%
A- 0 0%
B+ 0 0%
B 0 0%
B- 0 0%
C+ 0 0%
C 3 30%
C- 3 30%
D+ 2 20%
D 0 0%
D- 1 10%
F(50% or below) 1 10%
What suggestions or comments do you have regarding student
comprehension Unit 6?
Students don't seem to take the time needed to understand all the content in this unit. They are rushing
through and not too interested in Jacksonian Democracy. I just don't think the students care enough
about doing well on this. They know they can keep their grade up if they do better on the next unit.
There's so much information in the unit that they lose focus. Breaking the unit down into smaller parts
might help. Give them handouts and video help. They don't take the time to do the work themselves. If we
want them to be successful in the time they're willing to spend in the unit then we have to meet them
halfway. Live lessons with some background information and then some follow up questions worded like
the exam questions. Students have a hard time understanding what the questions are asking even when
they know the content. Help needed with test taking strategies, and reading comprehension. Get some
tips from reading coach and add strategies to some live instruction. Students don't come with the
background knowledge needed to understand the unit content well enough. Perhaps creating some

12. P a g e 11 | 15
supplemental lesson handouts, video instructions, and/or doing some live lessons would help. These kids
just don't have the background knowledge to be successful in this unit. There are a lot of different
connections to make in this unit and they can't figure it out. We have to help them by guiding them
through it one step at a time. Some kids do really well but, for most, this unit is just too much. Students
struggle with the amount of complex text in the unit and on the exam. Possibly provide some help to
students with understanding the text and unit content. We should offer some help with understanding the
unit content. There's a lot of challenging text and students have trouble with it. Maybe live sessions and/or
teacher-created video instruction. I think that giving students some one on one instruction or maybe a
handout with a lower reading complexity level would help them understand the content a lot better. They
struggle with the reading and miss the content.
Appendix B
Week 1 Week2 Week 3 Week 4
Average
by
session
(Liverotti)602 91.1 84.3 85.5 94 89.1 83.6 91.1 89.1 88.475
(Perez)604 88 85.5 82.2 90 84.4 90 91.1 86.7 87.2375
(Gaulier)606 82 84.4 90 92 91.1 86 90 87.2 87.8375
Average by week 87.03333 84.73333 85.9 92 88.2 86.53333 90.73333 87.66667
MJUSH Unit 6 Program Live Session Quiz Averages

13. P a g e 12 | 15
Appendix C

14. P a g e 13 | 15
Appendix D

15. P a g e 14 | 15
Student Note: Due to the featured online school’s regulations, publishing of exam questions
in any form is prohibited. For this reason, the evaluation includes only compiled data and
not the exam itself.