15Quantitative Research Study ReportInsert You

1
5
Quantitative Research Study Report
Insert Your Name Here
School of Public Service and Education, Capella University
EDD8040: Research Design for Practitioners
Insert the Instructor’s Name Here
Insert the Due Date Here (Month, Day, Year)
Introduction
Research Theory Framework
1. What were the key concepts of the research framework that
supported the development of the research questions?

2. What are your reflections on the connections between
theoretical or conceptual frameworks and research questions as
they relate to developing an Applied Improvement Project
(AIP)?
Methodology
3. Describe the intervention?
Results
4. From the results, describe the answers to each of the four
research questions and the rationale or supporting evidence for
those answers.
Discussion/Conclusion
5. What ideas, concepts, or processes from this study did you
find interesting or useful and how might those ideas, concepts,
or processes be incorporated into an applied research pr oject.

References
Chen, M., Hwang, G., & Chang, Y. (2019). A reflective
thinking-promoting approach to enhancing graduate students’
flipped learning engagement, participation behaviors, reflective
thinking and project learning outcomes. British Journal of
Educational Technology, 50(5), 2288-2307.https://doi-
org.library.capella.edu/10.1111/bjet.12823
Creswell, J. W., & Creswell, J. D. (2018). Research design:
Qualitative, quantitative, and mixed methods approaches (5th
ed.). Sage.
1
2
Quantitative Research Study Report
Insert your Name Here
School of Public Service and Education, Capella University
EDD8040: Research Design for Practitioners
Insert the Instructor’s Name Here
Insert the Due Date Here (Month, Day, Year)

[Important Writing Instructions]
[This assignment needs be written in the third person voice. Do
not write in the first-person voice (I . . .). There should be none
of you and your voice in this assignment or the course project.
However, for those questions that ask you your opinion or how
something applies to your Applied Improvement Project, you
can answer in the first-person voice. Do not use awkward
language such as The researcher . . . or The learner when
referring to yourself. Do not refer to yourself unless you are
answering those questions that ask you your opinion or how
something applies to your potential Applied Improvement
Project (AIP). Do not write in the second person voice (writing
that uses or implies the language you or your).]
[Always present the study and other literature with past tense
verbs (APA 7th ed. section 4.12 pp. 117-118); for example,
Chen et al. (2019) conducted . . .]
[Scholarly writing is meant to be read and interpreted literally.
Therefore, please avoid slang, colloquialisms,
anthropomorphisms, and conversational writing (refer to APA
7th ed. pp. 113-125). Instead, be clear, precise, and accurate in
your writing.]
[At the doctoral level, most of your writing should involve
summarizing or paraphrasing the literature. However, for an
assignment like this one in which you conduct an in-depth
review and analysis of a single study, there will be instances
when you will likely use direct quotes. For direct quotes with
fewer than 40 words, put quotation marks around the quoted
text and include within the in-text citation, the author’s name,
year, and page or para. number from which the quote came. For
direct quotes with 40 or more words, put in block format (See

APA 7th ed. pp. 272-273 for guidance and examples) and
include within the in-text citation, the author’s name, year, and
page or para. number from which the quote came.]
[There might be instances in which you use a direct quote that
came from the article’s literature review. If the article’s
authors use a quote or cite another author and you want to use
that text as a direct quote, be sure to quote your article’s
authors as the secondary sources (see APA 7th ed. p. 258).]
[Do not write with bullet points. Instead use complete
sentences developed within coherent paragraphs. Use
transitional language to smoothly move the flow of the thought
along throughout.]
[Apply APA formatting rules and adhere to APA writing style
guidelines.]
[Here are two important self-assessment final steps to help
ensure you do as well as you can with the assignment: When
you complete your draft, read it aloud to yourself. This step can
catch typos, grammar errors, awkward writing, etc. For a final
step – self-assess your assignment by reviewing the
corresponding scoring guide and compare the proficient and
distinguished columns criteria to your draft and revise as
necessary.]
Introduction
[Briefly identify the quantitative study by following APA
writing style, which means citing the author’s last name and
year of publication. When identifying and discussing the study
do not include the article’s title in your text as that is not how
APA style writing is done. The title can be found in the
reference citations. Instead follow APA writing style and
include only the author’s last name and the year the article was
published when referring to the study.]
[When describing the study do not write with
anthropomorphisms. An anthropomorphism is when you assign
human capabilities to a study or other non-human entities. Here
is an example of anthropomorphism: The study found . . .
Please note that studies are incapable of taking human action.

Instead write, for example: Jones (2018) found . . . Avoid
writing like this: The authors found . . . or The researchers
found . . . Instead write, for example: Smith and Jones (2019)
found . . . Smith and Jones (2019) explained . . . Note that in
APA writing style you use past tense verbs to present or
describe a study (see APA 7th ed. pp 117-118).]
[Follow APA writing style and identify the authors each time
you refer to the study. Within a paragraph in which you cite the
same author or authors more than once, include the year in the
first citation and omit the year in subsequent citations within
the paragraph (refer to APA 7th ed. section 8.16 pp. 265-266). ]
[Briefly describe the type of quantitative study; that is, what
specific quantitative design was used (e.g., quasi-experimental,
correlational, predictive correlational, comparative, descriptive
survey, etc.). Describe the purpose of the study (what was the
study designed to accomplish?). Identify the research
question(s) and variables. Use direct quotes sparingly; however,
if there is a research question, copy it and provide attribution
via quotation marks and an in-text citation that includes the
author, year and page or para. number. If there is a purpose
statement, you can copy that as well provide attribution via
quotation marks and an in-text citation that includes the author,
year and page number. At the doctoral level your writing
should be comprised primarily of summarizing and
paraphrasing. If you must use a direct quote ensure the quoted
text is in quotation marks followed by an in-text citation that
includes the author’s name, year and page (refer to your APA
manual 7th ed. Ch. 8 pp. 269-278). Conclude this section by
analyzing how the Chen et al. (2019) study represents a
quantitative design.]
Research Theory Framework
[Briefly describe the study’s theoretical or conceptual
framework. What is theoretical or conceptual foundation for the
study? In other words, what is the theory or previous literature
that provides the basis or foundation for the study? Did the
researcher test a hypothesis that stems from a theory or a model

or a body of research or a combination of these components?
What construct or constructs were measured? “A construct is a
hypothetical concept that is typically developed from a
theoretical framework. Constructs are names for things that
cannot be seen (intelligence, motivation, self-esteem)” (Lodico,
Spaulding, & Voegtle, 2010, p. 13). Constructs, concepts, and
variables are derived from theories or models. Describe in
detail the relationship among the theoretical of conceptual
framework and the formation of measurement of variables?
Please do not refer to the researcher’s hypothesis or expectation
as a theory. To be a theory a theory must be published as a
theory.]
[In addition, answer the following questions: Be sure to leave
these questions (prompts) in the assignment. Within your
assignment answer each question below the question.]
1. What were the key concepts of the research framework that
supported the development of the research questions? [Keep in
mind that your answer to this question can be repetitive with the
research theory framework described above. Ensure your
response includes all components of the research theory
framework and describe the support and development
relationship between these key components and the research
questions.]
2. What are your reflections on the connections between
theoretical or conceptual frameworks and research questions as
they relate to developing an Applied Improvement Project
(AIP)?
Methodology
[Identify the research design (e.g., correlational, quasi -
experimental, etc.) and briefly describe the study’s sample
(number of participants, where they were studied, and their
demographics), the study's instruments, and the procedures
used. Note how threats to validity and any legal or ethical
issues or considerations were addressed, referring to the
Creswell and Creswell (2018) text and/or your CITI training.
Typically, researchers as authors will not state assertions in

their journal articles reporting studies such as: Here is how I
countered threats to validity or Here is what I did to address
ethical considerations. Instead, they describe how they
countered threat of validity and ethical considerations as a part
of their methodology descriptions. Therefore, you need to be
able to recognize how threats to validity were countered and
ethical considerations addressed. Creswell and Creswell’s
(2018) threats to internal and external validity (Ch. 8 Tables 8.5
and 8.6) apply to experimental studies only and not to other
forms of quantitative research. In other non-experimental
quantitative studies, internal validity is the summation of the
reasons why the researcher/author was confident that any
conclusions the researcher made were warranted from the data
collected and the method used. Did the researcher(s)
demonstrate that the methods used to collect and analyze the
data lead to the trustworthiness of conclusions drawn? One
potential threat to validity to consider for quantitative research
is the validity of the instrument or instruments used (see
Creswell & Creswell p. 153). How did the Kim describe the
validity of the instrument? How do we know the instrument is
valid (that is, it measures what it purports to measure)? If the
validity and ethical issues and considerations were not
discussed, that omission is a weakness and limitation in the
study and indicate the omission and which of these components
were missing. In reflecting demographics ensure you use
properly formatted in text citations. When you cite figures from
a study, include an in-text citation.]
[In addition, answer the following question: Leave this prompt
(question) in your assignment and answer below it. ]
3. Describe the intervention??
Results
[Include a comprehensive summary of the major findings of the
study. Include key details. Remember – at the doctoral level
you should use direct quotes sparingly because the bulk of your
writing should consist of summarizing or paraphrasing.
Describe the results that were found.]

(question) in the assignment and answer below it.]
4. From the results, describe the answers to each of the four
research questions and the rationale or supporting evidence for
those answers.
Discussion/Conclusion
[Describe how the Chen et al. (2019) connected the study's
results to the literature presented in the literature review (the
term systems literature refers to the related relevant literature
presented in the study’s literature review). In other words, what
concepts, assumptions and or expectations from the literature
review were confirmed by the study’s results? Describe and
analyze the strengths and weaknesses of the findings (results).
How Chen et al.’ study be improved? Include your ideas for
improving the study. Describe and analyze Chen et al.’s
recommendations for future research and implications for
practice. If Chen et al.’s recommendations and implications are
missing or not sufficiently addressed in the study, indicate what
is missing or not sufficiently addressed as part of your review
and critique. Conclude by describing how Chen et al.’s findings
could be applied to practice. In other words, what did your
learn from Chen et al.’s study that could be applied by
practitioners (organizational leaders)?]
(question) in the assignment and answer below it.]
5. What ideas, concepts, or processes from this study did you
find interesting or useful and how might those ideas, concepts,
or processes be incorporated into an applied research project.

References
[Include a properly formatted list of references cited in this
assignment. References go on a separate page. Include all
references cited. Ensure references are in the hanging indent
format and are properly APA formatted; refer to APA
Publication Manual 7th edition (2020) Chapters 9 and 10 for
guidance and examples. Do not include references contained
with the Chen et al. article unless you have read these sources.]
Chen, M., Hwang, G., & Chang, Y. (2019). A reflective
thinking-promoting approach to enhancing graduate students’
flipped learning engagement, participation behaviors, reflective
thinking and project learning outcomes. British Journal of
Educational Technology, 50(5), 2288-2307.https://doi-
org.library.capella.edu/10.1111/bjet.12823
Creswell, J. W., & Creswell, J. D. (2018). Research design:
Qualitative, quantitative, and mixed methods approaches (5th
ed.). Sage.
British Journal of Educational Technology

doi:10.1111/bjet.12823
Vol 50 No 5 2019 2288–2307
© 2019 British Educational Research Association
A reflective thinking-promoting approach to enhancing graduate
students' flipped learning engagement, participation behaviors,
reflective thinking and project learning outcomes
Mei-Rong Alice Chen, Gwo-Jen Hwang and Yu-Ying Chang
Mei-Rong Alice Chen is a PhD candidate at the Graduate
Institute of Digital Learning and Education, National
Taiwan University of Science and Technology. Her research
interests include flipped learning and digital game-based
learning. Gwo-Jen Hwang is a chair professor at the Graduate
Institute of Digital Learning and Education, National
Taiwan University of Science and Technology. His research
interests include mobile learning, digital game-based
learning, flipped classroom and AI in education. Yu-Ying
Chang is an assistant professor at the Department of
English, Tamkang University, Taiwan. Her research interests
include EFL learning, flipped learning, and digital
learning. Address for correspondence: Gwo-Jen Hwang,
Graduate Institute of Digital Learning and Education,
National Taiwan University of Science and Technology, 43,
Sec.4, Keelung Rd., Taipei, 106, Taiwan. Email:
[email protected]
Introduction
Flipped learning is an evolutionary educational approach that
combines lecture-based and in-
teractive teaching methods (Johnson & Renner, 2012; Strayer,
2012). It reverses the instruc-
tional focus of teachers, helping students to foster lower level
thinking skills (ie, remembering

and understanding) at home and developing their higher order
thinking skills (ie, analyzing,
evaluating and creating) in the classroom (Anderson,
Krathwohl, & Bloom, 2001; Bergmann
& Sams, 2012). In recent years, millions of videos of online
educational resources from subject
experts on a wide range of topics have been made freely
available online. This ever-increasing
Abstract
Although flipped learning has been recognized as being a
potential approach enabling
students to learn at their own pace before the class and
facilitating in-depth peer-
to-peer and student-to-teacher interactions in the class, it
remains a challenge to
promote students’ active learning in the before-class stage,
which could significantly
affect their in-class engagement and learning performance. In
this study, a reflective
thinking-promoting approach is proposed to facilitate students’
learning design project
performance, technology-enhanced active engagement, and their
reflective thinking
and participation in the before-class stage of flipped learning. A
quasi-experiment was
conducted on a flipped Digital Learning course of a Master’s
program in a university
to evaluate the effects of the approach on students’ learning
design performance,
engagement, reflective thinking and participation. A total of 19
students (7 male
and 12 female) were in the experimental group learning with the
reflective thinking-
promoting approach, while 19 (4 male and 15 female) were in
the control group

learning with the conventional flipped learning approach. The
results indicated that
the proposed approach significantly enhanced not only the
students’ learning design
project outcomes and reflective thinking, but also their
engagement and participation
in the before-class stage of flipped learning.
mailto:
https://orcid.org/0000-0001-5155-276X
mailto:[email protected]
Reflective thinking-promoting for flipped learning 2289
access to lecture capture technology has made flipped learning
much easier. Flipped learning
is no longer a buzzword in education but a tendency and even a
must (Bishop & Verleger, 2013;
Hoffman, 2014; Lage, Platt, & Treglia, 2000). Many agree that
it enhances learning practice and
makes full use of the class time, focusing on student-centered
interactive activities and individ-
ual scaffolding (Bergmann & Sams, 2012; Tam, 2000).
Among various educational objectives, reflective thinking skills
involve complex judgmental
skills such as critical thinking and problem solving (Burton,
2010). These are indispensable skills
in today’s world of technologies and information abundance.
Therefore, it is believed that flipped
learning must be able to help students attain these skills in all
classrooms, at all education levels
and in all subjects. To acquire reflective thinking in flipped

learning, students need to fully par-
ticipate and be engaged in designed pre-class and in-class
activities and experience the process
of remembering, understanding, analysis, synthesis, evaluation
and finally conceptual change.
This process is recognized as the development of reflective
thinking (Atkins & Murphy, 1993;
Practitioner Notes
What is already known about this topic
• Flipping learning is an effective teaching approach that shifts
the lecture time to the
before-class stage and hence teachers have more time to conduct
learning activities
to promote students’ higher order thinking as well as to deal
with individual students
learning problems.
• Students’ learning experience, motivation and belief could be
the factors that guide
students towards engagement and participation in content and
help them learn new
skills.
• Engaging students in reflective thinking is an important and
challenging issue. It
provides students with an opportunity to scrutinize their own
learning and hence
make progress.
What this paper adds
• A reflective thinking-promoting approach into flipped learning
is proposed to facil-

itate students’ flipped learning engagement and participation
behaviors as well as
their project performance and reflective thinking.
• In addition to promoting students’ learning outcomes, the
results indicated that the
proposed approach provides promising results on the
technology-enhanced active
learning experience and participation in online learning in the
before-class stage of
flipped learning.
Implications for practice and/or policy
• Via monitoring students’ online before-class progress,
instructors can recognize the
factors that affect students’ learning, adjust or differentiate
their instruction and
even provide students with more opportunities or with
additional support to meet
students’ needs for learning.
• The link between the video lectures and the classroom
activities can be examined in
future research to perceive the influence of video lectures on
students’ participation
behaviors in-class activities.
• Forming reflective thinking skill is important, but attainable;
it needs students’ en-
gagement and participation in time and effort.

2290 British Journal of Educational Technology Vol 50 No 5
2019
Hong & Choi, 2015), a natural form of learning from
experience and reflection. Moreover, it is
an active, persistent and careful consideration (Porntaweekul,
Raksasataya, & Nethanomsak,
2016). If students can be aware of and govern their learning by
actively participating in reflective
thinking, that is, if they can assess what they know, what they
need to know and how they bridge
that gap in learning contexts, learning will occur. Therefore, in
this study, we aimed to examine
the effect of a reflective thinking-promoting approach on
students’ learning design performance
and reflective thinking compared with conventional flipped
learning. Meanwhile, students’ tech-
nology-enhanced active engagement and participation in the
learning management system
(hereafter LMS) were scrutinized. Further, we hoped to build up
a systematic module, leading to
more replicable and sustained flipped classrooms. Four research
questions are addressed:
1. Does the reflective thinking-promoting approach enhance
students’ learning design
performance?
2. Does the reflective thinking-promoting approach strengthen
students’ engagement?
3. Does the reflective thinking-promoting approach boost the
students’ reflective thinking?
4. How does the reflective thinking-promoting approach boost
the students’ participation in the
before-class stage of flipped learning?

Literature review
Flipped learning
The flipped classroom refers to the teaching mode which
reverses the traditional instruction
by delivering teachers’ lectures at the pre-class stage in the
form of instructional videos or
other media to enable teachers to have more time in the class to
help students do exercises and
solve the learning problems they encounter (Bergmann & Sams,
2012). In addition, the Flipped
Learning Network (https ://flipp edlea rning.org/) has stated the
importance of conducting effec-
tive “flipped learning” by taking four components (ie, flexible
environment, learning culture,
intentional content and professional educator) into account.
Researchers have further indicated
the need to employ effective learning strategies in flipped
classrooms rather than only paying at-
tention to the development of pre-class instructional videos to
draw on learners’ active learning
and engagement (Bishop & Verleger, 2013). Jensen, Kummer,
and Godoy (2015) defined that ac-
tive learning is using additional technology, teaching materials
and peer instruction for mean-
ingful learning experience. Instead of teaching, the focus should
be on how to involve students in
autonomous learning, that is “the involvement of students in
activities and thinking about activ-
ities.” In addition, Hung’s study (2015) indicated that active
learning involves students in doing
activities and in thinking about the information, they are
learning. In her study, the experimen-
tal group participants had also invested more out-of-class study
time in demonstrating their
learning engagement. Many educators and teachers have

supported this revolution (Bergmann
& Sams, 2012; Butt, 2014; Hamdan, McKnight, McKnight, &
Artfstrom, 2013; Hwang, Lai, &
Wang, 2015), whereas some have raised doubts about its effects
(Hung, 2017; Johnson & Renner,
2012; Strayer, 2012), and some consider it time-consuming and
tedious to prepare video record-
ings and in-class activities. Nevertheless, it is believed that two
major aspects can make flipped
learning effective and practical: one is in-class activities that
can trigger students’ engagement
and the other is top-quality educational videos of an optimal
duration that suits students’ atten-
tion span.
The premise of flipping a class is the combination of digitally
based lectures as pre-class tasks
and learner-centered activities in reserved class time (Hoffman,
2014; Tam, 2000). Because of
its blended features, the flipped classroom approach involves
digital platforms, that is, learning
https://flippedlearning.org/
management systems, also known as LMSs (eg, Moodle and
1know) and real face-to-face human
interaction (Bishop & Verleger, 2013). In the traditional
teacher-centered learning theories,
learners watch the instructional videos or read the assigned
materials via the LMS out of class
without any limitation of time and space; in contrast, in class,

aside from assessing learners’
retention and understanding of the subject matter, teachers,
based on learners’ diversity and on
their knowledge and experiences, implement a learner-centered
learning approach to have inter-
active classroom activities (Soliman, 2016).
However, flipping a class does not really contribute to flipped
learning. Flipped learning requires
certain kinds of action (Hamdan et al., 2013; Hwang, Lai, &
Wang, 2015). The Flipped Learning
Network presented an effective flipped learning model with four
components: flexible environ-
ment, learning culture, intentional content and professional
educator (Hung, 2017). There are
still some implications worth restating. First of all, the learning
environment has to remain flexible
enough to create individual work areas, small group work
spaces, and whole class mingling and
demonstration stations (Bergmann & Sams, 2012). Next, a wide
variety of learning experiences,
approaches and academic-support strategies shift from the
teacher-centered to learner-centered
orientation, focusing on one-to-one differentiated instruction
and immediate feedback in a com-
fortable learning culture (Bergmann & Sams, 2012). A
scaffolding effect on cognitive and social
development occurs at any moment (Hamdan et al., 2013).
Moreover, the video content is not
simply “add-on” homework; it helps learners to build up a solid
foundation for in-class activities
and to develop their conceptual understanding and procedural
fluency (Seaboyer, 2013).
Over the past decade, numerous studies have shown a sequence
of positive results, including learn-

ers’ academic performance (Bergmann & Sams, 2012; Sergis,
Sampson, & Pelliccione, 2018), per-
ceptions of engagement and learning skills (Elmaadaway,
2018), satisfaction (Bergmann & Sams,
2012), self-regulation (Lai & Hwang, 2016; Sun, Wu, & Lee,
2017), preferences (Bates & Galloway,
2012), technology acceptance (Kissi, Nat, & Armah, 2018) and
so on. Nevertheless, the core
value of flipped learning, that is, higher-order thinking skills,
is less discussed and less satisfac-
torily resolved. Flipped learning, not just revolving around
lecture-based learning to attain lower
thinking skills, that is, understanding and memorization, also
encourages learners to develop
higher order thinking skills in class: to apply, synthesize and
even create knowledge (Seaboyer,
2013). Therefore, a sound, comprehensive flipped learning
module will be outlined in this paper.
Technology-enhanced active engagement and participation
Learning effectiveness is determined by students’ optimal
engagement, and this engage-
ment is achieved by factors that promote learning (Chuang,
Weng, & Chen, 2018; Shernoff,
Csikszentmihalyi, Schneider, & Shernoff, 2003). Students’
optimal engagement is their
self-awareness of and their commitment to their own learning
(Andrusyszyn & Yankou, 2004).
Chen and Wu (2012), for instance, indicated that in learners’
experience, motivation and be-
lief are the factors that guide learners towards engagement in
content and help them learn
new skills. The former is derived from the motive for learners’
actions, willingness and goals,
whereas the latter is a firmly held opinion related to their
learning strategies, academic perfor-

mance and motivation (Chen & Pajares, 2010). In fact, two
personal traits, namely motivation
and belief, were examined and were found to be influential in
learning in a flipped classroom
context (Chuang et al., 2018). On the other hand, researchers
(Ainley & Ainley, 2011; Shernoff
& Csikszentmihalyi, 2009), have particularly pointed out that
learning engagement is char-
acterized by crucial factors such as learners’ concentration,
interest and enjoyment. That is,
in a powerful learning engagement, learners will be at a high
energy level to engage in their
learning and ultimately they have positive outcomes or results
(Fink, 2003). However, with the
2019
rapid development and popularization of Web 2.0 and computer
technology, these digital native
students are faced with an unprecedented impact on their
learning experience. According to
Thompson (2013) and Fong and Wang (2007), digital natives’
learning motivation drops very
fast when they are involved in reading. Their attention and
interest reduces even more in ac-
ademic subjects (Koutropoulos, 2011; Prensky, 2001).
Therefore, technology-enhanced active
learning has become particularly important.
The concept of active learning refers to students’ active
engagement and participation in the

learning process (Peng, Wang, & Sampson, 2017). It seizes the
idea of learning by doing and
eventually leads students to knowledge construction and
continuous learning (Argote & Miron-
Spektor, 2011; Pahl & Kenny, 2008). Technology is an effective
tool that can facilitate the learn-
ing process and consecutively create an active environment for
learners to build their knowledge,
skills, experience and engagement. For this reason, a reflective
analysis of existing technology-en-
hanced active learning engagement becomes crucial in this
present study.
Learning engagement determines whether learners have learned
throughout the course, but it
is an individual attribute and needs to be examined (Felder &
Brent, 2005; Ventura & Moscoloni,
2015). Based on the theory of Flow, proposed by
Csikszentmihalyi (1990), Schwarzenberg,
Navon, Nussbaum, Pérez-Sanagustín, and Caballero (2018) set
up a more thorough, compre-
hensive assessment model to measure learning experience. In
this model, the constructs of learn-
ing experience were adopted from Shernoff, Csikszentmihalyi,
Schneider, and Shernoff ’s (2003)
engagement, which consists of feedback, challenge, peer
instruction, choice and enjoyment. In
the construct of feedback, three dimensions are focused on: (1)
the objectives and success, (2)
self-current performance and (3) task completeness. Challenge,
slightly beyond one’s current
level of ability, has a discernible effect on academic
performance. Peer instruction provides learn-
ers with an opportunity to interact and learn from each other; it
has a positive impact on learn-
ing achievement. The perceived choice is related to one’s

satisfaction and autonomous learning.
Enjoyment refers to the satisfaction with the expected outcomes
of the task.
In this study, we utilized Schwarzenberg and his colleagues’
(2018) experience model as a refer-
ence to investigate learners’ engagement because it combines
theories that describe the factors
motivating learners and the conditions needed to generate the
optimum engagement. With the
help of the LMSs that monitor learners’ participation in the
before-class stage of flipped learning,
it is believed that technology-enhanced active engagement most
likely represents a learner’s con-
ceptions of how, when and where flipped learning does and can
take place.
Reflective thinking
Reflective thinking (also known as RT) is rational thinking
realized by mental discipline (Kok,
2002). It is often used synonymously with critical thinking, but
unlike critical thinking which
includes various thinking skills leading to satisfying outcomes,
RT puts more emphasis on the
process of making decisions or stating opinions about what has
happened (Evans, 2010; Schön,
2017). RT provides learners with a structured opportunity to
scrutinize their own learning
(Verpoorten, Westera, & Specht, 2011). During a reflective
activity, learners can develop reflec-
tive thinking skills by (1) relating new knowledge to previous
understandings, (2) thinking in
abstract and conceptual ways, (3) applying specific strategies in
new tasks and (4) understand-
ing their own ideas and thoughts (Hwang, Wu, & Ke, 2011).

RT can help learners to employ thinking skills such as analysis,
synthesis and evaluation to reach
a conceptual change (Atkins & Murphy, 1993; Hong & Choi,
2015). It is especially a critical factor
in problem solving (Kok, 2002; Wang, Yuan, Kirschner,
Kushniruk, & Peng, 2018). Although
previous studies pointed out that RT is often associated with
post-practice methods of experience
recapture through self-assessment, such as portfolios or learning
diaries (Evans, 2010) or inter-
active activities (Hwang et al., 2011), whether flipping learning
activates RT skills and engages
learners in active reflection needs to be further explored.
Many researchers privilege RT and believe it can promote
reflection upon practice (Atkins &
Murphy, 1993; Evans, 2010; Hong & Choi, 2015; Hwang et al.,
2011; Kok, 2002), but not many
of them can actually measure RT and tell whether people are
meeting their goal of developing
RT and further give explanations (Kember et al., 2000).
Fortunately, Kember and his colleagues
offered a prominent, detailed questionnaire for assessing
different levels of reflective thinking.
The questionnaire consists of four constructs, namely habitual
action, understanding, reflection
and critical reflection. Habitual action refers to learners’
automatic performance with little con-
scious thought, whereas understanding means that learners can

understand and comprehend a
concept in academic learning. As for reflection, based on
Dewey’s definition (1933, p. 9), learners
engage in intellectual and affective activities, raise questions,
test the assumptions during the
process of problem solving, and finally gain new
understandings and appreciations. The last is
a critical reflection, part of a higher order of thinking. It is a
reasoning process which finally
reaches a perspective transformation. Having the identities of
scales, the next step is to draw up
the effect of a reflective thinking-promoting approach in
flipped learning.
Ref lective thinking-promoting approach for f lipped learning
The “reflective thinking-promoting approach” proposed in this
study emphasizes the guided
“reflection” in a project after engaging students in the learning
process of watch-annotate, sum-
marize-question, discuss-give feedback and reflect-project by
referring to Rath’s (2014) WSQ
(Watch-Summary-Question) framework and Bloom’s Taxonomy
Matrix (Anderson et al., 2001)
to serve as the theoretical framework of the flipped learning
activities. The proposed approach is
divided into watch-annotate and summarize-question in the pre-
class stage, discuss-give feed-
back in the in-class stage, and reflect-project in the after-class
stage, with six cognitive process
dimensions (remember, understand, apply, analyze, evaluate and
create) and four knowledge di-
mensions (factual, conceptual procedural, knowledge and
metacognitive) of Bloom’s Taxonomy
Matrix (Anderson et al., 2001), as shown in Figure 1. This is
intended to enhance learners’ reflec-
tive thinking with a positive learning experience.

Pardo et al. (2015) stated that video annotation could be a
beneficial strategy to help learners
achieve their learning outcomes. In conventional flipped
learning, video watching as a pre-class
activity is a less interactive and one-way form of learning (Lai
& Hwang, 2016). Fu and Hwang
(2018) stated that the adoption of technologies could promote
learners’ learning engagement,
reflection and reflective thinking. Because of more recent
advanced technologies, considering
the “1know” system (http://www.1know.net) as an LMS
example as mentioned earlier, it can
offer time-stamped annotation features, whereby learners can
watch course videos and take
notes. Meanwhile, learners can also pause, rewind, re-watch the
videos. In addition, a reflective
thinking-promoting approach was adopted not only to watch-
annotate but also to summarize
question as a series of tasks before class, and then the learners
discussed the question selected by
the instructor and then provided feedback in class. The purpose
is to help the learners engage in
reflective thinking based on Bloom’s Taxonomy Matrix.
Accordingly, they were given a chance to
develop reflective thinking skills (see Figure 2).
To evaluate the effect of incorporating the reflective thinking-
promoting approach into flipped
learning on learner’ learning performance, technology-enhanced
active engagement, reflec-
tive thinking and participation in the before-class stage of
flipped learning, the reflective think-
ing-promoting approach was implemented in an online learning
management system (LMS), the

http://www.1know.net
2019
1know system, to monitor learners’ progress, as shown in the
color-coded task list in Figure 2.
The colors help to keep learners alert and more interested in
their own learning progress as well
as their peers’, encouraging them to learn.
One of the most substantial advantages of using the LMS is
that it can monitor learners’ learning
progress on pre-class tasks, note taking and behavior (time
spent on each task), but it can also
gather data on students’ online learning behavior. It can also
offer time-stamped annotation fea-
tures whereby students can watch course videos and take notes.
In addition, it allows students to
propose questions (as shown in Figure 3).
Figure 1: Structure of a reflective thinking-promoting approach
for flipped learning
[Colour figure can be viewed at wileyonlinelibrary.com]
Figure 2: Illustrative example of the flipped learning
management system
www.wileyonlinelibrary.com

Experimental design
To evaluate the effectiveness of the innovative flipped learning
approach, an experiment was
conducted on a digital learning course to evaluate the learning
design performance, learner en-
gagement, reflective thinking and participation in the before-
class stage of flipped learning with
two different flipped learning approaches. The objectives of the
Analysis and Applying Strategies
of Digital Learning Literature course are to foster students’
literature reading and analyzing
competences as well as their abilities of applying digital
learning strategies to educational set-
tings and analyzing the learning effectiveness.
Participants
A quasi-experimental design with a digital survey was carried
out in an Analysis and Applying
Strategies of Digital Learning Literature course at a technology
university in northern Taiwan.
A total of 38 graduate students (11 male and 27 female)
participated in this study, of whom
19 (7 male and 12 female) were in the experimental group
learning with the reflective think-
ing-promoting approach and 19 (4 male and 15 female) were in
the control group learning with
the conventional flipped learning approach. In order to avoid
the influence of different teachers
on the experimental outcomes, the two classes were instructed
by the same senior professor. The
students in the experimental group learned with the reflective
thinking-promoting approach

incorporated into the flipped learning approach whereas those in
the control group learned with
the conventional flipped learning approach. During the pre-class
learning activity, both groups
were assigned to watch seven videos, take notes, write a
summary and post a question, while
the control group students were assigned similar activities, but
note-taking and question-posing
were not obligatory, so it was an option for them.
Experimental process
In this study, the Analysis and Applying Strategies of Digital
Learning Literature course was
held for 3 hours a week over a period of 18 weeks. According to
the aim of the graduate course,
it is to prepare graduate students for advanced research,
particularly for the graduate thesis and
Figure 3: Interface of question-posing after watching the
instructional video lecture
2019
doctoral dissertation (Austin, 2002; Boote & Beile, 2005).
Based on the objective of Analysis
and Applying Strategies of Digital Learning Literature course,
10 journal papers were assigned
as course materials and seven instructional videos (as in Table
1) providing a starting point for

students to cultivate their reading and research skills. Students
were instructed to study vari-
ous digital learning strategies (eg, peer assessment, project-
based learning and problem-based
learning) via literature review and analysis, and to analyze the
subjects’ learning performances
and patterns from various aspects by applying the strategies to
practical educational settings.
Journal readings were examples related to the learning design
and research design methods
introduced in the instructional videos, such as learning
strategies and pedagogical theories. In
class, the students were guided to implement learning design
and research design as well as to
analyze the data (eg, learning logs) provided by the instructor
using the learning strategies and
analysis methods learned in the pre-class stage. The
instructional videos were developed by an
experienced professor who had had more than 20 years’
experience teaching the course, and
were reviewed by another experienced educational technology
instructor and two flipped learn-
ing experts recognized by FLGlobal.
In this course, the participants gave graded reports during the
class to display their knowledge
and to present all aspects of their original research ideas to
fellow researchers in the field. The
class was carried out in the read-and-present format. That is, the
professor had a list of journal
articles on the syllabus for each week related to the strategies of
digital learning; each partici-
pant studied one of the articles and then gave an in-class
report. Each participant was asked to
summarize the objectives and findings of the journal article
during his or her report, and to dis-

cuss ideas and provide their own reflections on the readings
with other participants. During the
Table 1: Example of course content and videos
Week Journal papers and instructional video links Time
11 Paper 5: Improving learning achievements, motivations and
problem-solving skills
through a peer assessment-based game development approach
Video1: Peer assessment https ://www.youtu be.com/watch
?v=tAUqR wRgdE I&featu
re=youtu.be
7:34
12 Paper 6: Effects of different online peer-feedback
approaches on students' performance
skills, motivation and self-efficacy in a dance course
Paper 7: A concept map-embedded educational computer game
for improving stu-
dents' learning performance in natural science courses
Video 2: Video sharing https ://www.youtu be.com/watch
?v=9d-EHp8q gbc&featu
re=youtu.be
9:32
13 Paper 8: Effects of an integrated concept mapping and web-
based problem-solving ap-
proach on students' learning achievements, perceptions and

cognitive loads
Video 3: Digital Media Design Principle of Design—cognitive
load 3-1 https ://www.
youtu be.com/watch ?v=gLj7Z-InR8M &featu re=youtu.be
16:32
Cognitive Theory of Multimedia Learning 3-2 https
://www.youtu be.com/watch
?v=a7-eEDVR pvs&featu re=youtu.be
11:28
14 Paper 9: Influences of an inquiry-based ubiquitous gaming
design on students' learn-
ing achievements, motivation, behavioral patterns, and tendency
towards critical
thinking and problem solving
Video 4: Problem-based learning 1 https
://youtu.be/BJv_eNe74I0 08:49
Problem-based learning 2 https ://www.youtu be.com/watch
?v=qB_6rFK2-sg&featu
re=youtu.be
16:42
Inquiry-based learning https ://youtu.be/RnwSK zl9PiE 14:04
https://www.youtube.com/watch?v=tAUqRwRgdEI&feature=you
tu.be
https://www.youtube.com/watch?v=tAUqRwRgdEI&feature= you
tu.be

https://www.youtube.com/watch?v=9d-
EHp8qgbc&feature=youtu.be
https://www.youtube.com/watch?v=9d-
EHp8qgbc&feature=youtu.be
https://www.youtube.com/watch?v=gLj7Z-
InR8M&feature=youtu.be
https://www.youtube.com/watch?v=gLj7Z-
InR8M&feature=youtu.be
https://www.youtube.com/watch?v=a7-
eEDVRpvs&feature=youtu.be
https://www.youtube.com/watch?v=a7-
eEDVRpvs&feature=youtu.be
https://youtu.be/BJv_eNe74I0
https://www.youtube.com/watch?v=qB_6rFK2-
sg&feature=youtu.be
https://www.youtube.com/watch?v=qB_6rFK2-
sg&feature=youtu.be
https://youtu.be/RnwSKzl9PiE
3-hour class, a report was conducted by one or two participants
for around 30 minutes, and then
the professor provided feedback on the students’ reports and
discussed some other issues during
the remaining time. After each report, the participants were
assigned to discuss and answer the
questions from the pre-class tasks while 1 hour was allotted to
answer a question in class. The
participants were given a week to complete the pre-class tasks.
Both the experimental and control
groups watched the pre-class video lectures before class and
finished the assigned tasks prior to

the in-class group discussion sessions. The students in the
experimental group generated open-
ended questions based on the instructional videos related to the
topics of the lecture, such as,
“What kind of teaching strategies can be used to increase
germane cognitive load and promote
schema properties?” and “What teaching methods can be used to
transform short-term memory
into long-term memory?” These questions were posted by the
students of the experimental group
after they watched the instructional videos, and were then
selected and edited by the professor.
On some occasions (eg, when some important issues were
overlooked by the students), the pro-
fessor could propose questions as well. Then, both the
experimental group and the control group
students discussed these questions in class. The professor
explained and clarified any confusion
about the video lectures after their discussion.
Figure 4 shows the experimental design of this study. The
experiment was conducted on the four
units of an Analysis and Applying Strategies of Digital
Learning Literature course, which aims
to teach the graduate students the analysis and applying
strategies of digital learning literature.
Before the experiment, the two groups of students spent
3 weeks on the concepts of flipped class-
room strategies. Pre-learning assessments for evaluating their
knowledge of collecting digital
learning literature and reporting on a digital learning paper was
administered. Following that,
the participants in the experimental group were required to
watch seven videos, take notes, write
a summary, and post a question before class meetings; on the
other hand, those in the control

group were required to do similar activities except for taking
notes and posting a question before
class meetings. In the pre-class tasks, the students in the
experimental group and control group
were required to watch videos and then write a summary on the
LMS platform before the com-
mencement of the class. The videos were embedded with a few
short questions to enhance stu-
dents’ understanding of that topic. The duration for each pre-
class video was kept to a maximum
of 16:42 minutes and an average of 12:05 minutes because of
the constraint of students’ con-
centration time (Brame, 2016). Researchers have suggested that
the duration of the video lec-
tures in flipped learning should be class grade multiplied by 1.5
in minutes (Atwa, Din, & Hussin,
2018; Bergmann & Sams, 2012), with a maximum video length
of around 20 minutes for older
students (Handke & Franke, 2013; Phillips & Trainor, 2014).
The pre-class video was uploaded, and the pre-class tasks were
assigned each week in the LMS
platform. The pre-class tasks are the summary of the assigned
journal papers, and summary and
annotation of the video lecture, as shown in Figure 5. The
students in the experimental group
and control group were required to summarize and submit what
they had remembered and com-
prehended concerning the video’s content. However, in the
experimental group, the students
were required to post a question regarding what they had
watched from the video lectures. The
instructor would select 10 frequently asked questions and have
students in both the experimental
group and the control group discuss them as one of the in-class
tasks. After all the pre-class and

in-class learning activities, the students designed an e-learning
research project, presented it and
filled out the post-questionnaires for measuring their learning
outcomes. The process was carried
out to activate the students’ metacognitive knowledge for more
learning to take place. It helps
students experience the cognitive process of Bloom Taxonomy
Matrix’s create level.
2019
Measuring instruments
In this study, the students’ learning design performance,
technology-enhanced active engage-
ment, reflective thinking and participation in the before-class
stage of flipped learning were
assessed. Learning design performance was evaluated based on
the students’ learning design
project reports. The pre-project report aimed to evaluate the
students’ basic knowledge of de-
signing digital learning activities. The students were required to
present reports during the class
to display their knowledge and perspectives of their learning
design, including what they had
learned from the literature and their own ideas of learning
design. The post-learning design
report aimed to assess the students’ concepts of applying and
innovative and effective strate-
gies for digital learning. The total score of both reports was
100. Two experienced instructors
rated the students’ reports based on a rubric consisting of four

dimensions, that is, correctness,
completeness, innovation and effectiveness. The two
instructors’ ratings were found to have the
substantial agreement of inter-rater consistency .77 Cohen’s
Kappa (Cohen, 1960).
The technology-enhanced active student engagement
questionnaire was adopted from the learn-
ing experience assessment scale developed by Schwarzenberg et
al. (2018). The questionnaire
consisted of 18 items in five subscales, namely feedback (six
items), challenge (three items), peer
instruction (three items), choice (three items) and enjoyment
(three items), with a 5-point Likert
scale. The Cronbach’s alpha values of the subscales are, .90,
.76, .84, .73 and .60 respectively.
The questionnaire of reflective thinking was adopted based on
the questionnaire to measure the
level of reflective thinking developed by Kember et al. (2000).
It consists of 16 items, includ-
ing four dimensions: habitual action, reflection, critical
reflections and understanding. The
Cronbach’s alpha values of the four dimensions were .80, .87,
.91 and .81 respectively.
The time spent viewing video lectures and the words posted in
the assigned tasks were collected
for analysis of participation behaviors in the before-class stage
of flipped learning. There were
seven video lectures (total 5,081 seconds, average 726 seconds)
and 17 tasks published in LMS.
During the 4-week flipped learning, participants in the
experimental group and control group
Figure 4: Experiment procedure

watched video lectures and posted their summaries and
reflections on the video lectures as the
pre-class tasks. In order to identify students’ participation
behaviors in the before-class stage of
flipped learning, the data of all participants’ time spent (in
seconds) watching the video lectures
and words (Chinese characters) were analyzed.
Experimental results
There were three kinds of data collected, namely (1) the
students’ scores of the pre-reports and
post-reports, (2) the results of the technology-enhanced active
learning engagement question-
naire, (3) the results of the reflective thinking questionnaire and
(4) Data in LMS of participants’
time spent watching video lectures and words posted. SPSS
software was used to analyze the
data. The dependent variables were checked for normal
distribution and homogeneity of vari-
ance before analysis.
Learning design performances
The one-way ANCOVA was used to compare the two groups’
learning design performances by
adopting the reflective thinking-promoting approach into the
flipped learning model as an in-
dependent variable, while the posttest and pretest scores were

respectively the dependent vari-
able and covariate. After confirming the assumption of
homogeneity of regression with F = .023
(p = .88 > .05), ANCOVA was performed, as shown in Table
2. A significant difference was
Figure 5: A list of pre-class tasks
2019
found with F = 4.29 (p < .05), showing that the learning design
performances of the two groups
were significantly different because of the different flipped
learning models. The result showed
that the participants (adjusted mean = 89.05, Std. error =
1.48) who learned with the reflec-
tive thinking-promoting approach incorporated into the flipped
learning model outperformed
the participants (adjusted mean = 84.73, Std. error = 1.48) who
learned with the conventional
flipped learning mode.
Technology-enhanced active learning engagement
Table 3 shows the t-test result for the technology-enhanced
active learning engagement ratings
of the two groups. The means and standard deviations of the
ratings were 4.59 and .30 for the
experimental group, and 4.36 and .35 for the control group. The
ratings of the two groups were

significantly different with t = 2.21 (p < .05). There was a
significant difference between the
two groups with d = .71 (Cohen, 1988). This implies that there
was more positive effect on the
learning engagement of the participants in the experimental
group than on those in the control
group.
We also performed t-tests to assess the impact of specific
dimensions. As shown in Table 4, it is
recognized that peer instruction t = 2.65 (p < .05, d = .86) had
more positive effect, with a sig-
nificant difference, on the experimental group (M = 4.77 SD
= .33) than on the control group
(M = 4.33 SD = .64).
Reflective thinking
The t-test outcome of the comparison of reflective thinking for
the two learning approaches
is shown in Table 5. The mean of reflective thinking ratings for
the experimental group was
4.57 (t = 3.34, p < .01, d = 1.11), which was significantly
higher than that of the control group
(mean = 4.16). Furthermore, the result of reflective thinking for
the two groups represents a good
effect size according to Cohen’s (1988) criteria, where a
Cohen’s d value greater than .8 is con-
sidered as a large effect size. This means that with the reflective
thinking-promoting approach
incorporated into the flipped learning approach, the students
achieved significantly higher re-
flective thinking than those in the conventional flipped learning
approach. In other words, the
reflective thinking-promoting approach can successfully
promote students’ reflective thinking.

In addition, it was found that three of four dimensions showed
significant differences between
the two groups, as shown in Table 6, in particular, for habitual
action (t = 2.65, p < .05, d = .85),
reflection (t = 3.09, p < .001, d = 1.50) and critical reflection
(t = 2.51, p < .0001, d = .92). Such a
finding could be further evidence that the students who learned
with the reflective thinking-pro-
moting approach in the flipped learning approach showed better
habitual action, reflection and
critical thinking than those who learned with the conventional
flipped learning approach.
Participation behaviors in the before-class stage of flipped
learning
In order to clearly understand the students’ participation
behaviors in the before-class stage
of flipped learning, students’ average time spent (in seconds) on
online video lectures and the
Table 2: ANCOVA result of the post scores on students'
learning performance
Group N Mean SD Adjusted mean Std. error F η 2
Experimental group 19 89.05 5.411 89.05 1.48 4.29* .106
Control group 19 84.73 7.327 84.73 1.48
*p < .05.

words (Chinese characters) they posted was analyzed. Table 7
shows the t-test results of the com-
parison of the experimental group (Mean = 1217.72, t = 3.14,
p < .01, d = 1.01) which spent
significantly more time than the control group (Mean =
885.99) watching video lectures. In
addition, the t-test results also reveal that the experimental
group (Mean = 1344.26, t = 5.66,
p < .001, d = 1.84) posted significantly more words than the
control group (Mean = 1005.68).
Discussion and conclusions
The study aimed to explore students’ engagement and
participation in active learning with
learning management systems and to examine the effect of
whether a reflective thinking-pro-
moting approach would enhance students’ flipped learning
design performance and participa-
tion. The implications of the findings are discussed as follows.
In general, the findings provide evidence to support the
reflective thinking-promoting approach.
Concerning research questions 1 and 3: This study aimed to
measure the impact of the inte-
grated reflective thinking-promoting approach in flipped
learning on the students’ learning
design performance and reflective thinking of graduate students
in the fall 2018 Analysis and
Applying Strategies of Digital Learning Literature course. In
the experimental group, the partici-
pants showed a significantly higher level of learning design
performance and reflective thinking,
Table 3: Independent sample t-test results of the technology-
enhanced active learning experience

Group N Mean SD t Cohen’s d
Experimental group 19 4.59 .30 2.21* .71
Control group 19 4.36 .35 19
*p < .05.
Table 4: Independent sample t-test results of the technology-
enhanced active learning experience in each
dimension
Dimensions Group N Mean SD t Cohen's d
Feedback Experimental group 19 4.67 .39 .86 .30
Control group 19 4.55 .43
Challenge Experimental group 19 4.31 .44 1.64 .51
Peer instruction Experimental group 19 4.77 .33 2.65* .86
Choice Experimental group 19 4.70 .43 .86 .27
Enjoyment Experimental group 19 4.39 .47 1.42 .46
*p < .05.
Table 5: Independent sample t-test results of reflective thinking
Group N Mean SD t Cohen's d
Experimental group 19 4.57 .35 3.34** 1.11
**p < .01.

2019
indicating that integrating the reflective thinking-promoting
approach in flipped learning has a
positive impact on students’ learning design performance and
reflective thinking capacity. In par-
ticular, the results of three of four dimensions: habitual action,
reflection and critical reflection,
demonstrated significant differences between the experimental
group and the control group.
One of the most important pre-class tasks in this study was
question-posing, and this task
provided an instruction way for both teaching and learning.
Participants in the experimental
group were required to pose a question after watching each
video lecture, and then the instruc-
tors would select 10 suitable questions and have participants in
the experimental group and
the control group discuss them in the class. Lan and Lin (2011)
mentioned that good quality
question-posing requires more reflective thinking, and
suggested that engaging in the practice
of question-posing can foster students’ cognitive development.
In this study, reflective think-
ing tasks in flipped learning offered participants an organized
approach to examine their own
learning, so they had an opportunity to develop reflective
thinking skills. Various studies have
shown that promoting students’ reflective thinking via activities
is valuable for learning design
performance (Bigge & Shermis, 1999; Cheng & Chau, 2013;

Hwang, Wu, & Ke, 2011; Phan,
2009; Verpoorten, Westera, & Specht, 2011). This is consistent
with a considerable amount of
literature demonstrating the correlation of higher order skills
with students’ success in learning
(Ghanizadeh, 2017; Kealey et al., 2005). Phan (2009) further
demonstrated that the effective-
ness of the learning process is influenced by reflective
thinking; likewise, learning achievement
is influenced by reflective thinking. Reflective thinking fosters
knowledge that can be applied to a
variety of situations and contexts. Students have the capability
to reflect initially, to acquire and
then to generate the new knowledge, so they tend to be more
successful academically.
Flipped learning combines video lectures on LMSs as pre-class
content instruction and in-class
hands-on activities that encourage students to practice, apply,
reflect and master the content that
Table 6: Independent sample t-test results of the reflection
between the two groups
Dimensions Group N Mean SD t Cohen's d
Habitual action Experimental group 19 4.67 .35 2.65* .85
Reflection Experimental group 19 4.61 .38 3.09** 1.50
Critical reflection Experimental group 19 4.46 .38 2.51*** .92
Understanding Experimental group 19 4.53 .53 .47 .16
*p < .05; **p < .01; ***p < .001.

Table 7: Independent sample t-test results for students'
participation behaviors in the before-class stage of flipped
learning
Variable Group N Mean SD t Cohen's d
Time spent on
videos
Experimental group 19 1217.72 4.38.62 3.14** 1.02
Control group 19 885.99 139.01
Words posted Experimental group 19 1344.26 236.33 5.66***
1.84
Control group 19 1005.68 110.39
**p < .01; ***p < .001.
they have learned from the pre-class requirements (Hoffman,
2014; Tam, 2000). This blended
approach provides flexibility for students who can work at their
own pace and take on the respon-
sibility for learning; however, it may not be easy for teachers to
keep students more engaged with
learning, which could lead to dissatisfaction in the end (Johnson
& Renner, 2012; Strayer, 2012).
Therefore, students’ engagement and participation in active
learning with the LMS becomes
more important. The results in Table 3 for question 2 present

that, in the technology-enhanced
active engagement, the students with a reflective thinking-
promoting approach overall outper-
formed the students in the conventional flipped learning
environment. As Chuang, Weng, and
Chen (2018) mentioned, students’ optimal engagement that
promotes learning depends on
certain factors. In this study, taking notes and posting questions
are such behaviors of learning
engagement.
In terms of detailed constructs of the engagement, the students
with a reflective thinking-
promoting approach had a statistically significant difference in
peer instruction, compared
with the students in the conventional flipped learning
environment. Students’ learning engage-
ment and active learning were examined in this study. Gilboy et
al. (2015) claimed that learn-
ing engagement is students’ commitment to and responsibility
for learning as they contribute
their time and effort to their learning production. Jensen,
Kummer, and Godoy (2015) defined
that active learning is using additional technology, teaching
materials and peer instruction for
a meaningful learning experience. Instead of teaching, the
focus should be on how to involve
students in autonomous learning, that is, the involvement of
students in activities and think-
ing about activities. In addition, Hung (2015) indicated that
active learning involves students in
doing activities and thinking about the information they are
learning. In her study, the partici-
pants of the experimental group also invested more out-of-class
study time to demonstrate their
learning engagement. In this study, the participants in the

experimental group increased their
participation time in the before-class stage of flipped learning,
where they needed to take notes
and reflect when raising questions. In class, they had deeper
discussions on their own questions
with peers and the instructor. The process of student
engagement was the reason for their active
learning (Pahl & Kenny, 2008). Lastly, it was found that the
students in both groups had high
values for feedback, challenge, peer instruction, choice and
enjoyment (with means of more than
four on the 5-point Likert scale). Perhaps that is because the
LMS can show students’ color-coded
progress on each task list, which resulted in their commitment
and enhanced their attentiveness
to their own and their peers’ learning progress.
The present study examined students’ engagement in active
learning with learning manage-
ment, and the findings were positive. Meanwhile, this is
evidence that a reflective thinking-
promoting approach can enhance students’ flipped learning
design performance and participa-
tion. A reflective-thinking promoting approach amplifies the
core value of flipped learning, that
is, higher-order thinking. Students can activate their intuitive
reflection and gain learning expe-
rience skills by analyzing, synthesizi ng and evaluating what
they have learned when interacting
with the video content instruction and discussions from their
peers and instructor. In addition,
with the LMSs that monitor students’ progress, instructors can
recognize the factors that affect
students’ learning, adjust or differentiate their instruction and
even provide students with more
opportunities or with additional support to meet students’ needs

for learning. Flipped learning
with a reflective thinking-promoting approach is one of the
examples that encourages students
to become more responsible for their learning and that aids
them in learning their subject matter.
On the other hand, there are several limitations and issues that
need to be considered for future
research. First, the sample size was insufficient, making it
difficult to make generalizations.
However, it was not easy to have a great number of graduate
students in this present study.
2019
Therefore, we suggest that in future research, a variety of
subjects can be considered as partic-
ipants. Second, the time of the study was not long enough. If
the time were longer, the results
should be more substantial because reflective thinking and
active learning are important, but
they take time to become habits. Third, the activities carried out
in the class, such as students’
discussion and feedback, were not entirely assessed in this
study. Overall, the most significant
purpose of this study is to illustrate the fact that forming
reflective thinking skills is both essential
and attainable. Most importantly, it needs students’
engagement, participation, time and effort.
Acknowledgements

This study is supported in part by the Ministry of Science and
Technology of the Republic of
China under contract number MOST-105-2511-S-011-008-MY3.
Statements on open data, ethics and conf lict of interest
The data can be obtained by sending request e-mails to the
corresponding author.
The participants were protected by hiding their personal
information during the research pro-
cess. They knew that the participation was voluntary and they
could withdraw from the study at
any time.
There is no potential conflict of interest in this study.
References
Ainley, M., & Ainley, J. (2011). Student engagement with
science in early adolescence: The contribution
of enjoyment to students' continuing interest in learning about
science. Contemporary Educational
Psychology, 36(1), 4–12.
Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A
taxonomy for learning, teaching, and assessing: A
revision of Bloom's taxonomy of educational objectives. New
York, NY: Longman.
Andrusyszyn, M.-A., & Yankou, D. (2004). Attention fatigue
and the effect of debriefing in a web-enhanced
graduate nursing course. International Journal of Instructional
Technology and Distance Learning, 1(4), 1–7.
Argote, L., & Miron-Spektor, E. (2011). Organizational
learning: From experience to knowledge.

Organization Science, 22(5), 1123–1137.
Atkins, S., & Murphy, K. (1993). Reflections: A review of the
literature. Journal of Advanced Nursing, 18,
1188–1192.
Atwa, Z. M., Din, R., & Hussin, M. (2018). Effectiveness of
flipped learning in physics education on
Palestinian high school students' achievement. Journal of
Personalized Learning, 2(1), 73–85.
Austin, A. E. (2002). Preparing the next generation of faculty:
Graduate school as socialization to the aca-
demic career. The Journal of Higher Education, 73(1), 94–122.
Bates, S., & Galloway, R. (2012). The inverted classroom in a
large enrolment introductory physics course:
A case study. Paper presented at the Proceedings of the HEA
STEM learning and teaching conference.
Bergmann, J., & Sams, A. (2012). Flip your classroom: Reach
every student in every class every day. Washington,
DC: International Society for Technology in Education.
Bigge, M. L., & Shermis, S. S. (1999). Learning theories for
teachers (6th ed.). New York: Longman.
Bishop, J. L., & Verleger, M. A. (2013). The flipped classroom:
A survey of the research. Paper presented at
the ASEE national conference proceedings, Atlanta, GA.
Boote, D. N., & Beile, P. (2005). Scholars before researchers:
On the centrality of the dissertation literature
review in research preparation. Educational Researcher, 34(6),
3–15.
Brame, C. J. (2016). Effective educational videos: Principles

and guidelines for maximizing student learn-
ing from video content. CBE—Life Sciences Education, 15(4),
es6.
Burton, E. (2010). High level thinking and questioning
strategies. Cross Cultural Management, 17(1), 5–9.
Butt, A. (2014). Student views on the use of a flipped classroom
approach: Evidence from Australia. Business
Education & Accreditation, 6(1), 33–43.
Chen, C. H., & Wu, I. C. (2012). The interplay between
cognitive and motivational variables in a supportive
online learning system for secondary physical education.
Computers & Education, 58, 542–550.
Chen, J. A., & Pajares, F. (2010). Implicit theories of ability of
grade 6 science students: Relation to epis-
temological beliefs and academic motivation and achievement
in science. Contemporary Educational
Psychology, 35, 75–87.
Cheng, G., & Chau, J. (2013). Exploring the relationship
between students' self-regulated learning ability
and their ePortfolio achievement. Internet and Higher
Education, 17, 9–15.
Chuang, H., Weng, C., & Chen, C. (2018). Which students
benefit most from a flipped classroom approach

to language learning? British Journal of Educational
Technology, 49(1), 56–68.
Cohen, J. (1960). A coefficient of agreement for nominal scales.
Educational and Psychological Measurement,
20(1), 37–46.
Cohen, J. (1988). Set correlation and contingency tables.
Applied Psychological Measurement, 12(4),
425–434.
Csikszentmihalyi, M. (1990). Flow: The psychology of optimal
experience. New York, NY: Harper Perennial.
Dewey, J. (1933). How we think: A restatement of the relation
of reflective thinking to the educative process.
Boston, MA: D.C. Heath & Co Publishers.
Elmaadaway, M. A. N. (2018). The effects of a flipped
classroom approach on class engagement and skill
performance in a Blackboard course. British Journal of
Educational Technology, 49(3), 479–491.
Evans, J. S. B. (2010). Intuition and reasoning: A dual-process
perspective. Psychological Inquiry, 21(4),
313–326.
Felder, R. M., & Brent, R. (2005). Understanding student
differences. Journal of Engineering Education, 94(1),
57–72.
Fink, L. D. (2003). Creating significant learning experiences:
An integrated approach to designing college courses.
San Francisco, CA: John Wiley & Sons.
Fong, J., & Wang, F. L. (Eds.). (2007). Blended learning.
Edinburgh, United Kingdom 15–17 August, 2007

Proceedings. City University of Hong Kong.
Fu, Q.-K., & Hwang, G.-J. (2018). Trends in mobile
technology-supported collaborative learning: A system-
atic review of journal publications from 2007 to 2016.
Ghanizadeh, A. (2017). The interplay between reflective
thinking, critical thinking, self-monitoring, and
academic achievement in higher education. Higher Education,
74(1), 101–114.
Gilboy, M. B., Heinerichs, S., & Pazzaglia, G. (2015).
Enhancing student engagement using the flipped
classroom. Journal of Nutrition Education and Behavior, 47(1),
109–114.
Hamdan, N., McKnight, P. E., McKnight, K., & Artfstrom, K.
M. (2013). The flipped learning model: A white
paper based on the literature review. Retrieved from the Flipped
Learning Network website.
Handke, J., & Franke, P. (2013). xMOOCs im virtual linguistics
campus. In: Schulmeister R, Herausgeber.
(Ed.), MOOCs-massive open online courses: Offene Bildung
oder Geschäftsmodell (pp. 101–126). Münster:
Waxmann Verlag.
Hoffman, E. S. (2014). Beyond the flipped classroom:
Redesigning a research methods course for e-instruc-
tion. Contemporary Issues in Education Research, 7, 51–62.
Hong, Y., & Choi, I. (2015). Assessing reflective thinking in
solving design problems: The development of a
questionnaire. British Journal of Educational Technology,

46(4), 848–863.
Hung, H.-T. (2015). Flipping the classroom for English
language learners to foster active learning. Computer
Assisted Language Learning, 28(1), 81–96.
Hung, H. (2017). Design-based research: Redesign of an
English language course using a flipped class-
room approach. TESOL, 51(1), 180–192.
Hwang, G., Lai, C., & Wang, S. (2015). Seamless flipped
learning: A mobile technology-enhanced flipped
classroom with effective learning strategies. Journal of
Computer in Education, 2(4), 449–473.
Hwang, G.-J., Wu, P.-H., & Ke, H.-R. (2011). An interactive
concept map approach to supporting mobile
learning activities for natural science courses. Computers &
Education, 57(4), 2272–2280.
2019
Jensen, J. L., Kummer, T. A., & Godoy, P. D. D. M. (2015).
Improvements from a flipped classroom may sim-
ply be the fruits of active learning. CBE—Life Sciences
Education, 14(1), ar5.
Johnson, L. W., & Renner, J. D. (2012). Effect of the flipped
classroom model on a secondary computer applica-
tions course: Student and teacher perceptions, questions, and
student achievement. Louisville, KY: University

of Louisville.
Kealey, B. T., Holland, J., & Watson, M. (2005). Preliminary
evidence on the association between critical
thinking and performance in principles of accounting. Issues in
Accounting Education, 20(1), 33–49.
Kember, D., Leung, D. Y. P., Jones, A., Loke, A. Y., McKay, J.,
Sinclair, K., … Yeung, E. (2000). Development of
a questionnaire to measure the level of reflective thinking.
Assessment & Evaluation in Higher Education,
25(4), 381–395.
Kissi, P. S., Nat, M., & Armah, R. B. (2018). The effects of
learning-family conflict, perceived control over
time and task-fit technology factors on urban-rural high school
students' acceptance of video-based
instruction in flipped learning approach. Educational
Technology Research and Development, 66(6),
1547–1569.
Kok, J. (2002). Reflective journal writing: How it promotes
reflective thinking in clinical nursing educa-
tion: A students' perspective. Curationis, 25(3), 35–42.
Koutropoulos, A. (2011). Digital natives: Ten years after.
MERLOT Journal of Online Learning and Teaching,
7(4), 525–538.
Lage, M. J., Platt, G. J., & Treglia, M. (2000). Inverting the
classroom: A gateway to creating an inclusive
learning environment. The Journal of Economic Education,
31(1), 30–43.
Lan, Y.-F., & Lin, P.-C. (2011). Evaluation and improvement of
student's question-posing ability in a web-

based learning environment. Australasian Journal of
Lai, C., & Hwang, G. (2016). A self-regulated flipped
classroom approach to improving students' learning
performance in a mathematics course. Computers & Education,
100, 126–140.
Pahl, C. & Kenny, C. (2008). The future of technology
enhanced active learning – A roadmap. In M. L.
Lytras, D. Gasevic, P. Ordonez de Pablos, & W. Huang (Eds.),
Technology enhanced learning (pp. 348–375).
Hershey, PA: IGI Global.
Pardo, A., Mirriahi, N., Dawson, S., Zhao, Y., Zhao, A., &
Gašević, D. (2015). Identifying learning strate-
gies associated with active use of video annotation software.
Paper presented at the Proceedings of the Fifth
International Conference on Learning Analytics and Knowledge.
Peng, J., Wang, M., & Sampson, D. (2017). Visualizing the
complex process for deep learning with an au-
thentic programming project. Journal of Educational
Technology & Society, 20(4), 275–287.
Phan, H. P. (2009). Exploring students' reflective thinking
practice, deep processing strategies, effort, and
achievement goal orientations. Educational Psychology, 29(3),
297–313.
Phillips, C. R., & Trainor, J. E. (2014). Millennial students and
the flipped classroom. ASBBS Proceedings,
21(1), 519.
Porntaweekul, S., Raksasataya, S., & Nethanomsak, T. (2016).
Developing reflective thinking instructional

model for enhancing students' desirable learning outcomes.
Educational Research and Reviews, 11(6),
238–251.
Prensky, M. (2001). Digital natives, digital immigrants part 1.
The Horizon, 9(5), 1–6. https ://doi.
org/10.1108/10748 12011 0424816
Raths, D. (2014). Nine video tips for a better flipped classroom.
The Education Digest, 79(6), 15.
Schön, D. A. (2017). The reflective practitioner: How
professionals think in action. Abingdon: Routledge.
Schwarzenberg, P., Navon, J., Nussbaum, M., Pérez-Sanagustín,
M., & Caballero, D. (2018). Learning expe-
rience assessment of flipped courses. Journal of Computing in
Higher Education, 30(2), 237–258.
Seaboyer, J. (2013). The role of technology-assisted assessment
in fostering critical reading in undergradu-
ate literary studies. International Computer Assisted Assessment
Conference. UK: Computer Assisted
Assessment.
Sergis, S., Sampson, D. G., & Pelliccione, L. (2018).
Investigating the impact of flipped classroom on stu-
dents' learning experiences: A self-determination theory
approach. Computers in Human Behavior, 78,
368–378.
https://doi.org/10.1108/10748120110424816
https://doi.org/10.1108/10748120110424816

Shernoff, D. J., & Csikszentmihalyi, M. (2009). Flow in
schools: Cultivating engaged learners and optimal
learning environments. In R. Gilman, E. S. Heubner, & M. J.
Furlong (Eds.), Handbook of positive psychol-
ogy in schools (pp. 131–145). New York, NY: Routledge.
Shernoff, D. J., Csikszentmihalyi, M., Schneider, B., &
Shernoff, E. S. (2003). Student engagement in high
school classrooms from the perspective of flow theory. School
Psychology Quarterly, 18(2), 158–176.
Soliman, N. A. (2016). Teaching English for academic purposes
via the flipped learning approach. Procedia
- Social and Behavioral Sciences, 232, 122–129.
Strayer, J. (2012). The flipped classroom: Turning the
traditional classroom on its head. Retrieved from http://
www.knewt on.com/flipp ed-class room/
Sun, J. C. Y., Wu, Y. T., & Lee, W. I. (2017). The effect of the
flipped classroom approach to Open Course
Ware instruction on students' self-regulation. British Journal of
Tam, M. (2000). Constructivism, instructional design, and
technology: Implications for transforming dis-
tance learning. Educational Technology & Society, 3(2), 50–60.
Thompson, P. (2013). The digital natives as learners:
Technology use patterns and approaches to learning.
Ventura, A. C., & Moscoloni, N. (2015). Learning styles and
disciplinary differences: A cross-sectional

study of undergraduate students. International Journal of
Learning and Teaching, 1(2), 88–93.
Verpoorten, D., Westera, W., & Specht, M. (2011). Infusing
reflective practice in eLearning courses—Can
widgets help? International Journal of Technology Enhanced
Learning, 3(1), 93–109.
Wang, M., Yuan, B., Kirschner, P. A., Kushniruk, A. W., &
Peng, J. (2018). Reflective learning with com-
plex problems in a visualization-based learning environment
with expert support. Computers in Human
Behavior, 87, 406–415.
http://www.knewton.com/flipped-classroom/
http://www.knewton.com/flipped-classroom/
Copyright of British Journal of Educational Technology is the
property of Wiley-Blackwell
and its content may not be copied or emailed to multiple sites or
posted to a listserv without
the copyright holder's express written permission. However,
users may print, download, or
email articles for individual use.

15Quantitative Research Study ReportInsert You

Recommended

Recommended

More Related Content

Similar to 15Quantitative Research Study ReportInsert You

Similar to 15Quantitative Research Study ReportInsert You (14)

More from KiyokoSlagleis

More from KiyokoSlagleis (20)

Recently uploaded

Recently uploaded (20)

15Quantitative Research Study ReportInsert You