SlideShare a Scribd company logo
International Journal
of
Learning, Teaching
And
Educational Research
p-ISSN:
1694-2493
e-ISSN:
1694-2116
IJLTER.ORG
Vol.19 No.6
PUBLISHER
Society for Research and Knowledge Management
Port Louis
Republic of Mauritius
www.ijlter.org
Chief Editor
Dr. Antonio Silva Sprock, Universidad Central de
Venezuela, Venezuela, Bolivarian Republic of
Editorial Board
Prof. Cecilia Junio Sabio
Prof. Judith Serah K. Achoka
Prof. Mojeed Kolawole Akinsola
Dr Jonathan Glazzard
Dr Marius Costel Esi
Dr Katarzyna Peoples
Dr Christopher David Thompson
Dr Arif Sikander
Dr Jelena Zascerinska
Dr Gabor Kiss
Dr Trish Julie Rooney
Dr Esteban Vázquez-Cano
Dr Barry Chametzky
Dr Giorgio Poletti
Dr Chi Man Tsui
Dr Alexander Franco
Dr Habil Beata Stachowiak
Dr Afsaneh Sharif
Dr Ronel Callaghan
Dr Haim Shaked
Dr Edith Uzoma Umeh
Dr Amel Thafer Alshehry
Dr Gail Dianna Caruth
Dr Menelaos Emmanouel Sarris
Dr Anabelie Villa Valdez
Dr Özcan Özyurt
Assistant Professor Dr Selma Kara
Associate Professor Dr Habila Elisha Zuya
International Journal of Learning, Teaching and
Educational Research
The International Journal of Learning, Teaching
and Educational Research is an open-access
journal which has been established for the dis-
semination of state-of-the-art knowledge in the
field of education, learning and teaching. IJLTER
welcomes research articles from academics, ed-
ucators, teachers, trainers and other practition-
ers on all aspects of education to publish high
quality peer-reviewed papers. Papers for publi-
cation in the International Journal of Learning,
Teaching and Educational Research are selected
through precise peer-review to ensure quality,
originality, appropriateness, significance and
readability. Authors are solicited to contribute
to this journal by submitting articles that illus-
trate research results, projects, original surveys
and case studies that describe significant ad-
vances in the fields of education, training, e-
learning, etc. Authors are invited to submit pa-
pers to this journal through the ONLINE submis-
sion system. Submissions must be original and
should not have been published previously or
be under consideration for publication while
being evaluated by IJLTER.
International Journal of Learning, Teaching and Educational Research
(IJLTER)
Vol. 19, No. 6 (June 2020)
Print version: 1694-2493
Online version: 1694-2116
IJLTER
International Journal of Learning, Teaching and Educational Research (IJLTER)
Vol. 19, No. 6
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part
of the material is concerned, specifically those of translation, reprinting, re-use of illustrations,
broadcasting, reproduction by photocopying machines or similar means, and storage in data banks.
Society for Research and Knowledge Management
International Journal of Learning, Teaching and Educational Research
The International Journal of Learning, Teaching and Educational
Research is a peer-reviewed open-access journal which has been
established for the dissemination of state-of-the-art knowledge in the
fields of learning, teaching and educational research.
Aims and Objectives
The main objective of this journal is to provide a platform for educators,
teachers, trainers, academicians, scientists and researchers from over the
world to present the results of their research activities in the following
fields: innovative methodologies in learning, teaching and assessment;
multimedia in digital learning; e-learning; m-learning; e-education;
knowledge management; infrastructure support for online learning;
virtual learning environments; open education; ICT and education;
digital classrooms; blended learning; social networks and education; e-
tutoring: learning management systems; educational portals, classroom
management issues, educational case studies, etc.
Indexing and Abstracting
The International Journal of Learning, Teaching and Educational
Research is indexed in Scopus since 2018. The Journal is also indexed in
Google Scholar and CNKI. All articles published in IJLTER are assigned
a unique DOI number.
Foreword
We are very happy to publish this issue of the International Journal of
Learning, Teaching and Educational Research.
The International Journal of Learning, Teaching and Educational
Research is a peer-reviewed open-access journal committed to
publishing high-quality articles in the field of education. Submissions
may include full-length articles, case studies and innovative solutions to
problems faced by students, educators and directors of educational
organisations. To learn more about this journal, please visit the website
http://www.ijlter.org.
We are grateful to the editor-in-chief, members of the Editorial Board
and the reviewers for accepting only high quality articles in this issue.
We seize this opportunity to thank them for their great collaboration.
The Editorial Board is composed of renowned people from across the
world. Each paper is reviewed by at least two blind reviewers.
We will endeavour to ensure the reputation and quality of this journal
with this issue.
Editors of the June 2020 Issue
VOLUME 19 NUMBER 6 June 2020
Table of Contents
The Emergency Remote Learning Experience of University Students in Indonesia amidst the COVID-19 Crisis....1
Maila D.H. Rahiem
Examining the Fairness of Language Test Across Gender with IRT-based Differential Item and Test Functioning
Methods ................................................................................................................................................................................. 27
Burhanettin Ozdemir and Abdulrahman Hadi Alshamrani
Main Approaches of Business English Teaching to Future Lawyers: A Case Study of Ukrainian Higher
Institutions............................................................................................................................................................................. 46
Oksana P. Bykonia, Iryna V. Borysenko, Tamila L. Gruba, Iurii L. Mosenkis and Dmytro O. Chystiak
Debate as a Tool for Learning and Facilitating Based on Higher Order Thinking Skills in The Process of
Argumentative Essay Writing............................................................................................................................................. 62
Marzni Mohamed Mokhtar, Marni Jamil, Rohizani Yaakub and Fadzilah Amzah
Exploring Teachers’ Pedagogical Practices in Teaching Mandarin as a Foreign Language in MARA Educational
Institutions, Malaysia ........................................................................................................................................................... 76
Nuraini Jafri, Umi Kalthom Abd Manaf and Fazilah Razali
The Effectiveness of Differentiated Instruction by Streaming: A preliminary Study of Current Practices in the
UAE ........................................................................................................................................................................................ 95
Ibrahim Suleiman Ibrahim Magableh and Amelia Abdullah
Technology-Driven Self-Directed Learning of Graduate Pharmaceutists: Adding Value through Entrepreneurship
............................................................................................................................................................................................... 111
Iryna Nizhenkovska, Tatyana Reva, Oksana Chkhalo and Oksana Holovchenko
Teachers’ Covid-19 Awareness, Distance Learning Education Experiences and Perceptions towards Institutional
Readiness and Challenges ................................................................................................................................................. 127
Aris Alea Lapada, Frosyl Fabrea Miguel, Dave Arthur Roldan Robledo and Zeba F Alam
Enhancing Malaysian Primary Pupils’ Vocabulary Skills using Pocable Game and Pear Deck .............................. 145
Chai Kar Ni, Bonaventure Jong, Mary Anne Dison, Sylvia Anak Thomas, Melor Md Yunus and Ashairi Suliman
Content Validity of West African Examination Council Financial Accounting Questions....................................... 161
Basil C. E. Oguguo, John J. Agah, Catherine U. Ene, Vivian N. ACholonu, Roseline N. Azubuike, Mary A. Okeke and
Lourita P. Agbo
Assessing the Relationship and Prediction of Manifold Facets of Analytic Relations to Academic Reading
Comprehension................................................................................................................................................................... 179
Md Kamrul Hasan, Md. Didar Hossain and Abdul Karim
Towards a Principled Use of L1 – Observing an EFL Teacher’s L1 Use in Rural Sabah, Malaysia.......................... 206
J. W. Ong and A. J. Ahmad Tajuddin
A Study of Saudi Advanced Academic Writing Students’ Perceptions of Research Essays, and Gaps in Their
Knowledge........................................................................................................................................................................... 223
Nida Qayoom and Mohammad Saleem
Mass Media Internships in Vocational Training of Students Majoring in Journalism............................................... 238
Nataliia A. Tsymbal, Nataliia M. Savchuk, Valentina I. Avramenko, Svitlana A. Sichkar and Iryna A. Denysiuk
Assessing the Efficacy of Extensive Reading during Study Abroad: A Time and Place for ER? ............................. 251
Byron O'Neill and Christopher Edelman
The Extent of Bullying Against Students with Learning Disabilities According to the Age Variable..................... 267
Mohamad Ahmad Saleem Khasawneh
Development of Instruments to Measure Mathematical Anxiety of Elementary School Students.......................... 282
Hafiziani Eka Putri, Mukhammad Ady Wahyudy, Aan Yuliyanto and Fitri Nuraeni
Situating "children-supporting-children" Platform in the Context of the Inclusive Agenda: A Phenomenological
Exploration .......................................................................................................................................................................... 303
Princess Zarla J. Raguindin and Li Yan Ping
The Impact of Educational Technologies on University Teachers’ Self-efficacy ........................................................ 323
Nataliia Saienko, Yuliana Lavrysh and Valentyna Lukianenko
UAE Elementary Teachers’ Use of ADHD Referral and Management Strategies...................................................... 337
Hala Elhoweris, Ahmed Mohamed, Osha Almuhairy, Rachel Takriti, Najwa Alhosani and Abdelaziz Sartawi
Exploring Accounting Teachers’ Views on the Quality of Accounting Prescribed Textbooks in South Africa ..... 353
Jabulisile C. Ngwenya and S’khumbuzo H. Mbili
Rurality and Exclusion in Ordinary Level Mathematics in Zimbabwe: A Document Analysis............................... 370
Simon Vurayai
Emergency Online Teaching in Economic and Management Sciences Necessitated by the COVID-19 Pandemic:
The Need for Healthy Relations in a Rural Schooling Context..................................................................................... 387
Habasisa Molise and Bekithemba Dube
1
©2020 The author and IJLTER.ORG. All rights reserved.
International Journal of Learning, Teaching and Educational Research
Vol. 19, No. 6, pp. 1-26, June 2020
https://doi.org/10.26803/ijlter.19.6.1
The Emergency Remote Learning Experience of
University Students in Indonesia amidst the
COVID-19 Crisis
Maila D. H. Rahiem
UIN Syarif Hidayatullah Jakarta
Jakarta, Indonesia
http://orcid.org/0000-0002-5618-2486
Abstract. This study aimed to explore and interpret the lived experience
of Indonesian university students in emergency remote learning (ERL)
during the COVID-19. Methods of the investigation was a qualitative
phenomenological approach involving 80 students from the Social
Science Education Program at a public university in Jakarta.
Understanding their experience was achieved through a rigorous analysis
of the participants’ diaries and reflective essays and an online focus
group. Results revealed that the students’ experiences fell into two
overarching themes, each with related sub-themes. The two identified
themes and subthemes were: (a) blended learning, with the subthemes of
e-learning, m-learning and conventional learning, and (b) paradoxical
learning, with the sub-themes of flexible learning and challenging
learning. By studying how university students learned during COVID-19,
we could help ensure the efficacy of ongoing ERL and better incorporate
similar programs in the future if this ever happens again.
Keywords: emergency remote learning; online learning; higher
education; COVID-19
1. Introduction
The COVID-19 global pandemic has created entirely unprecedented situations
that have greatly affected people’s lives. With doubts remaining at present over
how and when it will end, the question remains whether things will ever go back
to the way they were before the pandemic began with many skeptics remaining
doubtful (Lee et al., 2010).The World Health Organization (WHO) has warned
that COVID-19 is likely to haunt the world for a long time and that the planet will
only return to normal once a viral strain vaccine has been discovered. There are
currently ten candidate vaccines in clinical trials worldwide and 126 candidate
vaccines in pre-clinical trials (World Health Organization, 2020b). Anthony Fauci,
MD, director of the US National Institute of Allergy and Infectious Disease, said
that if all goes well, a vaccine might be available in November or December of this
2
©2020 The author and IJLTER.ORG. All rights reserved.
year (McCarthy, 2020; Reynolds, 2020). However, even after a vaccine has been
approved, there remains an immense challenge to generate enough of it for the
world's population (Felter, 2020).
Presently (Early June 2020), in Indonesia and some other countries, the number of
cases is still increasing at an alarming rate. The number of patients nationwide
who have tested positive is up to 36,406, with patients who have recovered
recorded as 13,213 people, and patients who have died totaling 2,048 people (Task
Force for the Acceleration of COVID-19, 2020). COVID-19 has spread to 34
provinces, meaning that it has spread to all of the provinces across the Indonesian
archipelago, and 391 out of 514 municipal districts have been infected (National
Agency for Disaster Management, 2020). So far, the percentage of Indonesia's
death rate, which is above 6 percent, is rated in the high category. The total
number of cases worldwide is also still increasing, with 7,127,753 confirmed cases
of COVID-19 spread across 216 countries, including 407,159 deaths, reported to
WHO (World Health Organization, 2020c).
As a result of the threat posed by COVID-19, educational institutions have hurried
to shift courses to virtual classrooms. Many institutions have chosen to cancel all
face-to-face classes, including laboratories and other learning activities, and have
required students to study remotely from home to help avoid the spread of the
virus that causes COVID-19. The UN Educational, Scientific, and Cultural
Organization(UNESCO, 2020) estimated that 129 countries had implemented
national school closures (data from June 10, 2020). These nationwide closures have
impacted almost 63.3 percent of the world's student population or 1,109,020
learners. Several other countries have implemented localized closures impacting
millions of additional learners (approximately 488,198,235 learners). The total
number of affected learners due to nationwide closures and localized closures
totals 1,725,082,528. The trend of nationwide closures has decreased from
previous figures recorded on April 1 – 4, 2020, where 1,598,099,000 learners, or
91.3% of total enrolled learners, were recorded as being affected due to the
nationwide closures policy in 195 countries. However, the total affected number
(nationwide and localized) is relatively the same, starting from March 26, 2020.
This data indicates that although the national closures policy has begun to be
revoked in some areas and regions, schools and local governments have
continued to implement remote learning.
In Indonesia, nationwide school closures began on March 23, 2020, while localized
closures in some provinces, e.g., DKI Jakarta, Central Java, Banten, West Java, and
Aceh, started early on March 16, 2020 (Kumparan, 2020). The national school
closures have impacted 60.2 million learners and 2.3 million educators who study
or teach at 425,451 educational institutions from early childhood to higher
education.
3
©2020 The author and IJLTER.ORG. All rights reserved.
Table 1: Number of affected academic institutions, learners and
educators in Indonesia
(Ministry of Education and Culture, 2019, 2020c, 2020a, 2020d, 2020b)
Level
Academic
Institutions
Learners Educators
Early Childhood Education 202,991 6,543,758 666,678
Elementary School 149,435 25,203.371 556,969
Junior High School 40,559 10,112,022 302,116
Senior High School 13,495 4,976,127 212,715
Vocational School (Equivalent to Senior High
School)
14,301 5,242,851 217,986
College/University 4,670 8,043,480 294,820
TOTAL 425,451 60,121,609 2,251,284
School closures are based on the previous studies of influenza outbreaks that
resulted from social interactions between students and teachers that disrupted the
learning process (Jackson et al., 2016). Cauchemez et al., (2008, 2009) quantified
the role of schools in influenza epidemics and predicted the effect of school
closures during a pandemic in France. They found that sustained school closures
during a pandemic could reduce the total number of cases by 13–17 percent (18–
23 percent in children), while during the height of the attack the rate could be as
much as 39–45 percent (47–52 percent in children). The effect of school closures
would be minimized if it proved difficult to sustain low contact rates among
children for a prolonged time. Earn (2012) looked at the associations between the
occurrence of H1N1 (pH1N1) pandemic influenza in Alberta, Canada in 2009 and
school closures or weather changes, and calculated the impact of school closures
and weather changes on the transmission of pH1N1. Mathematical models
indicated that school closures decreased transmission among school-aged
children by more than 50 percent, which was a critical factor in interrupting
transmission. The models also showed that seasonal changes in weather
conditions had a significant effect on the epidemic's temporal structure.
School closure is one of the critical components of numerous non-pharmaceutical
prevention strategies in many countries to help reduce the number of cases and
slow the spread of the disease, mainly if vaccines or antivirals are not available or
are ineffective due to resistance (Glass et al., 2006; Institute of Medicine, 2006;
Moscona, 2005; Sadique et al., 2008). School closure policies in various countries
occur in quick succession as a measure to contain the COVID-19 virus. Lee (2020)
argues that school closures alone may not be enough to stop the outbreak, but that
if it continues for at least eight weeks, could delay the outbreak peak by up to a
week, providing adequate time for a second more successful intervention, such as
vaccination.
In this study, the researcher used the term "Emergency Remote Learning (ERL)" to
illustrate the education that took place during school closure, not online or virtual
4
©2020 The author and IJLTER.ORG. All rights reserved.
learning, since well-planned online learning experiences are substantially
different from those that are delivered online in response to a crisis or catastrophe.
Hodges et al. (2020) call it Emergency Remote Teaching (ERT). ERT is a temporary
change from instruction delivery to alternative delivery due to crisis
circumstances. It requires the use of entirely remote teaching approaches for
instruction or education that would otherwise be provided face-to-face or as
blended or hybrid courses, and which will revert to that model once the crisis or
emergency has finished. The primary objective in this context is not to re-create a
stable educational environment, but rather to provide immediate access to
education and training in a manner that is easy to develop and easily accessible
during an emergency or crisis. Millman (2020) described the situation as
emergency remote teaching and learning — or "pandemic pedagogy."
This study focused on understanding how remote learning during COVID-19
pandemic has been conducted in a tertiary education institution by analyzing
university students' perspectives and experiences. The limited research was
conducted recently in Indonesia during the midst of the COVID-19 pandemic.
Most of the existing research conducted is in Indonesian, published in Indonesian
journals, focuses on either primary or secondary education, or education in
general (the level of education is not explained). Examples of this include,
"COVID-19 impact on the implementation of online learning in primary schools"
(Aji et al., 2020); "Education during the COVID-19 pandemic" (Khasanah et al.,
2020); "Studying Mathematics in the COVID-19 era" (Abidin, 2020); and "Online
learning in the middle of the COVID-19 pandemic" (Firman & Rahayu, 2020).
There are also several articles on learning in higher education during the COVID-
19 pandemic. These articles are written in Indonesian, with limited data and have
not been published through peer-reviewed processes such as “Students’ learning
typology in online learning”(Ramdhan et al., 2020); “Google Classroom-online
learning of Biology education students during the COVID-19 outbreak”(Suhada
et al., 2020); “Ordinal Regression analysis to see the effect of online learning media
on students’ enthusiasm in the COVID pandemic era”(Meiza et al., 2020).
Some articles published in peer-reviewed Indonesian journals focus on online
learning of COVID-19 at the college level. However, they were published in April,
where the learning from home experience had only been in operation for a few
weeks. Experiences from a short implementation time are unlikely to offer a full
picture of home-based teaching and learning activities, while there was also
minimal use of references. The published articles are “The impact of COVID-19
online learning activities in a Christian university in Indonesia”(Windhiyana,
2020); “Implementation of the performance model of DeLone and McLean
Knowledge Systems For Zoom application-based learning systems during the
COVID-19 pandemic”(Hidayatullah et al., 2020); “Perception of students
majoring in Early Childhood Islamic Education towards online Lectures in the
COVID-19 pandemic period”(Anhusadar, 2020); and “Variations of models and
learning platforms for prospective teachers during the COVID-19 pandemic
period”(Gunawan et al., 2020).
5
©2020 The author and IJLTER.ORG. All rights reserved.
Figure 1. Theoretical Model
The theoretical model outlined in Figure 1 shows what the related studies have
been performed, what this study was intended to do, and the research questions
raised by the context analysis and previous research. Recent studies have
described school closure as one of the key components of a range of non-
pharmaceutical prevention approaches aimed at minimizing the number of cases
and preventing the spread of disease. Some of the previous emergency remote
learning studies concluded that ERL is a temporary learning mechanism, a
transition from in-person instruction to substitute delivery due to an emergency.
Nonetheless, little is known on how to apply remote learning in higher education
during the outbreak of COVID-19 and how students perceived and experienced
ERL implementation.
Research on the implementation of home learning during COVID-19 in Indonesia
has been listed above. None of those papers use the term remote learning or ERL.
The use of the term affects the analysis and understanding that is predicated in
these studies. It is crucial to make a distinction, as the term refers to a unique and
specific event (Manfuso, 2020). This research filled the gap in knowledge (theory)
of ERL at the university level in Indonesia with insights and observations
collected from students who had been studying from home for an extended
period (more than ten weeks), had undergone an online mid-term exam, and was
at the end of the semester. Other than that, this research disseminated its
findings in English articles in international journals, so that more audiences could
read and learn from the research findings.
6
©2020 The author and IJLTER.ORG. All rights reserved.
This research also filled the practical gap on how to successfully implement ERL
in higher education to meet learners' needs. This study's main question is: how
university students experienced ERL due to the COVID-19 crisis? Through
knowing how university students learned in the emergency COVID-19, we could
improve on in-going practice. The students' feedback is also critical to improve
the learning process in the future if such a situation was ever to occur again. It is
unknown how long ERL will continue or what kind of other problems we may
face in the future, so educational institutions and students alike must be better
prepared for any such eventuality.
2. Materials and Methods
The current study is presented as part of a larger project on ERL in tertiary
education in Indonesia during the COVID-19 outbreak. The overall aim of the
study was to explore the implementation of ERL in tertiary education in
Indonesia. The study employed a qualitative method of the phenomenological
approach. This phenomenological analysis aims to collect and examine the lived
experience of university students who have learned from home, due to the
COVID-19 crisis, for more than two months. These experiences provided a new
level of insight into ERL's implementation due to the outbreak of COVID-19.
A phenomenology is a philosophical approach to the analysis of phenomena that
are part of human awareness (Arbnor & Bjerke, 2011). In phenomenology, the
researcher attempts to provide a direct description of someone's experience as it
is, without considering his or her psychological origin and the causal explanations
that the scientist may be able to provide (Merleau-Ponty & Landes, 2013).
Phenomenology is equipped to understand the subjective interpretations of the
participants of the fundamental object of the study by throwing light on the
significance applied to their lived experience and exploring the specific meanings
underlying the empiric variants of a given phenomenon rather than merely
applying the technique (Casmir, 1983; MacDermott, 2002).
Participants in this research are 80 students who have studied social science
education at a public university in Jakarta. The researcher employed a purposive
sampling method to select the participants. The researcher understood the
purpose of the study in such a way that she relied on her judgment to classify
qualified participants in a particular profile. The various explanations why the
researcher chose the participants are as follows: the first reason was the ease of
access for the researcher to collect data due to the COVID-19 situation, which
restricted the researcher's ability to gather data. The second reason, they were
semester four students, which means that they were in the middle of their
course (undergraduate education usually takes eight semesters). These students
already have enough learning experience and were still taking compulsory and
additional courses (semester seven students practice teaching, and in semester
eight, write a thesis). Third, students at this university come from Jakarta and
other provinces and different backgrounds. This diversity will help enrich the
data further. Lastly, the researcher intentionally chose students from educational
programs because they were prepared to become teachers, and their opinions are
important because their answers address how an educator should teach at a time
like this.
7
©2020 The author and IJLTER.ORG. All rights reserved.
These students are in the fourth semester and came from two classes, 4A and 4B.
These two groups have different concentrations, 4A is Sociology, and 4B is
Geography. The name of the university and the names of the students have been
hidden; to protect the identity of the participants and to provide them with the
opportunity to speak freely. Participants were aware of the scope and intent of the
study, and they were allowed to withdraw from the research anytime they chose
if they felt dissatisfied with it.
Data collection was carried out using creative methods because the researcher was
unable to conduct direct interviews due to the large-scale social restrictions that
were in place during the research period, data collection was completed by asking
participants to write a diary of their daily learning activities for two weeks (4 - 18
May 2020. See Appendix 1. Students’ Diary). They were then asked to compose a
reflective essay about ERL (collected on May 20, 2020. See Appendix 2. Reflective
Essay).
The researcher paid close attention to data triangulation to develop a
comprehensive understanding of the phenomena. Therefore, the researcher used
multiple methods or data sources, as suggested by Patton (Patton, 1999). The data
source utilized, other than diaries and essays, was focus group discussions using
the Google Meet application. These discussions were conducted twice; each
meeting was for 120 minutes and 40 participants (on May 26 and 27, 2020). The
focus group discussions were also aimed as a member check to increase the
precision, reliability, validity, and transferability of the findings. The researcher
presented the study's overall findings and then opened a discussion session.
Participants were free to question if any incorrect information had been recorded
and were allowed to add or remove details. The participants either confirmed that
the summaries represented their beliefs, feelings, and experiences or did not
reflect their experiences. The researcher concluded that the study seems credible
after the participants confirmed the accuracy and completeness of the study.
The complete elimination of non-response bias from the study is impossible.
Mixing data collection methods was one of the researchers' efforts to avoid non-
response impact from the sample. The researcher also sought to eliminate bias by
designing strategies that would promote the involvement of selected individuals.
The researcher used the NVivo program for data management and analysis.
NVivo is useful in organizing data and helped the researcher make sense of it
throughout the research since the data for this qualitative research was extensive
(80 participants). The researcher also made a memo in the NVivo system that
allowed for the possibility to document ideas and analyze thoughts, perspectives,
and observations of students on online learning. This analytical memo was
completed on an ongoing basis, every time data (journal or essay) was submitted.
Analytical memos provide a means for the researcher to record their thoughts
during the research process and to code memos as additional data for the study
(Saldaña, 2016).
In analyzing the data, the researcher adopted a two-stage coding model
conducted by Miles, Huberman and Saldaña (Miles et al., 2014) and Saldaña
(Saldana, 2009; Saldaña, 2016). The first cycle coding was initially assigned to the
8
©2020 The author and IJLTER.ORG. All rights reserved.
data, followed by the second cycle codes in which the initial codes were grouped
into meaningful categories, themes, or constructs. The two stages of coding are
not a single linear event; the qualitative analytical process is cyclical. In practice,
in the first cycle, the researcher coded each essay and journal individually. After
that, the researcher compared the data and detailed the code into a sub-code if
necessary. In the second cycle, the researcher reorganized and re-analyzed the
data coded in the first coding cycle. The primary aim of second-cycle coding was
to establish a sense of categorical, thematic, conceptual, and theoretical structure
from the first-cycle code series. The researcher changed codes, added new codes,
and dropped a few codes to conclude themes of research findings. Finally, the
researcher made the summative synthesis assumptions and declarative claims.
The assertions are based on the researcher's insights and observations, supported
by evidence from the data corpus.
3. Results
How did university students experience ERL during the COVID-19 crisis? Results
revealed that students’ experiences fell into two overarching themes, each with
related sub-themes. The two identified themes and subthemes are: (a) blended
learning, with the subthemes of e-learning, m-learning, and conventional
learning, and (b) paradoxical learning, with the sub-themes of flexible learning
and challenging learning.
Figure 2. Research findings
The researcher classified 25 codes, in the first cycle coding, from the data gathered
from 80 diaries, 80 reflective essays, and two focus group discussions. The
researcher then grouped related codes into five groups, which included: e-
learning, m-learning, traditional learning, flexible learning, and challenging
learning. The researcher summarized two key themes of this study's results in this
second coding process: blended learning and paradoxical learning. The researcher
concluded that during ERL, students experienced blended learning approaches,
and their experiences were paradoxical. They enjoyed learning from home, but
also saw it as challenging too. The following is an illustration of the process
of codification, thematic discovery, and the construction of assertions.
Emergency Remote Learning during
COVID-19 Outbreak
Blended
Learning
e-learning m-learning
Conventional
Learning
Paradoxical
Learning
Flexible
Learning
Challenging
Learning
9
©2020 The author and IJLTER.ORG. All rights reserved.
Figure 3. Data analysis process
10
©2020 The author and IJLTER.ORG. All rights reserved.
3.1 Blended Learning
Of the twenty-five codes found in the first coding cycle (as shown in Figure 2
above), thirteen are related to the media or methods used during the ERL. The
researcher then grouped codes that show similar meaning in categories: e-
learning, m-learning, and conventional learning. The researcher concluded from
the categories that learning remotely during the COVID-19 situation involved a
mix of technological learning media and conventional methods, or using blended
materials.
Table 2. Learning media/methods & frequency of being mentioned by participants
Media/Methods Frequency Categories
Reading E-book & E-Journal 56 e-learning & m-learning
YouTube & Video 52 e-learning & m-learning
Online Meeting (Zoom or Google Meet) 50 e-learning & m-learning
Browsing Internet 42 e-learning & m-learning
Educational Website 35 e-learning & m-learning
Podcast 33 e-learning & m-learning
Google Classroom 35 e-learning
WhatsApp 42 m-learning
Social Media 36 m-learning
Audio Call 32 m-learning
Assignments from Lecturers (exercises,
quizzes, and exams)
58 Conventional Learning
Making Notes 40 Conventional Learning
Module/Unit 33 Conventional Learning
From the coding of the data in the NVivo program, researchers have discovered
how many frequencies a single code of all the existing data appears. Table 2 shows
which media or methods are most mentioned repeatedly by students.
Reading e-books and e-journals was the most common learning methods
experienced by the students. Students also learned from YouTube and TV series
or movie documentaries. Usually, lecturers provided students with YouTube
links, asked them to study the video, and make a summary report. E-Books, E-
Journals, and YouTube videos were used to help explain learning about the
subject or giving instructions. Other platforms used to deliver the lecturers'
instructions or explanations were through WhatsApp (the lecturer recorded voice
notes and shared them in the WhatsApp group) and through the module or unit
of work being studied.
11
©2020 The author and IJLTER.ORG. All rights reserved.
Three platforms were commonly used for classroom discussions: online meetings
in Zoom or Google Meet, WhatsApp, and Social Media. They interchangeably
used the term Social Media for WhatsApp also, while a few referred to Facebook
when mentioning about Social Media. Not all the lecturers arranged synchronous
discussions using online platforms. Many students complained about it; they felt
that learning was ineffective without discussion. WhatsApp was the most
preferred media for discussion. In WhatsApp, the discussion could be
synchronous or asynchronous. The discussion could be made using either voice
notes or texts.
In order to enrich learning, students have tried to browse for more reading
material online by searching the internet and by visiting educational websites. If
they did not understand the subject, they would invariably contact their
classmates, asking them to explain the subject. The respondents believe that
YouTube videos are a good source of learning. Some listened to podcasts and
explained that this was as a result of needing to be more creative in expanding
their understanding. In measuring student learning progress, lecturers have
delivered assignments and quizzes to students through WhatsApp and Google
Classroom. Also, conventional methods were used for evaluation. The most
common conventional tasks assigned to students are the memorization of texts
and the writing of a summary.
The data reveals that emergency remote learning has not always been entirely
conducted online using sophisticated information technology; students have
often learned using conventional media and methods. The students take notes,
use paper, and pens to help them understand learning materials more efficiently.
They took notes on the video, sound recordings, e-books, PowerPoint
presentations, and journals provided by the lecturers. They felt that, in general,
they had not learned anything if they did not write it in a book. They seemed to
be able to understand more quickly when they wrote down what they had learned
in a notebook. Participant 4B24 explained about making a summary of learning
using paper and colorful pens and markers:
"If the lecturer has given material in the form of power points, modules,
and so on. I put together a summary of the material. And if there was a
discussion on WhatsApp or Zoom, I try to write notes about things that I
thought were important. Just like the previous way of learning (before
the COVID-19 crisis), I used colored pens or markers to write notes, so I
wouldn't get bored while reading notes." (4B24)
Whereas 4B35 said that she had made a mind map of the content she had learned
in order to be able to pass the mid-term and final exam. She added that she placed
her notes on the wall, and every morning she memorized and learned them:
“Although the mid-term test and end semester exams will be done online,
I will still make a mind map to test my understanding of the material... for
the final exam, I will make keywords from each material that I have
studied, I will make notes in my paper and paste them on the wall, and
every morning I will memorize and learn.” (4B35)
Students also conventionally learnt using modules or units, not a printed version,
but an e-module in PDF format. Two lecturers provided these e-modules
12
©2020 The author and IJLTER.ORG. All rights reserved.
containing discussions, quizzes, and a summary. The aim was to help students
learn independently. They can test their comprehension by answering the quiz in
the module and checking its accuracy by matching the answer key on the last page
of the module. The lecturers shared the module every 1-2 weeks. Some students
printed the modules or made a hand-written summary of the modules. The
students also explained that they felt more like they had studied when they have
paper, pen, and writing in a book.
"In the pandemic from the COVID-19 crisis, my learning is using
modules or PowerPoint presentations that I have obtained from lecturers
or notes from my colleagues." (4C27)
"I try to understand the tasks or modules provided, read all through them,
yet I still don't get it." (4C30)
Another conventional learning method is lecturers giving students assignments.
Students complained they actually had to finish too many assignments.
“Lecturers have been giving assignments online, a lot of them. Some
lecturers only give assignments continuously.” (4B21)
In ERL, blended learning was used for instruction and explanation, discussion,
evaluation, and enrichment. The researcher mapped out the thirteen
media/learning methods mentioned by the students in diaries and essays and the
purpose of their use. Here is the following map:
Figure 4. Blended media in ERL and their purpose of use
13
©2020 The author and IJLTER.ORG. All rights reserved.
WhatsApp is a prevalent medium used for all these learning purposes.
WhatsApp is used to explain, discuss, evaluate and enhance learning. In addition,
WhatsApp is also used for almost all courses during the ERL period. Only one of
the eight courses in Class 4A and Class 4C did not use the WhatsApp. This
assertion is derived from tabulations made on the types of learning platforms
used for each subject.
Table 3. Courses and learning platforms
No CLASS 4A CLASS 4B
COURSES LEARNING
PLATFORMS
COURSES LEARNING
PLATFORMS
1. The New
Indonesian
History
• YouTube
• Assignment via
WhatsApp
• Assignment via
Email
The New
Indonesian
History
• YouTube
• Assignment via
WhatsApp
• Assignment via
Email
2. Study &
Learning
• Google Classroom
• YouTube
• Google Meet
• WhatsApp
Study &
Learning
• Google Classroom
• YouTube
• Google Meet
• WhatsApp
3. Educational
Psychology
• Module
• Email
• Google Meet
Educational
Psychology
• Module
• Email
• Google Meet
4. Regional
Geography of
the World
• Reading Material
• Academic
Instruction System
• WhatsApp
Regional
Geography of
the World
• Reading Material
• Academic
Instruction System
• WhatsApp
5. Learning
Strategies
• Assignment
• Google Classroom
• WhatsApp
Learning
Strategies
Google
Classroom
WhatsApp
• Assignment
• Google Classroom
• WhatsApp
6. Practicum
Reading the
Quran
• Assignment
• WhatsApp Email
Practicum
Reading the
Quran
• Assignment
• WhatsApp Email
7. Entrepreneurs
hip
• Google Scholar
• WhatsApp Group
• Voice note
Entrepreneur
ship
• Google Scholar
• WhatsApp Group
• Voice note
8. Sociology of
Religion
• Email
• Voice Note
• WhatsApp
• PowerPoint
Presentation
Cartography • Lecturing in Zoom
Meeting
• Google Meet
• Assignment
• WhatsApp
During the semester, students from class 4 A and 4 B studied eight courses, seven
of which were the same for both classes and taught by the same lecturers. The
seven courses taught in Class 4A and Class 4B were: New Indonesian History,
Study and Learning; Educational Psychology; Regional Geography of the World;
Learning Strategies; Practicum Reading the Quran; and Entrepreneurship. The
distinctions were: Class 4A took Sociology of Religion, and Class 4A took
Cartography.
14
©2020 The author and IJLTER.ORG. All rights reserved.
3.2 Paradoxical Learning
The second main theme of the findings is “paradoxical learning.” The researcher
used this concept to describe how students had paradoxical perspectives and
insights into learning. On the one hand, they saw ERL as flexible, but on the other
hand, they also saw it as challenging.
Table 4. Students’ insights & observations on Paradoxical Learning
Insights/Observations Frequency Reasons
Time Management (Flexibility) 83 Flexible Learning
Family Time 52 Flexible Learning
Exercising 51 Flexible Learning
Comfortable & Quiet Place 42 Flexible Learning
Break & Rest 40 Flexible Learning
Refreshing 36 Flexible Learning
Technology Barriers 85 Challenging Learning
Cost of Internet 83 Challenging Learning
Overload Assignments 73 Challenging Learning
Trouble Finding Materials 51 Challenging Learning
Tiring 49 Challenging Learning
Noisy & Disturbing 42 Challenging Learning
Table 4, we can see how many times students reported certain insights or
observations about ERL paradoxically of all the existing data. The researcher used
color codes to show how they saw learning as two polar opposite scales. The two-
yellow color coded cells are time management and overload tasks. The three-
orange color coded cells are family time, comfortable and quiet, and noisy and
disturbing. The three-green coded cells are exercising, break and rest, and are
tiring. Four-blue coded cells are refreshing, technology barriers, internet costs,
and trouble finding materials. The segregated color-coded data indicates how
students had paradoxical perspectives and insights into learning.
On the one hand, students saw ERL as flexible; on the other hand, they also saw
it as challenging.
Figure 5. Flexibility versus overloaded
Flexibility
and time
management
Overloaded
with
assignments
Flexible
Learning
Challenging
Learning
15
©2020 The author and IJLTER.ORG. All rights reserved.
Students said that learning remotely at home gave them the flexibility to manage
their time.
“The flexible learning schedule matches our lifestyle; young people.
Lecturers and students are increasingly turning to online learning as a
viable alternative to study anywhere, anytime.” (4B26)
Contradictory to the amount of flexibility, they also argued that the lecturers
overloaded them with assignments, and they, therefore, found it difficult to
manage their time.
"It's not necessarily online learning or lectures that we were given, but
now there are remote assignments (online). And the tasks that were given
never stopped every week, we were given assignments by each subject, and
we were chased by deadlines. In my view, it can make students
frustrated and depressed. As we now know that we need to retain our
body's immunity to avoid the virus, and the stress or pressure caused by
the assignments may, in my opinion, disrupt the students' immunity
from the body." (4C33)
Figure 6. Family time, comfortable & Quiet versus noisy & disturbing
The students said that during remote learning, they have a lot of family time and
could study in a comfortable and quiet place.
“Studying online makes it easy for us to find places that make it easier to
think about, such as open spaces, indoor rooms, and family rooms at
home.” (4B17)
In contrast to this, they also felt disturbed by their siblings and the noise at home.
“The home environment cannot be controlled when there is a live lecture
that uses applications such as Google Meet and synchronous conversation.
The situation of the house is crowded. It's very annoying, in my opinion,
and it makes me unable to focus.” (4B18)
Family Time
Comfortable
& Quiet Place
Noisy and
Disturbing
Flexible
Learning
Challenging
Learning
16
©2020 The author and IJLTER.ORG. All rights reserved.
Figure 7. Exercising, break & rest versus tiring
Some students did some regular exercise at home. They spared extra time for self-
care.
"If there are no assignments, I will take the time to exercise and bask in
the healthy morning sun." (4B28)
"After I wake up, I do regular exercise or aerobic exercise. I'm doing this
regularly. Learning from home allows me to exercise more." (4C38)
Because the students were staying at home, they were able to take a break during
learning. They did not want to push themselves too hard to study. Their main
focus was to keep healthy and not be too tired.
“In addition, I also tried not to be too tired and forced to study because of
the consequences of the Covid-19 pandemic, I had to be careful not to fall
ill. I also don't push myself too much when I'm tired of studying.” (4B15)
Other respondents, however, felt that compared to face-to-face learning, remote
was much more tiring.
"I want to complain that this distance learning is very ineffective and
makes me even more exhausted." (4C20)
“A lot of tasks! The lecturer should consider the fact that a student is
staying at home to avoid illness or infection from happening. The lecturer
gave us lots of assignments excessively. It kills us. Not because of the
coronavirus that causes us to die. May Allah Amen protect us.” (4C18)
Figure 8. Refreshing versus problems
Exercising
Break &
Rest
Tiring
Flexible
Learning
Challenging
Learning
Refreshing Technological
barriers
Cost of the internet
Trouble finding
materials
Flexible
Learning
Challenging
Learning
17
©2020 The author and IJLTER.ORG. All rights reserved.
The students said they tried to manage their time and refresh themselves
whenever they were tired. They said this is what made learning from home
more flexible.
“Learning activities must be interspersed with activities that are fun,
refreshing. In order not to be boring, I always prepare fun activities (in my
opinion). This activity can also be a reward or reward when I have
completed an assignment or studied material.” (4C21)
They also complained, however, about technological disruption while studying,
expensive internet costs and difficulties in obtaining learning materials during the
ERL.
"The difficulty of accessing the Internet, I was at home during this
distance learning process. This makes me experience delays in getting
information because access to the Internet in the village is a bit difficult,
the internet connection is bad in my area." (4B18)
"It's very pricey to spend money on the internet data plan. I'm a student
who gets minimal pocket money, especially on staying at home like this, I
don't get any income from being a Scoutmaster. That makes it hard for me
to buy a data plan. I'm also envious of campuses that provide a free data
plan of up to 150 thousand rupiahs per month." (4B12)
"This policy has a side where I don't like it, for example, because I can't
go anywhere, so I have trouble finding references that I need to complete
my assignments. Learning resources are also limited and can only refer to
the internet and journals that aren't always right" (4B19).
4. Discussion
Indonesia reported the first two confirmed cases of COVID-19 in a Jakarta suburb
on March 2, 2020 (World Health Organization, 2020a). The situation became
increasingly dangerous in mid-March. President Jokowi, in a press conference at
the Bogor Presidential Palace on Sunday, March 15, 2020, called on local
governments to issue policies to prevent the spread of the coronavirus. One of the
policies recommended was to temporarily halt teaching and learning in schools
and universities and encourage students to study at home. As a follow-up to the
president's direction, on March 20, 2020, the Minister of Education and Culture
issued a Circular Number 36962/MPK.A/HK/2020 concerning online learning
and working from home in the context of the prevention of coronary disease
spread. Following this rule, the campus, which was the focus of this research,
began to close on March 23, 2020.
Learning was suddenly moved to remote learning. Remote learning has not been
anticipated, and most of the course syllabus was not designed to be an online or
for a distance learning experience. Without preparation or training, lecturers
designed and implemented remote learning programs. Similarly, students did not
have the opportunity to be prepared for this transition in learning. Despite all of
these limitations, everybody has been trying their best to make the situation a
success, quick and reliable temporary access to education in unprecedented time.
18
©2020 The author and IJLTER.ORG. All rights reserved.
Lecturers began using the media or the methods they were familiar with and
comfortable using. It may not be perfect, but students are resilient, the educator is
resourceful, and everyone has to make the most of what they have (Juliani, 2020).
This study reveals that learning remotely during the COVID-19 situation involved
a mix of both conventional and non-conventional methods. Technological
learning media was used combined with pen and paper methods. The use of
blended materials is what the researcher concluded as the first theme.
Blended learning is the application of more than one method, strategy, technique,
or media in education (Sadeghi et al., 2014; Thiele, 2003). It is a mixture of
conventional lectures or tutorials and web-based material (Concannon et al.,
2005); when delivering information, traditional face-to-face learning is combined
with technology (Farrel, 2006). In line with the government's policy of social
distancing, learning has taken place at home. Teachers and students separated by
physical distance but connected through technology, equipment, and resources.
Nonetheless, the data indicate that learning was not entirely online; students were
still studying in conventional ways. Conventional here does not mean that there
have been face-to-face lectures, but conventional learning methods for taking
notes, summarizing learning and memorizing notes.
In this study, the researcher distinguished between e-learning and m-learning
based on the devices used. E-learning means that the educational model used is
based on the use of electronic media and devices as tools for improving the
availability of training, communication, and interaction, and that helps to accept
new ways of understanding and learning (Salloum et al., 2019). An essential
element of e-learning is the use of electronic media, and e-learning is currently
explained as learning that is used through various computer devices, such as
computers, mobile phones, tablets, and virtual environments. M-learning focuses
on learning mobility, engagement with portable technology, and learning that
represents an emphasis on how society and its institutions could accommodate
and sustain an increasingly mobile population (Mehdipour & Zerehkafi, 2013). By
using mobile apps, learners can learn anywhere and anytime (Crescente & Lee,
2011). In Indonesia, internet users have reached 150 million people, with a 56
percent penetration spread across the region. There is only a small difference with
the number of mobile internet users, which amounts to 142.8 million people, with
a penetration percentage of 53 percent (Ministry of Communication and
Information Technology Republic of Indonesia, 2020). So, most people are linked
to the internet through mobile phones. Many homes do not have laptops or PCs,
but they are still connected online using mobile devices. The distinction between
e-learning and m-learning clarifies this phenomenon. In case of emergencies,
remote learning students have explained that they are connected using mobile
devices or tethered from a cell phone when they need to work from a laptop.
Blended learning is supposed to make learning more efficient by integrating
technology-based learning with traditional style learning. Some research has
identified many benefits of blended learning. Many studies have been carried out
to compare conventional teaching methods with blended learning, and different
results have been obtained. In some research, the participants' test scores did not
differ between the two groups (Khan, 2001). Thiele's study showed that the
19
©2020 The author and IJLTER.ORG. All rights reserved.
student knowledge scores in the blended teaching method were higher than the
lecturing method, but that the difference was not statistically significant (Thiele,
2003). Sedeghi et al. contrasted students' learning and satisfaction with teaching
and e-learning with traditional teaching methods. The results showed that the
blended method is effective in increasing the student learning rate (Sadeghi et al.,
2014). McPhee and Pickern (2017) concluded how ICT could help international
students' learning experience. Kirkwood (2009) claimed that ICT could make
doable learning tasks or situations that would otherwise be extremely difficult to
achieve and encourage an incremental improvement in learning outcomes.
Stephenson, Brown, and Griffin (Stephenson et al., 2008) argued that most e-
learning seemed to mimic or complement established academic practices, mainly
when used in 'blended' contexts. Another study suggests that a combination of
face-to-face training with e-learning is more flexible than other methods (Garrison
& Kanuka, 2004).
This ERL might not have been the ideal one. Blended learning has been
questioned as to the best solution to studying during the pandemic when
universities are closed. The advantages of blended learning described in the
studies, as mentioned above, are in contrast to the experience of students.
Afterward, we must remember that this blended learning took place in an
emergency with limited resources. As highlighted in the introduction, the
researcher refers to it as "emergency remote learning," not just "remote learning"
using blended materials. Before COVID-19, the classes studied by the researcher
rarely used e-learning and m-learning. E-learning and m-learning have emerged
as one of the alternatives to the problems of education during an outbreak of
coronavirus. E-learning and m-learning offer expanded opportunities for learning
versatility when students and educators are under the stay-at-home order.
However, there were challenges in using technology in learning. Students talked
about the technical barriers that hindered their learning due to technical problems
or the inability to use technological resources.
Students also discussed the shift in learning styles. Students were already
accustomed to face-to-face conventional teaching approaches. They claimed in
their writings that they felt that they were not learning since there was no lecturer
to guide them to understand the lesson. E-learning and m-learning involve
students working interdependently, in groups or independently to solve
problems, to work on projects, to meet individual needs, and to encourage
students to speak and choose. Students complained about this kind of
independent learning, doing assignments, and reading on their own. They were
familiar with the teaching-learning model, where lecturers gave learning
materials; students listened and took notes.
Another thing to think about is the availability of learning resources in Bahasa
Indonesia. Students said that they have trouble getting learning materials since
the libraries are closed. Universities have no access to paid online journals. So,
students are only using open-access materials.
The findings of this study reveal that university and its community were not well
prepared to face an emergency, such as the closure of the campus due to a
pandemic. Courses designed for conventional learning, the unusual use of
20
©2020 The author and IJLTER.ORG. All rights reserved.
technology in teaching and learning programs, students were not able to study
independently, limited access to learning materials in the native language of
learners, and lack access to high-quality reading resources (e.g., paid e-journals
and e-books). All of these issues need to be highlighted and addressed. Now and
in the future, the university should be more involved in information
communication technology. University learning should encourage students to
learn independently through a wide range of methods and media. Resources for
remote learning need to be strengthened, and students should get more access to
reading materials
Another theme that arose from this study is ERL during the outbreak of COVID-
19 is paradoxical. Paradoxical, according to the Cambridge dictionary 2020), is,
"Seeming impossible or difficult to understand because of containing two
opposite facts or characteristics." If something is paradoxical, it involves two facts
or qualities that seem to contradict each other (Collins Dictionary, 2020). The
university moved in-classroom-learning to online-based learning without enough
preparation. This rapid change was a test of organizational agility (Wu, 2020).
Students and lecturers were still adapting and looking for ways of teaching and
learning that were much better and easier. Educational planning in times of crisis
needs creative problem solving (Hodges et al., 2020). ERL needs much hard work,
but this is the only way forward. After some time, it is essential to review, develop
the current practice, and plan for future programs.
Learning from home might continue in the next few months, and this emergency
could happen again in the future. What are we supposed to do to make the ERL
better? Bao (2020) observed online teaching during the outbreak of COVID-19.
Bao has identified six educational strategies to improve student concentration and
engagement to achieve a smooth transition to online learning:
1. Preparing emergency preparedness plans for unexpected problems. If the
teaching plan A is not working because of a technical problem, try Plan B,
C, or any other plan.
2. Distributing the teaching content into smaller units to help the student
focus. The lecturers should divide the teaching content into several small
modules lasting approximately 20–25 minutes.
3. Emphasizing the use of "voice" in online teaching. Body language and
facial expressions are restricted in online instruction, and only the "voice"
is entirely functional. Therefore, in online teaching, the faculty should
properly slow down its speech to allow students to capture critical
knowledge points.
4. Working with teaching assistants and get online help from them. This
arrangement is also missing from the students' courses in this research.
There are no teaching assistants for the lecturers. Online teaching includes
the technology and professional expertise of those involved. Most faculties
at universities in Indonesia are not sufficiently trained or equipped to
operate online educational platforms, and support from teaching
assistants is especially relevant.
5. Strengthening the students' ability to learn outside the classroom. The
faculty has less control over online teaching. As a result, the progress of
21
©2020 The author and IJLTER.ORG. All rights reserved.
online teaching and its learning efficiency depends mainly on high-level
active learning outside the classroom. To this end, the faculty should use
various methods to moderately modify students' homework and reading
requirements to strengthen students' active learning outside the
classroom. Students complained about assignments in this research.
Lecturers and students need a dialogue to address what is currently
expected and should be achieved by two parties, students and lecturers,
to make learning more productive.
6. Efficiently combining online learning and offline self-learning. In the
offline self-learning process, students are expected to read the course-
specific literature and submit short papers based on their reading of
essential materials before class. The faculty should provide feedback on
student assignments and learn about students' cognitive learning levels.
In the online teaching phase, the faculty should use the student discussion
section to exchange their understanding based on their reading. Thus,
students will not learn ambiguous, fragmented, and superficial
knowledge. Instead, deep learning will be experienced during the
discussion. Students in this research complained that not enough lectures
and lecturers had given them too many assignments until they felt
overloaded. This should be communicated as to why assignments are part
of learning. However, lecturers should give feedback on the work of
students, so that they know the progress of their learning.
Bao (2020) also stressed the need to provide psychological support to the
university community. In this research, students viewed remote learning in both
a positive and bad light. In their diaries, they wrote about confusion, depression,
a devastating feeling, and some negative emotions about being isolated and
learning remotely. These feelings have to be addressed. The university should
provide psychological support to help students and staff manage their emotions.
Their anxieties need to be relieved in a variety of ways to ensure that they can
engage in online learning actively.
5. Conclusion
This study concluded that students experienced an ERL that utilized blended
strategies of e-learning, m-learning, and conventional learning techniques. Their
response to the process was paradoxical as they had both positive and negative
experiences. Lecturers used significantly limited media and methods to
implement learning. Moving instruction online could make teaching and learning
accessible anywhere, at any time, but the speed at which this transition to online
instruction is expected to happen is unparalleled. After experiencing ERL for
almost a semester, and while currently waiting for government policies on how
to learn next semester, it is an excellent time to learn from multiple viewpoints,
including students, about their experience. Some of the feedback from this
research are: preparing teachers and students for emergency learning, training
lecturers on using blended resources, designing learning curricula that can be
converted into online learning, developing student-independent learning skills,
and providing access to a wide variety of digital reading materials. This study's
findings cannot be generalized because they were based on a relatively small
number of university students from one department at one university. However,
22
©2020 The author and IJLTER.ORG. All rights reserved.
we assume that this work presents students' views and experiences on ERL in
several universities in Indonesia and probably in other countries, particularly in
developing countries. More research involving more students from various
universities should be initiated as a follow-up. It would also be useful to gain
insights from lecturers and students at different levels of education and discuss
the viewpoints of parents and other education stakeholders.
Funding: This research received no external funding.
Conflicts of Interest: The authors declare no conflict of interest in the design of
the study; in the collection, analysis, or interpretation of data; in the writing of the
manuscript, or in the decision to publish the results.
6. References
Abidin, Z. (2020). Belajar Matematika di Era Covid-19 [Studying Mathematics in the COVID-
19 era]. OSF Preprints. Retrieved from https://osf.io/nrbu7/
Aji, W., Dewi, F., Kristen, U., & Wacana, S. (2020). Dampak Covid-19 Terhadap
Implementasi Pembelajaran Daring Di Sekolah Dasar [COVID-19 impact on the
implementation of online learning in primary schools]. Edukatif: Jurnal Ilmu
Pendidikan, 2(1), 55–61.
https://doi.org/https://doi.org/10.31004/edukatif.v2i1.89
Anhusadar, L. O. (2020). Persepsi Mahasiswa PIAUD terhadap Kuliah Online di Masa
Pandemi [Perception of students majoring in Early Childhood Islamic Education
towards online Lectures in the COVID-19 pandemic period]. KINDERGARTEN:
Journal of Islamic Early Childhood Education, 3(1), 44–58.
Arbnor, I., & Bjerke, B. (2011). Methodology for Creating Business Knowledge. In
Methodology for Creating Business Knowledge.
https://doi.org/10.4135/9780857024473
Bao, W. (2020). COVID ‐19 and online teaching in higher education: A case study of
Peking University. Human Behavior and Emerging Technologies, 2(2), 113–115.
https://doi.org/10.1002/hbe2.191
Cambridge Dictionary. (2020). Paradoxical.
Casmir, F. L. (1983). Phenomenology and hermeneutics: Evolving approaches to the study
of intercultural and international communication. International Journal of
Intercultural Relations, 7(3), 309–324. https://doi.org/10.1016/0147-
1767(83)90035-4
Cauchemez, S., Ferguson, N. M., Wachtel, C., Tegnell, A., Saour, G., Duncan, B., & Nicoll,
A. (2009). Closure of schools during an influenza pandemic. The Lancet Infectious
Diseases, 9(8), 473–481. https://doi.org/10.1016/S1473-3099(09)70176-8
Cauchemez, S., Valleron, A.-J., Boëlle, P.-Y., Flahault, A., & Ferguson, N. M. (2008).
Estimating the impact of school closure on influenza transmission from Sentinel
data. Nature, 452(7188), 750–754. https://doi.org/10.1038/nature06732
Collins Dictionary. (2020). Paradoxical.
Concannon, F., Flynn, A., & Campbell, M. (2005). What campus-based students think
about the quality and benefits of e-learning. British Journal of Educational
Technology, 36(3), 501–512. https://doi.org/10.1111/j.1467-8535.2005.00482.x
Crescente, M. L., & Lee, D. (2011). Critical issues of m-learning: design models, adoption
processes, and future trends. Journal of the Chinese Institute of Industrial Engineers,
28(2), 111–123. https://doi.org/10.1080/10170669.2010.548856
23
©2020 The author and IJLTER.ORG. All rights reserved.
Earn, D. J. D. (2012). Effects of School Closure on Incidence of Pandemic Influenza in
Alberta, Canada. Annals of Internal Medicine, 156(3), 173.
https://doi.org/10.7326/0003-4819-156-3-201202070-00005
Farrel, M. (2006). Learning differently: e-learning in nurse education. Nursing
Management, 13(6), 14–17. https://doi.org/10.7748/nm.13.6.14.s12
Felter, C. (2020). What Is the World Doing to Create a COVID-19 Vaccine? Retrieved from
https://www.cfr.org/backgrounder/what-world-doing-create-covid-19-
vaccine
Firman, F., & Rahayu, S. (2020). Online learning in the middle of the COVID-19 pandemic
(Pembelajaran Online di Tengah Pandemi Covid-19). Indonesian Journal of
Educational Science (IJES), 2(2), 81–89. https://doi.org/10.31605/ijes.v2i2.659
Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative
potential in higher education. The Internet and Higher Education, 7(2), 95–105.
https://doi.org/10.1016/j.iheduc.2004.02.001
Glass, R. J., Glass, L. M., Beyeler, W. E., & Min, H. J. (2006). Targeted social distancing
design for pandemic influenza. Emerging Infectious Diseases, 12(11), 1671–1681.
https://doi.org/10.3201/eid1211.060255
Task Force for the Acceleration of COVID-19. (2020). Infografis COVID-19 (12 Juni 2020).
Retrieved from https://covid19.go.id/p/berita/infografis-covid-19-12-juni-
2020
Gunawan, Suranti, N. M. Y., & Fathoroni. (2020). Variations of models and learning
platforms for prospective teachers during the COVID-19 pandemic period.
Indonesian Journal of Teacher Education, 1(2), 61–70.
Hidayatullah, S., Khouroh, U., Windhyastiti, I., Patalo, R. G., & Waris, A. (2020).
Implementasi Model Kesuksesan Sistem Informasi DeLone And McLean
Terhadap Sistem Pembelajaran Berbasis Aplikasi Zoom Di Saat Pandemi Covid-
19 [Implementation of the performance model of DeLone and McLean
Knowledge Systems For Zoom application-based learning systems during the
COVID-19 pandemic]. Jurnal Teknologi Dan Manajemen Informatika, 6(1).
https://doi.org/10.26905/jtmi.v6i1.4165
Hodges, C., Moore, S., Lockee, B., Trust, T., & Bond, A. (2020). The Difference Between
Emergency Remote Teaching and Online Learning. Educause Review.
Institute of Medicine. (2006). Modeling Community Containment for Pandemic Influenza.
National Academies Press. https://doi.org/10.17226/11800
Jackson, C., Vynnycky, E., & Mangtani, P. (2016). The Relationship Between School
Holidays and Transmission of Influenza in England and Wales. American Journal
of Epidemiology, 184(9), 644–651. https://doi.org/10.1093/aje/kww083
Juliani, A. J. (2020). THIS IS NOT ONLINE OR DISTANCE LEARNING. Retrieved from
http://ajjuliani.com/this-is-not-online-or-distance-learning/
Khan, F. A. (2001). Problem-based Learning Variant: Transition Phase for a Large
Institution. National Library of Medicine, J. Pak Med Assoc, 51(8), 271–274. Retrieved
from https://pubmed.ncbi.nlm.nih.gov/11715887/
Khasanah, D. R. A. U., Pramudibyanto, H., & Widuroyekti, B. (2020). Pendidikan Dalam
Masa Pandemi Covid-19 [Education during the COVID-19 pandemic]. Jurnal
Sinestesia, 10(1), 41–48.
Kirkwood, A. (2009). E‐learning: you don’t always get what you hope for. Technology,
Pedagogy and Education, 18(2), 107–121.
https://doi.org/10.1080/14759390902992576
Kumparan. (2020). Anies Tutup Sekolah di DKI Jakarta Selama 2 Minggu untuk Cegah Corona
[Anies closes school in DKI Jakarta for 2 weeks to prevent Corona]. Retrieved from
24
©2020 The author and IJLTER.ORG. All rights reserved.
https://kumparan.com/kumparannews/anies-tutup-sekolah-di-dki-jakarta-
selama-2-minggu-untuk-cegah-corona-1t1V7X57VID
Lee, B. Y., Brown, S. T., Cooley, P., Potter, M. A., Wheaton, W. D., Voorhees, R. E.,
Stebbins, S., Grefenstette, J. J., Zimmer, S. M., Zimmerman, R. K., Assi, T.-M.,
Bailey, R. R., Wagener, D. K., & Burke, D. S. (2010). Simulating School Closure
Strategies to Mitigate an Influenza Epidemic. Journal of Public Health Management
and Practice, 16(3), 252–261. https://doi.org/10.1097/PHH.0b013e3181ce594e
Lee, T. H. (2020). Innovations in Care Creating the New Normal: The Clinician Response
to Covid-19. NEJM Catalyst, 19–21. https://doi.org/10.1056/CAT.20.0076
MacDermott, A. F. N. (2002). Living with angina pectoris - A phenomenological study.
European Journal of Cardiovascular Nursing. https://doi.org/10.1016/S1474-
5151(02)00047-6
Manfuso, L. G. (2020). From Emergency Remote Teaching to Rigorous Online Learning. Ed
Tech. Retrieved from
https://edtechmagazine.com/higher/article/2020/05/emergency-remote-
teaching-rigorous-online-learning-perfcon
McCarthy, M. (2020, June). Dr. Anthony Fauci: COVID-19 Will End and We Will Get Control
Over It. Healthline. Retrieved from https://www.healthline.com/health-
news/dr-anthony-fauci-this-will-end-and-we-will-get-control-over-it#6
McPhee, S., & Pickren, G. (2017). Blended learning with international students: a
multiliteracies approach. Journal of Geography in Higher Education, 41(3), 418–433.
https://doi.org/10.1080/03098265.2017.1331208
Mehdipour, Y., & Zerehkafi, H. (2013). Mobile Learning for Education: Benefits and
Challenges. International Journal of Computational Engineering Research, 03(06), 93-
101.
Meiza, A., Hanifah, F. S., & Natanel, Y. (2020). Analisis Regresi Ordinal untuk melihat
Pengaruh Media Pembelajaran Daring terhadap Antusiasme Mahasiswa Era Pandemi
Covid [Ordinal Regression analysis to see the effect of online learning media on students’
enthusiasm in the COVID pandemic era]. Retrieved from
http://sinta.ristekbrin.go.id/covid/penelitian/detail/361
Merleau-Ponty, M., & Landes, D. A. (2013). Phenomenology of perception. In
Phenomenology of Perception. https://doi.org/10.4324/9780203720714
Miles, M. B., Huberman, M. a, & Saldana, J. (2014). Drawing and Verying Conclusions.
Qualitative Data Analysis: A Methods Sourcebook.
Milman, N. B. (2020). Pandemic pedagogy. Phi Delta Kappan.
Ministry of Communication and Information Technology Republic of Indonesia. (2020).
Penggunaan Internet di Indonesia. Ministry of Communication and Information
Technology Republic of Indonesia.
https://aptika.kominfo.go.id/2019/08/penggunaan-internet-di-indonesia/
Ministry of Education and Culture. (2019). The 2019/2020 Early Childhood Education Statis.
Retrieved from https://doi.org/10.1017/CBO9781107415324.004
Ministry of Education and Culture. (2020a). 2019/2020 Junior High School Statistic.
Ministry of Education and Culture. (2020b). 2019/2020 Primary Education Statistics.
Retrieved from http://statistik.data.kemdikbud.go.id/index.php/page/sd
Ministry of Education and Culture. (2020c). 2019/2020 Senior High School Statistic.
Ministry of Education and Culture. (2020d). 2019/2020 Vocational School Statistic.
Moscona, A. (2005). Oseltamivir Resistance — Disabling Our Influenza Defenses. New
England Journal of Medicine, 353(25), 2633–2636.
https://doi.org/10.1056/NEJMp058291
25
©2020 The author and IJLTER.ORG. All rights reserved.
National Agency for Disaster Management. (2020). Infeksi COVID-19 Telah Menyebar di 34
Provinsi di Indonesia, Total Positif Jadi 3.512 Kasus [COVID-19 infections have
spread in 34 provinces in Indonesia, a total of positive to 3,512 cases]. Retrieved
from https://bnpb.go.id/berita/infeksi-covid19-telah-menyebar-di-34-
provinsi-di-indonesia-total-positif-jadi-3-512-kasus
Patton, M. Q. (1999). Enhancing the quality and credibility of qualitative analysis. Health
Services Research.
Ramdhan, D., Nasihudin, Rohaniawati, D., & Pratiwi, I. M. (2020). TIPOLOGI BELAJAR
MAHASISWA PADA PEMBELAJARAN DARING [Students’ learning typology in
online learning]. Digilib UIN Sunan Gunung Djati. Retrieved from
http://digilib.uinsgd.ac.id/30594/1/Artikel KTI dadan f ramdhan dkk .pdf
Reynolds, K. A. (2020). Coronavirus: Fauci says vaccine possible by end of 2020. Medical
Economics. https://www.medicaleconomics.com/news/coronavirus-fauci-
says-vaccine-possible-end-2020
Sadeghi, R., Sadaghat, M. M., & Ahmadi, F. S. (2014). Comparison of the Effect of Lecture
and Blended Teaching Methods on Students’ Learning and Satisfaction. J. Adv
Med Ed Prof, 2(4), 146–150.
Sadique, M. Z., Adams, E. J., & Edmunds, W. J. (2008). Estimating the costs of school
closure for mitigating an influenza pandemic. BMC Public Health, 8(1), 135.
https://doi.org/10.1186/1471-2458-8-135
Saldana, J. (2009). An Introduction to Codes and Coding. The Coding Manual for Qualitative
Researchers. https://doi.org/10.1519/JSC.0b013e3181ddfd0a
Saldaña, J. (2016). The Coding Manual for Qualitative Researchers (No. 14). Sage.
Salloum, S. A., Qasim M. A. A., Al-Emran, M., Abdel M. A., & Shaalan, K. (2019).
Exploring Students’ Acceptance of E-Learning Through the Development of a
Comprehensive Technology Acceptance Model. IEEE Access, 7, 128445–128462.
https://doi.org/10.1109/ACCESS.2019.2939467
Stephenson, J. E., Brown, C., & Griffin, D. K. (2008). Electronic delivery of lectures in the
university environment: An empirical comparison of three delivery styles.
Computers & Education, 50(3), 640–651.
https://doi.org/10.1016/j.compedu.2006.08.007
Suhada, I., Kurniati, T., Pramadi, R. A., & Listiawati, M. (2020). Google Pembelajaran Daring
Berbasis Google Classroom Mahasiswa Pendidikan Biologi Pada Masa Wabah Covid-19
[Classroom-online learning of Biology education students during the COVID-19
outbreak]. Retrieved from
http://sinta.ristekbrin.go.id/covid/penelitian/detail/363
Thiele, J. E. (2003). Learning Patterns of Online Students. Journal of Nursing Education,
42(8), 364–366.
UNESCO. (2020). COVID-19 Impact on Education. Retrieved from
https://en.unesco.org/covid19
Windhiyana, E. (2020). DAMPAK COVID-19 TERHADAP KEGIATAN
PEMBELAJARAN ONLINE DI PERGURUAN TINGGI KRISTEN DI
INDONESIA [The impact of COVID-19 online learning activities in a Christian
university in Indonesia]. Perspektif Ilmu Pendidikan, 34(1), 1–8.
https://doi.org/10.21009/PIP.341.1
World Health Organization. (2020a). Coronavirus disease 2019 (COVID-19) Situation
Report – 42. World Health Organization. Retrieved from
https://www.who.int/docs/default-source/coronaviruse/situation-
reports/20200302-sitrep-42-covid-19.pdf?sfvrsn=224c1add_2
World Health Organization. (2020b). Draft landscape of COVID-19 candidate vaccines -
15 May 2020. World Health Organisation. Retrieved from
26
©2020 The author and IJLTER.ORG. All rights reserved.
https://www.who.int/who-documents-detail/draft-landscape-of-covid-19-
candidate-vaccines
World Health Organization. (2020c). World Health Organization COVID-19 Dashboard.
Retrieved from https://covid19.who.int
Wu, Z. (2020). How a top Chinese university is responding to coronavirus. World Economic
Forum.
Appendix 1. Students’ Diary
Write every day about your daily learning routine. Illustrate and describe your
experiences in 300-500 words a day. If you like, you can add a picture.
Name:
Day/Date Learning Activities
Monday/4 May 2020
Tuesday/5 May 2020
Wednesday/6 May 2020
Thursday/7 May 2020
Friday/8 May 2020
Saturday/9 May 2020
Sunday/10 May 2020
Monday/11 May 2020
Tuesday/12 May 2020
Wednesday/13 May 2020
Thursday/14 May 2020
Friday/15 May 2020
Saturday/16 May 2020
Sunday/17 May 2020
Monday/18 May 2020
Appendix 2. Reflective Essay
Write your thoughts on the implementation of emergency remote learning that has been
going on for more than 2 months in 500-1000 words. Write down what you like and do
not like about ERL.
27
©2020 The authors and IJLTER.ORG. All rights reserved.
International Journal of Learning, Teaching and Educational Research
Vol. 19, No. 6, pp. 27-45, June 2020
https://doi.org/10.26803/ijlter.19.6.2
Examining the Fairness of Language Test Across
Gender with IRT-based Differential Item and
Test Functioning Methods
Burhanettin Ozdemir
Prince Sultan University, Riyadh, Saudi Arabia
https://orcid.org/0000-0001-7716-2700
Abdulrahman Hadi Alshamrani
National Center for Assessment, Riyadh, Saudi Arabia
https://orcid.org/0000-0002-6560-3422
Abstract. Test fairness is an important indicator of the validity of test
results. The fairness and equity require ensuring that the background
characteristics of test-takers, such as ethnicity and gender, do not affect
their test scores. Differential item functioning (DIF) methods are
commonly used to detect potentially biased items that lead to the unfair
assessment of the performance of test-takers with the same ability levels
coming from the different cultural, social, demographic, and linguistic
backgrounds. This study aims at detecting potentially biased items
across gender and examining their effect on test scores to ensure the
fairness of test results for each domain and the entire test. Item response
theory (IRT) based Lord’s chi-square DIF method at item level and
Mantel-Haenszel/Liu-Agresti differential test functioning (DTF) method
at test level were implemented to the English Placement Tests (EPT)
administered to high school graduates by the National Center for
Assessment. The results show that 6 items of the EPT exhibit DIF for the
entire test. Two of them are related to reading comprehension and four
to the structure domain, while none of the compositional analysis
methods shows DIF. These results indicate the existence of content
specific DIF effect. Additionally, two items exhibit uniform DIF, one of
which shows DIF favoring male students and the offer favoring female
students. The small to moderate DTF effect associated with sub-domains
and the entire test imply that DIF effects cancel each out, assuring the
fairness of results at test level. However, the items with substantially
high DIF values need to be examined by content experts to determine
the possible cause of DIF effects to avoid gender bias and unfair test
outcomes. We also suggest conducting further studies to investigate the
reasons behind the content specific DIF effects in language tests.
Keywords: test fairness and validity; gender bias; language testing;
differential item functioning; differential test functioning
28
©2020 The authors and IJLTER.ORG. All rights reserved.
1. Introduction
The major concern of the stakeholders in education and test-takers is to ensure
the fairness tests. The best way to provide fairness regarding the decision made
upon a test is to increase the validity and reliability of test results. Therefore, any
effort to minimize confounding factors such as random and systematic errors,
and increase validity and reliability of test will serve the purpose of developing
fair tests and valid test scores for examinees belonging to different groups.
Examining the factorial structure of a test and differential functioning at the item
level and test level are commonly used methods to assess the reliability and
validity of test scores. Differential functioning may occur when items and tests
produce different results for different groups consistently and therefore lead to
invalid test scores and decisions made based on these scores.
The stakeholders that take part in educational test development and assessment
processes explicitly emphasize the importance of fairness in test results
regarding different subgroups. They put a substantial amount of effort to detect
irrelevant factors threatening the construct validity of the test. They are aware of
the necessity and importance of collecting evidence to justify the validity and
fairness of the tests and change the testing policies accordingly. Recently, the
European Federation of Psychological Association has proposed a model for
collecting evidence of construct validity (Evers, Muñiz, Hagemeister,
Høstmælingen, Lindley, Sjöberg, & Bartram, 2013; Hope, Adamson, McManus,
Chris, & Elder, 2018) in which using differential item functioning (DIF) is
considered as an important method for assessing the quality of the test.
Moreover, the Test Commission of the Spanish Psychological Association has
emphasized the critical role of DIF analysis in the context of test fairness
(Hernández, Tomás, Ferreres, & Lloret, 2015; Hope et al., 2018).
A common definition of differential item functioning (DIF) is that an item is said
to exhibit DIF when the probability of correct response to an item differs across
subgroups with the same ability level (Hambleton & Rogers, 1989). DIF types are
classified into two groups that are uniform DIF and non-uniform DIF. An item
exhibits uniform-DIF when the difference in item performance is consistent and
in favor of certain subgroups across the entire range of ability. However, if this
difference between subgroups is not consistent, then DIF is identified as non-
uniform (Hambleton, Clauser, Mazor & Jones, 1993).
The existence of DIF is an indicator of item bias and the presence of the
secondary latent trait besides the primary latent trait that an item aims to
measure. However, this secondary latent trait does not always imply bias or
cause unfair assessment. If the secondary latent trait is related to the primary
trait and occurs due to the nature of the measured structure, then the item is not
labeled as unfair regardless of the differing performance of sub-groups. This
situation was illustrated in a study conducted by Drabinová and Martinková
(2016). They found that one DIF item related to childhood illness in which
females showed better performance than males. However, a detailed
investigation of content experts revealed that this performance difference
occurred since women are more experienced than men since they spend more
time with their children in the Czech Republic (Martinková et al., 2017).
29
©2020 The authors and IJLTER.ORG. All rights reserved.
Therefore, the performance difference between women and men in this example
reflects the true ability difference and does not cause unfairness. Therefore, an
item may display DIF, however, this finding does not provide enough evidence
to classify this item as a biased item. Bias is related to systematic error in test
administration and contents and relies on both statistical tests and expert
opinions (Camilli & Shepard, 1994; Clauser & Mazor, 1998; Wiberg, 2006), while
DIF only relies on statistical tests.
Although there are many different parametric and non-parametric methods to
detect DIF, which method to utilize is a main concern of researchers, since each
method has advantages and shortcomings (Anastasi & Urbina, 1997; Hunter,
2014). For instance, some methods fail to detect non-uniform DIF but effective
when the sample size is small, such as the Mantel Haenszel and Rasch methods,
while some methods are capable of detecting non-uniform DIF but requires
large sample size (Ferne & Rupp, 2007; Lai, Teresi, & Gershon,2005) such as IRT
based Raju’s area method and Lord’s chi-square method. These aforementioned
methods are the most commonly used exploratory methods utilized to identify
differential item functioning for categorical variables that represent existing
subgroups such as gender, nationality, and age groups (Aryadoust & Zhang,
2016). The next step after detecting items exhibiting DIF is to investigate the
possible source and cause of occurrence of DIF (Zhu & Aryadost, 2020).
The EPT is administered to high school graduates by the National Center for
Assessment (NCA) in Saudi Arabia. The results of the EPT has been used by
several colleges, universities, and institutes to measure students’ language skills,
to screen their improvements across different levels or to determine their
required language proficiencies (Education & Training Evaluation Commission
[ETEC], 2020). Luo and Al-Harbi (2016) examined the factorial structure of the
EPT with unidimensional and DIMTEST methods. They found strong evidence
supporting the unidimensionality of the EPT which justified the usage of the
IRT-based models instead of the classical test theory method (CTT).
1.1.Literature review
There have been many studies conducted to ensure test fairness across different
subgroups and to define the potential source of DIF effects. Drabinová and
Martinková (2016) conducted a study to determine the potential sources of DIF
effects concerning the presence of the secondary latent trait besides the primary
latent trait. They concluded that the existence of the DIF effect for some items
did not mean that these items were biased because some of the DIF effects
reflected the relationship between the secondary latent trait and the primary
latent trait. Thus, they suggested that one should avoid labeling DIF items as
biased items without the investigation of the content experts.
Chubbuck et al. (2016) studied DIF effects in the context of differing contents
across gender groups. They employed the Mantel–Haenszel and standardized
DIF methods to detect DIF items for each content domain. They found that the
males showed better performance than females in reading comprehension items.
They also defined the lack of sufficient context in the sentence completion items
as a potential source of DIF effects. Finally, they recommended utilizing more
than one DIF methods to increase the accuracy of the results. Wedman (2018)
30
©2020 The authors and IJLTER.ORG. All rights reserved.
examined if the language ability of non-native test takers that took the test in a
language other than their mother tongue affected their performance compared
to the native speakers. It was found that the deficiency in the language skill of
non-native test takers caused the DIF. Moreover, He defined the failure in
wording the content clearly in an item as a potential source of DIF effects.
(Siegel, 2007, Wedman, 2018).
In one study, Stage (2005) investigated the SweSAT test items administered in
spring concerning DIF across gender groups. The Mantel–Haenszel DIF method
was employed to detect DIF items and It was found that 21 out of 122 items
exhibited DIF across gender groups. Among these DIF items, 10 items related to
the quantitative and verbal domains were in favor of female students. However,
this study did not find any patterns among DIF items and did not suggest
anything about the potential source of DIF effects. Federer and her colleagues
(2016) employed the Mantel-Haenszel DIF method for detecting potential DIF
items in the context of natural selection across gender groups. They specifically
focused on open-ended questions. It was found that women outperformed men
for the items that require applying the knowledge to the new conditions.
Admitting the fact that the developed measurement instrument showed gender
bias and, they did not suggest anything about the potential source of DIF effects
due to the complex nature of DIF structure.
Similarly, Lin and Wu (2003) used DIF and differential bundle functioning (DBF)
to detect items that function differently across gender on the EPT administered
to Chinese EFL learners. For this purpose, they used the SIBTEST methods to
detect DIF items. The results of this study indicated that the testlets (item
bundles) containing the listening comprehension items showed DIF in favor of
females, while the testlets containing the grammar and vocabulary exhibited DIF
in favor of males. Thus, these findings provide strong evidence about content
specific DIF. Pae (2012) studied the trends in the magnitude of DIF on the
English subtest administered to the Korean students across gender groups for
the nine-year period. He used the Mantel-Haenszel and IRT-based likelihood
ratio test methods to detect the DIF items. Moreover, the study examined the
effects of reading strategies and perceived interest on the magnitude of DIF. The
results of this study showed the strong evidence about the relationship between
the type of items and DIF, and a substantial interaction between the test takers
interest in the items and the magnitude of DIF across gender.
It is substantially important to run DTF analyses along with DIF since items are
small and unreliable compared to the test (Gierl, Bisanz, Bisanz, Boughton, &
Khaliq, 2001) and the total amount of DIF provides an overall effect of DIF on
test scores even when there is no item detected as DIF in a test (Hunter, 2014;
Shealy & Stout, 1993). Additionally, DTF values can be negligibly small when
these DIF items are in favor of different subgroups or in a different direction
where DIF effects cancel each out (Borsboom, 2006; Zhu & Aryadoust, 2020).
DTF is also important since decisions about examinees are not made at item-
level, but test-level (Ellis & Raju, 2003; Roznowski & Reith, 1999; Pae & Park,
2006; Zumbo, 2003). More detailed information about the DIF and DTF methods
is provided in the following sections.
31
©2020 The authors and IJLTER.ORG. All rights reserved.
1.2.IRT based DIF methods
The IRT based DIF methods are suggested in the case of a large sample size. The
latent variable (ability estimate-θ) estimated by IRT models is used as a
matching variable for subgroups rather than observed scores. There are many
different IRT based methods to detect DIF items, some of which are test of b
difference, Lord’s chi-square, Raju’s area method, likelihood ratio test (LRT
method), and item drift method. The Lord’s chi-square DIF method (Lord, 1980)
is an extension of the test of b difference and takes the other parameters into
account. The major shortcoming of Raju’s area method is that the exact areas
between ICCs are infinite when guessing parameters are not equal (Hunter,
2014). In this project, Lord’s chi-square DIF method was used to detect DIF
items, because it takes more than one parameter into account when calculating
DIF statistics and capable of detecting both uniform and non-uniform DIF in the
presence of large sample size. The “difR” package installed in the R program
was used to run DIF analyses. The formula for Lord’s chi-square DIF methods is
as follows:
(1)
where VjR = (ajR, bjR, cjR) and VjF = (ajF, bjF, cjF) are the vectors of item parameters
related to the reference group and focal group, respectively. Besides, the
variance-covariance matrices of reference and focal groups are denoted by ∑ 𝑗𝑅
and ∑ 𝑗𝐹, respectively. The 𝑄1-statistic has chi-square distribution and its
degrees of freedom is equal to the number of estimated parameters (Camilli,
2006; Lord, 1980). Previously research show that DIF results obtained from
Lord’s chi-squared test and Raju’s unsigned area method are highly correlated
(Millsap & Everson, 1993; Shepard, Camilli, &Williams, 1985). The most
important disadvantage of the Lord’s chi-squared test is that it tends to reject the
null hypothesis of no DIF even when the discrepancy between ICCs of sub-
groups is small in the presence of a large sample size (Camilli & Shephard,1994;
Wiberg, 2006). Thus, a more stringent criterion should be used in the presence of
a large sample size.
1.3 Differential test functioning (DTF)
Differential test functioning (DTF) values correspond to the total amount of DIF
for the entire test. Therefore, it is equal to the sum of item DIF statistics in a test
(Donovan, Drasgow & Probst, 2000; Ellis & Mead, 2000; Hunter, 2014;
Nandakumar, 1993). There are different methods to calculate DTF such as Raju’s
DFIT (Raju, van der Linden & Fleer, 1995), and Mantel-Haenszel/Liu-Agresti
method (Penfield & Algina, 2006). Raju’s DFIT estimates DTF through
calculating the squared difference between test characteristic curves, while the
Mantel-Haenszel/Liu-Agresti method is based on variance estimates and tend to
have higher DTF rates than Raju’s DFIT (Hunter, 2014). Penfield (2005, 2013)
developed a program called DIFAS which enables us to calculate both DIF and
the Mantel-Haenszel/Liu-Agresti statistics. In this study, the MH-LA method
was used to evaluate DTF associated with the EPT tests.
( ) ( ) ( )
1
j jR jF jR jF
Q v v jR jF v v
 −
= − − −
 
32
©2020 The authors and IJLTER.ORG. All rights reserved.
The formula for MH-LA DTF method proposed by Camilli and Penfield (1997) is
as follows:
2 2
1
2 1 1
( )
I I
i
i i
s
I
 
 = =
− −
=
  (2)
where “I” represents the number of items; 𝜓𝑖
̂ denotes MH log-odds ratio
statistics; µ represents mean and 𝑠𝑖
2
represent the error variance of ψ. Some
studies report weighted 𝜏2
statistics along with 𝜏2
statistics. The formula for
weighted 𝜏2
is as follows:
2 2
1
2 1 1
1
( )
I I
i i
i i
I
i
i
W W
W
 
 = =
=
− −
=
 

(3)
where 𝑤𝑖 is equal to 𝑠𝑖
−2
.
1.4 The purpose of the study
This study aims at examining the presence of items that function differently for
the entire test and across different sub-domains for gender in English Placement
Tests (EPT) to ensure the fairness of the test. For this purpose, item response
theory (IRT) based Differential Item Functioning (DIF) at the item level and
Differential Test Functioning (DTF) methods at the test level were implemented,
respectively, to examine whether items function differently across different
subgroups.
Considering the findings of previously conducted studies, this study aims to test
five different hypotheses. The first hypothesis is that the factorial structure of the
EPT remains unchanged across gender groups. The second hypothesis is that
some of the EPT items are likely to exhibit DIF across gender. The third
hypothesis assumes the existence of content specific DIF items at the item level.
The fourth and fifth hypotheses are that the existence of DIF items affects the test
scores for the entire test and each subdomain.
1.5 Research questions
1. Do the factorial structure of EPT for the entire test and each gender group
support the unidimensionaliy?
2. Do items of the EPT function differently across gender (female vs. male)?
3. What is the distribution of DIF items across sub-domains (Reading
Comprehension, Structure, and Compositional Analysis), when each domain is
treated as a separate test?
4. Do test scores of the EPT exhibit differential test functioning (DTF) across
gender (Female vs. Male)?
5. Do test scores of the EPT exhibit differential test functioning (DTF) across
gender, when each domain is treated as a separate test?
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020
IJLTER.ORG Vol 19 No 6 June 2020

More Related Content

Similar to IJLTER.ORG Vol 19 No 6 June 2020

IJLTER.ORG Vol 21 No 7 July 2022
IJLTER.ORG Vol 21 No 7 July 2022IJLTER.ORG Vol 21 No 7 July 2022
IJLTER.ORG Vol 21 No 7 July 2022
ijlterorg
 
IJLTER.ORG Vol 20 No 8 August 2021
IJLTER.ORG Vol 20 No 8 August 2021IJLTER.ORG Vol 20 No 8 August 2021
IJLTER.ORG Vol 20 No 8 August 2021
ijlterorg
 
IJLTER.ORG Vol 22 No 5 May 2023
IJLTER.ORG Vol 22 No 5 May 2023IJLTER.ORG Vol 22 No 5 May 2023
IJLTER.ORG Vol 22 No 5 May 2023
ijlterorg
 
IJLTER.ORG Vol 21 No 10 October 2022
IJLTER.ORG Vol 21 No 10 October 2022IJLTER.ORG Vol 21 No 10 October 2022
IJLTER.ORG Vol 21 No 10 October 2022
ijlterorg
 
IJLTER.ORG Vol 19 No 4 April 2020
IJLTER.ORG Vol 19 No 4 April 2020IJLTER.ORG Vol 19 No 4 April 2020
IJLTER.ORG Vol 19 No 4 April 2020
ijlterorg
 
UNDERSTANDING_WHAT_WORKS_IN_ORAL_READING.pdf
UNDERSTANDING_WHAT_WORKS_IN_ORAL_READING.pdfUNDERSTANDING_WHAT_WORKS_IN_ORAL_READING.pdf
UNDERSTANDING_WHAT_WORKS_IN_ORAL_READING.pdf
mj markes
 
UNDERSTANDING WHAT WORKS IN ORAL READING.pdf
UNDERSTANDING WHAT WORKS IN ORAL READING.pdfUNDERSTANDING WHAT WORKS IN ORAL READING.pdf
UNDERSTANDING WHAT WORKS IN ORAL READING.pdf
MJoMarques
 
IJLTER.ORG Vol 22 No 2 February 2023
IJLTER.ORG Vol 22 No 2 February 2023IJLTER.ORG Vol 22 No 2 February 2023
IJLTER.ORG Vol 22 No 2 February 2023
ijlterorg
 
IJLTER.ORG Vol 20 No 12 December 2021
IJLTER.ORG Vol 20 No 12 December 2021IJLTER.ORG Vol 20 No 12 December 2021
IJLTER.ORG Vol 20 No 12 December 2021
ijlterorg
 
IJLTER.ORG Vol 20 No 3 March 2021
IJLTER.ORG Vol 20 No 3 March 2021IJLTER.ORG Vol 20 No 3 March 2021
IJLTER.ORG Vol 20 No 3 March 2021
ijlterorg
 
Vol 13 No 2 - September 2015
Vol 13 No 2 - September 2015Vol 13 No 2 - September 2015
Vol 13 No 2 - September 2015
ijlterorg
 
IJLTER.ORG Vol 22 No 4 April 2023
IJLTER.ORG Vol 22 No 4 April 2023IJLTER.ORG Vol 22 No 4 April 2023
IJLTER.ORG Vol 22 No 4 April 2023
ijlterorg
 
ILJTER.ORG Volume 23 Number 4 April 2024
ILJTER.ORG Volume 23 Number 4 April 2024ILJTER.ORG Volume 23 Number 4 April 2024
ILJTER.ORG Volume 23 Number 4 April 2024
ijlterorg
 
IJLTER.ORG Vol 20 No 10 October 2021
IJLTER.ORG Vol 20 No 10 October 2021IJLTER.ORG Vol 20 No 10 October 2021
IJLTER.ORG Vol 20 No 10 October 2021
ijlterorg
 
Vol 15 No 7 - June 2016
Vol 15 No 7 - June 2016Vol 15 No 7 - June 2016
Vol 15 No 7 - June 2016
ijlterorg
 
IJLTER.ORG Vol 21 No 11 November 2022
IJLTER.ORG Vol 21 No 11 November 2022IJLTER.ORG Vol 21 No 11 November 2022
IJLTER.ORG Vol 21 No 11 November 2022
ijlterorg
 
IJLTER.ORG Vol 20 No 9 September 2021
IJLTER.ORG Vol 20 No 9 September 2021IJLTER.ORG Vol 20 No 9 September 2021
IJLTER.ORG Vol 20 No 9 September 2021
ijlterorg
 
Vol 15 No 13 - December 2016
Vol 15 No 13 - December 2016Vol 15 No 13 - December 2016
Vol 15 No 13 - December 2016
ijlterorg
 
IJLTER.ORG Vol 19 No 1 January 2020
IJLTER.ORG Vol 19 No 1 January 2020IJLTER.ORG Vol 19 No 1 January 2020
IJLTER.ORG Vol 19 No 1 January 2020
ijlterorg
 
IJLTER.ORG Vol 19 No 9 September 2020
IJLTER.ORG Vol 19 No 9 September 2020IJLTER.ORG Vol 19 No 9 September 2020
IJLTER.ORG Vol 19 No 9 September 2020
ijlterorg
 

Similar to IJLTER.ORG Vol 19 No 6 June 2020 (20)

IJLTER.ORG Vol 21 No 7 July 2022
IJLTER.ORG Vol 21 No 7 July 2022IJLTER.ORG Vol 21 No 7 July 2022
IJLTER.ORG Vol 21 No 7 July 2022
 
IJLTER.ORG Vol 20 No 8 August 2021
IJLTER.ORG Vol 20 No 8 August 2021IJLTER.ORG Vol 20 No 8 August 2021
IJLTER.ORG Vol 20 No 8 August 2021
 
IJLTER.ORG Vol 22 No 5 May 2023
IJLTER.ORG Vol 22 No 5 May 2023IJLTER.ORG Vol 22 No 5 May 2023
IJLTER.ORG Vol 22 No 5 May 2023
 
IJLTER.ORG Vol 21 No 10 October 2022
IJLTER.ORG Vol 21 No 10 October 2022IJLTER.ORG Vol 21 No 10 October 2022
IJLTER.ORG Vol 21 No 10 October 2022
 
IJLTER.ORG Vol 19 No 4 April 2020
IJLTER.ORG Vol 19 No 4 April 2020IJLTER.ORG Vol 19 No 4 April 2020
IJLTER.ORG Vol 19 No 4 April 2020
 
UNDERSTANDING_WHAT_WORKS_IN_ORAL_READING.pdf
UNDERSTANDING_WHAT_WORKS_IN_ORAL_READING.pdfUNDERSTANDING_WHAT_WORKS_IN_ORAL_READING.pdf
UNDERSTANDING_WHAT_WORKS_IN_ORAL_READING.pdf
 
UNDERSTANDING WHAT WORKS IN ORAL READING.pdf
UNDERSTANDING WHAT WORKS IN ORAL READING.pdfUNDERSTANDING WHAT WORKS IN ORAL READING.pdf
UNDERSTANDING WHAT WORKS IN ORAL READING.pdf
 
IJLTER.ORG Vol 22 No 2 February 2023
IJLTER.ORG Vol 22 No 2 February 2023IJLTER.ORG Vol 22 No 2 February 2023
IJLTER.ORG Vol 22 No 2 February 2023
 
IJLTER.ORG Vol 20 No 12 December 2021
IJLTER.ORG Vol 20 No 12 December 2021IJLTER.ORG Vol 20 No 12 December 2021
IJLTER.ORG Vol 20 No 12 December 2021
 
IJLTER.ORG Vol 20 No 3 March 2021
IJLTER.ORG Vol 20 No 3 March 2021IJLTER.ORG Vol 20 No 3 March 2021
IJLTER.ORG Vol 20 No 3 March 2021
 
Vol 13 No 2 - September 2015
Vol 13 No 2 - September 2015Vol 13 No 2 - September 2015
Vol 13 No 2 - September 2015
 
IJLTER.ORG Vol 22 No 4 April 2023
IJLTER.ORG Vol 22 No 4 April 2023IJLTER.ORG Vol 22 No 4 April 2023
IJLTER.ORG Vol 22 No 4 April 2023
 
ILJTER.ORG Volume 23 Number 4 April 2024
ILJTER.ORG Volume 23 Number 4 April 2024ILJTER.ORG Volume 23 Number 4 April 2024
ILJTER.ORG Volume 23 Number 4 April 2024
 
IJLTER.ORG Vol 20 No 10 October 2021
IJLTER.ORG Vol 20 No 10 October 2021IJLTER.ORG Vol 20 No 10 October 2021
IJLTER.ORG Vol 20 No 10 October 2021
 
Vol 15 No 7 - June 2016
Vol 15 No 7 - June 2016Vol 15 No 7 - June 2016
Vol 15 No 7 - June 2016
 
IJLTER.ORG Vol 21 No 11 November 2022
IJLTER.ORG Vol 21 No 11 November 2022IJLTER.ORG Vol 21 No 11 November 2022
IJLTER.ORG Vol 21 No 11 November 2022
 
IJLTER.ORG Vol 20 No 9 September 2021
IJLTER.ORG Vol 20 No 9 September 2021IJLTER.ORG Vol 20 No 9 September 2021
IJLTER.ORG Vol 20 No 9 September 2021
 
Vol 15 No 13 - December 2016
Vol 15 No 13 - December 2016Vol 15 No 13 - December 2016
Vol 15 No 13 - December 2016
 
IJLTER.ORG Vol 19 No 1 January 2020
IJLTER.ORG Vol 19 No 1 January 2020IJLTER.ORG Vol 19 No 1 January 2020
IJLTER.ORG Vol 19 No 1 January 2020
 
IJLTER.ORG Vol 19 No 9 September 2020
IJLTER.ORG Vol 19 No 9 September 2020IJLTER.ORG Vol 19 No 9 September 2020
IJLTER.ORG Vol 19 No 9 September 2020
 

More from ijlterorg

ILJTER.ORG Volume 23 Number 3 March 2024.pdf
ILJTER.ORG Volume 23 Number 3 March 2024.pdfILJTER.ORG Volume 23 Number 3 March 2024.pdf
ILJTER.ORG Volume 23 Number 3 March 2024.pdf
ijlterorg
 
ILJTER.ORG Volume 23 Number 2 February 2024
ILJTER.ORG Volume 23 Number 2 February 2024ILJTER.ORG Volume 23 Number 2 February 2024
ILJTER.ORG Volume 23 Number 2 February 2024
ijlterorg
 
ILJTER.ORG Volume 22 Number 12 December 2023
ILJTER.ORG Volume 22 Number 12 December 2023ILJTER.ORG Volume 22 Number 12 December 2023
ILJTER.ORG Volume 22 Number 12 December 2023
ijlterorg
 
ILJTER.ORG Volume 22 Number 11 November 2023
ILJTER.ORG Volume 22 Number 11 November 2023ILJTER.ORG Volume 22 Number 11 November 2023
ILJTER.ORG Volume 22 Number 11 November 2023
ijlterorg
 
ILJTER.ORG Volume 22 Number 10 October 2023
ILJTER.ORG Volume 22 Number 10 October 2023ILJTER.ORG Volume 22 Number 10 October 2023
ILJTER.ORG Volume 22 Number 10 October 2023
ijlterorg
 
ILJTER.ORG Volume 22 Number 09 September 2023
ILJTER.ORG Volume 22 Number 09 September 2023ILJTER.ORG Volume 22 Number 09 September 2023
ILJTER.ORG Volume 22 Number 09 September 2023
ijlterorg
 
ILJTER.ORG Volume 22 Number 06 June 2023
ILJTER.ORG Volume 22 Number 06 June 2023ILJTER.ORG Volume 22 Number 06 June 2023
ILJTER.ORG Volume 22 Number 06 June 2023
ijlterorg
 
IJLTER.ORG Vol 22 No 3 March 2023
IJLTER.ORG Vol 22 No 3 March 2023IJLTER.ORG Vol 22 No 3 March 2023
IJLTER.ORG Vol 22 No 3 March 2023
ijlterorg
 
IJLTER.ORG Vol 22 No 1 January 2023
IJLTER.ORG Vol 22 No 1 January 2023IJLTER.ORG Vol 22 No 1 January 2023
IJLTER.ORG Vol 22 No 1 January 2023
ijlterorg
 
IJLTER.ORG Vol 21 No 12 December 2022
IJLTER.ORG Vol 21 No 12 December 2022IJLTER.ORG Vol 21 No 12 December 2022
IJLTER.ORG Vol 21 No 12 December 2022
ijlterorg
 
IJLTER.ORG Vol 21 No 9 September 2022
IJLTER.ORG Vol 21 No 9 September 2022IJLTER.ORG Vol 21 No 9 September 2022
IJLTER.ORG Vol 21 No 9 September 2022
ijlterorg
 
IJLTER.ORG Vol 21 No 8 August 2022
IJLTER.ORG Vol 21 No 8 August 2022IJLTER.ORG Vol 21 No 8 August 2022
IJLTER.ORG Vol 21 No 8 August 2022
ijlterorg
 
IJLTER.ORG Vol 21 No 5 May 2022
IJLTER.ORG Vol 21 No 5 May 2022IJLTER.ORG Vol 21 No 5 May 2022
IJLTER.ORG Vol 21 No 5 May 2022
ijlterorg
 
IJLTER.ORG Vol 21 No 4 April 2022
IJLTER.ORG Vol 21 No 4 April 2022IJLTER.ORG Vol 21 No 4 April 2022
IJLTER.ORG Vol 21 No 4 April 2022
ijlterorg
 
IJLTER.ORG Vol 19 No 11 November 2020
IJLTER.ORG Vol 19 No 11 November 2020IJLTER.ORG Vol 19 No 11 November 2020
IJLTER.ORG Vol 19 No 11 November 2020
ijlterorg
 
IJLTER.ORG Vol 19 No 7 July 2020
IJLTER.ORG Vol 19 No 7 July 2020IJLTER.ORG Vol 19 No 7 July 2020
IJLTER.ORG Vol 19 No 7 July 2020
ijlterorg
 

More from ijlterorg (16)

ILJTER.ORG Volume 23 Number 3 March 2024.pdf
ILJTER.ORG Volume 23 Number 3 March 2024.pdfILJTER.ORG Volume 23 Number 3 March 2024.pdf
ILJTER.ORG Volume 23 Number 3 March 2024.pdf
 
ILJTER.ORG Volume 23 Number 2 February 2024
ILJTER.ORG Volume 23 Number 2 February 2024ILJTER.ORG Volume 23 Number 2 February 2024
ILJTER.ORG Volume 23 Number 2 February 2024
 
ILJTER.ORG Volume 22 Number 12 December 2023
ILJTER.ORG Volume 22 Number 12 December 2023ILJTER.ORG Volume 22 Number 12 December 2023
ILJTER.ORG Volume 22 Number 12 December 2023
 
ILJTER.ORG Volume 22 Number 11 November 2023
ILJTER.ORG Volume 22 Number 11 November 2023ILJTER.ORG Volume 22 Number 11 November 2023
ILJTER.ORG Volume 22 Number 11 November 2023
 
ILJTER.ORG Volume 22 Number 10 October 2023
ILJTER.ORG Volume 22 Number 10 October 2023ILJTER.ORG Volume 22 Number 10 October 2023
ILJTER.ORG Volume 22 Number 10 October 2023
 
ILJTER.ORG Volume 22 Number 09 September 2023
ILJTER.ORG Volume 22 Number 09 September 2023ILJTER.ORG Volume 22 Number 09 September 2023
ILJTER.ORG Volume 22 Number 09 September 2023
 
ILJTER.ORG Volume 22 Number 06 June 2023
ILJTER.ORG Volume 22 Number 06 June 2023ILJTER.ORG Volume 22 Number 06 June 2023
ILJTER.ORG Volume 22 Number 06 June 2023
 
IJLTER.ORG Vol 22 No 3 March 2023
IJLTER.ORG Vol 22 No 3 March 2023IJLTER.ORG Vol 22 No 3 March 2023
IJLTER.ORG Vol 22 No 3 March 2023
 
IJLTER.ORG Vol 22 No 1 January 2023
IJLTER.ORG Vol 22 No 1 January 2023IJLTER.ORG Vol 22 No 1 January 2023
IJLTER.ORG Vol 22 No 1 January 2023
 
IJLTER.ORG Vol 21 No 12 December 2022
IJLTER.ORG Vol 21 No 12 December 2022IJLTER.ORG Vol 21 No 12 December 2022
IJLTER.ORG Vol 21 No 12 December 2022
 
IJLTER.ORG Vol 21 No 9 September 2022
IJLTER.ORG Vol 21 No 9 September 2022IJLTER.ORG Vol 21 No 9 September 2022
IJLTER.ORG Vol 21 No 9 September 2022
 
IJLTER.ORG Vol 21 No 8 August 2022
IJLTER.ORG Vol 21 No 8 August 2022IJLTER.ORG Vol 21 No 8 August 2022
IJLTER.ORG Vol 21 No 8 August 2022
 
IJLTER.ORG Vol 21 No 5 May 2022
IJLTER.ORG Vol 21 No 5 May 2022IJLTER.ORG Vol 21 No 5 May 2022
IJLTER.ORG Vol 21 No 5 May 2022
 
IJLTER.ORG Vol 21 No 4 April 2022
IJLTER.ORG Vol 21 No 4 April 2022IJLTER.ORG Vol 21 No 4 April 2022
IJLTER.ORG Vol 21 No 4 April 2022
 
IJLTER.ORG Vol 19 No 11 November 2020
IJLTER.ORG Vol 19 No 11 November 2020IJLTER.ORG Vol 19 No 11 November 2020
IJLTER.ORG Vol 19 No 11 November 2020
 
IJLTER.ORG Vol 19 No 7 July 2020
IJLTER.ORG Vol 19 No 7 July 2020IJLTER.ORG Vol 19 No 7 July 2020
IJLTER.ORG Vol 19 No 7 July 2020
 

Recently uploaded

ZK on Polkadot zero knowledge proofs - sub0.pptx
ZK on Polkadot zero knowledge proofs - sub0.pptxZK on Polkadot zero knowledge proofs - sub0.pptx
ZK on Polkadot zero knowledge proofs - sub0.pptx
dot55audits
 
Mule event processing models | MuleSoft Mysore Meetup #47
Mule event processing models | MuleSoft Mysore Meetup #47Mule event processing models | MuleSoft Mysore Meetup #47
Mule event processing models | MuleSoft Mysore Meetup #47
MysoreMuleSoftMeetup
 
Présentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
Présentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptxPrésentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
Présentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
siemaillard
 
Constructing Your Course Container for Effective Communication
Constructing Your Course Container for Effective CommunicationConstructing Your Course Container for Effective Communication
Constructing Your Course Container for Effective Communication
Chevonnese Chevers Whyte, MBA, B.Sc.
 
বাংলাদেশ অর্থনৈতিক সমীক্ষা (Economic Review) ২০২৪ UJS App.pdf
বাংলাদেশ অর্থনৈতিক সমীক্ষা (Economic Review) ২০২৪ UJS App.pdfবাংলাদেশ অর্থনৈতিক সমীক্ষা (Economic Review) ২০২৪ UJS App.pdf
বাংলাদেশ অর্থনৈতিক সমীক্ষা (Economic Review) ২০২৪ UJS App.pdf
eBook.com.bd (প্রয়োজনীয় বাংলা বই)
 
A Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdfA Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdf
Jean Carlos Nunes Paixão
 
Main Java[All of the Base Concepts}.docx
Main Java[All of the Base Concepts}.docxMain Java[All of the Base Concepts}.docx
Main Java[All of the Base Concepts}.docx
adhitya5119
 
Film vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movieFilm vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movie
Nicholas Montgomery
 
Your Skill Boost Masterclass: Strategies for Effective Upskilling
Your Skill Boost Masterclass: Strategies for Effective UpskillingYour Skill Boost Masterclass: Strategies for Effective Upskilling
Your Skill Boost Masterclass: Strategies for Effective Upskilling
Excellence Foundation for South Sudan
 
How to Make a Field Mandatory in Odoo 17
How to Make a Field Mandatory in Odoo 17How to Make a Field Mandatory in Odoo 17
How to Make a Field Mandatory in Odoo 17
Celine George
 
Temple of Asclepius in Thrace. Excavation results
Temple of Asclepius in Thrace. Excavation resultsTemple of Asclepius in Thrace. Excavation results
Temple of Asclepius in Thrace. Excavation results
Krassimira Luka
 
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem studentsRHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
Himanshu Rai
 
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching AptitudeUGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
S. Raj Kumar
 
Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...
PsychoTech Services
 
clinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdfclinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdf
Priyankaranawat4
 
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptxNEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
iammrhaywood
 
The History of Stoke Newington Street Names
The History of Stoke Newington Street NamesThe History of Stoke Newington Street Names
The History of Stoke Newington Street Names
History of Stoke Newington
 
Advanced Java[Extra Concepts, Not Difficult].docx
Advanced Java[Extra Concepts, Not Difficult].docxAdvanced Java[Extra Concepts, Not Difficult].docx
Advanced Java[Extra Concepts, Not Difficult].docx
adhitya5119
 
Walmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdfWalmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdf
TechSoup
 
writing about opinions about Australia the movie
writing about opinions about Australia the moviewriting about opinions about Australia the movie
writing about opinions about Australia the movie
Nicholas Montgomery
 

Recently uploaded (20)

ZK on Polkadot zero knowledge proofs - sub0.pptx
ZK on Polkadot zero knowledge proofs - sub0.pptxZK on Polkadot zero knowledge proofs - sub0.pptx
ZK on Polkadot zero knowledge proofs - sub0.pptx
 
Mule event processing models | MuleSoft Mysore Meetup #47
Mule event processing models | MuleSoft Mysore Meetup #47Mule event processing models | MuleSoft Mysore Meetup #47
Mule event processing models | MuleSoft Mysore Meetup #47
 
Présentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
Présentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptxPrésentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
Présentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
 
Constructing Your Course Container for Effective Communication
Constructing Your Course Container for Effective CommunicationConstructing Your Course Container for Effective Communication
Constructing Your Course Container for Effective Communication
 
বাংলাদেশ অর্থনৈতিক সমীক্ষা (Economic Review) ২০২৪ UJS App.pdf
বাংলাদেশ অর্থনৈতিক সমীক্ষা (Economic Review) ২০২৪ UJS App.pdfবাংলাদেশ অর্থনৈতিক সমীক্ষা (Economic Review) ২০২৪ UJS App.pdf
বাংলাদেশ অর্থনৈতিক সমীক্ষা (Economic Review) ২০২৪ UJS App.pdf
 
A Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdfA Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdf
 
Main Java[All of the Base Concepts}.docx
Main Java[All of the Base Concepts}.docxMain Java[All of the Base Concepts}.docx
Main Java[All of the Base Concepts}.docx
 
Film vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movieFilm vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movie
 
Your Skill Boost Masterclass: Strategies for Effective Upskilling
Your Skill Boost Masterclass: Strategies for Effective UpskillingYour Skill Boost Masterclass: Strategies for Effective Upskilling
Your Skill Boost Masterclass: Strategies for Effective Upskilling
 
How to Make a Field Mandatory in Odoo 17
How to Make a Field Mandatory in Odoo 17How to Make a Field Mandatory in Odoo 17
How to Make a Field Mandatory in Odoo 17
 
Temple of Asclepius in Thrace. Excavation results
Temple of Asclepius in Thrace. Excavation resultsTemple of Asclepius in Thrace. Excavation results
Temple of Asclepius in Thrace. Excavation results
 
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem studentsRHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
 
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching AptitudeUGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
 
Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...
 
clinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdfclinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdf
 
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptxNEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
 
The History of Stoke Newington Street Names
The History of Stoke Newington Street NamesThe History of Stoke Newington Street Names
The History of Stoke Newington Street Names
 
Advanced Java[Extra Concepts, Not Difficult].docx
Advanced Java[Extra Concepts, Not Difficult].docxAdvanced Java[Extra Concepts, Not Difficult].docx
Advanced Java[Extra Concepts, Not Difficult].docx
 
Walmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdfWalmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdf
 
writing about opinions about Australia the movie
writing about opinions about Australia the moviewriting about opinions about Australia the movie
writing about opinions about Australia the movie
 

IJLTER.ORG Vol 19 No 6 June 2020

  • 1. International Journal of Learning, Teaching And Educational Research p-ISSN: 1694-2493 e-ISSN: 1694-2116 IJLTER.ORG Vol.19 No.6
  • 2. PUBLISHER Society for Research and Knowledge Management Port Louis Republic of Mauritius www.ijlter.org Chief Editor Dr. Antonio Silva Sprock, Universidad Central de Venezuela, Venezuela, Bolivarian Republic of Editorial Board Prof. Cecilia Junio Sabio Prof. Judith Serah K. Achoka Prof. Mojeed Kolawole Akinsola Dr Jonathan Glazzard Dr Marius Costel Esi Dr Katarzyna Peoples Dr Christopher David Thompson Dr Arif Sikander Dr Jelena Zascerinska Dr Gabor Kiss Dr Trish Julie Rooney Dr Esteban Vázquez-Cano Dr Barry Chametzky Dr Giorgio Poletti Dr Chi Man Tsui Dr Alexander Franco Dr Habil Beata Stachowiak Dr Afsaneh Sharif Dr Ronel Callaghan Dr Haim Shaked Dr Edith Uzoma Umeh Dr Amel Thafer Alshehry Dr Gail Dianna Caruth Dr Menelaos Emmanouel Sarris Dr Anabelie Villa Valdez Dr Özcan Özyurt Assistant Professor Dr Selma Kara Associate Professor Dr Habila Elisha Zuya International Journal of Learning, Teaching and Educational Research The International Journal of Learning, Teaching and Educational Research is an open-access journal which has been established for the dis- semination of state-of-the-art knowledge in the field of education, learning and teaching. IJLTER welcomes research articles from academics, ed- ucators, teachers, trainers and other practition- ers on all aspects of education to publish high quality peer-reviewed papers. Papers for publi- cation in the International Journal of Learning, Teaching and Educational Research are selected through precise peer-review to ensure quality, originality, appropriateness, significance and readability. Authors are solicited to contribute to this journal by submitting articles that illus- trate research results, projects, original surveys and case studies that describe significant ad- vances in the fields of education, training, e- learning, etc. Authors are invited to submit pa- pers to this journal through the ONLINE submis- sion system. Submissions must be original and should not have been published previously or be under consideration for publication while being evaluated by IJLTER.
  • 3. International Journal of Learning, Teaching and Educational Research (IJLTER) Vol. 19, No. 6 (June 2020) Print version: 1694-2493 Online version: 1694-2116 IJLTER International Journal of Learning, Teaching and Educational Research (IJLTER) Vol. 19, No. 6 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically those of translation, reprinting, re-use of illustrations, broadcasting, reproduction by photocopying machines or similar means, and storage in data banks. Society for Research and Knowledge Management
  • 4. International Journal of Learning, Teaching and Educational Research The International Journal of Learning, Teaching and Educational Research is a peer-reviewed open-access journal which has been established for the dissemination of state-of-the-art knowledge in the fields of learning, teaching and educational research. Aims and Objectives The main objective of this journal is to provide a platform for educators, teachers, trainers, academicians, scientists and researchers from over the world to present the results of their research activities in the following fields: innovative methodologies in learning, teaching and assessment; multimedia in digital learning; e-learning; m-learning; e-education; knowledge management; infrastructure support for online learning; virtual learning environments; open education; ICT and education; digital classrooms; blended learning; social networks and education; e- tutoring: learning management systems; educational portals, classroom management issues, educational case studies, etc. Indexing and Abstracting The International Journal of Learning, Teaching and Educational Research is indexed in Scopus since 2018. The Journal is also indexed in Google Scholar and CNKI. All articles published in IJLTER are assigned a unique DOI number.
  • 5. Foreword We are very happy to publish this issue of the International Journal of Learning, Teaching and Educational Research. The International Journal of Learning, Teaching and Educational Research is a peer-reviewed open-access journal committed to publishing high-quality articles in the field of education. Submissions may include full-length articles, case studies and innovative solutions to problems faced by students, educators and directors of educational organisations. To learn more about this journal, please visit the website http://www.ijlter.org. We are grateful to the editor-in-chief, members of the Editorial Board and the reviewers for accepting only high quality articles in this issue. We seize this opportunity to thank them for their great collaboration. The Editorial Board is composed of renowned people from across the world. Each paper is reviewed by at least two blind reviewers. We will endeavour to ensure the reputation and quality of this journal with this issue. Editors of the June 2020 Issue
  • 6. VOLUME 19 NUMBER 6 June 2020 Table of Contents The Emergency Remote Learning Experience of University Students in Indonesia amidst the COVID-19 Crisis....1 Maila D.H. Rahiem Examining the Fairness of Language Test Across Gender with IRT-based Differential Item and Test Functioning Methods ................................................................................................................................................................................. 27 Burhanettin Ozdemir and Abdulrahman Hadi Alshamrani Main Approaches of Business English Teaching to Future Lawyers: A Case Study of Ukrainian Higher Institutions............................................................................................................................................................................. 46 Oksana P. Bykonia, Iryna V. Borysenko, Tamila L. Gruba, Iurii L. Mosenkis and Dmytro O. Chystiak Debate as a Tool for Learning and Facilitating Based on Higher Order Thinking Skills in The Process of Argumentative Essay Writing............................................................................................................................................. 62 Marzni Mohamed Mokhtar, Marni Jamil, Rohizani Yaakub and Fadzilah Amzah Exploring Teachers’ Pedagogical Practices in Teaching Mandarin as a Foreign Language in MARA Educational Institutions, Malaysia ........................................................................................................................................................... 76 Nuraini Jafri, Umi Kalthom Abd Manaf and Fazilah Razali The Effectiveness of Differentiated Instruction by Streaming: A preliminary Study of Current Practices in the UAE ........................................................................................................................................................................................ 95 Ibrahim Suleiman Ibrahim Magableh and Amelia Abdullah Technology-Driven Self-Directed Learning of Graduate Pharmaceutists: Adding Value through Entrepreneurship ............................................................................................................................................................................................... 111 Iryna Nizhenkovska, Tatyana Reva, Oksana Chkhalo and Oksana Holovchenko Teachers’ Covid-19 Awareness, Distance Learning Education Experiences and Perceptions towards Institutional Readiness and Challenges ................................................................................................................................................. 127 Aris Alea Lapada, Frosyl Fabrea Miguel, Dave Arthur Roldan Robledo and Zeba F Alam Enhancing Malaysian Primary Pupils’ Vocabulary Skills using Pocable Game and Pear Deck .............................. 145 Chai Kar Ni, Bonaventure Jong, Mary Anne Dison, Sylvia Anak Thomas, Melor Md Yunus and Ashairi Suliman Content Validity of West African Examination Council Financial Accounting Questions....................................... 161 Basil C. E. Oguguo, John J. Agah, Catherine U. Ene, Vivian N. ACholonu, Roseline N. Azubuike, Mary A. Okeke and Lourita P. Agbo Assessing the Relationship and Prediction of Manifold Facets of Analytic Relations to Academic Reading Comprehension................................................................................................................................................................... 179 Md Kamrul Hasan, Md. Didar Hossain and Abdul Karim Towards a Principled Use of L1 – Observing an EFL Teacher’s L1 Use in Rural Sabah, Malaysia.......................... 206 J. W. Ong and A. J. Ahmad Tajuddin
  • 7. A Study of Saudi Advanced Academic Writing Students’ Perceptions of Research Essays, and Gaps in Their Knowledge........................................................................................................................................................................... 223 Nida Qayoom and Mohammad Saleem Mass Media Internships in Vocational Training of Students Majoring in Journalism............................................... 238 Nataliia A. Tsymbal, Nataliia M. Savchuk, Valentina I. Avramenko, Svitlana A. Sichkar and Iryna A. Denysiuk Assessing the Efficacy of Extensive Reading during Study Abroad: A Time and Place for ER? ............................. 251 Byron O'Neill and Christopher Edelman The Extent of Bullying Against Students with Learning Disabilities According to the Age Variable..................... 267 Mohamad Ahmad Saleem Khasawneh Development of Instruments to Measure Mathematical Anxiety of Elementary School Students.......................... 282 Hafiziani Eka Putri, Mukhammad Ady Wahyudy, Aan Yuliyanto and Fitri Nuraeni Situating "children-supporting-children" Platform in the Context of the Inclusive Agenda: A Phenomenological Exploration .......................................................................................................................................................................... 303 Princess Zarla J. Raguindin and Li Yan Ping The Impact of Educational Technologies on University Teachers’ Self-efficacy ........................................................ 323 Nataliia Saienko, Yuliana Lavrysh and Valentyna Lukianenko UAE Elementary Teachers’ Use of ADHD Referral and Management Strategies...................................................... 337 Hala Elhoweris, Ahmed Mohamed, Osha Almuhairy, Rachel Takriti, Najwa Alhosani and Abdelaziz Sartawi Exploring Accounting Teachers’ Views on the Quality of Accounting Prescribed Textbooks in South Africa ..... 353 Jabulisile C. Ngwenya and S’khumbuzo H. Mbili Rurality and Exclusion in Ordinary Level Mathematics in Zimbabwe: A Document Analysis............................... 370 Simon Vurayai Emergency Online Teaching in Economic and Management Sciences Necessitated by the COVID-19 Pandemic: The Need for Healthy Relations in a Rural Schooling Context..................................................................................... 387 Habasisa Molise and Bekithemba Dube
  • 8. 1 ©2020 The author and IJLTER.ORG. All rights reserved. International Journal of Learning, Teaching and Educational Research Vol. 19, No. 6, pp. 1-26, June 2020 https://doi.org/10.26803/ijlter.19.6.1 The Emergency Remote Learning Experience of University Students in Indonesia amidst the COVID-19 Crisis Maila D. H. Rahiem UIN Syarif Hidayatullah Jakarta Jakarta, Indonesia http://orcid.org/0000-0002-5618-2486 Abstract. This study aimed to explore and interpret the lived experience of Indonesian university students in emergency remote learning (ERL) during the COVID-19. Methods of the investigation was a qualitative phenomenological approach involving 80 students from the Social Science Education Program at a public university in Jakarta. Understanding their experience was achieved through a rigorous analysis of the participants’ diaries and reflective essays and an online focus group. Results revealed that the students’ experiences fell into two overarching themes, each with related sub-themes. The two identified themes and subthemes were: (a) blended learning, with the subthemes of e-learning, m-learning and conventional learning, and (b) paradoxical learning, with the sub-themes of flexible learning and challenging learning. By studying how university students learned during COVID-19, we could help ensure the efficacy of ongoing ERL and better incorporate similar programs in the future if this ever happens again. Keywords: emergency remote learning; online learning; higher education; COVID-19 1. Introduction The COVID-19 global pandemic has created entirely unprecedented situations that have greatly affected people’s lives. With doubts remaining at present over how and when it will end, the question remains whether things will ever go back to the way they were before the pandemic began with many skeptics remaining doubtful (Lee et al., 2010).The World Health Organization (WHO) has warned that COVID-19 is likely to haunt the world for a long time and that the planet will only return to normal once a viral strain vaccine has been discovered. There are currently ten candidate vaccines in clinical trials worldwide and 126 candidate vaccines in pre-clinical trials (World Health Organization, 2020b). Anthony Fauci, MD, director of the US National Institute of Allergy and Infectious Disease, said that if all goes well, a vaccine might be available in November or December of this
  • 9. 2 ©2020 The author and IJLTER.ORG. All rights reserved. year (McCarthy, 2020; Reynolds, 2020). However, even after a vaccine has been approved, there remains an immense challenge to generate enough of it for the world's population (Felter, 2020). Presently (Early June 2020), in Indonesia and some other countries, the number of cases is still increasing at an alarming rate. The number of patients nationwide who have tested positive is up to 36,406, with patients who have recovered recorded as 13,213 people, and patients who have died totaling 2,048 people (Task Force for the Acceleration of COVID-19, 2020). COVID-19 has spread to 34 provinces, meaning that it has spread to all of the provinces across the Indonesian archipelago, and 391 out of 514 municipal districts have been infected (National Agency for Disaster Management, 2020). So far, the percentage of Indonesia's death rate, which is above 6 percent, is rated in the high category. The total number of cases worldwide is also still increasing, with 7,127,753 confirmed cases of COVID-19 spread across 216 countries, including 407,159 deaths, reported to WHO (World Health Organization, 2020c). As a result of the threat posed by COVID-19, educational institutions have hurried to shift courses to virtual classrooms. Many institutions have chosen to cancel all face-to-face classes, including laboratories and other learning activities, and have required students to study remotely from home to help avoid the spread of the virus that causes COVID-19. The UN Educational, Scientific, and Cultural Organization(UNESCO, 2020) estimated that 129 countries had implemented national school closures (data from June 10, 2020). These nationwide closures have impacted almost 63.3 percent of the world's student population or 1,109,020 learners. Several other countries have implemented localized closures impacting millions of additional learners (approximately 488,198,235 learners). The total number of affected learners due to nationwide closures and localized closures totals 1,725,082,528. The trend of nationwide closures has decreased from previous figures recorded on April 1 – 4, 2020, where 1,598,099,000 learners, or 91.3% of total enrolled learners, were recorded as being affected due to the nationwide closures policy in 195 countries. However, the total affected number (nationwide and localized) is relatively the same, starting from March 26, 2020. This data indicates that although the national closures policy has begun to be revoked in some areas and regions, schools and local governments have continued to implement remote learning. In Indonesia, nationwide school closures began on March 23, 2020, while localized closures in some provinces, e.g., DKI Jakarta, Central Java, Banten, West Java, and Aceh, started early on March 16, 2020 (Kumparan, 2020). The national school closures have impacted 60.2 million learners and 2.3 million educators who study or teach at 425,451 educational institutions from early childhood to higher education.
  • 10. 3 ©2020 The author and IJLTER.ORG. All rights reserved. Table 1: Number of affected academic institutions, learners and educators in Indonesia (Ministry of Education and Culture, 2019, 2020c, 2020a, 2020d, 2020b) Level Academic Institutions Learners Educators Early Childhood Education 202,991 6,543,758 666,678 Elementary School 149,435 25,203.371 556,969 Junior High School 40,559 10,112,022 302,116 Senior High School 13,495 4,976,127 212,715 Vocational School (Equivalent to Senior High School) 14,301 5,242,851 217,986 College/University 4,670 8,043,480 294,820 TOTAL 425,451 60,121,609 2,251,284 School closures are based on the previous studies of influenza outbreaks that resulted from social interactions between students and teachers that disrupted the learning process (Jackson et al., 2016). Cauchemez et al., (2008, 2009) quantified the role of schools in influenza epidemics and predicted the effect of school closures during a pandemic in France. They found that sustained school closures during a pandemic could reduce the total number of cases by 13–17 percent (18– 23 percent in children), while during the height of the attack the rate could be as much as 39–45 percent (47–52 percent in children). The effect of school closures would be minimized if it proved difficult to sustain low contact rates among children for a prolonged time. Earn (2012) looked at the associations between the occurrence of H1N1 (pH1N1) pandemic influenza in Alberta, Canada in 2009 and school closures or weather changes, and calculated the impact of school closures and weather changes on the transmission of pH1N1. Mathematical models indicated that school closures decreased transmission among school-aged children by more than 50 percent, which was a critical factor in interrupting transmission. The models also showed that seasonal changes in weather conditions had a significant effect on the epidemic's temporal structure. School closure is one of the critical components of numerous non-pharmaceutical prevention strategies in many countries to help reduce the number of cases and slow the spread of the disease, mainly if vaccines or antivirals are not available or are ineffective due to resistance (Glass et al., 2006; Institute of Medicine, 2006; Moscona, 2005; Sadique et al., 2008). School closure policies in various countries occur in quick succession as a measure to contain the COVID-19 virus. Lee (2020) argues that school closures alone may not be enough to stop the outbreak, but that if it continues for at least eight weeks, could delay the outbreak peak by up to a week, providing adequate time for a second more successful intervention, such as vaccination. In this study, the researcher used the term "Emergency Remote Learning (ERL)" to illustrate the education that took place during school closure, not online or virtual
  • 11. 4 ©2020 The author and IJLTER.ORG. All rights reserved. learning, since well-planned online learning experiences are substantially different from those that are delivered online in response to a crisis or catastrophe. Hodges et al. (2020) call it Emergency Remote Teaching (ERT). ERT is a temporary change from instruction delivery to alternative delivery due to crisis circumstances. It requires the use of entirely remote teaching approaches for instruction or education that would otherwise be provided face-to-face or as blended or hybrid courses, and which will revert to that model once the crisis or emergency has finished. The primary objective in this context is not to re-create a stable educational environment, but rather to provide immediate access to education and training in a manner that is easy to develop and easily accessible during an emergency or crisis. Millman (2020) described the situation as emergency remote teaching and learning — or "pandemic pedagogy." This study focused on understanding how remote learning during COVID-19 pandemic has been conducted in a tertiary education institution by analyzing university students' perspectives and experiences. The limited research was conducted recently in Indonesia during the midst of the COVID-19 pandemic. Most of the existing research conducted is in Indonesian, published in Indonesian journals, focuses on either primary or secondary education, or education in general (the level of education is not explained). Examples of this include, "COVID-19 impact on the implementation of online learning in primary schools" (Aji et al., 2020); "Education during the COVID-19 pandemic" (Khasanah et al., 2020); "Studying Mathematics in the COVID-19 era" (Abidin, 2020); and "Online learning in the middle of the COVID-19 pandemic" (Firman & Rahayu, 2020). There are also several articles on learning in higher education during the COVID- 19 pandemic. These articles are written in Indonesian, with limited data and have not been published through peer-reviewed processes such as “Students’ learning typology in online learning”(Ramdhan et al., 2020); “Google Classroom-online learning of Biology education students during the COVID-19 outbreak”(Suhada et al., 2020); “Ordinal Regression analysis to see the effect of online learning media on students’ enthusiasm in the COVID pandemic era”(Meiza et al., 2020). Some articles published in peer-reviewed Indonesian journals focus on online learning of COVID-19 at the college level. However, they were published in April, where the learning from home experience had only been in operation for a few weeks. Experiences from a short implementation time are unlikely to offer a full picture of home-based teaching and learning activities, while there was also minimal use of references. The published articles are “The impact of COVID-19 online learning activities in a Christian university in Indonesia”(Windhiyana, 2020); “Implementation of the performance model of DeLone and McLean Knowledge Systems For Zoom application-based learning systems during the COVID-19 pandemic”(Hidayatullah et al., 2020); “Perception of students majoring in Early Childhood Islamic Education towards online Lectures in the COVID-19 pandemic period”(Anhusadar, 2020); and “Variations of models and learning platforms for prospective teachers during the COVID-19 pandemic period”(Gunawan et al., 2020).
  • 12. 5 ©2020 The author and IJLTER.ORG. All rights reserved. Figure 1. Theoretical Model The theoretical model outlined in Figure 1 shows what the related studies have been performed, what this study was intended to do, and the research questions raised by the context analysis and previous research. Recent studies have described school closure as one of the key components of a range of non- pharmaceutical prevention approaches aimed at minimizing the number of cases and preventing the spread of disease. Some of the previous emergency remote learning studies concluded that ERL is a temporary learning mechanism, a transition from in-person instruction to substitute delivery due to an emergency. Nonetheless, little is known on how to apply remote learning in higher education during the outbreak of COVID-19 and how students perceived and experienced ERL implementation. Research on the implementation of home learning during COVID-19 in Indonesia has been listed above. None of those papers use the term remote learning or ERL. The use of the term affects the analysis and understanding that is predicated in these studies. It is crucial to make a distinction, as the term refers to a unique and specific event (Manfuso, 2020). This research filled the gap in knowledge (theory) of ERL at the university level in Indonesia with insights and observations collected from students who had been studying from home for an extended period (more than ten weeks), had undergone an online mid-term exam, and was at the end of the semester. Other than that, this research disseminated its findings in English articles in international journals, so that more audiences could read and learn from the research findings.
  • 13. 6 ©2020 The author and IJLTER.ORG. All rights reserved. This research also filled the practical gap on how to successfully implement ERL in higher education to meet learners' needs. This study's main question is: how university students experienced ERL due to the COVID-19 crisis? Through knowing how university students learned in the emergency COVID-19, we could improve on in-going practice. The students' feedback is also critical to improve the learning process in the future if such a situation was ever to occur again. It is unknown how long ERL will continue or what kind of other problems we may face in the future, so educational institutions and students alike must be better prepared for any such eventuality. 2. Materials and Methods The current study is presented as part of a larger project on ERL in tertiary education in Indonesia during the COVID-19 outbreak. The overall aim of the study was to explore the implementation of ERL in tertiary education in Indonesia. The study employed a qualitative method of the phenomenological approach. This phenomenological analysis aims to collect and examine the lived experience of university students who have learned from home, due to the COVID-19 crisis, for more than two months. These experiences provided a new level of insight into ERL's implementation due to the outbreak of COVID-19. A phenomenology is a philosophical approach to the analysis of phenomena that are part of human awareness (Arbnor & Bjerke, 2011). In phenomenology, the researcher attempts to provide a direct description of someone's experience as it is, without considering his or her psychological origin and the causal explanations that the scientist may be able to provide (Merleau-Ponty & Landes, 2013). Phenomenology is equipped to understand the subjective interpretations of the participants of the fundamental object of the study by throwing light on the significance applied to their lived experience and exploring the specific meanings underlying the empiric variants of a given phenomenon rather than merely applying the technique (Casmir, 1983; MacDermott, 2002). Participants in this research are 80 students who have studied social science education at a public university in Jakarta. The researcher employed a purposive sampling method to select the participants. The researcher understood the purpose of the study in such a way that she relied on her judgment to classify qualified participants in a particular profile. The various explanations why the researcher chose the participants are as follows: the first reason was the ease of access for the researcher to collect data due to the COVID-19 situation, which restricted the researcher's ability to gather data. The second reason, they were semester four students, which means that they were in the middle of their course (undergraduate education usually takes eight semesters). These students already have enough learning experience and were still taking compulsory and additional courses (semester seven students practice teaching, and in semester eight, write a thesis). Third, students at this university come from Jakarta and other provinces and different backgrounds. This diversity will help enrich the data further. Lastly, the researcher intentionally chose students from educational programs because they were prepared to become teachers, and their opinions are important because their answers address how an educator should teach at a time like this.
  • 14. 7 ©2020 The author and IJLTER.ORG. All rights reserved. These students are in the fourth semester and came from two classes, 4A and 4B. These two groups have different concentrations, 4A is Sociology, and 4B is Geography. The name of the university and the names of the students have been hidden; to protect the identity of the participants and to provide them with the opportunity to speak freely. Participants were aware of the scope and intent of the study, and they were allowed to withdraw from the research anytime they chose if they felt dissatisfied with it. Data collection was carried out using creative methods because the researcher was unable to conduct direct interviews due to the large-scale social restrictions that were in place during the research period, data collection was completed by asking participants to write a diary of their daily learning activities for two weeks (4 - 18 May 2020. See Appendix 1. Students’ Diary). They were then asked to compose a reflective essay about ERL (collected on May 20, 2020. See Appendix 2. Reflective Essay). The researcher paid close attention to data triangulation to develop a comprehensive understanding of the phenomena. Therefore, the researcher used multiple methods or data sources, as suggested by Patton (Patton, 1999). The data source utilized, other than diaries and essays, was focus group discussions using the Google Meet application. These discussions were conducted twice; each meeting was for 120 minutes and 40 participants (on May 26 and 27, 2020). The focus group discussions were also aimed as a member check to increase the precision, reliability, validity, and transferability of the findings. The researcher presented the study's overall findings and then opened a discussion session. Participants were free to question if any incorrect information had been recorded and were allowed to add or remove details. The participants either confirmed that the summaries represented their beliefs, feelings, and experiences or did not reflect their experiences. The researcher concluded that the study seems credible after the participants confirmed the accuracy and completeness of the study. The complete elimination of non-response bias from the study is impossible. Mixing data collection methods was one of the researchers' efforts to avoid non- response impact from the sample. The researcher also sought to eliminate bias by designing strategies that would promote the involvement of selected individuals. The researcher used the NVivo program for data management and analysis. NVivo is useful in organizing data and helped the researcher make sense of it throughout the research since the data for this qualitative research was extensive (80 participants). The researcher also made a memo in the NVivo system that allowed for the possibility to document ideas and analyze thoughts, perspectives, and observations of students on online learning. This analytical memo was completed on an ongoing basis, every time data (journal or essay) was submitted. Analytical memos provide a means for the researcher to record their thoughts during the research process and to code memos as additional data for the study (Saldaña, 2016). In analyzing the data, the researcher adopted a two-stage coding model conducted by Miles, Huberman and Saldaña (Miles et al., 2014) and Saldaña (Saldana, 2009; Saldaña, 2016). The first cycle coding was initially assigned to the
  • 15. 8 ©2020 The author and IJLTER.ORG. All rights reserved. data, followed by the second cycle codes in which the initial codes were grouped into meaningful categories, themes, or constructs. The two stages of coding are not a single linear event; the qualitative analytical process is cyclical. In practice, in the first cycle, the researcher coded each essay and journal individually. After that, the researcher compared the data and detailed the code into a sub-code if necessary. In the second cycle, the researcher reorganized and re-analyzed the data coded in the first coding cycle. The primary aim of second-cycle coding was to establish a sense of categorical, thematic, conceptual, and theoretical structure from the first-cycle code series. The researcher changed codes, added new codes, and dropped a few codes to conclude themes of research findings. Finally, the researcher made the summative synthesis assumptions and declarative claims. The assertions are based on the researcher's insights and observations, supported by evidence from the data corpus. 3. Results How did university students experience ERL during the COVID-19 crisis? Results revealed that students’ experiences fell into two overarching themes, each with related sub-themes. The two identified themes and subthemes are: (a) blended learning, with the subthemes of e-learning, m-learning, and conventional learning, and (b) paradoxical learning, with the sub-themes of flexible learning and challenging learning. Figure 2. Research findings The researcher classified 25 codes, in the first cycle coding, from the data gathered from 80 diaries, 80 reflective essays, and two focus group discussions. The researcher then grouped related codes into five groups, which included: e- learning, m-learning, traditional learning, flexible learning, and challenging learning. The researcher summarized two key themes of this study's results in this second coding process: blended learning and paradoxical learning. The researcher concluded that during ERL, students experienced blended learning approaches, and their experiences were paradoxical. They enjoyed learning from home, but also saw it as challenging too. The following is an illustration of the process of codification, thematic discovery, and the construction of assertions. Emergency Remote Learning during COVID-19 Outbreak Blended Learning e-learning m-learning Conventional Learning Paradoxical Learning Flexible Learning Challenging Learning
  • 16. 9 ©2020 The author and IJLTER.ORG. All rights reserved. Figure 3. Data analysis process
  • 17. 10 ©2020 The author and IJLTER.ORG. All rights reserved. 3.1 Blended Learning Of the twenty-five codes found in the first coding cycle (as shown in Figure 2 above), thirteen are related to the media or methods used during the ERL. The researcher then grouped codes that show similar meaning in categories: e- learning, m-learning, and conventional learning. The researcher concluded from the categories that learning remotely during the COVID-19 situation involved a mix of technological learning media and conventional methods, or using blended materials. Table 2. Learning media/methods & frequency of being mentioned by participants Media/Methods Frequency Categories Reading E-book & E-Journal 56 e-learning & m-learning YouTube & Video 52 e-learning & m-learning Online Meeting (Zoom or Google Meet) 50 e-learning & m-learning Browsing Internet 42 e-learning & m-learning Educational Website 35 e-learning & m-learning Podcast 33 e-learning & m-learning Google Classroom 35 e-learning WhatsApp 42 m-learning Social Media 36 m-learning Audio Call 32 m-learning Assignments from Lecturers (exercises, quizzes, and exams) 58 Conventional Learning Making Notes 40 Conventional Learning Module/Unit 33 Conventional Learning From the coding of the data in the NVivo program, researchers have discovered how many frequencies a single code of all the existing data appears. Table 2 shows which media or methods are most mentioned repeatedly by students. Reading e-books and e-journals was the most common learning methods experienced by the students. Students also learned from YouTube and TV series or movie documentaries. Usually, lecturers provided students with YouTube links, asked them to study the video, and make a summary report. E-Books, E- Journals, and YouTube videos were used to help explain learning about the subject or giving instructions. Other platforms used to deliver the lecturers' instructions or explanations were through WhatsApp (the lecturer recorded voice notes and shared them in the WhatsApp group) and through the module or unit of work being studied.
  • 18. 11 ©2020 The author and IJLTER.ORG. All rights reserved. Three platforms were commonly used for classroom discussions: online meetings in Zoom or Google Meet, WhatsApp, and Social Media. They interchangeably used the term Social Media for WhatsApp also, while a few referred to Facebook when mentioning about Social Media. Not all the lecturers arranged synchronous discussions using online platforms. Many students complained about it; they felt that learning was ineffective without discussion. WhatsApp was the most preferred media for discussion. In WhatsApp, the discussion could be synchronous or asynchronous. The discussion could be made using either voice notes or texts. In order to enrich learning, students have tried to browse for more reading material online by searching the internet and by visiting educational websites. If they did not understand the subject, they would invariably contact their classmates, asking them to explain the subject. The respondents believe that YouTube videos are a good source of learning. Some listened to podcasts and explained that this was as a result of needing to be more creative in expanding their understanding. In measuring student learning progress, lecturers have delivered assignments and quizzes to students through WhatsApp and Google Classroom. Also, conventional methods were used for evaluation. The most common conventional tasks assigned to students are the memorization of texts and the writing of a summary. The data reveals that emergency remote learning has not always been entirely conducted online using sophisticated information technology; students have often learned using conventional media and methods. The students take notes, use paper, and pens to help them understand learning materials more efficiently. They took notes on the video, sound recordings, e-books, PowerPoint presentations, and journals provided by the lecturers. They felt that, in general, they had not learned anything if they did not write it in a book. They seemed to be able to understand more quickly when they wrote down what they had learned in a notebook. Participant 4B24 explained about making a summary of learning using paper and colorful pens and markers: "If the lecturer has given material in the form of power points, modules, and so on. I put together a summary of the material. And if there was a discussion on WhatsApp or Zoom, I try to write notes about things that I thought were important. Just like the previous way of learning (before the COVID-19 crisis), I used colored pens or markers to write notes, so I wouldn't get bored while reading notes." (4B24) Whereas 4B35 said that she had made a mind map of the content she had learned in order to be able to pass the mid-term and final exam. She added that she placed her notes on the wall, and every morning she memorized and learned them: “Although the mid-term test and end semester exams will be done online, I will still make a mind map to test my understanding of the material... for the final exam, I will make keywords from each material that I have studied, I will make notes in my paper and paste them on the wall, and every morning I will memorize and learn.” (4B35) Students also conventionally learnt using modules or units, not a printed version, but an e-module in PDF format. Two lecturers provided these e-modules
  • 19. 12 ©2020 The author and IJLTER.ORG. All rights reserved. containing discussions, quizzes, and a summary. The aim was to help students learn independently. They can test their comprehension by answering the quiz in the module and checking its accuracy by matching the answer key on the last page of the module. The lecturers shared the module every 1-2 weeks. Some students printed the modules or made a hand-written summary of the modules. The students also explained that they felt more like they had studied when they have paper, pen, and writing in a book. "In the pandemic from the COVID-19 crisis, my learning is using modules or PowerPoint presentations that I have obtained from lecturers or notes from my colleagues." (4C27) "I try to understand the tasks or modules provided, read all through them, yet I still don't get it." (4C30) Another conventional learning method is lecturers giving students assignments. Students complained they actually had to finish too many assignments. “Lecturers have been giving assignments online, a lot of them. Some lecturers only give assignments continuously.” (4B21) In ERL, blended learning was used for instruction and explanation, discussion, evaluation, and enrichment. The researcher mapped out the thirteen media/learning methods mentioned by the students in diaries and essays and the purpose of their use. Here is the following map: Figure 4. Blended media in ERL and their purpose of use
  • 20. 13 ©2020 The author and IJLTER.ORG. All rights reserved. WhatsApp is a prevalent medium used for all these learning purposes. WhatsApp is used to explain, discuss, evaluate and enhance learning. In addition, WhatsApp is also used for almost all courses during the ERL period. Only one of the eight courses in Class 4A and Class 4C did not use the WhatsApp. This assertion is derived from tabulations made on the types of learning platforms used for each subject. Table 3. Courses and learning platforms No CLASS 4A CLASS 4B COURSES LEARNING PLATFORMS COURSES LEARNING PLATFORMS 1. The New Indonesian History • YouTube • Assignment via WhatsApp • Assignment via Email The New Indonesian History • YouTube • Assignment via WhatsApp • Assignment via Email 2. Study & Learning • Google Classroom • YouTube • Google Meet • WhatsApp Study & Learning • Google Classroom • YouTube • Google Meet • WhatsApp 3. Educational Psychology • Module • Email • Google Meet Educational Psychology • Module • Email • Google Meet 4. Regional Geography of the World • Reading Material • Academic Instruction System • WhatsApp Regional Geography of the World • Reading Material • Academic Instruction System • WhatsApp 5. Learning Strategies • Assignment • Google Classroom • WhatsApp Learning Strategies Google Classroom WhatsApp • Assignment • Google Classroom • WhatsApp 6. Practicum Reading the Quran • Assignment • WhatsApp Email Practicum Reading the Quran • Assignment • WhatsApp Email 7. Entrepreneurs hip • Google Scholar • WhatsApp Group • Voice note Entrepreneur ship • Google Scholar • WhatsApp Group • Voice note 8. Sociology of Religion • Email • Voice Note • WhatsApp • PowerPoint Presentation Cartography • Lecturing in Zoom Meeting • Google Meet • Assignment • WhatsApp During the semester, students from class 4 A and 4 B studied eight courses, seven of which were the same for both classes and taught by the same lecturers. The seven courses taught in Class 4A and Class 4B were: New Indonesian History, Study and Learning; Educational Psychology; Regional Geography of the World; Learning Strategies; Practicum Reading the Quran; and Entrepreneurship. The distinctions were: Class 4A took Sociology of Religion, and Class 4A took Cartography.
  • 21. 14 ©2020 The author and IJLTER.ORG. All rights reserved. 3.2 Paradoxical Learning The second main theme of the findings is “paradoxical learning.” The researcher used this concept to describe how students had paradoxical perspectives and insights into learning. On the one hand, they saw ERL as flexible, but on the other hand, they also saw it as challenging. Table 4. Students’ insights & observations on Paradoxical Learning Insights/Observations Frequency Reasons Time Management (Flexibility) 83 Flexible Learning Family Time 52 Flexible Learning Exercising 51 Flexible Learning Comfortable & Quiet Place 42 Flexible Learning Break & Rest 40 Flexible Learning Refreshing 36 Flexible Learning Technology Barriers 85 Challenging Learning Cost of Internet 83 Challenging Learning Overload Assignments 73 Challenging Learning Trouble Finding Materials 51 Challenging Learning Tiring 49 Challenging Learning Noisy & Disturbing 42 Challenging Learning Table 4, we can see how many times students reported certain insights or observations about ERL paradoxically of all the existing data. The researcher used color codes to show how they saw learning as two polar opposite scales. The two- yellow color coded cells are time management and overload tasks. The three- orange color coded cells are family time, comfortable and quiet, and noisy and disturbing. The three-green coded cells are exercising, break and rest, and are tiring. Four-blue coded cells are refreshing, technology barriers, internet costs, and trouble finding materials. The segregated color-coded data indicates how students had paradoxical perspectives and insights into learning. On the one hand, students saw ERL as flexible; on the other hand, they also saw it as challenging. Figure 5. Flexibility versus overloaded Flexibility and time management Overloaded with assignments Flexible Learning Challenging Learning
  • 22. 15 ©2020 The author and IJLTER.ORG. All rights reserved. Students said that learning remotely at home gave them the flexibility to manage their time. “The flexible learning schedule matches our lifestyle; young people. Lecturers and students are increasingly turning to online learning as a viable alternative to study anywhere, anytime.” (4B26) Contradictory to the amount of flexibility, they also argued that the lecturers overloaded them with assignments, and they, therefore, found it difficult to manage their time. "It's not necessarily online learning or lectures that we were given, but now there are remote assignments (online). And the tasks that were given never stopped every week, we were given assignments by each subject, and we were chased by deadlines. In my view, it can make students frustrated and depressed. As we now know that we need to retain our body's immunity to avoid the virus, and the stress or pressure caused by the assignments may, in my opinion, disrupt the students' immunity from the body." (4C33) Figure 6. Family time, comfortable & Quiet versus noisy & disturbing The students said that during remote learning, they have a lot of family time and could study in a comfortable and quiet place. “Studying online makes it easy for us to find places that make it easier to think about, such as open spaces, indoor rooms, and family rooms at home.” (4B17) In contrast to this, they also felt disturbed by their siblings and the noise at home. “The home environment cannot be controlled when there is a live lecture that uses applications such as Google Meet and synchronous conversation. The situation of the house is crowded. It's very annoying, in my opinion, and it makes me unable to focus.” (4B18) Family Time Comfortable & Quiet Place Noisy and Disturbing Flexible Learning Challenging Learning
  • 23. 16 ©2020 The author and IJLTER.ORG. All rights reserved. Figure 7. Exercising, break & rest versus tiring Some students did some regular exercise at home. They spared extra time for self- care. "If there are no assignments, I will take the time to exercise and bask in the healthy morning sun." (4B28) "After I wake up, I do regular exercise or aerobic exercise. I'm doing this regularly. Learning from home allows me to exercise more." (4C38) Because the students were staying at home, they were able to take a break during learning. They did not want to push themselves too hard to study. Their main focus was to keep healthy and not be too tired. “In addition, I also tried not to be too tired and forced to study because of the consequences of the Covid-19 pandemic, I had to be careful not to fall ill. I also don't push myself too much when I'm tired of studying.” (4B15) Other respondents, however, felt that compared to face-to-face learning, remote was much more tiring. "I want to complain that this distance learning is very ineffective and makes me even more exhausted." (4C20) “A lot of tasks! The lecturer should consider the fact that a student is staying at home to avoid illness or infection from happening. The lecturer gave us lots of assignments excessively. It kills us. Not because of the coronavirus that causes us to die. May Allah Amen protect us.” (4C18) Figure 8. Refreshing versus problems Exercising Break & Rest Tiring Flexible Learning Challenging Learning Refreshing Technological barriers Cost of the internet Trouble finding materials Flexible Learning Challenging Learning
  • 24. 17 ©2020 The author and IJLTER.ORG. All rights reserved. The students said they tried to manage their time and refresh themselves whenever they were tired. They said this is what made learning from home more flexible. “Learning activities must be interspersed with activities that are fun, refreshing. In order not to be boring, I always prepare fun activities (in my opinion). This activity can also be a reward or reward when I have completed an assignment or studied material.” (4C21) They also complained, however, about technological disruption while studying, expensive internet costs and difficulties in obtaining learning materials during the ERL. "The difficulty of accessing the Internet, I was at home during this distance learning process. This makes me experience delays in getting information because access to the Internet in the village is a bit difficult, the internet connection is bad in my area." (4B18) "It's very pricey to spend money on the internet data plan. I'm a student who gets minimal pocket money, especially on staying at home like this, I don't get any income from being a Scoutmaster. That makes it hard for me to buy a data plan. I'm also envious of campuses that provide a free data plan of up to 150 thousand rupiahs per month." (4B12) "This policy has a side where I don't like it, for example, because I can't go anywhere, so I have trouble finding references that I need to complete my assignments. Learning resources are also limited and can only refer to the internet and journals that aren't always right" (4B19). 4. Discussion Indonesia reported the first two confirmed cases of COVID-19 in a Jakarta suburb on March 2, 2020 (World Health Organization, 2020a). The situation became increasingly dangerous in mid-March. President Jokowi, in a press conference at the Bogor Presidential Palace on Sunday, March 15, 2020, called on local governments to issue policies to prevent the spread of the coronavirus. One of the policies recommended was to temporarily halt teaching and learning in schools and universities and encourage students to study at home. As a follow-up to the president's direction, on March 20, 2020, the Minister of Education and Culture issued a Circular Number 36962/MPK.A/HK/2020 concerning online learning and working from home in the context of the prevention of coronary disease spread. Following this rule, the campus, which was the focus of this research, began to close on March 23, 2020. Learning was suddenly moved to remote learning. Remote learning has not been anticipated, and most of the course syllabus was not designed to be an online or for a distance learning experience. Without preparation or training, lecturers designed and implemented remote learning programs. Similarly, students did not have the opportunity to be prepared for this transition in learning. Despite all of these limitations, everybody has been trying their best to make the situation a success, quick and reliable temporary access to education in unprecedented time.
  • 25. 18 ©2020 The author and IJLTER.ORG. All rights reserved. Lecturers began using the media or the methods they were familiar with and comfortable using. It may not be perfect, but students are resilient, the educator is resourceful, and everyone has to make the most of what they have (Juliani, 2020). This study reveals that learning remotely during the COVID-19 situation involved a mix of both conventional and non-conventional methods. Technological learning media was used combined with pen and paper methods. The use of blended materials is what the researcher concluded as the first theme. Blended learning is the application of more than one method, strategy, technique, or media in education (Sadeghi et al., 2014; Thiele, 2003). It is a mixture of conventional lectures or tutorials and web-based material (Concannon et al., 2005); when delivering information, traditional face-to-face learning is combined with technology (Farrel, 2006). In line with the government's policy of social distancing, learning has taken place at home. Teachers and students separated by physical distance but connected through technology, equipment, and resources. Nonetheless, the data indicate that learning was not entirely online; students were still studying in conventional ways. Conventional here does not mean that there have been face-to-face lectures, but conventional learning methods for taking notes, summarizing learning and memorizing notes. In this study, the researcher distinguished between e-learning and m-learning based on the devices used. E-learning means that the educational model used is based on the use of electronic media and devices as tools for improving the availability of training, communication, and interaction, and that helps to accept new ways of understanding and learning (Salloum et al., 2019). An essential element of e-learning is the use of electronic media, and e-learning is currently explained as learning that is used through various computer devices, such as computers, mobile phones, tablets, and virtual environments. M-learning focuses on learning mobility, engagement with portable technology, and learning that represents an emphasis on how society and its institutions could accommodate and sustain an increasingly mobile population (Mehdipour & Zerehkafi, 2013). By using mobile apps, learners can learn anywhere and anytime (Crescente & Lee, 2011). In Indonesia, internet users have reached 150 million people, with a 56 percent penetration spread across the region. There is only a small difference with the number of mobile internet users, which amounts to 142.8 million people, with a penetration percentage of 53 percent (Ministry of Communication and Information Technology Republic of Indonesia, 2020). So, most people are linked to the internet through mobile phones. Many homes do not have laptops or PCs, but they are still connected online using mobile devices. The distinction between e-learning and m-learning clarifies this phenomenon. In case of emergencies, remote learning students have explained that they are connected using mobile devices or tethered from a cell phone when they need to work from a laptop. Blended learning is supposed to make learning more efficient by integrating technology-based learning with traditional style learning. Some research has identified many benefits of blended learning. Many studies have been carried out to compare conventional teaching methods with blended learning, and different results have been obtained. In some research, the participants' test scores did not differ between the two groups (Khan, 2001). Thiele's study showed that the
  • 26. 19 ©2020 The author and IJLTER.ORG. All rights reserved. student knowledge scores in the blended teaching method were higher than the lecturing method, but that the difference was not statistically significant (Thiele, 2003). Sedeghi et al. contrasted students' learning and satisfaction with teaching and e-learning with traditional teaching methods. The results showed that the blended method is effective in increasing the student learning rate (Sadeghi et al., 2014). McPhee and Pickern (2017) concluded how ICT could help international students' learning experience. Kirkwood (2009) claimed that ICT could make doable learning tasks or situations that would otherwise be extremely difficult to achieve and encourage an incremental improvement in learning outcomes. Stephenson, Brown, and Griffin (Stephenson et al., 2008) argued that most e- learning seemed to mimic or complement established academic practices, mainly when used in 'blended' contexts. Another study suggests that a combination of face-to-face training with e-learning is more flexible than other methods (Garrison & Kanuka, 2004). This ERL might not have been the ideal one. Blended learning has been questioned as to the best solution to studying during the pandemic when universities are closed. The advantages of blended learning described in the studies, as mentioned above, are in contrast to the experience of students. Afterward, we must remember that this blended learning took place in an emergency with limited resources. As highlighted in the introduction, the researcher refers to it as "emergency remote learning," not just "remote learning" using blended materials. Before COVID-19, the classes studied by the researcher rarely used e-learning and m-learning. E-learning and m-learning have emerged as one of the alternatives to the problems of education during an outbreak of coronavirus. E-learning and m-learning offer expanded opportunities for learning versatility when students and educators are under the stay-at-home order. However, there were challenges in using technology in learning. Students talked about the technical barriers that hindered their learning due to technical problems or the inability to use technological resources. Students also discussed the shift in learning styles. Students were already accustomed to face-to-face conventional teaching approaches. They claimed in their writings that they felt that they were not learning since there was no lecturer to guide them to understand the lesson. E-learning and m-learning involve students working interdependently, in groups or independently to solve problems, to work on projects, to meet individual needs, and to encourage students to speak and choose. Students complained about this kind of independent learning, doing assignments, and reading on their own. They were familiar with the teaching-learning model, where lecturers gave learning materials; students listened and took notes. Another thing to think about is the availability of learning resources in Bahasa Indonesia. Students said that they have trouble getting learning materials since the libraries are closed. Universities have no access to paid online journals. So, students are only using open-access materials. The findings of this study reveal that university and its community were not well prepared to face an emergency, such as the closure of the campus due to a pandemic. Courses designed for conventional learning, the unusual use of
  • 27. 20 ©2020 The author and IJLTER.ORG. All rights reserved. technology in teaching and learning programs, students were not able to study independently, limited access to learning materials in the native language of learners, and lack access to high-quality reading resources (e.g., paid e-journals and e-books). All of these issues need to be highlighted and addressed. Now and in the future, the university should be more involved in information communication technology. University learning should encourage students to learn independently through a wide range of methods and media. Resources for remote learning need to be strengthened, and students should get more access to reading materials Another theme that arose from this study is ERL during the outbreak of COVID- 19 is paradoxical. Paradoxical, according to the Cambridge dictionary 2020), is, "Seeming impossible or difficult to understand because of containing two opposite facts or characteristics." If something is paradoxical, it involves two facts or qualities that seem to contradict each other (Collins Dictionary, 2020). The university moved in-classroom-learning to online-based learning without enough preparation. This rapid change was a test of organizational agility (Wu, 2020). Students and lecturers were still adapting and looking for ways of teaching and learning that were much better and easier. Educational planning in times of crisis needs creative problem solving (Hodges et al., 2020). ERL needs much hard work, but this is the only way forward. After some time, it is essential to review, develop the current practice, and plan for future programs. Learning from home might continue in the next few months, and this emergency could happen again in the future. What are we supposed to do to make the ERL better? Bao (2020) observed online teaching during the outbreak of COVID-19. Bao has identified six educational strategies to improve student concentration and engagement to achieve a smooth transition to online learning: 1. Preparing emergency preparedness plans for unexpected problems. If the teaching plan A is not working because of a technical problem, try Plan B, C, or any other plan. 2. Distributing the teaching content into smaller units to help the student focus. The lecturers should divide the teaching content into several small modules lasting approximately 20–25 minutes. 3. Emphasizing the use of "voice" in online teaching. Body language and facial expressions are restricted in online instruction, and only the "voice" is entirely functional. Therefore, in online teaching, the faculty should properly slow down its speech to allow students to capture critical knowledge points. 4. Working with teaching assistants and get online help from them. This arrangement is also missing from the students' courses in this research. There are no teaching assistants for the lecturers. Online teaching includes the technology and professional expertise of those involved. Most faculties at universities in Indonesia are not sufficiently trained or equipped to operate online educational platforms, and support from teaching assistants is especially relevant. 5. Strengthening the students' ability to learn outside the classroom. The faculty has less control over online teaching. As a result, the progress of
  • 28. 21 ©2020 The author and IJLTER.ORG. All rights reserved. online teaching and its learning efficiency depends mainly on high-level active learning outside the classroom. To this end, the faculty should use various methods to moderately modify students' homework and reading requirements to strengthen students' active learning outside the classroom. Students complained about assignments in this research. Lecturers and students need a dialogue to address what is currently expected and should be achieved by two parties, students and lecturers, to make learning more productive. 6. Efficiently combining online learning and offline self-learning. In the offline self-learning process, students are expected to read the course- specific literature and submit short papers based on their reading of essential materials before class. The faculty should provide feedback on student assignments and learn about students' cognitive learning levels. In the online teaching phase, the faculty should use the student discussion section to exchange their understanding based on their reading. Thus, students will not learn ambiguous, fragmented, and superficial knowledge. Instead, deep learning will be experienced during the discussion. Students in this research complained that not enough lectures and lecturers had given them too many assignments until they felt overloaded. This should be communicated as to why assignments are part of learning. However, lecturers should give feedback on the work of students, so that they know the progress of their learning. Bao (2020) also stressed the need to provide psychological support to the university community. In this research, students viewed remote learning in both a positive and bad light. In their diaries, they wrote about confusion, depression, a devastating feeling, and some negative emotions about being isolated and learning remotely. These feelings have to be addressed. The university should provide psychological support to help students and staff manage their emotions. Their anxieties need to be relieved in a variety of ways to ensure that they can engage in online learning actively. 5. Conclusion This study concluded that students experienced an ERL that utilized blended strategies of e-learning, m-learning, and conventional learning techniques. Their response to the process was paradoxical as they had both positive and negative experiences. Lecturers used significantly limited media and methods to implement learning. Moving instruction online could make teaching and learning accessible anywhere, at any time, but the speed at which this transition to online instruction is expected to happen is unparalleled. After experiencing ERL for almost a semester, and while currently waiting for government policies on how to learn next semester, it is an excellent time to learn from multiple viewpoints, including students, about their experience. Some of the feedback from this research are: preparing teachers and students for emergency learning, training lecturers on using blended resources, designing learning curricula that can be converted into online learning, developing student-independent learning skills, and providing access to a wide variety of digital reading materials. This study's findings cannot be generalized because they were based on a relatively small number of university students from one department at one university. However,
  • 29. 22 ©2020 The author and IJLTER.ORG. All rights reserved. we assume that this work presents students' views and experiences on ERL in several universities in Indonesia and probably in other countries, particularly in developing countries. More research involving more students from various universities should be initiated as a follow-up. It would also be useful to gain insights from lecturers and students at different levels of education and discuss the viewpoints of parents and other education stakeholders. Funding: This research received no external funding. Conflicts of Interest: The authors declare no conflict of interest in the design of the study; in the collection, analysis, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results. 6. References Abidin, Z. (2020). Belajar Matematika di Era Covid-19 [Studying Mathematics in the COVID- 19 era]. OSF Preprints. Retrieved from https://osf.io/nrbu7/ Aji, W., Dewi, F., Kristen, U., & Wacana, S. (2020). Dampak Covid-19 Terhadap Implementasi Pembelajaran Daring Di Sekolah Dasar [COVID-19 impact on the implementation of online learning in primary schools]. Edukatif: Jurnal Ilmu Pendidikan, 2(1), 55–61. https://doi.org/https://doi.org/10.31004/edukatif.v2i1.89 Anhusadar, L. O. (2020). Persepsi Mahasiswa PIAUD terhadap Kuliah Online di Masa Pandemi [Perception of students majoring in Early Childhood Islamic Education towards online Lectures in the COVID-19 pandemic period]. KINDERGARTEN: Journal of Islamic Early Childhood Education, 3(1), 44–58. Arbnor, I., & Bjerke, B. (2011). Methodology for Creating Business Knowledge. In Methodology for Creating Business Knowledge. https://doi.org/10.4135/9780857024473 Bao, W. (2020). COVID ‐19 and online teaching in higher education: A case study of Peking University. Human Behavior and Emerging Technologies, 2(2), 113–115. https://doi.org/10.1002/hbe2.191 Cambridge Dictionary. (2020). Paradoxical. Casmir, F. L. (1983). Phenomenology and hermeneutics: Evolving approaches to the study of intercultural and international communication. International Journal of Intercultural Relations, 7(3), 309–324. https://doi.org/10.1016/0147- 1767(83)90035-4 Cauchemez, S., Ferguson, N. M., Wachtel, C., Tegnell, A., Saour, G., Duncan, B., & Nicoll, A. (2009). Closure of schools during an influenza pandemic. The Lancet Infectious Diseases, 9(8), 473–481. https://doi.org/10.1016/S1473-3099(09)70176-8 Cauchemez, S., Valleron, A.-J., Boëlle, P.-Y., Flahault, A., & Ferguson, N. M. (2008). Estimating the impact of school closure on influenza transmission from Sentinel data. Nature, 452(7188), 750–754. https://doi.org/10.1038/nature06732 Collins Dictionary. (2020). Paradoxical. Concannon, F., Flynn, A., & Campbell, M. (2005). What campus-based students think about the quality and benefits of e-learning. British Journal of Educational Technology, 36(3), 501–512. https://doi.org/10.1111/j.1467-8535.2005.00482.x Crescente, M. L., & Lee, D. (2011). Critical issues of m-learning: design models, adoption processes, and future trends. Journal of the Chinese Institute of Industrial Engineers, 28(2), 111–123. https://doi.org/10.1080/10170669.2010.548856
  • 30. 23 ©2020 The author and IJLTER.ORG. All rights reserved. Earn, D. J. D. (2012). Effects of School Closure on Incidence of Pandemic Influenza in Alberta, Canada. Annals of Internal Medicine, 156(3), 173. https://doi.org/10.7326/0003-4819-156-3-201202070-00005 Farrel, M. (2006). Learning differently: e-learning in nurse education. Nursing Management, 13(6), 14–17. https://doi.org/10.7748/nm.13.6.14.s12 Felter, C. (2020). What Is the World Doing to Create a COVID-19 Vaccine? Retrieved from https://www.cfr.org/backgrounder/what-world-doing-create-covid-19- vaccine Firman, F., & Rahayu, S. (2020). Online learning in the middle of the COVID-19 pandemic (Pembelajaran Online di Tengah Pandemi Covid-19). Indonesian Journal of Educational Science (IJES), 2(2), 81–89. https://doi.org/10.31605/ijes.v2i2.659 Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. The Internet and Higher Education, 7(2), 95–105. https://doi.org/10.1016/j.iheduc.2004.02.001 Glass, R. J., Glass, L. M., Beyeler, W. E., & Min, H. J. (2006). Targeted social distancing design for pandemic influenza. Emerging Infectious Diseases, 12(11), 1671–1681. https://doi.org/10.3201/eid1211.060255 Task Force for the Acceleration of COVID-19. (2020). Infografis COVID-19 (12 Juni 2020). Retrieved from https://covid19.go.id/p/berita/infografis-covid-19-12-juni- 2020 Gunawan, Suranti, N. M. Y., & Fathoroni. (2020). Variations of models and learning platforms for prospective teachers during the COVID-19 pandemic period. Indonesian Journal of Teacher Education, 1(2), 61–70. Hidayatullah, S., Khouroh, U., Windhyastiti, I., Patalo, R. G., & Waris, A. (2020). Implementasi Model Kesuksesan Sistem Informasi DeLone And McLean Terhadap Sistem Pembelajaran Berbasis Aplikasi Zoom Di Saat Pandemi Covid- 19 [Implementation of the performance model of DeLone and McLean Knowledge Systems For Zoom application-based learning systems during the COVID-19 pandemic]. Jurnal Teknologi Dan Manajemen Informatika, 6(1). https://doi.org/10.26905/jtmi.v6i1.4165 Hodges, C., Moore, S., Lockee, B., Trust, T., & Bond, A. (2020). The Difference Between Emergency Remote Teaching and Online Learning. Educause Review. Institute of Medicine. (2006). Modeling Community Containment for Pandemic Influenza. National Academies Press. https://doi.org/10.17226/11800 Jackson, C., Vynnycky, E., & Mangtani, P. (2016). The Relationship Between School Holidays and Transmission of Influenza in England and Wales. American Journal of Epidemiology, 184(9), 644–651. https://doi.org/10.1093/aje/kww083 Juliani, A. J. (2020). THIS IS NOT ONLINE OR DISTANCE LEARNING. Retrieved from http://ajjuliani.com/this-is-not-online-or-distance-learning/ Khan, F. A. (2001). Problem-based Learning Variant: Transition Phase for a Large Institution. National Library of Medicine, J. Pak Med Assoc, 51(8), 271–274. Retrieved from https://pubmed.ncbi.nlm.nih.gov/11715887/ Khasanah, D. R. A. U., Pramudibyanto, H., & Widuroyekti, B. (2020). Pendidikan Dalam Masa Pandemi Covid-19 [Education during the COVID-19 pandemic]. Jurnal Sinestesia, 10(1), 41–48. Kirkwood, A. (2009). E‐learning: you don’t always get what you hope for. Technology, Pedagogy and Education, 18(2), 107–121. https://doi.org/10.1080/14759390902992576 Kumparan. (2020). Anies Tutup Sekolah di DKI Jakarta Selama 2 Minggu untuk Cegah Corona [Anies closes school in DKI Jakarta for 2 weeks to prevent Corona]. Retrieved from
  • 31. 24 ©2020 The author and IJLTER.ORG. All rights reserved. https://kumparan.com/kumparannews/anies-tutup-sekolah-di-dki-jakarta- selama-2-minggu-untuk-cegah-corona-1t1V7X57VID Lee, B. Y., Brown, S. T., Cooley, P., Potter, M. A., Wheaton, W. D., Voorhees, R. E., Stebbins, S., Grefenstette, J. J., Zimmer, S. M., Zimmerman, R. K., Assi, T.-M., Bailey, R. R., Wagener, D. K., & Burke, D. S. (2010). Simulating School Closure Strategies to Mitigate an Influenza Epidemic. Journal of Public Health Management and Practice, 16(3), 252–261. https://doi.org/10.1097/PHH.0b013e3181ce594e Lee, T. H. (2020). Innovations in Care Creating the New Normal: The Clinician Response to Covid-19. NEJM Catalyst, 19–21. https://doi.org/10.1056/CAT.20.0076 MacDermott, A. F. N. (2002). Living with angina pectoris - A phenomenological study. European Journal of Cardiovascular Nursing. https://doi.org/10.1016/S1474- 5151(02)00047-6 Manfuso, L. G. (2020). From Emergency Remote Teaching to Rigorous Online Learning. Ed Tech. Retrieved from https://edtechmagazine.com/higher/article/2020/05/emergency-remote- teaching-rigorous-online-learning-perfcon McCarthy, M. (2020, June). Dr. Anthony Fauci: COVID-19 Will End and We Will Get Control Over It. Healthline. Retrieved from https://www.healthline.com/health- news/dr-anthony-fauci-this-will-end-and-we-will-get-control-over-it#6 McPhee, S., & Pickren, G. (2017). Blended learning with international students: a multiliteracies approach. Journal of Geography in Higher Education, 41(3), 418–433. https://doi.org/10.1080/03098265.2017.1331208 Mehdipour, Y., & Zerehkafi, H. (2013). Mobile Learning for Education: Benefits and Challenges. International Journal of Computational Engineering Research, 03(06), 93- 101. Meiza, A., Hanifah, F. S., & Natanel, Y. (2020). Analisis Regresi Ordinal untuk melihat Pengaruh Media Pembelajaran Daring terhadap Antusiasme Mahasiswa Era Pandemi Covid [Ordinal Regression analysis to see the effect of online learning media on students’ enthusiasm in the COVID pandemic era]. Retrieved from http://sinta.ristekbrin.go.id/covid/penelitian/detail/361 Merleau-Ponty, M., & Landes, D. A. (2013). Phenomenology of perception. In Phenomenology of Perception. https://doi.org/10.4324/9780203720714 Miles, M. B., Huberman, M. a, & Saldana, J. (2014). Drawing and Verying Conclusions. Qualitative Data Analysis: A Methods Sourcebook. Milman, N. B. (2020). Pandemic pedagogy. Phi Delta Kappan. Ministry of Communication and Information Technology Republic of Indonesia. (2020). Penggunaan Internet di Indonesia. Ministry of Communication and Information Technology Republic of Indonesia. https://aptika.kominfo.go.id/2019/08/penggunaan-internet-di-indonesia/ Ministry of Education and Culture. (2019). The 2019/2020 Early Childhood Education Statis. Retrieved from https://doi.org/10.1017/CBO9781107415324.004 Ministry of Education and Culture. (2020a). 2019/2020 Junior High School Statistic. Ministry of Education and Culture. (2020b). 2019/2020 Primary Education Statistics. Retrieved from http://statistik.data.kemdikbud.go.id/index.php/page/sd Ministry of Education and Culture. (2020c). 2019/2020 Senior High School Statistic. Ministry of Education and Culture. (2020d). 2019/2020 Vocational School Statistic. Moscona, A. (2005). Oseltamivir Resistance — Disabling Our Influenza Defenses. New England Journal of Medicine, 353(25), 2633–2636. https://doi.org/10.1056/NEJMp058291
  • 32. 25 ©2020 The author and IJLTER.ORG. All rights reserved. National Agency for Disaster Management. (2020). Infeksi COVID-19 Telah Menyebar di 34 Provinsi di Indonesia, Total Positif Jadi 3.512 Kasus [COVID-19 infections have spread in 34 provinces in Indonesia, a total of positive to 3,512 cases]. Retrieved from https://bnpb.go.id/berita/infeksi-covid19-telah-menyebar-di-34- provinsi-di-indonesia-total-positif-jadi-3-512-kasus Patton, M. Q. (1999). Enhancing the quality and credibility of qualitative analysis. Health Services Research. Ramdhan, D., Nasihudin, Rohaniawati, D., & Pratiwi, I. M. (2020). TIPOLOGI BELAJAR MAHASISWA PADA PEMBELAJARAN DARING [Students’ learning typology in online learning]. Digilib UIN Sunan Gunung Djati. Retrieved from http://digilib.uinsgd.ac.id/30594/1/Artikel KTI dadan f ramdhan dkk .pdf Reynolds, K. A. (2020). Coronavirus: Fauci says vaccine possible by end of 2020. Medical Economics. https://www.medicaleconomics.com/news/coronavirus-fauci- says-vaccine-possible-end-2020 Sadeghi, R., Sadaghat, M. M., & Ahmadi, F. S. (2014). Comparison of the Effect of Lecture and Blended Teaching Methods on Students’ Learning and Satisfaction. J. Adv Med Ed Prof, 2(4), 146–150. Sadique, M. Z., Adams, E. J., & Edmunds, W. J. (2008). Estimating the costs of school closure for mitigating an influenza pandemic. BMC Public Health, 8(1), 135. https://doi.org/10.1186/1471-2458-8-135 Saldana, J. (2009). An Introduction to Codes and Coding. The Coding Manual for Qualitative Researchers. https://doi.org/10.1519/JSC.0b013e3181ddfd0a Saldaña, J. (2016). The Coding Manual for Qualitative Researchers (No. 14). Sage. Salloum, S. A., Qasim M. A. A., Al-Emran, M., Abdel M. A., & Shaalan, K. (2019). Exploring Students’ Acceptance of E-Learning Through the Development of a Comprehensive Technology Acceptance Model. IEEE Access, 7, 128445–128462. https://doi.org/10.1109/ACCESS.2019.2939467 Stephenson, J. E., Brown, C., & Griffin, D. K. (2008). Electronic delivery of lectures in the university environment: An empirical comparison of three delivery styles. Computers & Education, 50(3), 640–651. https://doi.org/10.1016/j.compedu.2006.08.007 Suhada, I., Kurniati, T., Pramadi, R. A., & Listiawati, M. (2020). Google Pembelajaran Daring Berbasis Google Classroom Mahasiswa Pendidikan Biologi Pada Masa Wabah Covid-19 [Classroom-online learning of Biology education students during the COVID-19 outbreak]. Retrieved from http://sinta.ristekbrin.go.id/covid/penelitian/detail/363 Thiele, J. E. (2003). Learning Patterns of Online Students. Journal of Nursing Education, 42(8), 364–366. UNESCO. (2020). COVID-19 Impact on Education. Retrieved from https://en.unesco.org/covid19 Windhiyana, E. (2020). DAMPAK COVID-19 TERHADAP KEGIATAN PEMBELAJARAN ONLINE DI PERGURUAN TINGGI KRISTEN DI INDONESIA [The impact of COVID-19 online learning activities in a Christian university in Indonesia]. Perspektif Ilmu Pendidikan, 34(1), 1–8. https://doi.org/10.21009/PIP.341.1 World Health Organization. (2020a). Coronavirus disease 2019 (COVID-19) Situation Report – 42. World Health Organization. Retrieved from https://www.who.int/docs/default-source/coronaviruse/situation- reports/20200302-sitrep-42-covid-19.pdf?sfvrsn=224c1add_2 World Health Organization. (2020b). Draft landscape of COVID-19 candidate vaccines - 15 May 2020. World Health Organisation. Retrieved from
  • 33. 26 ©2020 The author and IJLTER.ORG. All rights reserved. https://www.who.int/who-documents-detail/draft-landscape-of-covid-19- candidate-vaccines World Health Organization. (2020c). World Health Organization COVID-19 Dashboard. Retrieved from https://covid19.who.int Wu, Z. (2020). How a top Chinese university is responding to coronavirus. World Economic Forum. Appendix 1. Students’ Diary Write every day about your daily learning routine. Illustrate and describe your experiences in 300-500 words a day. If you like, you can add a picture. Name: Day/Date Learning Activities Monday/4 May 2020 Tuesday/5 May 2020 Wednesday/6 May 2020 Thursday/7 May 2020 Friday/8 May 2020 Saturday/9 May 2020 Sunday/10 May 2020 Monday/11 May 2020 Tuesday/12 May 2020 Wednesday/13 May 2020 Thursday/14 May 2020 Friday/15 May 2020 Saturday/16 May 2020 Sunday/17 May 2020 Monday/18 May 2020 Appendix 2. Reflective Essay Write your thoughts on the implementation of emergency remote learning that has been going on for more than 2 months in 500-1000 words. Write down what you like and do not like about ERL.
  • 34. 27 ©2020 The authors and IJLTER.ORG. All rights reserved. International Journal of Learning, Teaching and Educational Research Vol. 19, No. 6, pp. 27-45, June 2020 https://doi.org/10.26803/ijlter.19.6.2 Examining the Fairness of Language Test Across Gender with IRT-based Differential Item and Test Functioning Methods Burhanettin Ozdemir Prince Sultan University, Riyadh, Saudi Arabia https://orcid.org/0000-0001-7716-2700 Abdulrahman Hadi Alshamrani National Center for Assessment, Riyadh, Saudi Arabia https://orcid.org/0000-0002-6560-3422 Abstract. Test fairness is an important indicator of the validity of test results. The fairness and equity require ensuring that the background characteristics of test-takers, such as ethnicity and gender, do not affect their test scores. Differential item functioning (DIF) methods are commonly used to detect potentially biased items that lead to the unfair assessment of the performance of test-takers with the same ability levels coming from the different cultural, social, demographic, and linguistic backgrounds. This study aims at detecting potentially biased items across gender and examining their effect on test scores to ensure the fairness of test results for each domain and the entire test. Item response theory (IRT) based Lord’s chi-square DIF method at item level and Mantel-Haenszel/Liu-Agresti differential test functioning (DTF) method at test level were implemented to the English Placement Tests (EPT) administered to high school graduates by the National Center for Assessment. The results show that 6 items of the EPT exhibit DIF for the entire test. Two of them are related to reading comprehension and four to the structure domain, while none of the compositional analysis methods shows DIF. These results indicate the existence of content specific DIF effect. Additionally, two items exhibit uniform DIF, one of which shows DIF favoring male students and the offer favoring female students. The small to moderate DTF effect associated with sub-domains and the entire test imply that DIF effects cancel each out, assuring the fairness of results at test level. However, the items with substantially high DIF values need to be examined by content experts to determine the possible cause of DIF effects to avoid gender bias and unfair test outcomes. We also suggest conducting further studies to investigate the reasons behind the content specific DIF effects in language tests. Keywords: test fairness and validity; gender bias; language testing; differential item functioning; differential test functioning
  • 35. 28 ©2020 The authors and IJLTER.ORG. All rights reserved. 1. Introduction The major concern of the stakeholders in education and test-takers is to ensure the fairness tests. The best way to provide fairness regarding the decision made upon a test is to increase the validity and reliability of test results. Therefore, any effort to minimize confounding factors such as random and systematic errors, and increase validity and reliability of test will serve the purpose of developing fair tests and valid test scores for examinees belonging to different groups. Examining the factorial structure of a test and differential functioning at the item level and test level are commonly used methods to assess the reliability and validity of test scores. Differential functioning may occur when items and tests produce different results for different groups consistently and therefore lead to invalid test scores and decisions made based on these scores. The stakeholders that take part in educational test development and assessment processes explicitly emphasize the importance of fairness in test results regarding different subgroups. They put a substantial amount of effort to detect irrelevant factors threatening the construct validity of the test. They are aware of the necessity and importance of collecting evidence to justify the validity and fairness of the tests and change the testing policies accordingly. Recently, the European Federation of Psychological Association has proposed a model for collecting evidence of construct validity (Evers, Muñiz, Hagemeister, Høstmælingen, Lindley, Sjöberg, & Bartram, 2013; Hope, Adamson, McManus, Chris, & Elder, 2018) in which using differential item functioning (DIF) is considered as an important method for assessing the quality of the test. Moreover, the Test Commission of the Spanish Psychological Association has emphasized the critical role of DIF analysis in the context of test fairness (Hernández, Tomás, Ferreres, & Lloret, 2015; Hope et al., 2018). A common definition of differential item functioning (DIF) is that an item is said to exhibit DIF when the probability of correct response to an item differs across subgroups with the same ability level (Hambleton & Rogers, 1989). DIF types are classified into two groups that are uniform DIF and non-uniform DIF. An item exhibits uniform-DIF when the difference in item performance is consistent and in favor of certain subgroups across the entire range of ability. However, if this difference between subgroups is not consistent, then DIF is identified as non- uniform (Hambleton, Clauser, Mazor & Jones, 1993). The existence of DIF is an indicator of item bias and the presence of the secondary latent trait besides the primary latent trait that an item aims to measure. However, this secondary latent trait does not always imply bias or cause unfair assessment. If the secondary latent trait is related to the primary trait and occurs due to the nature of the measured structure, then the item is not labeled as unfair regardless of the differing performance of sub-groups. This situation was illustrated in a study conducted by Drabinová and Martinková (2016). They found that one DIF item related to childhood illness in which females showed better performance than males. However, a detailed investigation of content experts revealed that this performance difference occurred since women are more experienced than men since they spend more time with their children in the Czech Republic (Martinková et al., 2017).
  • 36. 29 ©2020 The authors and IJLTER.ORG. All rights reserved. Therefore, the performance difference between women and men in this example reflects the true ability difference and does not cause unfairness. Therefore, an item may display DIF, however, this finding does not provide enough evidence to classify this item as a biased item. Bias is related to systematic error in test administration and contents and relies on both statistical tests and expert opinions (Camilli & Shepard, 1994; Clauser & Mazor, 1998; Wiberg, 2006), while DIF only relies on statistical tests. Although there are many different parametric and non-parametric methods to detect DIF, which method to utilize is a main concern of researchers, since each method has advantages and shortcomings (Anastasi & Urbina, 1997; Hunter, 2014). For instance, some methods fail to detect non-uniform DIF but effective when the sample size is small, such as the Mantel Haenszel and Rasch methods, while some methods are capable of detecting non-uniform DIF but requires large sample size (Ferne & Rupp, 2007; Lai, Teresi, & Gershon,2005) such as IRT based Raju’s area method and Lord’s chi-square method. These aforementioned methods are the most commonly used exploratory methods utilized to identify differential item functioning for categorical variables that represent existing subgroups such as gender, nationality, and age groups (Aryadoust & Zhang, 2016). The next step after detecting items exhibiting DIF is to investigate the possible source and cause of occurrence of DIF (Zhu & Aryadost, 2020). The EPT is administered to high school graduates by the National Center for Assessment (NCA) in Saudi Arabia. The results of the EPT has been used by several colleges, universities, and institutes to measure students’ language skills, to screen their improvements across different levels or to determine their required language proficiencies (Education & Training Evaluation Commission [ETEC], 2020). Luo and Al-Harbi (2016) examined the factorial structure of the EPT with unidimensional and DIMTEST methods. They found strong evidence supporting the unidimensionality of the EPT which justified the usage of the IRT-based models instead of the classical test theory method (CTT). 1.1.Literature review There have been many studies conducted to ensure test fairness across different subgroups and to define the potential source of DIF effects. Drabinová and Martinková (2016) conducted a study to determine the potential sources of DIF effects concerning the presence of the secondary latent trait besides the primary latent trait. They concluded that the existence of the DIF effect for some items did not mean that these items were biased because some of the DIF effects reflected the relationship between the secondary latent trait and the primary latent trait. Thus, they suggested that one should avoid labeling DIF items as biased items without the investigation of the content experts. Chubbuck et al. (2016) studied DIF effects in the context of differing contents across gender groups. They employed the Mantel–Haenszel and standardized DIF methods to detect DIF items for each content domain. They found that the males showed better performance than females in reading comprehension items. They also defined the lack of sufficient context in the sentence completion items as a potential source of DIF effects. Finally, they recommended utilizing more than one DIF methods to increase the accuracy of the results. Wedman (2018)
  • 37. 30 ©2020 The authors and IJLTER.ORG. All rights reserved. examined if the language ability of non-native test takers that took the test in a language other than their mother tongue affected their performance compared to the native speakers. It was found that the deficiency in the language skill of non-native test takers caused the DIF. Moreover, He defined the failure in wording the content clearly in an item as a potential source of DIF effects. (Siegel, 2007, Wedman, 2018). In one study, Stage (2005) investigated the SweSAT test items administered in spring concerning DIF across gender groups. The Mantel–Haenszel DIF method was employed to detect DIF items and It was found that 21 out of 122 items exhibited DIF across gender groups. Among these DIF items, 10 items related to the quantitative and verbal domains were in favor of female students. However, this study did not find any patterns among DIF items and did not suggest anything about the potential source of DIF effects. Federer and her colleagues (2016) employed the Mantel-Haenszel DIF method for detecting potential DIF items in the context of natural selection across gender groups. They specifically focused on open-ended questions. It was found that women outperformed men for the items that require applying the knowledge to the new conditions. Admitting the fact that the developed measurement instrument showed gender bias and, they did not suggest anything about the potential source of DIF effects due to the complex nature of DIF structure. Similarly, Lin and Wu (2003) used DIF and differential bundle functioning (DBF) to detect items that function differently across gender on the EPT administered to Chinese EFL learners. For this purpose, they used the SIBTEST methods to detect DIF items. The results of this study indicated that the testlets (item bundles) containing the listening comprehension items showed DIF in favor of females, while the testlets containing the grammar and vocabulary exhibited DIF in favor of males. Thus, these findings provide strong evidence about content specific DIF. Pae (2012) studied the trends in the magnitude of DIF on the English subtest administered to the Korean students across gender groups for the nine-year period. He used the Mantel-Haenszel and IRT-based likelihood ratio test methods to detect the DIF items. Moreover, the study examined the effects of reading strategies and perceived interest on the magnitude of DIF. The results of this study showed the strong evidence about the relationship between the type of items and DIF, and a substantial interaction between the test takers interest in the items and the magnitude of DIF across gender. It is substantially important to run DTF analyses along with DIF since items are small and unreliable compared to the test (Gierl, Bisanz, Bisanz, Boughton, & Khaliq, 2001) and the total amount of DIF provides an overall effect of DIF on test scores even when there is no item detected as DIF in a test (Hunter, 2014; Shealy & Stout, 1993). Additionally, DTF values can be negligibly small when these DIF items are in favor of different subgroups or in a different direction where DIF effects cancel each out (Borsboom, 2006; Zhu & Aryadoust, 2020). DTF is also important since decisions about examinees are not made at item- level, but test-level (Ellis & Raju, 2003; Roznowski & Reith, 1999; Pae & Park, 2006; Zumbo, 2003). More detailed information about the DIF and DTF methods is provided in the following sections.
  • 38. 31 ©2020 The authors and IJLTER.ORG. All rights reserved. 1.2.IRT based DIF methods The IRT based DIF methods are suggested in the case of a large sample size. The latent variable (ability estimate-θ) estimated by IRT models is used as a matching variable for subgroups rather than observed scores. There are many different IRT based methods to detect DIF items, some of which are test of b difference, Lord’s chi-square, Raju’s area method, likelihood ratio test (LRT method), and item drift method. The Lord’s chi-square DIF method (Lord, 1980) is an extension of the test of b difference and takes the other parameters into account. The major shortcoming of Raju’s area method is that the exact areas between ICCs are infinite when guessing parameters are not equal (Hunter, 2014). In this project, Lord’s chi-square DIF method was used to detect DIF items, because it takes more than one parameter into account when calculating DIF statistics and capable of detecting both uniform and non-uniform DIF in the presence of large sample size. The “difR” package installed in the R program was used to run DIF analyses. The formula for Lord’s chi-square DIF methods is as follows: (1) where VjR = (ajR, bjR, cjR) and VjF = (ajF, bjF, cjF) are the vectors of item parameters related to the reference group and focal group, respectively. Besides, the variance-covariance matrices of reference and focal groups are denoted by ∑ 𝑗𝑅 and ∑ 𝑗𝐹, respectively. The 𝑄1-statistic has chi-square distribution and its degrees of freedom is equal to the number of estimated parameters (Camilli, 2006; Lord, 1980). Previously research show that DIF results obtained from Lord’s chi-squared test and Raju’s unsigned area method are highly correlated (Millsap & Everson, 1993; Shepard, Camilli, &Williams, 1985). The most important disadvantage of the Lord’s chi-squared test is that it tends to reject the null hypothesis of no DIF even when the discrepancy between ICCs of sub- groups is small in the presence of a large sample size (Camilli & Shephard,1994; Wiberg, 2006). Thus, a more stringent criterion should be used in the presence of a large sample size. 1.3 Differential test functioning (DTF) Differential test functioning (DTF) values correspond to the total amount of DIF for the entire test. Therefore, it is equal to the sum of item DIF statistics in a test (Donovan, Drasgow & Probst, 2000; Ellis & Mead, 2000; Hunter, 2014; Nandakumar, 1993). There are different methods to calculate DTF such as Raju’s DFIT (Raju, van der Linden & Fleer, 1995), and Mantel-Haenszel/Liu-Agresti method (Penfield & Algina, 2006). Raju’s DFIT estimates DTF through calculating the squared difference between test characteristic curves, while the Mantel-Haenszel/Liu-Agresti method is based on variance estimates and tend to have higher DTF rates than Raju’s DFIT (Hunter, 2014). Penfield (2005, 2013) developed a program called DIFAS which enables us to calculate both DIF and the Mantel-Haenszel/Liu-Agresti statistics. In this study, the MH-LA method was used to evaluate DTF associated with the EPT tests. ( ) ( ) ( ) 1 j jR jF jR jF Q v v jR jF v v  − = − − −  
  • 39. 32 ©2020 The authors and IJLTER.ORG. All rights reserved. The formula for MH-LA DTF method proposed by Camilli and Penfield (1997) is as follows: 2 2 1 2 1 1 ( ) I I i i i s I    = = − − =   (2) where “I” represents the number of items; 𝜓𝑖 ̂ denotes MH log-odds ratio statistics; µ represents mean and 𝑠𝑖 2 represent the error variance of ψ. Some studies report weighted 𝜏2 statistics along with 𝜏2 statistics. The formula for weighted 𝜏2 is as follows: 2 2 1 2 1 1 1 ( ) I I i i i i I i i W W W    = = = − − =    (3) where 𝑤𝑖 is equal to 𝑠𝑖 −2 . 1.4 The purpose of the study This study aims at examining the presence of items that function differently for the entire test and across different sub-domains for gender in English Placement Tests (EPT) to ensure the fairness of the test. For this purpose, item response theory (IRT) based Differential Item Functioning (DIF) at the item level and Differential Test Functioning (DTF) methods at the test level were implemented, respectively, to examine whether items function differently across different subgroups. Considering the findings of previously conducted studies, this study aims to test five different hypotheses. The first hypothesis is that the factorial structure of the EPT remains unchanged across gender groups. The second hypothesis is that some of the EPT items are likely to exhibit DIF across gender. The third hypothesis assumes the existence of content specific DIF items at the item level. The fourth and fifth hypotheses are that the existence of DIF items affects the test scores for the entire test and each subdomain. 1.5 Research questions 1. Do the factorial structure of EPT for the entire test and each gender group support the unidimensionaliy? 2. Do items of the EPT function differently across gender (female vs. male)? 3. What is the distribution of DIF items across sub-domains (Reading Comprehension, Structure, and Compositional Analysis), when each domain is treated as a separate test? 4. Do test scores of the EPT exhibit differential test functioning (DTF) across gender (Female vs. Male)? 5. Do test scores of the EPT exhibit differential test functioning (DTF) across gender, when each domain is treated as a separate test?