IJLTER.ORG Vol 19 No 6 June 2020

International Journal
of
Learning, Teaching
And
Educational Research
p-ISSN:
1694-2493
e-ISSN:
1694-2116
IJLTER.ORG
Vol.19 No.6

PUBLISHER
Society for Research and Knowledge Management
Port Louis
Republic of Mauritius
www.ijlter.org
Chief Editor
Dr. Antonio Silva Sprock, Universidad Central de
Venezuela, Venezuela, Bolivarian Republic of
Editorial Board
Prof. Cecilia Junio Sabio
Prof. Judith Serah K. Achoka
Prof. Mojeed Kolawole Akinsola
Dr Jonathan Glazzard
Dr Marius Costel Esi
Dr Katarzyna Peoples
Dr Christopher David Thompson
Dr Arif Sikander
Dr Jelena Zascerinska
Dr Gabor Kiss
Dr Trish Julie Rooney
Dr Esteban Vázquez-Cano
Dr Barry Chametzky
Dr Giorgio Poletti
Dr Chi Man Tsui
Dr Alexander Franco
Dr Habil Beata Stachowiak
Dr Afsaneh Sharif
Dr Ronel Callaghan
Dr Haim Shaked
Dr Edith Uzoma Umeh
Dr Amel Thafer Alshehry
Dr Gail Dianna Caruth
Dr Menelaos Emmanouel Sarris
Dr Anabelie Villa Valdez
Dr Özcan Özyurt
Assistant Professor Dr Selma Kara
Associate Professor Dr Habila Elisha Zuya
International Journal of Learning, Teaching and
Educational Research
The International Journal of Learning, Teaching
and Educational Research is an open-access
journal which has been established for the dis-
semination of state-of-the-art knowledge in the
field of education, learning and teaching. IJLTER
welcomes research articles from academics, ed-
ucators, teachers, trainers and other practition-
ers on all aspects of education to publish high
quality peer-reviewed papers. Papers for publi-
cation in the International Journal of Learning,
Teaching and Educational Research are selected
through precise peer-review to ensure quality,
originality, appropriateness, significance and
readability. Authors are solicited to contribute
to this journal by submitting articles that illus-
trate research results, projects, original surveys
and case studies that describe significant ad-
vances in the fields of education, training, e-
learning, etc. Authors are invited to submit pa-
pers to this journal through the ONLINE submis-
sion system. Submissions must be original and
should not have been published previously or
be under consideration for publication while
being evaluated by IJLTER.

International Journal of Learning, Teaching and Educational Research
(IJLTER)
Vol. 19, No. 6 (June 2020)
Print version: 1694-2493
Online version: 1694-2116
IJLTER
International Journal of Learning, Teaching and Educational Research (IJLTER)
Vol. 19, No. 6
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part
of the material is concerned, specifically those of translation, reprinting, re-use of illustrations,
broadcasting, reproduction by photocopying machines or similar means, and storage in data banks.
Society for Research and Knowledge Management

The International Journal of Learning, Teaching and Educational
Research is a peer-reviewed open-access journal which has been
established for the dissemination of state-of-the-art knowledge in the
fields of learning, teaching and educational research.
Aims and Objectives
The main objective of this journal is to provide a platform for educators,
teachers, trainers, academicians, scientists and researchers from over the
world to present the results of their research activities in the following
fields: innovative methodologies in learning, teaching and assessment;
multimedia in digital learning; e-learning; m-learning; e-education;
knowledge management; infrastructure support for online learning;
virtual learning environments; open education; ICT and education;
digital classrooms; blended learning; social networks and education; e-
tutoring: learning management systems; educational portals, classroom
management issues, educational case studies, etc.
Indexing and Abstracting
Research is indexed in Scopus since 2018. The Journal is also indexed in
Google Scholar and CNKI. All articles published in IJLTER are assigned
a unique DOI number.

Foreword
We are very happy to publish this issue of the International Journal of
Learning, Teaching and Educational Research.
Research is a peer-reviewed open-access journal committed to
publishing high-quality articles in the field of education. Submissions
may include full-length articles, case studies and innovative solutions to
problems faced by students, educators and directors of educational
organisations. To learn more about this journal, please visit the website
http://www.ijlter.org.
We are grateful to the editor-in-chief, members of the Editorial Board
and the reviewers for accepting only high quality articles in this issue.
We seize this opportunity to thank them for their great collaboration.
The Editorial Board is composed of renowned people from across the
world. Each paper is reviewed by at least two blind reviewers.
We will endeavour to ensure the reputation and quality of this journal
with this issue.
Editors of the June 2020 Issue

VOLUME 19 NUMBER 6 June 2020
Table of Contents
The Emergency Remote Learning Experience of University Students in Indonesia amidst the COVID-19 Crisis....1
Maila D.H. Rahiem
Examining the Fairness of Language Test Across Gender with IRT-based Differential Item and Test Functioning
Methods ................................................................................................................................................................................. 27
Burhanettin Ozdemir and Abdulrahman Hadi Alshamrani
Main Approaches of Business English Teaching to Future Lawyers: A Case Study of Ukrainian Higher
Institutions............................................................................................................................................................................. 46
Oksana P. Bykonia, Iryna V. Borysenko, Tamila L. Gruba, Iurii L. Mosenkis and Dmytro O. Chystiak
Debate as a Tool for Learning and Facilitating Based on Higher Order Thinking Skills in The Process of
Argumentative Essay Writing............................................................................................................................................. 62
Marzni Mohamed Mokhtar, Marni Jamil, Rohizani Yaakub and Fadzilah Amzah
Exploring Teachers’ Pedagogical Practices in Teaching Mandarin as a Foreign Language in MARA Educational
Institutions, Malaysia ........................................................................................................................................................... 76
Nuraini Jafri, Umi Kalthom Abd Manaf and Fazilah Razali
The Effectiveness of Differentiated Instruction by Streaming: A preliminary Study of Current Practices in the
UAE ........................................................................................................................................................................................ 95
Ibrahim Suleiman Ibrahim Magableh and Amelia Abdullah
Technology-Driven Self-Directed Learning of Graduate Pharmaceutists: Adding Value through Entrepreneurship
............................................................................................................................................................................................... 111
Iryna Nizhenkovska, Tatyana Reva, Oksana Chkhalo and Oksana Holovchenko
Teachers’ Covid-19 Awareness, Distance Learning Education Experiences and Perceptions towards Institutional
Readiness and Challenges ................................................................................................................................................. 127
Aris Alea Lapada, Frosyl Fabrea Miguel, Dave Arthur Roldan Robledo and Zeba F Alam
Enhancing Malaysian Primary Pupils’ Vocabulary Skills using Pocable Game and Pear Deck .............................. 145
Chai Kar Ni, Bonaventure Jong, Mary Anne Dison, Sylvia Anak Thomas, Melor Md Yunus and Ashairi Suliman
Content Validity of West African Examination Council Financial Accounting Questions....................................... 161
Basil C. E. Oguguo, John J. Agah, Catherine U. Ene, Vivian N. ACholonu, Roseline N. Azubuike, Mary A. Okeke and
Lourita P. Agbo
Assessing the Relationship and Prediction of Manifold Facets of Analytic Relations to Academic Reading
Comprehension................................................................................................................................................................... 179
Md Kamrul Hasan, Md. Didar Hossain and Abdul Karim
Towards a Principled Use of L1 – Observing an EFL Teacher’s L1 Use in Rural Sabah, Malaysia.......................... 206
J. W. Ong and A. J. Ahmad Tajuddin

A Study of Saudi Advanced Academic Writing Students’ Perceptions of Research Essays, and Gaps in Their
Knowledge........................................................................................................................................................................... 223
Nida Qayoom and Mohammad Saleem
Mass Media Internships in Vocational Training of Students Majoring in Journalism............................................... 238
Nataliia A. Tsymbal, Nataliia M. Savchuk, Valentina I. Avramenko, Svitlana A. Sichkar and Iryna A. Denysiuk
Assessing the Efficacy of Extensive Reading during Study Abroad: A Time and Place for ER? ............................. 251
Byron O'Neill and Christopher Edelman
The Extent of Bullying Against Students with Learning Disabilities According to the Age Variable..................... 267
Mohamad Ahmad Saleem Khasawneh
Development of Instruments to Measure Mathematical Anxiety of Elementary School Students.......................... 282
Hafiziani Eka Putri, Mukhammad Ady Wahyudy, Aan Yuliyanto and Fitri Nuraeni
Situating "children-supporting-children" Platform in the Context of the Inclusive Agenda: A Phenomenological
Exploration .......................................................................................................................................................................... 303
Princess Zarla J. Raguindin and Li Yan Ping
The Impact of Educational Technologies on University Teachers’ Self-efficacy ........................................................ 323
Nataliia Saienko, Yuliana Lavrysh and Valentyna Lukianenko
UAE Elementary Teachers’ Use of ADHD Referral and Management Strategies...................................................... 337
Hala Elhoweris, Ahmed Mohamed, Osha Almuhairy, Rachel Takriti, Najwa Alhosani and Abdelaziz Sartawi
Exploring Accounting Teachers’ Views on the Quality of Accounting Prescribed Textbooks in South Africa ..... 353
Jabulisile C. Ngwenya and S’khumbuzo H. Mbili
Rurality and Exclusion in Ordinary Level Mathematics in Zimbabwe: A Document Analysis............................... 370
Simon Vurayai
Emergency Online Teaching in Economic and Management Sciences Necessitated by the COVID-19 Pandemic:
The Need for Healthy Relations in a Rural Schooling Context..................................................................................... 387
Habasisa Molise and Bekithemba Dube

1
©2020 The author and IJLTER.ORG. All rights reserved.
Vol. 19, No. 6, pp. 1-26, June 2020
https://doi.org/10.26803/ijlter.19.6.1
The Emergency Remote Learning Experience of
University Students in Indonesia amidst the
COVID-19 Crisis
Maila D. H. Rahiem
UIN Syarif Hidayatullah Jakarta
Jakarta, Indonesia
http://orcid.org/0000-0002-5618-2486
Abstract. This study aimed to explore and interpret the lived experience
of Indonesian university students in emergency remote learning (ERL)
during the COVID-19. Methods of the investigation was a qualitative
phenomenological approach involving 80 students from the Social
Science Education Program at a public university in Jakarta.
Understanding their experience was achieved through a rigorous analysis
of the participants’ diaries and reflective essays and an online focus
group. Results revealed that the students’ experiences fell into two
overarching themes, each with related sub-themes. The two identified
themes and subthemes were: (a) blended learning, with the subthemes of
e-learning, m-learning and conventional learning, and (b) paradoxical
learning, with the sub-themes of flexible learning and challenging
learning. By studying how university students learned during COVID-19,
we could help ensure the efficacy of ongoing ERL and better incorporate
similar programs in the future if this ever happens again.
Keywords: emergency remote learning; online learning; higher
education; COVID-19
1. Introduction
The COVID-19 global pandemic has created entirely unprecedented situations
that have greatly affected people’s lives. With doubts remaining at present over
how and when it will end, the question remains whether things will ever go back
to the way they were before the pandemic began with many skeptics remaining
doubtful (Lee et al., 2010).The World Health Organization (WHO) has warned
that COVID-19 is likely to haunt the world for a long time and that the planet will
only return to normal once a viral strain vaccine has been discovered. There are
currently ten candidate vaccines in clinical trials worldwide and 126 candidate
vaccines in pre-clinical trials (World Health Organization, 2020b). Anthony Fauci,
MD, director of the US National Institute of Allergy and Infectious Disease, said
that if all goes well, a vaccine might be available in November or December of this

2
year (McCarthy, 2020; Reynolds, 2020). However, even after a vaccine has been
approved, there remains an immense challenge to generate enough of it for the
world's population (Felter, 2020).
Presently (Early June 2020), in Indonesia and some other countries, the number of
cases is still increasing at an alarming rate. The number of patients nationwide
who have tested positive is up to 36,406, with patients who have recovered
recorded as 13,213 people, and patients who have died totaling 2,048 people (Task
Force for the Acceleration of COVID-19, 2020). COVID-19 has spread to 34
provinces, meaning that it has spread to all of the provinces across the Indonesian
archipelago, and 391 out of 514 municipal districts have been infected (National
Agency for Disaster Management, 2020). So far, the percentage of Indonesia's
death rate, which is above 6 percent, is rated in the high category. The total
number of cases worldwide is also still increasing, with 7,127,753 confirmed cases
of COVID-19 spread across 216 countries, including 407,159 deaths, reported to
WHO (World Health Organization, 2020c).
As a result of the threat posed by COVID-19, educational institutions have hurried
to shift courses to virtual classrooms. Many institutions have chosen to cancel all
face-to-face classes, including laboratories and other learning activities, and have
required students to study remotely from home to help avoid the spread of the
virus that causes COVID-19. The UN Educational, Scientific, and Cultural
Organization(UNESCO, 2020) estimated that 129 countries had implemented
national school closures (data from June 10, 2020). These nationwide closures have
impacted almost 63.3 percent of the world's student population or 1,109,020
learners. Several other countries have implemented localized closures impacting
millions of additional learners (approximately 488,198,235 learners). The total
number of affected learners due to nationwide closures and localized closures
totals 1,725,082,528. The trend of nationwide closures has decreased from
previous figures recorded on April 1 – 4, 2020, where 1,598,099,000 learners, or
91.3% of total enrolled learners, were recorded as being affected due to the
nationwide closures policy in 195 countries. However, the total affected number
(nationwide and localized) is relatively the same, starting from March 26, 2020.
This data indicates that although the national closures policy has begun to be
revoked in some areas and regions, schools and local governments have
continued to implement remote learning.
In Indonesia, nationwide school closures began on March 23, 2020, while localized
closures in some provinces, e.g., DKI Jakarta, Central Java, Banten, West Java, and
Aceh, started early on March 16, 2020 (Kumparan, 2020). The national school
closures have impacted 60.2 million learners and 2.3 million educators who study
or teach at 425,451 educational institutions from early childhood to higher
education.

3
Table 1: Number of affected academic institutions, learners and
educators in Indonesia
(Ministry of Education and Culture, 2019, 2020c, 2020a, 2020d, 2020b)
Level
Academic
Institutions
Learners Educators
Early Childhood Education 202,991 6,543,758 666,678
Elementary School 149,435 25,203.371 556,969
Junior High School 40,559 10,112,022 302,116
Senior High School 13,495 4,976,127 212,715
Vocational School (Equivalent to Senior High
School)
14,301 5,242,851 217,986
College/University 4,670 8,043,480 294,820
TOTAL 425,451 60,121,609 2,251,284
School closures are based on the previous studies of influenza outbreaks that
resulted from social interactions between students and teachers that disrupted the
learning process (Jackson et al., 2016). Cauchemez et al., (2008, 2009) quantified
the role of schools in influenza epidemics and predicted the effect of school
closures during a pandemic in France. They found that sustained school closures
during a pandemic could reduce the total number of cases by 13–17 percent (18–
23 percent in children), while during the height of the attack the rate could be as
much as 39–45 percent (47–52 percent in children). The effect of school closures
would be minimized if it proved difficult to sustain low contact rates among
children for a prolonged time. Earn (2012) looked at the associations between the
occurrence of H1N1 (pH1N1) pandemic influenza in Alberta, Canada in 2009 and
school closures or weather changes, and calculated the impact of school closures
and weather changes on the transmission of pH1N1. Mathematical models
indicated that school closures decreased transmission among school-aged
children by more than 50 percent, which was a critical factor in interrupting
transmission. The models also showed that seasonal changes in weather
conditions had a significant effect on the epidemic's temporal structure.
School closure is one of the critical components of numerous non-pharmaceutical
prevention strategies in many countries to help reduce the number of cases and
slow the spread of the disease, mainly if vaccines or antivirals are not available or
are ineffective due to resistance (Glass et al., 2006; Institute of Medicine, 2006;
Moscona, 2005; Sadique et al., 2008). School closure policies in various countries
occur in quick succession as a measure to contain the COVID-19 virus. Lee (2020)
argues that school closures alone may not be enough to stop the outbreak, but that
if it continues for at least eight weeks, could delay the outbreak peak by up to a
week, providing adequate time for a second more successful intervention, such as
vaccination.
In this study, the researcher used the term "Emergency Remote Learning (ERL)" to
illustrate the education that took place during school closure, not online or virtual

4
learning, since well-planned online learning experiences are substantially
different from those that are delivered online in response to a crisis or catastrophe.
Hodges et al. (2020) call it Emergency Remote Teaching (ERT). ERT is a temporary
change from instruction delivery to alternative delivery due to crisis
circumstances. It requires the use of entirely remote teaching approaches for
instruction or education that would otherwise be provided face-to-face or as
blended or hybrid courses, and which will revert to that model once the crisis or
emergency has finished. The primary objective in this context is not to re-create a
stable educational environment, but rather to provide immediate access to
education and training in a manner that is easy to develop and easily accessible
during an emergency or crisis. Millman (2020) described the situation as
emergency remote teaching and learning — or "pandemic pedagogy."
This study focused on understanding how remote learning during COVID-19
pandemic has been conducted in a tertiary education institution by analyzing
university students' perspectives and experiences. The limited research was
conducted recently in Indonesia during the midst of the COVID-19 pandemic.
Most of the existing research conducted is in Indonesian, published in Indonesian
journals, focuses on either primary or secondary education, or education in
general (the level of education is not explained). Examples of this include,
"COVID-19 impact on the implementation of online learning in primary schools"
(Aji et al., 2020); "Education during the COVID-19 pandemic" (Khasanah et al.,
2020); "Studying Mathematics in the COVID-19 era" (Abidin, 2020); and "Online
learning in the middle of the COVID-19 pandemic" (Firman & Rahayu, 2020).
There are also several articles on learning in higher education during the COVID-
19 pandemic. These articles are written in Indonesian, with limited data and have
not been published through peer-reviewed processes such as “Students’ learning
typology in online learning”(Ramdhan et al., 2020); “Google Classroom-online
learning of Biology education students during the COVID-19 outbreak”(Suhada
et al., 2020); “Ordinal Regression analysis to see the effect of online learning media
on students’ enthusiasm in the COVID pandemic era”(Meiza et al., 2020).
Some articles published in peer-reviewed Indonesian journals focus on online
learning of COVID-19 at the college level. However, they were published in April,
where the learning from home experience had only been in operation for a few
weeks. Experiences from a short implementation time are unlikely to offer a full
picture of home-based teaching and learning activities, while there was also
minimal use of references. The published articles are “The impact of COVID-19
online learning activities in a Christian university in Indonesia”(Windhiyana,
2020); “Implementation of the performance model of DeLone and McLean
Knowledge Systems For Zoom application-based learning systems during the
COVID-19 pandemic”(Hidayatullah et al., 2020); “Perception of students
majoring in Early Childhood Islamic Education towards online Lectures in the
COVID-19 pandemic period”(Anhusadar, 2020); and “Variations of models and
learning platforms for prospective teachers during the COVID-19 pandemic
period”(Gunawan et al., 2020).

5
Figure 1. Theoretical Model
The theoretical model outlined in Figure 1 shows what the related studies have
been performed, what this study was intended to do, and the research questions
raised by the context analysis and previous research. Recent studies have
described school closure as one of the key components of a range of non-
pharmaceutical prevention approaches aimed at minimizing the number of cases
and preventing the spread of disease. Some of the previous emergency remote
learning studies concluded that ERL is a temporary learning mechanism, a
transition from in-person instruction to substitute delivery due to an emergency.
Nonetheless, little is known on how to apply remote learning in higher education
during the outbreak of COVID-19 and how students perceived and experienced
ERL implementation.
Research on the implementation of home learning during COVID-19 in Indonesia
has been listed above. None of those papers use the term remote learning or ERL.
The use of the term affects the analysis and understanding that is predicated in
these studies. It is crucial to make a distinction, as the term refers to a unique and
specific event (Manfuso, 2020). This research filled the gap in knowledge (theory)
of ERL at the university level in Indonesia with insights and observations
collected from students who had been studying from home for an extended
period (more than ten weeks), had undergone an online mid-term exam, and was
at the end of the semester. Other than that, this research disseminated its
findings in English articles in international journals, so that more audiences could
read and learn from the research findings.

6
This research also filled the practical gap on how to successfully implement ERL
in higher education to meet learners' needs. This study's main question is: how
university students experienced ERL due to the COVID-19 crisis? Through
knowing how university students learned in the emergency COVID-19, we could
improve on in-going practice. The students' feedback is also critical to improve
the learning process in the future if such a situation was ever to occur again. It is
unknown how long ERL will continue or what kind of other problems we may
face in the future, so educational institutions and students alike must be better
prepared for any such eventuality.
2. Materials and Methods
The current study is presented as part of a larger project on ERL in tertiary
education in Indonesia during the COVID-19 outbreak. The overall aim of the
study was to explore the implementation of ERL in tertiary education in
Indonesia. The study employed a qualitative method of the phenomenological
approach. This phenomenological analysis aims to collect and examine the lived
experience of university students who have learned from home, due to the
COVID-19 crisis, for more than two months. These experiences provided a new
level of insight into ERL's implementation due to the outbreak of COVID-19.
A phenomenology is a philosophical approach to the analysis of phenomena that
are part of human awareness (Arbnor & Bjerke, 2011). In phenomenology, the
researcher attempts to provide a direct description of someone's experience as it
is, without considering his or her psychological origin and the causal explanations
that the scientist may be able to provide (Merleau-Ponty & Landes, 2013).
Phenomenology is equipped to understand the subjective interpretations of the
participants of the fundamental object of the study by throwing light on the
significance applied to their lived experience and exploring the specific meanings
underlying the empiric variants of a given phenomenon rather than merely
applying the technique (Casmir, 1983; MacDermott, 2002).
Participants in this research are 80 students who have studied social science
education at a public university in Jakarta. The researcher employed a purposive
sampling method to select the participants. The researcher understood the
purpose of the study in such a way that she relied on her judgment to classify
qualified participants in a particular profile. The various explanations why the
researcher chose the participants are as follows: the first reason was the ease of
access for the researcher to collect data due to the COVID-19 situation, which
restricted the researcher's ability to gather data. The second reason, they were
semester four students, which means that they were in the middle of their
course (undergraduate education usually takes eight semesters). These students
already have enough learning experience and were still taking compulsory and
additional courses (semester seven students practice teaching, and in semester
eight, write a thesis). Third, students at this university come from Jakarta and
other provinces and different backgrounds. This diversity will help enrich the
data further. Lastly, the researcher intentionally chose students from educational
programs because they were prepared to become teachers, and their opinions are
important because their answers address how an educator should teach at a time
like this.

7
These students are in the fourth semester and came from two classes, 4A and 4B.
These two groups have different concentrations, 4A is Sociology, and 4B is
Geography. The name of the university and the names of the students have been
hidden; to protect the identity of the participants and to provide them with the
opportunity to speak freely. Participants were aware of the scope and intent of the
study, and they were allowed to withdraw from the research anytime they chose
if they felt dissatisfied with it.
Data collection was carried out using creative methods because the researcher was
unable to conduct direct interviews due to the large-scale social restrictions that
were in place during the research period, data collection was completed by asking
participants to write a diary of their daily learning activities for two weeks (4 - 18
May 2020. See Appendix 1. Students’ Diary). They were then asked to compose a
reflective essay about ERL (collected on May 20, 2020. See Appendix 2. Reflective
Essay).
The researcher paid close attention to data triangulation to develop a
comprehensive understanding of the phenomena. Therefore, the researcher used
multiple methods or data sources, as suggested by Patton (Patton, 1999). The data
source utilized, other than diaries and essays, was focus group discussions using
the Google Meet application. These discussions were conducted twice; each
meeting was for 120 minutes and 40 participants (on May 26 and 27, 2020). The
focus group discussions were also aimed as a member check to increase the
precision, reliability, validity, and transferability of the findings. The researcher
presented the study's overall findings and then opened a discussion session.
Participants were free to question if any incorrect information had been recorded
and were allowed to add or remove details. The participants either confirmed that
the summaries represented their beliefs, feelings, and experiences or did not
reflect their experiences. The researcher concluded that the study seems credible
after the participants confirmed the accuracy and completeness of the study.
The complete elimination of non-response bias from the study is impossible.
Mixing data collection methods was one of the researchers' efforts to avoid non-
response impact from the sample. The researcher also sought to eliminate bias by
designing strategies that would promote the involvement of selected individuals.
The researcher used the NVivo program for data management and analysis.
NVivo is useful in organizing data and helped the researcher make sense of it
throughout the research since the data for this qualitative research was extensive
(80 participants). The researcher also made a memo in the NVivo system that
allowed for the possibility to document ideas and analyze thoughts, perspectives,
and observations of students on online learning. This analytical memo was
completed on an ongoing basis, every time data (journal or essay) was submitted.
Analytical memos provide a means for the researcher to record their thoughts
during the research process and to code memos as additional data for the study
(Saldaña, 2016).
In analyzing the data, the researcher adopted a two-stage coding model
conducted by Miles, Huberman and Saldaña (Miles et al., 2014) and Saldaña
(Saldana, 2009; Saldaña, 2016). The first cycle coding was initially assigned to the

8
data, followed by the second cycle codes in which the initial codes were grouped
into meaningful categories, themes, or constructs. The two stages of coding are
not a single linear event; the qualitative analytical process is cyclical. In practice,
in the first cycle, the researcher coded each essay and journal individually. After
that, the researcher compared the data and detailed the code into a sub-code if
necessary. In the second cycle, the researcher reorganized and re-analyzed the
data coded in the first coding cycle. The primary aim of second-cycle coding was
to establish a sense of categorical, thematic, conceptual, and theoretical structure
from the first-cycle code series. The researcher changed codes, added new codes,
and dropped a few codes to conclude themes of research findings. Finally, the
researcher made the summative synthesis assumptions and declarative claims.
The assertions are based on the researcher's insights and observations, supported
by evidence from the data corpus.
3. Results
How did university students experience ERL during the COVID-19 crisis? Results
revealed that students’ experiences fell into two overarching themes, each with
related sub-themes. The two identified themes and subthemes are: (a) blended
learning, with the subthemes of e-learning, m-learning, and conventional
learning, and (b) paradoxical learning, with the sub-themes of flexible learning
and challenging learning.
Figure 2. Research findings
The researcher classified 25 codes, in the first cycle coding, from the data gathered
from 80 diaries, 80 reflective essays, and two focus group discussions. The
researcher then grouped related codes into five groups, which included: e-
learning, m-learning, traditional learning, flexible learning, and challenging
learning. The researcher summarized two key themes of this study's results in this
second coding process: blended learning and paradoxical learning. The researcher
concluded that during ERL, students experienced blended learning approaches,
and their experiences were paradoxical. They enjoyed learning from home, but
also saw it as challenging too. The following is an illustration of the process
of codification, thematic discovery, and the construction of assertions.
Emergency Remote Learning during
COVID-19 Outbreak
Blended
Learning
e-learning m-learning
Conventional
Learning
Paradoxical
Learning
Flexible
Learning
Challenging
Learning

9
Figure 3. Data analysis process

10
3.1 Blended Learning
Of the twenty-five codes found in the first coding cycle (as shown in Figure 2
above), thirteen are related to the media or methods used during the ERL. The
researcher then grouped codes that show similar meaning in categories: e-
learning, m-learning, and conventional learning. The researcher concluded from
the categories that learning remotely during the COVID-19 situation involved a
mix of technological learning media and conventional methods, or using blended
materials.
Table 2. Learning media/methods & frequency of being mentioned by participants
Media/Methods Frequency Categories
Reading E-book & E-Journal 56 e-learning & m-learning
YouTube & Video 52 e-learning & m-learning
Online Meeting (Zoom or Google Meet) 50 e-learning & m-learning
Browsing Internet 42 e-learning & m-learning
Educational Website 35 e-learning & m-learning
Podcast 33 e-learning & m-learning
Google Classroom 35 e-learning
WhatsApp 42 m-learning
Social Media 36 m-learning
Audio Call 32 m-learning
Assignments from Lecturers (exercises,
quizzes, and exams)
58 Conventional Learning
Making Notes 40 Conventional Learning
Module/Unit 33 Conventional Learning
From the coding of the data in the NVivo program, researchers have discovered
how many frequencies a single code of all the existing data appears. Table 2 shows
which media or methods are most mentioned repeatedly by students.
Reading e-books and e-journals was the most common learning methods
experienced by the students. Students also learned from YouTube and TV series
or movie documentaries. Usually, lecturers provided students with YouTube
links, asked them to study the video, and make a summary report. E-Books, E-
Journals, and YouTube videos were used to help explain learning about the
subject or giving instructions. Other platforms used to deliver the lecturers'
instructions or explanations were through WhatsApp (the lecturer recorded voice
notes and shared them in the WhatsApp group) and through the module or unit
of work being studied.

11
Three platforms were commonly used for classroom discussions: online meetings
in Zoom or Google Meet, WhatsApp, and Social Media. They interchangeably
used the term Social Media for WhatsApp also, while a few referred to Facebook
when mentioning about Social Media. Not all the lecturers arranged synchronous
discussions using online platforms. Many students complained about it; they felt
that learning was ineffective without discussion. WhatsApp was the most
preferred media for discussion. In WhatsApp, the discussion could be
synchronous or asynchronous. The discussion could be made using either voice
notes or texts.
In order to enrich learning, students have tried to browse for more reading
material online by searching the internet and by visiting educational websites. If
they did not understand the subject, they would invariably contact their
classmates, asking them to explain the subject. The respondents believe that
YouTube videos are a good source of learning. Some listened to podcasts and
explained that this was as a result of needing to be more creative in expanding
their understanding. In measuring student learning progress, lecturers have
delivered assignments and quizzes to students through WhatsApp and Google
Classroom. Also, conventional methods were used for evaluation. The most
common conventional tasks assigned to students are the memorization of texts
and the writing of a summary.
The data reveals that emergency remote learning has not always been entirely
conducted online using sophisticated information technology; students have
often learned using conventional media and methods. The students take notes,
use paper, and pens to help them understand learning materials more efficiently.
They took notes on the video, sound recordings, e-books, PowerPoint
presentations, and journals provided by the lecturers. They felt that, in general,
they had not learned anything if they did not write it in a book. They seemed to
be able to understand more quickly when they wrote down what they had learned
in a notebook. Participant 4B24 explained about making a summary of learning
using paper and colorful pens and markers:
"If the lecturer has given material in the form of power points, modules,
and so on. I put together a summary of the material. And if there was a
discussion on WhatsApp or Zoom, I try to write notes about things that I
thought were important. Just like the previous way of learning (before
the COVID-19 crisis), I used colored pens or markers to write notes, so I
wouldn't get bored while reading notes." (4B24)
Whereas 4B35 said that she had made a mind map of the content she had learned
in order to be able to pass the mid-term and final exam. She added that she placed
her notes on the wall, and every morning she memorized and learned them:
“Although the mid-term test and end semester exams will be done online,
I will still make a mind map to test my understanding of the material... for
the final exam, I will make keywords from each material that I have
studied, I will make notes in my paper and paste them on the wall, and
every morning I will memorize and learn.” (4B35)
Students also conventionally learnt using modules or units, not a printed version,
but an e-module in PDF format. Two lecturers provided these e-modules

12
containing discussions, quizzes, and a summary. The aim was to help students
learn independently. They can test their comprehension by answering the quiz in
the module and checking its accuracy by matching the answer key on the last page
of the module. The lecturers shared the module every 1-2 weeks. Some students
printed the modules or made a hand-written summary of the modules. The
students also explained that they felt more like they had studied when they have
paper, pen, and writing in a book.
"In the pandemic from the COVID-19 crisis, my learning is using
modules or PowerPoint presentations that I have obtained from lecturers
or notes from my colleagues." (4C27)
"I try to understand the tasks or modules provided, read all through them,
yet I still don't get it." (4C30)
Another conventional learning method is lecturers giving students assignments.
Students complained they actually had to finish too many assignments.
“Lecturers have been giving assignments online, a lot of them. Some
lecturers only give assignments continuously.” (4B21)
In ERL, blended learning was used for instruction and explanation, discussion,
evaluation, and enrichment. The researcher mapped out the thirteen
media/learning methods mentioned by the students in diaries and essays and the
purpose of their use. Here is the following map:
Figure 4. Blended media in ERL and their purpose of use

13
WhatsApp is a prevalent medium used for all these learning purposes.
WhatsApp is used to explain, discuss, evaluate and enhance learning. In addition,
WhatsApp is also used for almost all courses during the ERL period. Only one of
the eight courses in Class 4A and Class 4C did not use the WhatsApp. This
assertion is derived from tabulations made on the types of learning platforms
used for each subject.
Table 3. Courses and learning platforms
No CLASS 4A CLASS 4B
COURSES LEARNING
PLATFORMS
COURSES LEARNING
PLATFORMS
1. The New
Indonesian
History
• YouTube
• Assignment via
WhatsApp
• Assignment via
Email
The New
Indonesian
History
• YouTube
• Assignment via
WhatsApp
• Assignment via
Email
2. Study &
Learning
• Google Classroom
• YouTube
• Google Meet
• WhatsApp
Study &
Learning
• YouTube
• Google Meet
• WhatsApp
3. Educational
Psychology
• Module
• Email
• Google Meet
Educational
Psychology
• Module
• Email
• Google Meet
4. Regional
Geography of
the World
• Reading Material
• Academic
Instruction System
• WhatsApp
Regional
Geography of
the World
• Reading Material
• Academic
Instruction System
• WhatsApp
5. Learning
Strategies
• Assignment
• WhatsApp
Learning
Strategies
Google
Classroom
WhatsApp
• Assignment
• WhatsApp
6. Practicum
Reading the
Quran
• Assignment
• WhatsApp Email
Practicum
Reading the
Quran
• Assignment
• WhatsApp Email
7. Entrepreneurs
hip
• Google Scholar
• WhatsApp Group
• Voice note
Entrepreneur
ship
• Google Scholar
• WhatsApp Group
• Voice note
8. Sociology of
Religion
• Email
• Voice Note
• WhatsApp
• PowerPoint
Presentation
Cartography • Lecturing in Zoom
Meeting
• Google Meet
• Assignment
• WhatsApp
During the semester, students from class 4 A and 4 B studied eight courses, seven
of which were the same for both classes and taught by the same lecturers. The
seven courses taught in Class 4A and Class 4B were: New Indonesian History,
Study and Learning; Educational Psychology; Regional Geography of the World;
Learning Strategies; Practicum Reading the Quran; and Entrepreneurship. The
distinctions were: Class 4A took Sociology of Religion, and Class 4A took
Cartography.

14
3.2 Paradoxical Learning
The second main theme of the findings is “paradoxical learning.” The researcher
used this concept to describe how students had paradoxical perspectives and
insights into learning. On the one hand, they saw ERL as flexible, but on the other
hand, they also saw it as challenging.
Table 4. Students’ insights & observations on Paradoxical Learning
Insights/Observations Frequency Reasons
Time Management (Flexibility) 83 Flexible Learning
Family Time 52 Flexible Learning
Exercising 51 Flexible Learning
Comfortable & Quiet Place 42 Flexible Learning
Break & Rest 40 Flexible Learning
Refreshing 36 Flexible Learning
Technology Barriers 85 Challenging Learning
Cost of Internet 83 Challenging Learning
Overload Assignments 73 Challenging Learning
Trouble Finding Materials 51 Challenging Learning
Tiring 49 Challenging Learning
Noisy & Disturbing 42 Challenging Learning
Table 4, we can see how many times students reported certain insights or
observations about ERL paradoxically of all the existing data. The researcher used
color codes to show how they saw learning as two polar opposite scales. The two-
yellow color coded cells are time management and overload tasks. The three-
orange color coded cells are family time, comfortable and quiet, and noisy and
disturbing. The three-green coded cells are exercising, break and rest, and are
tiring. Four-blue coded cells are refreshing, technology barriers, internet costs,
and trouble finding materials. The segregated color-coded data indicates how
students had paradoxical perspectives and insights into learning.
On the one hand, students saw ERL as flexible; on the other hand, they also saw
it as challenging.
Figure 5. Flexibility versus overloaded
Flexibility
and time
management
Overloaded
with
assignments
Flexible
Learning
Challenging
Learning

15
Students said that learning remotely at home gave them the flexibility to manage
their time.
“The flexible learning schedule matches our lifestyle; young people.
Lecturers and students are increasingly turning to online learning as a
viable alternative to study anywhere, anytime.” (4B26)
Contradictory to the amount of flexibility, they also argued that the lecturers
overloaded them with assignments, and they, therefore, found it difficult to
manage their time.
"It's not necessarily online learning or lectures that we were given, but
now there are remote assignments (online). And the tasks that were given
never stopped every week, we were given assignments by each subject, and
we were chased by deadlines. In my view, it can make students
frustrated and depressed. As we now know that we need to retain our
body's immunity to avoid the virus, and the stress or pressure caused by
the assignments may, in my opinion, disrupt the students' immunity
from the body." (4C33)
Figure 6. Family time, comfortable & Quiet versus noisy & disturbing
The students said that during remote learning, they have a lot of family time and
could study in a comfortable and quiet place.
“Studying online makes it easy for us to find places that make it easier to
think about, such as open spaces, indoor rooms, and family rooms at
home.” (4B17)
In contrast to this, they also felt disturbed by their siblings and the noise at home.
“The home environment cannot be controlled when there is a live lecture
that uses applications such as Google Meet and synchronous conversation.
The situation of the house is crowded. It's very annoying, in my opinion,
and it makes me unable to focus.” (4B18)
Family Time
Comfortable
& Quiet Place
Noisy and
Disturbing
Flexible
Learning
Challenging
Learning

16
Figure 7. Exercising, break & rest versus tiring
Some students did some regular exercise at home. They spared extra time for self-
care.
"If there are no assignments, I will take the time to exercise and bask in
the healthy morning sun." (4B28)
"After I wake up, I do regular exercise or aerobic exercise. I'm doing this
regularly. Learning from home allows me to exercise more." (4C38)
Because the students were staying at home, they were able to take a break during
learning. They did not want to push themselves too hard to study. Their main
focus was to keep healthy and not be too tired.
“In addition, I also tried not to be too tired and forced to study because of
the consequences of the Covid-19 pandemic, I had to be careful not to fall
ill. I also don't push myself too much when I'm tired of studying.” (4B15)
Other respondents, however, felt that compared to face-to-face learning, remote
was much more tiring.
"I want to complain that this distance learning is very ineffective and
makes me even more exhausted." (4C20)
“A lot of tasks! The lecturer should consider the fact that a student is
staying at home to avoid illness or infection from happening. The lecturer
gave us lots of assignments excessively. It kills us. Not because of the
coronavirus that causes us to die. May Allah Amen protect us.” (4C18)
Figure 8. Refreshing versus problems
Exercising
Break &
Rest
Tiring
Flexible
Learning
Challenging
Learning
Refreshing Technological
barriers
Cost of the internet
Trouble finding
materials
Flexible
Learning
Challenging
Learning

17
The students said they tried to manage their time and refresh themselves
whenever they were tired. They said this is what made learning from home
more flexible.
“Learning activities must be interspersed with activities that are fun,
refreshing. In order not to be boring, I always prepare fun activities (in my
opinion). This activity can also be a reward or reward when I have
completed an assignment or studied material.” (4C21)
They also complained, however, about technological disruption while studying,
expensive internet costs and difficulties in obtaining learning materials during the
ERL.
"The difficulty of accessing the Internet, I was at home during this
distance learning process. This makes me experience delays in getting
information because access to the Internet in the village is a bit difficult,
the internet connection is bad in my area." (4B18)
"It's very pricey to spend money on the internet data plan. I'm a student
who gets minimal pocket money, especially on staying at home like this, I
don't get any income from being a Scoutmaster. That makes it hard for me
to buy a data plan. I'm also envious of campuses that provide a free data
plan of up to 150 thousand rupiahs per month." (4B12)
"This policy has a side where I don't like it, for example, because I can't
go anywhere, so I have trouble finding references that I need to complete
my assignments. Learning resources are also limited and can only refer to
the internet and journals that aren't always right" (4B19).
4. Discussion
Indonesia reported the first two confirmed cases of COVID-19 in a Jakarta suburb
on March 2, 2020 (World Health Organization, 2020a). The situation became
increasingly dangerous in mid-March. President Jokowi, in a press conference at
the Bogor Presidential Palace on Sunday, March 15, 2020, called on local
governments to issue policies to prevent the spread of the coronavirus. One of the
policies recommended was to temporarily halt teaching and learning in schools
and universities and encourage students to study at home. As a follow-up to the
president's direction, on March 20, 2020, the Minister of Education and Culture
issued a Circular Number 36962/MPK.A/HK/2020 concerning online learning
and working from home in the context of the prevention of coronary disease
spread. Following this rule, the campus, which was the focus of this research,
began to close on March 23, 2020.
Learning was suddenly moved to remote learning. Remote learning has not been
anticipated, and most of the course syllabus was not designed to be an online or
for a distance learning experience. Without preparation or training, lecturers
designed and implemented remote learning programs. Similarly, students did not
have the opportunity to be prepared for this transition in learning. Despite all of
these limitations, everybody has been trying their best to make the situation a
success, quick and reliable temporary access to education in unprecedented time.

18
Lecturers began using the media or the methods they were familiar with and
comfortable using. It may not be perfect, but students are resilient, the educator is
resourceful, and everyone has to make the most of what they have (Juliani, 2020).
This study reveals that learning remotely during the COVID-19 situation involved
a mix of both conventional and non-conventional methods. Technological
learning media was used combined with pen and paper methods. The use of
blended materials is what the researcher concluded as the first theme.
Blended learning is the application of more than one method, strategy, technique,
or media in education (Sadeghi et al., 2014; Thiele, 2003). It is a mixture of
conventional lectures or tutorials and web-based material (Concannon et al.,
2005); when delivering information, traditional face-to-face learning is combined
with technology (Farrel, 2006). In line with the government's policy of social
distancing, learning has taken place at home. Teachers and students separated by
physical distance but connected through technology, equipment, and resources.
Nonetheless, the data indicate that learning was not entirely online; students were
still studying in conventional ways. Conventional here does not mean that there
have been face-to-face lectures, but conventional learning methods for taking
notes, summarizing learning and memorizing notes.
In this study, the researcher distinguished between e-learning and m-learning
based on the devices used. E-learning means that the educational model used is
based on the use of electronic media and devices as tools for improving the
availability of training, communication, and interaction, and that helps to accept
new ways of understanding and learning (Salloum et al., 2019). An essential
element of e-learning is the use of electronic media, and e-learning is currently
explained as learning that is used through various computer devices, such as
computers, mobile phones, tablets, and virtual environments. M-learning focuses
on learning mobility, engagement with portable technology, and learning that
represents an emphasis on how society and its institutions could accommodate
and sustain an increasingly mobile population (Mehdipour & Zerehkafi, 2013). By
using mobile apps, learners can learn anywhere and anytime (Crescente & Lee,
2011). In Indonesia, internet users have reached 150 million people, with a 56
percent penetration spread across the region. There is only a small difference with
the number of mobile internet users, which amounts to 142.8 million people, with
a penetration percentage of 53 percent (Ministry of Communication and
Information Technology Republic of Indonesia, 2020). So, most people are linked
to the internet through mobile phones. Many homes do not have laptops or PCs,
but they are still connected online using mobile devices. The distinction between
e-learning and m-learning clarifies this phenomenon. In case of emergencies,
remote learning students have explained that they are connected using mobile
devices or tethered from a cell phone when they need to work from a laptop.
Blended learning is supposed to make learning more efficient by integrating
technology-based learning with traditional style learning. Some research has
identified many benefits of blended learning. Many studies have been carried out
to compare conventional teaching methods with blended learning, and different
results have been obtained. In some research, the participants' test scores did not
differ between the two groups (Khan, 2001). Thiele's study showed that the

19
student knowledge scores in the blended teaching method were higher than the
lecturing method, but that the difference was not statistically significant (Thiele,
2003). Sedeghi et al. contrasted students' learning and satisfaction with teaching
and e-learning with traditional teaching methods. The results showed that the
blended method is effective in increasing the student learning rate (Sadeghi et al.,
2014). McPhee and Pickern (2017) concluded how ICT could help international
students' learning experience. Kirkwood (2009) claimed that ICT could make
doable learning tasks or situations that would otherwise be extremely difficult to
achieve and encourage an incremental improvement in learning outcomes.
Stephenson, Brown, and Griffin (Stephenson et al., 2008) argued that most e-
learning seemed to mimic or complement established academic practices, mainly
when used in 'blended' contexts. Another study suggests that a combination of
face-to-face training with e-learning is more flexible than other methods (Garrison
& Kanuka, 2004).
This ERL might not have been the ideal one. Blended learning has been
questioned as to the best solution to studying during the pandemic when
universities are closed. The advantages of blended learning described in the
studies, as mentioned above, are in contrast to the experience of students.
Afterward, we must remember that this blended learning took place in an
emergency with limited resources. As highlighted in the introduction, the
researcher refers to it as "emergency remote learning," not just "remote learning"
using blended materials. Before COVID-19, the classes studied by the researcher
rarely used e-learning and m-learning. E-learning and m-learning have emerged
as one of the alternatives to the problems of education during an outbreak of
coronavirus. E-learning and m-learning offer expanded opportunities for learning
versatility when students and educators are under the stay-at-home order.
However, there were challenges in using technology in learning. Students talked
about the technical barriers that hindered their learning due to technical problems
or the inability to use technological resources.
Students also discussed the shift in learning styles. Students were already
accustomed to face-to-face conventional teaching approaches. They claimed in
their writings that they felt that they were not learning since there was no lecturer
to guide them to understand the lesson. E-learning and m-learning involve
students working interdependently, in groups or independently to solve
problems, to work on projects, to meet individual needs, and to encourage
students to speak and choose. Students complained about this kind of
independent learning, doing assignments, and reading on their own. They were
familiar with the teaching-learning model, where lecturers gave learning
materials; students listened and took notes.
Another thing to think about is the availability of learning resources in Bahasa
Indonesia. Students said that they have trouble getting learning materials since
the libraries are closed. Universities have no access to paid online journals. So,
students are only using open-access materials.
The findings of this study reveal that university and its community were not well
prepared to face an emergency, such as the closure of the campus due to a
pandemic. Courses designed for conventional learning, the unusual use of

20
technology in teaching and learning programs, students were not able to study
independently, limited access to learning materials in the native language of
learners, and lack access to high-quality reading resources (e.g., paid e-journals
and e-books). All of these issues need to be highlighted and addressed. Now and
in the future, the university should be more involved in information
communication technology. University learning should encourage students to
learn independently through a wide range of methods and media. Resources for
remote learning need to be strengthened, and students should get more access to
reading materials
Another theme that arose from this study is ERL during the outbreak of COVID-
19 is paradoxical. Paradoxical, according to the Cambridge dictionary 2020), is,
"Seeming impossible or difficult to understand because of containing two
opposite facts or characteristics." If something is paradoxical, it involves two facts
or qualities that seem to contradict each other (Collins Dictionary, 2020). The
university moved in-classroom-learning to online-based learning without enough
preparation. This rapid change was a test of organizational agility (Wu, 2020).
Students and lecturers were still adapting and looking for ways of teaching and
learning that were much better and easier. Educational planning in times of crisis
needs creative problem solving (Hodges et al., 2020). ERL needs much hard work,
but this is the only way forward. After some time, it is essential to review, develop
the current practice, and plan for future programs.
Learning from home might continue in the next few months, and this emergency
could happen again in the future. What are we supposed to do to make the ERL
better? Bao (2020) observed online teaching during the outbreak of COVID-19.
Bao has identified six educational strategies to improve student concentration and
engagement to achieve a smooth transition to online learning:
1. Preparing emergency preparedness plans for unexpected problems. If the
teaching plan A is not working because of a technical problem, try Plan B,
C, or any other plan.
2. Distributing the teaching content into smaller units to help the student
focus. The lecturers should divide the teaching content into several small
modules lasting approximately 20–25 minutes.
3. Emphasizing the use of "voice" in online teaching. Body language and
facial expressions are restricted in online instruction, and only the "voice"
is entirely functional. Therefore, in online teaching, the faculty should
properly slow down its speech to allow students to capture critical
knowledge points.
4. Working with teaching assistants and get online help from them. This
arrangement is also missing from the students' courses in this research.
There are no teaching assistants for the lecturers. Online teaching includes
the technology and professional expertise of those involved. Most faculties
at universities in Indonesia are not sufficiently trained or equipped to
operate online educational platforms, and support from teaching
assistants is especially relevant.
5. Strengthening the students' ability to learn outside the classroom. The
faculty has less control over online teaching. As a result, the progress of

21
online teaching and its learning efficiency depends mainly on high-level
active learning outside the classroom. To this end, the faculty should use
various methods to moderately modify students' homework and reading
requirements to strengthen students' active learning outside the
classroom. Students complained about assignments in this research.
Lecturers and students need a dialogue to address what is currently
expected and should be achieved by two parties, students and lecturers,
to make learning more productive.
6. Efficiently combining online learning and offline self-learning. In the
offline self-learning process, students are expected to read the course-
specific literature and submit short papers based on their reading of
essential materials before class. The faculty should provide feedback on
student assignments and learn about students' cognitive learning levels.
In the online teaching phase, the faculty should use the student discussion
section to exchange their understanding based on their reading. Thus,
students will not learn ambiguous, fragmented, and superficial
knowledge. Instead, deep learning will be experienced during the
discussion. Students in this research complained that not enough lectures
and lecturers had given them too many assignments until they felt
overloaded. This should be communicated as to why assignments are part
of learning. However, lecturers should give feedback on the work of
students, so that they know the progress of their learning.
Bao (2020) also stressed the need to provide psychological support to the
university community. In this research, students viewed remote learning in both
a positive and bad light. In their diaries, they wrote about confusion, depression,
a devastating feeling, and some negative emotions about being isolated and
learning remotely. These feelings have to be addressed. The university should
provide psychological support to help students and staff manage their emotions.
Their anxieties need to be relieved in a variety of ways to ensure that they can
engage in online learning actively.
5. Conclusion
This study concluded that students experienced an ERL that utilized blended
strategies of e-learning, m-learning, and conventional learning techniques. Their
response to the process was paradoxical as they had both positive and negative
experiences. Lecturers used significantly limited media and methods to
implement learning. Moving instruction online could make teaching and learning
accessible anywhere, at any time, but the speed at which this transition to online
instruction is expected to happen is unparalleled. After experiencing ERL for
almost a semester, and while currently waiting for government policies on how
to learn next semester, it is an excellent time to learn from multiple viewpoints,
including students, about their experience. Some of the feedback from this
research are: preparing teachers and students for emergency learning, training
lecturers on using blended resources, designing learning curricula that can be
converted into online learning, developing student-independent learning skills,
and providing access to a wide variety of digital reading materials. This study's
findings cannot be generalized because they were based on a relatively small
number of university students from one department at one university. However,

22
we assume that this work presents students' views and experiences on ERL in
several universities in Indonesia and probably in other countries, particularly in
developing countries. More research involving more students from various
universities should be initiated as a follow-up. It would also be useful to gain
insights from lecturers and students at different levels of education and discuss
the viewpoints of parents and other education stakeholders.
Funding: This research received no external funding.
Conflicts of Interest: The authors declare no conflict of interest in the design of
the study; in the collection, analysis, or interpretation of data; in the writing of the
manuscript, or in the decision to publish the results.
6. References
Abidin, Z. (2020). Belajar Matematika di Era Covid-19 [Studying Mathematics in the COVID-
19 era]. OSF Preprints. Retrieved from https://osf.io/nrbu7/
Aji, W., Dewi, F., Kristen, U., & Wacana, S. (2020). Dampak Covid-19 Terhadap
Implementasi Pembelajaran Daring Di Sekolah Dasar [COVID-19 impact on the
implementation of online learning in primary schools]. Edukatif: Jurnal Ilmu
Pendidikan, 2(1), 55–61.
https://doi.org/https://doi.org/10.31004/edukatif.v2i1.89
Anhusadar, L. O. (2020). Persepsi Mahasiswa PIAUD terhadap Kuliah Online di Masa
Pandemi [Perception of students majoring in Early Childhood Islamic Education
towards online Lectures in the COVID-19 pandemic period]. KINDERGARTEN:
Journal of Islamic Early Childhood Education, 3(1), 44–58.
Arbnor, I., & Bjerke, B. (2011). Methodology for Creating Business Knowledge. In
Methodology for Creating Business Knowledge.
https://doi.org/10.4135/9780857024473
Bao, W. (2020). COVID ‐19 and online teaching in higher education: A case study of
Peking University. Human Behavior and Emerging Technologies, 2(2), 113–115.
https://doi.org/10.1002/hbe2.191
Cambridge Dictionary. (2020). Paradoxical.
Casmir, F. L. (1983). Phenomenology and hermeneutics: Evolving approaches to the study
of intercultural and international communication. International Journal of
Intercultural Relations, 7(3), 309–324. https://doi.org/10.1016/0147-
1767(83)90035-4
Cauchemez, S., Ferguson, N. M., Wachtel, C., Tegnell, A., Saour, G., Duncan, B., & Nicoll,
A. (2009). Closure of schools during an influenza pandemic. The Lancet Infectious
Diseases, 9(8), 473–481. https://doi.org/10.1016/S1473-3099(09)70176-8
Cauchemez, S., Valleron, A.-J., Boëlle, P.-Y., Flahault, A., & Ferguson, N. M. (2008).
Estimating the impact of school closure on influenza transmission from Sentinel
data. Nature, 452(7188), 750–754. https://doi.org/10.1038/nature06732
Collins Dictionary. (2020). Paradoxical.
Concannon, F., Flynn, A., & Campbell, M. (2005). What campus-based students think
about the quality and benefits of e-learning. British Journal of Educational
Technology, 36(3), 501–512. https://doi.org/10.1111/j.1467-8535.2005.00482.x
Crescente, M. L., & Lee, D. (2011). Critical issues of m-learning: design models, adoption
processes, and future trends. Journal of the Chinese Institute of Industrial Engineers,
28(2), 111–123. https://doi.org/10.1080/10170669.2010.548856

23
Earn, D. J. D. (2012). Effects of School Closure on Incidence of Pandemic Influenza in
Alberta, Canada. Annals of Internal Medicine, 156(3), 173.
https://doi.org/10.7326/0003-4819-156-3-201202070-00005
Farrel, M. (2006). Learning differently: e-learning in nurse education. Nursing
Management, 13(6), 14–17. https://doi.org/10.7748/nm.13.6.14.s12
Felter, C. (2020). What Is the World Doing to Create a COVID-19 Vaccine? Retrieved from
https://www.cfr.org/backgrounder/what-world-doing-create-covid-19-
vaccine
Firman, F., & Rahayu, S. (2020). Online learning in the middle of the COVID-19 pandemic
(Pembelajaran Online di Tengah Pandemi Covid-19). Indonesian Journal of
Educational Science (IJES), 2(2), 81–89. https://doi.org/10.31605/ijes.v2i2.659
Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative
potential in higher education. The Internet and Higher Education, 7(2), 95–105.
https://doi.org/10.1016/j.iheduc.2004.02.001
Glass, R. J., Glass, L. M., Beyeler, W. E., & Min, H. J. (2006). Targeted social distancing
design for pandemic influenza. Emerging Infectious Diseases, 12(11), 1671–1681.
https://doi.org/10.3201/eid1211.060255
Task Force for the Acceleration of COVID-19. (2020). Infografis COVID-19 (12 Juni 2020).
Retrieved from https://covid19.go.id/p/berita/infografis-covid-19-12-juni-
2020
Gunawan, Suranti, N. M. Y., & Fathoroni. (2020). Variations of models and learning
platforms for prospective teachers during the COVID-19 pandemic period.
Indonesian Journal of Teacher Education, 1(2), 61–70.
Hidayatullah, S., Khouroh, U., Windhyastiti, I., Patalo, R. G., & Waris, A. (2020).
Implementasi Model Kesuksesan Sistem Informasi DeLone And McLean
Terhadap Sistem Pembelajaran Berbasis Aplikasi Zoom Di Saat Pandemi Covid-
19 [Implementation of the performance model of DeLone and McLean
Knowledge Systems For Zoom application-based learning systems during the
COVID-19 pandemic]. Jurnal Teknologi Dan Manajemen Informatika, 6(1).
https://doi.org/10.26905/jtmi.v6i1.4165
Hodges, C., Moore, S., Lockee, B., Trust, T., & Bond, A. (2020). The Difference Between
Emergency Remote Teaching and Online Learning. Educause Review.
Institute of Medicine. (2006). Modeling Community Containment for Pandemic Influenza.
National Academies Press. https://doi.org/10.17226/11800
Jackson, C., Vynnycky, E., & Mangtani, P. (2016). The Relationship Between School
Holidays and Transmission of Influenza in England and Wales. American Journal
of Epidemiology, 184(9), 644–651. https://doi.org/10.1093/aje/kww083
Juliani, A. J. (2020). THIS IS NOT ONLINE OR DISTANCE LEARNING. Retrieved from
http://ajjuliani.com/this-is-not-online-or-distance-learning/
Khan, F. A. (2001). Problem-based Learning Variant: Transition Phase for a Large
Institution. National Library of Medicine, J. Pak Med Assoc, 51(8), 271–274. Retrieved
from https://pubmed.ncbi.nlm.nih.gov/11715887/
Khasanah, D. R. A. U., Pramudibyanto, H., & Widuroyekti, B. (2020). Pendidikan Dalam
Masa Pandemi Covid-19 [Education during the COVID-19 pandemic]. Jurnal
Sinestesia, 10(1), 41–48.
Kirkwood, A. (2009). E‐learning: you don’t always get what you hope for. Technology,
Pedagogy and Education, 18(2), 107–121.
https://doi.org/10.1080/14759390902992576
Kumparan. (2020). Anies Tutup Sekolah di DKI Jakarta Selama 2 Minggu untuk Cegah Corona
[Anies closes school in DKI Jakarta for 2 weeks to prevent Corona]. Retrieved from

24
https://kumparan.com/kumparannews/anies-tutup-sekolah-di-dki-jakarta-
selama-2-minggu-untuk-cegah-corona-1t1V7X57VID
Lee, B. Y., Brown, S. T., Cooley, P., Potter, M. A., Wheaton, W. D., Voorhees, R. E.,
Stebbins, S., Grefenstette, J. J., Zimmer, S. M., Zimmerman, R. K., Assi, T.-M.,
Bailey, R. R., Wagener, D. K., & Burke, D. S. (2010). Simulating School Closure
Strategies to Mitigate an Influenza Epidemic. Journal of Public Health Management
and Practice, 16(3), 252–261. https://doi.org/10.1097/PHH.0b013e3181ce594e
Lee, T. H. (2020). Innovations in Care Creating the New Normal: The Clinician Response
to Covid-19. NEJM Catalyst, 19–21. https://doi.org/10.1056/CAT.20.0076
MacDermott, A. F. N. (2002). Living with angina pectoris - A phenomenological study.
European Journal of Cardiovascular Nursing. https://doi.org/10.1016/S1474-
5151(02)00047-6
Manfuso, L. G. (2020). From Emergency Remote Teaching to Rigorous Online Learning. Ed
Tech. Retrieved from
https://edtechmagazine.com/higher/article/2020/05/emergency-remote-
teaching-rigorous-online-learning-perfcon
McCarthy, M. (2020, June). Dr. Anthony Fauci: COVID-19 Will End and We Will Get Control
Over It. Healthline. Retrieved from https://www.healthline.com/health-
news/dr-anthony-fauci-this-will-end-and-we-will-get-control-over-it#6
McPhee, S., & Pickren, G. (2017). Blended learning with international students: a
multiliteracies approach. Journal of Geography in Higher Education, 41(3), 418–433.
https://doi.org/10.1080/03098265.2017.1331208
Mehdipour, Y., & Zerehkafi, H. (2013). Mobile Learning for Education: Benefits and
Challenges. International Journal of Computational Engineering Research, 03(06), 93-
101.
Meiza, A., Hanifah, F. S., & Natanel, Y. (2020). Analisis Regresi Ordinal untuk melihat
Pengaruh Media Pembelajaran Daring terhadap Antusiasme Mahasiswa Era Pandemi
Covid [Ordinal Regression analysis to see the effect of online learning media on students’
enthusiasm in the COVID pandemic era]. Retrieved from
http://sinta.ristekbrin.go.id/covid/penelitian/detail/361
Merleau-Ponty, M., & Landes, D. A. (2013). Phenomenology of perception. In
Phenomenology of Perception. https://doi.org/10.4324/9780203720714
Miles, M. B., Huberman, M. a, & Saldana, J. (2014). Drawing and Verying Conclusions.
Qualitative Data Analysis: A Methods Sourcebook.
Milman, N. B. (2020). Pandemic pedagogy. Phi Delta Kappan.
Ministry of Communication and Information Technology Republic of Indonesia. (2020).
Penggunaan Internet di Indonesia. Ministry of Communication and Information
Technology Republic of Indonesia.
https://aptika.kominfo.go.id/2019/08/penggunaan-internet-di-indonesia/
Ministry of Education and Culture. (2019). The 2019/2020 Early Childhood Education Statis.
Retrieved from https://doi.org/10.1017/CBO9781107415324.004
Ministry of Education and Culture. (2020a). 2019/2020 Junior High School Statistic.
Ministry of Education and Culture. (2020b). 2019/2020 Primary Education Statistics.
Retrieved from http://statistik.data.kemdikbud.go.id/index.php/page/sd
Ministry of Education and Culture. (2020c). 2019/2020 Senior High School Statistic.
Ministry of Education and Culture. (2020d). 2019/2020 Vocational School Statistic.
Moscona, A. (2005). Oseltamivir Resistance — Disabling Our Influenza Defenses. New
England Journal of Medicine, 353(25), 2633–2636.
https://doi.org/10.1056/NEJMp058291

25
National Agency for Disaster Management. (2020). Infeksi COVID-19 Telah Menyebar di 34
Provinsi di Indonesia, Total Positif Jadi 3.512 Kasus [COVID-19 infections have
spread in 34 provinces in Indonesia, a total of positive to 3,512 cases]. Retrieved
from https://bnpb.go.id/berita/infeksi-covid19-telah-menyebar-di-34-
provinsi-di-indonesia-total-positif-jadi-3-512-kasus
Patton, M. Q. (1999). Enhancing the quality and credibility of qualitative analysis. Health
Services Research.
Ramdhan, D., Nasihudin, Rohaniawati, D., & Pratiwi, I. M. (2020). TIPOLOGI BELAJAR
MAHASISWA PADA PEMBELAJARAN DARING [Students’ learning typology in
online learning]. Digilib UIN Sunan Gunung Djati. Retrieved from
http://digilib.uinsgd.ac.id/30594/1/Artikel KTI dadan f ramdhan dkk .pdf
Reynolds, K. A. (2020). Coronavirus: Fauci says vaccine possible by end of 2020. Medical
Economics. https://www.medicaleconomics.com/news/coronavirus-fauci-
says-vaccine-possible-end-2020
Sadeghi, R., Sadaghat, M. M., & Ahmadi, F. S. (2014). Comparison of the Effect of Lecture
and Blended Teaching Methods on Students’ Learning and Satisfaction. J. Adv
Med Ed Prof, 2(4), 146–150.
Sadique, M. Z., Adams, E. J., & Edmunds, W. J. (2008). Estimating the costs of school
closure for mitigating an influenza pandemic. BMC Public Health, 8(1), 135.
https://doi.org/10.1186/1471-2458-8-135
Saldana, J. (2009). An Introduction to Codes and Coding. The Coding Manual for Qualitative
Researchers. https://doi.org/10.1519/JSC.0b013e3181ddfd0a
Saldaña, J. (2016). The Coding Manual for Qualitative Researchers (No. 14). Sage.
Salloum, S. A., Qasim M. A. A., Al-Emran, M., Abdel M. A., & Shaalan, K. (2019).
Exploring Students’ Acceptance of E-Learning Through the Development of a
Comprehensive Technology Acceptance Model. IEEE Access, 7, 128445–128462.
https://doi.org/10.1109/ACCESS.2019.2939467
Stephenson, J. E., Brown, C., & Griffin, D. K. (2008). Electronic delivery of lectures in the
university environment: An empirical comparison of three delivery styles.
Computers & Education, 50(3), 640–651.
https://doi.org/10.1016/j.compedu.2006.08.007
Suhada, I., Kurniati, T., Pramadi, R. A., & Listiawati, M. (2020). Google Pembelajaran Daring
Berbasis Google Classroom Mahasiswa Pendidikan Biologi Pada Masa Wabah Covid-19
[Classroom-online learning of Biology education students during the COVID-19
outbreak]. Retrieved from
http://sinta.ristekbrin.go.id/covid/penelitian/detail/363
Thiele, J. E. (2003). Learning Patterns of Online Students. Journal of Nursing Education,
42(8), 364–366.
UNESCO. (2020). COVID-19 Impact on Education. Retrieved from
https://en.unesco.org/covid19
Windhiyana, E. (2020). DAMPAK COVID-19 TERHADAP KEGIATAN
PEMBELAJARAN ONLINE DI PERGURUAN TINGGI KRISTEN DI
INDONESIA [The impact of COVID-19 online learning activities in a Christian
university in Indonesia]. Perspektif Ilmu Pendidikan, 34(1), 1–8.
https://doi.org/10.21009/PIP.341.1
World Health Organization. (2020a). Coronavirus disease 2019 (COVID-19) Situation
Report – 42. World Health Organization. Retrieved from
https://www.who.int/docs/default-source/coronaviruse/situation-
reports/20200302-sitrep-42-covid-19.pdf?sfvrsn=224c1add_2
World Health Organization. (2020b). Draft landscape of COVID-19 candidate vaccines -
15 May 2020. World Health Organisation. Retrieved from

26
https://www.who.int/who-documents-detail/draft-landscape-of-covid-19-
candidate-vaccines
World Health Organization. (2020c). World Health Organization COVID-19 Dashboard.
Retrieved from https://covid19.who.int
Wu, Z. (2020). How a top Chinese university is responding to coronavirus. World Economic
Forum.
Appendix 1. Students’ Diary
Write every day about your daily learning routine. Illustrate and describe your
experiences in 300-500 words a day. If you like, you can add a picture.
Name:
Day/Date Learning Activities
Monday/4 May 2020
Tuesday/5 May 2020
Wednesday/6 May 2020
Thursday/7 May 2020
Friday/8 May 2020
Saturday/9 May 2020
Sunday/10 May 2020
Monday/11 May 2020
Tuesday/12 May 2020
Wednesday/13 May 2020
Thursday/14 May 2020
Friday/15 May 2020
Saturday/16 May 2020
Sunday/17 May 2020
Monday/18 May 2020
Appendix 2. Reflective Essay
Write your thoughts on the implementation of emergency remote learning that has been
going on for more than 2 months in 500-1000 words. Write down what you like and do
not like about ERL.

27
©2020 The authors and IJLTER.ORG. All rights reserved.
Vol. 19, No. 6, pp. 27-45, June 2020
https://doi.org/10.26803/ijlter.19.6.2
Examining the Fairness of Language Test Across
Gender with IRT-based Differential Item and
Test Functioning Methods
Burhanettin Ozdemir
Prince Sultan University, Riyadh, Saudi Arabia
https://orcid.org/0000-0001-7716-2700
Abdulrahman Hadi Alshamrani
National Center for Assessment, Riyadh, Saudi Arabia
https://orcid.org/0000-0002-6560-3422
Abstract. Test fairness is an important indicator of the validity of test
results. The fairness and equity require ensuring that the background
characteristics of test-takers, such as ethnicity and gender, do not affect
their test scores. Differential item functioning (DIF) methods are
commonly used to detect potentially biased items that lead to the unfair
assessment of the performance of test-takers with the same ability levels
coming from the different cultural, social, demographic, and linguistic
backgrounds. This study aims at detecting potentially biased items
across gender and examining their effect on test scores to ensure the
fairness of test results for each domain and the entire test. Item response
theory (IRT) based Lord’s chi-square DIF method at item level and
Mantel-Haenszel/Liu-Agresti differential test functioning (DTF) method
at test level were implemented to the English Placement Tests (EPT)
administered to high school graduates by the National Center for
Assessment. The results show that 6 items of the EPT exhibit DIF for the
entire test. Two of them are related to reading comprehension and four
to the structure domain, while none of the compositional analysis
methods shows DIF. These results indicate the existence of content
specific DIF effect. Additionally, two items exhibit uniform DIF, one of
which shows DIF favoring male students and the offer favoring female
students. The small to moderate DTF effect associated with sub-domains
and the entire test imply that DIF effects cancel each out, assuring the
fairness of results at test level. However, the items with substantially
high DIF values need to be examined by content experts to determine
the possible cause of DIF effects to avoid gender bias and unfair test
outcomes. We also suggest conducting further studies to investigate the
reasons behind the content specific DIF effects in language tests.
Keywords: test fairness and validity; gender bias; language testing;
differential item functioning; differential test functioning

28
1. Introduction
The major concern of the stakeholders in education and test-takers is to ensure
the fairness tests. The best way to provide fairness regarding the decision made
upon a test is to increase the validity and reliability of test results. Therefore, any
effort to minimize confounding factors such as random and systematic errors,
and increase validity and reliability of test will serve the purpose of developing
fair tests and valid test scores for examinees belonging to different groups.
Examining the factorial structure of a test and differential functioning at the item
level and test level are commonly used methods to assess the reliability and
validity of test scores. Differential functioning may occur when items and tests
produce different results for different groups consistently and therefore lead to
invalid test scores and decisions made based on these scores.
The stakeholders that take part in educational test development and assessment
processes explicitly emphasize the importance of fairness in test results
regarding different subgroups. They put a substantial amount of effort to detect
irrelevant factors threatening the construct validity of the test. They are aware of
the necessity and importance of collecting evidence to justify the validity and
fairness of the tests and change the testing policies accordingly. Recently, the
European Federation of Psychological Association has proposed a model for
collecting evidence of construct validity (Evers, Muñiz, Hagemeister,
Høstmælingen, Lindley, Sjöberg, & Bartram, 2013; Hope, Adamson, McManus,
Chris, & Elder, 2018) in which using differential item functioning (DIF) is
considered as an important method for assessing the quality of the test.
Moreover, the Test Commission of the Spanish Psychological Association has
emphasized the critical role of DIF analysis in the context of test fairness
(Hernández, Tomás, Ferreres, & Lloret, 2015; Hope et al., 2018).
A common definition of differential item functioning (DIF) is that an item is said
to exhibit DIF when the probability of correct response to an item differs across
subgroups with the same ability level (Hambleton & Rogers, 1989). DIF types are
classified into two groups that are uniform DIF and non-uniform DIF. An item
exhibits uniform-DIF when the difference in item performance is consistent and
in favor of certain subgroups across the entire range of ability. However, if this
difference between subgroups is not consistent, then DIF is identified as non-
uniform (Hambleton, Clauser, Mazor & Jones, 1993).
The existence of DIF is an indicator of item bias and the presence of the
secondary latent trait besides the primary latent trait that an item aims to
measure. However, this secondary latent trait does not always imply bias or
cause unfair assessment. If the secondary latent trait is related to the primary
trait and occurs due to the nature of the measured structure, then the item is not
labeled as unfair regardless of the differing performance of sub-groups. This
situation was illustrated in a study conducted by Drabinová and Martinková
(2016). They found that one DIF item related to childhood illness in which
females showed better performance than males. However, a detailed
investigation of content experts revealed that this performance difference
occurred since women are more experienced than men since they spend more
time with their children in the Czech Republic (Martinková et al., 2017).

29
Therefore, the performance difference between women and men in this example
reflects the true ability difference and does not cause unfairness. Therefore, an
item may display DIF, however, this finding does not provide enough evidence
to classify this item as a biased item. Bias is related to systematic error in test
administration and contents and relies on both statistical tests and expert
opinions (Camilli & Shepard, 1994; Clauser & Mazor, 1998; Wiberg, 2006), while
DIF only relies on statistical tests.
Although there are many different parametric and non-parametric methods to
detect DIF, which method to utilize is a main concern of researchers, since each
method has advantages and shortcomings (Anastasi & Urbina, 1997; Hunter,
2014). For instance, some methods fail to detect non-uniform DIF but effective
when the sample size is small, such as the Mantel Haenszel and Rasch methods,
while some methods are capable of detecting non-uniform DIF but requires
large sample size (Ferne & Rupp, 2007; Lai, Teresi, & Gershon,2005) such as IRT
based Raju’s area method and Lord’s chi-square method. These aforementioned
methods are the most commonly used exploratory methods utilized to identify
differential item functioning for categorical variables that represent existing
subgroups such as gender, nationality, and age groups (Aryadoust & Zhang,
2016). The next step after detecting items exhibiting DIF is to investigate the
possible source and cause of occurrence of DIF (Zhu & Aryadost, 2020).
The EPT is administered to high school graduates by the National Center for
Assessment (NCA) in Saudi Arabia. The results of the EPT has been used by
several colleges, universities, and institutes to measure students’ language skills,
to screen their improvements across different levels or to determine their
required language proficiencies (Education & Training Evaluation Commission
[ETEC], 2020). Luo and Al-Harbi (2016) examined the factorial structure of the
EPT with unidimensional and DIMTEST methods. They found strong evidence
supporting the unidimensionality of the EPT which justified the usage of the
IRT-based models instead of the classical test theory method (CTT).
1.1.Literature review
There have been many studies conducted to ensure test fairness across different
subgroups and to define the potential source of DIF effects. Drabinová and
Martinková (2016) conducted a study to determine the potential sources of DIF
effects concerning the presence of the secondary latent trait besides the primary
latent trait. They concluded that the existence of the DIF effect for some items
did not mean that these items were biased because some of the DIF effects
reflected the relationship between the secondary latent trait and the primary
latent trait. Thus, they suggested that one should avoid labeling DIF items as
biased items without the investigation of the content experts.
Chubbuck et al. (2016) studied DIF effects in the context of differing contents
across gender groups. They employed the Mantel–Haenszel and standardized
DIF methods to detect DIF items for each content domain. They found that the
males showed better performance than females in reading comprehension items.
They also defined the lack of sufficient context in the sentence completion items
as a potential source of DIF effects. Finally, they recommended utilizing more
than one DIF methods to increase the accuracy of the results. Wedman (2018)

30
examined if the language ability of non-native test takers that took the test in a
language other than their mother tongue affected their performance compared
to the native speakers. It was found that the deficiency in the language skill of
non-native test takers caused the DIF. Moreover, He defined the failure in
wording the content clearly in an item as a potential source of DIF effects.
(Siegel, 2007, Wedman, 2018).
In one study, Stage (2005) investigated the SweSAT test items administered in
spring concerning DIF across gender groups. The Mantel–Haenszel DIF method
was employed to detect DIF items and It was found that 21 out of 122 items
exhibited DIF across gender groups. Among these DIF items, 10 items related to
the quantitative and verbal domains were in favor of female students. However,
this study did not find any patterns among DIF items and did not suggest
anything about the potential source of DIF effects. Federer and her colleagues
(2016) employed the Mantel-Haenszel DIF method for detecting potential DIF
items in the context of natural selection across gender groups. They specifically
focused on open-ended questions. It was found that women outperformed men
for the items that require applying the knowledge to the new conditions.
Admitting the fact that the developed measurement instrument showed gender
bias and, they did not suggest anything about the potential source of DIF effects
due to the complex nature of DIF structure.
Similarly, Lin and Wu (2003) used DIF and differential bundle functioning (DBF)
to detect items that function differently across gender on the EPT administered
to Chinese EFL learners. For this purpose, they used the SIBTEST methods to
detect DIF items. The results of this study indicated that the testlets (item
bundles) containing the listening comprehension items showed DIF in favor of
females, while the testlets containing the grammar and vocabulary exhibited DIF
in favor of males. Thus, these findings provide strong evidence about content
specific DIF. Pae (2012) studied the trends in the magnitude of DIF on the
English subtest administered to the Korean students across gender groups for
the nine-year period. He used the Mantel-Haenszel and IRT-based likelihood
ratio test methods to detect the DIF items. Moreover, the study examined the
effects of reading strategies and perceived interest on the magnitude of DIF. The
results of this study showed the strong evidence about the relationship between
the type of items and DIF, and a substantial interaction between the test takers
interest in the items and the magnitude of DIF across gender.
It is substantially important to run DTF analyses along with DIF since items are
small and unreliable compared to the test (Gierl, Bisanz, Bisanz, Boughton, &
Khaliq, 2001) and the total amount of DIF provides an overall effect of DIF on
test scores even when there is no item detected as DIF in a test (Hunter, 2014;
Shealy & Stout, 1993). Additionally, DTF values can be negligibly small when
these DIF items are in favor of different subgroups or in a different direction
where DIF effects cancel each out (Borsboom, 2006; Zhu & Aryadoust, 2020).
DTF is also important since decisions about examinees are not made at item-
level, but test-level (Ellis & Raju, 2003; Roznowski & Reith, 1999; Pae & Park,
2006; Zumbo, 2003). More detailed information about the DIF and DTF methods
is provided in the following sections.

31
1.2.IRT based DIF methods
The IRT based DIF methods are suggested in the case of a large sample size. The
latent variable (ability estimate-θ) estimated by IRT models is used as a
matching variable for subgroups rather than observed scores. There are many
different IRT based methods to detect DIF items, some of which are test of b
difference, Lord’s chi-square, Raju’s area method, likelihood ratio test (LRT
method), and item drift method. The Lord’s chi-square DIF method (Lord, 1980)
is an extension of the test of b difference and takes the other parameters into
account. The major shortcoming of Raju’s area method is that the exact areas
between ICCs are infinite when guessing parameters are not equal (Hunter,
2014). In this project, Lord’s chi-square DIF method was used to detect DIF
items, because it takes more than one parameter into account when calculating
DIF statistics and capable of detecting both uniform and non-uniform DIF in the
presence of large sample size. The “difR” package installed in the R program
was used to run DIF analyses. The formula for Lord’s chi-square DIF methods is
as follows:
(1)
where VjR = (ajR, bjR, cjR) and VjF = (ajF, bjF, cjF) are the vectors of item parameters
related to the reference group and focal group, respectively. Besides, the
variance-covariance matrices of reference and focal groups are denoted by ∑ 𝑗𝑅
and ∑ 𝑗𝐹, respectively. The 𝑄1-statistic has chi-square distribution and its
degrees of freedom is equal to the number of estimated parameters (Camilli,
2006; Lord, 1980). Previously research show that DIF results obtained from
Lord’s chi-squared test and Raju’s unsigned area method are highly correlated
(Millsap & Everson, 1993; Shepard, Camilli, &Williams, 1985). The most
important disadvantage of the Lord’s chi-squared test is that it tends to reject the
null hypothesis of no DIF even when the discrepancy between ICCs of sub-
groups is small in the presence of a large sample size (Camilli & Shephard,1994;
Wiberg, 2006). Thus, a more stringent criterion should be used in the presence of
a large sample size.
1.3 Differential test functioning (DTF)
Differential test functioning (DTF) values correspond to the total amount of DIF
for the entire test. Therefore, it is equal to the sum of item DIF statistics in a test
(Donovan, Drasgow & Probst, 2000; Ellis & Mead, 2000; Hunter, 2014;
Nandakumar, 1993). There are different methods to calculate DTF such as Raju’s
DFIT (Raju, van der Linden & Fleer, 1995), and Mantel-Haenszel/Liu-Agresti
method (Penfield & Algina, 2006). Raju’s DFIT estimates DTF through
calculating the squared difference between test characteristic curves, while the
Mantel-Haenszel/Liu-Agresti method is based on variance estimates and tend to
have higher DTF rates than Raju’s DFIT (Hunter, 2014). Penfield (2005, 2013)
developed a program called DIFAS which enables us to calculate both DIF and
the Mantel-Haenszel/Liu-Agresti statistics. In this study, the MH-LA method
was used to evaluate DTF associated with the EPT tests.
( ) ( ) ( )
1
j jR jF jR jF
Q v v jR jF v v
 −
= − − −
 

32
The formula for MH-LA DTF method proposed by Camilli and Penfield (1997) is
as follows:
2 2
1
2 1 1
( )
I I
i
i i
s
I
 
 = =
− −
=
  (2)
where “I” represents the number of items; 𝜓𝑖
̂ denotes MH log-odds ratio
statistics; µ represents mean and 𝑠𝑖
2
represent the error variance of ψ. Some
studies report weighted 𝜏2
statistics along with 𝜏2
statistics. The formula for
weighted 𝜏2
is as follows:
2 2
1
2 1 1
1
( )
I I
i i
i i
I
i
i
W W
W
 
 = =
=
− −
=
 

(3)
where 𝑤𝑖 is equal to 𝑠𝑖
−2
.
1.4 The purpose of the study
This study aims at examining the presence of items that function differently for
the entire test and across different sub-domains for gender in English Placement
Tests (EPT) to ensure the fairness of the test. For this purpose, item response
theory (IRT) based Differential Item Functioning (DIF) at the item level and
Differential Test Functioning (DTF) methods at the test level were implemented,
respectively, to examine whether items function differently across different
subgroups.
Considering the findings of previously conducted studies, this study aims to test
five different hypotheses. The first hypothesis is that the factorial structure of the
EPT remains unchanged across gender groups. The second hypothesis is that
some of the EPT items are likely to exhibit DIF across gender. The third
hypothesis assumes the existence of content specific DIF items at the item level.
The fourth and fifth hypotheses are that the existence of DIF items affects the test
scores for the entire test and each subdomain.
1.5 Research questions
1. Do the factorial structure of EPT for the entire test and each gender group
support the unidimensionaliy?
2. Do items of the EPT function differently across gender (female vs. male)?
3. What is the distribution of DIF items across sub-domains (Reading
Comprehension, Structure, and Compositional Analysis), when each domain is
treated as a separate test?
4. Do test scores of the EPT exhibit differential test functioning (DTF) across
gender (Female vs. Male)?
5. Do test scores of the EPT exhibit differential test functioning (DTF) across
gender, when each domain is treated as a separate test?

IJLTER.ORG Vol 19 No 6 June 2020

IJLTER.ORG Vol 19 No 6 June 2020

Recommended

Recommended

More Related Content

Similar to IJLTER.ORG Vol 19 No 6 June 2020

Similar to IJLTER.ORG Vol 19 No 6 June 2020 (20)

More from ijlterorg

More from ijlterorg (16)

Recently uploaded

Recently uploaded (20)

IJLTER.ORG Vol 19 No 6 June 2020