AUTOMATED SYSTEM FOR GENERATING AND VALIDATION A LEARNING TESTS

AUTOMATED SYSTEM FOR GENERATING AND VALIDATION A
LEARNING TESTS
Iva Kostadinova, K. Rasheva-Yordanova, Ivan Ivanov, Pepa Petrova
University of Library Studies and Information Technologies (BULGARIA)
Abstract
The advent of modern ICT has enhanced the process of automation of activities that emerged in the
era of industrial society. Gradually, more and more activities are taken over by computers making
human intervention squeezed to a periodic operability check of the same constructed system. In the
educational system and the process of training, the automation of the test generation process at the
stage of creating the test questions and facilitating the intellectual efforts made by lecturers are still
insufficiently developed. The issue stems from the lack of an integrated automated system for testing
what has been learned whereby the manual job of the lecturers in creating test questions can be
replaced by an automated one. A way to neutralize the existing inconvenience in the process of
learning is the development of a special system for generating tests.
The current paper represents an automated system for generating and checking tests through the use
of templates, which suggests an accelerated generation of exam questions as well as economizing on
the precious time of lecturers. The introduction and implementation of such a system provides a
progressive generation of a sufficient number of exam questions, which will help prepare individual
tests for the participants in the learning process. Research has shown that the integration of the
system into the process of testing the knowledge that has been acquired enhances to a great extent
the time for generating individual exam tests for each student.
Keywords: computer systems and technologies, automated system, progressive generate test
questions, generating test, template, individual test, automated validation.
1 INTRODUCTION
The penetration of modern ICT has strengthened the process of automating the activities that have
emerged in the industrial society. In many human activities, the process of automation has replaced
human labor with machine engineering. This in turn has resulted in better productivity and lower cost
of product or service. However, there are still areas where the degree of automation is relatively weak.
One of these areas is the use of intellectual work by the teachers in the process of preparing the tests.
Testing worldwide has already been successfully implemented since the end of the last 20th century.
The introduction of the test as a form of testing in Bulgaria was required by the National Program for
the Development of School Education and Preschool Education and Training (2006-2015). By
adopting this program, the test was the basis of an "effective system of internal evaluation and
national standardized external evaluation" in Bulgaria and is defined as "a leading form of ongoing
control in the learning process and at the end of each educational stage" [9].
There are many advantages of the test method for both the individual learner and the development of
education as a whole. These can be attributed to the much larger scope and depth of the questions on
the material studied; easier testing and analysis of the learning outcomes, using the tests as a
simulator in order to strengthen the knowledge and increase the skills and competencies of the
learners.
Using the test as a tool for checking and controlling knowledge requires, in turn, the preparation of
multiple tests and, subsequently, their examination and analysis of the results. The use of computer
tests and automated knowledge testing systems greatly facilitates the learning process. The
introduction and use of an automated system to ensure the progressive generation of a sufficient
number of exam questions and the development of individual tests for each learner will facilitate the
teachers and improve the learner's knowledge.
Proceedings of ICERI2017 Conference
16th-18th November 2017, Seville, Spain
ISBN: 978-84-697-6957-7
0414

2 DEGREE OF AUTOMATION IN THE PROCESS OF GENERATING TESTS
AND CHECKING THEM
The test as a means of verifying knowledge dates back to 1350-1400 [4]. Standardized testing began
to be a test method in British universities after 1850, replacing the Oral Medieval Test. [10] The test is
also used in Bulgaria, but since its introduction as a test and control form in the Bulgarian School [9],
their use has increased considerably. In the history has can be distinguished a several types of tests:
• Primary paper test - created, filled, checked and processed entirely on paper. Due to the
penetration of computers in all spheres of life, in middle and high-developed economies is no
longer used.
• Contemporary paper test - paper-based, digitized, but provided for completion and subsequent
verification as a paper test.
• Computer Test - Off-line Digitized Test.
• On-line test - a digitized test that is available on-line or on a local network. Typically, such a test
is tied to retrieving test queries from a database.
The preparation of each test passes through several stages, forming the so-called test cycle: setting
the objectives of the test, creating, specifying, editing and arranging the test tasks, pilot test, a priori
analysis, processing the test tasks, apostilary analysis, normalization, instructions, posting, and
sharing [1][13][14][6]. The computerization and automation process also reflects the stages of
creating, completing, and checking the tests. From preparation of a paper test, hand-filling and manual
verification of results, today is the introduction of automation at every stage of the test's life cycle until
its testing and processing of the subsequent results.
2.1 Automation in the process of creating tests
In addition to manual (non-automated) testing technology, automation has begun to take place in the
testing stages. New technologies allow for the creation of tests that completely exclude the paper
transfer stage. There is automation from the very beginning of the process - from filling the test
question till its verification and analysis.
At present, two approaches are basically used in the process of creating test questions valid for all
types of tests:
• Create a new test question;
• Creating a test question by changing value or concept in an existing one.
Both approaches to creating a test question require the lecturer (s) to perform intellectual work to
create each test question individually. Intellectual work consists of the preparation of the specific
question - a way of asking, clarity and unambiguousness of a short response, etc., which take a
significant teaching time. The extent of automation at this stage is limited to the digitization of test
questions.
The test questions in modern paper tests and computer off-line tests are used once. For each next
test, a new test item should be created.
Questions compiling on-line tests are stored in a database. Once created and entered into the
database, they can be used repeatedly to create many different tests. For this purpose, each question
in the database must be bound to a certain weight or theme. For the questions already created,
computer programs have the ability to group questions into tests under certain laws and levels of
complexity. In the case of closed questions, it is also possible through the computer program to
automatically arrange the Random principle responses so as to provide some degree of individuality to
the question and test.
The implementation of the intellectual work on creating any test question still remains not automated in
any of the existing test-creation capabilities.
2.2 Automation in the process of completing and checking of tests
The introduction of automation into the process of completing and checking the tests saved teachers'
time. It greatly accelerates the processing time and analysis of the results of the exams. Computer off-
line and on-line tests, which are completed with a click of a mouse, provide information as soon as the
0415

response is answered. On the one hand, learners can immediately understand their results and the
mistakes they have made. They can also focus on the appropriate literature to help them learn the
missing knowledge. On the other hand, appraisers have the opportunity immediately to learn about the
testing and its results.
Developed testing applications allow for a variety of statistics, after testing, both locally on a particular
computer and on a web-based application. In use are the many and varied products and programs
offering electronic tests[16][6]. These are systems such as CMS Moodle [7], CMS Joomla with its
addition AriQuiz [2], QuizLet [12], Google Forms, programs such as MyTest [8], TCEXAM [15],
ClassMarker [3] and other [5][11].
Figure 1 shows the main stages and processes of the life cycle of a test and the degree of intrusion of
automation into them.
Fig. 1 Degree of automation in the stages and substages of the modern paper and computer test lifecycle.
2.3 Advantages of the automated knowledge testing systems
• Ability to store and reuse the created issues in a database
• Ability to facilitate the bulk import (adding) of file issues
• Automatically check responses
• Automatic notification of student completion
• Automatically running statistics on test results
• Ability to test at any time using on-line systems
3 DESIGN AND CONSTRUCTION OF AN AUTOMATED SYSTEM FOR
GENERATING AND CHECKING TESTS BY MEANS OF TEMPLATES (ASGT)
The introduction of the test as a leading form of testing has put the training institutions in the challenge
of a relatively short time to prepare a number of tests to test the knowledge of their students. The
0416

process of preparing a test depends on its subject, the number of questions in it, the type of answer
(open or closed questions). The preparation of a test requires that its creator be competent in the field.
Today, a variety of courses and forms of training are available to suit the learner. The use of computer
or on-line tests, their automated checking and processing, is a great facilitation for trainers in terms of
grouping the questions and automatic checking.
After reviewing and evaluating the techniques for creating testing software and examination, including
the aforementioned programs, it can be concluded that the process of automation in creating tests is
limited to automatically retrieving questions from a database on random principle. The way the
questions are entered into the database is done by a human, question by question or they may be
imported from a file, but usually in some specific format and structure. The creation of the questions
themselves is still the work of the intellectual work of the lecturers.
3.1 Designing the system
The construction of an automated system for generating tests (ASGT) aims at facilitating the teaching
work in the process of creating and testing test exams. The direct potential users of this system are
the teachers, and as a result of their activity, users become the learners themselves.
Building the ASGT is based on a model for creating a system for automated generation of tests [1].
The basis of the approach is to shape and preserve templates that have fields with areas or lists of
values (fig.2). On the basis of the already prepared templates with the fields enclosed in them, in the
process of generating a test, random templates with random values given in each field are derived,
and a test is generated. In this way, multiple tests are generated from a template in a much shorter
time, which significantly alleviates intellectual work in the process of creation. The generation of each
test is linked to the process of compiling the static part of the template with the dynamic one that is
being performed at the time the test is created.
Generating a test is done in two stages:
• Stage 1. Preparing and writing templates in a template database
• Stage 2. Select a number of templates from a category to generate a test
Fig.2. Technology for Automated Test Generation trough templates.
3.2 Generating templates
The preparation of templates is at the heart of the ASGT. The template presents a template of a
question that consists of two parts - a static part and a dynamic part (Figure 3). The static section sets
the constant values of the question, while in the dynamic part are set the areas with values. The
dynamic part of each template is bound to a compiler that extracts a random value from each field,
and forms a set of questions based on these values. Fig. 3 shows a template of a C ++ programming
0417

template. The Condition field represents the constant part of the template, and the Template field is
the dynamic field.
The preparation of a template from another discipline (literature, English, etc.) according to the
discipline's specificity could contain several static and several dynamic parts and by their grouping to
generate a template. Each template gets a category and weight. The category is used to select the
type of exam preparation (initial, intermediate, or final). The weight of the template is used in the
process of adaptive self-test.
After the template is prepared, it is generated to check if it is properly constructed and returns a
response, and then it is saved in a database.
Figure 3 Generate a template by setting fields with ranges of values.
3.3 Generating tests
The process of generating a test is to select templates from the database and compile arbitrary values
from the fields in the database. It takes only two steps - choosing categories and number of questions.
The teacher forms the test by selecting the category (topic) and number of questions from it, then sets
the desired number of tests and presses the "Generate" button.
The test generation process retrieves templates selected from the database from the specified
categories, randomly selects values from the fields assigned to each field, and generates the
questions and answers they ask for.
Based on the number of fields and the number of values in each field of each template, the set of
questions is generated and the number of tests required will be formed. As a result of the generation,
individual tests with individual questions are prepared for each learner. Each test receives an
individual number on the basis of which the automatic check is subsequently performed.
0418

Fig. 4. Generate a test by selecting templates and a number of questions from them.
3.4 Meta language structure
At the core of the automatic test generation based on the set values in the fields in each template
stands the meta-language of the ASGT. It consists of the use of acceptable symbols, constructions
and an analysis mechanism through which the system functions.
3.4.1 Symbols
Table 1. The ASGT metaelectric symbols are.
Symbols Description
Latin letters a-z, A-Z
National letters а-я; А-Я
Whole Numbers 0-9
Fractional Numbers 0.0-9.9
Special Symbols „>“,“<“,“!“,“==“,“=“,“&“,”(.)“,“[ ]“, „{ }“,“;“ и др.
The saved words in the programming language
used (language specific)
Official words and operators ( int, string, double,
return, for, if, break; continue )
3.4.2 Constructions
In the currently used meta-language in ASGT only two relatively easy constructions are used.
Table 2. Constructions built into the meta-language of the ASGT.
Construction Description
#[…, …] set a range of values
#[…; …] set values through a list
 Structure for setting value ranges
Assigning value ranges is convenient for templates of queries containing numeric values. They are
easy to define, have a broader scope and are clearly defined. Assigning a range of values is used in
0419

the dynamic part of the template. A specific symbol is used that defines the value field (Figure 5). To
set a range of values, the symbol "#" is used, and in the [ ] it is set the start and end values of the
value range. The interval setting is valid for both integer and fractional numbers.
 Structure to set values through a list
Setting values using a list is valid when there is no clearly defined interval. The list of values can be of
numbers, symbols, even concepts. The character "#" is set again, and into the [] square brackets the
options are listed. The use of a set list value is more limited than the value range, but is successfully
applied to non-mathematical disciplines and languages.
Fig.5. Meta-language in ASGT.
The proposed meta-language in ACGT allows the successful combination of both types of
constructions as many times as needed in each template. This provides the system with the ability to
generate unlimited variations of questions from any template.
 Analyzing mechanism
The use of an analysis mechanism is required when creating a template with a fractional number. The
analyzing mechanism sets a suitable step of incrementing a fractional number to an exact symbol. In
the ACGT, an analysis mechanism of fractional increment is set to two decimal places. When
generating fragments using fractional numbers, the analysis mechanism determines the next number
(Table 3).
Table 3. Example of an integral and fractional analysis mechanism.
Template Step of the
analyzing
mechanism
Sample Generated Issues Number of
Generated
Issues
What result will the
program output?
int main()
{
int a=#[2,4];
if(a>3) cout<<”OK”;
return 0;
}
1
What result will the program output?
int main()
{ int a=2;
return 0;}
3
int main()
{ int a=3;
return 0;}
0420

int main()
{ int a=4;
return 0;}
What result will the
program output?
int main()
{
double a=#[2.0, 4.3];
return 0;
}
0.01
int main()
{ double =2.0;
return 0;}
23
int main()
{ double a=2,01;
return 0;}
int main()
{ double a=2.02;
return 0;}
………..
………..
int main()
{ double a=2.03;
return 0;}
4 INVESTIGATION OF THE EFFECTIVENESS OF ASGT TEST GENERATION
The preparation of test questions via ACGT through templates is especially useful in preparing tests
with computations in disciplines such as mathematics, statistics and programming because of clarity
when setting value ranges
The preparation of test questions in a different field or discipline is also feasible, but in this case the
value assortments are actually formed as enumeration lists. The variants of the test question are
formed by combining the static part of the text with the value set in the list box. Due to the fact that
listing options are smaller, the number of test questions generated will be smaller.
The output of the number of fields and the number of values assigned to each one in a template
returns the number of possible combinations in the question generation process. For the calculation of
the possible number of generated questions from a template, the following formula can be used:
Generating a question in which a template uses two fields to set values, with 5 options in each field
after combining, generates 10 variants of the question. By increasing the number of fields and the set
0421

value ranges for each field, the different possible variations for each question are increased. Thus, to
generate questions using 10 templates, each with 2 fields with set intervals of 10 values without
matching and without repetition, 200 questions are generated. The time to generate these questions,
however, is to select the category of templates and the number of questions they have.
Testing time is limited to selecting a category and number of questions from it and pressing a button.
The only requirement is that there are enough templates to be entered for each category with the
specified value fields in them. In automated test generation, it is important to select and combine the
prepared templates in the test template. Setting a larger range of values for each field in the selected
template results in a larger number of issues.
The technology embedded in the automated pattern generation approach using templates can be
used for both automated test creation and non-automated (manual) testing. At the teacher's choice,
the tests so prepared can be downloaded and provided for filling in the paper (Figure 6), and may also
be provided for computer replenishment according to the size of the test group and the availability of
ICT for conducting the test.
Fig. 6. Sample C ++ Programming Test.
Benefits of ASGT through templates
• There is less storage space necessary than other test generating systems because it keeps the
templates rather than the generated issues.
• More secure as test questions and tests are generated at the time the test is created.
• Easily generate test questions (the introduction of a template with at least 2 fields with 5 values
generates 10 different questions.
• The test preparation process takes considerably less time for the teacher.
• The Individual Tests prepared by ASGT for each learner reduce the ability to share knowledge
during the exam.
• Ensured automatic check of completed tests saves time for verification.
0422

5 CONCLUSION
Developing a system for automatically generating test questions and testing through templates would
help alleviate intellectual work in the process of generating questions from test makers.
Built on this principle, the auto-generating system is well-suited both to the preparation of tests with
computations in disciplines such as mathematics, statistics, and programming because of the clarity of
setting value intervals, as well as in different areas or disciplines.
The system currently allows generating tests for C ++ and C # disciplines, but provides the ability to
generate tests for Java programming, mathematical tasks, and languages
Using the test as a form of testing provides a broader scope for verifying learner knowledge.
Generating more questions is a way to generate individual tests for each learner.
REFERENCES
[1] Kostadinova, I. Ivan Ivanov, Katia Rasheva – Yordanova. An Approach to constructing a model
for creating a system for automated generation of tests//Proceedings of the 2nd world multi
conference on global science (ATTENTIS 2016) 14-16 Dec. 2016, , Publishing Institution of
University of Zilina, 2017, pp 35-41, ISBN 978-80-554-1302-0
[2] Ari Soft [on-line resource]16.10.2016 http://www.ari-soft.com/Joomla-Components/ARI-
Quiz/Detailed-product-flyer.html?_
[3] Class marker [on-line resource] 16.11.2016. https://www.classmarker.com/
[4] Dictionary.com [on-line resource].16.10.2016. http://www.dictionary.com/browse/test
[5] Huang, Si; Myra B. Cohen ; Atif M. Memon (2010). Repairing GUI Test Suites Using a Genetic
Algorithm. Third International Conference on Software Testing, Verification and Validation
(ICST2010), 6-10 April 2010 Electronic ISBN: 978-1-4244-6436-4, DOI: 10.1109/ICST.2010.39
[6] Koleva, N, Designing a knowledge management system in an integrated information
environment. Proceedings of Uniwersity of Ruse “Angel Kanchev” - 2012, volume 51, book 6.2 ,
pp-175-180, ISSN 1311-3321
[7] Moodle [on-line resource] 16.10.2016 https://docs.moodle.org/31/en/Teacher_quick_guide
[8] MyTestXPro [on-line resource] 17.11.2016 http://mytest.klyaksa.net/
[9] National Programme for Development of School Education and Primary education and training
2006 – 2015 year. // The Public Procurement portal Council of Ministers. [on-line
resource]17.10.2016. http://www.strategy.bg/StrategicDocuments/View.aspx?lang=bg-
BG&Id=393
[10] North Central Regional Educational Laboratory [on-line resource] 15.10.2016
http://www.ncrel.org/sdrs/areas/issues/students/earlycld/ea5lk3.htm
[11] Qian Kai, Mate Sztipanovits ; Xiang Fu (2008) Automated Testing and Smart Tutoring System
for Web Application. International Workshop on Education Technology and Training (2008) and
International Workshop on Geoscience and Remote Sensing(2008). ETT and GRS 2008. Print
ISBN: 978-0-7695-3563-0, DOI: 10.1109/ETTandGRS.2008.232
[12] Quizlet [on-line resource] 16.11.2016 https://quizlet.com/latest
[13] Shotlekov, I. (2011). Drawing up a pilot test online Information Technology. i- Continuing
education, 25. [ on-line resource] 16.10.2016.
http://www.diuu.bg/ispisanie/broi25/25kt/25kt1.htm
[14] Shotlekov, I. (2015). From pilot to the final version of the online test in Information Technology.
i- Continuing education [on-line resource] 17.10.2016 http://diuu.bg/emag/2919/6/
[15] TCExam Open Source Computer-Based Assessment Software [on-line resource] 16.11.2016.
https://tcexam.org/
0423

[16] Pashev, G.,Georgi Totkov., Automatized generation of personalized learning paths thought
aspects in multidimensional spaces. Proceeding of ІX National conferences „ Education and
research in the information society” 26 – 27 may 2016, Plovdiv, Bulgaria (ERIS-2016), pp.43-52,
ISBN 978-954-8986-45-8.
0424

AUTOMATED SYSTEM FOR GENERATING AND VALIDATION A LEARNING TESTS

Recommended

Recommended

More Related Content

Similar to AUTOMATED SYSTEM FOR GENERATING AND VALIDATION A LEARNING TESTS

Similar to AUTOMATED SYSTEM FOR GENERATING AND VALIDATION A LEARNING TESTS (20)

More from Gina Rizzo

More from Gina Rizzo (20)

Recently uploaded

Recently uploaded (20)

AUTOMATED SYSTEM FOR GENERATING AND VALIDATION A LEARNING TESTS