K5. E. mokymas(is) XVIII mokslinė kompiuterininkų konferencija „Kompiuterininkų dienos – 2017“, Kaune, VU KnF 2017-09-22

1. On Methodology to Evaluate
Acceptance, Use and Suitability of
Personalised Learning Units
Eugenijus Kurilovas, Saulius Minkevičius, Julija Kurilova, Irina Vinogradova
KoDi 2017, Kaunas, 22 Sep. 2017

2. • The aim of the paper
• Related research
• Research methodology
• Research results
• Conclusion
Outline

3. The aim of the
paper

4.  The aim of the paper is to present methodology (i.e. model
and method) to evaluate acceptance, use and suitability of
personalised learning units / scenarios for particular students.
 Learning units / scenarios are referred here as
methodological sequences of learning components (learning
objects, learning activities, and learning environment).
 High-quality learning units should consist of the learning
components optimised to particular students according to
their personal needs, e.g. learning styles.
 In the paper, optimised learning units mean learning units
composed of the components having the highest probabilistic
suitability indexes to particular students according to Felder-
Silverman learning styles model.

5. Related research

6. • Learning Package / Scenario / Unit – an aggregation of learning
activities that take place in particular (virtual) learning environments
(VLEs) using particular learning objects (LOs)
• LO – any digital resource that can be reused to support learning
• Learning activities (LAs) describe a role they have to undertake
within a specified environment composed of LOs and services.
Activities take place in a so-called “environment”, which is a
structured collection of LOs, services, and sub-environments
• VLE – a single piece of software, accessed via standard Web browser,
which provides an integrated online learning environment

7. 1. Question: What methods and techniques on evaluating the quality of
personalised learning exist in scientific literature?
2. Question: What methods and techniques of multiple criteria decision
making in education exist in scientific literature?

8.  The main idea of personalisation is to reach an abstract common goal: to provide
users with what they want or need without expecting them to ask for it explicitly.
 From the educational point of view, personalisation attempts to provide for an
individual tailored products, services, information, etc.
 A more technical standpoint to personalisation is linked with the modelling of Web
objects (products and pages) and subjects (users), their categorisation, organising
them to achieve the desired personalisation.
 Personalisation provides training programmes that are customised to individual
learners, based on an analysis of the learners’ objectives, current status of skills /
knowledge, LS preferences, as well as constant monitoring of progress.
 The concept of personalised learning advocates that instruction should not be
restricted by time, place or any other barriers, and should be tailored to the
continuously modified individual learner’s requirements, abilities, preferences,
background knowledge, interests, skills, etc.
 The personalised learning concept signifies a radical departure in educational theory
and technology, from “traditional” interactive learning environments to personalised
learning environments.

9. Fig. 1. UTAUT model (according to Venkatesh et. al. (2003))

10. Fig. 2. ETAS-M (according to Poelmans et. al. (2009))

11. Research
methodology

12.  In order to propose psychologically, pedagogically, mathematically, and
technologically sound methodology to create the whole optimal
personalised learning package, several approaches, concepts and
methods are applied:
 The concept of personalised learning package / unit
 Learning personalisation method based on application of intelligent
technologies (creation of learners’ profiles, ontologies to interrelate
learners’ profiles with learning components, recommender system,
intelligent agents)
 Stochastic approach for automatic and dynamic modelling of
students’ learning styles
 Personalised learning components’ recommendation method
based on expert evaluation techniques

13. Felder-Silverman LS Model is known as the most suitable for engineering
education and e-learning. FSLSM classifies students according to where they fit on
4 scales pertaining to the ways they receive and process information (dimensions)
as follows:
• By Information type: Sensory (SEN) vs Intuitive (INT);
• By Sensory channel: Visual (VIS) vs Verbal (VER);
• By Information processing: Active (ACT) vs Reflective (REF), and
• By Understanding: Sequential (SEQ) vs Global (GLO).

14. Felder-Silverman Learning Styles Model
If a student answers 7 questions favourable to the Sensory LS, and 4
questions favourable to the Intuitive LS: PRSEN = 7 / 11 = 0.64, and PRINT
= 4 / 11 = 0.36, and further on to all dimensions of FSLSM.
Thus, one could obtain e.g. the following LS initially stored in his / her
student profile / model:
Table 1. Example of LS initially stored in the student profile / model
Learning styles
By Information
type
By Sensory
channel
By Information
processing
By Understanding
SEN INT VIS VER ACT REF SEQ GLO
0.64 0.36 0.82 0.18 0.73 0.27 0.45 0.55

15. • Evaluation – a process by which people make
judgements about value and worth
• Quality evaluation – a systematic examination of the
extent to which an entity (part, product, service or
organisation) is capable of meeting specified
requirements
• Expert evaluation – a multiple criteria evaluation of
learning software aimed at the selection of the best
alternative based on score-ranking results

17. Due to the probabilistic nature of LS in the FSLSM, our approach is based on probabilistic
LS combinations. Each LS combination is a 4-tuple composed by one preference from each
FSLSM dimension. Students’ probable LS are stored in student profile / model as values of
the interval [0,1]. Those values represent probabilities of preference in each of FSLSM
dimension. Therefore, students’ LS are stored as probability distributions considering each
FSLSM learning dimension. Considering this kind of model, students’ LS are stored in their
profiles / models according to Definition 1:
LS = {(PRSEN = x; PRINT = 1 – x), (PRVIS = y; PRVER = 1 – y), (PRACT = z; PRREF = 1 – z),
(PRSEQ = v, PRGLO = 1 – v)}, where
PRSEN is the probability of the student’s preference for the Sensory LS;
PRINT is the probability of the student’s preference for the Intuitive LS;
PRVIS is the probability of the student’s preference for the Visual LS;
PRVER is the probability of the student’s preference for the Verbal LS;
PRACT is the probability of the student’s preference for the Active LS;
PRREF is the probability of the student’s preference for the Reflective LS; and
PRSEQ is the probability of the student’s preference for the Sequential LS; and PRGLO is
the probability of the student’s preference for the Global LS.

18. Consequently,
PRSEN + PRINT = 1;
PRVIS + PRVER = 1;
PRACT + PRREF = 1;
PRSEQ + PRGLO = 1.
Calculations of probabilities should be done according to Formula 1:
Formula (1) divides by 11 the number of favourable answers to LS
(Ai), considering that Index of LS has 11 questions for each FSLSM
dimension, totalling 44 questions in Soloman-Felder Index of Learning
Styles Questionnaire.
In (1), i represent a LS in FSLSM dimension, and Ai represent the
number of favourable answers to a LS.
PRi is a probability of preference to a LS by the student in a FSLSM
dimension, according to aforementioned Definition 1.
11
i
i
A
PR = (1)

19. Suitability of inquiry-based learning (IBL) activities and sub-activities to FSLSM is
presented in Table 2. IBL activities are divided into sub-activities, and all those sub-
activities are evaluated by the experts in terms of their suitability to students’ LS.
Expert evaluation method is applied here. Suitability ratings mean the aggregated
level of suitability of particular IBL sub-activities to particular LS.

20. If one should multiply these suitability ratings by probabilities of
particular students’ LS according to Table 1, he / she should obtain
probabilistic ratings / values of suitability of particular IBL sub-activities
to particular student’s (i.e. Active) LS according to Formula 2:
PRVACT = PRACT * VACT (2)
where PRVACT means probabilistic value (level) of suitability of particular
IBL sub-activity to particular student according to his / her preference
to Activist LS,
PRACT means probabilistic value of the student’s preference to Activist
LS (e.g. 0.73 according to Table 1), and
VACT means the value of suitability of particular IBL sub-activity to
Activist LS.
This Formula should be applied for each IBL sub-activity.

21. Accordingly, PRVs mean the indexes of particular learning component’s suitability to
particular student.
These Suitability Indexes should be included in the recommender system, and all
learning components should be linked to particular students according to those
Suitability Indexes.
The higher Suitability Index the better the learning component fits particular student’s
needs.
Thus, optimal learning package (i.e. learning package of the highest quality) for
particular student means a methodological sequence of learning components (LAs,
LOs and VLE) having the highest Suitability Indexes.
The level of students’ competences, i.e. knowledge / understanding, skills and
attitudes / values directly depends on the level of application of optimal learning
packages in real pedagogical practice.
Thus, in order to create a probabilistic model for a whole personalised learning
package consisting of suitable learning components optimal to particular students
according to their profiles, one should apply Formula 1, appropriate Table 1, and
Formula 2 for all components of the learning packages.

22. Research results

23. Fig. 3. Proposed UoL suitability, acceptance and use evaluation
model

24. Thus, in order to identify numerical value of UoL evaluation function, we would:
(1)multiply the values of all ETAS-M-based evaluation criteria by their weights,
(2)add these numbers together and identify the sum,
(3)multiply all these sums by probabilistic suitability indexes of corresponding
learning components, and
(4)identify the total sum. The higher the numerical value of f(x) the better is the UoL
for particular learner.

25. Conclusion

26. In the paper, the authors propose personalised learning units / scenarios acceptance, use and suitability
for particular students model based on MCDA criteria identification principles, learning components’-
based evaluation model, and Educational Technology Acceptance & Satisfaction Model (ETAS-M) based
on UTAUT model.
Personalisation of UoL components and the whole UoL should be guaranteed by correct identifying
corresponding probabilistic suitability indexes.
The proposed model is components’ based, on the one hand, and ETAS-M-based, on the other. It’s
more convenient in comparison with purely components-based model because it is based only on
evaluation of UoLs suitability and use made by the users, and fully reflects their needs and points of
view. Additionally, this kind of model does not require specific high-level technological expertise from
experts-evaluators.
On the other hand, proposed model is better than pure ETAS-M / UTAUT-based model because it’s
more flexible since it takes into consideration corresponding suitability indexes of all learning
components of UoL, i.e. the proposed model reflects suitability of given UoL to particular students.
Personalised UoLs evaluation method was proposed by formula (3).

27. The examples have shown that TOPSIS method could also be successfully
used in formula (3) to identify the values of suitability, acceptance and use of
UoL.
Proposed methodology is feasible to be applied in real-life pedagogical
situations in educational institutions.
In order to easily create and evaluate personalised UoLs, educational
institutions should establish FSLSM-based students’ profiles, use high quality
vocabularies of learning components, and have enough expertise to identify
corresponding suitability indexes.

28. Thank you for your attention !!!
Questions ?
Eugenijus Kurilovas
Vilnius University
Vilnius Gediminas Technical University, Professor