Presentation of Prof.dr.sc. Diana Šimić at the Digital Skills Gap PLA (Peer Learning Activity) hosted by SRCE in Zagreb, Croatia

1. Faculty of
Organization and
Informatics

2. Faculty of
Organization and
Informatics
Pavlinska 2
42 000 Varaždin
Croatia
Analysis of usage of
LMS activities and
resources as an
aspect of e-course
quality
1962. - 2016.
Prof.dr.sc. Diana Šimić
diana.simic@foi.hr
Darko Grabar, mag.inf.
darko.grabar@foi.hr

3. UNIVERSITY OF ZAGREB
- UNIZG
The oldest and
the largest
university in
Croatia
1669.
Founded
29
Faculties
3
Art
academies
3
University
centres
• SRCE- IT Centre
• University & National Library
• Student Centre
80 000
Students
(all study levels)
9 000
Teaching staff
(all study levels)
1874.
Became
State university

4. FACULTY OF ORGANIZATION
AND INFORMATICS - FOI
• Medium sized faculty
• Leading higher
education institution in
Croatia to provide
education in applied
information
technology and
information sciences
1974.
Founded

5. FOI STUDENTS IN NUMBERS
OVER
3000
STUDENTS
MORE THAN
500
DEGREES
PER YEAR
FROM MORE THAN
100
SCHOOLS
FROM REGION

6. FOI E-Learning in numbers
OVER
300
STUDENTS
MORE THAN
500
DEGREES
PER YEAR
FROM MORE THAN
100
SCHOOLS
FROM REGION
OVER
300 online courses
EACH YEAR
MORE THAN
3000
ACTIVE ONLINE USERS

7. Main LMS - Moodle
7https://elf.foi.hr/
2006/2007

8. 8

9. Blended learning at FOI
• Three levels of usage of e-learning at FOI
• Level 1:
To enhance availability of teaching materials and
teacher to student communication
• Level 2:
To facilitate better acquisition of knowledge
through advanced integration of LMS system
with traditional classroom
• Level 3:
Improvement of teaching methods and
techniques of teaching through hybrid course
organization according to the instruction design
principles
9

10. Blended learning at FOI
• Three levels of usage of e-learning at FOI
• Level 1:
To enhance availability of teaching materials and
teacher to student communication
• General course information
• Learning outcomes
• Course plan and work program
• Literature
• Selected teaching materials
• Student to teacher communication by e-mail
• General discussion forum
10

11. Can quantitative analysis give us some
useful information
11

12. Can quantitative analysis give us some
useful information
12

13. What if we dig deeper into LMS
13

14. 14

15. 15
Typical FOI Moodle
database has around 390
tables containing more than
5 GB of data for each
academic year
~ 15 000 average books

16. What we can use for analysis
• Student and teacher overall activity
• Student usage of specific resources
• Number od specific resources, modules
used in course
• Number of forum posts, student discussions,..
• Student grades, points, badges, activity
completions, …
• …
16

17. Faculty of
Organization
and Informatics
Pavlinska 2
42 000 Varaždin
Croatia
Rich or sparse
structure?
1962. - 2016.

18. Goal
• To identify structurally / organizationally
poor e-courses
• To provide additional training and support to
authors of structurally poor e-courses
18

19. Questions …
• When is an e-course well structured (makes good
use of LMS)?
• … uses more types of resources and activities
• … use LMS for two-way communication with students
• … LMS is used for formative assessment and self-
assessment
• … various types of activities and resources are used …
19

20. How do we compare e-courses?
• Use AHP to combine multiple criteria for
deciding which e-courses have richer
structure
• Create a composite indicator and rank e-
courses
20

21. Composite indicators
• Used to compare countries, universities etc.
with multiple criteria
• Combine criteria into a single number
• Examples:
• Competitiveness index
• e-Readiness index
• ARWU … Academic Ranking of World Universities
21

22. Methodology
• OECD and JRC (2008) Handbook on
constructing composite indicators
methodology and user guide
https://composite-indicators.jrc.ec.europa.eu/
• Overview of methodology
• Checklist for building a composite indicator
(CI)
22

23. CI building workflow
1. Theoretical framework
2. Data selection
3. Imputation of missing data
4. Multivariate analyses
5. Normalization
6. Weighting and aggregation
7. Uncertainty and sensitivity analysis
8. Back to data
9. Links to other indicators
10. Visualization of results
23

24. 1. Theoretical framework
• What constitutes richness of e-course
structure?
• Various forms of content and activities
(educational content)
• Various types of assignments and assessments
(assessment)
• Rich communication with students
(communication)
• Use of LMS elements to create structure in
educational content / activities (structure)
• …. what else? … we will try to figure it out together
24

25. 2.-4. Next steps
• Selecting data
• Information routinely available from an LMS
system
• Imputation of missing data
• Not necessary in our case
• Multivariate analyses
• Descriptive statistics and visualization
• Principal components analysis
• Variable clustering
25

26. 5-7 Next steps
• Normalization
• Weighting and aggregation
• Uncertainty and sensitivity analysis
• we illustrate some of the steps …
26

27. Descriptive statistics
27
vars n mean sd skew kurtosis se Q0.25 Q0.75
assign 1 210 6.73 6.89 2.23 6.63 0.48 2.25 9.00
broj_aktivnosti_na_predmetu 2 210 7.38 2.59 0.59 0.88 0.18 6.00 9.00
broj_forum_diskusija_samo_vijesti 3 210 11.97 10.29 1.19 1.51 0.71 4.00 17.00
broj_grade_itema 4 210 13.68 13.89 2.65 9.15 0.96 5.00 18.00
broj_labela_na_predmetu 5 210 6.44 9.19 1.87 4.25 0.63 0.00 10.00
broj_nastavnika 6 210 3.07 1.53 1.07 1.64 0.11 2.00 4.00
Outliers
Skewed
distribution

28. Principal components
analysis (PCA)
28
Variable RC1 RC2 RC3 RC4 RC5
l.broj_testova_na_predmetu 0,93
l.quiz 0,92
l.broj_pitanja_na_predmetu 0,88
l.page 0,62
broj_aktivnosti_na_predmetu 0,56 0,37 0,38 0,44
l.broj_pracenih_aktivnosti_na_predmetu 0,36 -0,32
l.assign 0,83
broj_tipa_zadaca_novi 0,71
l.broj_grade_itema 0,53 0,70
l.tjedni_broj_zadaca 0,68
l.razlika_gradeitem_ocijenjeno 0,41 0,57 -0,30
l.broj_skrivenih_nastavnih_cjelina 0,57 -0,52
l.Omjer.labela.i.tema 0,94
l.broj_labela_na_predmetu 0,90
l.broj_vidljivih_nastavnih_cjelina 0,59
l.broj_resursa_na_predmetu 0,57
l.broj_forum_diskusija_samo_vijesti 0,56 0,48
l.choice 0,54
l.url 0,30 0,46
l.broj_stvarno_ocjenjenih_stavaka 0,37 0,40 0,65
broj_forum_postova_bez_vijesti_po_diskusiji 0,41
broj_forum_diskusija_bez_vijesti_po_studentu 0,36
Identify:
• direction of
maximal variability
• variables that
contribute most to
differentiational
• variables that vary
together
Reliability – Cronbach 𝛼
Sampling adequacy KMO

29. Variable clustering
29

30. Normalization
• Need to bring all variables to the same scale
before aggregation
• Look-out for outliers, very skewed data
• Some approaches:
• Ranking
• Standardization (z-score)
• Min-max (resulting in range 0 to 1)
• Distance to a reference course
• Transform to categorical variable using quantiles
30

31. Weighting
• Choosing weights
• From statistical models:
• Equal weights
• PCA / FA
• Data Envelopment Analysis (DEA)
• Participatory methods
• Budget Allocation Process (BPA)
• Analytical Hierarchical Process (AHP)
• Conjoint Analysis
31

32. At the end
• Aggregation
• Linear
• Geometric
• ….
• Sensitivity analysis
• Vary choices from each step to see how they
influence the final results
32

33. Back to the Theoretical
Framework
• How do we choose which variables / data
should enter into the CI, and what is the
structure?
• From literature
• From own experience
• From other experts … like you
33

34. Q-Sort
• Developed by Stephenson (1953) for personality research
• Often used in development of conceptual frameworks
• Experts provided with a set of statements / items
• Asked to provide measure of importance for each item
and dimension.
Stephenson, W (1953) The Study of Behavior. Chicago: University of Chicago Press.
34

35. Q-method
• Analysis of consistency of responses
• Analysis of typical profiles of attitudes
• Selection of relevant features for the
theoretical framework
35

36. Faculty of
Organization
and Informatics
Pavlinska 2
42 000 Varaždin
Croatia
Practical activities
1962. - 2016.

37. Qsort example
URL: http://courses.foi.hr/qsort
Password: 123456
37