Case study data to support workshop activities for Learner Enhanced Technology: Can activity analytics support understanding engagement a measurable process? Delivered 20th Jan at HETL SOTE 2015, Utah Valley University.

1. Case Study
Can activity analytics support understanding engagement a measurable process?
James Ballard – HETL SOTE 2015, Utah Valley University
jameslballard
JamesBallard2
@jameslballard

2. Background

3. Description
The data for this case study comes from the Virtual Learning Environment (Moodle) and e-
Portfolio (Mahara) of a Further Education college in the UK. The college curriculum is vocational
and employer led supporting around 18,000 students annually with an average age of 29. As a
community college the majority of learners live in the same borough while others travel from
adjoining boroughs.
Comparatively this is one of the most ethnically diverse and youngest populations in London
with census data reporting 47% of the population as being of Black or Minority Ethnicity (BME)
and 4 in 5 people under 50;
English is a second language for many students with over 170 mother tongue languages
spoken in the area;
Population turnover is high with 20% moving in and out of the borough annually;
It is in the top 10% of deprived areas of the UK.

4. Descriptive Statistics
MEASURE ALL STUDENTS
Total Students 7,647
Total Activity (logs) 1,466,907
Mean Activity (logs per learner) 192
Standard Deviation 348
Median 65
Mode 2
Min 0
Max 7,263
Data is taken from the 2013/14 Academic Year

5. Histogram
Summarised frequencies of logs per user
884
491
1014
1020
1168
1439
800
579
230
21
1
0
0 200 400 600 800 1000 1200 1400 1600
5
10
25
50
100
250
500
1000
2500
5000
7500
More

6. Activity 1 – Instrument
Formation
What might we measure?

7. Log data – Page 1/5
Date Course Tool Action
03/01/2014 10:01 Functional Skills - Construction course view
03/01/2014 10:02 Functional Skills - Construction forum view forum
03/01/2014 10:02 Functional Skills - Construction course view
03/01/2014 10:02 Functional Skills - Construction forum view forum
03/01/2014 10:03 CIH L3 Cert in Housing Practice
Programme
course view
03/01/2014 10:03 CIH L3 Cert in Housing Practice
Programme
course view section
03/01/2014 10:03 CIH L3 Cert in Housing Practice
Programme
forum view forum
03/01/2014 10:03 CIH L3 Cert in Housing Practice
Programme
forum view discussion
03/01/2014 10:04 CIH L3 Cert in Housing Practice
Programme
forum add post
03/01/2014 10:04 CIH L3 Cert in Housing Practice
Programme
forum view discussion
04/01/2014 01:18 CIH L3 Cert in Housing Practice
Programme
forum view forum

8. Log data – Page 2/5
Date Course Tool Action
04/01/2014 01:18 CIH L3 Cert in Housing Practice
Programme
forum view discussion
04/01/2014 01:18 CIH L3 Cert in Housing Practice
Programme
forum view forum
04/01/2014 01:28 CIH L3 Cert in Housing Practice
Programme
course view
04/01/2014 01:29 CIH L3 Cert in Housing Practice
Programme
course view section
04/01/2014 01:29 CIH L3 Cert in Housing Practice
Programme
course view section
04/01/2014 01:30 CIH L3 Cert in Housing Practice
Programme
resource view
04/01/2014 01:32 CIH L3 Cert in Housing Practice
Programme
folder view
04/01/2014 01:32 CIH L3 Cert in Housing Practice
Programme
course view section
04/01/2014 01:34 CIH L3 Cert in Housing Practice
Programme
resource view
04/01/2014 01:35 CIH L3 Cert in Housing Practice
Programme
user view all
04/01/2014 01:37 CIH L3 Cert in Housing Practice
Programme
user view

9. Log data – Page 3/5
Date Course Tool Action
04/01/2014 01:37 CIH L3 Cert in Housing Practice
Programme
user view
04/01/2014 01:38 CIH L3 Cert in Housing Practice
Programme
assign view
04/01/2014 01:39 CIH L3 Cert in Housing Practice
Programme
course view section
04/01/2014 01:39 CIH L3 Cert in Housing Practice
Programme
folder view
09/01/2014 08:10 CIH L3 Cert in Housing Practice
Programme
course view
09/01/2014 08:10 CIH L3 Cert in Housing Practice
Programme
user view all
09/01/2014 08:10 CIH L3 Cert in Housing Practice
Programme
forum view forums
09/01/2014 08:11 CIH L3 Cert in Housing Practice
Programme
forum view forum
09/01/2014 08:11 CIH L3 Cert in Housing Practice
Programme
forum view discussion
09/01/2014 08:51 CIH L3 Cert in Housing Practice
Programme
assign view
09/01/2014 08:52 CIH L3 Cert in Housing Practice
Programme
assign view submit assignment form

10. Log Data - Page 4/5
Date Course Tool Action
09/01/2014 08:53 CIH L3 Cert in Housing Practice
Programme
assign submit
09/01/2014 08:53 CIH L3 Cert in Housing Practice
Programme
assign view
09/01/2014 08:54 CIH L3 Cert in Housing Practice
Programme
assign view confirm submit assignment form
09/01/2014 08:54 CIH L3 Cert in Housing Practice
Programme
assign submit for grading
09/01/2014 08:54 CIH L3 Cert in Housing Practice
Programme
assign view
09/01/2014 12:39 CIH L3 Cert in Housing Practice
Programme
forum view forum
09/01/2014 12:39 CIH L3 Cert in Housing Practice
Programme
forum view discussion
09/01/2014 12:52 CIH L3 Cert in Housing Practice
Programme
forum add post
09/01/2014 12:52 CIH L3 Cert in Housing Practice
Programme
forum view discussion
09/01/2014 12:52 CIH L3 Cert in Housing Practice
Programme
course view
09/01/2014 12:53 CIH L3 Cert in Housing Practice
Programme
course view section

11. Log Data - Page 5/5
Date Course Tool Action
09/01/2014 12:53 CIH L3 Cert in Housing Practice
Programme
course view
09/01/2014 12:54 CIH L3 Cert in Housing Practice
Programme
course view section
09/01/2014 12:54 CIH L3 Cert in Housing Practice
Programme
course view
09/01/2014 12:54 CIH L3 Cert in Housing Practice
Programme
forum view forum
09/01/2014 12:54 CIH L3 Cert in Housing Practice
Programme
course view

12. Worksheet 1/2
TASK TYPE TALLY TOTAL
Assimilative: attending and understanding content
Information Handling: gathering and classifying
resource or manipulating data
Adaptive: use of modelling or simulation software
Communicative: dialogic activities (e.g. pair
dialogues or group based discussion)
Productive: construction of an artefact such as a
written essay or a sculpture
Experiential: practicing skills in a particular context
or undertaking an investigation

13. PIE CHART COLUMN CHART
Worksheet 2/2
0

14. Activity 2 – Practical
Application
Creating an Engagement Profile

15. CIH L3 Cert in Housing
Practice Programme
Total Activity Logs:
1,458
Student 1 – Jo March

16. Activity Over Time
17
236
156
118
222
265
143
101 97 103
Sep-13 Oct-13 Nov-13 Dec-13 Jan-14 Feb-14 Mar-14 Apr-14 May-14 Jun-14
Summarised count of the number of logs per month

17. Site Areas
Summary of the areas of the site that were accessed
15
210
139
107
173
218
119
85 84 89
2
3
1
1
41
1
2
2
23
16
11
44
43
22
16 13 14
SEP-13 OCT-13 NOV-13 DEC-13 JAN-14 FEB-14 MAR-14 APR-14 MAY-14 JUN-14
Portal Homepage
Learning Centre
Induction
Functional Skills - Construction
Cross College English
Construction Study Skills
CIH L3 Cert in Housing Practice Programme

18. Module Key
Affordances of the tools as a key for understanding the log summary table next
Module Description Module Description
Assignment Assessed assignment submission Label Inline course text
Book Chaptered web content Gallery Image gallery
Chat Synchronous discussion NLN Content Packaged content module
Course Course page Page Teacher created page
Folder Folder of teacher uploaded documents Resource Teacher uploaded file
Forum Asynchronous threaded discussion Web link URL to external site
IMS CP Packaged content module User User profile

19. Log Summary
Summarised table showing how the logs were distributed across specific LMS modules.
Assignment
Book
Chat
Course
Folder
Forum
IMSCP
Label
Gallery
NLN
Content
Page
Resource
Weblink
User
Total
add discussion 2 2
add post 68 68
submission statement accepted 1 1
submit 6 6
submit for grading 3 3
subscribe 7 7
subscribeall 1 1
talk 6 6
update post 3 3
view 71 1 91 83 1 59 3 3 312
view all 15 14 14 14 14 14 14 14 14 14 12 153
view confirm submit assignment form 3 3
view discussion 292 292
view forum 252 252
view forums 18 18
view section 112 112
view submit assignment form 15 15
Total 114 14 7 203 97 643 14 14 14 14 15 73 17 15 1254

20. Task Type Analysis – Page 1/3
Summary of the different task types the learner is involved in over time.
0
5
10
15
20
25
30
35
Oct-13 Nov-13 Dec-13 Jan-14 Feb-14 Mar-14 Apr-14 May-14 Jun-14
Productive
Communicative
Assimilative

21. Task Type Analysis – Page 2/3
Summary of the percentage of different task types the learner is involved in
Communicative
Assimilative
Productive
Information Handling
Adaptive
Experiential

22. Task Type Analysis – Page 3/3
Summary of the distribution of different task types the learner is involved in
0
50
100
150
200
250
300
350
400
Communicative
Assimilative
Productive
Information Handling
Adaptive
Experiential

23. Course Information – Page 1/2
Summary of the different task types per learner in the CIH L3 Cert in Housing Practice Programme.
0
200
400
600
800
1000
1200
1400
1600
29316 32603 31349 25331 34323 36027 34673 36231 Jo
March
13799 35727 35827 33758 33595 35469
Productive
Communicative
Assimilative

24. SOCIAL NETWORK ANALYSIS KEY
Each node represents a user in the course (blue
= learner and red = teacher). A connected arrow
indicates a forum post in reply to that user. The
weight of the arrow indicates the number of
replies.
Teachers:
29316
25331
Course Information – Page 2/2
Summary of the network analysis for the forum.

25. INVOLVEMENT INTERACTION INTIMACY INFLUENCE
Activities – list indicators of positive or negative engagement
Level of intensity – mark on the scale from low to high
Low High Low High Low High Low High
Worksheet
Student Profile 1: Jo March

26. IT Level 2 Diploma
student
Total Activity Logs:
4,958
Student 2 – Oliver Twist

27. Activity Over Time
Summarised count of the number of logs per month
32 104 12 0 0 89
4713
8 0 0
Sep-13 Oct-13 Nov-13 Dec-13 Jan-14 Feb-14 Mar-14 Apr-14 May-14 Jun-14

28. Site Areas
Summary of the areas of the site that were accessed
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Sep-13 Oct-13 Nov-13 Dec-13 Jan-14 Feb-14 Mar-14 Apr-14 May-14 Jun-14
Portal homepage
Mystery Shopping
Level 2 IT
Learning Centre
Get Involved
CG Award for IT Users (ITQ) E-QUALS

29. Module Key
Affordances of the tools as a key for understanding the log summary table next
Module Description Module Description
Assignment Assessed assignment submission Label Inline course text
Book Chaptered web content Gallery Image gallery
Chat Synchronous discussion NLN Content Packaged content module
Course Course page Page Teacher created page
Folder Folder of teacher uploaded documents Resource Teacher uploaded file
Forum Asynchronous threaded discussion Web link URL to external site
IMS CP Packaged content module User User profile

30. Log Summary
Summarised table showing how the logs were distributed across specific LMS modules.
Assignment
Course
Folder
Resource
Total
view 1 76 17 3328 3422
view section 1476 1476
view submit
assignment form
1 1
Total 2 1552 17 3328 4899

31. Task Type Analysis – Page 1/3
Summary of the different task types the learner is involved in over time.
0
500
1000
1500
2000
2500
3000
3500
Sep-13 Oct-13 Nov-13 Dec-13 Jan-14 Feb-14 Mar-14 Apr-14 May-14 Jun-14
Assimilative

32. Task Type Analysis – Page 2/3
Summary of the percentage of different task types the learner is involved in
Communicative
Assimilative
Productive
Information Handling
Adaptive
Experiential

33. Task Type Analysis – Page 3/3
Summary of the distribution of different task types the learner is involved in
0
500
1000
1500
2000
2500
3000
3500
Communicative
Assimilative
Productive
Information Handling
Adaptive
Experiential

34. Course Information – Page 1/2
Summary of the different task types per learner in the IT Level 2 Diploma student.
0
500
1000
1500
2000
2500
3000
3500
4000
OliverTwist
34568
3855
25364
11281
20395
33850
35759
13694
37361
35388
7253
39542
13203
34280
34210
3606
33335
5445
39501
13806
32380
6002
24326
34810
42742
33688
35824
40230
33965
16898
36361
3909
34023
4774
14153
41128
26311
Productive
Communicative
Assimilative

35. SOCIAL NETWORK ANALYSIS KEY
There are no discussions in this course so the
Social Network Analysis is not included.
Course Information – Page 2/2
Summary of the network analysis for the forum.

36. INVOLVEMENT INTERACTION INTIMACY INFLUENCE
Activities – list indicators of positive or negative engagement
Level of intensity – mark on the scale from low to high
Low High Low High Low High Low High
Worksheet
Student Profile 2: Oliver Twist

37. Learner Enhanced Technology
Understanding engagement as a
measurable process
Thank You
James Ballard
jameslballard@outlook.com
JamesBallard2
@jameslballard
jameslballard