The document presents a framework for facilitating feedback processes at scale in educational contexts through personalized support actions, emphasizing the importance of data-driven personalization to enhance student success. It discusses various methodologies and technologies utilized in implementing real-time feedback systems, including user engagement strategies and data analytics. The initiative aims to aid instructors in providing effective feedback to large and diverse student groups, fostering improved learning outcomes.

1. Facilitating feedback processes at scale
through personalised support actions
Abelardo Pardo (@abelardopardo)
Division of Information Technology, Engineering and the Environment
slideshare.net/abelardo_pardo
LauraLaRose
Open Apereo Conference
5 June 2018, Montreal Canada

2. Abelardo Pardo Facilitating feedback processes at scale 2
Pedagogically
sound
Technologically
Effective

3. Abelardo Pardo Facilitating feedback processes at scale 3
Help learners
succeed

4. Abelardo Pardo Facilitating feedback processes at scale 4
PaoloMazzoleni
Data
Learning
analytics

5. Abelardo Pardo Facilitating feedback processes at scale 5
The allure
of
dashboards
PNNLPacificNorthwest
• Information density
• Assumed expertise

6. Abelardo Pardo Facilitating feedback processes at scale 6
Thaler, R. H., & Sunstein, C. R. (2008). Nudge. Great Britain: Yale University Press.
“People make good choices in contexts in
which they have experience, good
information, and prompt feedback”
DerekBruffflickr.com

7. Abelardo Pardo Facilitating feedback processes at scale 7
Paul A. Kirschner, Mirjam Neelen, https://3starlearningexperiences.wordpress.com/2018/06/05/no-feedback-no-learning/

8. Abelardo Pardo Facilitating feedback processes at scale 8
Krause, K.-L., Hartley, R., James, R., & McInnis, C. (2005). The First Year Experience in Australian Universities: Findings from a
decade of National Studies. University of Melbourne: Centre for the Study of Higher Education.
Eleafflickr.com
The feedback question gets systematically
low values in student surveys

9. Abelardo Pardo Facilitating feedback processes at scale 9
Liu, D. Y.-T., Bartimote-Aufflick, K., Pardo, A., & Bridgeman, A. J. (2016). Data-driven Personalization of Student Learning Support in
Higher Education. In A. Peña-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends: A view of the current state of the
art. In preparation: Springer.
• Upload student listing
• Upload indicators per
student
• Write filters or rules
• Contact selected
students with
personalised email
Student Relationship
Engagement System (SRES)

10. Abelardo Pardo Facilitating feedback processes at scale 10
Report to students
Kahn, I., & Pardo, A. (2016). Data2U: Scalable Real time Student Feedback in Active Learning Environments. Paper presented at the
International Conference on Learning Analytics and Knowledge, Edinburgh, UK. doi:10.1145/2883851.2883911

11. Abelardo Pardo Facilitating feedback processes at scale 11
Calsidyrose
• Direct connection with
data
• Intuitive operations for
academics
• Immediate sense of
purpose
• Solving a clearly
identified pedagogical
problem
• Aware of wealth of data
sources
• LMS agnostic —
Ecosystem
• Support human and
machine integration
• Extensible with
upcoming AI results

12. Abelardo Pardo Facilitating feedback processes at scale 12
ontasklearning.org

13. Abelardo Pardo Facilitating feedback processes at scale 13
Data
Table
Data
Frame
Data frame
to SQL
CSV to
data frame
Data frame
operations
Pandas
Web
Application
Admin
site
Security
features
API
implementation
Easy to scale
to high load
MVP and
scalable
Solid
ORM
Django

14. Abelardo Pardo Facilitating feedback processes at scale 14
SimonClancy
Quick user tour
Three roles
• Administrator
• Instructor
• Learner

15. Abelardo Pardo Facilitating feedback processes at scale 15
• Users manage a set
of workflow
• Workflow is data +
actions
• Fully functional admin
pages
• Manage configuration
options, users, etc.
• System-wide
connections
• Allow generic data
import

16. Abelardo Pardo Facilitating feedback processes at scale 16
• Export workflow
• Share with other users
• Clone to replicate
• Global attributes
• Key (unique)
columns
• Automatically
identified
Automatic type
detection

17. Abelardo Pardo Facilitating feedback processes at scale 17
• Download CSV
file
• Allow “Views"

18. Abelardo Pardo Facilitating feedback processes at scale 18
Arbitrary subsets
of rows and
columns

19. Abelardo Pardo Facilitating feedback processes at scale 19

20. Abelardo Pardo Facilitating feedback processes at scale 20
• Personalised
content
• Data capture
Import actions,
promote sharing

21. Abelardo Pardo Facilitating feedback processes at scale 21
• Select a subset of
learners
• Arbitrary expression
on any column value

22. Abelardo Pardo Facilitating feedback processes at scale 22
Columns in
data table

23. Abelardo Pardo Facilitating feedback processes at scale 23
• Select a subset of
learners
• Arbitrary expression
on any column value
Conditions to
include/ignore text

24. Abelardo Pardo Facilitating feedback processes at scale 24

25. Abelardo Pardo Facilitating feedback processes at scale 25
• Select a subset of
learners
• Arbitrary expression
on any column value
Replace by
column value
Conditions to
include/ignore text
Text included only
if condition
No_Video_1 is truePreview the
content for each
learner

26. Abelardo Pardo Facilitating feedback processes at scale 26
Replace by
column value
Personalised
text

27. Abelardo Pardo Facilitating feedback processes at scale 27
• Send summary email to
instructor
• Track email reading
• Download/export
workflow snapshot

28. Abelardo Pardo Facilitating feedback processes at scale 28

29. Abelardo Pardo Facilitating feedback processes at scale 29
Include in LMS
Time window
availability
Offer content
through URL

30. Abelardo Pardo Facilitating feedback processes at scale 30
Data capture
actions

31. Abelardo Pardo Facilitating feedback processes at scale 31
Description
Subset of
learners
Questions

32. Abelardo Pardo Facilitating feedback processes at scale 32
New question
Allowed values

33. Abelardo Pardo Facilitating feedback processes at scale 33
New question

34. Abelardo Pardo Facilitating feedback processes at scale 34

35. Abelardo Pardo Facilitating feedback processes at scale 35
Instructor lead
data entry

36. Abelardo Pardo Facilitating feedback processes at scale 36
Select learner

37. Abelardo Pardo Facilitating feedback processes at scale 37
Enter data

38. Abelardo Pardo Facilitating feedback processes at scale 38
Schedule Email
Messages

39. Abelardo Pardo Facilitating feedback processes at scale 39

40. Abelardo Pardo Facilitating feedback processes at scale 40
Obtain data and
proceed to merge

41. Abelardo Pardo Facilitating feedback processes at scale 41
Match keys to
merge
Choose merge
version

42. Abelardo Pardo Facilitating feedback processes at scale 42
Basic
visualisation

43. Abelardo Pardo Facilitating feedback processes at scale 43

44. Abelardo Pardo Facilitating feedback processes at scale 44
Basic
visualisation

45. Abelardo Pardo Facilitating feedback processes at scale 45
Value with respect
to population

46. Abelardo Pardo Facilitating feedback processes at scale 46
TaymTaym
Quick technical tour

47. Abelardo Pardo Facilitating feedback processes at scale 47

48. Abelardo Pardo Facilitating feedback processes at scale 48
Calsidyrose
• Several authentication
options (SAML2)
• LTI authentication
• Connection to
corporate SMTP server
• User identification
through email address
• Execution of admin
scripts through Django
shell (ORM capable)
• PostgreSQL (tighter
integration with django)
• Plugins to apply
arbitrary
transformations
Deployment

49. Abelardo Pardo Facilitating feedback processes at scale 49
• DRF (Django Rest
Framework)
• Token
Authentication

50. Abelardo Pardo Facilitating feedback processes at scale 50
EdenBrackstone
Does it work?

51. Abelardo Pardo Facilitating feedback processes at scale 51
Helpful feedback
Effect size (Cohen’s d) = 0.49
Medium positive effect
Midterm Scores
Effect size (Cohen’s d) = 0.21
Small positive effect
Pardo, A., Jovanović, J., Dawson, S., Gašević, D., & Mirriahi, N. (In press). Using Learning Analytics to Scale the Provision of
Personalised Feedback. British Journal of Educational Technology. doi:10.1111/bjet.12592

52. Abelardo Pardo Facilitating feedback processes at scale 52
Focus groups
• “It helps me to validate where I am; do I need to
freak out right now?”
• “…gives you a nudge — Stop procrastinating and
playing games!”
• A reminder to study “across the board” (flow-on
effect)
• “The wording makes you want to do it. Like an
encouragement.”

53. Abelardo Pardo Facilitating feedback processes at scale 53
• Support instructors to
manage personalised
feedback processes
• Simple rule-base
knowledge encoding
• Provide appropriate
view of data sources
• Scale to large and highly
diverse cohorts
• Open-source project
• Pilots in 2018 finishing
• Demo servers available
ontasklearning.org

54. Abelardo Pardo Facilitating feedback processes at scale 54
Meet (a subset of) the team

55. Facilitating feedback processes at scale
through personalised support actions
Abelardo Pardo (@abelardopardo)
Division of Information Technology, Engineering and the Environment
slideshare.net/abelardo_pardo
LauraLaRose
Open Apereo Conference
5 June 2018, Montreal Canada