Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Large-scale Learning Analytics at TU Delft

533 views

Published on

Meetup Data Science Northeast NL, October 13, 2016.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Large-scale Learning Analytics at TU Delft

  1. 1. Claudia Hauff Web Information Systems, TU Delft Large-scale Learning Analytics
  2. 2. It’s data that is on the Web … Web data … lets find the 
 Web Information Systems people! ✤ 40+ MOOCs ✤ 1+ Million enrollments ✤ From primary school to PhD level ✤ Lots of user data (click logs)
  3. 3. Our goals Data Knowledge Application to learning Gain actionable insights into learner behaviours at scale. a. Data Science b. Big data processing Increase our knowledge about learners by looking beyond the learning platform a. Web data analytics Design technology interventions that enable adaptive learning at scale. a. Web data analytics b. Human-centered design c. Learning technologies
  4. 4. Learner profiling beyond the MOOC platform ACM WebScience 2016 Guanliang Chen, Dan Davis, Jun Lin, Claudia Hauff, and Geert-Jan Houben. Beyond the MOOC platform: Gaining Insights about Learners from the Social Web, ACM WebScience, pp. 15-24, 2016.
  5. 5. Whythis research? Learner Before the MOOC NOTHING Engagement, retention, … During the MOOC NOTHING After the MOOC
  6. 6. Howto solve the problem? We propose: a deeper understanding about learners can be gained by exploring their traces on the Social Web.
  7. 7. Whatresearch questions? 1 On what Social Web platforms can a significant fraction of MOOC learners be identified? 
 Are learners who demonstrate specific traits on the Social Web drawn to certain types of MOOCs? 
2 To what extent do Social Web platforms enable us to observe (specific) user attributes that are relevant to the online learning experience? 
 3
  8. 8. Learner identification across Social Web platforms edX learners Email Login name Full name+ + 1. Explicit Matching Profile images & links Identification via emails 2. Direct Matching Identification via profile links from Step 1 3. Fuzzy Matching Search learners by their login & full names Compare: 1. profile link 2. profile image 3. login & full names
  9. 9. Social Web platforms involved in our work
  10. 10. Matching results for 18 DelftX MOOCs Lowest Highest Overall Gravatar 4.37% 23.49% 7.81% Twitter 4.99% 17.58% 7.78% Linkedin 3.90% 11.05% 5.89% StackExchange 1.23% 21.91% 4.58% GitHub 3.43% 41.93% 10.92%
  11. 11. Matching results for 18 DelftX MOOCs Lowest Highest Overall Gravatar 4.37% 23.49% 7.81% Twitter 4.99% 17.58% 7.78% Linkedin 3.90% 11.05% 5.89% StackExchange 1.23% 21.91% 4.58% GitHub 3.43% 41.93% 10.92% On average, 5% of learners can be identified on globally popular Social Web platforms. 

  12. 12. Learners on Linkedin - Using job titles & skills to characterise learners Spreadsheet MOOC - Software Engineer - Business Analyst - … Design Approach MOOC - Co founder - UX designer - …
  13. 13. Learners on StackExchange - Functional Programming learners in StackOverflow - To what extent do learners change their question/answering behaviour during and after a MOOC?
  14. 14. Take-home Messages On average, 5% of learners from 18 DelftX MOOCs can be identified on 5 globally popular Social Web platforms. 
1 Learners with specific traits prefer different types of MOOCs.2 Learners’ post-course behaviour can be investigated by using their external Social Web traces.3
  15. 15. Learning Transfer: does it take place? Best Paper Nominee at ACM Learning At Scale 2016 An Investigation into the Uptake of Functional Programming in Practice Guanliang Chen, Dan Davis, Claudia Hauff and Geert-Jan Houben, Learning Transfer: does it take place in MOOCs?, ACM Learning At Scale, pp. 409-418, 2016.
  16. 16. Whatis learning transfer? Learning transfer is the application 
 of knowledge or skills gained in a learning environment to another context.
  17. 17. Whydo we care? Learning transfer is a more important measure of learning in MOOCs than retention, success or engagement.
  18. 18. FP101x @flickr:christiaan_008 Course programming language: Haskell Run as a typical video-lecture based MOOC Assessment: 288 Multiple Choice questions Introduction to Functional Programming 37,485 learners registered. 41% engaged with the course. 5% completed the course. 33% were active on GitHub (1.1M events).
  19. 19. Whatdid we do? FP101x logs surveys coding activities 3 months 2.5 years + 0.5 years + + email address Are changes made in a functional language?
  20. 20. GitHub 10+ million registered users hosting, collaboration and organisation the most popular social coding platform founded in 2007 long-term large-scale detailed
  21. 21. detailed logs code changes project meta-data
  22. 22. A sanity check
  23. 23. Are “GitHub learners” different? GitHub learners Non-GitHub learners #Learners 12,415 25,070 Completion rate 7.71% 4.03% Avg. time watching videos 49.1 min 27.7 min Avg. #questions attempted 31.3 17.5 Avg. accuracy of learners’ answers 23.4% 12.9% GitHub learners are more engaged than non-GitHub learners and exhibit higher levels of knowledge.
  24. 24. Are “Expert learners” different? Expert GitHub learners Novice GitHub learners #Learners 1,721 10,694 Completion rate 15% 6.5% Avg. time watching videos 78.6 min 44.4 min Avg. #questions attempted 57.9 27.0 Avg. accuracy of learners’ answers 38.0% 21.1% Expert learners are more engaged than Novice learners and exhibit higher levels of knowledge.
  25. 25. To what extent do engaged learners exhibit learning transfer? 5-10% >30%10-30%<5%
  26. 26. To what extent do engaged learners exhibit learning transfer? 5-10%
  27. 27. Which type of learner is more likely to display learning transfer? flickr@ConalGallagher Intrinsically motivated Extrinsically motivated
  28. 28. Which type of learner is more likely to display learning transfer? flickr@ConalGallagher Intrinsically motivated
  29. 29. Which type of learner is more likely to display learning transfer? Experienced Inexperienced
  30. 30. Which type of learner is more likely to display learning transfer? Experienced
  31. 31. Which type of learner is more likely to display learning transfer? High-spacing
 learning routine Low-spacing
 learning routine
  32. 32. Which type of learner is more likely to display learning transfer? High-spacing
 learning routine
  33. 33. Learners who transfer quickly move to Scala FP101x
  34. 34. Conclusions Most transfer learning findings from the classroom hold. The observed transfer rate is low: 8.5%. Learners quickly moved on after the course to industrially-relevant functional languages. @flickr:torsten-reuschling
  35. 35. From Learners to Earners: Enabling MOOC Learners to Apply their Skills and Earn Money in an Online Market Place IEEE Transactions on Learning Technologies Guanliang Chen, Dan Davis, Markus Krause, Efthimia Aivaloglou, Claudia Hauff and Geert-Jan Houben. Can Learners be Earners? Investigating a Design to Enable MOOC Learners to Apply their Skills and Earn Money in an Online Market Place, IEEE Transactions on Learning Technologies.
  36. 36. What MOOCs aim to educate the world. Most successful learners are already highly educated. Learners from developing countries 
 are underrepresented. is the problem?
  37. 37. Whatis the problem? EX101x: Data Analysis to the MAX()
  38. 38. How Pay learner at scale: recommend tasks from online market places to learners that are relevant to the course material. can we tackle it?
  39. 39. Howcan we tackle it?
  40. 40. What 1) To what extent do online market 
 places contain relevant tasks? 2) Are learners able to solve 
 real-world tasks with high quality? do we need to look at?
  41. 41. Setup 1) Weekly spreadsheet “bonus 
 exercises” drawn from Upwork 
 (manually checked) in EX101x 2) Accuracy check 3) Quality check (code smells)
  42. 42. Howare learners doing? Good accuracy & quality.
  43. 43. Built a working recommender. Deployed in
 a MOOC by the
 end of October.
  44. 44. Our goals one more time… Data Knowledge Application to learning Gain actionable insights into learner behaviours at scale. Increase our knowledge about learners by looking beyond the learning platform Design technology interventions that enable adaptive learning at scale.
  45. 45. MOOCs are vital to bring higher education to the world. Lots of unexplored potential. Plenty of data. Many users. http://bit.ly/lambda-lab Overall …

×