Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

LAK19 - Towards Value-Sensitive Learning Analytics Design

301 views

Published on

LAK19 Full Paper. Abstract: To support ethical considerations and system integrity in learning analytics, this paper introduces two cases of applying the Value Sensitive Design methodology to learning analytics design. The first study applied two methods of Value Sensitive Design, namely stakeholder analysis and value analysis, to a conceptual investigation of an existing learning analytics tool. This investigation uncovered a number of values and value tensions, leading to design trade-offs to be considered in future tool refinements. The second study holistically applied Value Sensitive Design to the design of a recommendation system for the Wikipedia WikiProjects. To proactively consider values among stakeholders, we derived a multi-stage design process that included literature analysis, empirical investigations, prototype development, community engagement, iterative testing and refinement, and continuous evaluation. By reporting on these two cases, this paper responds to a need of practical means to support ethical considerations and human values in learning analytics systems. These two cases demonstrate that Value Sensitive Design could be a viable approach for balancing a wide range of human values, which tend to encompass and surpass ethical issues, in learning analytics design.

Published in: Data & Analytics
  • DOWNLOAD FULL. BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

LAK19 - Towards Value-Sensitive Learning Analytics Design

  1. 1. Towards Value-Sensitive Learning Analytics Design Bodong Chen, Curriculum & Instruction Haiyi Zhu, Computer Science & Engineering LAK Conference | 2019-03-08 1 / 37
  2. 2. Ethical considerations in learning analytics (Slade & Prinsloo, 2013; Ferguson et al., 2016) 2 / 37
  3. 3. Ethical frameworks Codes of ethical practice Ethical consideration checklists Empirical work of stakeholders (Drachsler & Greller, 2016; Ifenthaler & Tracey, 2016; Sclater, 2016; Willis, Slade, & Prinsloo, 2016) 3 / 37
  4. 4. 2017 - The ACM Statement on Algorithmic Transparency and Accountability 2018– - The ACM FAT* Conference 2018 - #LAK18 & JLA Special Section on Human-Centered Learning Analytics 2019 - #LAK19 FairLAK workshop; papers on evaluating fairness (Photo Credit) Transparency, Accountability, Fairness 4 / 37
  5. 5. Learning Analytics System Integrity (Buckingham Shum, 2017, Black box learning analytics? Beyond algorithmic transparency) 5 / 37
  6. 6. Ethical Design Critique (EDC) (Buckingham Shum, 2018, Ethical Design Critique for Learning Analytics Dashboards) 6 / 37
  7. 7. Value-Sensitve Design Another toolkit for a very hard problem 7 / 37
  8. 8. "Something important, worthwhile, or useful;" "one's judgment of what is important in life" (Oxford English Dictionaries, 2018). "What a person or group of people consider important in life" (Borning & Muller 2012). "What is important to people in their lives, with a focus on ethics and morality" (Friedman, Hendry, & Borning, 2017) (Source: "It's about power", ACM Communications) Values (vs. Ethics) 8 / 37
  9. 9. Value Sensitive Design (VSD) is a methodology and an established set of methods for addressing values a pioneering endeavor to proactively consider human values throughout the process of technology design (Davis & Nathan. 2013) offers a systemic approach with speci c strategies and methods to explicitly incorporate the consideration of human values into design (Friedman et al., 2017) Value-Sensitve Design 9 / 37
  10. 10. Tripartite Methodology (Boston, 2016; Friedman et al., 2017) 10 / 37
  11. 11. Direct and Indirect Stakeholder Analysis Value Source Analysis Co-evolution of Technology and Social Structure Value Scenario Value Sketch Value-oriented Semi-structured Interview Value-oriented Coding Manual Value-oriented Mock-up, Prototype or Field Deployment Value "Dams" and "Flows" (Friedman et al., 2017, A Survey of Value Sensitive Design Methods) VSD Methods 11 / 37
  12. 12. Study 1: A conceptual investigation of a social learning analytics tool Study 2: A holistic application to the design of a recommendation system for WikiProjects 12 / 37
  13. 13. Social Media for Learning (Greenhow & Lewin, 2016; Haythornthwaite, Priya, et al., 2018) 13 / 37
  14. 14. Yellowdig (Photo Credit) 14 / 37
  15. 15. (Photo Credit) Provide another way of engaging with the class discussion, Enhance student learning and social interaction, and Assist the instructor to grasp discussion content in order to make informed instructional decisions Yellowdig Visualization Tool 15 / 37
  16. 16. Tool Features (demo) Multi-mode, interactive Temporal Lexical (instrutor only) 16 / 37
  17. 17. Methods 1. Stakeholder analysis 2. Value analysis 17 / 37
  18. 18. Direct Instructors Students Active In-active Indirect Future cohorts of the class Academic advisors 1. The Stakeholders 18 / 37
  19. 19. 2. Values Design Choices Instructors Active Students Inactive Students What to show Richer data Utility: navigating, sense­making Utility: assessing Freedom from bias Autonomy Privacy Usability: cognitive load (Bereiter, 2014, Principled Practical Knowledge) 19 / 37
  20. 20. 2. Values Design Choices Instructors Active Students Inactive Students How to show Node highlighting  (student or post) Utility: navigating, sense­making Utility: intervening Social interaction Self­image Self­image Privacy Force­directed layout Utility: intervening Sense of community Self­image Freedom from bias Content filter  (instructor only) Utility: sense­making Utility: sense­making Fair access 20 / 37
  21. 21. Design trade-offs Value tensions Analysis Representation Action utility <­­> autonomy; privacy; self­image     active <­­> inactive students make student nodes not highlightable provide the option of not revealing one's name freedom from bias   <­­> self­image consider other graph layout algorithms study student perceptions of layouts fair access   <­­> utility: sense­ making   <­­> privacy give student access to text mining features reserve the student filter to the instructor (Siemens, 2013) 21 / 37
  22. 22. Study 1: A conceptual investigation of a social learning analytics tool Study 2: A holistic application to the design of a recommendation system for WikiProjects 22 / 37
  23. 23. WikiProjects Self-organized groups of contributors within the English Wikipedia community. 23 / 37
  24. 24. Problem and Goal  WikiProjects have been declining Only 12% projects have 10+ active members  To build a recommendation system to help WikiProjects identify and recruit suitable new members 24 / 37
  25. 25. A Five-Stage Approach (Zhu et al., 2018, CSCW) 25 / 37
  26. 26. Stage 1. Understand Stakeholders 26 / 37
  27. 27. Stage 1. Understand Stakeholders A literature review + A survey study --> Stakeholders' values Design Choices New project comers Current project members Wikipedia community Who to recruit Experienced editors Collaboration Productivity Member retention Brand­new editors Mentorship Productivity New member retention How to define fit Interest­based Mutual interest Mutual interest Relation­based Personal connection Productivity How to decide Automation Control Human­in­the­ loop Control 27 / 37
  28. 28. Stage 2. Prototyping - "Who to Recruit" Design Choices New project comers Current project members Wikipedia community Who to recruit Experienced editors Collaboration Productivity Member retention Brand­new editors Mentorship Productivity New member retention Value Tension -> Design Choice Evaluateing both brand-new and experienced editors Ranking the two types of editors separately 28 / 37
  29. 29. Interest-based Rule-based Category-based Relationship-based Bond-based Co-edit-based Stage 2. Prototyping - "How to de ne t" Design Choices New project comers Current project members Wikipedia community How to define fit Interest­ based Mutual interest Mutual interest Relation­ based Personal connection Productivity Four algorithms (see Zhu et al., 2018, CSCW) 29 / 37
  30. 30. Stage 2. Prototyping - "How to decide" Design Choices New project comers Current project members Wikipedia community How to decide Automation Control Human­in­the­ loop Control Pitfall -> Design choice Design a dashboard to communicate recommendations to project members 30 / 37
  31. 31. Stage 2. Prototyping - "How to decide" Analytic results communicated to current WikiProjects members: 31 / 37
  32. 32. Stage 3-4. Deploy, Iterate and Re ne Over a six-month period: The analytic system analyzed information from 16,000 editors delivered 4 distinct batches of 385 recommendations to 18 Wikiprojects Re ned the system prototype based on stakeholder feedback 32 / 37
  33. 33. Stage 5: Evaluate Messages from current WikiProjects members "This puts some science behind recommendations, and will be a great supplement to the current processes. " Messages from invited new members "Thank you for reaching out to me and thank you for informing me about the WikiProject Africa talk page... I appreciate it. " (see Zhu et al., 2018, CSCW) 33 / 37
  34. 34. Stage 5: Evaluate Response from the Wikipedia community: a Signpost (news story) about the design 34 / 37
  35. 35. Summary Study 1: A conceptual investigation of a social learning analytics tool Identi ed stakeholders and their values Elicited value tensions Suggested design tradeoffs to address value tensions Study 2: A holistic application to the design of a recomendation system Developed a ve-stage approach that integrate multiple methods Designed with end-users Involved the user community in the design process Iterated over an extended period of time 35 / 37
  36. 36. Implications and Future Directions Value Sensitive Design provides a toolkit to: 1. guide the evaluation and re nement of existing learning analytics applications 2. help holistically addressing values in the design of new learning analytics Currently applying VSD to the design of the Personal Learning Compass tool ... 36 / 37
  37. 37. Thank You! Full Paper: https://doi.org/10.1145/3303772.3303798 Preprint: https://arxiv.org/abs/1812.08335  chenbd@umn.edu  @bod0ng  Personal website: http://meefen.github.io/  Research group: https://colig.github.io/ 37 / 37

×