Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Towards Value-Sensitive
Learning Analytics Design
Bodong Chen, Curriculum & Instruction
Haiyi Zhu, Computer Science & Engi...
Ethical considerations
in learning analytics
(Slade & Prinsloo, 2013; Ferguson et al., 2016)
2 / 37
Ethical frameworks
Codes of ethical practice
Ethical consideration
checklists
Empirical work of
stakeholders
(Drachsler & ...
2017 - The ACM Statement on Algorithmic
Transparency and Accountability
2018– - The ACM FAT* Conference
2018 - #LAK18 & JL...
Learning Analytics System Integrity
(Buckingham Shum, 2017, Black box learning analytics? Beyond algorithmic transparency)...
Ethical Design Critique (EDC)
(Buckingham Shum, 2018, Ethical Design Critique for Learning Analytics Dashboards)
6 / 37
Value-Sensitve Design
Another toolkit for a very hard problem
7 / 37
"Something important, worthwhile, or
useful;" "one's judgment of what is
important in life" (Oxford English
Dictionaries, ...
Value Sensitive Design (VSD) is a
methodology and an established set of
methods for addressing values
a pioneering endeavo...
Tripartite Methodology
(Boston, 2016; Friedman et al., 2017)
10 / 37
Direct and Indirect Stakeholder
Analysis
Value Source Analysis
Co-evolution of Technology and Social
Structure
Value Scena...
Study 1: A conceptual
investigation
of a social learning analytics tool
Study 2: A holistic
application
to the design of a...
Social Media for Learning
(Greenhow & Lewin, 2016; Haythornthwaite, Priya, et al., 2018)
13 / 37
Yellowdig
(Photo Credit)
14 / 37
(Photo Credit)
Provide another way of engaging with
the class discussion,
Enhance student learning and social
interaction,...
Tool Features (demo)
Multi-mode, interactive
Temporal
Lexical (instrutor only)
16 / 37
Methods
1. Stakeholder analysis
2. Value analysis
17 / 37
Direct
Instructors
Students
Active
In-active
Indirect
Future cohorts of the class
Academic advisors
1. The Stakeholders
18...
2. Values
Design Choices Instructors Active Students Inactive Students
What to show Richer data
Utility: navigating, sense...
2. Values
Design Choices Instructors Active Students Inactive Students
How to show
Node highlighting 
(student or post)
Ut...
Design trade-offs
Value tensions Analysis Representation Action
utility <­­> autonomy;
privacy; self­image  
 
active <­­>...
Study 1: A conceptual
investigation
of a social learning analytics tool
Study 2: A holistic
application
to the design of a...
WikiProjects
Self-organized groups of contributors within the English Wikipedia community.
23 / 37
Problem and Goal
 WikiProjects have been declining
Only 12% projects have 10+ active members
 To build a recommendation ...
A Five-Stage Approach
(Zhu et al., 2018, CSCW) 25 / 37
Stage 1. Understand Stakeholders
26 / 37
Stage 1. Understand Stakeholders
A literature review + A survey study --> Stakeholders' values
Design Choices
New project
...
Stage 2. Prototyping - "Who to Recruit"
Design Choices
New project
comers
Current project
members
Wikipedia
community
Who ...
Interest-based
Rule-based
Category-based
Relationship-based
Bond-based
Co-edit-based
Stage 2. Prototyping - "How to de ne ...
Stage 2. Prototyping - "How to decide"
Design Choices
New project
comers
Current project
members
Wikipedia
community
How t...
Stage 2. Prototyping - "How to decide"
Analytic results communicated to current WikiProjects members:
31 / 37
Stage 3-4. Deploy, Iterate and Re ne
Over a six-month period:
The analytic system
analyzed information from 16,000 editors...
Stage 5: Evaluate
Messages from current WikiProjects members
"This puts some science behind recommendations, and will be a...
Stage 5: Evaluate
Response from the Wikipedia community: a Signpost (news story) about the design
34 / 37
Summary
Study 1: A conceptual investigation of a social
learning analytics tool
Identi ed stakeholders and their values
El...
Implications and Future Directions
Value Sensitive Design provides a toolkit to:
1. guide the evaluation and re nement of ...
Thank You!
Full Paper: https://doi.org/10.1145/3303772.3303798
Preprint: https://arxiv.org/abs/1812.08335
 chenbd@umn.edu...
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0

Share

LAK19 - Towards Value-Sensitive Learning Analytics Design

Download to read offline

LAK19 Full Paper. Abstract: To support ethical considerations and system integrity in learning analytics, this paper introduces two cases of applying the Value Sensitive Design methodology to learning analytics design. The first study applied two methods of Value Sensitive Design, namely stakeholder analysis and value analysis, to a conceptual investigation of an existing learning analytics tool. This investigation uncovered a number of values and value tensions, leading to design trade-offs to be considered in future tool refinements. The second study holistically applied Value Sensitive Design to the design of a recommendation system for the Wikipedia WikiProjects. To proactively consider values among stakeholders, we derived a multi-stage design process that included literature analysis, empirical investigations, prototype development, community engagement, iterative testing and refinement, and continuous evaluation. By reporting on these two cases, this paper responds to a need of practical means to support ethical considerations and human values in learning analytics systems. These two cases demonstrate that Value Sensitive Design could be a viable approach for balancing a wide range of human values, which tend to encompass and surpass ethical issues, in learning analytics design.

  • Be the first to like this

LAK19 - Towards Value-Sensitive Learning Analytics Design

  1. 1. Towards Value-Sensitive Learning Analytics Design Bodong Chen, Curriculum & Instruction Haiyi Zhu, Computer Science & Engineering LAK Conference | 2019-03-08 1 / 37
  2. 2. Ethical considerations in learning analytics (Slade & Prinsloo, 2013; Ferguson et al., 2016) 2 / 37
  3. 3. Ethical frameworks Codes of ethical practice Ethical consideration checklists Empirical work of stakeholders (Drachsler & Greller, 2016; Ifenthaler & Tracey, 2016; Sclater, 2016; Willis, Slade, & Prinsloo, 2016) 3 / 37
  4. 4. 2017 - The ACM Statement on Algorithmic Transparency and Accountability 2018– - The ACM FAT* Conference 2018 - #LAK18 & JLA Special Section on Human-Centered Learning Analytics 2019 - #LAK19 FairLAK workshop; papers on evaluating fairness (Photo Credit) Transparency, Accountability, Fairness 4 / 37
  5. 5. Learning Analytics System Integrity (Buckingham Shum, 2017, Black box learning analytics? Beyond algorithmic transparency) 5 / 37
  6. 6. Ethical Design Critique (EDC) (Buckingham Shum, 2018, Ethical Design Critique for Learning Analytics Dashboards) 6 / 37
  7. 7. Value-Sensitve Design Another toolkit for a very hard problem 7 / 37
  8. 8. "Something important, worthwhile, or useful;" "one's judgment of what is important in life" (Oxford English Dictionaries, 2018). "What a person or group of people consider important in life" (Borning & Muller 2012). "What is important to people in their lives, with a focus on ethics and morality" (Friedman, Hendry, & Borning, 2017) (Source: "It's about power", ACM Communications) Values (vs. Ethics) 8 / 37
  9. 9. Value Sensitive Design (VSD) is a methodology and an established set of methods for addressing values a pioneering endeavor to proactively consider human values throughout the process of technology design (Davis & Nathan. 2013) offers a systemic approach with speci c strategies and methods to explicitly incorporate the consideration of human values into design (Friedman et al., 2017) Value-Sensitve Design 9 / 37
  10. 10. Tripartite Methodology (Boston, 2016; Friedman et al., 2017) 10 / 37
  11. 11. Direct and Indirect Stakeholder Analysis Value Source Analysis Co-evolution of Technology and Social Structure Value Scenario Value Sketch Value-oriented Semi-structured Interview Value-oriented Coding Manual Value-oriented Mock-up, Prototype or Field Deployment Value "Dams" and "Flows" (Friedman et al., 2017, A Survey of Value Sensitive Design Methods) VSD Methods 11 / 37
  12. 12. Study 1: A conceptual investigation of a social learning analytics tool Study 2: A holistic application to the design of a recommendation system for WikiProjects 12 / 37
  13. 13. Social Media for Learning (Greenhow & Lewin, 2016; Haythornthwaite, Priya, et al., 2018) 13 / 37
  14. 14. Yellowdig (Photo Credit) 14 / 37
  15. 15. (Photo Credit) Provide another way of engaging with the class discussion, Enhance student learning and social interaction, and Assist the instructor to grasp discussion content in order to make informed instructional decisions Yellowdig Visualization Tool 15 / 37
  16. 16. Tool Features (demo) Multi-mode, interactive Temporal Lexical (instrutor only) 16 / 37
  17. 17. Methods 1. Stakeholder analysis 2. Value analysis 17 / 37
  18. 18. Direct Instructors Students Active In-active Indirect Future cohorts of the class Academic advisors 1. The Stakeholders 18 / 37
  19. 19. 2. Values Design Choices Instructors Active Students Inactive Students What to show Richer data Utility: navigating, sense­making Utility: assessing Freedom from bias Autonomy Privacy Usability: cognitive load (Bereiter, 2014, Principled Practical Knowledge) 19 / 37
  20. 20. 2. Values Design Choices Instructors Active Students Inactive Students How to show Node highlighting  (student or post) Utility: navigating, sense­making Utility: intervening Social interaction Self­image Self­image Privacy Force­directed layout Utility: intervening Sense of community Self­image Freedom from bias Content filter  (instructor only) Utility: sense­making Utility: sense­making Fair access 20 / 37
  21. 21. Design trade-offs Value tensions Analysis Representation Action utility <­­> autonomy; privacy; self­image     active <­­> inactive students make student nodes not highlightable provide the option of not revealing one's name freedom from bias   <­­> self­image consider other graph layout algorithms study student perceptions of layouts fair access   <­­> utility: sense­ making   <­­> privacy give student access to text mining features reserve the student filter to the instructor (Siemens, 2013) 21 / 37
  22. 22. Study 1: A conceptual investigation of a social learning analytics tool Study 2: A holistic application to the design of a recommendation system for WikiProjects 22 / 37
  23. 23. WikiProjects Self-organized groups of contributors within the English Wikipedia community. 23 / 37
  24. 24. Problem and Goal  WikiProjects have been declining Only 12% projects have 10+ active members  To build a recommendation system to help WikiProjects identify and recruit suitable new members 24 / 37
  25. 25. A Five-Stage Approach (Zhu et al., 2018, CSCW) 25 / 37
  26. 26. Stage 1. Understand Stakeholders 26 / 37
  27. 27. Stage 1. Understand Stakeholders A literature review + A survey study --> Stakeholders' values Design Choices New project comers Current project members Wikipedia community Who to recruit Experienced editors Collaboration Productivity Member retention Brand­new editors Mentorship Productivity New member retention How to define fit Interest­based Mutual interest Mutual interest Relation­based Personal connection Productivity How to decide Automation Control Human­in­the­ loop Control 27 / 37
  28. 28. Stage 2. Prototyping - "Who to Recruit" Design Choices New project comers Current project members Wikipedia community Who to recruit Experienced editors Collaboration Productivity Member retention Brand­new editors Mentorship Productivity New member retention Value Tension -> Design Choice Evaluateing both brand-new and experienced editors Ranking the two types of editors separately 28 / 37
  29. 29. Interest-based Rule-based Category-based Relationship-based Bond-based Co-edit-based Stage 2. Prototyping - "How to de ne t" Design Choices New project comers Current project members Wikipedia community How to define fit Interest­ based Mutual interest Mutual interest Relation­ based Personal connection Productivity Four algorithms (see Zhu et al., 2018, CSCW) 29 / 37
  30. 30. Stage 2. Prototyping - "How to decide" Design Choices New project comers Current project members Wikipedia community How to decide Automation Control Human­in­the­ loop Control Pitfall -> Design choice Design a dashboard to communicate recommendations to project members 30 / 37
  31. 31. Stage 2. Prototyping - "How to decide" Analytic results communicated to current WikiProjects members: 31 / 37
  32. 32. Stage 3-4. Deploy, Iterate and Re ne Over a six-month period: The analytic system analyzed information from 16,000 editors delivered 4 distinct batches of 385 recommendations to 18 Wikiprojects Re ned the system prototype based on stakeholder feedback 32 / 37
  33. 33. Stage 5: Evaluate Messages from current WikiProjects members "This puts some science behind recommendations, and will be a great supplement to the current processes. " Messages from invited new members "Thank you for reaching out to me and thank you for informing me about the WikiProject Africa talk page... I appreciate it. " (see Zhu et al., 2018, CSCW) 33 / 37
  34. 34. Stage 5: Evaluate Response from the Wikipedia community: a Signpost (news story) about the design 34 / 37
  35. 35. Summary Study 1: A conceptual investigation of a social learning analytics tool Identi ed stakeholders and their values Elicited value tensions Suggested design tradeoffs to address value tensions Study 2: A holistic application to the design of a recomendation system Developed a ve-stage approach that integrate multiple methods Designed with end-users Involved the user community in the design process Iterated over an extended period of time 35 / 37
  36. 36. Implications and Future Directions Value Sensitive Design provides a toolkit to: 1. guide the evaluation and re nement of existing learning analytics applications 2. help holistically addressing values in the design of new learning analytics Currently applying VSD to the design of the Personal Learning Compass tool ... 36 / 37
  37. 37. Thank You! Full Paper: https://doi.org/10.1145/3303772.3303798 Preprint: https://arxiv.org/abs/1812.08335  chenbd@umn.edu  @bod0ng  Personal website: http://meefen.github.io/  Research group: https://colig.github.io/ 37 / 37

LAK19 Full Paper. Abstract: To support ethical considerations and system integrity in learning analytics, this paper introduces two cases of applying the Value Sensitive Design methodology to learning analytics design. The first study applied two methods of Value Sensitive Design, namely stakeholder analysis and value analysis, to a conceptual investigation of an existing learning analytics tool. This investigation uncovered a number of values and value tensions, leading to design trade-offs to be considered in future tool refinements. The second study holistically applied Value Sensitive Design to the design of a recommendation system for the Wikipedia WikiProjects. To proactively consider values among stakeholders, we derived a multi-stage design process that included literature analysis, empirical investigations, prototype development, community engagement, iterative testing and refinement, and continuous evaluation. By reporting on these two cases, this paper responds to a need of practical means to support ethical considerations and human values in learning analytics systems. These two cases demonstrate that Value Sensitive Design could be a viable approach for balancing a wide range of human values, which tend to encompass and surpass ethical issues, in learning analytics design.

Views

Total views

837

On Slideshare

0

From embeds

0

Number of embeds

270

Actions

Downloads

2

Shares

0

Comments

0

Likes

0

×