Supporting Social Deliberative Skills-StudiesDashboardTextanalysis-Murray
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Supporting Social Deliberative Skills-StudiesDashboardTextanalysis-Murray

on

  • 311 views

2013 EEE and HICC conferences--3 presentations

2013 EEE and HICC conferences--3 presentations
See socialdeliberativeskills.com

Statistics

Views

Total Views
311
Views on SlideShare
311
Embed Views
0

Actions

Likes
0
Downloads
2
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Differences in: Goals, values, needs; Background/worldview/identity; Beliefs, ideas, positions; Power, roles
  • https://docs.google.com/document/d/1nb_OsOqpQMPZbov4Uo1PWwqbFyJoW5jz_1hyiIYobug/edit
  • 8 males and 14 females ranging in undergraduate grade level from sophomores to seniors, with one non-degree studen
  • Students who posted fewer than 5 times for both topics combined are excluded ;One student failed to follow instructions (did not use the sliders). This student dominated the discussion, contributing over a third of the total posts. This student’s posts were longer than average, constituting 41% of the total length of the conversation of this group, as gauged by the total number of characters typed. Two other students in this group did not post enough to be included in the analysis. One student wrote a note to the facilitator claiming that one student in this group seemed overly critical and not respectful, which affected her feeling of safety. The tension here may have put a damper on the entire group
  • (not surprising since INTERSUB was strongly correlated with Total SD Skill)
  • https://docs.google.com/document/d/1nb_OsOqpQMPZbov4Uo1PWwqbFyJoW5jz_1hyiIYobug/edit
  • (not surprising since INTERSUB was strongly correlated with Total SD Skill)
  • 8 males and 14 females ranging in undergraduate grade level from sophomores to seniors, with one non-degree studen

Supporting Social Deliberative Skills-StudiesDashboardTextanalysis-Murray Presentation Transcript

  • 1. 1. Supporting Social Deliberative Skills Online: the Effects of Reflective Scaffolding Tools 2. A Prototype Facilitators Dashboard: Assessing and visualizing dialogue quality in online deliberation for education and work 3. Text Analysis of Deliberative Skills in Undergraduate Online Dialogue: Using L1 Regularized Logistic Regression with Psycholinguistic Features Tom Murray, Xiaoxi Xu, Beverly Woolf, Leah Wing, Lynn Stephens, Natasha Shrikant, Lori Clarke, Lee Osterweil EEE 2013 & HICC 2013 Supporting Social Deliberative Skills in Online Contexts
  • 2. Project Overview 2
  • 3. Overview 1. Background–Social Deliberative Skills 2. Classroom Studies 3. Facilitator Dashboard 4. Automated Text Analysis/Classification 5. Conclusions
  • 4. 1. Background on Social Deliberative Skills
  • 5. Group & Collaborative Learning/Work Conflict Resolution Meaning Negotiation Problem solving Planning Brainstorming Inquiry Decision making Knowledge building Group dynamics (form, storm, norm) Peer help/tutoring
  • 6. Meaning Negotiation/ Conflict Resolution Scope: Support the skills needed to bridge different perspectives to build mutual understanding and mutual regard
  • 7. Educational Priorities • King & Baxter (2005) note that “In times of increased global interdependence, producing interculturally competent citizens who can engage in informed, ethical decision-making when confronted with problems that involve a diversity of perspectives is becoming an urgent educational priority [however, these skills] are what corporations find in shortest supply among entry-level candidates"
  • 8. Areas of Application: Dialog/Deliberation Dispute/Conflict Resolution • Civic engagement/public dialogue • International & inter-group conflict • Labor/management, consumer disputes alternative dispute resolution • Interpersonal disputes / mediation • Deliberative decision making (school, work, home)
  • 9. Social Deliberative Skills: Social/Emotional/Reflective • 1. Social perspective taking (cognitive empathy, reciprocal role taking...) • 2. Social perspective seeking (social inquiry, question asking skills...) • 3. Social perspective monitoring (self-reflection, meta-dialogue...) • 4. Social perspective weighing (reflective reasoning; comparing and contrasting views...) 9
  • 10. Text Coding Scheme 10
  • 11. Social Deliberative Skill: application of HOSs to me/you/we Higher Order Skills • argumentation • critical thinking • explanation & clarification • inquiry/curiosity (question asking & investigation) • reflective judgment • meta-cognition • epistemic reasoning Apply these skills, not to EXTERNAL REALITY (“IT”/problem domain) but to the INTERSUBJECTIVE domain Higher Order Skills applied to: SELF goals; level of certainty; feelings, values, assumptions… YOU goals, assumptions, feelings, values; perspective taking; "believing" & cognitive empathy… WE agreements, goals; quality of the discourse/collaboration; differences and similarities in values, beliefs, goals, power, roles…
  • 12. Corpora and Rater Agreement
  • 13. Examples of Social Deliberative Skills/Behavior From authentic dialogues in our online corpora “ I am probably extremely biased because I am under 21 years old and in college. I wonder if as a 45 year old I will feel differently. ” (self reflection) “I can’t help but imagine what that is like, for her and for her family.” (perspective taking) 13
  • 14. Support/Scaffolding (vs. “Education”) Online Dialogue & DELIBERATION Outcomes: - Agreements/solutions - Relationship, Trust (social capital) - SKILL USE (and practice) Existing Skills Adaptive Support (4th party) Passive Support (interface) Facilitator Support (Dashboard)
  • 15. 2. Classroom Studies
  • 16. [CURRENT] WEEK 1: Discuss the pros and cons of leg... UPDATE PROFILE LOG OUT HOME Logged in as tomm [CURRENT] WEEK 1: Discuss the pros and cons of legalizing marijuana.[CURRENT] WEEK 1: Discuss the pros and cons of legalizing marijuana. To focus the conversation, we invite you to assume you are on an advisory panel for the state legislature, having some preliminary conversations online, and you will eventually be drafting a group recommendation. Consider not only your own preferences but what is best for the state (or society). edit delete CONTRIBUTE YOUR THOUGHTS 14:53 EDT Sunday, November 13 by tomm tomm has joined the conversation 23:53 EDT Saturday, November 12 by ines- v ines-v added a resource: 'Getting a Fix' 23:52 EDT Saturday, November 12 by ines- v I have to disagree with your third point that marijuana is a gateway drug. Of all the people I know that smoke marijuana, they do not do any hard drugs. I do agree that gateway drugs exist, however I feel like that typically happens from one hard drug to another when one doesn't seem to be enough. But if you want to talk about gateway drugs we would also have to mention alcohol and cigarettes which many people consume and smoke. Alcohol and cigarettes are also drugs and often considered gateway drugs. They are both legal so that option is void in regards to marijuana. You also mentioned cancer and other lung related issues. Marijuana is a natural plant. Cigarettes are made up of extremely harmful chemicals that cause lung related issues and cancer much faster than marijuana ever could. Yet, they are still legal. If anything, cigarettes should be illegal when considering public health. Marijuana is a lot safer than cigarettes. I do appreciate you playing Devil's advocate though! I'd like to explain how I see it differently (ines-v) 18:26 EDT Friday, November 11 by arthur- x It seems like the vast majority is supportive of the legalization of marijuana, so I'm going to play devil's advocate in order to bring the opposition's side to the table. First off, research has demonstrated that marijuana use reduces learning ability by limiting the capacity to absorb and retain information. A 1995 study of college students discovered that the inability of heavy marijuana users to focus, sustain attention, and organize data persists for as long as 24 hours after their last use of the drug. Earlier research, comparing cognitive abilities of adult marijuana users with non-using adults, found that users fall short on memory as well as math and verbal skills. Although it has yet to be proven conclusively that heavy marijuana use can cause irreversible loss of intellectual capacity, animal studies have shown marijuana-induced ines-v arthur-x joseph-t laura-t rtwells matthew-s tomm DIALOGUE TABLE Everyone (no demographics set) 16
  • 17. 17 Mediem Opinion Sliders
  • 18. Study: Classroom Dialog • 26 College Students in 2 week online discussion • 2 Topics: Trayvon Martin Shooting & Gun Control • 3 Experimental conditions/ 3 discussion groups • 829 text segments from 369 posts • 43% of the segments coded as "deliberate skill”
  • 19. Experimental Conditions Exp Group N Gender Grade Vanilla 8 (5 Female, 3 Male) 4 soph, 4 juniors, 0 seniors Reflective Tools 8 (5 Female, 3 Male) 4 soph, 2 juniors, 2 seniors (Sliders) 8 (Group omitted due to interaction issues) 19 • Sliders group omitted (did not use tools; poor group dynamics) • V&R groups: 241 posts and 516 segments (average of 15.06 (SD = 7.45) posts/student) • Mean words/post = 54 (SD = 42); mean characters/post = 299 (SD = 242)
  • 20. Total Skill score adds: • Intersubjectivity: perspective taking or question asking • Meta-dialogue, discussing the quality of the dialogue • Meta-Topic: Birds eye or systemic view of the topic • Appreciation (Gratitude, affirmation of another's idea or situation) • Source Reference (Mentioning a source, with a reference or description; without a fact) • Apology 20
  • 21. Total Skill vs. Condition 21
  • 22. Intersub vs. Condition 22
  • 23. Main Effect Exp. Group Total_ SD_Skill Intersubjective speech acts Vanilla (N = 8) 0.29 (0.07) 0.20 (0.09) Reflective Tools (N = 8) 0.40 (0.08) 0.30 (0.08) 23 • A significant difference and main effect between Total-SD-Score and grouping, F(1, 14) = 6.89, p = 0.02*, d = 1.46 (a large effect) in favor of the Reflective Tools group • A significant relationship between Intersub and grouping, F(1, 14) = 4.81, p = 0.05*, d = 1.05 (a large effect) in favor of the Reflective Tools group
  • 24. Comparison of Discussion Topics 24 Reflective tool use vs. topic Story words vs. topic
  • 25. Other Results • No effects of sub-skill vs. gender, except females scored higher on Appreciation • Positive correlation between Total Skill and post-survey scores on self-scored Engagement (r = 0.44) and Learning (r = 0.21) (no correlation vs. Enjoyment question) • No correlation between Replies-from and Replied-to vs. experimental group • The main effect of Condition vs. Total-skill came from the Trayvon discussion (Gun Control topic had less engagement) • Most of main effect of Total-skill from the Intersub sub- skill
  • 26. 3. Facilitator Dashboard
  • 27. 27
  • 28. View by Gender 28
  • 29. Linguistic Features – LIWC 80+ features 5 categories Linguistic process (e.g., total words per sentence, % of pronouns) Psychological process (e.g., affect, cognition) Paralinguistic dimensions (e.g., assents, fillers) Punctuation (e.g., quotation marks, exclamation marks) Contents (excluded from this study) 29
  • 30. Dashboard Text Tagging 30
  • 31. 31
  • 32. Advice Screen
  • 33. Settings
  • 34. Next: Linked Representations 34
  • 35. Future: Additional Metrics Common problems encountered in online facilitation • Low or no participation of individuals or groups, or silences or lulls on the part of individuals, the entire group, or sub-groups • Conversation domination by an individual or group • Inappropriate or disrespectful behavior • Off-topic conversation • Tension-filled disagreements, or high emotional content • Too much agreement or politeness • Misunderstanding due to missing communication skills normally available in face-to-face communication
  • 36. 4. Automated Text Analysis/Classification Text Analysis of Deliberative Skill: Using L1 Regularized Logistic Regression with Psycholinguistic Features
  • 37. Research Approach • Analyze online dialogues through a variety of lexical, discourse, and gender demographic features • Create machine learning classifiers to recognize social deliberative skills – “Total Skill” – Individual Skills (future work) 37
  • 38. Text Coding Scheme 38
  • 39. Total Skill score adds: • Intersubjectivity: perspective taking or question asking • Meta-dialogue, discussing the quality of the dialogue • Meta-Topic: Birds eye or systemic view of the topic • Appreciation (Gratitude, affirmation of another's idea or situation) • Source Reference (Mentioning a source, with a reference or description; without a fact) • Apology 39
  • 40. Corpora and Rater Agreement
  • 41. Demographic Features – Gender • Data distribution 41 Motivation: Woolley et. al, have shown that women score higher on social sensitivity than men do.
  • 42. Code Frequencies (classroom) Inter- sub Meta_ Dialogue Meta_ Topic Apology Apprecia tion Fact_ Source Source_ Ref #students 22 5 15 1 8 1 4 %segmts 25% 0.9% 5.5% 0.2% 1.3% 0.3% 1.2% 42
  • 43. Code Frequencies in Several Domains Exp. Group Total_ SD_Skill Intersubjective speech acts Vanilla (N = 8) 0.29 (0.07) 0.20 (0.09) Reflective Tools (N = 8) 0.40 (0.08) 0.30 (0.08) 43 • A significant difference and main effect between Total-SD-Score and grouping, F(1, 14) = 6.89, p = 0.02*, d = 1.46 (a large effect) in favor of the Reflective Tools group • A significant relationship between Intersub and grouping, F(1, 14) = 4.81, p = 0.05*, d = 1.05 (a large effect) in favor of the Reflective Tools group
  • 44. Linguistic Features – LIWC 80+ features 5 categories Linguistic process (e.g., total words per sentence, % of pronouns) Psychological process (e.g., affect, cognition) Paralinguistic dimensions (e.g., assents, fillers) Punctuation (e.g., quotation marks, exclamation marks) Contents (excluded from this study) 44
  • 45. Discourse Features – Coh-Metrix 100+ features 8 categories Narrativity Referential cohesion Syntactic simplicity Word concreteness Causal cohesion Verb cohesion Logical cohesion Temporal cohesion 45
  • 46. Machine Learning Method • L1 Regularized Logistic Regression –Auto-select features while learning –High generalizability via minimizing training loss and selecting a sparse model –High transparency like a “glass-box” model 46
  • 47. Performance Metrics • Accuracy What percent of all predictions were correct? Precision What percent of the positive predictions were correct? • Recall What percent of the positive cases were caught? • F2 Weighted average of precision and recall that weights recall twice as high 47
  • 48. Study 1. Single Classroom Dialog • 26 College Students in 2 week online discussion • 3 small discussion groups (of 8 or 9) • 2 Topics: Trayvon Martin Shooting & Gun Control • 829 text segments from 369 posts • 43% of the segments coded as "deliberate skill”
  • 49. Predictive performance (in %) of L1 regularized logistic regression built using different type of features
  • 50. Results • Moderate Recall (68%) and F2 (65%) • LIWC features outperformed Coh-Metrix • Adding gender and grade level features did not improve performance • Possibly encoded within LIWC/Coh-Metrix
  • 51. Study 2: Multi-Domain Dialogue Analysis • College dialogues – 4 college classes – Posts from college students from a variety of disciplines participating in e-discussions, about controversial topics. • Civic deliberation – E-Democracy.org, neighborhood discussion about ethnic tensions in a multi-racial community. • Professional community negotiation – Email exchanges among faculty of two academic communities deciding where to schedule a meeting 51
  • 52. Preliminary ML methods comparison 52 Conclusion: • L1-LRL slightly outperformed Naive Bayes and SVM
  • 53. Study 2 Experimental Design (using L1-RLR) • Goals – Study feature effects on prediction performance of machine learning models • Design of 2 scenarios and 54 experiments – In-domain analysis (6*3 evaluations) • Evaluate six possible feature configurations in each domain – Cross-domain analysis (6*6 evaluations) • Evaluate six possible feature configurations in six domain pairs (training and testing) 53
  • 54. 54 Precision 52.7 Recall 100.0 F2 84.8 Accuracy 56.2 Precision 56.9 Recall 69.7 F2 66.7 Accuracy 53.4 Precision 53.5 Recall 90.0 F2 79.2 Accuracy 56.2 Precision 56.8 Recall 70.6 F2 67.3 Accuracy 53.9 Precision 53.9 Recall 87.0 F2 77.5 Accuracy 54.1 Precision 54.0 Recall 87.0 F2 77.5 Accuracy 52.7 Precision 53.1 Recall 88.7 All LIWC+Gender Cohmetrix +Gender LIWC +Cohmetrix Cohmetrix Gender LIWC
  • 55. 55 In-Domain Training
  • 56. In-domain training: Results • Gender did not predict; nor did adding it to other features improve prediction • Classroom domain had poor performance (probably due to data skew) • LIWC performed better in Faculty dialogue recall 90%) • Coh-Metrix performed better in Civic dialogue (recall 84%)
  • 57. Data Skew per Domain
  • 58. Cross-Domain Training Training corpus Civic deliberation Civic deliberation Professional community negotiation Professional community negotiation College dialogues College dialogues Testing corpus Professional community negotiation College dialogues Civic deliberation College dialogues Civic deliberation Professional community negotiation Accuracy 52.7 31.7 56.8 31.7 43.2 47.3 Precision 52.7 31.7 56.8 31.7 0.0 0.0 Recall 100.0 100.0 100.0 100.0 0.0 0.0 F2 84.8 69.9 86.8 69.9 0.0 0.0 Accuracy 56.2 47.1 59.6 39.3 42.4 47.3 Precision 56.9 33.4 59.6 32.7 46.8 50.0 Recall 69.7 67.6 89.3 86.9 9.8 2.6 F2 66.7 56.1 81.2 65.2 11.6 3.2 Accuracy 53.4 36.4 53.3 39.0 41.9 47.0 Precision 53.5 30.7 58.3 30.2 36.8 42.9 Recall 90.0 80.0 62.2 70.3 3.1 1.3 F2 79.2 60.5 61.4 55.5 3.8 1.6 Cohmetrix Cross-domain Gender LIWC
  • 59. Cross-Domain Training Results • College domain prediction much better when training in other domains • Faculty domain best for training overall (Recall: 89% Civic; 87% College; 90% Faculty) • In general LIWC features do slightly better than CohMetrix, and combining them does not improve performance
  • 60. 60
  • 61. Conclusions
  • 62. Future Work-Text Analysis Create multi-task machine learning models with advanced regularizes (e.g., sparse group Lasso) to simultaneously identify each component social deliberative skills from online communication 62
  • 63. extra slides
  • 64. Top 10 LIWC features learned by L1 regularized logistic regression
  • 65. Total Skill component correlations