ODR 2013 SDSkills dashboard umass
Upcoming SlideShare
Loading in...5
×
 

ODR 2013 SDSkills dashboard umass

on

  • 160 views

 

Statistics

Views

Total Views
160
Slideshare-icon Views on SlideShare
160
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Mediem made by idealogue
  • PCP, e-democracyhttp://www.franklinpierce.edu/institutes/neccl/IF: “expand the scope and health of our public discussions”
  • Students who posted fewer than 5 times for both topics combined are excluded ;One student failed to follow instructions (did not use the sliders). This student dominated the discussion, contributing over a third of the total posts. This student’s posts were longer than average, constituting 41% of the total length of the conversation of this group, as gauged by the total number of characters typed. Two other students in this group did not post enough to be included in the analysis. One student wrote a note to the facilitator claiming that one student in this group seemed overly critical and not respectful, which affected her feeling of safety. The tension here may have put a damper on the entire group
  • https://docs.google.com/document/d/1nb_OsOqpQMPZbov4Uo1PWwqbFyJoW5jz_1hyiIYobug/edit
  • (not surprising since INTERSUB was strongly correlated with Total SD Skill)
  • https://docs.google.com/document/d/1nb_OsOqpQMPZbov4Uo1PWwqbFyJoW5jz_1hyiIYobug/edit
  • Features that are predictive for the classification task are unknown from the literature -- we are the first to explore this research territory of identifying social deliberative skills. Social deliberative skills have a hierarchical structure, which implies that we are tasked with doing hierarchical classification with multi-classes. Our data annotated with social deliberative skills are highly skewed, because some skills are used much more often than others.
  • Tom Murray, 2012
  • Fall 2011 experiment

ODR 2013 SDSkills dashboard umass ODR 2013 SDSkills dashboard umass Presentation Transcript

  • An Online DeliberationFacilitatorsDashboard: Visualizations andText analysis to support qualitydialoguesWing, L., Murray, T., Woolf, B., Katsh, E.The Twelfth International Online Dispute Resolution Forum,Montreal, June 2013
  • “The Fourth Party: Improving Computer-MediatedDeliberation through Cognitive, Social andEmotional Support”• 3-Year NSF Social Computing grant, startedFall 2010• Description atwww.socialdeliberativeskills.com2
  • Project collaborators• Beverly Woolf: CompSci, PI (intelligent and collaborativeeducational systems)• Tom Murray: CompSci; project manager/co-PI, principalvisionary and instigator (ed-tech, cog-psych & D&D)• Ethan Katsh (ODR), Legal Studies, co-PI• Leah Wing (social justice and ODR), Legal Studies, co-PI• Linda Tropp, Psychology of Peace and Violence, advisor(intergroup relations/conflict)• Zan Goncalves, New England Center for Civic Life(teaching, practice and study of deliberative democracy)• Idealogue Inc.; iCohere. Inc. softwareplatforms(Advanced dialogue)3
  • Thanks for consultationand/or data from:• DemarsAssociates.com/PayPal/ebay (e-commerce)• Juripax.com (online workplace and divorcesettlements)• National Mediation Board (transportationmanagement/labor disputes)• Modria.com (e-commerce plus)• Idealogue.com (depth-oriented online dialogueplatform)• iCohere.com (online communities and work groups)• Mass Dept. of Dispute Resolution (civic engagement)• New England Center for Civic Life (teaching, practiceand study of deliberative democracy)4
  • Social Deliberative Skills:Social/Emotional/ReflectivePerspective taking & cognitive empathyPerspective seeking (curiosity/inquiry)Self-reflection: on ones biases, intentions,emotional stateMeta-dialog: Reflect on the quality of thedialogEpistemic skill: e.g. treating facts/datadifferently from opinions/hypothesesTolerance for uncertainty, ambiguity,disagreement, paradox…5
  • Support/Scaffolding vs.“Education”FacilitatedOnlineDELIBERATIONOutcomes:- Agreements/solutions- Relationship, Trust (social capital),understanding- SKILL USE (and practice)ExistingSkillsAdaptiveSupport(4th party)InterfaceSupportFacilitatorSupport(Dashboard)
  • Example: Topics Chosen by Students—set context in Spring ’12• Week 1: Discuss the pros and cons of legalizingmarijuana• Week 2: Sex – whats the big deal? What values aremost important in making sexual choices?• Week 3: Discus the pros and cons of the death penalty(capital punishment)7
  • [CURRENT] WEEK 1: Discuss the pros and cons of leg...UPDATE PROFILELOG OUTHOMELogged in as tomm[CURRENT] WEEK 1: Discuss the pros and cons of legalizing marijuana.[CURRENT] WEEK 1: Discuss the pros and cons of legalizing marijuana.To focus the conversation, we invite you to assume you are on an advisory panel for the statelegislature, having some preliminary conversations online, and you will eventually be draftinga group recommendation. Consider not only your own preferences but what is best for thestate (or society).edit deleteCONTRIBUTE YOUR THOUGHTS14:53 EDT Sunday, November 13 by tommtomm has joined the conversation23:53 EDT Saturday, November 12 by ines- vines-v added a resource: Getting a Fix23:52 EDT Saturday, November 12 by ines- vI have to disagree with your third point that marijuana is a gateway drug. Ofall the people I know that smoke marijuana, they do not do any hard drugs.I do agree that gateway drugs exist, however I feel like that typicallyhappens from one hard drug to another when one doesnt seem to beenough. But if you want to talk about gateway drugs we would also have tomention alcohol and cigarettes which many people consume and smoke.Alcohol and cigarettes are also drugs and often considered gateway drugs.They are both legal so that option is void in regards to marijuana.You also mentioned cancer and other lung related issues. Marijuana is anatural plant. Cigarettes are made up of extremely harmful chemicals thatcause lung related issues and cancer much faster than marijuana ever could.Yet, they are still legal. If anything, cigarettes should be illegal whenconsidering public health. Marijuana is a lot safer than cigarettes.I do appreciate you playing Devils advocate though!Id like to explain how I see it differently (ines-v)18:26 EDT Friday, November 11 by arthur- xIt seems like the vast majority is supportive of the legalization of marijuana,so Im going to play devils advocate in order to bring the oppositions sideto the table.First off, research has demonstrated that marijuana use reduces learningability by limiting the capacity to absorb and retain information. A 1995study of college students discovered that the inability of heavy marijuanausers to focus, sustain attention, and organize data persists for as long as 24hours after their last use of the drug. Earlier research, comparing cognitiveabilities of adult marijuana users with non-using adults, found that users fallshort on memory as well as math and verbal skills. Although it has yet to beproven conclusively that heavy marijuana use can cause irreversible loss ofintellectual capacity, animal studies have shown marijuana-inducedines-varthur-xjoseph-tlaura-trtwellsmatthew-stommDIALOGUE TABLEEveryone (no demographics set)8
  • 9MediemOpinion Sliders
  • Experimental ConditionsExp Group N Gender GradeVanilla 8 (5 Female,3 Male)4 soph, 4 juniors,0 seniorsReflective Tools 8 (5 Female,3 Male)4 soph, 2 juniors,2 seniors(Sliders) 8 (Group omitted due to interaction issues)10• V&R groups: 241 posts and 516 segments (averageof 15.06 (SD = 7.45) posts/student)• Mean words/post = 54 (SD = 42); meancharacters/post = 299 (SD = 242)
  • Text Coding11Soc. DELIBERATIONSkill EvidenceMISC CODES ACTIONNEGOTIATIONARGUMENT CODESSELF_REFLection_INTERSUBictiveQ_INTERLocutorREF_INTERLocutorPERSPECTIVE_taking_META_DialogMEDIATEMETA_CONSMETA_CONFLMETA_SUMMETA_CHECK_META_TOPICWEIGHSYSTEMs_thinkingFACT_cite_SouRCeSOURCE_REFerenceAPPRECiationQ_TOPICCHANGE_mindUNCERtaintyOTHERS_THNKAPOLOGYHELPREQ_HELPPROCESSAGREEDISAGREE_NEGative-emotionNEGEMO_INTerlocutorNEGEMO_Topic_OFFTOPICTECHnicalSOCIAL(External actions)ActRequestActProposeActAcceptActDeclineActNegot(Dialogue_Actions)DI_ActRequestDI_ActProposeDI_ActAcceptDI_ActDeclineDI_ActNegot(Facilitators only)WELCOMINGPROC_EXPLMOTIVATE_ARGument_GENericFACT_NOSRCGENERAL_SOLUTNEXPER_OBSERVARG_OPINIONOPINION_ONLYOVER_GENSUPPORTSUM_MY-argumtEXAMPLEELAB
  • Main EffectExp. Group Total_SD_SkillIntersubjectivespeech actsVanilla (N = 8) 0.29 (0.07) 0.20 (0.09)Reflective Tools (N = 8) 0.40 (0.08) 0.30 (0.08)12• A significant difference and main effect betweenTotal-SD-Score and grouping, F(1, 14) = 6.89, p =0.02*, d = 1.46 (a large effect) in favor of theReflective Tools group• A significant relationship between Intersub andgrouping, F(1, 14) = 4.81, p = 0.05*, d = 1.05 (a largeeffect) in favor of the Reflective Tools group
  • 13Facilitator’s Dashboard
  • Dashboard Text Tagging14
  • Advice Screen
  • Applicability and UseFacilitators• To identify individual and group participationlevels for assessing useful interventions(ie: to stimulate more involvement, moreconstructive engagement, etc.)• To identify patterns of ‘silence’ or intensity(ie: to ID topics/relationships to attend to;use as clues to search text analysis forconflicts or breakthroughs)
  • Applicability and UseParticipants• For self/group assessment of engagement,topical, and relationship patterns re:participation, intensity, silence• For clues to search text analysis forconflicts or breakthroughs
  • CohMetrixdiscourse & coherence18LIWClexical categories
  • Skill vs. text metrics correlations
  • LIWC/Cohmetrix Correlations withTotal-SD-Skill
  • Cross-DomainTrainingPrelim. Machine Learning ResultsFor Predicting Total Skill
  • Future ApplicationsCommon problems encountered in online facilitation• Low or no participation of individuals or groups, orsilences or lulls on the part of individuals, the entiregroup, or sub-groups• Conversation domination by an individual or group• Inappropriate or disrespectful behavior• Off-topic conversation• Tension-filled disagreements, or high emotionalcontent• Too much agreement or politeness• Misunderstanding due to missing communication skillsnormally available in face-to-face communication• Violation of rules (e.g. confidentiality, no advertising,etc.)
  • Thank you
  • Extra slides
  • Automated Text AnalysisLIWC (Pennebaker et al.) – Dictionary-based (linguisticinquiry word count)– 4,500 words/STEMS; 80 word categories– we focus on 19 of them– 80 >> 4 general descriptor categories (word count, words per sentence, % of words captured, and %of words >6 letters), 22 standard linguistic dimensions (e.g., % pronouns, articles, auxiliaryverbs, etc.), 32 psychological constructs (e.g., affect, cognition, biological processes), 7 personalconcern categories (e.g., work, home, leisure activities), 3 paralinguistic dimensions(assents, fillers, nonfluencies), and 12 punctuation categories (periods, commas, etc).– Relate the categories to things like aggression, used in theraputic contexts, to ID lying,• Coh-Metrix (Graesser et al.)– syntax, referential cohesion, semantic cohesion, rhetorical composition…– 100 measurements categories– We focus on 4 composite measurements (or majorfactors): Narrativity, Referential Cohesion, SyntacticSimplicity, and Word Concreteness
  • Experimental Groups27
  • Code FrequenciesIntersub Meta_DialogueMeta_TopicApology AppreciationFact_SourceSource_Ref#students22 5 15 1 8 1 4% ofsegs 25% 0.9% 5.5% 0.2% 1.3% 0.3% 1.2%28
  • Total Skill score adds:• Appreciation (Gratitude, affirmation of anothers ideaor situation)• Apology• Fact--sourced (stating a fact and noting the source inthe same post)• Source Reference (Mentioning a source, with areference or description; without a fact)• Intersubjectivity: perspective taking or question asking• Meta-dialogue, discussing the quality of the dialogue• Meta-Topic: Birds eye or systemic view of the topic29
  • Next: Linked Representations• clicking on the name of an individual or group in a chart ornetwork diagram will focus (or filer or highlight) all tools onthat individual or group;• clicking on a link in the network diagram will show postsbetween the relevant interlocutors;• clicking on a word in the word cloud will highlight postsincluding that word;• clicking on a location in the time-axis of a trend line willnavigate to posts in the Timeline at that time;• hovering over an agent trigger will show the Advice or Alertassociated with that event; and clicking on Advice or Alertswill navigate to the place in the dialogue timeline for thetriggering event(s).
  • Text Coding31Soc. DELIBERATIONSkill EvidenceMISC CODES ACTIONNEGOTIATIONARGUMENT CODESSELF_REFLection_INTERSUBictiveQ_INTERLocutorREF_INTERLocutorPERSPECTIVE_taking_META_DialogMEDIATEMETA_CONSMETA_CONFLMETA_SUMMETA_CHECK_META_TOPICWEIGHSYSTEMs_thinkingFACT_cite_SouRCeSOURCE_REFerenceAPPRECiationQ_TOPICCHANGE_mindUNCERtaintyOTHERS_THNKAPOLOGYHELPREQ_HELPPROCESSAGREEDISAGREE_NEGative-emotionNEGEMO_INTerlocutorNEGEMO_Topic_OFFTOPICTECHnicalSOCIAL(External actions)ActRequestActProposeActAcceptActDeclineActNegot(Dialogue_Actions)DI_ActRequestDI_ActProposeDI_ActAcceptDI_ActDeclineDI_ActNegot(Facilitators only)WELCOMINGPROC_EXPLMOTIVATE_ARGument_GENericFACT_NOSRCGENERAL_SOLUTNEXPER_OBSERVARG_OPINIONOPINION_ONLYOVER_GENSUPPORTSUM_MY-argumtEXAMPLEELAB
  • Social Deliberative Skills:Social/Emotional/Reflective• 1. Social perspective taking(cognitive empathy, reciprocal roletaking...)• 2. Social perspective seeking (socialinquiry, question asking skills...)• 3. Social perspective monitoring(self-reflection, meta-dialogue...)• 4. Social perspective weighing(reflective reasoning; comparing andcontrasting views...)32
  • Domain statistics andinter-rater agreement
  • Codoole—coding tools34Coole – coding tools
  • Social Deliberative Skill:application of HOSs to me/you/weHigher Order Skills• argumentation• critical thinking• explanation & clarification• inquiry/curiosity(question asking & investigation)• reflective judgment• meta-cognition• epistemic reasoningApply these skills, not toEXTERNAL REALITY (“IT”/problemdomain) but to theINTERSUBJECTIVE domainHigher Order Skills applied to:SELFgoals; level of certainty;feelings, values, assumptions…YOUgoals, assumptions, feelings,values; perspective taking;"believing" & cognitive empathy…WEagreements, goals; quality ofthe discourse/collaboration;differences and similarities invalues, beliefs, goals, power, roles…
  • Survey Questions29 of the 36 participants took the survey
  • By Demographic
  • Settings
  • Post number vs. size