2013 AIED SRL workshop--Social Deliberatie Skills Defined_ Murray


Published on

Published in: Education, Health & Medicine
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • New domain, still catching up on interdscipl literature; hear to learn
  • & meta-memory; meta affect;
  • when we slice the world into categories we simplify and may ignore what is between or outside them
  • so we should take caution; or periodically revisit our construct definitions and ontologies
  • at the algorithm level: adjusting class weights, cost, or priors for different classifiers, including support vector machine (SVM) with different hyper parameters (e.g., kernels, degree), SVM-one class, naïve Bayes, decision tree, random forest, bagging trees, and boosting treesat the data level: using different sampling techniques (e.g., up-sampling and down-sampling) to balance the data before training a classifier. When using a SVM classifier with a polynomial kernel, we have achieved similar result as we have done using the first approach.A high variance model – training error is much lower than testing errorBagging is often an effective technique for solving the high variance problem However, because our data is highly skewed, it is not surprising that bagging does not work wellA new adaptive sampling algorithm for imbalanced datathe algorithm works as follows: (1) it iteratively performs down-sampling without replacement in order to fully utilize the training data and ensure that there are great overlapping in different sampling spaces, (2) it runs SVM multiple times using different samples, and (3) it decides the class labels for testing data based on majority vote using classifiers built in step (2). Co-training algorithms for imbalanced multi-class classificationthe key idea of co-training is that two classifiers trained on two different views train each other using the unlabeled data two views: cohesion-based features (coh-metrix), lexical features(LIWC)Latent Deliberative Skill Model (LDSM) for Skill Classificationa mixture of only 5 skill labels are modeled, e.g, perspective taking, big-picture thinking the generative processchoose a distribution over 5 skill labels for a documentpick up a skill label from the skill label distribution draw a word according to the skill label-word distribution.
  • 2013 AIED SRL workshop--Social Deliberatie Skills Defined_ Murray

    1. 1. Toward Defining, Justifying, Measuring, and Supporting Social Deliberative Skills Tom Murray UMass Amherst At AIED July 2013, Memphis: Workshop on Self-Regulated Learning in Educational Technologies: Supporting, modeling, evaluating, and fostering metacognition with computer-based learning environment
    2. 2. Group & Collaborative Work/Leaning Conflict Resolution Meaning Negotiation Problem solving Planning Brainstorming & Creativity Inquiry Decision making Knowledge building Group dynamics (form, storm, norm) Peer help/tutoring
    3. 3. Social Deliberative Skills: Social/Emotional/Reflective • 1. Social perspective taking (cognitive empathy, reciprocal role taking...) • 2. Social perspective seeking (social inquiry, question asking skills...) • 3. Social perspective monitoring (self-reflection, meta-dialogue...) • 4. Social perspective weighing (reflective reasoning; comparing and contrasting views...) 3
    4. 4. Social Deliberative Skills Perspective Taking Social Metacognition Reflective Reasoning Empathy Dialogue and Deliberation skills
    5. 5. Meaning Negotiation Conflict Resolution Skills needed to bridge different perspectives to build mutual understanding and mutual regard Social Deliberative Skills: The capacity to deal productively with heterogeneous goals, values, or perspectives in dialogue and deliberation – Including: collaboration, problem solving, knowledge building, inquiry learning...
    6. 6. Overview • Background • Supporting them online – Participants – Facilitators • Measuring them – Human coding – Machine classification • Deeper exploration of their meaning – And issues with construct definitions (ontology)
    7. 7. 1. Background
    8. 8. Text Coding Scheme 8
    9. 9. Corpora and Rater Agreement
    10. 10. Examples of Social Deliberative Skills/Behavior From authentic dialogues in our online corpora “ I am probably extremely biased because I am under 21 years old and in college. I wonder if as a 45 year old I will feel differently. ” (self reflection) “I can’t help but imagine what that is like, for her and for her family.” (perspective taking) 10
    11. 11. Code Frequencies in Several Domains Exp. Group Total_ SD_Skill Intersubjective speech acts Vanilla (N = 8) 0.29 (0.07) 0.20 (0.09) Reflective Tools (N = 8) 0.40 (0.08) 0.30 (0.08) 11 • A significant difference and main effect between Total-SD-Score and grouping, F(1, 14) = 6.89, p = 0.02*, d = 1.46 (a large effect) in favor of the Reflective Tools group • A significant relationship between Intersub and grouping, F(1, 14) = 4.81, p = 0.05*, d = 1.05 (a large effect) in favor of the Reflective Tools group
    12. 12. Support/Scaffolding (vs. “Education”) Online Dialogue & DELIBERATION Outcomes: - Agreements/solutions - Relationship, Trust (social capital) - SKILL USE (and practice) Existing Skills Adaptive Support (4th party) Passive Support (interface) Facilitator Support (Dashboard)
    13. 13. [CURRENT] WEEK 1: Discuss the pros and cons of leg... UPDATE PROFILE LOG OUT HOME Logged in as tomm [CURRENT] WEEK 1: Discuss the pros and cons of legalizing marijuana.[CURRENT] WEEK 1: Discuss the pros and cons of legalizing marijuana. To focus the conversation, we invite you to assume you are on an advisory panel for the state legislature, having some preliminary conversations online, and you will eventually be drafting a group recommendation. Consider not only your own preferences but what is best for the state (or society). edit delete CONTRIBUTE YOUR THOUGHTS 14:53 EDT Sunday, November 13 by tomm tomm has joined the conversation 23:53 EDT Saturday, November 12 by ines- v ines-v added a resource: 'Getting a Fix' 23:52 EDT Saturday, November 12 by ines- v I have to disagree with your third point that marijuana is a gateway drug. Of all the people I know that smoke marijuana, they do not do any hard drugs. I do agree that gateway drugs exist, however I feel like that typically happens from one hard drug to another when one doesn't seem to be enough. But if you want to talk about gateway drugs we would also have to mention alcohol and cigarettes which many people consume and smoke. Alcohol and cigarettes are also drugs and often considered gateway drugs. They are both legal so that option is void in regards to marijuana. You also mentioned cancer and other lung related issues. Marijuana is a natural plant. Cigarettes are made up of extremely harmful chemicals that cause lung related issues and cancer much faster than marijuana ever could. Yet, they are still legal. If anything, cigarettes should be illegal when considering public health. Marijuana is a lot safer than cigarettes. I do appreciate you playing Devil's advocate though! I'd like to explain how I see it differently (ines-v) 18:26 EDT Friday, November 11 by arthur- x It seems like the vast majority is supportive of the legalization of marijuana, so I'm going to play devil's advocate in order to bring the opposition's side to the table. First off, research has demonstrated that marijuana use reduces learning ability by limiting the capacity to absorb and retain information. A 1995 study of college students discovered that the inability of heavy marijuana users to focus, sustain attention, and organize data persists for as long as 24 hours after their last use of the drug. Earlier research, comparing cognitive abilities of adult marijuana users with non-using adults, found that users fall short on memory as well as math and verbal skills. Although it has yet to be proven conclusively that heavy marijuana use can cause irreversible loss of intellectual capacity, animal studies have shown marijuana-induced ines-v arthur-x joseph-t laura-t rtwells matthew-s tomm DIALOGUE TABLE Everyone (no demographics set) 13
    14. 14. 14 Mediem Opinion Sliders
    15. 15. Graduate Class 2012—Total Skill 15
    16. 16. 16
    17. 17. CohMetrix discourse & coherence 17 LIWC lexical categories
    18. 18. 18
    19. 19. feature comparisons 19 In-domain Cross-domain Training corpus Civic deliberation Professional community negotiation Civic deliberation Professional community negotiation Testing corpus Civic deliberation Professional community negotiation Professional community negotiation Civic deliberation Gender Accuracy 56.8 52.7 52.7 56.8 Precision 56.8 52.7 52.7 56.8 Recall 100.0 100.0 100.0 100.0 F2 86.8 84.8 84.8 86.8 LIWC Accuracy 55.3 55.3 56.2 59.6 Precision 58.9 54.6 56.9 59.6 Recall 70.7 90.0 69.7 89.3 F2 68.0 79.7 66.7 81.2 Cohmetrix Accuracy 62.1 54.8 53.4 53.3 Precision 61.4 55.3 53.5 58.3 Recall 83.6 74.0 90.0 62.2 F2 77.9 69.3 79.2 61.4 LIWC+Gender Accuracy 52.3 54.6 56.2 60.9 Precision 56.8 54.2 56.8 60.6
    20. 20. Exploring Social Deliberative Skills
    21. 21. Social Deliberative Skills: Literature BACKGROUND • Dialogue Acts, Argumentation • Reflective & Critical Reasoning • Meta-skills • Civic Deliberation • Accountable Talk • Research into Perspective taking and Empathy
    22. 22. Dialogue Act Schemes • M. Baker (Rainbow, 2007) – Social relation – Interaction management – Task management – Opinions – Argumentation – Broaden and deepen • Dejong (Tr-Diagram, 2005) – Argument – Explanation – Challenge (but) – Evidence – Support/Refute • Kim (2007) – Acknowledge/Support/Compl ement – Inform, Command, Announce – Answer/Suggestion/Advice – Correction/Objection – Elaboration/Description – Question
    23. 23. Reflective Reasoning, Critical Thinking, Argumentation • Reflective Reasoning – Ill-structured problems – about which reasonable people reasonably disagree • Critical Thinking – analyze – interpret – integrate
    24. 24. D. Kuhn: Ill-defined constructs in Higher Order Thinking Argumentation Inquiry Critical thinking Metacognition Reflective Judgment Epistemological Knowledge
    25. 25. Metacognition (Winne; Azedevo; Wellman) • Phases: planning, monitoring, control/regulation, reflection/assessment (repeat) • Domains: cognitive, motivational/affective, behavioral, contextual; memory, attention; problem solving state • Self-reflection; self-explanation; help/info- seeking and SR learning
    26. 26. Social Metacognition (White; Scardemalia; Collins) • Meta-knowledge/skill about: – Self expertise, incl. collab. and communic. skill – Roles & relationships – Social context (and mediation) – Knowledge-building: monitoring; improving; rise- aboves
    27. 27. Civic Deliberation – Dahlberg (2001) (& Habermas): – reflexivity, ideal role taking, sincerity, inclusion – exchange of criticizable moral-practical validity claims
    28. 28. Analytic Aspects of Deliberation: Stromer-Galley • Introduction - welcoming, context, etc. • Deliberation Process • Technical Process • Meta-Talk (consensus, conflict, clarity) • Summary • Disagree/Agree • Off-topic move • Reasoned Opinion (fact, question, elab. opinion) • Sourcing • Intervention • Invite Others • Social (praise, apology, off topic)
    29. 29. Analytic/Cognitive vs. Social/Emotional Gastil & Black: Key features of deliberative dialogue 29
    30. 30. Accountable Talk • Social: Listen (pay attention to…), Summarize, Build upon, Refer to • Knowledge: Verify (check), Unpack (explain), Support (examples, evidence), Link • Reason: Defend, Challenge, Combine, Predict
    31. 31. Social Deliberative Skills – v1
    32. 32. Primary SD-Skills • Differentiating facts/opinions (knowing how to reason about each) • Reflecting on biases and assumptions (mostly one's own but also others) • Perspective taking (of actual interlocutors; of other groups/identities/cultures, etc.) • Reflecting on the dialog as a whole (meta- dialogue)
    33. 33. Social Deliberative Skills V-2 Social/Emotional/Reflective • 1. Social perspective taking (cognitive empathy, reciprocal role taking...) • 2. Social perspective seeking (social inquiry, question asking skills...) • 3. Social perspective monitoring (self-reflection, meta-dialogue...) • 4. Social perspective weighing (reflective reasoning; comparing and contrasting views...) 33
    34. 34. Text Coding Scheme 34
    35. 35. Social Deliberative Skill: application of HOSs to me/you/we Higher Order Skills • argumentation • critical thinking • explanation & clarification • inquiry/curiosity (questioning) • reflective judgment • meta-cognition • epistemic reasoning Apply these skills, not to EXTERNAL REALITY (“IT”/problem domain) but to the INTERSUBJECTIVE domain Higher Order Skills applied to: SELF goals; level of certainty; feelings, values, assumptions… YOU goals, assumptions, feelings, values; perspective taking; "believing" & cognitive empathy… WE agreements, goals; quality of the discourse/collaboration; differences and similarities in values, beliefs, goals, power, roles…
    36. 36. Next: Support & Interventions • “Teaching:” role-play, simulated dialogues... • Research on Perspective taking & Empathy? • Computer-based – Scripts – Sentence Openers – Reification (& passive reminders) – Awareness Tools
    37. 37. Project Collaborators: Bev Woolf, Xiaoxi Xu, Lynn Stephens, Leah Wing, Natasha Shrikant And thanks to Art and Nia from CohMetrics lab.
    38. 38. Project Overview 38
    39. 39. A journey through ontological conundrums • Below is additional material related to the unavoidability of construct overlap and indeterminacy in defining and using abstract concepts such as metacognition, inquiry, etc. (See overlap in Kuhn slide above)
    40. 40. “ ...to Understand, Measure & Support SD-skills” • Reviewer: “How can study them if you don’t understand them and can’t define them precisely?”
    41. 41. • Scholarly work is "notoriously fraught with definitional disagreement" (Shermer, 2011) An invitation to co-explore Ontological Indeterminacy
    42. 42. Kurt Fischer’s Dynamic Skill Theory • Skills develop in response to real tasks • Higher order skills are built doing complex tasks; no isolated task, therefore no isolated skill (compare “leadership” skill with “territoriality” biological drive) • ...skills [and knowledge] are not isolated units, but rather function together in complex structures of inter participation...an ecosystem...any given skill requires the existence of various others as component [or interacting] parts...
    43. 43. Therefore.... • Definition of task takes precedence over definition of the skill
    44. 44. Example: “intelligence” and IQ • Real life target Task • Skills (& knowledge) • Measurement Caution: the construct definition can be reduced to the measurement definition or enactment
    45. 45. SD-Skills: task-oriented definition Skills needed to bridge different perspectives to build mutual understanding and mutual regard Social Deliberative Skills: The capacity to deal productively with heterogeneous goals, values, or perspectives in dialogue and deliberation – Including: collaboration, problem solving, knowledge building, inquiry learning...
    46. 46. Concept Indeterminacy (Lakoff) • Abstract Concepts: –Graded (fuzzy) –Have “metaphorical pluralism” –Metaphorical (limited by embodiment) • More abstract => More indeterminate Tom Murray | www.perspegrity.com | August 2010 46
    47. 47. Two sources of indeterminacy • 1. Task structure: Higher order skills are built doing complex tasks; no isolated task therefore no isolated skill • 2. Nature of categories: Abstract concepts are indeterminate by nature
    48. 48. Real phenomena tend to be chaotic, organic; not categorized
    49. 49. extra slides
    50. 50. Stromer-Galley SD-Skills Strommer-Galley WELCOMING Introduction - welcoming, context, etc. PROC-EXPL - process-explanation Deliberation Process TECH Technical Process META_SUM Summarize AGREE Disagree/Agree DISAGREE Disagree/Agree OFF_TOPIC Off-topic move MEDIATE Intervention || Invite Others SOCIAL Social
    51. 51. Code-descriptions and frequencies Perspective taking (15%), Big picture thinking (8%), meta-dialogue (5%), self-reflection (3%), Arg_gen (~30%)
    52. 52. Debate, Dialogue and Deliberation Deliberation: “thoughtful, careful, or lengthy consideration by individuals; and formal discussion and debate in groups” (Davies & Chandler 2011)