Extending  Sense-Making Models  with Ideas from  Cognition and Learning Theories Pengyi Zhang Dagobert Soergel Judith L. K...
Outline <ul><li>What is sense-making? </li></ul><ul><li>Theoretical framework </li></ul><ul><li>A comprehensive sense-maki...
Sense-making <ul><li>Sense-making is  </li></ul><ul><li>the process of creating an understanding of a problem or task  </l...
Sense-making <ul><ul><li>Sense-making is a pre-requisite for many other tasks such as decision making and problem solving;...
A Sample Sense-making Scenario <ul><li>Task description </li></ul><ul><li>An intelligence analyst is tasked to gather, ana...
Theoretical Framework <ul><li>Sense-making models </li></ul><ul><ul><li>Generic sense-making models  </li></ul></ul><ul><u...
Sense-making Elements Processes Sensing Making sense Acti vities Mechanisms Accretion Tuning Restructuring <ul><li>Inducti...
An Iterative Sense-making Model Data gap Structure gap Search: exploratory / focused Outcomes Updated knowledge Structure ...
Pilot Test <ul><li>Participants </li></ul><ul><li>Data collection </li></ul><ul><li>Data analysis </li></ul><ul><li>Findin...
Participants <ul><li>6 information science students as surrogate subjects for intelligence analysts </li></ul><ul><li>Task...
Data Collection <ul><li>6 2-hour task sessions between March 2007 and May 2007, with 2 to 5 participants attending each </...
A Sample Task Scenario <ul><li>Task T1: al-Bashir  (Abridged Version) </li></ul><ul><li>Omar Hasan Ahmad al-Bashir is a Su...
Data Analysis – Coding Scheme A  Processes Search Sense-making  Exploratory search Gap identification Exploratory search f...
Data Analysis – Coding Scheme C  Cognitive Mechanisms  Inductive mechanisms Deductive mechanisms Key item extraction Defin...
Sample Think-aloud Protocol with Coding Protocol (Energy Security Task) Loops Processes Conceptual Changes Cognitive Mecha...
Findings – Search and Sense-making Loops <ul><li>The overall search and sense-making loops followed four stages: </li></ul...
Findings – Cognitive Mechanisms <ul><li>Participants used a two-way approach:  data-driven (bottom-up)  80% structure-driv...
Findings – Dealing with Conflicts <ul><li>Disregard  –  “… I wanted that article to say something else.  I have to disrega...
Findings – Role of Instantiated Structures <ul><li>Entities (represented as names) and key concepts, (represented as keywo...
Conclusions <ul><li>The empirical think-aloud data seem to be consistent with the iterative sense-making model </li></ul><...
Implications and  Future Work <ul><li>Theoretical </li></ul><ul><ul><li>Better understanding of sense-making processes by ...
<ul><li>Pengyi Zhang </li></ul><ul><li>[email_address]   </li></ul><ul><li>terpconnect.umd.edu/~pengyi/sensemaking/ </li><...
Upcoming SlideShare
Loading in …5
×

Extending Sense-Making Models with Ideas from Cognition and Learning Theories

3,613 views
3,434 views

Published on

by Pengyi Zhang
Dagobert Soergel
Judith L. Klavans
Douglas W. Oard

Published in: Education, Technology
0 Comments
13 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,613
On SlideShare
0
From Embeds
0
Number of Embeds
44
Actions
Shares
0
Downloads
0
Comments
0
Likes
13
Embeds 0
No embeds

No notes for slide
  • Extending Sense-Making Models with Ideas from Cognition and Learning Theories

    1. 1. Extending Sense-Making Models with Ideas from Cognition and Learning Theories Pengyi Zhang Dagobert Soergel Judith L. Klavans Douglas W. Oard
    2. 2. Outline <ul><li>What is sense-making? </li></ul><ul><li>Theoretical framework </li></ul><ul><li>A comprehensive sense-making model </li></ul><ul><li>Pilot testing </li></ul><ul><li>Conclusions </li></ul><ul><li>Future work </li></ul>
    3. 3. Sense-making <ul><li>Sense-making is </li></ul><ul><li>the process of creating an understanding of a problem or task </li></ul><ul><li>so that further actions may be taken in an informed manner (Stefik et al., 1999). </li></ul>
    4. 4. Sense-making <ul><ul><li>Sense-making is a pre-requisite for many other tasks such as decision making and problem solving; </li></ul></ul><ul><ul><li>An important part of sense-making involves making clear the interrelated concepts and their relationships in a problem or task space. </li></ul></ul>
    5. 5. A Sample Sense-making Scenario <ul><li>Task description </li></ul><ul><li>An intelligence analyst is tasked to gather, analyze, and synthesize information related to Al-Bashir and to make recommendations for action in the form of a report. </li></ul><ul><li>Background </li></ul><ul><li>The Darfur conflict is a crisis in the Darfur region of western Sudan. Al-Bashir, the Sudanese president, is one of the key players in the area who is believed to have significant responsibility for continuous conflicts in the region. As part of an effort to resolve these armed conflicts, the administration needs to know as much as possible about al-Bashir in order to better negotiate with the involved parties and strategize its efforts. </li></ul>
    6. 6.
    7. 7. Theoretical Framework <ul><li>Sense-making models </li></ul><ul><ul><li>Generic sense-making models </li></ul></ul><ul><ul><li>Sense-making models of intelligence analysis </li></ul></ul><ul><ul><li>Conducting research as sense-making </li></ul></ul><ul><ul><li>Organizational sense-making </li></ul></ul><ul><li>Learning </li></ul><ul><ul><li>Schema theory </li></ul></ul><ul><ul><li>Assimilation theory </li></ul></ul><ul><ul><li>Generative Learning Theory </li></ul></ul><ul><ul><li>Structural knowledge acquisition </li></ul></ul><ul><li>Cognition </li></ul><ul><ul><li>Types of conceptual changes </li></ul></ul><ul><ul><li>Cognitive mechanisms </li></ul></ul><ul><ul><li>Cognitive structures of knowledge </li></ul></ul><ul><li>Design of related tools </li></ul><ul><ul><li>Information extraction </li></ul></ul><ul><ul><li>Representation and visualization tools </li></ul></ul><ul><ul><li>Note-taking, analysis tools </li></ul></ul><ul><ul><li>Task-specific and task-independent tools </li></ul></ul>
    8. 8. Sense-making Elements Processes Sensing Making sense Acti vities Mechanisms Accretion Tuning Restructuring <ul><li>Inductive, data-driven </li></ul><ul><li>Key item extraction </li></ul><ul><li>Comparison </li></ul><ul><li>Schema induction </li></ul><ul><li>Generalization </li></ul><ul><li>Deductive, structure-driven </li></ul><ul><li>Definition </li></ul><ul><li>Specification </li></ul><ul><li>Elimination </li></ul><ul><li>Explanation </li></ul><ul><li>Inference </li></ul>Outcomes Identification of gaps Search Exploratory search Focused search for data for structure Building structure Instantiating structure Consuming instantiated structure <ul><li>Other </li></ul><ul><li>Metaphor </li></ul><ul><li>Classification </li></ul><ul><li>Semantic fit </li></ul><ul><li>Socratic dialogues </li></ul>
    9. 9. An Iterative Sense-making Model Data gap Structure gap Search: exploratory / focused Outcomes Updated knowledge Structure loop Data loop Identification of Gaps Searching for data Instantiating structure Accretion: Instantiated structure Tuning: Adapted structure Re-structuring: New structure Searching for structure Building structure Existing Knowledge Structures and their instantiations with data The iterations proceed from exploratory to focused search and sense-making. Task / Problem Decision / Solution / Task completion
    10. 10. Pilot Test <ul><li>Participants </li></ul><ul><li>Data collection </li></ul><ul><li>Data analysis </li></ul><ul><li>Findings </li></ul>
    11. 11. Participants <ul><li>6 information science students as surrogate subjects for intelligence analysts </li></ul><ul><li>Tasked with synthesizing data from a variety of sources, assessing the credibility of information, and evaluating claims based on supporting evidence </li></ul><ul><li>Trained in using Rosetta - a multilingual, multimedia news retrieval system </li></ul>
    12. 12. Data Collection <ul><li>6 2-hour task sessions between March 2007 and May 2007, with 2 to 5 participants attending each </li></ul><ul><li>Data consists of 17 think-aloud protocols – participants were instructed to verbalize their thoughts as they work on the tasks </li></ul><ul><li>Sessions were logged and participants were interviewed after each session. </li></ul>
    13. 13. A Sample Task Scenario <ul><li>Task T1: al-Bashir (Abridged Version) </li></ul><ul><li>Omar Hasan Ahmad al-Bashir is a Sudanese military leader, dictator, and current president of Sudan. Your task is to produce a report identifying information to assess the influence of al-Bashir as a basis for policy decisions and diplomatic actions. Requested information includes: </li></ul><ul><li>key figures, organizations, and countries who have been associated with al-Bashir; </li></ul><ul><li>his rise to power; and </li></ul><ul><li>groups who have resisted him and the level of success in their resistance. </li></ul>
    14. 14. Data Analysis – Coding Scheme A Processes Search Sense-making Exploratory search Gap identification Exploratory search for data Data gap Exploratory search for structure Structural gap Focused search Building structures Focused search for data Using automatically extracted results Focused search for structure Extracting relationships manually Instantiating structures Updating knowledge B Conceptual Changes Sense-making success Sense-making failure Accretion Unable to fit data into structure Tuning Re-structuring Unable to build new structure
    15. 15. Data Analysis – Coding Scheme C Cognitive Mechanisms Inductive mechanisms Deductive mechanisms Key item extraction Definition Comparison Specification Similarity Explanation-based mechanisms Differentiation or discrimination Elimination Analogy and metaphor Semantic fit Classification Socratic dialogues Schema induction Inference Generalization D Emerging Codes Added During Analysis Reasons starting a new loop Resolution of conflicts New lead Disregard conflicting evidence Sense-making success Compromise Sense-making failure Accept new evidence Search failure Confusion
    16. 16. Sample Think-aloud Protocol with Coding Protocol (Energy Security Task) Loops Processes Conceptual Changes Cognitive Mechanisms Okay that was actually a very useful search. So let’s still take this query and look at Algeria, ‘cause obviously Algeria and Nigeria are very close… Loop 4 Focused Search for data Comparison I understand some of the keywords in the article but I don’t understand what the article… Sense-making Failure Key item extraction Okay this has to do with Algeria, southern Algeria. The minister of energy… OPEC meeting… so I am going to see what their connections are with OPEC. Building structures Key item extraction … with all the violence in Nigeria, I was expecting to find the same types of political outrage in Algeria… Instantiating structures Comparison and analogy and I’m not seeing any notice of that at all. Updating of knowledge Re-structuring
    17. 17. Findings – Search and Sense-making Loops <ul><li>The overall search and sense-making loops followed four stages: </li></ul><ul><ul><li>Task analysis </li></ul></ul><ul><ul><li>exploratory stage </li></ul></ul><ul><ul><li>focused stage </li></ul></ul><ul><ul><li>updates of knowledge representation. </li></ul></ul><ul><li>Reasons for starting a new loop of search and sense-making: </li></ul><ul><ul><li>Success of previous sense-making </li></ul></ul><ul><ul><li>Failure of previous sense-making </li></ul></ul><ul><ul><li>New lead </li></ul></ul><ul><ul><li>Failure of search </li></ul></ul>
    18. 18. Findings – Cognitive Mechanisms <ul><li>Participants used a two-way approach: data-driven (bottom-up) 80% structure-driven (top-down) 20% </li></ul><ul><li>Key item extraction and comparison were used most often. </li></ul>
    19. 19. Findings – Dealing with Conflicts <ul><li>Disregard – “… I wanted that article to say something else. I have to disregard it.” </li></ul><ul><li>Compromise – “…it is not my understanding that this has anything to do with oil. But okay, this is a different take.” </li></ul><ul><li>Acceptance – “…I thought these countries had similar serious problems. But [in fact, they] seem to be relatively stable and put a lot of effort into establishing their economy.” </li></ul><ul><li>Confusion – “…Obviously I have no idea what this is about...” </li></ul>
    20. 20. Findings – Role of Instantiated Structures <ul><li>Entities (represented as names) and key concepts, (represented as keywords) were often the basis for relevance judgments. </li></ul><ul><li>The relationships embedded in new information and between the new information and participants’ previous knowledge seemed to play an important role in structure building and data fitting. </li></ul><ul><li>Both concepts and relationships seemed to be crucial for updating knowledge. </li></ul>
    21. 21. Conclusions <ul><li>The empirical think-aloud data seem to be consistent with the iterative sense-making model </li></ul><ul><li>Users used mostly data-driven cognitive mechanisms </li></ul><ul><li>Instantiated structure elements play an important role in sense-making </li></ul>
    22. 22. Implications and Future Work <ul><li>Theoretical </li></ul><ul><ul><li>Better understanding of sense-making processes by extending the existing models with ideas from cognition and learning </li></ul></ul><ul><ul><li>Further testing and refinement. </li></ul></ul><ul><li>Empirical </li></ul><ul><ul><li>Better system design based on findings from user studies. </li></ul></ul><ul><ul><li>Making design suggestions to support sense-making, not just search. </li></ul></ul>
    23. 23. <ul><li>Pengyi Zhang </li></ul><ul><li>[email_address] </li></ul><ul><li>terpconnect.umd.edu/~pengyi/sensemaking/ </li></ul>

    ×