Extending Sense-Making Models with Ideas from Cognition and Learning Theories

Extending Sense-Making Models with Ideas from Cognition and Learning Theories Pengyi Zhang Dagobert Soergel Judith L. Klavans Douglas W. Oard

Outline
What is sense-making?
Theoretical framework
A comprehensive sense-making model
Pilot testing
Conclusions
Future work

Sense-making
Sense-making is
the process of creating an understanding of a problem or task
so that further actions may be taken in an informed manner (Stefik et al., 1999).

Sense-making
Sense-making is a pre-requisite for many other tasks such as decision making and problem solving;
An important part of sense-making involves making clear the interrelated concepts and their relationships in a problem or task space.

A Sample Sense-making Scenario
Task description
An intelligence analyst is tasked to gather, analyze, and synthesize information related to Al-Bashir and to make recommendations for action in the form of a report.
Background
The Darfur conflict is a crisis in the Darfur region of western Sudan. Al-Bashir, the Sudanese president, is one of the key players in the area who is believed to have significant responsibility for continuous conflicts in the region. As part of an effort to resolve these armed conflicts, the administration needs to know as much as possible about al-Bashir in order to better negotiate with the involved parties and strategize its efforts.

Theoretical Framework
Sense-making models
Generic sense-making models
Sense-making models of intelligence analysis
Conducting research as sense-making
Organizational sense-making
Learning
Schema theory
Assimilation theory
Generative Learning Theory
Structural knowledge acquisition
Cognition
Types of conceptual changes
Cognitive mechanisms
Cognitive structures of knowledge
Design of related tools
Information extraction
Representation and visualization tools
Note-taking, analysis tools
Task-specific and task-independent tools

Sense-making Elements Processes Sensing Making sense Acti vities Mechanisms Accretion Tuning Restructuring
Inductive, data-driven
Key item extraction
Comparison
Schema induction
Generalization
Deductive, structure-driven
Definition
Specification
Elimination
Explanation
Inference
Outcomes Identification of gaps Search Exploratory search Focused search for data for structure Building structure Instantiating structure Consuming instantiated structure
Other
Metaphor
Classification
Semantic fit
Socratic dialogues

An Iterative Sense-making Model Data gap Structure gap Search: exploratory / focused Outcomes Updated knowledge Structure loop Data loop Identification of Gaps Searching for data Instantiating structure Accretion: Instantiated structure Tuning: Adapted structure Re-structuring: New structure Searching for structure Building structure Existing Knowledge Structures and their instantiations with data The iterations proceed from exploratory to focused search and sense-making. Task / Problem Decision / Solution / Task completion

Pilot Test
Participants
Data collection
Data analysis
Findings

Participants
6 information science students as surrogate subjects for intelligence analysts
Tasked with synthesizing data from a variety of sources, assessing the credibility of information, and evaluating claims based on supporting evidence
Trained in using Rosetta - a multilingual, multimedia news retrieval system

Data Collection
6 2-hour task sessions between March 2007 and May 2007, with 2 to 5 participants attending each
Data consists of 17 think-aloud protocols – participants were instructed to verbalize their thoughts as they work on the tasks
Sessions were logged and participants were interviewed after each session.

A Sample Task Scenario
Task T1: al-Bashir (Abridged Version)
Omar Hasan Ahmad al-Bashir is a Sudanese military leader, dictator, and current president of Sudan. Your task is to produce a report identifying information to assess the influence of al-Bashir as a basis for policy decisions and diplomatic actions. Requested information includes:
key figures, organizations, and countries who have been associated with al-Bashir;
his rise to power; and
groups who have resisted him and the level of success in their resistance.

Data Analysis – Coding Scheme A Processes Search Sense-making Exploratory search Gap identification Exploratory search for data Data gap Exploratory search for structure Structural gap Focused search Building structures Focused search for data Using automatically extracted results Focused search for structure Extracting relationships manually Instantiating structures Updating knowledge B Conceptual Changes Sense-making success Sense-making failure Accretion Unable to fit data into structure Tuning Re-structuring Unable to build new structure

Data Analysis – Coding Scheme C Cognitive Mechanisms Inductive mechanisms Deductive mechanisms Key item extraction Definition Comparison Specification Similarity Explanation-based mechanisms Differentiation or discrimination Elimination Analogy and metaphor Semantic fit Classification Socratic dialogues Schema induction Inference Generalization D Emerging Codes Added During Analysis Reasons starting a new loop Resolution of conflicts New lead Disregard conflicting evidence Sense-making success Compromise Sense-making failure Accept new evidence Search failure Confusion

Sample Think-aloud Protocol with Coding Protocol (Energy Security Task) Loops Processes Conceptual Changes Cognitive Mechanisms Okay that was actually a very useful search. So let’s still take this query and look at Algeria, ‘cause obviously Algeria and Nigeria are very close… Loop 4 Focused Search for data Comparison I understand some of the keywords in the article but I don’t understand what the article… Sense-making Failure Key item extraction Okay this has to do with Algeria, southern Algeria. The minister of energy… OPEC meeting… so I am going to see what their connections are with OPEC. Building structures Key item extraction … with all the violence in Nigeria, I was expecting to find the same types of political outrage in Algeria… Instantiating structures Comparison and analogy and I’m not seeing any notice of that at all. Updating of knowledge Re-structuring

Findings – Search and Sense-making Loops
The overall search and sense-making loops followed four stages:
Task analysis
exploratory stage
focused stage
updates of knowledge representation.
Reasons for starting a new loop of search and sense-making:
Success of previous sense-making
Failure of previous sense-making
New lead
Failure of search

Findings – Cognitive Mechanisms
Participants used a two-way approach: data-driven (bottom-up) 80% structure-driven (top-down) 20%
Key item extraction and comparison were used most often.

Findings – Dealing with Conflicts
Disregard – “… I wanted that article to say something else. I have to disregard it.”
Compromise – “…it is not my understanding that this has anything to do with oil. But okay, this is a different take.”
Acceptance – “…I thought these countries had similar serious problems. But [in fact, they] seem to be relatively stable and put a lot of effort into establishing their economy.”
Confusion – “…Obviously I have no idea what this is about...”

Findings – Role of Instantiated Structures
Entities (represented as names) and key concepts, (represented as keywords) were often the basis for relevance judgments.
The relationships embedded in new information and between the new information and participants’ previous knowledge seemed to play an important role in structure building and data fitting.
Both concepts and relationships seemed to be crucial for updating knowledge.

Conclusions
The empirical think-aloud data seem to be consistent with the iterative sense-making model
Users used mostly data-driven cognitive mechanisms
Instantiated structure elements play an important role in sense-making

Implications and Future Work
Theoretical
Better understanding of sense-making processes by extending the existing models with ideas from cognition and learning
Further testing and refinement.
Empirical
Better system design based on findings from user studies.
Making design suggestions to support sense-making, not just search.

Pengyi Zhang
[email_address]
terpconnect.umd.edu/~pengyi/sensemaking/

http://www.slideshare.net	20
http://asispvc.blogspot.com	14
http://health.medicbd.com	1

Extending Sense-Making Models with Ideas from Cognition and Learning Theories

by PVC.ASIST on Sep 24, 2008

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 35

Statistics

Extending Sense-Making Models with Ideas from Cognition and Learning Theories — Presentation Transcript