Extending Sense-Making Models with Ideas from Cognition and Learning Theories

Extending Sense-Making Models with Ideas from Cognition and Learning Theories Pengyi Zhang Dagobert Soergel Judith L. Klavans Douglas W. Oard

Outline

What is sense-making?

Theoretical framework

A comprehensive sense-making model

Pilot testing

Conclusions

Future work

Sense-making

Sense-making is

the process of creating an understanding of a problem or task

so that further actions may be taken in an informed manner (Stefik et al., 1999).

Sense-making

Sense-making is a pre-requisite for many other tasks such as decision making and problem solving;

An important part of sense-making involves making clear the interrelated concepts and their relationships in a problem or task space.

A Sample Sense-making Scenario

Task description

An intelligence analyst is tasked to gather, analyze, and synthesize information related to Al-Bashir and to make recommendations for action in the form of a report.

Background

The Darfur conflict is a crisis in the Darfur region of western Sudan. Al-Bashir, the Sudanese president, is one of the key players in the area who is believed to have significant responsibility for continuous conflicts in the region. As part of an effort to resolve these armed conflicts, the administration needs to know as much as possible about al-Bashir in order to better negotiate with the involved parties and strategize its efforts.

Theoretical Framework

Sense-making models

Generic sense-making models

Sense-making models of intelligence analysis

Conducting research as sense-making

Organizational sense-making

Learning

Schema theory

Assimilation theory

Generative Learning Theory

Structural knowledge acquisition

Cognition

Types of conceptual changes

Cognitive mechanisms

Cognitive structures of knowledge

Design of related tools

Information extraction

Representation and visualization tools

Note-taking, analysis tools

Task-specific and task-independent tools

Sense-making Elements Processes Sensing Making sense Acti vities Mechanisms Accretion Tuning Restructuring

Inductive, data-driven

Key item extraction

Comparison

Schema induction

Generalization

Deductive, structure-driven

Definition

Specification

Elimination

Explanation

Inference

Outcomes Identification of gaps Search Exploratory search Focused search for data for structure Building structure Instantiating structure Consuming instantiated structure

Other

Metaphor

Classification

Semantic fit

Socratic dialogues

An Iterative Sense-making Model Data gap Structure gap Search: exploratory / focused Outcomes Updated knowledge Structure loop Data loop Identification of Gaps Searching for data Instantiating structure Accretion: Instantiated structure Tuning: Adapted structure Re-structuring: New structure Searching for structure Building structure Existing Knowledge Structures and their instantiations with data The iterations proceed from exploratory to focused search and sense-making. Task / Problem Decision / Solution / Task completion

Pilot Test

Participants

Data collection

Data analysis

Findings

Participants

6 information science students as surrogate subjects for intelligence analysts

Tasked with synthesizing data from a variety of sources, assessing the credibility of information, and evaluating claims based on supporting evidence

Trained in using Rosetta - a multilingual, multimedia news retrieval system

Data Collection

6 2-hour task sessions between March 2007 and May 2007, with 2 to 5 participants attending each

Data consists of 17 think-aloud protocols – participants were instructed to verbalize their thoughts as they work on the tasks

Sessions were logged and participants were interviewed after each session.

A Sample Task Scenario

Task T1: al-Bashir (Abridged Version)

Omar Hasan Ahmad al-Bashir is a Sudanese military leader, dictator, and current president of Sudan. Your task is to produce a report identifying information to assess the influence of al-Bashir as a basis for policy decisions and diplomatic actions. Requested information includes:

key figures, organizations, and countries who have been associated with al-Bashir;

his rise to power; and

groups who have resisted him and the level of success in their resistance.

Data Analysis – Coding Scheme A Processes Search Sense-making Exploratory search Gap identification Exploratory search for data Data gap Exploratory search for structure Structural gap Focused search Building structures Focused search for data Using automatically extracted results Focused search for structure Extracting relationships manually Instantiating structures Updating knowledge B Conceptual Changes Sense-making success Sense-making failure Accretion Unable to fit data into structure Tuning Re-structuring Unable to build new structure

Data Analysis – Coding Scheme C Cognitive Mechanisms Inductive mechanisms Deductive mechanisms Key item extraction Definition Comparison Specification Similarity Explanation-based mechanisms Differentiation or discrimination Elimination Analogy and metaphor Semantic fit Classification Socratic dialogues Schema induction Inference Generalization D Emerging Codes Added During Analysis Reasons starting a new loop Resolution of conflicts New lead Disregard conflicting evidence Sense-making success Compromise Sense-making failure Accept new evidence Search failure Confusion

Sample Think-aloud Protocol with Coding Protocol (Energy Security Task) Loops Processes Conceptual Changes Cognitive Mechanisms Okay that was actually a very useful search. So let’s still take this query and look at Algeria, ‘cause obviously Algeria and Nigeria are very close… Loop 4 Focused Search for data Comparison I understand some of the keywords in the article but I don’t understand what the article… Sense-making Failure Key item extraction Okay this has to do with Algeria, southern Algeria. The minister of energy… OPEC meeting… so I am going to see what their connections are with OPEC. Building structures Key item extraction … with all the violence in Nigeria, I was expecting to find the same types of political outrage in Algeria… Instantiating structures Comparison and analogy and I’m not seeing any notice of that at all. Updating of knowledge Re-structuring

Findings – Search and Sense-making Loops

The overall search and sense-making loops followed four stages:

Task analysis

exploratory stage

focused stage

updates of knowledge representation.

Reasons for starting a new loop of search and sense-making:

Success of previous sense-making

Failure of previous sense-making

New lead

Failure of search

Findings – Cognitive Mechanisms

Participants used a two-way approach: data-driven (bottom-up) 80% structure-driven (top-down) 20%

Key item extraction and comparison were used most often.

Findings – Dealing with Conflicts

Disregard – “… I wanted that article to say something else. I have to disregard it.”

Compromise – “…it is not my understanding that this has anything to do with oil. But okay, this is a different take.”

Acceptance – “…I thought these countries had similar serious problems. But [in fact, they] seem to be relatively stable and put a lot of effort into establishing their economy.”

Confusion – “…Obviously I have no idea what this is about...”

Findings – Role of Instantiated Structures

Entities (represented as names) and key concepts, (represented as keywords) were often the basis for relevance judgments.

The relationships embedded in new information and between the new information and participants’ previous knowledge seemed to play an important role in structure building and data fitting.