A Topic map-based ontology IR system versus Clustering-based IR System: A Comparative Study in Security Domain Myongho Yi Texas Woman’s University, TX, USA, myi@twu.edu Sam Gyun Oh SungKyunKwan University, Seoul, Korea, samoh@skku.edu

Agenda 1. Related Works 2. Research Questions 3. Research Designs 4. Research Results 5. Conclusion

Background Many information organization approaches such as taxonomy, thesaurus, classification, and ontology have been attempted to provide effective searching. Among them, clustering and ontology approaches have received much attention. However, there have not been many studies which compare in terms of user performance.

Many information organization approaches such as taxonomy, thesaurus, classification, and ontology have been attempted to provide effective searching.

Among them, clustering and ontology approaches have received much attention. However, there have not been many studies which compare in terms of user performance.

Three Information Org. Approaches Term Lists: Synonym Rings* Authority Files* Glossaries/Dictionaries Gazetteers* Natural language Controlled language Weakly-structured Strongly-structured Classification & Categorization: Subject Headings Classification schemes* Taxonomies* Categorization schemes Relationship Groups : Ontologies* Semantic networks* Concept maps* Thesauri* Pick lists* (Zeng, 2005)

Clustering 2.0 Classification of data into different subtopic categories. Clustering shows related items according to their similarity. Classify related search results into topic folders Clustering 2.0 Remix clustering Shows other subtle topics

Classification of data into different subtopic categories.

Clustering shows related items according to their similarity.

Classify related search results into topic folders

Clustering 2.0

Remix clustering

Shows other subtle topics

Works in Medical Domain Less Polysemes Mainly Hierarchical Relationships Cancer Breast Cancer Prostate Cancer Colon Cancer Lung Cancer Skin Cancer … .

Less Polysemes

Mainly Hierarchical Relationships

Cancer

Breast Cancer

Prostate Cancer

Colon Cancer

Lung Cancer

Skin Cancer

… .

Norwegian Electronic Health Library United States Nat’l Lib of Medicine

How about Other Domains? Social Sciences Humanities Polysemes Bank Financial institution Rely on

Social Sciences

Humanities

Polysemes

Bank

Financial institution

Rely on

Clustering - Limitations Relevant results ? Loosely related associative relationships Same / Different category Examples Security Information Security Network Security PGP Customers (?) Valuable (?) Other Topics (?) No classified? Loosely related terms? Term Lists?

Relevant results ?

Loosely related associative relationships

Same / Different category

Examples

Security

Information Security

Network Security

PGP

Customers (?)

Valuable (?)

Other Topics (?)

Purpose of Study To measure the efficiency on representations of associative relationships To compare the user performance of our Topic Maps-based method with the Clustering-based method.

To measure the efficiency on representations of associative relationships

To compare the user performance of our Topic Maps-based method with the Clustering-based method.

Related Works Yi (2008) Compared on Ontology based System to thesaurus based system 40 subjects, 8 queries, 2 types of queries, search time and recall An ontology system showed a better recall and search time for relationship based queries Oh (2006) Compared on Topic Map-Based Korean Folk Music (Pansori) Retrieval System (TMPRS) to Current Pansori Retrieval System (CPRS) Twenty LIS Students in Korea, 7 different search tasks and own query TMPRS showed higher performance for objective and subjective measurements in general E.K.F. Dang, Luk, Ho, Chan, & Lee, (2008) Clustering algorithms Partitioning and hierarchical

Yi (2008)

Compared on Ontology based System to thesaurus based system

40 subjects, 8 queries, 2 types of queries, search time and recall

An ontology system showed a better recall and search time for relationship based queries

Oh (2006)

Compared on Topic Map-Based Korean Folk Music (Pansori) Retrieval System (TMPRS) to Current Pansori Retrieval System (CPRS)

Twenty LIS Students in Korea, 7 different search tasks and own query

TMPRS showed higher performance for objective and subjective measurements in general

E.K.F. Dang, Luk, Ho, Chan, & Lee, (2008)

Clustering algorithms

Partitioning and hierarchical

Research Questions Are there recall/precision differences between TMIR and CIR? Are there search time differences between TMIR and CIR? Are there search steps differences between TMIR and CIR?

Are there recall/precision differences between TMIR and CIR?

Are there search time differences between TMIR and CIR?

Are there search steps differences between TMIR and CIR?

Research Design Subjects Information Technology Major Students Data Collection Questionnaire Screen Recording TMIR and CIR Topic Maps-based Security Information Retrieval (TMIR) system and Clustering-based Security Information Retrieval (CIR) system.

Subjects

Information Technology Major Students

Data Collection

Questionnaire

Screen Recording

TMIR and CIR

Topic Maps-based Security Information Retrieval (TMIR) system and Clustering-based Security Information Retrieval (CIR) system.

Research Variables Independent Variables Dependent Variables Two Retrieval Systems Topic Map-Based Ontology Information Retrieval System Clustering-Based Information Retrieval System Quantitative Measurement Search steps, Search Time

Search Task Types Task # Degree of Relationships Task 1 Simple Task List all the security software 2 Complex Task Name all the Security engineers who work for Cisco 3 Complex Task Find Vendors providing security training service 4 Association and Cross Reference Related Task List all the security hardware supported by IBM Consultants 5 Association and Cross Reference Related Task List all software using RSA cryptography and find engineers who specialize in these software packages. 6 Association and Cross Reference Related Task Find security system engineers who specialize in firewall and their supervisor and sale representatives 7 Association and Cross Reference Related Task Assume that your organization is interested in security training. Who will be the right people to contact? Please provide their e-mail addresses

Ontology Development Process Code using XML Topic Maps (XTM) Identify Equivalent Relationships Identify Hierarchical Relationships Identify Associative Relationships - Same categories - Different categories List Terms by Ontology Engineer Classify/Categorize by Ontology Engineer Add Semantic Relationships by Ontology Engineer Normalize by Domain Expert Implemented by Programmer Domain Experts Identify index terms List the index terms Do not distinguish between preferred and non-preferred terms Classify terms Categorize terms One term can be in multiple categories Verify three relationships Add additional relationships

Code using XML Topic Maps (XTM)

Identify Equivalent Relationships

Identify Hierarchical Relationships

Identify Associative Relationships

- Same categories

- Different categories

Domain Experts

Identify index terms

List the index terms

Do not distinguish between preferred and non-preferred terms

Classify terms

Categorize terms

One term can be in multiple categories

Verify three relationships

Add additional relationships

Ontology Modeling Associations Works for Maintains Applied to Embed in Provides Complies with Designs Makes Provides

Associations

Works for

Maintains

Applied to

Embed in

Provides

Complies with

Designs

Makes

Provides

Embed in Cryptography embed in Hardware

Cryptography embed in Hardware

Two Retrieval Systems Compared Search for “Firewall” Clustering-based Information Retrieval System Show Related Terms

Search for “Firewall”

Clustering-based Information Retrieval System

Two Retrieval Systems Compared Clustering-based Information Retrieval System Firewall Software Listed No Related Information Provided Such as Vendor, Engineers for Firewall

Clustering-based Information Retrieval System

Two Retrieval Systems Compared Converted to the Identical Interface using Omnigator

Converted to the Identical Interface using Omnigator

Two Retrieval Systems Compared Search for “Firewall” Topic Maps-based Information Retrieval System Show Topic Types and Associative Relationships

Search for “Firewall”

Topic Maps-based Information Retrieval System

Two Retrieval Systems Compared Search for “Firewall” Topic Maps-based Information Retrieval System Shows the type of information and related information such as developers and sales person

Search for “Firewall”

Topic Maps-based Information Retrieval System

Research Results There was a significant difference in recall between the two groups. The estimate value shows the recall on TMIR was higher than CIR. The estimate value also has shown that the search time/search steps in the experimental group was less than in the control group.

There was a significant difference in recall between the two groups.

The estimate value shows the recall on TMIR was higher than CIR.

The estimate value also has shown that the search time/search steps in the experimental group was less than in the control group.

Discussion There were significant differences between the two groups and in terms of recall, precision, search time, and search steps. Overall, recall was higher when performing simple task than when performing complex tasks. Performing complex-tasks took more search time than performing simple tasks across the two groups. The control group took more total search time than the experimental group.

There were significant differences between the two groups and in terms of recall, precision, search time, and search steps.

Overall, recall was higher when performing simple task than when performing complex tasks.

Performing complex-tasks took more search time than performing simple tasks across the two groups. The control group took more total search time than the experimental group.

Conclusion This study illustrates that the positive influences of a Topic map-based ontology IR system are improved recall/precision, shorter search time and search steps for given search tasks than the clustering-based IR system. The results of this study attest to the potential of Topic Maps-based ontology to improve information retrieval system performance through better support for associative relationships between terms belonging to different hierarchies by providing explicit relationships among resources.

This study illustrates that the positive influences of a Topic map-based ontology IR system are improved recall/precision, shorter search time and search steps for given search tasks than the clustering-based IR system.

The results of this study attest to the potential of Topic Maps-based ontology to improve information retrieval system performance through better support for associative relationships between terms belonging to different hierarchies by providing explicit relationships among resources.

Q & A Myongho Yi Texas Woman’s University, TX, USA, myi@twu.edu Sam Gyun Oh SungKyunKwan University, Seoul, Korea, samoh@skku.edu

Myongho Yi

Texas Woman’s University, TX, USA, myi@twu.edu

Sam Gyun Oh

SungKyunKwan University, Seoul, Korea, samoh@skku.edu