This document summarizes incremental conceptual clustering, specifically Fisher's Cobweb algorithm. It begins by discussing hierarchical clustering approaches like single and complete link clustering. It then focuses on Cobweb, which uses a hill climbing search strategy to incrementally construct concept hierarchies from observations. Cobweb's category utility function measures intra-class similarity and inter-class predictiveness to evaluate candidate clusterings. The algorithm can classify observations into existing clusters, create new clusters, or merge and split clusters to construct the hierarchy.

1. Incremental Conceptual
Clustering
Kalpa Gunaratna
Reading group discussions @Kno.e.sis
Based on Fisher’s Cobweb algorithm

2. Clustering *
• Clustering is the unsupervised classification of patterns into groups.
* Jain, Anil K., M. Narasimha Murty, and Patrick J. Flynn. "Data clustering: a review." ACM computing surveys (CSUR) 31, no. 3 (1999): 264-323.
2

3. 3

4. Focus on hierarchical clustering
• Single link clustering
The distance between two clusters is the minimum of the distances between all
pairs of patterns drawn from the two clusters.
In other words, evaluates dissimilarity between two clusters as the dissimilarity
of the nearest patterns, one from each cluster.
• Complete link clustering
The distance between two clusters is the maximum of all pairs between the two
clusters.
In other words, evaluates dissimilarity between two clusters as the greatest
distance between any two patterns, one from each cluster.
• Produces compact clusters.
4

5. • Single link algorithm can extract concentric clusters as shown below
whereas complete link cannot.
5

6. • But single link algorithm suffers from chaining effect as shown below whereas
complete link does not have this effect. Therefore, researchers believe complete
link gives more useful clusters in real problems.
6

7. • Dendrogram
7

8. Our focus – Incremental Conceptual Clustering
(Cobweb) 1, 2
Given a set of observations, humans acquire concepts that organize
those observations and use them in classifying future experiences. This
type of concept formation can occur in the absence of a tutor and it can
take place despite irrelevant and incomplete information.
81. Fisher, Douglas H. "Knowledge acquisition via incremental conceptual clustering." Machine learning 2, no. 2 (1987): 139-172.
2. Gennari, John H., Pat Langley, and Doug Fisher. "Models of incremental concept formation." Artificial intelligence 40, no. 1 (1989): 11-61.

9. • Cobweb
• Uses a hill climbing search strategy having operators enabling bi-directional
travel in the space.
• Hill climbing is a classic AI search method in which one applies all operator instantiations,
compares the resulting states using an evaluation function, selects the best state, and
iterates until no more progress can be made.
• Has a function called Category Utility to decide on what action to take in the
hill climbing search.
• Computes similarity within clusters and dissimilarity between clusters.
9

10. • Category utility function
• Intra-class similarity is measured by P(Ai=Vij/Ck). - predictability
• The larger this probability, the greater the proportion of class members sharing the value
and the more predictable the value is of class members.
• Inter-class similarity is measured by P(Ck/Ai=Vij). - predictiveness
• The larger this probability, the fewer the objects in contrasting classes that share this
value and the more predictive the value is of the class.
10

11. 𝑘 𝑖 𝑗
𝑃 𝐴𝑖 = 𝑉𝑖𝑗 𝑃 𝐶 𝑘/𝐴𝑖 = 𝑉𝑖𝑗 𝑃 𝐴𝑖 = 𝑉𝑖𝑗/𝐶 𝑘
Using Bayes’ theorem
𝑘
𝑃(𝐶𝑘)
𝑖 𝑗
𝑃 𝐴𝑖 = 𝑉𝑖𝑗/𝐶𝑘 2
This is the expected number of attribute values that one
can correctly guess for an arbitrary member of class Ck.
11

12. • They further went on to say that CU as the increase in the expected
number of attribute values that can be correctly guessed, given a set
of n categories, over the expected number of correct guesses without
such knowledge.
• Divided by K so that merging, splitting, or adding nodes is taken care
of (will discuss now).
12

13. • There are four main operators in creating the hierarchy.
• Classify into an existing class.
• Create a new class.
• Combine two classes into one (merging).
• Divide a class into several classes (splitting).
• Because of the last two operations, this is normally not sensitive to
the order of items to be clustered.
13

14. • Merging
14

15. • Splitting
15

16. 16

17. 17

18. • Positive points about Incremental Conceptual Clustering (as I see)
• Unsupervised
• Input order does not matter
• Efficient – does not compute similarity/dissimilarity between all
pairs/combinations
• Good for dynamic environments
• Bi-directional search space walk in the hierarchy construction
• Try to mimic human categorization behavior
• Clustering is based on probability – not just a similarity score
18