Unsupervised learning discovers patterns in unlabeled input data through clustering. Clustering groups similar instances together based on a similarity metric without supervision. Conceptual clustering forms clusters based on semantic relationships between attributes rather than just similarity. Several algorithms were discussed, including COBWEB which uses conceptual clustering to incrementally build a hierarchical classification structure by optimizing category utility. UNIMEM also builds a hierarchical structure but stores instances at nodes to function as a type of memory.

1. 5 - Unsupervised Learning Introduction
• Unsupervised Learning
• Learner receives no explicit information about
• Introduction classification of input examples.
• Statistical Clustering • Information is implicit.
• Aim of learning process - to discover regularities in the input
• Conceptual Clustering data.
• UNIMEM • Typically, consists of partitioning instances into classes
(based on some similarity metric).
• COBWEB • ie finding clusters of instances in the instance space.
• Not surprising that unsupervised learning systems sometimes
closely resemble statistical clustering systems.
What is Clustering ? Simple Clustering
Algorithm
• Initialize
• Common problem - construction of meaningful • Set D to be the set of singleton sets such that each
classifications of observed objects or situations. set contains a unique set.
• Often known as numerical taxonomy - since it • Until D contains only 1 element, do the following:
involves production of a class hierarchy • Form a matrix of similarity values for all
(classification scheme) using a mathematical elements of D
measure of similarity over the instances. • Using some given similarity function
• Merge those elements of D which have a
maximum similarity value.
• Often known as agglomerative clustering.
• Works bottom-up - trying to build larger clusters.
• Alternative - divisive clustering.
• Works top-down (cf ID3)

2. Clustering Clustering
• Traditional techniques • Consider this example:
• Often inadequate - as they arrange objects into classes solely
on the basis of a numerical measure of object similarity.
• Only information used is that contained in the instances A B
themselves.
• Algorithms unable to take account of semantic relationships
among instance attributes or global concepts that might be of
relevance in forming a classification scheme.
• Conceptual Clustering • WE would not cluster A and B together - but would
• Idea first introduced by R S Michalski - 1980 cluster them into the 2 diamonds.
• Defined as process of constructing a concept network • Partitioning using concept membership rather
characterizing a collection of objects with nodes marked by than distance.
concepts describing object classes & links marked by the • Points are placed in the same cluster if
relationships between the classes. collectively they represent the same concept.
• This is basis of conceptual clustering
Conceptual Clustering Conceptual Clustering
• Can be regarded as:
name body-cover heart-chamber body-temp fertilisation
mammal hair four regulated internal
• Given: • Given animal bird feathers four regulated internal
• A set of objects descriptors: reptile cornified-skin imperfect-four unregulated internal
• A set of attributes to be used to characterise objects amphibian moist-skin three unregulated external
fish scales two unregulated external
• A body of background knowledge - includes problem
constraints, properties of attributes, criteria for evaluating
quality of constructed classifications. animals
• Find: • Classification
• A hierarchy of object classes hierarchy mammals/bird reptile amphibian/fish
• Each node should form a coherent concept produced:
• Compact
• Easily represented in terms of a definition or rule that mammal bird amphibian fish
has a natural interpretation for humans

3. Conceptual Clustering UNIMEM
• Lebowitz - 1987
• Michalski - 1980
• Essentially a divisive clustering algorithm
• Conjunctive conceptual clustering • Uses a decision tree structure as its basic representation.
• Concept class consists of conjunctive statements
involving relations on selected object attributes.
• If asked to classify an instance - searches down through the
• Method arranges objects into a hierarchy of classes. tree, testing attributes & returns a classification based on the
• CLUSTER/2 relevant leaf nodes.
• Used to construct classification hierarchy of a large
collection of Spanish folk songs. • If asked to update the tree so as to represent a new instance
- searches down through the tree looking for a suitable place
to add in new structure.
UNIMEM UNIMEM
• Basic clustering principle:
• Add new nodes into tree as & when they appear
to be warranted by the presented instances. • Instance matches a node if it is covered by that node (concept)
• UNIMEM actually stores each presented instance • Matching determined by testing to see what proportion of
at all nodes which cover it. the instance's attributes are associated with the node.
• Search process returns all the most specific nodes that explain
• If two instances stored at a node that are (cover) the new instance.
particularly similar - then create an extra child • UNIMEM then generalizes each node in this set as necessary
node whose definition covers the two instances in in order to account for the new instance.
question. • The new instance is then classified with all other instances
• Two instances are then relocated to this node. stored at the node.
• As new instances are processed - new nodes are
created & hierarchy grows downwards.

4. UNIMEM Algorithm UNIMEM as Memory
• UNIMEM actually stores new instances inside the tree.
• Initialize decision tree to be an empty root node. • Can thus be viewed as a type of memory.
• Apply following steps to each instance: • GBM - Generalisation-Based Memory
• Search the tree depth-first for most specific concept • Structure of hierarchy enables classes of instances to be
nodes that the instance matches. accessed much more efficiently than would be the case
• Add new instance to the tree at or below these nodes if all instances were stored in a linear memory
• Involves comparing new instance to ones already structure.
stored there & creating new subnodes if appropriate.
COBWEB COBWEB
• Incremental system for hierarchical conceptual
• Fisher - 1987 clustering
• Based on principle that a good clustering should • Carries out hill-climbing search through a space of
minimize distance between two points within a cluster & hierarchical classification schemes using operators
maximize distance between points in different clusters. which enable bidirectional travel through this space.
• Good clustering defined as: • Features of COBWEB:
• One which maximizes intra-cluster similarity & • Heuristic evaluation function to guide search.
minimizes inter-cluster similarity.
• State representation - structure of hierarchies &
representation of concepts.
• Goal of COBWEB - to find optimum tradeoff between
these two ! • Operators used to build classification schemes
• Control strategy.

5. Category Utility Representation
• Can be viewed as a function which rewards • Choice of category utility as heuristic measure dictates a
similarity of objects within same class & concept representation different to logical, typically
dissimilarity of objects in different classes. conjunctive representations used in AI.
• Probabilistic representation of {fish, amphibian, mammal}
• Gluck & Corter - 1985
Attributes Values & Probabilities
• Category utility function:
body-cover scales (0.33), moist-skin (0.33), hair (0.33)
n heart-chamber two (0.33), three (0.33), four (0.33)
∑k=1 P(Ck) [ ∑i ∑j P(Ai = Vij/Ck)2 - ∑i ∑j P(Ai = Vij)2 ]
body-temp unregulated (0.67), regulated (0.33)
n fertilisation external (0.67), internal (0.33)
• Each node in the classification tree is a probabilistic concept
which represents an object class & summarises the objects
classified under the node.
Operators Operators contd ...
• Classifying object in existing class
• Incorporation of a new object into the tree is a process of • To determine which category best "hosts" a new object,
classifying an object by descending the tree along an COBWEB tentatively places the object in each category.
appropriate path & performing one of several operations at • Partition which results from adding object to a given node
each level. is evaluated using category utility function.
• Operators include: • Node which results in the best partition (highest CU) is
• Classifying object with respect to an existing class. identified as the best existing host for the new object.
• Creating a new class. • Creating a new class
• Combining two classes into a single class. • Quality of the partition resulting from placing the object
• Dividing a class into several classes. in the best existing host is compared to partition resulting
from creation of a new singleton class containing the
object.
• Depending on which partition is best - object is placed in the
best existing class or a new class is created.

6. Example Operators contd ...
• Add "mammal":
P(C0) = 1.0 • While the first two operators are effective in many
P(scales | C0) = 0.33
...
ways - by themselves they are very sensitive to
ordering of input data.
P(C0) = 1.0
P(C1) = 0.33
P(scales | C1) = 1.0
P(C2) = 0.33
P(moist | C2) = 1.0
P(C3) = 0.33
P(hair | C3) = 1.0
• Merging & splitting operators implemented to guard
P(scales | C0) = 0.5 ... ... ... against these effects.
...
• Merging
P(C1) = 0.5 P(C2) = 0.5 • Add "bird": • Two nodes of a level are combined in hope that
P(scales | C1) = 1.0 P(moist | C2) = 1.0
... ...
P(C0) = 1.0 the resultant partition is of better quality.
P(scales | C0) = 0.25
...
• Involves creating a new node
Existing Classification Structure
P(C1) = 0.25
• Two original nodes are made children of newly
P(C2) = 0.25 P(C3) = 0.5
P(scales | C1) = 1.0 P(moist | C2) = 1.0 P(hair | C3) = 0.5 created node.
... ... ...
• Splitting
P(C4) = 0.5 P(C5) = 0.5 • Node may be deleted and its children promoted.
P(hair | C4) = 1.0 P(feath | C5) = 1.0
... ...
Merging & Splitting COBWEB Control
Operators Structure
P
COBWEB ( Object , Root of classification tree )
P
1. Update counts of the Root
• Node Merging
New node 2. IF Root is a leaf
A B THEN Return the expanded leaf to accommodate Object
ELSE Find the child of Root which best hosts Object & perform
A B one of the following:
a. Consider creating a new class & do so if appropriate
b. Consider node merging & do so if appropriate, call
P
COBWEB ( Object, Merged node )
P c. Consider node splitting & do so if appropriate, call
• Node Splitting COBWEB ( Object, Root )
A d. IF None of the above were performed
B
THEN Call COBWEB ( Object, Best child of Root )
A B

7. AutoClass
• Cheeseman et al - 1988
• Bayesian statistical technique
• Bayes' theorem - formula for combining probabilities
• Technique determines:
• Most probable number of classes
• Their probabilistic descriptions
• Probability that each object is a member of each class
• AutoClass does not do absolute partitioning of data into
classes.
• Calculates the probability of each object's membership in
each class.