More Related Content
Similar to Chapter 4 Clustering I02142018 Intro
Similar to Chapter 4 Clustering I02142018 Intro (20)
More from TawnaDelatorrejs
More from TawnaDelatorrejs (20)
Recently uploaded
Recently uploaded (20)
Chapter 4 Clustering I02142018 Intro
- 1. Chapter 4: Clustering I 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› What is Cluster Analysis? Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups
- 2. Inter-cluster distances are maximized Intra-cluster distances are minimized 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Applications of Cluster Analysis Understanding Group related documents for browsing, group genes and proteins that have similar functionality, or group stocks with similar price fluctuations Summarization Reduce the size of large data sets
- 3. Clustering precipitation in Australia 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› What is not Cluster Analysis? Simple segmentation Dividing students into different registration groups alphabetically, by last name Results of a query Groupings are a result of an external specification Clustering is a grouping of objects based on the data Supervised classification Have class label information Association Analysis Local vs. global connections 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Notion of a Cluster can be Ambiguous
- 4. How many clusters? Four Clusters
- 5. Two Clusters
- 6. Six Clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Types of Clusterings A clustering is a set of clusters Important distinction between hierarchical and partitional sets of clusters Partitional Clustering A division of data objects into non-overlapping subsets (clusters) such that each data object is in exactly one subset Hierarchical clustering A set of nested clusters organized as a hierarchical tree 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Partitional Clustering
- 7. Original Points A Partitional Clustering 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Hierarchical Clustering Traditional Hierarchical Clustering Non-traditional Hierarchical Clustering Non-traditional Dendrogram Traditional Dendrogram 02/14/2018 Introduction to Data Mining, 2nd Edition
- 8. ‹#› Types of Clusters Well-separated clusters Center-based clusters Contiguous clusters Density-based clusters Property or Conceptual Described by an Objective Function 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Types of Clusters: Well-Separated Well-Separated Clusters: A cluster is a set of points such that any point in a cluster is closer (or more similar) to every other point in the cluster than to any point not in the cluster. 3 well-separated clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#›
- 9. Types of Clusters: Center-Based Center-based A cluster is a set of objects such that an object in a cluster is closer to the “center” of a cluster, than to the center of any other cluster The center of a cluster is a centroid. The average of all the points in the cluster is a medoid 4 center-based clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Types of Clusters: Contiguity-Based Contiguous Cluster (Nearest neighbor or Transitive) A point in a cluster is closer (or more similar) to one or more other points in the cluster than to any point not in the cluster. 8 contiguous clusters
- 10. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Types of Clusters: Density-Based Density-based A cluster is a dense region of points, which is separated by low - density regions, from other regions of high density. Used when the clusters are irregular or intertwined, and when noise and outliers are present. 6 density-based clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Types of Clusters: Conceptual Clusters Shared Property or Conceptual Clusters Finds clusters that share some common property or represent a particular concept. . 2 Overlapping Circles
- 11. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Types of Clusters: Objective Function Clusters Defined by an Objective Function Finds clusters that minimize or maximize an objective function. Enumerate all possible ways of dividing the points into clusters and evaluate the `goodness' of each potential set of clusters by using the given objective function. (NP Hard) Can have global or local objectives. Hierarchical clustering algorithms typically have local objectives Partitional algorithms typically have global objectives A variation of the global objective function approach is to fit the data to a parameterized model. Parameters for the model are determined from the data. Mixture models assume that the data is a ‘mixture' of a number of statistical distributions. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Characteristics of the Input Data Are Important Type of proximity or density measure Central to clustering Depends on data and application Data characteristics that affect proximity and/or density are Dimensionality Sparseness Attribute type Special relationships in the data
- 12. For example, autocorrelation Distribution of the data Noise and Outliers Often interfere with the operation of the clustering algorithm 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Clustering Algorithms K-means and its variants Hierarchical clustering Density-based clustering 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› K-means Clustering Partitional clustering approach Number of clusters, K, must be specified Each cluster is associated with a centroid (center point) Each point is assigned to the cluster with the closest centroid The basic algorithm is very simple
- 13. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Example of K-means Clustering 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Example of K-means Clustering 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› K-means Clustering – Details Initial centroids are often chosen randomly.
- 14. Clusters produced vary from one run to another. The centroid is (typically) the mean of the points in the cluster. ‘Closeness’ is measured by Euclidean distance, cosine similarity, correlation, etc. K-means will converge for common similarity measures mentioned above. Most of the convergence happens in the first few iterations. Often the stopping condition is changed to ‘Until relatively few points change clusters’ Complexity is O( n * K * I * d ) n = number of points, K = number of clusters, I = number of iterations, d = number of attributes 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Evaluating K-means Clusters Most common measure is Sum of Squared Error (SSE) For each point, the error is the distance to the nearest cluster To get SSE, we square these errors and sum them. x is a data point in cluster Ci and mi is the representative point for cluster Ci can show that mi corresponds to the center (mean) of the cluster Given two sets of clusters, we prefer the one with the smallest error One easy way to reduce SSE is to increase K, the number of clusters A good clustering with smaller K can have a lower SSE than a poor clustering with higher K
- 15. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Two different K-means Clusterings Sub-optimal Clustering Optimal Clustering Original Points 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Limitations of K-means K-means has problems when clusters are of differing Sizes Densities Non-globular shapes K-means has problems when the data contains outliers. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Limitations of K-means: Differing Sizes
- 16. Original Points K-means (3 Clusters) 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Limitations of K-means: Differing Density Original Points K-means (3 Clusters) 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Limitations of K-means: Non-globular Shapes Original Points K-means (2 Clusters)
- 17. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Overcoming K-means Limitations Original PointsK-means Clusters One solution is to use many clusters. Find parts of clusters, but need to put together. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Overcoming K-means Limitations Original PointsK-means Clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Overcoming K-means Limitations
- 18. Original PointsK-means Clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Importance of Choosing Initial Centroids 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Importance of Choosing Initial Centroids 02/14/2018 Introduction to Data Mining, 2nd Edition
- 19. ‹#› Importance of Choosing Initial Centroids … 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Importance of Choosing Initial Centroids … 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Problems with Selecting Initial Points If there are K ‘real’ clusters then the chance of selecting one centroid from each cluster is small. Chance is relatively small when K is large If clusters are the same size, n, then For example, if K = 10, then probability = 10!/1010 = 0.00036 Sometimes the initial centroids will readjust themselves in
- 20. ‘right’ way, and sometimes they don’t Consider an example of five pairs of clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› 10 Clusters Example Starting with two initial centroids in one cluster of each pair of clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› 10 Clusters Example Starting with two initial centroids in one cluster of each pair of clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#›
- 21. Solution s to Initial Centroids Problem Multiple runs Helps, but probability is not on your side Sample and use hierarchical clustering to determine initial centroids Select more than k initial centroids and then select among these initial centroids Select most widely separated Generate a larger number of clusters and then perform a hierarchical clustering 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Hierarchical Clustering Produces a set of nested clusters organized as a hierarchical tree Can be visualized as a dendrogram A tree like diagram that records the sequences of merges or splits
- 22. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Strengths of Hierarchical Clustering Do not have to assume any particular number of clusters Any desired number of clusters can be obtained by ‘cutting’ the dendrogram at the proper level They may correspond to meaningful taxonomies Example in biological sciences (e.g., animal kingdom) 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Hierarchical Clustering Two main types of hierarchical clustering Agglomerative:
- 23. Start with the points as individual clusters At each step, merge the closest pair of clusters until only one cluster (or k clusters) left Divisive: Start with one, all-inclusive cluster At each step, split a cluster until each cluster contains an individual point (or there are k clusters) Traditional hierarchical algorithms use a similarity or distance matrix Merge or split one cluster at a time 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Agglomerative Clustering Algorithm Most popular hierarchical clustering technique Basic algorithm is straightforward Compute the proximity matrix Let each data point be a cluster
- 24. Repeat Merge the two closest clusters Update the proximity matrix Until only a single cluster remains Key operation is the computation of the proximity of two clusters Different approaches to defining the distance between clusters distinguish the different algorithms 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Proximity Matrix 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Starting Situation
- 25. Start with clusters of individual points and a proximity matrix
- 26. p1 p3 p5 p4 p2 p1 p2 p3 p4 p5 . . . . . . Proximity Matrix 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#›
- 27. Intermediate Situation After some merging steps, we have some clusters C1 C4 C2 C5 C3 C2 C1 C1 C3 C5 C4 C2 C3
- 28. C4 C5 Proximity Matrix 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Intermediate Situation We want to merge the two closest clusters (C2 and C5) and update the proximity matrix.
- 30. Proximity Matrix 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› After Merging The question is “How do we update the proximity matrix?”
- 31. C1 C4 C2 U C5 C3 ? ? ? ? ? ? ? C2 U C5 C1 C1 C3 C4 C2 U C5 C3 C4
- 32. Proximity Matrix 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› How to Define Inter-Cluster Distance
- 33. p1 p3 p5 p4 p2 p1 p2 p3 p4 p5 . . . . . . Similarity? MIN MAX Group Average Distance Between Centroids Other methods driven by an objective function Ward’s Method uses squared error
- 34. Proximity Matrix 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› How to Define Inter-Cluster Similarity
- 36. Proximity Matrix MIN MAX Group Average Distance Between Centroids Other methods driven by an objective function Ward’s Method uses squared error 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› How to Define Inter-Cluster Similarity
- 38. Proximity Matrix MIN MAX Group Average Distance Between Centroids Other methods driven by an objective function Ward’s Method uses squared error 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› How to Define Inter-Cluster Similarity
- 41. Proximity Matrix MIN MAX Group Average Distance Between Centroids Other methods driven by an objective function Ward’s Method uses squared error 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› How to Define Inter-Cluster Similarity
- 43. Proximity Matrix MIN MAX Group Average Distance Between Centroids Other methods driven by an objective function Ward’s Method uses squared error 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› MIN or Single Link Proximity of two clusters is based on the two closest points in the different clusters Determined by one pair of points, i.e., by one link in the proximity graph Example:
- 44. Distance Matrix: 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Hierarchical Clustering: MIN Nested Clusters Dendrogram 1 2 3 4 5 6 1
- 45. 2 3 4 5 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Strength of MIN Original Points Six Clusters Can handle non-elliptical shapes 02/14/2018 Introduction to Data Mining, 2nd Edition
- 46. ‹#› Limitations of MIN Original Points Two Clusters Sensitive to noise and outliers Three Clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› MAX or Complete Linkage Proximity of two clusters is based on the two most distant points in the different clusters Determined by all pairs of points in the two clusters Distance Matrix:
- 47. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Hierarchical Clustering: MAX Nested Clusters Dendrogram 1 2 3 4 5 6 1 2
- 48. 5 3 4 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Strength of MAX Original Points Two Clusters Less susceptible to noise and outliers 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Limitations of MAX
- 49. Original Points Two Clusters Tends to break large clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Group Average Proximity of two clusters is the average of pairwise proximity between points in the two clusters. Need to use average connectivity for scalability since total proximity favors large clusters Distance Matrix:
- 50. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Hierarchical Clustering: Group Average Nested Clusters Dendrogram 1 2 3 4 5 6 1
- 51. 2 5 3 4 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Hierarchical Clustering: Group Average Compromise between Single and Complete Link Strengths Less susceptible to noise and outliers Limitations Biased towards globular clusters 02/14/2018 Introduction to Data Mining, 2nd Edition
- 52. ‹#› Cluster Similarity: Ward’s Method Similarity of two clusters is based on the increase in squared error when two clusters are merged Similar to group average if distance between points is distance squared Less susceptible to noise and outliers Biased towards globular clusters Hierarchical analogue of K-means Can be used to initialize K-means 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Hierarchical Clustering: Comparison Group Average Ward’s Method
- 56. 4 5 6 1 2 3 4 5 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Hierarchical Clustering: Time and Space requirements O(N2) space since it uses the proximity matrix. N is the number of points. O(N3) time in many cases There are N steps and at each step the size, N2, proximity
- 57. matrix must be updated and searched Complexity can be reduced to O(N2 log(N) ) time with some cleverness 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Hierarchical Clustering: Problems and Limitations Once a decision is made to combine two clusters, it cannot be undone No global objective function is directly minimized Different schemes have problems with one or more of the following: Sensitivity to noise and outliers Difficulty handling clusters of different sizes and non-globular shapes Breaking large clusters
- 58. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Cluster Validity For supervised classification we have a variety of measures to evaluate how good our model is Accuracy, precision, recall For cluster analysis, the analogous question is how to evaluate the “goodness” of the resulting clusters? But “clusters are in the eye of the beholder”! Then why do we want to evaluate them? To avoid finding patterns in noise To compare clustering algorithms To compare two sets of clusters To compare two clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#›
- 59. Clusters found in Random Data Random Points K-means Complete Link 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Determining the clustering tendency of a set of data, i.e., distinguishing whether non-random structure actually exists in the data. Comparing the results of a cluster analysis to externally known results, e.g., to externally given class labels. Evaluating how well the results of a cluster analysis fit the data without reference to external information. - Use only the data Comparing the results of two different sets of cluster analyses to determine which is better. Determining the ‘correct’ number of clusters.
- 60. For 2, 3, and 4, we can further distinguish whether we want to evaluate the entire clustering or just individual clusters. Different Aspects of Cluster Validation 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Numerical measures that are applied to judge various aspects of cluster validity, are classified into the following three types. External Index: Used to measure the extent to which cluster labels match externally supplied class labels. Entropy Internal Index: Used to measure the goodness of a clustering structure without respect to external information. Sum of Squared Error (SSE) Relative Index: Used to compare two different clusterings or clusters. Often an external or internal index is used for this functi on, e.g., SSE or entropy Sometimes these are referred to as criteria instead of indices However, sometimes criterion is the general strategy and index
- 61. is the numerical measure that implements the criterion. Measures of Cluster Validity 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Two matrices Proximity Matrix Ideal Similarity Matrix One row and one column for each data point An entry is 1 if the associated pair of points belong to the same cluster An entry is 0 if the associated pair of points belongs to different clusters Compute the correlation between the two matrices Since the matrices are symmetric, only the correlation between n(n-1) / 2 entries needs to be calculated. High correlation indicates that points that belong to the same cluster are close to each other. Not a good measure for some density or contiguity based clusters. Measuring Cluster Validity Via Correlation
- 62. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Measuring Cluster Validity Via Correlation Correlation of ideal similarity and proximity matrices for the K- means clusterings of the following two data sets. Corr = -0.9235 Corr = -0.5810 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Order the similarity matrix with respect to cluster labels and inspect visually. Using Similarity Matrix for Cluster Validation
- 63. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Using Similarity Matrix for Cluster Validation Clusters in random data are not so crisp K-means 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Using Similarity Matrix for Cluster Validation Clusters in random data are not so crisp Complete Link
- 64. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Clusters in more complicated figures aren’t well separated Internal Index: Used to measure the goodness of a clustering structure without respect to external information SSE SSE is good for comparing two clusterings or two clusters (average SSE). Can also be used to estimate the number of clusters Internal Measures: SSE 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Internal Measures: SSE SSE curve for a more complicated data set
- 65. SSE of clusters found using K-means 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Correlation of ideal similarity and proximity matrices for the K- means clusterings of the following two data sets. Statistical Framework for Correlation Corr = -0.9235 Corr = -0.5810 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Cluster Cohesion: Measures how closely related are objects in a
- 66. cluster Example: SSE Cluster Separation: Measure how distinct or well-separated a cluster is from other clusters Example: Squared Error Cohesion is measured by the within cluster sum of squares (SSE) Separation is measured by the between cluster sum of squares Where |Ci| is the size of cluster i Internal Measures: Cohesion and Separation 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#›
- 67. Internal Measures: Cohesion and Separation Example: SSE BSS + WSS = constant 1 2 3 4 5 m1 m2 m
- 68. K=2 clusters: K=1 cluster: 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› A proximity graph based approach can also be used for cohesion and separation. Cluster cohesion is the sum of the weight of all links within a cluster. Cluster separation is the sum of the weights between nodes in the cluster and nodes outside the cluster. Internal Measures: Cohesion and Separation
- 70. cohesion separation 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› “The validation of clustering structures is the most difficult and frustrating part of cluster analysis. Without a strong effort in this direction, cluster analysis will remain a black art accessible only to those true believers who have experience and great courage.” Algorithms for Clustering Data, Jain and Dubes Final Comment on Cluster Validity
- 71. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Discovered Clusters Industry Group 1 Applied-Matl-DOWN,Bay-Network-Down,3-COM-DOWN, Cabletron-Sys-DOWN,CISCO-DOWN,HP-DOWN, DSC-Comm-DOWN,INTEL-DOWN,LSI-Logic-DOWN, Micron-Tech-DOWN,Texas-Inst-Down,Tellabs-Inc-Down, Natl-Semiconduct-DOWN,Oracl-DOWN,SGI-DOWN, Sun-DOWN Technology1-DOWN 2 Apple-Comp-DOWN,Autodesk-DOWN,DEC-DOWN, ADV-Micro-Device-DOWN,Andrew-Corp-DOWN, Computer-Assoc-DOWN,Circuit-City-DOWN, Compaq-DOWN, EMC-Corp-DOWN, Gen-Inst-DOWN, Motorola-DOWN,Microsoft-DOWN,Scientific-Atl-DOWN Technology2-DOWN 3 Fannie-Mae-DOWN,Fed-Home-Loan-DOWN,
- 73. JOURNAL OF APPLIED BEHAVIOR ANALYSIS WHAT MAKES EXTINCTION WORK: AN ANALYSIS OF PROCEDURAL FORM AND FUNCTION BRIAN A. IWATA THE UNIVERSITY OF FLORIDA GARY M. PACE THE KENNEDY INSTITUTE, BALTIMORE, MARYLAND GLYNNIS EDWARDS COWDERY WE CARE, INC., BERKELEY, CALIFORNIA AND RAYMOND G. MILTENBERGER NORTH DAKOTA STATE UNIVERSITY We examined methods for determining how extinction should be applied to different functions of self-injurious behavior (SIB). Assessment data indicated that
- 74. the head banging of 3 children with developmental disabilities was maintained by different reinforcement contingencies: One subject's SIB was positively reinforced by attention from adults, the 2nd subject's SIB was negatively reinforced by escape from educational tasks, and the 3rd subject's SIB appeared to be automatically reinforced or "self-stimulatory" in nature. Three functional variations of extinction-EXT (attention), EXT (escape), and EXT (sensory)-were evaluated, and each subject was exposed to at least two of these variations in reversal or multiple baseline designs. Reductions in SIB were observed only when implementation of "extinction" involved the discontinuation of reinforcement previously shown to be responsible for maintaining the behavior. These results highlight important differences among treatment techniques based on the same behavioral principle (extinction) when applied to topo- graphically similar but functionally dissimilar responses, and further illustrate the practical impli- cations of a functional analysis of behavior disorders for designing, selecting, and classifying ther- apeutic interventions. DESCRIPTORS: extinction, functional analysis, self-injurious
- 75. behavior Research on the functional analysis of severe behavior disorders has produced a variety of inter- ventions based on the modification of antecedent events that occasion behavior problems as well as the consequences that maintain them (see Iwata, Vollmer, & Zarcone, 1990, and Mace, Lalli, & Lalli, 1991, for recent reviews). Much of the em- phasis in treatment has been on strengthening new stimulus-response-consequence relationships. By contrast, relatively little attention has been paid to eliminating reinforcement that maintained the be- This research was supported in part by a grant from the Developmental Disabilities Planning Council and the Florida Department of Health and Rehabilitative Services. Reprints may be obtained from Brian Iwata, Psychology Department, University of Florida, Gainesville, Florida 3261 1. havior problem in the first place, even though many of the interventions described in the literature im- plicitly contained such a provision. That is, pun- ishment of Response A is almost always combined
- 76. with the cessation of reinforcement; likewise, treat- ment involving reinforcement of Response B as a replacement for Response A usually coincides with the termination of reinforcement for Response A. Moreover, data from several recent studies suggest that the effects of reinforcement-based interventions may be limited unless extinction (withholding the behavior's maintaining reinforcers) is included as part of the treatment program. It has been shown, for example, that interventions based on the de- velopment of communicative responses (Carr & Durand, 1985) and on behavioral momentum (Mace & Belfiore, 1990) produced either mixed 131 1994127,131-144 NUMBER I (SPRING 1994) BRIAN A. IWATA et al. results (Wacker et al., 1990) or no treatment effect (Zarcone, Iwata, Hughes, & Vollmer, 1993) when implemented without extinction, and that the crit-
- 77. ical feature of procedures such as differential rein- forcement of other behavior (DRO) contingencies appears to be the extinction component rather than the reinforcement component (Mazaleski, Iwata, Vollmer, Zarcone, & Smith, 1993). The assessment and treatment of self-injurious behavior (SIB) provide an important context for the examination of extinction, because it has been demonstrated that SIB can be maintained through a variety of operant mechanisms. Nevertheless, be- cause the "discontinuation of reinforcement" for behavior problems in general often involves ces- sation of ongoing events, this procedure has become so common that some textbooks define extinction solely through reference to "ignoring," time-out, and other examples of stimulus termination (e.g., LaVigna & Donnellan, 1986). However, research has shown that procedural time-out can serve dif- ferent functions, not all of which are behavior re- ducing (Plummer, Baer, & LeBlanc, 1977; Solnick, Rincover, & Peterson, 1977), and the same is true of procedural approaches to defining extinction. Ignoring misbehavior may represent extinction in some cases but not in others; conversely, the correct
- 78. application of extinction may require termination of events in some cases but continuation in others. The procedures that define extinction in a given situation are determined by the specific nature of the reinforcement to be "discontinued." In cases in which SIB was maintained by social- positive reinforcement in the form of adult atten- tion, extinction consisted of withholding attention or terminating it contingent on the occurrence of SIB (e.g., Day, Rea, Schussler, Larsen, & Johnson, 1988; Lovaas & Simmons, 1969). By contrast, SIB maintained through social-negative reinforcement in the form of escape from task demands has been extinguished by preventing escape; in other words, continuing and not terminating the ongoing sit- uation (e.g., Iwata, Pace, Kalsher, Cowdery, & Cataldo, 1990; Repp, Felce, & Barton, 1988). Finally, SIB apparently maintained by nonsocial, automatic reinforcement (e.g., sensory stimulation) has shown extinction-like decreases when the in- dividuals wore equipment that allowed the behav- ior to occur, but attenuated its consequences (e.g., Rincover & Devany, 1982). Thus, at least three
- 79. functional variations of extinction have been used as treatment for SIB, each designed to terminate a different source of reinforcement, and each ame- nable to a number of procedural modifications and descriptive labels. Although extinction procedures are clearly unique from the standpoint of both form (specific therapist actions) and function (the maintaining contingen- cies to which they apply), these variations are not well differentiated through existing terminology. Planned ignoring, the label perhaps most often applied to extinction of attention-maintained be- havior (Nelson & Rutherford, 1983), provides an adequate description of the therapist's response, but it does not describe the underlying behavioral pro- cess (extinction), nor does it identify the source of reinforcement being withheld (attention). Escape extinction, a term used to describe extinction of escape-maintained behavior (Iwata, Pace, Kalsher, Cowdery, & Cataldo, 1990), specifies the relevant process as well as the reinforcer, but not the pro- cedure. Finally, sensory extinction, which refers to a variety of techniques designed to attenuate stim- ulation directly produced by a response (Rincover,
- 80. 1978; Rincover, Cook, Peoples, & Packard, 1979), has the same descriptive characteristics as escape extinction (it specifies process and reinforcer, but not procedure); in addition, the wording is awk- ward because nothing "sensory" is extinguished. In an attempt to promote brevity as well as con- sistency of terminology, we have adopted through- out this paper the convention of referring to dif- ferentfunctional variations of extinction by using the abbreviation "EXT" followed in parentheses by the source of reinforcement withheld, as in EXT (attention), EXT (escape), and EXT (sensory). We chose "EXT" (attention) over "attention EXT" because the former places emphasis on behavioral process, with the reinforcer incidental, whereas the latter connotes that attention is extinguished. This terminology is still incomplete because it does not differentiate procedural variations within 132 VARIATIONS OF EXTINCTION
- 81. a given functional class of extinction techniques. For example, EXT (attention) may take different forms depending on the situational context in which it is applied. If the target behavior occurs in the absence of ongoing social interaction, EXT (atten- tion) requires no response from the therapist and simply consists of withholding attention that pre- viously followed the behavior. A more complicated situation exists if the target behavior occurs while the therapist is interacting with the client. In this case, withholding reinforcement [EXT (attention)] would consist of terminating the interaction.' Be- cause similar variations are also characteristic of both EXT (escape) and EXT (sensory), we did not attempt to specify any particular procedure by way of descriptive label; this practice is consistent with terms such as DRO, time-out, punishment, and so forth, all of which require further elaboration. A reasonable conclusion based on descriptions of behavior, procedure, and outcome reported in the above studies is that extinction designed for SIB serving one function may have little or no therapeutic effect on SIB serving a different func-
- 82. tion. Given the variations of extinction described previously and the maintaining contingencies for which they might be used, it is possible to construct a contingency matrix such as that found in Figure 1, which allows predictions about behavioral out- come. For example, elimination of social reinforce- ment for SIB using EXT (attention) or EXT (es- cape) will not interfere with the delivery of sensory (automatic) reinforcement, and EXT (sensory) would have no effect on social sources of reinforce- ment. Most seriously, EXT (attention) applied to escape-maintained SIB (i.e., stimulus removal con- tingent on escape behavior) and, conversely, EXT (escape) applied to attention-maintained SIB (i.e., stimulus continuation contingent on attention-get- ting behavior) may be countertherapeutic and se- riously exacerbate the behavior problem. At the In this example, it is unclear if the subsequent change in behavior reflects withholding of contingent reinforcement that maintained SIB (extinction) or removal of ongoing re- inforcement (time-out). In either case, and from a practical standpoint, the only means of not reinforcing the behavior consists of terminating interaction.
- 83. FUNCTIONAL VARIATIONS OF EXTINCTION 0 LL 0 z LU 0 z 1 z 0 0 z Ir.2 f ifis_---_..Treatment Figure 1. Expected outcomes when functional variations of extinction are applied to different maintaining contingen- cies for SIB. present time, evidence directly supporting these pre- dictions can be found in only a few studies. Research on the treatment of self-injurious escape behavior,
- 84. for example, sometimes has included a baseline condition in which time-out from educational tasks was made contingent on SIB (Iwata, Pace, Kalsher, Cowdery, & Cataldo, 1990; Repp et al., 1988; Steege et al., 1990). In effect, this condition pro- cedurally amounted to EXT (attention) but was associated with increases in escape-maintained SIB. In the present study, we provide an empirical evaluation of the key predictions found in Figure 1 by conducting a comparative analysis of extinc- tion techniques. The relative effects of three func- tional variations of extinction were examined when applied to the same topography of SIB maintained by three different contingencies of reinforcement. METHOD Subjects and Setting Subjects were selected for participation based on (a) their having similar topographies of SIB, and (b) the outcomes of functional analysis baselines indicating that their SIB was maintained by dif- ferent sources of reinforcement. Three children with
- 85. developmental disabilities who met these criteria were included as participants. All of them exhibited head banging that produced contusions or lacera- tions on the head and face. Donnie, a 7-year-old boy with severe mental retardation, was ambulatory 133 BRIAN A. IWATA et al. and could feed himself. He did not follow vocal instructions, although he had a fairly good imitative repertoire, and he communicated his needs by pointing. Donnie's parents could not identify any situations that were predictive of either high or low levels of SIB. His previous treatments consisted of response interruption, redirection to another activ- ity, time-out, and differential reinforcement. Jack, a 12-year-old boy with severe mental retardation, was nonambulatory but could move his wheelchair over short distances, feed himself, and manipulate small objects. He followed a few instructions and communicated primarily through idiosyncratic ges-
- 86. ture (i.e., reaching and making a high-pitched vocal noise). Jack's parents and teachers reported that his SIB seemed to occur as part of a tantrum, although they could not reliably describe the types of situ- ations in which tantrums occurred. His previous treatments consisted of time-out and differential reinforcement. Millie, an 8-year-old girl with mod- erate mental retardation, was ambulatory, could feed herself, and occasionally followed simple in- structions. Although she did not exhibit any ex- pressive language, she appeared to be socially re- sponsive to adults (e.g., she approached adults when they were nearby, smiled at them, etc.). Millie's parents indicated that her SIB occurred "most of the time," but became worse when she did not "get her way." Her previous treatments consisted of verbal reprimands, response interruption, man- ual restraint, redirection, and differential reinforce- ment. She was also placed in a hard seizure helmet as a means of protection when her SIB was deemed to be "uncontrollable." Sessions were usually conducted individually in therapy rooms containing one-way observation win- dows, but occasionally (due to scheduling neces-
- 87. sities) were run in a large group area where two or three other clients and one or two other staff mem- bers were present but located in a different part of the room (i.e., at least 6 m away). This periodic change in location seemed to have no effect on the subjects' behavior. Sessions were 15 min in duration and usually were conducted 5 days per week, with four to eight sessions daily and at least 15-min breaks between sessions. Response Measurement and Reliability SIB was defined as any audible contact between the head or face and either a fist or an object (e.g., furniture or walls). Data were recorded using one of two methods: (a) on paper during continuous 10-s intervals, which were cued by cassette tape; or (b) on a hand-held computer (Panasonic Model RL-H1800). All data were converted into per- centage of 10-s intervals during which one or more instances of SIB occurred. Data were also collected on a variety of experimenter behaviors (delivery of instructions or attention, withdrawal of instructions or attention, placement and removal of apparatus) as a means of monitoring procedural consistency;
- 88. these data indicated that experimenter compliance exceeded 90% across all categories. Interobserver agreement was assessed by having a second observer simultaneously but independently record data during 23% of all sessions. Agreement percentages were calculated based on interval-by- interval comparison of observers' records by divid- ing the number of agreements by the number of agreements plus disagreements and multiplying by 100%. Mean agreement was 92% across subjects, with all values exceeding 86%. Experimental Designs and Sequence During the initial baseline, subjects were exposed to a series of assessment conditions in a multiele- ment design (Sidman, 1960; Ulman & Sulzer-Aza- roff, 1975) to identify the maintaining variables for their SIB. Subsequently, each subject was ex- posed to two or more functional variations of ex- tinction by way of reversal or multiple baseline designs. Because the specific conditions and their order of presentation were determined by subjects' baseline performances, more complete details are
- 89. provided in the Results section for each subject. Functional Analysis Baseline Subjects were exposed to four conditions based on Iwata, Dorsey, Slifer, Bauman, and Richman (1982). A brief description of each condition is provided here. Attention. This condition was designed to test 134 VARIATIONS OF EXTINCTION for positive reinforcement (attention) as a main- taining variable for SIB. An experimenter initially placed the subject in contact with toys that were in the room, then proceeded to ignore the subject while seeming to do paperwork. Contingent on the occurrence of SIB, the experimenter expressed con- cern (e.g., "Stop. Don't do that; you'll hurt your- self.") and briefly interrupted the response.
- 90. Demand. This condition was a test for negative reinforcement (escape from demands) as a main- taining variable. An experimenter presented aca- demic tasks (e.g., object identification, sorting, drawing, and other tasks requiring simple motor responses) comparable to those found in the sub- ject's individual educational plan. Tasks were pre- sented in a discrete-trial format approximately once every 30 s. The experimenter used modeling and physical guidance as supplementary prompts to produce compliance, delivered praise contingent on correct responses, and implemented a 30-s time- out contingent on the occurrence of SIB. The time- out consisted of removing the materials from the table and turning away from the subject. Alone. By restricting access to social and material sources of stimulation, this condition provided a test for SIB maintained by nonsocial (automatic) reinforcement. The subject was observed while alone in the room (which was empty except for a chair or couch). Play. This condition served as a control. The
- 91. experimenter provided continuous access to toys, delivered praise approximately every 30 s contin- gent on the absence of SIB, and ignored occurrences of SIB. Extinction Conditions Three variations of extinction were evaluated, each designed to discontinue a different source of reinforcement for SIB. An attempt was made to implement each type of extinction with each sub- ject, but this was not always possible if (a) little or no SIB occurred in a given baseline condition, or (b) the baseline condition itself was not amenable to implementation of the procedure as described. For example, it was not possible to prevent escape from tasks (i.e., to withhold negative reinforce- ment) during the alone baseline, because no tasks were presented initially. EXT (attention). This variation of extinction was designed to discontinue positive reinforcement in the form of attention delivered by the experi- menter. EXT (attention) already was a component
- 92. of three of the baseline conditions: during the de- mand condition, attention was withdrawn contin- gent on SIB; during the alone condition, no adult was present to deliver any attention; and during the play condition, attention was withheld contin- gent on SIB. Thus, the attention condition was the only baseline on which EXT (attention) could be introduced as a new manipulation. This extinction contingency consisted of having the experimenter ignore (not attend to) SIB, and was combined with experimenter attention for the absence of SIB, which was delivered on a 30-s DRO schedule. Occur- rences of SIB reset the DRO interval. EXT (escape). This variation of extinction dis- continued negative reinforcement in the form of escape from tasks. It was implemented during the demand condition, which was the only condition containing a task requirement. During EXT (es- cape), learning trials continued as during baseline but were not terminated contingent on SIB. Instead, the experimenter physically guided the subject to comply with the instruction and continued the ses- sion according to schedule. It has been noted pre- viously (Iwata, Pace, Kalsher, Cowdery, & Cataldo,
- 93. 1990) that the physical guidance component of this intervention may represent a punishment con- tingency. However, (a) guidance was the inevitable consequence of noncompliance even during base- line, and (b) behavior change observed following implementation of this procedure in the above study as well as in others (e.g., Goh & Iwata, 1994; Repp et al., 1988) showed either temporary burst- ing or gradual decreases consistent with extinction processes. EXT (sensory). This variation of extinction was designed to attenuate the sensory consequences pro- duced by subjects' head banging, and could be implemented across any of the baseline conditions. Each subject was fitted with an oversized helmet (either a seizure or a lacrosse helmet), which was 135 BRIAN A. IWATA et al. reinforced with additional padding. During EXT
- 94. (sensory), the helmet was placed on the subject's head at the beginning of a session and remained in place throughout (subjects never attempted to remove the helmets). An extension of this "non- contingent" helmet condition was implemented for Donnie and is described below. RESULTS Donnie Figure 2 shows the results obtained for Donnie. He exhibited a considerable amount of SIB across all of the baseline conditions, thus permitting a series of manipulations on each baseline. Moderate levels of SIB were observed during the demand, attention, and play conditions (lower three panels), whereas SIB occurred almost continuously during the alone condition (upper panel) containing no access to social stimulation, play materials, or tasks. These data suggested that Donnie's SIB was main- tained by factors independent of the social and physical environment and were consistent with be- havior reinforced by directly produced, automatic consequences.
- 95. On the alone baseline (top panel), EXT (sensory) was implemented in a reversal (ABAB) design, and results showed decreases in SIB associated with the helmet intervention. The final condition on the alone baseline involved two modifications of the EXT (sensory) procedure. First, play materials that Donnie was observed to manipulate during the original play baseline were introduced. Second, the protective helmet was applied only contingent on the occurrence of SIB. As long as SIB did not occur, Donnie was free to play with the toys and move about without wearing the helmet. When SIB oc- curred, an experimenter, who was in the room but did not interact with Donnie during the session, placed the helmet on Donnie for a 2-min time- out, during which the toys were removed from sight. These procedural modifications were based on results reported by Dorsey, Iwata, Reid, and Davis (1982), who found that it was possible to transfer control of SIB from a noncontingent to a contingent helmet condition. SIB increased initially when this change was made on the alone baseline, but eventually decreased to a level similar to that
- 96. observed during the EXT (sensory) conditions. On the demand baseline (second panel from the top), the introduction of EXT (sensory) was asso- ciated with a rapid and large decrease in Donnie's SIB. This reduction occurred in spite of the fact that the original baseline contingency for SIB- EXT (attention) or, alternatively, escape from the task contingent on SIB-remained in effect. Next, EXT (sensory) was removed and EXT (escape) was implemented; this resulted in an increase in SIB to its baseline level. Finally, the contingent helmet procedure was combined with EXT (escape). The experimenter presented learning trials as before and, contingent on SIB, guided Donnie through the task and then placed the helmet on him. The helmet remained on for approximately 2 min (four learning trials). This condition was associated with a decrease in SIB similar to that seen during the previous EXT (sensory) condition. On the attention baseline (third panel from the top), the introduction of EXT (attention) had no effect on Donnie's SIB. During the next condition, EXT (sensory), SIB decreased even though it pro-
- 97. duced attention from the experimenter. The final procedure implemented on this baseline consisted of EXT (attention) combined with a 2-min time- out in the helmet contingent on occurrences of SIB, and was associated with a reduction in SIB similar to that seen in the EXT (sensory) condition. On the play baseline (bottom panel), which con- tained an EXT (attention) component, no decrease in SIB was observed until EXT (sensory) was in- troduced. Subsequently, SIB increased temporarily but decreased again when the procedure was changed to a 2-min time-out in the helmet contingent on SIB. Jack The initial baseline of Figure 3 shows the results of Jack's assessment. He exhibited little or no SIB except during the demand condition, which con- tained two distinctive features: the presentation of task demands and a brief time-out from the task contingent on SIB. The time-out component pro- 136
- 98. DONNIE 100- Baseline 800 n:n -a 40 - 20 - VARIATIONS OF EXTINCTION r EXT(sensory) 1 BL I Toys + Conting( -a- Ati -a-- -D -.*-- Al4 -a- Pli
- 99. ent Helmet ttention emand lone lay 1 ---- -i- 40 EXT Contingent Helmet Baseline EXT(sensory) (escape) + EXT(escape) 20 o r , . . 40 Contingent Helmet Baseline EXT(attn) EXT(sensory) + EXT(attn) 20 40 Contingent Baseline EXT(sensory) Helmet
- 100. 20 0 20 40 60 80 100 137 co) -J cn w H z uL 0 Lu w z
- 101. w 0L 20 40 60 80 1 00 SESSIONS Figure 2. Percentage of intervals of SIB exhibited by Donnie across experimental conditions. ou -1 A II BRIAN A. IWATA et al. EXT(sensory) 100 co
- 103. 20 0 -H0. SESSIONS Figure 3. Percentage of intervals of SIB exhibited by Jack across experimental conditions. cedurally resembled EXT (attention) but appeared to function as (negative) reinforcement for escape behavior. Because little or no SIB was observed during other conditions, all subsequent manipu- lations were made on the demand baseline. EXT (sensory) and EXT (escape) were implemented con- currently and resulted in a large decrease in Jack's SIB. A partial reversal was then conducted in which EXT (escape) was removed while EXT (sensory) remained in effect; this change resulted in an in- crease in Jack's SIB to its original baseline level. SIB decreased again when EXT (escape) was re- introduced, and it remained low following the re- moval of EXT (sensory).
- 104. Millie Figure 4 shows the results obtained for Millie. During the initial baseline assessment, her SIB oc- curred almost exclusively during the attention con- dition, suggesting that the behavior was maintained by social-positive reinforcement. Very little SIB was observed during the play and demand conditions, both of which contained a brief time-out from attention contingent on SIB, and SIB decreased throughout the alone condition, which contained no attention whatsoever. Thus, Millie's assessment data revealed not only a reinforcement effect during the attention condition but also an apparent ex- tinction effect during others. In light of these results, subsequent interventions were implemented only on the attention baseline. EXT (attention) with attention delivered a 30-s DRO schedule was introduced concurrent with EXT (sensory) and resulted in a large decrease in Millie's SIB. Next, while EXT (sensory) remained in effect, a reversal was conducted in which EXT (attention)
- 105. + DRO was removed; this partial return to base- line was associated with an immediate and large increase in SIB. When EXT (attention) + DRO was reinstated, SIB decreased to near-zero levels. EXT (sensory) was then removed while EXT (at- tention) + DRO remained in effect, and no increase was observed in SIB. During this final condition, the length of the DRO interval was gradually in- creased to 1, 2, and finally 5 min, and SIB remained low across all schedule changes. 138 VARIATIONS OF EXTINCTION EXT(sensory) _,4,0MILLIE EXT(attn) + DRO Baseline BL -a Attention *- Demand
- 106. -0- Alone 0 Play 20 40 60 80 SESSIONS Figure 4. Percentage of intervals of SIB exhibited by Millie across experimental conditions. Summary of Key Comparisons Table 1 contains a summary of the results from selected baseline and extinction conditions for Don- nie, Jack, and Millie. The data reflect reductions or increases in SIB expressed as mean percentages of baseline responding (EXT mean/baseline mean) for relevant and irrelevant extinction conditions compared to their appropriate baselines. Because SIB during relevant extinction conditions usually showed decreasing trends, comparisons based on overall condition means would not accurately depict end-of-treatment responding. Therefore, all per- centages are based on the last five sessions of a condition. For example, under "Relevant EXT,"
- 107. Donnie's mean percentage of SIB during the last five sessions of EXT (sensory) occurred at 0.5% of the mean SIB during the last five sessions of his alone baseline, whereas under "Irrelevant EXT," his five-session mean percentage of SIB during EXT (escape) occurred at 81.9% of his demand baseline mean. These comparisons show that SIB for all subjects was reduced to below 15% of its baseline Table 1 Mean Change (Reduction or Increase) in SIB Expressed as a Percentage of Baseline Responding for Relevant and Irrelevant Extinction Conditions Compared to Their Relevant Baselines. Percentages Are Based on the Last Five Sessions of a Condition Relevant EXT / baseline Irrelevant EXT / baseline Subject Conditions compared % of BL Conditions compared % of BL Donnie EXT (sensory) / alone BL 0.5 EXT (escape) / demand BL 81.9 EXT (sensory) / demand BL 6.9 EXT (attn) / attn …
- 108. Chapter 4: Clustering I 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› What is Cluster Analysis? Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups
- 109. Inter-cluster distances are maximized
- 110. Intra-cluster distances are minimized 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Applications of Cluster Analysis Understanding Group related documents for browsing, group genes and proteins that have similar functionality, or group stocks with similar price fluctuations Summarization Reduce the size of large data sets Clustering precipitation in Australia
- 111. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› What is not Cluster Analysis? Simple segmentation Dividing students into different registration groups alphabetically, by last name Results of a query Groupings are a result of an external specification Clustering is a grouping of objects based on the data Supervised classification Have class label information 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Notion of a Cluster can be Ambiguous
- 112. How many clusters?
- 113. Four Clusters
- 114. Two Clusters
- 115. Six Clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Types of Clusterings A clustering is a set of clusters Important distinction between hierarchical and partitional sets of clusters Partitional Clustering A division of data objects into non-overlapping subsets (clusters) such that each data object is in exactly one subset
- 116. Hierarchical clustering A set of nested clusters organized as a hierarchical tree 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Partitional Clustering
- 117. Original Points A Partitional Clustering 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Hierarchical Clustering Traditional Hierarchical Clustering Non-traditional Hierarchical Clustering Non-traditional Dendrogram Traditional Dendrogram
- 118. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Other Distinctions Between Sets of Clusters Exclusive versus non-exclusive In non-exclusive clusterings, points may belong to multiple clusters. Can represent multiple classes or ‘border’ points Fuzzy versus non-fuzzy In fuzzy clustering, a point belongs to every cluster with some weight between 0 and 1 Weights must sum to 1 Probabilistic clustering has similar characteristics Partial versus complete In some cases, we only want to cluster some of the data Heterogeneous versus homogeneous Clusters of widely different sizes, shapes, and densities 02/14/2018 Introduction to Data Mining, 2nd Edition
- 119. ‹#› Types of Clusters Well-separated clusters Center-based clusters Contiguous clusters Density-based clusters Property or Conceptual Described by an Objective Function 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Types of Clusters: Well-Separated Well-Separated Clusters: A cluster is a set of points such that any point in a cluster is closer (or more similar) to every other point in the cluster than to any point not in the cluster.
- 120. 3 well-separated clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Types of Clusters: Center-Based Center-based A cluster is a set of objects such that an object in a cluster is closer to the “center” of a cluster, than to the center of any other cluster The center of a cluster is a centroid. The average of all the points in the cluster is a medoid 4 center-based clusters
- 121. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Types of Clusters: Contiguity-Based Contiguous Cluster (Nearest neighbor or Transitive) A point in a cluster is closer (or more similar) to one or more other points in the cluster than to any point not in the cluster. 8 contiguous clusters
- 122. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Types of Clusters: Density-Based Density-based A cluster is a dense region of points, which is separated by low - density regions, from other regions of high density. Used when the clusters are irregular or intertwined, and when noise and outliers are present. 6 density-based clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Types of Clusters: Conceptual Clusters
- 123. Shared Property or Conceptual Clusters Finds clusters that share some common property or represent a particular concept. . 2 Overlapping Circles 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Types of Clusters: Objective Function Clusters Defined by an Objective Function Finds clusters that minimize or maximize an objective function. Enumerate all possible ways of dividing the points into clusters and evaluate the `goodness' of each potential set of clusters by using the given objective function. (NP Hard) Can have global or local objectives. Hierarchical clustering algorithms typically have local objectives Partitional algorithms typically have global objectives A variation of the global objective function approach is to fit the data to a parameterized model.
- 124. Parameters for the model are determined from the data. Mixture models assume that the data is a ‘mixture' of a number of statistical distributions. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Characteristics of the Input Data Are Important Type of proximity or density measure Central to clustering Depends on data and application Data characteristics that affect proximity and/or density are Dimensionality Sparseness Attribute type Special relationships in the data For example, autocorrelation Distribution of the data Noise and Outliers Often interfere with the operation of the clustering algorithm
- 125. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Clustering Algorithms K-means and its variants Hierarchical clustering Density-based clustering 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› K-means Clustering Partitional clustering approach
- 126. Number of clusters, K, must be specified Each cluster is associated with a centroid (center point) Each point is assigned to the cluster with the closest centroid The basic algorithm is very simple 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Example of K-means Clustering 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#›
- 127. Example of K-means Clustering 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› K-means Clustering – Details Initial centroids are often chosen randomly. Clusters produced vary from one run to another. The centroid is (typically) the mean of the points in the cluster. ‘Closeness’ is measured by Euclidean distance, cosine similarity, correlation, etc. K-means will converge for common similarity measures mentioned above. Most of the convergence happens in the first few iterations. Often the stopping condition is changed to ‘Until relatively few points change clusters’
- 128. Complexity is O( n * K * I * d ) n = number of points, K = number of clusters, I = number of iterations, d = number of attributes 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Evaluating K-means Clusters Most common measure is Sum of Squared Error (SSE) For each point, the error is the distance to the nearest cluster To get SSE, we square these errors and sum them. x is a data point in cluster Ci and mi is the representative point for cluster Ci can show that mi corresponds to the center (mean) of the cluster Given two sets of clusters, we prefer the one with the smallest error One easy way to reduce SSE is to increase K, the number of clusters A good clustering with smaller K can have a lower SSE than a
- 129. poor clustering with higher K 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Two different K-means Clusterings Sub-optimal Clustering Optimal Clustering Original Points 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Limitations of K-means K-means has problems when clusters are of differing
- 130. Sizes Densities Non-globular Shapes K-means has problems when the data contains outliers/noise. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Limitations of K-means: Differing Sizes Original Points K-means (3 Clusters) 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#›
- 131. Limitations of K-means: Differing Density Original Points K-means (3 Clusters) 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Limitations of K-means: Non-globular Shapes Original Points K-means (2 Clusters) 02/14/2018 Introduction to Data Mining, 2nd Edition
- 132. ‹#› Overcoming K-means Limitations Original PointsK-means Clusters One solution is to use many clusters. Find parts of clusters, but need to put together. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Overcoming K-means Limitations Original PointsK-means Clusters
- 133. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Overcoming K-means Limitations Original PointsK-means Clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Importance of Choosing Initial Centroids
- 134. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Importance of Choosing Initial Centroids 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Importance of Choosing Initial Centroids …
- 135. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Importance of Choosing Initial Centroids … 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› Problems with Selecting Initial Points If there are K ‘real’ clusters then the chance of selecting one
- 136. centroid from each cluster is small. Chance is relatively small when K is large If clusters are the same size, n, then For example, if K = 10, then probability = 10!/1010 = 0.00036 Sometimes the initial centroids will readjust themselves in ‘right’ way, and sometimes they don’t Consider an example of five pairs of clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› 10 Clusters Example Starting with two initial centroids in one cluster of each pair of clusters
- 137. 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#› 10 Clusters Example Starting with two initial centroids in one cluster of each pair of clusters 02/14/2018 Introduction to Data Mining, 2nd Edition ‹#›