Upcoming SlideShare
×

# Association Rule Mining

895 views

Published on

0 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

Views
Total views
895
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
26
0
Likes
0
Embeds 0
No embeds

No notes for slide

### Association Rule Mining

1. 1. Survey on Data Mining -- Association Rules Chen-jung Stella Li csl15@cornell.edu Introduction Data mining has become a research area with increasing importance due to its capability of helping end users extract useful information from large databases. With rapid growth in adapting high-technological tools, businesses can now generate and collect massive amount of data, which they could not have done before. From this myriad collection of data, businesses would like to "discover" certain useful and interesting information, which could assist them in marketing strategies, decision making, etc. This can all be accomplished with various data mining techniques, such as association rules, characterization, classification, clustering, and so forth. This survey paper will present the method on association rules and explore various algorithms that have been studied in this field. The mining association rules basically is to find important associations among items in a given database of sales transactions such that the presence of some items will imply the presence of other items in the same transaction. A formal model is introduced in [AIS93]. Let I={i1, i2, … , im} be a set of binary attributes, called items. Let D a set of transactions and each transaction T is a set of items such that T⊆I. Let X be a set of items. A transaction T is said to contain X if and only if X⊆T. An association rule is an implication of the form X⇒Y, where X⊂ I, Y⊂ I, and X∩Y = ∅. (X is called the antecedent of the rule, while Y is called the consequent of the rule.) Furthermore, the rule X⇒Y is said to hold in the transaction set D with confidence c if there are c% of the transaction set D containing X also containing Y. The rule X⇒Y is said to have support s in the transaction set D if there are s% of transactions in D containing X∪Y. The confidence factor indicates the strength of the implication rules; whereas the support factor indicates the frequencies of the occurring patterns in the rule. The goal of mining association rules is then to find "strong" rules, which are rules with high confidence and strong support, in a large databases. The problem of mining association rules can further be decomposed into two sub-problems using the above formulation: 1) discovering the large itemsets, and 2) 1
2. 2. using the large itemsets to generate the association rules for the database. The first sub-problem, discovering the large itemsets, is said to generate all combinations of items that have fractional transaction support above a certain threshold. All other combinations that fall below the support threshold are called small itemsets. The second sub-problem, generating association rules, uses the large itemsets found in the first problem and generates the desired rules. A straightforward way to generate rules is the following: for every large itemset X, find all non-empty subsets of X. And for every such subset Y, output a rule of the form Y⇒(X-Y) if the ratio of support of X to support of Y is exceeds c, the minimum confidence factor. Since there is a straightforward way in solving the second sub-problem, the major emphasis has been placed on finding efficient algorithm in finding the large itemsets. The algorithms that will be discussed in the following sections are the AIS algorithm [AIS93], Apriori and AprioriTid [AS94], DHP [PCY95], Partition Algorithm [SON95], and SETM [HS95]. AIS Algorithm Before introducing the AIS algorithm, there are two terms needed to be defined. The "frontier set" of a pass is a set which consists of itemsets that are extended during the pass. During each pass, a measurement of the support for certain itemsets is taken. These itemsets are called "candidate itemsets." They are derived from the tuples in the database and the itemsets contained in the frontier set. The AIS algorithm makes multiple passes over the database. Initially, the frontier set is empty. During each pass over the database, the candidate sets are generated from taking the extensions of the frontier set and the tuples in the database. At the end of each pass, if the support for a candidate set is equal or above the minimum support required, this candidate set is kept and considered as a large itemset which will then be used in the following passes. Meanwhile, this itemset will be determined whether it should be added to the frontier set for the next pass. That is, those candidate itemsets that were expected to be small but turned out to be large in the current pass would be included in the frontier set for the next pass. The entire algorithm terminates when the frontier set becomes empty. At the end, the remaining candidate itemsets are the large itemsets that we are supposed to discover. Apriori, AprioriTid, and AprioriHybrid Algorithms The Apriori algorithm first counts occurrences of items to determine the large 1- itemsets. (itemset with the cardinality of 1) Then there are two phases in the 2
3. 3. subsequent passes. First of all, the large itemsets Lk-1 found in the (k-1)th pass are used to generate the candidate itemsets Ck, using the "apriori-gen" function. Next, the support of candidates in Ck is counted by scanning the database. The final answer is obtained by taking the union of all Lk itemsets. The "apriori-gen" function takes Lk-1 as argument and returns a superset of the set of all large k-itemsets. Firstly, the join step is taken by joining Lk-1 with Lk-1. The next step is the prune step, which deletes all itemsets c∈Ck such that some (k-1)-subset of c is not in Lk-1. The intuition behind the "apriori-gen" function is that every subset of a large itemset must be large; thus we can combine almost-matching pairs of large (k-1)-itemsets, i.e. Lk-1, and then prune out those with non-large (k-1)-subsets. The AprioriTid algorithm is a variation of the Apriori algorithm. The AprioriTid algorithm also uses the "apriori-gen" function to determine the candidate itemsets before the pass begins. The main difference from the Apriori algorithm is that the AprioriTid algorithm does not use the database for counting support after the first pass. Instead, the set <TID, {Xk}> is used for counting. (Each Xk is a potentially large k-itemset in the transaction with identifier TID.) The benefit of using this scheme for counting support is that at each pass other than the first pass, the scanning of the entire database is avoided. But the downside of this is that the set <TID, {Xk}> that would have been generated at each pass may be huge. Another algorithm, called AprioriHybrid, is introduced in [AS94]. The basic idea of the AprioriHybird algorithm is to run the Apriori algorithm initially, and then switch to the AprioriTid algorithm when the generated database (i.e. <TID, {Xk}>) would fit in the memory. DHP Algorithm The DHP (Direct Hashing and Pruning) algorithm is an effective hash-based algorithm for the candidate set generation. The DHP algorithm consists of three steps. The first step is to get a set of large 1-itemsets and constructs a hash table for 2-itemsets. (The hash table contains the information regarding the support of each itemset.) The second step generates the set of candidate itemsets Ck, but it only adds the k-itemset into Ck if that k-itemset is hashed into a hash entry whose value is greater than or equal to the minimum transaction support. The third part is essentially the same as the second part except it does not use the hash table in determining whether to include a particular itemset into the candidate itemsets. The second part is designed for the use in early iteration, whereas the third part should be used for later iterations when the number of hash buckets with a support count greater than or equal to s (the minimum transaction support required) is less than a pre-defined threshold. 3
4. 4. Note that the DHP algorithm has two major features: one is its efficiency in generation of large itemsets; the other is effectiveness in reduction on transaction database size. The generation of smaller candidate sets is the key to effectively trim the transaction database size at earlier stages of the iterations such that the computational cost for the later iterations is significantly reduced. Therefore, the DHP algorithm is particularly powerful to determine large itemsets in early stages, and improves the performance bottleneck in a great deal. Partition Algorithm The Partition algorithm logically partition the database D into n partitions, and only reads the entire database at most two times to generate the association rules. The reason for using the partition scheme is that any potential large itemset would appear as a large itemset in at least one of the partitions. The algorithm consists of two phases. In the first phase, the algorithm iterates n times, and during each iteration, only one partition is considered. At any given iteration, the function "gen_large_itemsets" takes a single partition and generates local large itemsets of all lengths from this partition. All of these local large itemsets of the same lengths in all n partitions are merged and then combined to generate the global candidate itemsets. In the second phase, the algorithm counts the support of each global candidate itemsets and generates the global large itemsets. Note that the database is read twice during the process: once in the first phase and the other in the second phase, which the support counting requires a scan of the entire database. By taking minimal number of passes through the entire database drastically saves the time used for doing I/O. SETM Algorithm The SETM algorithm is a set-oriented algorithm for mining association rules. The algorithms introduced previously are mostly tuple-oriented. The SETM differs from the previous algorithms in a great deal; particularly, it incorporates the simple database primitives such as sorting and merge-scan join. In the first iteration, the SETM algorithm sorts R1 (denoted as the relation in the 1st iteration) on item, and C1 is the count relation generated for R1. In the following iterations, there are two sort operations and one merge-scan join performed. After the first sort operation, a temporary relation, R'k, in the kth iteration is generated by performing merge-scan join on Rk-1 and R1. Then, the second sort operation is to sort R'k on items, which is used for efficient generation of the support counts. The generation of counts involves a sequential scan over Rk. At the end of each iteration, deleting tuples from Rk that 4
5. 5. do not meet the minimum support generates Rk. Note that this is done via table look- ups on relation Ck. Another major contribution of the paper [HS95] is that it proposes ways to express the generation of association rules using SQL queries. Thus, besides introducing the SETM algorithm, the paper [HS95] also presents two formulations of their set-oriented data mining strategy that would naturally leads to nested-loop joins and sort-merge joins. Let suppose that there is a relation, SALES(trans_id, item). The first step in the formulation for leading to nested-loop joins is to generate the counts for each item x and to store the result in relation C1(item, count). INSERT INTO C1 SELECT r1.item, COUNT(*) FROM SQLES r1 GROUP BY r1.item HAVING COUNT(*) >= :min_support; Then the next step is to generate all patterns (x,y) and check whether they meet the minimum support criterion. For ease of expressing, let's take a specific item X. All the patterns (A,y) can be generated using a self-join of SALES. SELECT r1.item, r2.item, COUNT(*) FROM SALES r1, SALES r2 WHERE r1.trans_id = r2.trans_id AND R1.item = 'A' AND r2.item <> 'A' GROUP BY r1.item, r2.item HAVING COUNT(*) >= :min_support; Another formulation leads to sort-merge joins. The first step is to generate ordered patterns of length k: INSERT INTO R'k SELECT p.trans_id, p.item1, …, p.item[k-1], q.item FROM Rk-1 p, SALES q WHERE q.trans_id = p.trans_id AND q.item > p.item[k-1]; The next step is to generate the counts relations for those patterns in R'k that meet the minimum support criterion. INSERT INTO Ck 5
6. 6. SELECT p.item1, …, p.item[k], COUNT(*) FROM R'k p GROUP BY p.item,…, p.item[k] HAVING COUNT(*) >= :min_support; Then, select those tuples from R'k that meet the minimum support such that a further extension of the relations can be done. INSERT INTO Rk SELECT p.trans_id, p.item1, … p.item[k] FROM R'k p, Ck q WHERE p.item1 = q.item1 AND … p.item[k] = q.item[k] ORDER BY p.trans_id, p.item1, …, p.item[k]; This final step is repeated until Rk=∅. Therefore, the paper [HS95] proves the point that "at least some aspects of data mining can be carried out by using general query languages such as SQL, rather than by developing specialized black box algorithms." Conclusion The goal of mining association rules is to discover important associations among items in a database of transactions such that the presence of some items will imply the presence of other items. The problem of mining association rules has been decomposed into two sub-problems: discovering the large itemsets, and then generate rules based on these large itemsets. The attention has been placed on the first sub- problem since the second sub-problem is quite straightforward. Thus, there have been several algorithms proposed to solve the first sub-problem. These researches in algorithms of mining association rules are basically motivated by the fact that the amount of the processed data in mining association rules is huge; thus it is crucial to devise efficient algorithms to conduct mining on such data. The classical one is introduced in [AIS93] and is often referred as the AIS algorithm. It first layouts the framework of mining association rules and introduces a terminology called candidate itemsets, which is used throughout in the later work. However, the Apriori and AprioriTid algorithms [AS94] are the most well known ones in the field. Both of them use the "apriori-gen" function in generating the candidate itemsets. The DHP algorithm [PCY95] is very similar to the Apriori algorithm except that it employs a 6
7. 7. hash table scheme to make the generation more efficiently and effectively. The Partition algorithm [SON95] reduces the I/O overhead, in comparison with the Apriori algorithm, by first dividing the database into partitions and generating large itemsets from those partitions. The last algorithm being looked at is the SETM algorithm [HS95]. The SETM algorithm is inherently different from the others because it is a set-oriented algorithm. It is developed from realizing that there are certain formulations in SQL statements that could lead to nested-loop joins and sort- merge joins. Since the efficiency in mining association rules is application dependent, up to this point, there is no clear performance modeling to determine which algorithm performs ultimately better than others. References [AIS93] R. Agrawal, T. Imielinski, and A. Swami, "Mining Association Rules between Sets of Items in Large Databases," Proc. ACM SIGMOD, May 1993. [AS94] R. Agrawal and R. Srikant, "Fast Algorithms for Mining Association Rules in Large Databases," Proc. 20th Int'l Conf. of Very Large Data Bases, Sept. 1994. [PCY95] J.-S Park, M.-S. Chen, and P.S. Yu, "An Effective Hash Based Algorithm for Mining Association Rules," Proc. ACM SIGMOD, May 1995. [SON95] A. Savasere, E. Omiecinski, and S. Navathe, "An Efficient Algorithm for Mining Association Rules in large Databases," Proc. 21st Int'l Conf. of Very Large Data Bases, 1995 [HS95] M. Houtsma and A. Swami, "Set-Oriented Mining for Association Rules in Relational Databases," IEEE 11th Int'l Conf. on Data Engineering, 1995 7