Rule-based classification uses a set of if-then rules to classify tuples. Each rule has a condition part (if) and a consequent part (then) that assigns a class. The document discusses evaluating rule coverage, accuracy, and characteristics of rule sets such as being mutually exclusive or exhaustive. It also describes direct and indirect methods for building classification rules, including sequential covering algorithms and extracting rules from decision trees.
Date: October 9, 2017
Course: UiS DAT630 - Web Search and Data Mining (fall 2017) (https://github.com/kbalog/uis-dat630-fall2017)
Presentation based on resources from the 2016 edition of the course (https://github.com/kbalog/uis-dat630-fall2016) and the resources shared by the authors of the book used through the course (https://www-users.cs.umn.edu/~kumar001/dmbook/index.php).
Please cite, link to or credit this presentation when using it or part of it in your work.
Hierarchical Novelty Detection for Visual Object RecognitionNAVER Engineering
발표자: 이기복 (Univ. of Michigan 박사과정)
발표일: 2018.6.
Deep neural networks have achieved impressive success in large-scale visual object recognition tasks with a predefined set of classes. However, recognizing objects of novel classes unseen during training still remains challenging. The problem of detecting such novel classes has been addressed in the literature, but most prior works have focused on providing simple binary or regressive decisions, e.g., the output would be "known," "novel," or corresponding confidence intervals. In this paper, we study more informative novelty detection schemes based on a hierarchical classification framework. For an object of a novel class, we aim for finding its closest super class in the hierarchical taxonomy of known classes. To this end, we propose two different approaches termed top-down and flatten methods, and their combination as well. The essential ingredients of our methods are confidence-calibrated classifiers, data relabeling, and the leave-one-out strategy for modeling novel classes under the hierarchical taxonomy. Furthermore, our method can generate a hierarchical embedding that leads to improved generalized zero-shot learning performance in combination with other commonly-used semantic embeddings.
Date: October 9, 2017
Course: UiS DAT630 - Web Search and Data Mining (fall 2017) (https://github.com/kbalog/uis-dat630-fall2017)
Presentation based on resources from the 2016 edition of the course (https://github.com/kbalog/uis-dat630-fall2016) and the resources shared by the authors of the book used through the course (https://www-users.cs.umn.edu/~kumar001/dmbook/index.php).
Please cite, link to or credit this presentation when using it or part of it in your work.
Hierarchical Novelty Detection for Visual Object RecognitionNAVER Engineering
발표자: 이기복 (Univ. of Michigan 박사과정)
발표일: 2018.6.
Deep neural networks have achieved impressive success in large-scale visual object recognition tasks with a predefined set of classes. However, recognizing objects of novel classes unseen during training still remains challenging. The problem of detecting such novel classes has been addressed in the literature, but most prior works have focused on providing simple binary or regressive decisions, e.g., the output would be "known," "novel," or corresponding confidence intervals. In this paper, we study more informative novelty detection schemes based on a hierarchical classification framework. For an object of a novel class, we aim for finding its closest super class in the hierarchical taxonomy of known classes. To this end, we propose two different approaches termed top-down and flatten methods, and their combination as well. The essential ingredients of our methods are confidence-calibrated classifiers, data relabeling, and the leave-one-out strategy for modeling novel classes under the hierarchical taxonomy. Furthermore, our method can generate a hierarchical embedding that leads to improved generalized zero-shot learning performance in combination with other commonly-used semantic embeddings.
Franz Et Al. Using ASP to Simulate the Interplay of Taxonomic and Nomenclatur...taxonbytes
Answer Set Programming (ASP) is a declarative, stable model approach to logic programming with an under-realized potential for representing and reasoning over biological information. ASP is particularly suited to address reasoning challenges with complex starting conditions and rule sets. One such challenge is the interplay of taxonomic and nomenclatural change in biological taxonomy that often results when a taxonomy is revised based on a previously published perspective. Depending on the nature of the taxonomic changes to be undertaken, one or more Code-mandated principles will apply to regulate specific and concomitant name changes. In the case of the International Code of Zoological Nomenclature, two principles of significance include the Principles of Priority and Typification. Although the relationship between the number of taxonomic and nomenclatural adjustments under a given transition scenario is not linear, the application of the name-changing rules is usually unambiguous and therefore amenable to logic representation. Here we explore the modeling of the taxonomy/nomenclature interplay in ASP with a simple, abstract nine-taxon use case that contains four terminal species of which two are type-bearers for their respective genera. Four distinct one-taxon transfer scenarios are simulated through a transition system approach, requiring 1-7 concomitant nomenclatural changes depending (1) on the priority relationships among the terminal taxa being repositioned and (2) the type-bearing name dependencies of their higher-level parents. ASP can simulate these rules faithfully and thus reason over situations that range from a one-to-one match of taxonomic and nomenclatural changes to situations where they two kinds of change become increasingly disconnected (e.g., transfer of non-type genera among tribes without name change, or "transfer" [in reverse direction] of a single priority-carrying name/taxon into a larger yet junior entity with numerous required name changes). Our results, though very preliminary, illustrate how ASP logic approach may be utilized to perform optimizations at the taxonomy/nomenclature intersection, and generally represent a novel step towards translating Code-mandated naming rules into logic, with potential benefits for virtual taxonomic domains.
This is a presentation to explain the concepts described in the paper "A Concept Analysis Inspired Greedy Algorithm for Test Suite Minimization" by Sriraman Tallam and Neelam Gupta.
Abstract: Software testing and retesting occurs continuously during the soft- ware development lifecycle to detect errors as early as possible and to ensure that changes to existing software do not break the soft- ware. Test suites once developed are reused and updated frequently as the software evolves. As a result, some test cases in the test suite may become redundant as the software is modified over time since the requirements covered by them are also covered by other test cases. Due to the resource and time constraints for re-executing large test suites, it is important to develop techniques to minimize available test suites by removing redundant test cases. In general, the test suite minimization problem is NP complete. In this paper, we present a new greedy heuristic algorithm for selecting a minimal subset of a test suite T that covers all the requirements covered by T . We show how our algorithm was inspired by the concept analy- sis framework. We conducted experiments to measure the extent of test suite reduction obtained by our algorithm and prior heuristics for test suite minimization. In our experiments, our algorithm al- ways selected same size or smaller size test suite than that selected by prior heuristics and had comparable time performance.
What is the Covering (Rule-based) algorithm?
Classification Rules- Straightforward
1. If-Then rule
2. Generating rules from Decision Tree
Rule-based Algorithm
1. The 1R Algorithm / Learn One Rule
2. The PRISM Algorithm
3. Other Algorithm
Application of Covering algorithm
Discussion on e/m-learning application
Clustering and Classification Algorithms Ankita DubeyAnkita Dubey
Clustering is a process of partitioning a set of data (or objects) into a set of meaningful sub-classes, called clusters. Help users understand the natural grouping or structure in a data set. Used either as a stand-alone tool to get insight into data distribution or as a preprocessing step for other algorithms.
Franz Et Al. Using ASP to Simulate the Interplay of Taxonomic and Nomenclatur...taxonbytes
Answer Set Programming (ASP) is a declarative, stable model approach to logic programming with an under-realized potential for representing and reasoning over biological information. ASP is particularly suited to address reasoning challenges with complex starting conditions and rule sets. One such challenge is the interplay of taxonomic and nomenclatural change in biological taxonomy that often results when a taxonomy is revised based on a previously published perspective. Depending on the nature of the taxonomic changes to be undertaken, one or more Code-mandated principles will apply to regulate specific and concomitant name changes. In the case of the International Code of Zoological Nomenclature, two principles of significance include the Principles of Priority and Typification. Although the relationship between the number of taxonomic and nomenclatural adjustments under a given transition scenario is not linear, the application of the name-changing rules is usually unambiguous and therefore amenable to logic representation. Here we explore the modeling of the taxonomy/nomenclature interplay in ASP with a simple, abstract nine-taxon use case that contains four terminal species of which two are type-bearers for their respective genera. Four distinct one-taxon transfer scenarios are simulated through a transition system approach, requiring 1-7 concomitant nomenclatural changes depending (1) on the priority relationships among the terminal taxa being repositioned and (2) the type-bearing name dependencies of their higher-level parents. ASP can simulate these rules faithfully and thus reason over situations that range from a one-to-one match of taxonomic and nomenclatural changes to situations where they two kinds of change become increasingly disconnected (e.g., transfer of non-type genera among tribes without name change, or "transfer" [in reverse direction] of a single priority-carrying name/taxon into a larger yet junior entity with numerous required name changes). Our results, though very preliminary, illustrate how ASP logic approach may be utilized to perform optimizations at the taxonomy/nomenclature intersection, and generally represent a novel step towards translating Code-mandated naming rules into logic, with potential benefits for virtual taxonomic domains.
This is a presentation to explain the concepts described in the paper "A Concept Analysis Inspired Greedy Algorithm for Test Suite Minimization" by Sriraman Tallam and Neelam Gupta.
Abstract: Software testing and retesting occurs continuously during the soft- ware development lifecycle to detect errors as early as possible and to ensure that changes to existing software do not break the soft- ware. Test suites once developed are reused and updated frequently as the software evolves. As a result, some test cases in the test suite may become redundant as the software is modified over time since the requirements covered by them are also covered by other test cases. Due to the resource and time constraints for re-executing large test suites, it is important to develop techniques to minimize available test suites by removing redundant test cases. In general, the test suite minimization problem is NP complete. In this paper, we present a new greedy heuristic algorithm for selecting a minimal subset of a test suite T that covers all the requirements covered by T . We show how our algorithm was inspired by the concept analy- sis framework. We conducted experiments to measure the extent of test suite reduction obtained by our algorithm and prior heuristics for test suite minimization. In our experiments, our algorithm al- ways selected same size or smaller size test suite than that selected by prior heuristics and had comparable time performance.
What is the Covering (Rule-based) algorithm?
Classification Rules- Straightforward
1. If-Then rule
2. Generating rules from Decision Tree
Rule-based Algorithm
1. The 1R Algorithm / Learn One Rule
2. The PRISM Algorithm
3. Other Algorithm
Application of Covering algorithm
Discussion on e/m-learning application
Clustering and Classification Algorithms Ankita DubeyAnkita Dubey
Clustering is a process of partitioning a set of data (or objects) into a set of meaningful sub-classes, called clusters. Help users understand the natural grouping or structure in a data set. Used either as a stand-alone tool to get insight into data distribution or as a preprocessing step for other algorithms.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...John Andrews
SlideShare Description for "Chatty Kathy - UNC Bootcamp Final Project Presentation"
Title: Chatty Kathy: Enhancing Physical Activity Among Older Adults
Description:
Discover how Chatty Kathy, an innovative project developed at the UNC Bootcamp, aims to tackle the challenge of low physical activity among older adults. Our AI-driven solution uses peer interaction to boost and sustain exercise levels, significantly improving health outcomes. This presentation covers our problem statement, the rationale behind Chatty Kathy, synthetic data and persona creation, model performance metrics, a visual demonstration of the project, and potential future developments. Join us for an insightful Q&A session to explore the potential of this groundbreaking project.
Project Team: Jay Requarth, Jana Avery, John Andrews, Dr. Dick Davis II, Nee Buntoum, Nam Yeongjin & Mat Nicholas
Data Centers - Striving Within A Narrow Range - Research Report - MCG - May 2...pchutichetpong
M Capital Group (“MCG”) expects to see demand and the changing evolution of supply, facilitated through institutional investment rotation out of offices and into work from home (“WFH”), while the ever-expanding need for data storage as global internet usage expands, with experts predicting 5.3 billion users by 2023. These market factors will be underpinned by technological changes, such as progressing cloud services and edge sites, allowing the industry to see strong expected annual growth of 13% over the next 4 years.
Whilst competitive headwinds remain, represented through the recent second bankruptcy filing of Sungard, which blames “COVID-19 and other macroeconomic trends including delayed customer spending decisions, insourcing and reductions in IT spending, energy inflation and reduction in demand for certain services”, the industry has seen key adjustments, where MCG believes that engineering cost management and technological innovation will be paramount to success.
MCG reports that the more favorable market conditions expected over the next few years, helped by the winding down of pandemic restrictions and a hybrid working environment will be driving market momentum forward. The continuous injection of capital by alternative investment firms, as well as the growing infrastructural investment from cloud service providers and social media companies, whose revenues are expected to grow over 3.6x larger by value in 2026, will likely help propel center provision and innovation. These factors paint a promising picture for the industry players that offset rising input costs and adapt to new technologies.
According to M Capital Group: “Specifically, the long-term cost-saving opportunities available from the rise of remote managing will likely aid value growth for the industry. Through margin optimization and further availability of capital for reinvestment, strong players will maintain their competitive foothold, while weaker players exit the market to balance supply and demand.”
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
3. Data Mining 3
Rule-Based Classifier
Using IF-THEN Rules for Classification
• Classify tuples by using a collection of “if…then…” rules.
– Represent the knowledge in the form of IF-THEN rules
Rule: IF (Condition) THEN Consequent
– where
• Condition is a conjunction of attributes
• Consequent is a class label
(rule antecedent or condition)
(rule consequent)
• Examples of classification rules:
IF (age=youth AND student=yes) THEN buys_computer = yes
IF (BloodType=warm AND LayEggs=yes) THEN class = bird
4. Data Mining 4
Rule-Based Classifier:
Rule Coverage and Accuracy
• A rule r covers an instance x if the attributes of the instance satisfy the condition of
the rule.
Coverage of a rule:
• Fraction of tuples that satisfy the condition of a rule.
Accuracy of a rule:
• Fraction of tuples that satisfy the condition that also satisfy the consequent of a rule.
ncovers : # of tuples covered by rule R
ncorrect : # of tuples correctly classified by rule R
coverage(R) = ncovers / |D| /* D: training data set */
accuracy(R) = ncorrect / ncovers
5. Data Mining 5
Rule-Based Classifier:
Rule Coverage and Accuracy - Example
IF (Status=Single) THEN Class=No
Coverage = 4/10 = 40%
Accuracy = 2/4 = 50%
Tid Refund Marital
Status
Taxabl
e
Income
Class
1 Yes Single 125K No
2 No Married 100K No
3 No Single 70K No
4 Yes Married 120K No
5 No Divorced 95K Yes
6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No
10 No Single 90K Yes
10
6. Data Mining 6
Rule-Based Classifier - Example
Name Blood Type Give Birth Can Fly Live in Water Class
human warm yes no no mammals
python cold no no no reptiles
salmon cold no no yes fishes
whale warm yes no yes mammals
frog cold no no sometimes amphibians
komodo cold no no no reptiles
bat warm yes yes no mammals
pigeon warm no yes no birds
cat warm yes no no mammals
leopard shark cold yes no yes fishes
turtle cold no no sometimes reptiles
penguin warm no no sometimes birds
porcupine warm yes no no mammals
eel cold no no yes fishes
salamander cold no no sometimes amphibians
gila monster cold no no no reptiles
platypus warm no no no mammals
owl warm no yes no birds
dolphin warm yes no yes mammals
eagle warm no yes no birds
R1: IF (Give Birth = no)AND (Can Fly = yes) THEN Class=Birds
R2: IF (Give Birth = no)AND (Live in Water = yes) THEN Class=Fishes
R3: IF (Give Birth = yes)AND (Blood Type = warm) THEN Class=Mammals
R4: IF (Give Birth = no)AND (Can Fly = no) THEN Class=Reptiles
R5: IF (Live in Water = sometimes) THEN Class=Amphibians
7. Data Mining 7
How a Rule-Based Classifier Works?
• A rule-based classifier classifies a tuple based on the rule triggered by the tuple.
R1: IF (Give Birth = no)AND (Can Fly = yes) THEN Class=Birds
R2: IF (Give Birth = no)AND (Live in Water = yes) THEN Class=Fishes
R3: IF (Give Birth = yes) AND (Blood Type = warm) THEN Class=Mammals
R4: IF (Give Birth = no)AND (Can Fly = no) THEN Class=Reptiles
R5: IF (Live in Water = sometimes) THEN Class=Amphibians
• A lemur triggers rule R3, so it is classified as a mammal
• A turtle triggers both R4 and R5
– Since the classes predicted by the rules are contradictory (reptiles versus amphibians),
their conflicting classes must be resolved.
• A dogfish shark triggers none of the rules
– we need to ensure that the classifier can still make a reliable prediction even though a tuple
is not covered by any rule.
Name Blood Type Give Birth Can Fly Live in Water Class
lemur warm yes no no ?
turtle cold no no sometimes ?
dogfish shark cold yes no yes ?
8. Data Mining 8
Characteristics of Rule Sets
Mutually Exclusive Rules
• The rules in a rule set R are mutually exclusive if no two rules in R are triggered by
the same tuple.
• Every tuple is covered by at most one rule in R.
Exhaustive Rules
• A rule set R has exhaustive coverage if there is a rule for each combination of
attribute values.
• Every record is covered by at least one rule in R.
9. Data Mining 9
Characteristics of Rule Sets
Rules are not mutually exclusive:
• A tuple may trigger more than one rule
• Solution 1: Use ordered rule set
– The rules in a rule set are ordered in decreasing order of their priority (e.g., based on
accuracy, coverage, or the order in which the rules are generated).
– An ordered rule set is also known as a decision list.
– When a tuple is presented, it is classified by the highest-ranked rule that covers the tuple.
• Solution 2: Use unordered rule set and a voting scheme
– A tuple triggers multiple rules and considers the consequent of each rule as a vote for a
particular class.
– The tuple is usually assigned to the class that receives the highest number of votes. In some
cases, the vote may be weighted by the rule’s accuracy
10. Data Mining 10
Characteristics of Rule Sets
Rules are not exhaustive
• A tuple may not trigger any rules
• Solution: Use a default class
– If the rule set is not exhaustive, then a default rule must be added to cover the remaining
cases.
DefaultRule: IF () THEN Class = defaultclass
– A default rule has an empty antecedent (TRUE) and is triggered when all other rules have
failed.
– defaultclass is known as the default class and is typically assigned to the majority class of
training records not covered by the existing rules.
11. Data Mining 11
Rule Ordering Schemes
• Rule ordering can be implemented on a rule-by-rule basis or on a class-by-class basis.
Rule-Based Ordering Scheme
• Individual rules are ranked based on their quality.
– Every tuple is classified by the “best” rule covering it.
– Lower-ranked rules are much harder to interpret because they assume the negation of the
rules preceding them.
Class-Based Ordering Scheme
• Rules that belong to the same class appear together in the rule set.
• The rules are then collectively sorted on the basis of their class information.
– The relative ordering among the rules from the same class is not important; as long as one
of the rules fires, the class will be assigned to the test tuple.
– A high-quality rule to be overlooked in favor of an inferior rule that happens to predict the
higher-ranked class.
13. Data Mining 13
Building Classification Rules
• To build a rule-based classifier, we need to extract a set of rules that identifies key
relationships between the attributes of a data set and the class label.
Direct Methods:
• Extract classification rules directly from data.
• Examples: RIPPER, CN2, …
Indirect Methods:
• Extract rules from other classification models (e.g. decision trees, etc).
• Examples: C4.5rules
14. Direct Method: Sequential Covering
• Sequential covering algorithm extracts rules directly from training data
• Typical sequential covering algorithms: RIPPER, FOIL, CN2,
• Rules are learned sequentially, each for a given class Ci will cover many tuples of Ci
but none (or few) of the tuples of other classes.
Sequential CoveringAlgorithm
Data Mining 14
15. Sequential Covering - Example
• Find the first rule R1 and remove the tuples covered by R1 from the training data set.
• Add the rule to the rule list.
Data Mining 15
original data set tuples covered by R1 remove tuples covered
by R1
17. Data Mining 17
Sequential Covering
How to Learn-One-Rule?
• The objective of Learn-One-Rule function is to extract a classification rule that covers
many of the positive examples and none (or very few) of the negative examples in the
training set.
• Finding an optimal rule is computationally expensive given the exponential size of the
search space.
• Learn-One-Rule function addresses the exponential search problem by growing the
rules in a greedy fashion.
– It generates an initial rule r and keeps refining the rule until a certain stopping
criterion is met.
– The rule is then pruned to improve its generalization error.
18. Data Mining 18
Learn-One-Rule: Rule-Growing Strategy
•Two common rule-growing strategies: general-to-specific or specific-to-general.
General-to-specific:
• An initial rule r : {} → y is created, where the left-hand side is an empty set and the
right-hand side contains the target class.
– The rule has poor quality because it covers all the examples in the training set.
• New conjuncts are subsequently added to improve the rule’s quality.
– Algorithm then explores all possible candidates and greedily chooses the next conjunct.
– Process continues until the stopping criterion is met (e.g., when the added conjunct does
not improve the quality of the rule).
Specific-to-general
• One of the positive examples is randomly chosen as the initial seed for the rule-
growing process.
• During the refinement step, the rule is generalized by removing one of its conjuncts so
that it can cover more positive examples.
20. Data Mining 20
Learn-One-Rule: Rule Evaluation
• An evaluation metric is needed to determine which conjunct should be added (or
removed) during the rule-growing process.
– Accuracy is an obvious choice because it explicitly measures the fraction of training
examples classified correctly by the rule.
– A potential limitation of accuracy is that it does not take into account the rule’s coverage.
• Training data set has: 60 positive examples, 100 negative examples
• R1: covers 50 positive examples and 5 negative examples
• R2: covers 2 positive examples and no negative examples.
• Although the accuracy of R2 (100%) is higher than the accuracy of R1 (90.9%), R1 is a better
rule because of the coverage of the rule.
• We need evaluation metrics take into account the rule’s coverage:
– Foil’s Information Gain, and other metrics.
21. Learn-One-Rule: Rule Evaluation
• Foil’s Information Gain is a rule-quality measure which considers both coverage
and accuracy.
R0: {A} class
R1: {A and B} class
(initial rule)
(rule after adding conjunct B)
Foil’s Information Gain(R0, R1) = 𝐩𝟏 × ( 𝐥𝐨𝐠
𝐩𝟏
𝟐 𝐩𝟏+𝐧𝟏
− 𝐥𝐨𝐠
𝐩𝟎
𝟐 𝐩𝟎+𝐧𝟎
Data Mining 21
)
where
p0: number of positive instances covered by R0
n0: number of negative instances covered by R0
p1: number of positive instances covered by R1
n1: number of negative instances covered by R1
22. Indirect Method for Rule Extraction
Rule Extraction from a Decision Tree
• Rules are easier to understand than large trees
• One rule is created for each path from the root to a leaf
• Each attribute-value pair along a path forms a
conjunction: the leaf holds the class prediction
• Rules are mutually exclusive and exhaustive
• Example: Rule extraction from our buys_computer decision-tree
IF age = young AND student = no
IF age = young AND student = yes
IF age = mid-age
THEN buys_computer = no
THEN buys_computer = yes
THEN buys_computer = yes
IF age = old AND credit_rating = excellent THEN buys_computer = no
IF age = old AND credit_rating = fair THEN buys_computer = yes
age?
student? credit rating?
<=30 >40
no yes yes
yes
31..40
fair
excellent
yes
Data Mining 22
no
23. Data Mining 23
Indirect Method for Rule Extraction
C4.5 Rules
• Extract rules from an unpruned decision tree
• For each rule, r: A y,
– Consider an alternative rule r′: A′ y where A′ is obtained by removing one of
the conjuncts in A
– Compare the pessimistic error rate for r against all r’s
– Prune if one of the alternative rules has lower pessimistic error rate
– Repeat until we can no longer improve generalization error
• Instead of ordering the rules, order subsets of rules (class ordering)
– Each subset is a collection of rules with the same rule consequent (class)
25. Data Mining 25
Bayesian Classification
• Bayesian classifiers are statistical classifiers.
– They can predict class membership probabilities such as the probability that a given tuple
belongs to a particular class.
• Bayesian classification is based on Bayes Theorem.
• A simple Bayesian classifier known as the Naïve Bayesian Classifier to be
comparable in performance with decision tree and selected neural network classifiers.
– Bayesian classifiers exhibits high accuracy and speed when applied to large databases.
• Even when Bayesian methods are computationally intractable, they can provide a
standard of optimal decision making against which other methods can be measured
26. Data Mining 26
Bayes Theorem
• P(A) is prior probability (unconditional probability) of event A.
• P(A|B) is posterior probability (conditional probability) of event A given that
event B holds.
• P(A,B) is the joint probability of two events A and B.
– The (unconditional) probability of the events A and B occurring together.
– P(A,B) = P(B,A)
27. Data Mining 27
Bayes Theorem
P(A|B) = P(A,B) / P(B)
P(B|A) = P(B,A) / P(A)
P(A,B) = P(A|B)*P(B)
P(B,A) = P(B|A)*P(A)
Since P(A,B) = P(B,A), we have P(A|B)*P(B) = P(B|A)*P(A)
Thus, we have Bayes Theorem
P(A|B) = P(B|A)*P(A) / P(B)
P(B|A) = P(A|B)*P(B) / P(A)
29. Bayes Theorem - Example
• Given:
– A doctor knows that meningitis causes stiff neck 50% of the
time P(S|M) = 0.5
– Prior probability of any patient having meningitis is
1/50,000 P(M) = 1/50,000
– Prior probability of any patient having stiff neck is 1/20
P(S) = 1/20
• If a patient has stiff neck, what’s the probability he/she has meningitis? P(M|S) ?
𝐏 𝐌 𝐒 = =
𝐏 𝐒 𝐌 𝐏(𝐌) 𝟎. 𝟓 × 𝟏/𝟓𝟎𝟎𝟎𝟎
𝐏(𝐒) 𝟏/𝟐𝟎
Data Mining 29
= 𝟎. 𝟎𝟎𝟎𝟐
30. Data Mining 30
Independence of Events
• The events A and B are INDEPENDENT if and only if P(A,B) = P(A)*P(B)
Example: Bit strings of length 3 is {000,001,010,011,100,101,110,111}
Event A: A randomly generated bit string of length three begins with a 1.
Event B: A randomly generated bit string of length three ends with a 1.
P(A) = 4/8 100,101,110,111 P(B) = 4/8 001,011,101,111
P(A,B) = 2/8 101,111 Are A and B independent?
P(A)*P(B) = (4/8) * (4/8) = 16/64 = 2/8 = P(A,B)
A and B are independent.
Event C: A randomly generated bit string of length three contains with two 1s.
P(C) = 3/8
P(A,C) = 2/8
011,101,110
101,110 Are A and C independent?
P(A)*P(C) = (4/8)*(3/8) = 12/64 = 3/16 ≠ 2/8
A and C are NOT independent.
31. Bayes Theorem for Prediction
• Let X be a data sample: its class label is unknown.
• Let H be a hypothesis that X belongs to class C.
• Classification is to determine P(H|X), (i.e., posteriori probability): the probability
that the hypothesis holds given the observed data sample X.
• P(H) (prior probability): the initial probability of H
– E.g., X will buy computer, regardless of age, income, …
• P(X): probability that sample data is observed
• P(X|H) (likelihood): the probability of observing the sample X, given that the
hypothesis H holds.
Bayes Theorem:
• Predicts X belongs to Ci iff the probability P(Ci|X) is the highest among all the
P(Ck|X) for all the k classes.
𝐏 𝐇 𝐗
𝐏 𝐗 𝐇 𝐏(𝐇)
=
𝐏(𝐗)
Data Mining 31
32. Data Mining 32
Naïve Bayes Classifier
• Let D be a training set of tuples and their associated class labels, and each tuple is
represented by an attribute vector (x1, x2, …, xn)
– AttributesA1, A2, …, An have values A1=x1, A2=x2, …, An=xn
• Suppose there are m classes C1, C2, …, Cm.
• We are looking the classification of the tuple (x1, x2, …, xn).
• The classification of this tuple will be the class Ci that maximizes the following
conditional probability.
P(Ci | x1, x2, …, xn )
33. Data Mining 33
Naïve Bayes Classifier
• To compute P(Ci | x1, x2, …, xn ) is almost impossible for a real data set.
• We use Bayes Theorem to find this conditional probability.
P(Ci | x1, x2, …, xn ) = P( x1, x2, …, xn | Ci ) * P(Ci) / P(x1, x2, …, xn)
• Since P(x1, x2, …, xn) is constant for all classes, we only look at the class Ci that
maximizes the following formula.
P( x1, x2, …, xn | Ci ) * P(Ci)
• We should compute P( x1, x2, …, xn | Ci ) and P(Ci) from the training dataset.
34. Data Mining 34
Naïve Bayes Classifier
Computing Probabilities
• To compute P(Ci) from the dataset is easy.
P(Ci) = NCi / N where NCi is the number of tuples belong to class Ci and
N is the number of the total tuples in the dataset.
• But, to compute P( x1, x2, …, xn | Ci ) from the dataset is NOT easy.
– In fact, it is almost impossible for a dataset with many attributes.
– If we have n binary attributes, the number of possible tuples is 2n.
35. Data Mining 35
Naïve Bayes Classifier
Computing Probabilities – Independence Assumption
• In order to compute P( x1, x2, …, xn | Ci ), we make independence assumption for
attributes although this assumption may not be true.
Independence Assumption: Attributes are conditionally independent (i.e., no
dependence relation between attributes)
P( x1, x2, …, xn | Ci ) = P( x1|Ci ) * P(x2|Ci ) * … * P(xn |Ci )
• If Ak is categorical,
P(xk|Ci) is the # of tuples in Ci having value xk for Ak divided by
|Ci| (# of tuples of Ci in the dataset)
36. Naïve Bayes Classifier
Computing Probabilities – continuous-valued attribute
• If Ak is a continuous-valued attribute,
P(xk|Ci) is usually computed based on Gaussian distribution with
a mean μ and standard deviation σ
and P(xk|Ci) is g(𝐱𝐤, 𝛍𝐂𝐢, 𝜎𝐂𝐢)
• Or we can discretize the continuous-valued attribute first.
𝟏
𝟐𝛑𝜎𝟐
g 𝐱, 𝛍, 𝝈 = 𝐞
−𝟏 𝐱−𝛍
𝟐 𝜎
Data Mining 36
𝟐
37. Data Mining 37
Naïve Bayes Classifier
Computing Probabilities from Training Dataset
Dataset has 14 tuples.
Two classes:
buyscomputer=yes
buyscomputer=no
P(bc=yes) = 9/14
P(bc=no) = 5/14
age income student creditrating buyscomputer
<=30 high no fair no
<=30 high no excellent no
31…40 high no fair yes
>40 medium no fair yes
>40 low yes fair yes
>40 low yes excellent no
31…40 low yes excellent yes
<=30 medium no fair no
<=30 low yes fair yes
>40 medium yes fair yes
<=30 medium yes excellent yes
31…40 medium no excellent yes
31…40 high yes fair yes
>40 medium no excellent no
38. Naïve Bayes Classifier
Computing Probabilities from Training Dataset
P(age=b31|bc=yes)=2/9
P(age=i31|bc=yes)=4/9
P(age=g40|bc=yes)=3/9
P(age=b31|bc=no)=3/5
P(age=i31|bc=no)=0
P(age=g40|bc=no)=2/5
P(inc=high|bc=yes)=2/9
P(inc=med|bc=yes)=4/9
P(inc=low|bc=yes)=3/9
P(inc=high|bc=no)=2/5
P(inc=med|bc=no)=2/5
P(inc=low|bc=no)=1/5
P(std=yes|bc=yes)=6/9
P(std=no|bc=yes)=3/9
P(std=yes|bc=no)=1/5
P(std=no|bc=no)=4/5
P(cr=exc|bc=yes)=3/9
P(cr=fair|bc=yes)=6/9
P(cr=exc|bc=no)=3/5
P(cr=fair|bc=no)=2/5
P(bc=yes) = 9/14
P(bc=no) = 5/14
age income student creditrating buyscomputer
<=30 high no fair no
<=30 high no excellent no
31…40 high no fair yes
>40 medium no fair yes
>40 low yes fair yes
>40 low yes excellent no
31…40 low yes excellent yes
<=30 medium no fair no
<=30 low yes fair yes
>40 medium yes fair yes
<=30 medium yes excellent yes
31…40 medium no excellent yes
31…40 high yes fair yes
>40 medium no excellent no
Data Mining 38
40. Data Mining 40
Naïve Bayes Classifier - Example
• Compute all probabilities for Naïve Bayes Classifier.
• Find the classification of the tuple (A=T,B=F)
• What is the confidence of that classification?
A B Class
T F Yes
T F No
T T Yes
T F Yes
F T Yes
F T No
F F No
41. Data Mining 41
Naïve Bayes Classifier - Example
P(yes) = 4/7 P(no) = 3/7
P(A=T|yes) = 3/4
P(A=F|yes) = 1/4
P(A=T|no) = 1/3
P(A=F|no) = 2/3
P(B=T|yes) = 2/4
P(B=F|yes) = 2/4
P(B=T|no) = 1/3
P(B=F|no) = 2/3
P(A=T,B=F|yes) = P(A=T|yes)*P(B=F|yes) = 3/4 * 2/4 = 6/16
P(A=T,B=F|no) = P(A=T|no)*P(B=F|no) = 1/3 * 2/3 = 2/9
P(A=T,B=F|yes)*P(yes) = 6/16 * 4/7 = 24/112 = 0.214
P(A=T,B=F|no)*P(no) = 2/9 * 3/7 = 6/63 = 0.095
Classification is YES
Confidence: 0.214 / (0.214+0.095) = 0.69 69%
A B Class
T F Yes
T F No
T T Yes
T F Yes
F T Yes
F T No
F F No
42. Data Mining 42
Avoiding the Zero-Probability Problem
• Naïve Bayesian prediction requires each conditional probability to be a non-zero
value. Otherwise, the predicted probability will be zero
P( x1, x2, …, xn | Ci ) = P( x1|Ci ) * P(x2|Ci ) * … * P(xn |Ci )
• In order to avoid zero probability values, we apply smoothing techniques.
• One of these smoothing techniques is add-one smoothing (Laplacian correction).
Smoothed Values
P(A=v1|Ci) = (Nv1Ci + 1) / (Nci+3)
P(A=v2|Ci) = (Nv2Ci + 1) / (Nci+3)
P(A=v1|Ci) = (Nv3Ci + 1) / (Nci+3)
P(A=v1|Ci) = Nv1Ci / Nci
P(A=v2|Ci) = Nv2Ci / Nci
P(A=v3|Ci) = Nv3Ci / Nci
43. Data Mining 43
Avoiding the Zero-Probability Problem
P(age=b31|bc=no)=3/5
P(age=i31|bc=no)=0
P(age=g40|bc=no)=2/5
After add-one smoothing:
P(age=b31|bc=no)=(3+1)/(5+3) = 4/8
P(age=i31|bc=no)=(0+1)/(5+3) = 1/8
P(age=g40|bc=no)=(2+1)/(5+3) = 3/8
44. Data Mining 44
Naïve Bayes Classifier: Comments
• Advantages
– Easy to implement
– Good results obtained in most of the cases
• Disadvantages
– Assumption: class conditional independence, therefore loss of accuracy
– Practically, dependencies exist among variables
• E.g., hospitals: patients: Profile: age, family history, etc.
Symptoms: fever, cough etc., Disease: lung cancer, diabetes, etc.
• Dependencies among these cannot be modeled by Naïve Bayes Classifier
• How to deal with these dependencies? Bayesian Belief Networks