From: http://www.norsys.com/tutorials/netica/secA/tut_A1.htm As your clinic grows and you handle hundreds of patient cases, you learn that while the text books may have described the North American situation, the reality of your clinic and its population of patients is very different. This is what your data collection efforts reveal: 50% of your patients smoke. 1% have TB. 5.5% have lung cancer. 45% have some form of mild or chronic bronchitis. You enter these new figures into your net, and now you have a practical Bayes net, one that really describes the kind of patient you typically deal with. So, let us see how we would use this net in our daily medical practice. The first thing we should note is that the above net describes a new patient, one whom has just been referred to us, and for whom we have no knowledge whatsoever, other than that they are from our target population. As we acquire knowledge specific to each particular patient, the probabilities in the net will automatically adjust. This is the great beauty and power of Bayesian inference in action. And the great strength of the Bayes net approach is that the probabilities that result at each stage of knowlege buildup are mathematically and scientifically sound. In other words, given whatever knowledge we have about our patient, then based on the best mathematical and statistical knowledge to date, the net will tell us what we can legitimately conclude. This is a very powerful tool, indeed. Take a moment to think on it. You as a doctor are not just relying on hunches, or an intuitive sense of the likelihood of illness, as you may have in the past, but, rather, on a scientifically and provably accurate estimate of the likelihood of illness, one that gets more and more accurate as you gain knowledge about the particular patient, or about the particular population that the patient comes from. So, let us see how adding knowledge about a particular patient adjusts the probabilities. Let us say a woman walks in, a new patient, and we begin talking to her. She tells us that she is often short of breath (dyspnea). So, we enter that finding into our net. With Netica we shall see, this is as simple as pointing your mouse at a node and clicking on it once, whereupon a list of available states pops up, and you then click on the correct item in the list. After doing that, this is what the net looks like. Notice how the Dyspnea box is grayed, indicating that we have evidence for it being in one of its states. In this case, because our patient appears trustworthy, we say we are 100% certain that our patient has dyspnea. It is easy with Netica to enter an uncertain finding (also called a likelihood finding), say of 90% Present, but let's keep things simple for now. Observe how with this new finding, that our patient has dyspnea, that the probabilities for all three illnesses has increased. Why is this? Well, since all those illnesses have dyspnea as a symptom, because our patient is indeed exhibiting this symptom, it only makes sense that our belief in the possible presence of those illnesses should increase. Basically, the presence of the symptom has increased our belief that she might be seriously ill. Let's look at those inferences more closely. 1.The most significant jump is for Bronchitis, from 45% to 83.4%. Why such a large jump? Well, bronchitis is far more common than cancer or TB. So, once we have evidence for serious lung illness, it becomes our most likely candidate diagnosis. 2.The chances that our patient is a smoker has now increased substantially, from 50% to 63.4%. 3.The chances that she recently visited Asia has increased very slightly: from 1% to 1.03%, which is insignificant. 4.The chances of our getting an abnormal X-Ray from our patient has also gone up marginally, from 11% to 16%. If you think about this expansion of our knowledge, it is truly quite helpful. We have only entered one finding, the presence of Dyspnea, and this knowledge has &quot;propagated&quot; or spread its influence around the net, accurately updating all the other possible beliefs. Some of our beliefs are increased substantially, others hardly at all. And the beauty of it is that the amounts are precisely quantified. We still do not know what precisely is ailing our patient. Our current best belief is that she suffers from Bronchitis (probability of Present=83.4%). However, we would like to increase our chances of a correct diagnosis. If we stop here and diagnose her with Bronchitis and she really has Cancer, we would be a poor doctor indeed. We really need more information. So, being thorough, we run through our standard check-list of questions. We ask her if she has been to Asia recently. Surprisingly, she answers &quot;yes&quot;. Now, let us see how this knowledge affects the net. Suddenly, the chances of tuberculosis has increased substantially, from 2% to 9%. Note, interestingly, that the chances of lung cancer, bronchitis, or of our patient being a smoker all have decreased. Why is this? Well, this is because the explanation of dyspnea is now more strongly explained by tuberculosis than before (although bronchitis still remains the best candidate diagnosis). And because cancer and bronchitis are now less probable, so is smoking. This phenomenon is called &quot;explaining away&quot; in Bayes net circles. It says that when you have competing possible causes for some event, and the chances of one of those causes increases, the chances of the other causes must decline since they are being &quot;explained away&quot; by the first explanation. To continue with our example, suppose we ask more questions and find out that our patient is indeed a smoker. Here is the updated net. Note that our current best hypothesis still remains that the patient is suffering from Bronchitis, and not TB or lung cancer. But to be sure, we order a diagnostic X-Ray be performed. Let us say that the X-ray turns out normal. The result is: Note how this more strongly confirms Bronchitis and disconfirms TB or lung cancer. But suppose the X-ray were abnormal. The result is: Note the big difference. TB or Lung Cancer has shot up enormously in probability. Bronchitis is still the most probable of the three separate illnesses, but it is less than the combination hypothesis of TB or Lung Cancer. So, we would then decide to perform further tests, order blood tests, lung tissue biopsies, and so forth. Our current Bayes net does not cover those tests, but it would be easy to extend it by simply adding extra nodes as we acquire new statistics for those diagnostic procedures. And we do not need to throw away any part of the previous net. This is another powerful feature of Bayes nets. They are easily extended (or reduced, simplified) to suit your changing needs and your changing knowledge.
Transcript
1.
Data Mining - CSE5230 Classifiers 1 Bayesian Classification and Bayesian Networks CSE5230/DMS/2004/5
Whenever you are presented with a measure of classification performance, you should consider whether it is doing better than chance
A measure exists that indicates how much better than chance performance actually is. This is known as Cohen’s -statistic (kappa statistic) [Dun1989]:
where frequencies are in [0,1].
There are also other, more sophisticated measures of classifier performance. Good data mining packages, such as Weka, report several
Consider the Venn diagram at right. The area of the rectangle is 1, and the area of each region gives the probability of the event(s) associated with that region
P(A|B) means “the probability of observing event A given that event B has already been observed”, i.e.
how much of the time that we see B do we also see A? (i.e. the ratio of the purple region to the magenta region)
Bayes Theorem - 1 P(A|B) = P(A B)/P(B), and also P(B|A) = P(A B)/P(A), therefore P(A|B) = P(B|A)P(A)/P(B) (Bayes formula for two events) P(A) P(B) P(A B)
i.e. the probability that any given data is a kangaroo regardless of it method of locomotion or night time behaviour - i.e. before we know anything about X
Similarly, P(X|H) is the posterior probability of X conditioned on H
i.e the probability that X is a jumper and is nocturnal given that we know X is a kangaroo
Assumes that the effect of an attribute value on a given class is independent of the values of other attributes. This assumption is known as class conditional independence
This makes the calculations involved easier, but makes a simplistic assumption - hence the term “naïve”
Can you think of an real-life example where the class conditional independence assumption would break down?
Consider each data instance to be an n -dimensional vector of attribute values (i.e. features):
Given m classes C 1 ,C 2, …, C m , a data instance X is assigned to the class for which it has the greatest posterior probability, conditioned on X , i.e. X is assigned to C i if and only if
It is can be very expensive to compute the P(X|C i )
if each component x k can have one of c values, there are c n possible values of X to consider
Consequently, the (naïve) assumption of class conditional independence is often made, giving
The P(x 1 |C i ),…, P(x n |C i ) can be estimated from a training sample (using the proportions if the variable is categorical; using a normal distribution and the calculated mean and standard deviation of each class if it is continuous)
Fully computed Bayesian classifiers are provably optimal
i.e. under the assumptions given, no other classifier can give better performance
In practice, assumptions are made to simplify calculations (e.g. class conditional independence), so optimal performance is not achieved
sub-optimal performance is due to inaccuracies in the assumptions made
Nevertheless, the performance of the Naïve Bayes Classifier is often comparable to that decision trees and neural networks [p. 299, HaK2000], and has been shown to be optimal under conditions somewhat broader than class conditional independence [DoP1996]
Problem with the naïve Bayesian classifier: dependencies do exist between attributes
Bayesian Belief Networks (BBNs) allow for the specification of the joint conditional probability distributions : the class conditional dependencies can be defined between subsets of attributes
i.e. we can make use of prior knowledge
A BBN consists of two components. The first is a directed acyclic graph where
each node represents an variable; variables may correspond to actual data attributes or to “hidden variables”
each arc represents a probabilistic dependence
each variable is conditionally independent of its non-descendents, given its parents
A node within the BBN can be selected as an output node
output nodes represent class label attributes
there may be more than one output node
The classification process, rather than returning a single class label (as many classifiers, e.g. decision trees, do) can return a probability distribution for the class labels
i.e. an estimate of the probability that the data instance belongs to each class
A Machine learning algorithm is needed to find the CPTs, and possibly the network structure
If the network structure is known and all the variables are observable then training the network simply requires the calculation of Conditional Probability Table (as in naïve Bayesian classification)
When the network structure is given but some of the variables are hidden (variables believed to influence but not observable) a gradient descent method can be used to train the BBN based on the training data. The aim is to learn the values of the CPT entries
Let w ijk be a CPT entry for the variable Y i = y ij having parents U i = u ik
e.g. from our example, Y i may be LungCancer, y ij its value “True”, U i lists the parents of Y i , e.g. {FamilyHistory, Smoker}, and u ik lists the values of the parent nodes, e.g. {“True”, “True”}
The w ijk are analogous to weights in a neural network, and can be optimized using gradient descent (the same learning technique as backpropagation is based on). See [HaK2000] for details
An important advance in the training of BBNs was the development of Markov Chain Monte Carlo methods [Nea1993]
[ClN1989] Peter Clark and Tim Niblett, The CN2 Induction Algorithm , Machine Learning 3(4), pp. 261-283, 1989.
[DHS2000] Richard O. Duda, Peter E. Hart and David G. Stork, Pattern Classification (2nd Edn), Wiley, New York, NY, 2000
[DoP1996] Pedro Domingos and Michael Pazzani. Beyond independence: Conditions for the optimality of the simple Bayesian classiffer , In Proceedings of the 13th International Conference on Machine Learning, pp. 105-112, 1996.
[Dun1989] Graham Dunn, Design and Analysis of Reliability Studies: The Statistical Evaluation of Measurement Errors, Edward Arnold, London, 1989.
[Gra2003] Paul Graham, Better Bayesian Filtering , In Proceedings of the 2003 Spam Conference, Cambridge, MA, USA, January 17 2003
[HaK2000] Jiawei Han and Micheline Kamber, Data Mining: Concepts and Techniques , The Morgan Kaufmann Series in Data Management Systems, Jim Gray, Series (Ed.), Morgan Kaufmann Publishers, August 2000
[Nea2001] Radford Neal, What is Bayesian Learning?, in comp.ai.neural-nets FAQ, Part 3 of 7: Generalization, on-line resource, accessed September 2001 http://www. faqs .org/ faqs / ai - faq /neural-nets/part3/section-7.html
[Nea1993] Radford Neal, Probabilistic inference using Markov chain Monte Carlo methods . Technical Report CRG-TR-93-1, Department of Computer Science, University of Toronto, 1993
[PaL1998] Patrick Pantel and Dekang Lin, SpamCop: A Spam Classification & Organization Program , In AAAI Workshop on Learning for Text Categorization, Madison, Wisconsin, July 1998.
[SDH1998] Mehran Sahami, Susan Dumais, David Heckerman and Eric Horvitz, A Bayesian Approach to Filtering Junk E-mail , In AAAI Workshop on Learning for Text Categorization, Madison, Wisconsin, July 1998.
Be the first to comment