This document summarizes a paper about refining inductive knowledge-based systems through empirical methods. It describes a system called ARIS that interleaves learning and performance evaluation to allow accurate classifications on real-world datasets. ARIS employs an ordering of rules according to their learned weights, which are calculated using Bayes' theorem. The system focuses rule analysis and refinement by heuristics. It has been used to refine knowledge bases from C4.5 and RIPPER, showing improved accuracy and the ability to gradually enhance the knowledge base during refinement.

1. RULE REFINEMENT IN INDUCTIVE KNOWLEDGE-BASED
SYSTEMS
by
Mohamed Arteimi
Department of Computer Science
Witwatersrand University
Johannesburg, 1999

2. Abstract
This paper presents empirical methods for enhancing the accuracy of inductive learning systems. It
addresses the problems of: learning propositional production rules in multi-class classification tasks in
noisy domains, maintaining continuous learning when confronted with new situations after the learning
phase is completed, and classifying an object when no rule is satisfied for it.
It is shown that interleaving the learning and performance-evaluation processes, allows accurate
classifications to be made on real-world data sets. The paper presents the system ARIS which
implements this approach, and it is shown that the resulting classifications are often more accurate than
those made by the non-refined knowledge bases.
The core design decision that lies behind ARIS is that it employs an ordering of the rules according to
their weight. A rule’s weight is learned by using Bayes’ theorem to calculate weights for the rule’s
conditions and to combine them. This model focuses the analysis of the knowledge base and assists the
refinement process significantly.
The system is non-interactive, it relies on heuristics to focus the refinement on those experiments that
appear to be most consistent with the refinement data set. The design framework of ARIS consists of a
tabular model for expressing rule weights, and the relationship between refinement cases and the rules
satisfied for each case to focus the refinement process. The system has been used to refine knowledge
bases created by ARIS itself, as well as to refine knowledge bases created by the RIPPER and C4.5
systems [6,25] in ten selected domains. Two main advantages have been observed. First, the ability to
gradually improve the knowledge base as the refinement proceeds. Second, is the ability to learn strong
rules utilising condition’s weight using minimum covering algorithm. Thus, by taking account of this
information we improve the applicability and quality of refinement.
Keywords: knowledge-base refinement; inductive learning.
1.
Introduction
The growth in database technology and computer communication have resulted in the creation of huge,
efficient data stores in almost every domain. For example, credit-card transactions, medical images, and
large-scale sky surveys are all stored in massive and ever growing databases. These streams of data need
to be analysed to extract higher-level information that could be useful for decision making and for
understanding the process generating the data. For large systems of this type, we need automated
methods that acquire decision-making knowledge.
Humans attempt to understand their environment by using a simplification of this environment (called a
model). This model represents events in the environment, and similarities among objects [12]. Similar
objects are grouped in classes and rules are constructed to predict the behaviour of new objects of such a
1

3. class. In machine learning we try to automate this process and construct class descriptions (models) such
as a decision tree or a set of production rules using an iterative search strategy on a library of examples.
This is known as inductive learning.
There have been various methods for learning rules inductively by analysing a set of training examples
to extract interesting and relevant information from the databases. Among these are RIPPER [6], and
C4.5 [25].
This information is higher-level knowledge in the form of class descriptions that are
generalisations about what it means to be a member of a class or a category. This gives us the ability to
make inferences that are true for most or all members of the class.
In this paper we present a method for improving or refining existing knowledge bases by using a
refinement library of examples.
Section 2 gives an overview of two existing refinement systems, and inductive learning systems. Section
3 describes the ARIS knowledge-base refinement system. The application of ARIS to knowledge bases
generated by C4.5 and RIPPER systems is described in Section 4. Section 5 compares inductive
refinement and incremental learning. Section 6 concludes with discussion and future enhancements.
2
Related Work
2.1 Inductive systems
There has been a relatively long history of induction to solve complex problems, by creating a model of
the problem domain, and inferring from that model instead of from the domain itself. The constructed
model is a particularly useful tool if it is possible to perform classification by simple, easy-to-understand
tests, whose semantics are intuitively clear, such as decision trees [3,5,9] and production rules [1,6,25].
Learning algorithms vary in:
-
the way in which they search the training data set,
-
the way in which they generalise,
-
the way they represent class descriptions, and
-
the way in which they cope with errors and noise in the training data.
Generally, we can classify concept learners into two categories depending on the learning strategy
involved.
S
Simultaneous covering algorithms
At each step in the search, the algorithm selects among alternative attributes by comparing the
partitions of the data that each attribute generates. Decision-tree algorithms belong to this category,
examples include ID3 [24], C4.5 [25], and CART [3]. These algorithms use an overfit-and-simplify
learning strategy involving post pruning of the generated trees.
2

4. S
Sequential covering algorithms
Here, the algorithm chooses among alternative attribute-value pairs to include in rules by computing
the subsets of data they cover. Direct rule learning algorithms such as RIPPER [6], and HYDRA [1]
belong to this category.
Two variations on this approach have been explored:
1.
General-to-specific search
An approach to learning one rule at a time is to organise the search in the hypothesis space in
the same way as in simultaneous covering algorithms, but to follow only the most promising
branch in the tree at each step. The search starts by considering the most general condition
possible (i.e. the empty test that matches every instance), then adds the attribute test that most
improves rule performance measured over the training data. Once this test has been added, the
process is repeated by adding a second attribute test, and so on. This is different from
simultaneous covering approach since at each step only a single descendent is followed, rather
than growing a subtree that covers all possible values of the selected attributes.
Some learning systems such as BEXA [27] extend this approach to maintain a list of the k best
attribute tests at each step, rather than a single best attribute test.
2.
Specific-to-general search
Another way to organise the search space is to start the search process beginning with the most
specific rule and gradually generalise to cover more positive cases [18].
One way to see the significance of the difference between the above two approaches is to compare the
number of distinct choices made by the two algorithms in order to learn the same set of rules.
To learn a set of n rules, each containing k attribute-value tests in their conditions, sequential covering
algorithms will perform nk primitive search steps, making an independent decision to select each
condition of each rule. In contrast, simultaneous covering algorithms will make many fewer independent
choices, because each choice of a decision node in the decision tree corresponds to choosing the
condition for the multiple rules associated with that node. In other words, if the decision node tests an
attribute that has m possible values, the choice of the decision node corresponds to choosing a condition
for each of the m corresponding rules. Thus, sequential covering algorithms make a larger number of
independent choices than simultaneous covering algorithms.
The other difference is that in simultaneous covering algorithms different rules test the same attribute
while in sequential covering algorithms this behaviour is not required.
3

5. 2.2
Refinement systems
Previous knowledge-base refinement systems use knowledge bases elicited from experts in a fairly well
understood domains. The knowledge base is believed to be complete and only minor tweaking is
required. This section describes two such refinement systems SEEK [22] and KRUST [7]. They are the
forerunners of the ARIS refinement algorithm. The statistical approach of SEEK is the basis for ARIS’s
rating of rules and imbedded hypothesis. SEEK uses statistical evidence to apply suitable refinements.
The refinements are:
S
Generalise a rule which will fire if it is enabled.
S
Specialise the error-causing rule to allow an enabled correct rule to fire instead.
The rules in SEEK are represented in the EXPERT language but expressed in a tabular format that
divides the findings associated with a conclusion into two categories; major and minor, and the level of
confidence associated with a conclusion into three categories: definite, probable, and possible.
A list of major and minor symptoms is defined for each conclusion that the system can reach. For
example, major symptoms for connective tissue disease (one of the rheumatic diseases) include swollen
hands and severe myositis, while examples of minor symptoms include anemia and mild myositis.
The rule changes include the following operators:
1) Change the certainty of a diagnosis,
2) Increase/decrease the number of majors or minors (i.e. suggestive symptoms),
3) Add/delete an exclusion (i.e. a symptom that should not be present),
4) Add/delete a requirement (i.e. a necessary symptom) and,
5) Change the conclusion (RHS) of a rule.
The rule changes do not include numerical range changes. SEEK requires the involvement of an expert
to decide on whether or not a suggested change by SEEK can be implemented. The main criticism of
SEEK is that it uses a criterion table for representing the hypotheses. This works well in the domain of
Rheumatoid diseases, but may be difficult to implement in other domains. SEEK is also unable to add
new rules.
Politakis [22] gives a method to integrate condition weights and rule weights for classification. The
method used relies on a very specific set of weights regarding the relationship between a set of
conditions and a conclusion. The condition weight can either be minor or major while the weight of a
rule can have a value in the set {possible, probable, definite} depending on the combination of
conditions used in a rule. This technique is strongly connected to our implementation of condition and
rule weights as will be discussed in Section 3. The proposed scheme in this paper is dynamically
adjusted with respect to a probabilistic estimate (depending on the data distribution) and it is defined on
a continuous scale, while in SEEK system an expert is required to assign weights for both conditions and
rules.
4

6. The use of an exclusion component in SEEK is another important aspect that is similar to the proposed
approach. We use an UNLESS clause to rule out a conclusion for a particular situation taking care of
special cases in the domain of interest.
The KRUST system [7] refines a knowledge base when there is a conflict between the expert’s
conclusion and the system’s conclusion for a particular test case. This test case is one of a bank of tasksolution pairs used by KRUST. KRUST normally outputs a single knowledge base, based on the
performance of a set of proposed refined knowledge bases on all the task-solution pairs. In the event of a
tie for best knowledge base, or if the original knowledge base is judged best, the expert is consulted.
However, unlike SEEK, KRUST does not require the involvement of an expert during refinement.
KRUST follows the “first enabled is fired” conflict resolution strategy, and its refinement operations are:
S
Generalise a rule,
S
Specialise a rule,
S
Change the priority of a rule,
S
Add a new rule for the misdiagnosed case.
KRUST’s generation of a new rule is naïve [7], and needs to be improved. Again KRUST believes that
the original knowledge base is complete, and no redundant rules exist. Hence, there is no redundancy
checking to remove undesirable rules. Moreover, there is no way to handle or exclude special cases as is
the case with SEEK.
KRUST’s approach in generating many refinements is based on the realisation that the application of a
variety of refinement operations could lead to knowledge bases that have different performances. The
research in this paper emphasises the need for trying all possible refinement operations that are
applicable for the observed misdiagnoses and to choose one that best improves the knowledge-base
performance.
3.
The ARIS inductive refinement system
This section is devoted to a description of the mechanisms involved in representing/creating the
knowledge base, improving the knowledge base, and the associated weighting schemes. This is
implemented in a system called ARIS (standing for Automated Refinement of Inductive Systems) which
refines propositional production rules. The architecture of ARIS is shown in Figure 1, and the main
features are:
•
a rule weighting scheme for providing rule priorities and ordering,
•
ranking refinement of cases by combining the weights of rules satisfied for each case,
•
ability to classify an object, when no rule is satisfied, through partially satisfied rules, and
•
ability to perform all refinement operations on continuous and discrete attributes.
5

7. Refinement
data
Test data
Rule generator
Rule classifier
Rule
Tree generator
Training data
(inference engine)
base
Ordered
rules
Refinement
generator
Prediction result
Classifier
Refined rule set
Refined
knowledge base
Refinement module
Architecture
Figure 1. The ARIS architecture
ARIS initially generates a knowledge base using induction on a set of training examples, it then
proceeds to test the knowledge base on a separate set of testing data for refinement purposes called the
refinement data set. It is only after this testing, and if some cases in the refinement set are mis-classified,
that the refinement subsystem is invoked. Finally, the system is tested on a separate testing data set for
evaluation of the refinement process. The refinement subsystem identifies possible errors in the
knowledge base, and calls on a library of operators to develop possible refinements, guided by a global
heuristic. The best refinement is implemented, and the process repeats until no further refinements are
possible. A knowledge-based refinement system is successful to the extent that it can improve the
accuracy of its initial knowledge base on new and unseen cases and produce a more accurate and
comprehensible knowledge base than the initial knowledge base. ARIS performs a search through a
space of both specialisation and generalisation operators in an attempt to find a minimal refinement to a
knowledge base.
Conceptually, the refinement subsystem has three main phases, two of them are executed for every
hypothesis present in the knowledge base, while keeping the rules ordered with respect to their weights:
Phase 1: (Localisation process)
During the first phase, all misdiagnosed cases from the refinement set belonging to a particular
hypothesis (class) are identified. Each case receives a weight from the rules satisfied for the
case. This indicates the rule overlapping at this point (case) in hypothesis space. The case that
has the highest absolute weight among the misdiagnosed cases is selected, since it points to the
strongest rule from the set of erroneous rules (i.e. with highest weight) and frequently many
rules overlap at this point.
6

8. Phase 2: (Refinement generation, verification and selection)
During this phase, the rule responsible for the misdiagnoses is determined, and all possible
refinement operations applicable are tried. Namely; the erroneous rule is specialised, a similar
rule reaching the intended class is generalised, and a new rule is created. The resultant
performance is stored for each refinement operator and finally the refinement operator or group
of operators that produce the best performance is chosen. The process is repeated until no
further improvement is possible.
Phase 3: (Check completeness and remove superfluous rules)
Finally, the knowledge base is checked for incompleteness. Each case must be covered by at
least one rule. If there are cases that are not covered by the existing rules, new rules are created.
Moreover, any superfluous rules are removed. Figure 2 demonstrates the algorithm.
The main components of ARIS are: a tree generator, a rule generator, a refinement generator, a
judgement module, and an inference engine. The refinement generator is responsible for applying all
possible refinements to remedy the misdiagnoses. The set of refinements reflect how the rules interact.
Rules can be changed by allowing them to fire (called enabling), or preventing them from firing (called
disabling). The judgement module selects the best refinement operator or group of operators which
result in the best improvement of knowledge-base performance while correcting the misdiagnosed
cases.
7

9. Algorithm : ARIS
Inputs: A set of training data
A set of refinement data,
A set of test data,
Optionally, a set of rules of the form:
IF conditions THEN hypothesis
Output: Refined rule set of either
IF conditions THEN hypothesis
Or
IF conditions THEN hypothesis
UNLESS
Conditions
First:
If no rules are given generate a set of production rules by processing the training data,
Second:
1.
for hypothesis ←each hypothesis in the hypothesis space
{Identify rules that are not consistent with refinement data set. These may at later stages be
specialised but will not be generalised. }
•
Rank rules with respect to their weights.
•
For each misdiagnosed case in the refinement set, calculate combined weight of rules satisfied
for the case (see Section 3.1).
•
Select a case that has the highest absolute weight among the misdiagnosed cases.
•
Refine the knowledge base for the selected case (i.e. missed case) by generating all possible
refinements.
•
Select the best refinement, and implement it.
End for
2.
•
Maintain knowledge-base completeness, by guaranteeing that each case is covered by at least
one rule, use a minimum covering algorithm to generate missing rules (see Section 3.4).
•
Prune all superfluous rules.
Figure 2: The ARIS algorithm
3.1 Learning condition weights
Feature weights have been used in case-based learning [14,19] to denote the relevance of features in
similarity computations and to guide the selection of relevant features. The basic approach involves
storing cases and their associated classes in memory and then, when given a test instance, the task
8

10. becomes one of finding the training cases nearest to that test case and using them to predict the class of
the test instance. Several methods for constructing feature weights have been proposed, based on either a
naïve distance metric that weighs attributes equally [28] or a weighted Euclidean metric in which the
weight for each feature comes from a small set of options, for example [0,0.1,0.2,…,1], where 0 means
that the feature is irrelevant, and higher weight denotes greater relevance such as in [19]. Some methods
cannot handle numeric features directly, and have to discretise numeric values by classification; others
have to convert multi-valued symbolic features to a set of binary features with respect to the number of
its values (i.e. a symbolic feature with N values is mapped into N-1 binary features) such as
quantification method II [19].
Politakis [22] used a domain specific scale for weighing conditions and rules. His approach can be
summarised as follows:
The conditions in a rule are grouped under the following categories:
majors
considered to be very important conditions,
minors
less specific but suggestive for reaching a particular conclusion,
exclusions
conditions that must not be satisfied in order to reach a given
conclusion.
The certainty in the conclusion of a rule is measured by the confidence level it reaches. There are three
levels of confidence:
(1)
Possible,
(2)
Probable, and
(3)
Definite.
Different combinations of conditions could result in reaching the same conclusion but with different
certainty.
In contrast, the only source of information we have is the training data. In this research, conditional
probability theory is used to take advantage of the data distribution in the training set. The intention is to
measure the importance of a condition to a particular class, through the measure of change in belief in
the class with respect to that condition. This poses the following question. For a particular hypothesis h,
if given a piece of evidence e, how will the relation between h and e change?
To formalise the concept, let us consider a sample space that is partitioned by events E1,E2,…En.
Let H+ be an event in the space denoting a certain class or a concept with probability of P(H+)>0; Then,
by Bayes’ Theorem
+
P( E j | H ) =
P( H + | E j )P( E j )
n
∑ P( H
+
| E i )P( E i )
i =1
For a simple problem, with two conditions, and hypotheses H+ and HP( E 1 | H + ) =
P( H + | E 1 )P( E 1 )
P( H + | E 1 )P( E 1 ) + P( H + | E 2 )P( E 2 )
9

11. =
=
P( H+ | E1 )P( E1 )
P( H+ | E1 )P( E1 ) + P( H+ |~ E1 )P(~ E1 )
P( H+ | E1 )P( E1 )
P( H+ | E1 )P( E1 ) + P( H+ |~ E1 )(1 − P( E1 ))
We can rewrite this as
P( E 1 | H + ) = QA × P( E 1 )
(1)
Where
QA =
P( H + | E1 )
P( H + | E1 )P( E1 ) + P( H + |~ E1 )( 1 − P( E1 ))
Or using the terminology of Table 2.
QA =
X
1 X
3
X
+ X
1
2
(1 − X
3
)
If we take the logarithm of QA, the incremental nature of the updating process of the evidence on the
condition of the hypothesis becomes apparent. In other words, the posterior probability P(E1|H+) in
Equation 1 above, is dependent on the factor QA. If QA is greater than one, the posterior probability will
be greater than the prior probability P(E1); otherwise, if QA is less than one, it would result in P(E1|H+)
less than P(E1). We can thus view this quantity as a weight, carried by the evidence E1, which sways the
belief one way or the other. The weight QA calculated as above ranges from zero to infinity; however
we would like to map the values of the evidence weight within the interval [-1,+1] with a value of 1 for
QA mapping to 0 (central), so that it is easier to distinguish whether the given evidence is in favour of
the particular hypothesis or not. In other words positive weight denotes supporting evidence, and
negative weight indicates opposing evidence. Therefore the following function is used to map the value
to the desired range
W=F(QA)
Where
2 QA
−1
1 + QA
Another candidate for this transformation [23] is:
F ( QA ) =
F (QA) =
1
e
−QA
+1
−1
Some other possible forms have been suggested in the literature for this mapping [13].
10

12. Positive weight indicates a supporting evidence for the hypothesis, and negative weight indicates
opposing evidence against the hypothesis.
Symbol
Description
Ei
Indicates an event or evidence (eg. Age<=20)
~Ei
Indicates the complement of Ei (eg. Age>20)
H+
Denotes space of positive instances of hypothesis (eg. Healthy)
H
-
Denotes space of negative instances of hypothesis (eg. Sick)
P(Ei)
Represents the prior distribution of objects(cases) within the scope of the condition
relative to the total number of examples covered by the entire population (e.g.
P(Age>20)
X1 = P(H|Ei)
Fraction of positive instances of hypothesis covered by the condition Ei (i.e. True
positives TP)
X2 = P(H|~Ei)
Fraction of positive instances that are not covered by the condition Ei, relative to the
complement of Ei (i.e. False negatives FN)
X3=P(Ei)
Fraction of all instances covered by the evidence Ei, relative to the entire population
Table 2: Terminology for weight model
We emphasise that this weighting scheme defines class-specific weights, depending on class frequency
and according to the degree to which its distribution of values (for a class) differs from its distribution
for other classes.
3.2 Rule weights and ordering
In what follows, we discuss the rationale for weighting and ordering the rules in an inductive
knowledge-based system. The natural properties of production rules allow many ways for rules to
interact and overlap. In other words, a rule may collide with or subsume another rule, because the
available conditions are not usually independent. This requires some process to use the available
information to resolve any existing conflict and generalise well over the training data, while keeping the
rules as simple as possible.
Knowledge-based systems are characterised by the separation of rules, data, and control scheme [8], and
the task execution is accomplished through the following phases:
Phase1: Matching phase in which the knowledge-based system matches its rules against a
given
instance, to determine the conflict set (the set of rules satisfied),
Phase2: Conflict resolution. Here the system selects a rule to fire from the conflict set.
Phase3: Rule application, which entails execution of the selected rule by following its condition part.
11

13. The matching phase is known to be a time consuming operation, but the conflict resolution is
particularly important as it determines which rule will fire and hence how the case will be classified.
Several general schemes of conflict resolution have appeared in the literature utilising different kinds of
heuristics, for example:
1.
Rule specificity was used to favour specific rules [11].
2.
Connection strength [8] which measures rule strength that codes its “relatedness” to other rules in
the knowledge base for selecting the next rule to fire on the basis of experience. Their scheme
imposes an explicit priority ordering given the last rule satisfied during task execution. The
following heuristic is adopted:
Wherever rule A fires, then based on experience the most likely candidate rule to be fired and
eventually lead to success is rule B.
This idea is very different from our approach.
3
Logical sufficiency [2] is a probabilistic estimate of rule reliability, that measures the proportion of
the training examples of class c, for which the rule was true divided by the proportion of the training
examples of other classes for which the rule was true.
4
Belief certainty such as certainty factor model used in MYCIN [4] is used to represent uncertainty
to rules conclusion. This is based on expert estimates which are the integration of many factors
including probabilistic knowledge and attitude.
Traditionally, conflict resolution, requires computing the entire conflict set (except scheme 2 above),
and then selecting a rule according to certain property (like frequency of rule success). We propose the
addition of a different mechanism to the inference engine for ordering the rules with respect to a strength
of the association between rule’s conditions and the intended hypothesis (i.e. right hand side of the rule).
This is similar to Politakis’ approach [22] preferring rules with strongest confidence over rules with less
confidence.
The goal of refinement is to improve knowledge-base efficiency with as few modifications as possible.
This entails reducing the number of rules it has to consider for the purpose of improving the prediction
quality. For the proposed method we maintain a simple matcher that matches rules individually.
Selecting the next rule to match depends on its location (position in a list) and on its weight, thus
avoiding the expensive computation of the entire conflict set. This means that the first rule to match will
be fired, so there is never a conflict set. The rule weight can also be used for purging low utility rules
from the system.
Changing the priority of a rule affects its ability to fire, and it is the easiest operational change to
perform [7]. Several knowledge-based systems use a rule priority change as a plausible action in their
conflict resolution schemes [7,16]. Craw [8] advocates changing priority of a rule while rejecting the
change if it proves to be ineffective in later stages. Laird [16] supports the notion of changing control
12

14. rather than the conditions of the rules in his system Soar (i.e. modifying the decisions in which the
knowledge is used instead of modifying the rules that encode the knowledge) to recover from incorrect
decisions. In his system the decisions made on the basis of preferences are categorised into three classes
(acceptability, necessity, and desirability) to select a rule for firing. Other authors [29] took the opposite
extreme view point, namely, not to change the rule order under any circumstances; but there is no clear
evidence for keeping a specific ordering of the rules, if resorting to a small change like reordering could
improve knowledge-base efficiency. ARIS starts with an unordered set of rules, and imposes an order
required to keep the rules in line with their utility to the system, in the sense that:
(1) Provides as few matching operations as possible to produce the desired action without explicit
conflict resolution computation.
(2) Improves refinement efficiency by supplying a process for locating the rule responsible for
misdiagnoses directly.
For example, if we have a knowledge base containing rules that help making decisions about outdoor
activities for children, such as:
R1:
IF Today’s temperature < 22 THEN Play-Soccer
R2:
IF Today’s temperature < -10 THEN Build Snowman
As you can see, for a temperature of -12 degrees, the knowledge-base system will suggest playing
soccer, which is not correct for this situation. While there are several ways to amend this problem, the
simplest one is to reorder the rules; here we must use some criterion that allows the learning system to
perform the action (evaluation then ordering) automatically. We can provide the learning system with
semantic ability to relate each class with particular valid condition, so that the learning system can
distinguish irrelevant conditions, and order the rules with the proper context.
The other way is to assign a weight to a rule which relates the strength of the conditions to the
conclusion of the rule. For the example above, the condition “Today’s temperature<22”, covers some
false positive cases from the second rule’s class. Now suppose that the rule R2 is complete and
consistent. The intended scheme of rule weighting would result in the weight of rule R2 being higher
than the weight of rule R1. As a result R2 will be placed in the first location. This yields better
prediction performance. In Section 3.1, we described the weighting technique for rule conditions. We
must admit the fact that there is no consensus on how best to measure the combined effect of the
conditions on a conclusion in a rule, because in real-world applications, the conditions are not
independent, and the relation between conditions themselves is not clear. However, we use a heuristic
for combining condition weights to produce a weight for the rule.
13

15. The following combination function [4,13] will be used to calculate the weight of a rule
Rule' weight= w1 + w2 (1− w ) + w3[1−[w + w2 (1− w )]]+ ...
s
1
1
1
This computes the impact of the tests in a rule’s antecedent, where wi is the weight of condition i. Since
the weights of conditions can be either positive or negative evidence (for or against) the hypothesis, we
need to split the equation above as follows:
0 if no supporting evidence

MSE = 
for positive conditions
 w1 + w 2 (1 − w1 ) + w 3 [1 − [ w1 + w 2 (1 − w1 )]] + ...
0 if no opposing vidence

MOE = 
for negative conditions
 w1 + w2 (1 − w1 ) + w3 [1 − [ w1 + w2 (1 − w1 )]] + ...
The weight of the rule is then MSE-MOE
where
MSE= Measure of supporting evidence,
MOE= Measure of opposing evidence,
wi = Weight for evidence i.
3.3 Improving an existing knowledge base
It is our hypothesis that tackling the most difficult part of the knowledge base is a good approach, and
we feel that other misdiagnoses could be solved as a by-product of the process. To test this hypothesis,
we maintain a list of the available rules in the knowledge base, against a set of refinement cases selected
independently from training data set as shown for example in Table 3 below:
CASE
RULE
R2
R3
+
WEIGHT
R1
-
+
REFINEMENT CASE
1+
*
2 3+
*
*
*
.
.
Table 3: Generating refinement locations
14

16. For a particular class in the given domain, each rule maintains its weight and a label (+ or -) with respect
to the class under investigation. Each case is labelled + if it belongs to the class of interest or otherwise. A ‘*’ means that the case is covered by the rule. All the rules satisfied for a case in the
refinement data set collectively produce a weight for that case. To determine this weight, we follow the
formulation given in [4] which defines evidence combination as a special case of the Dempster-Shafer
theory of evidence, that is:
n
F ( w1 ,..., w n ) = 1 −
∏(1− w
j
), n >= 1
(2)
j =1
Where w is weight of rule J.
Since the rules in our approach are ordered with respect to their weight, we reformulate equation (2) as
follows
∏ (1 − w )+ Max[w], n >= 1
n
F ( w) = F ( w1 ,..., w n ) = 1 −
j
j =1
Where Max[w] is the highest weight of a rule among all the satisfied rules for a particular case. This
combined weight F(w) is an indication of rule overlapping. The case is misdiagnosed by the system if a
satisfied rule with the highest weight belongs to a different class (i.e. has a different sign). We would
like to find the points that have strongest overlapping (i.e. interaction) of rules where misdiagnoses exist.
The algorithm in Figure 5 describes how to assign case weights based on the rules satisfied. While the
weight of the rules is used as an ordering criterion to order the rules in the knowledge base, for aiding
classification, it also forms a basis for enhancing the refinement process.
Refinement locations are meant to be places in the knowledge base where misclassification may occur.
We identify refinement locations by a negative weight of misclassified instances. The number of
different rules which satisfy a particular refinement case represents its potential, i.e. the degree of
overlap in the knowledge base, which must be resolved. Our method of analysis calculates the
performance of rules in the knowledge base with respect to the refinement data set. For each stored case,
all the satisfied rules are identified. It is common to find multiple rules satisfied for a given case. The
problem of misclassification appears when rules of incorrect conclusions are satisfied for the case.
Because of this, we separate these rules into two groups one group corresponds to the active hypothesis
(the class of the misdiagnosed case) for the refinement location, and the second group contains rules
concluding other classes. We would like to determine the point at which the refinement would produce
the optimal result with respect to increased performance of the knowledge base, and least refinement
effort with respect to number of refinement operators implemented. Finding such an optimal refinement
is not possible, but the case weight as calculated in Figure 5 is a useful heuristic indicating areas of high
interest for possible refinements. Figure 3 summarises the refinement process.
15

17. Evaluate KB
performance
Stored
refinement
cases
KB system
conclusion
conclusion
Verify KB
completeness
Incorrect knowledge
base diagnoses ?
No
Yes
Verify KB
redundancy
Summarize rules
performance on each
refinement case
Return
Instantiate heuristics to
locate the most critical part
of the rule overlapping
Perform refinement
operations and select the
best
Apply refinement
operator
Evaluate the
KB
Update the
best
refinement
Implement the
best refinement
Figure 3: Refinement process
16

18. Refine (erroneous, missed case)
{The target rule is a rule satisfying missed case and concluding the correct hypothesis for the missed
case}
1.
Specialise the erroneous rule
S Select best specialisation
S If there is a target rule, accept the best specialisation
2.
If there is no target rule:
{
•
Find closest rule to generalise, then
{Two successive scans of the rule are performed in order to generalise it }
(1) Generalise by deleting a condition from the rule, such that the deletion does not degrade
the performance of the rule, so as to cover the missed case
(2) Generalise by expanding numerical scope of a condition if it is continuous, so that the
missed case will be correctly covered.
•
If the weight of closest rule after generalisation > weight of the erroneous rule, then adopt the
generalisation operation only,
Else
{
•
Augment erroneous rule with UNLESS clause
•
Apply both generalisation (to closest rule) and specialisation (to erroneous
rule) operators
•
Evaluate the knowledge-base performance using each of the above
refinements, and accept the one that better enhances the knowledge-base
performance.
}
}
3.
If there is no rule to generalise, then
•
Create a new rule using a minimum covering algorithm
•
If the weight of the created rule > weight of erroneous rule, then add new rule to the
knowledge base;
Else
{
•
Specialise the erroneous rule, so it no longer covers the missed case, and
•
Add the newly created rule.
}
Figure 4: The Refine algorithm
17

19. Combined weight (missed case)
Fp←Weight of rules supporting the hypothesis h, and satisfying missed case.
Fn←Weight of rules that do not support the hypothesis h, and satisfy the missed case.
Maxwp←Maximum weight carried by a rule, among those supporting the hypothesis h.
Maxwn←Maximum weight carried by a rule, among those satisfying the missed case but do not conclude
hypothesis h.
Wi←Weight of rule(i) { Calculated using the approach discussed in Section 3.2 }
•
Calculate combined weight of rules satisfying the missed case and concluding h
k
Fp ← 1−
∏ (1 − w )+ Max
i
wp
i =1
•
Calculate combined weight of rules satisfying the missed case, and do not conclude h
∏ (1 − w )+ Max
R
Fn ← 1 −
j
wn
j =1
•
Compute the combined weight for the case, to indicate whether the case is misdiagnosed or not. If
the combined weight of rules in favour of the hypothesis under consideration, is greater than the
combined weight of the other rules, then the case will be diagnosed correctly; otherwise it will be
misdiagnosed.
Case weight = Fp − Fn
•
Return case weight.
Figure 5: The combined weight algorithm
Example:
Table 4, describes a small classification system, to demonstrate our approach. It contains two final
hypotheses (+) and (−). There are 16 training cases C1,C2,…,C16. The cases are marked as (+) if they
belong to the hypothesis (+), or (−) otherwise. The rules are shown with their weights. The marks
indicate the cases to which the rules R1 to R4 are applicable (i.e. the conditions of the rule are satisfied
for the case). The rules are assigned (+) sign if they conclude hypothesis (+) or (−) otherwise
18

20. Case
Combined
R3 –
R2+
R5 –
R6 +
R1 +
R4 +
case weight
0.40386
0.28568
0.225048
0.000778
-0.0003322
-0.256594
1+
0.6864
*
2+
0.6864
*
3+
0.6864
*
4+
0.1709
*
5+
0.1709
*
6+
7−
8+

25. *
*
*
*
-0.5148
*
-0.5148
*
-0.5148
*

26. *
*
*
*
*
*
9+
0.5155
10−
-1.3064
*
11−
-1.3064
*
12−
-1.3064
*
13−
-1.3048
*
14+
0.2582
*
15+
0.2582
*
16−
0.2582
*

30. *
*
*
*

39. *
*