1. Symbolic Rules
Extraction From Trained
Neural Networks
Koushal Kumar
M .Tech CSE
Mob: +918968939621
26/5/2012 Koushal Kumar 1

2. What are Artificial Neural Networks?
• Artificial Neural Networks are powerful
computational systems consisting of many
simple processing elements connected together
to perform tasks analogously to biological
brains.
• They are massively parallel, which makes them
efficient, robust, fault tolerant and noise tolerant
• They can learn from training data and generalize
to new situations.
26/5/2012 Koushal Kumar 2

3. The limitation of Neural Network
The major criticism against Neural Network is that decision
given by neural networks is difficult to understand by a
human being. Reasons for this are
Knowledge in Neural Networks are stored as real values
parameters (weights and bias) of networks
Neural Networks are unable to explain its internal
processing how they come to particular decision
This behavior makes Neural Networks Black Box in
Nature
26/5/2012 Koushal Kumar 3

4. Continue..
26/5/2012 Koushal Kumar 4

5. Rules extraction From Neural
Networks.
So to overcome the Black Box nature of Neural
Networks we need to extract rules from Neural
Networks so that the user can gain a better
understanding of the decision process. following types
of rules can be extracted from neural networks
I) M OF N types rules
II) Fuzzy rules
26/5/2012 Koushal Kumar 5

6. continue..
III) IF THEN RULES
IV) Decision Rules
V) First order logic rules
From all above types of rules IF THEN RULES
and Decision rules are easy to understand then
others kind of rules.
26/5/2012 Koushal Kumar 6

7. J48 Algorithm for extracting
decision trees
• J48 is an algorithm used to generate a decision tree.
• Developed by quinlan and most widely used decision
tree induction algorithm.
• It is based upon greedy search approach i.e select the
best attribute and never looks back to reconsider early
choices.
• It select the best attribute according to its entropy
value.
• More preference will be given to that attribute which
has more value of entropy.
26/5/2012 Koushal Kumar 7

8. Before normalization
26/5/2012 Koushal Kumar 8

9. After normalization
26/5/2012 Koushal Kumar 9

10. MATLAB Simulator
• Matlab stands for matrix laboratory.
• It integrate computation, visualization, and
programming in an easy-to-use environment.
• MATLAB is a package that has been purpose-
designed to make computations easy, fast and
reliable.
• Matlab can be used in math and computation,
algorithm development, simulation purposes.
• MATLAB is a powerful system that can plot graphs
and perform a large variety of calculations with
numbers.
26/5/2012 Koushal Kumar 10

11. Weka simulator
• WEKA is abbreviation of Waikato Environment for
Knowledge Analysis.
• Weka is open source simulator with machine learning
algorithms.
• The Weka workbench contains a collection of
visualization tools and algorithms for data analysis
and predictive modelling.
26/5/2012 Koushal Kumar 11

12. Training data set and its Histogram
26/5/2012 Koushal Kumar 12

13. Histogram of input target values
26/5/2012 Koushal Kumar 13

14. Histogram of testing set as Inputs
26/5/2012 Koushal Kumar 14

15. Histogram of testing set as Target values
26/5/2012 Koushal Kumar 15

16. Neural networks Training with BPA
26/5/2012 Koushal Kumar 16

17. Best Training Performance curve
26/5/2012 Koushal Kumar 17

18. Gradient Performance Curve After Training &
Testing
26/5/2012 Koushal Kumar 18

19. Data import window in weka
26/5/2012 Koushal Kumar 19

20. J48 algorithm Output (pruned)
26/5/2012 Koushal Kumar 20

21. Results from pruned Tree
26/5/2012 Koushal Kumar 21

22. J48 algorithm Output (Unpruned)
26/5/2012 Koushal Kumar 22

23. Unpruned Tree Results
26/5/2012 Koushal Kumar 23

24. Results of classification from Decision Tree
26/5/2012 Koushal Kumar 24

25. Decision Tree Extracted From Exp no 7
26/5/2012 Koushal Kumar 25

26. The Following Rule Set Is Obtained From the above Decision
Tree
I. Applying Remove Redundancy Conditions
IF Children ≥ 1 AND Children >2AND Children >3 THEN Marital status =YES
Children ≥ 1 is more specific than Children >3 and Children > 2. So we remove all such
conditions
Rule 1:
a) IF Current_act = NO AND Age ≤ 48.0 AND Sex = FEMALE AND Children ≤ 0 THEN Region
Town
b) IF AGE > 48.0 AND Region Suburban AND Current_act = NO then Pep = NO
II. For every pair decision trees Remove redundancy rules. For example
Rule 1: IF Age ≤60 AND Salary ≤ 3500 AND Pep = NO THEN Mortage = YES
Rule 2: IF Age ≤ 50 AND Salary ≤ 3500 AND Pep = NO THEN Mortage = YES
New Rule: IF Age ≤ 50 AND Salary ≤ 3500 AND Pep = NO THEN Mortage =YES
III. Remove more specific rules. The rules with a condition set which is a superset of
another rule should be removed. For example
26/5/2012 Koushal Kumar 26

27. Continue….
• Rule 1: IF Age ≤ 60 AND Region = Rural AND Saving_ act =
YES THEN Pep = NO
• Rule 2: IF Age <= 60 AND Children <= 1 AND Region
=Rural AND saving act = YES THEN Pep= NO
• Rule 3: IF Region = Rural AND saving _ act =YES THEN
Pep = NO
• New Rule: IF Region = Rural AND saving _ act =YES THEN
Pep = NO
26/5/2012 Koushal Kumar 27

28. Comparison of J48 algorithm with Others Classifiers
26/5/2012 Koushal Kumar 28

29. Graphical comparisons of algorithms
26/5/2012 Koushal Kumar 29

30. Data Flow diagram of J48 algorithm
26/5/2012 Koushal Kumar 30

31. Paper Published
• “Extracting Explanation From Artificial Neural Networks” is
published in International Journal of Computer Science and
Information Technologies.
• “Advanced Applications of Neural Networks and Artificial
Intelligence: A Review” has been selected in International journal of
information technologies and computer science and it will published
on May June volume of journal.
• Seminar: Published Research paper on “Symbolic Rules Extraction
From Trained Neural Networks” in two days UGC Sponsored
National Seminar on Social Implications of Artificial Intelligence
organized in KMV College in Jalandhar.
26/5/2012 Koushal Kumar 31

32. References
• Olcay Boz AT & T Labs”Converting a trained neural network
to a decision tree DecText-Decision Tree Extractor.
• J.T Yao Dept of computer sci university of Regina “Knowledge
extracted from Trained neural networks whats next?
• Simon Haykin ”Neural networks a Comprehensive foundation”
Pearson education(second edition).
• R. Davis, B.G. Buchanan, and E. Shortcliff, “Production Rules
as a
• Representation for a Knowledge Based Consultation Progra”,
• Artificial Intelligence, 1977, vol. 8(1), pp.15-45.
• Knowledge Extraction from the Neural ‘Black Box’ in
Ecological Monitoring Journal of Industrial and Systems
Engineering Vol. 3, No. 1, pp 38-55 Spring 2009
26/5/2012 Koushal Kumar 32