Fuzzy Logic Seminar Knowledge extraction from support vector machines we introduce you to the SVM and why it is called SVM, then we demonstrate an algorithm that converts an SVM to an Artificial Neural Network, and how to obtain knowledge from it.

1. Knowledge Extraction from
Support Vector Machines:
A Fuzzy Logic Approach

2. “Certain class of SVMs is mathematically
equivalent to FARB”

3. What is SVM

4. What does it do?
Learns a hyper plane to classify data into
2 classes.

5. What is a hyperplane?
A hyperplane is a function like the equation for a line,
𝑦 = 𝑚𝑥 + 𝑏
In fact, for a simple classification task with just 2
features, the hyperplane can be a line.

6. SVM finds the optimal solution.

7. Support Vector Machine
SVM attempts to maximize
the margin, so that the
hyperplane is just as far away
from red ball as the blue ball.
In this way, it decreases the
chance of misclassification.

8. More Formally
Input:
set of (input, output) training pair samples.
Output:
set of weights w (or 𝑤𝑖), one for each feature, whose
linear combination predicts the value of y.

9. We use the optimization of maximizing the
margin (‘street width’) to reduce the number
of weights that are nonzero to just a few that
correspond to the important features that
‘matter’ in deciding the separating
line(hyperplane)…these nonzero weights
correspond to the support vectors (because
they ‘support’ the separating hyperplane)

10. The optimization problem
minimize 𝑓(𝑤) ≡ (1/2) ∥ 𝒘 ∥2
subject to 𝑔 𝑤, 𝑏 ≡ −𝑦𝑖 𝒘 ⋅ 𝒙 + 𝑏 + 1 ≤ 0, 𝑖 = 1 … 𝑚
we use Lagrange multipliers to get this
problem into a form that can be solved analytically

11. What if things get more complicated?

12. Throw the balls in the air. While the balls are
in the air and thrown up in just the right way,
you use a large sheet of paper to divide the
balls in the air.
mapping data to a high dimensional
space

13. Kernel
polynomial: (𝒙𝒊 ⋅ 𝒙𝒋 + 𝑐) 𝑝
Gaussian radial basis function: exp(−∥ 𝒙𝒊– 𝒙𝒋 ∥2
/2𝜎2
)
SVM does its thing, maps them into a higher dimension
and then finds the hyperplane to separate the classes.

15. Where does SVM get its name from?
• The decision function is fully specified by a (usually very small) subset
of training samples, the support vectors.
• Support vectors are the data points that lie closest to the decision
surface (or hyperplane)
• They are the data points most difficult to classify
• They have direct bearing on the optimum location of the decision
surface
• they ‘support’ the separating hyperplane

16. Knowledge Extraction
Extracting the knowledge learned by a black–box classifier and
representing it in a comprehensible form

17. Knowledge Extraction
Benefits :
• Validation
• Feature extraction
• Knowledge refinement and improvement
• Knowledge acquisition for symbolic AI systems
• Scientific discovery

18. Knowledge Extraction Rule Extraction
• Methods for RE from ANNs have been classified into three categories:
Decompositional
Pedagogical
Eclectic

19. decompositional approach for KE
SVM :The IO mapping of the trained SVM f : Rn → {−1, 1} is given by

20. decompositional approach for KE

23. “Certain class of SVMs is mathematically
equivalent to FARB”

24. What is FARB !?
Let’s take an example first !

25. Example:
Input: q ∈ R ,
Output: O ∈ R ,
And: a0, a1, k ∈ R, with k > 0.
Rules:
R1: If q is equal to k Then O = a0 + a1,
R2: If q is equal to −k Then O = a0 − a1,

26. - Linguistic terms: equal to k , equal to –k
- To express fuzziness, Gaussian membership function is used:

27. These Function Satisfy:

28. Applying Singleton fuzzifier and Centre of Gravity defuzzifier yields:
But, What does this Output mean !??

29. Take a deeper look !
It is a feedforward ANN with a single neuron, employing the activation
function tanh() !
So: this FRB is equivalent to ANN

30. This FRB , in particular, satisfy the definition of FARB, which is:

33. To get the same output, apply the same steps as in the example, which
is:
But how this output is any close to the one in the example ?!?

34. And many other MFs satisfy this output, given specific values of z,u,v,r
and g. Such as Logistic function and others.
Apply: zi = ui = 1,
vi = ri = 0,
and gi(x) = tanh(x).

35. Result
Kolman and Margaliot: Every standard ANN has a corresponding FARB.
There’s a transformation T:
This work extend that to: Certain class of SVMs satisfy the
transformation P:

36. “Certain class of SVMs is mathematically
equivalent to FARB”

37. The SVM-FARB Equivalence
*
1
( ) * ( , )
Nsv
i
i i
i
h x b y K x s

 
(2)
0
1 1
( ) ( )
m m
i i i i i i i i
i i
O q a ra z a g u q v
 
    
(8)

38. Theorem 2. (SVM-FARB equivalence)
condition
Find FARB with:
So these conditions would hold
0, , , ,i i i i
m Nsv
q a a   

*
0
1
*
,
,
( ) ( , )
m
i i
i
i i i i
i
i i i i
a ra b
z a y
g u q v K x s


 

 

(15)

39. Pause and Think
• Let’s say we have a FARB
• How many rules have we got?
1 1
0 1
...
...
m m
m
If q is and and q is
Then O a a a
  
   
(7)

40. Famous SVM Kernels
( , ) , (linear kernel)T
K x y x y
( , ) (1 / ) , , , (polynomial kernel)T d
K x y x y c c d   
( , ) tanh( ), 0, 0, (MLP kernel)T
K x y x y      
2 2
ˆ ˆ( , ) exp( / (2 )), , (RBF or Gaussian kernel)K x y x y     

41. Corollary 1 {MLP kernel}
These parameters will satisfy (15) conditions
( ) ( )
tanh(( ) )
( ) ( ) 2
k k
k k
q q
q k
q q
  
 
 
 

 

,
,
2 , /
i i
T i
i
i i
k k
i i
q x s
k
   
   
   

 
  
* *
1
( ) tanh( )
Nsv
T i
i i
i
h x y x s b  

   (17)
0
1 1
( ) ( )
m m
i i i i i i i i
i i
O q a ra z a g u q v
 
    
* *
0
1, 0,
2 ,
2 ,
( ) tanh( )
,
i i
i i
i i i
i
i i i
z r
u
v k
g x x
a b a y
 
 

 
 
  

 

42. Pause and Think
appear in the FARB if-part
What could this mean?
iq
cos
cos ; and are normalized
T i
i
i
i
i
i
q x s
q x s
q x s






43. Corollary 2 {MPL Kernel}
2
* *
2
1
( ) exp( )
ˆ2
Nsv
i i
i
x y
h x y b


   (18)
These parameters will
satisfy (15) conditions
0
1 1
( ) ( )
m m
i i i i i i i i
i i
O q a ra z a g u q v
 
    
2
2
( ) ( ) ( )
2exp( ) 1
( ) ( ) 2
k k
k k
q q q k
q q
 
  
 
 
 
  

0 0
,
,
ˆ, 0
T i
i
i i
q x s
k
   
 
   
 
 
 
2
2
* *
0 1
*
2, 1
1 2 ,
0,
( ) exp( ),
/ 2,
/ 2
i i
i
i
i
Nsv
i ii
i i i
z r
u
v
g x x
a b y
a y




  


 
 



44. Experimentation{Iris data set}
• 150 examples
• sepal length
• sepal width
• length
• petal width
• 3 classes
• Setosa
• Versicolor
• Virginica

45. SVM

46. Results {SVM1}

49. Results {SVM2}