This document summarizes Bigyan Bhar's seminar on using support vector machines (SVMs) for semi-supervised classification. It introduces classification and SVMs, describes how unlabeled data can be used for semi-supervised learning, and compares methods like transductive SVM and augmented Lagrangian techniques. Results show some semi-supervised methods like penalty methods are more robust than TSVM in certain cases. Future work could establish theoretical bounds for method accuracy and explore non-SVM semi-supervised classifiers.

1. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
SVM based Semi-Supervised Classication
Topics in Pattern Recognition
Bigyan Bhar
M.E. CSA, IISc
4710-410-091-07064
Oct 11th, 2010
Bigyan Bhar Seminar, Topics in PR

2. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Outline
1 Classication
2 Support Vector Machine (SVM)
3 Using SVM for Semi-Supervised Classication
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
4 Results
5 Conclusion
New Facts
Further Directions
Acknowledgments
References
Bigyan Bhar Seminar, Topics in PR

3. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Outline
1 Classication
2 Support Vector Machine (SVM)
3 Using SVM for Semi-Supervised Classication
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
4 Results
5 Conclusion
New Facts
Further Directions
Acknowledgments
References
Bigyan Bhar Seminar, Topics in PR

4. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
What is a Classication?
Classication refers to an algorithmic procedure for assigning a
given piece of input data into one of a given number of
categories
Class test Final Exam Project Seminar
13 35 16 18
10 31 5 19
11 21 9 11
12 29 10 15
Grade
A
B
C
B
Bigyan Bhar Seminar, Topics in PR

5. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Traditional Classier
Classifier
Builder
Labelled Data Classifier
Unlabelled Data Classifier Label for Data
Bigyan Bhar Seminar, Topics in PR

6. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Classier
Classier is supposed to classify unlabeled data
We have a lot of unlabeled data; typically much more than
number of labeled data
So far we have seen classiers being built using only labeled
data
What if we could also use the large set of unclassied data to
build a better classier?
Bigyan Bhar Seminar, Topics in PR

7. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Semi-supervised Classier
Classifier
Builder
Unlabelled
Classifier
Builder
Labelled Data Classifier
Labelled Data Classifier
Semi−supervised
Bigyan Bhar Seminar, Topics in PR

8. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
How to use the unlabeled data?
The separating plane has to pass through a low density region
Bigyan Bhar Seminar, Topics in PR

9. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
How to use the unlabeled data?
The separating plane has to pass through a low density region
Bigyan Bhar Seminar, Topics in PR

10. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Labeling Constraint
The low density region principle that we observed can be
realized using a fractional constraint
# of positive class examples
total # of of examples
= r
r is an user supplied input
We enforce the above constraint on unlabeled examples as
they are large in number
Bigyan Bhar Seminar, Topics in PR

11. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Outline
1 Classication
2 Support Vector Machine (SVM)
3 Using SVM for Semi-Supervised Classication
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
4 Results
5 Conclusion
New Facts
Further Directions
Acknowledgments
References
Bigyan Bhar Seminar, Topics in PR

12. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
What is SVM?
SVM = Support Vector Machine
Maximal Margin Classier
Bigyan Bhar Seminar, Topics in PR

13. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
SVM Continued
w
w
’x+b=1
w
’x+b=0
w
’x+b=−1
m
argin
Total margin= 1
w + 1
w = 2
w
Optimization problem
min
w
1
2
wTw
subject to,
yi wTxi +b ≥ 1 ∀1 ≤ i ≤ l
Bigyan Bhar Seminar, Topics in PR

14. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
SVM Formulation
Using KKT conditions, we get the nal SVM problem as:
w∗
= argmin
w
1
2
l
∑
i=1
loss yiwTxi +
λ
2
wTw
Bigyan Bhar Seminar, Topics in PR

15. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
Outline
1 Classication
2 Support Vector Machine (SVM)
3 Using SVM for Semi-Supervised Classication
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
4 Results
5 Conclusion
New Facts
Further Directions
Acknowledgments
References
Bigyan Bhar Seminar, Topics in PR

16. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
Transductive SVM (TSVM)
min
w,{yj }
u
j=1
λ
2
w 2
+
1
2l
l
∑
i=1
loss yiwTxi +
λ
2u
u
∑
j=1
loss yjwTxi
subject to:
1
u
u
∑
j=1
max 0,sign wTxj = r
Bigyan Bhar Seminar, Topics in PR

17. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
Modifying TSVM
What is the cost to importance ratio of the terms in TSVM
formulation?
min
w,{yj }
u
j=1
λ
2
w 2
+
1
2l
l
∑
i=1
loss yiwTxi +
λ
2u
u
∑
j=1
loss yjwTxi
Clearly the third term, unlabeled loss is the costliest
computation of yi for the large set of unlabeled terms
What if we can avoid it altogether?
Bigyan Bhar Seminar, Topics in PR

18. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
Modied TSVM
TSVM formulation:
min
w,{yj }
u
j=1
λ
2
w 2
+
1
2l
l
∑
i=1
loss yiwTxi +
λ
2u
u
∑
j=1
loss yjwTxi
Our formulation:
min
w
λ
2
w 2
+
1
2l
l
∑
i=1
loss yiwTxi
subject to:
1
u
u
∑
j=1
max 0,sign wTxj = r
Bigyan Bhar Seminar, Topics in PR

19. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
Augmented Lagrangian Technique
Augmented Lagrangian is a technique for solving minimization
problems with equality constraints
It converges faster than the generalized methods
Original problem: min f(x), subject to g(x) = 0
Can be written as an unconstrained minimization over:
L(x,λ,µ) = f(x)−λg(x)+
1
2µ
g(x) 2
Since f and the Lagrangian (for any λ) agree on the feasible
set g(x) = 0, the basic idea remains same as that of
Lagrangian
a small value of µ forces the minimizer(s) of L to lie close to
the feasible set
values of x that that reduce f are preferred
Bigyan Bhar Seminar, Topics in PR

20. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
Modied TSVM using Augmented Lagrangian
Our formulation:
min
w
[f(w)] =⇒ min
w
λ
2
w 2
+
1
2l
l
∑
i=1
loss yiwTxi
subject to:
g(w) = 0 =⇒
1
u
u
∑
j=1
max 0,sign wTxj −r = 0
Augmented Lagrangian:
min
x
[L(x,λ,µ)] = min
x
f(x)−λg(x)+
1
2µ
g(x) 2
Bigyan Bhar Seminar, Topics in PR

21. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
Penalty Method
Augmented Lagrangian:
min
x
[L(x,λ,µ)] = min
x
f(x)−λg(x)+
1
2µ
g(x) 2
Penalty Method:
min
x
f(x)+
1
2µ
g(x) 2
Bigyan Bhar Seminar, Topics in PR

22. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
SVM based
Supervised SVM (SSVM)
w∗
= arg min
w∈Rd
λ
2
w 2
+
1
2
l
∑
i=1
loss yiwTxi
SSVM with Threshold Adjustment
Obtain w∗ from SSVM
Adjust threshold to satisfy la belling constraint
1
u
u
∑
j=1
max 0,sign wTxj = r
Bigyan Bhar Seminar, Topics in PR

23. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
All Methods at a Glance
SVM based:
SSVM on labeled data
SSVM on labeled data with threshold adjustment
Methods proposed in this work:
Augmented Lagrangian
Penalty Method
TSVM
Deterministic Annealing
Switching
Bigyan Bhar Seminar, Topics in PR

24. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Outline
1 Classication
2 Support Vector Machine (SVM)
3 Using SVM for Semi-Supervised Classication
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
4 Results
5 Conclusion
New Facts
Further Directions
Acknowledgments
References
Bigyan Bhar Seminar, Topics in PR

25. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Accuracy Vs # of Labled Example (gcat)
Bigyan Bhar Seminar, Topics in PR

26. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Accuracy Vs # of Labled Example (aut-avn)
Bigyan Bhar Seminar, Topics in PR

27. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Accuracy Vs # of Noise in r (gcat)
Bigyan Bhar Seminar, Topics in PR

28. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
Accuracy Vs # of Noise in r (aut-avn)
Bigyan Bhar Seminar, Topics in PR

29. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
New Facts
Further Directions
Acknowledgments
References
Outline
1 Classication
2 Support Vector Machine (SVM)
3 Using SVM for Semi-Supervised Classication
Transductive SVM Modications
Augmented Lagrangian
Other Methods
All Methods
4 Results
5 Conclusion
New Facts
Further Directions
Acknowledgments
References
Bigyan Bhar Seminar, Topics in PR

30. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
New Facts
Further Directions
Acknowledgments
References
Some Results
Simple penalty method is the most robust method WRT
estimation of r
TSVM still leads in terms of accuracy
Augmented Lagrangian is a direction worth investigating due
to its faster computational time
Defeating the SSVM is possible only for reasonably accurate
estimation of r
If labeled dataset does not follow r, then alternate methods
perform better
Bigyan Bhar Seminar, Topics in PR

31. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
New Facts
Further Directions
Acknowledgments
References
Future Directions
Establish theoretical bounds for accuracy of our methods WRT
that of TSVM
Look at non-SVM based semi-supervised classiers (e.g.
decision tree) and come up with a way to express the
fractional constraint
Can we use something other than the fractional constraint to
enforce the low density criterion?
Bigyan Bhar Seminar, Topics in PR

32. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
New Facts
Further Directions
Acknowledgments
References
Acknowledgments
I thank the following persons for their able guidance and help in
this work:
S S Keerthi (Yahoo! Labs)
M N Murthy (IISc)
S Sundararajan (Yahoo! Labs)
S Shevade (IISc)
Bigyan Bhar Seminar, Topics in PR

33. Classication
Support Vector Machine (SVM)
Using SVM for Semi-Supervised Classication
Results
Conclusion
New Facts
Further Directions
Acknowledgments
References
References
MS Gockenbach. The augmented Lagrangian method for
equality-constrained optimizations
V Sindhwani, SS Keerthi. Newton Methods for Fast Solution
of Semi-supervised Linear SVMs
SS Keerthi, D DeCoste. A modied nite Newton method for
fast solution of large scale linear SVMs
Bigyan Bhar Seminar, Topics in PR