The document presents a method for actively evaluating classifiers on large datasets. It aims to estimate the true accuracy of a classifier on an unlabeled dataset given a small labeled subset. It proposes using a stratified estimate, where the labeled and unlabeled data are stratified into buckets based on a learned hash function of instance features. Weights for each bucket are estimated based on the fraction of unlabeled instances in that bucket. This allows accuracy estimation to scale to large unlabeled datasets.

1. Problem setup Accuracy estimation Results Summary References
Active Evaluation of Classi

2. ers on Large
Datasets
Namit Katariya, Arun Iyer, Sunita Sarawagi
IIT Bombay
December 13, 2012
International Conference on Data Mining, 2012
Active Evaluation of Classi

3. ers 1/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

4. Problem setup Accuracy estimation Results Summary References
Problem setup
Setup
A classi

5. er C(x) deployed on
A large unlabeled dataset D
Estimate true accuracy of C(x) on D
Given
A labeled set L : small or unrepresentative of D
Measured accuracy on labeled set6= True accuracy on data
A human labeler
Compelling problem in many real-life applications
Active Evaluation of Classi

6. ers 2/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

7. Problem setup Accuracy estimation Results Summary References
Problem setup
Setup
A classi

8. er C(x) deployed on
A large unlabeled dataset D
Estimate true accuracy of C(x) on D
Given
A labeled set L : small or unrepresentative of D
Measured accuracy on labeled set6= True accuracy on data
A human labeler
Compelling problem in many real-life applications
Active Evaluation of Classi

9. ers 2/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

10. Problem setup Accuracy estimation Results Summary References
Problem setup
Setup
A classi

11. er C(x) deployed on
A large unlabeled dataset D
Estimate true accuracy of C(x) on D
Given
A labeled set L : small or unrepresentative of D
Measured accuracy on labeled set6= True accuracy on data
A human labeler
Compelling problem in many real-life applications
Active Evaluation of Classi

12. ers 2/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

13. Problem setup Accuracy estimation Results Summary References
Problem setup
Setup
A classi

14. er C(x) deployed on
A large unlabeled dataset D
Estimate true accuracy of C(x) on D
Given
A labeled set L : small or unrepresentative of D
Measured accuracy on labeled set6= True accuracy on data
A human labeler
Compelling problem in many real-life applications
Active Evaluation of Classi

15. ers 2/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

16. Problem setup Accuracy estimation Results Summary References
Our goal
1 Handle arbitrary classi

17. er e.g. a user script
existing work assumes that C(x) is probabilistic and can output
well-calibrated Pr(yjx) values.
2 Provide interactive speed to user in loop even when D is very
large
even a single sequential scan on D may not be practical.
D can be accessed only via an index.
3 Require user to label as few additional instances as possible
Similar to active learning but dierent ...
Active learning usually used in the context of learning classi

18. ers
Task of learning classi

19. ers dierent from evaluating given classi

20. er
Active Evaluation of Classi

21. ers 3/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

22. Problem setup Accuracy estimation Results Summary References
Our goal
1 Handle arbitrary classi

23. er e.g. a user script
existing work assumes that C(x) is probabilistic and can output
well-calibrated Pr(yjx) values.
2 Provide interactive speed to user in loop even when D is very
large
even a single sequential scan on D may not be practical.
D can be accessed only via an index.
3 Require user to label as few additional instances as possible
Similar to active learning but dierent ...
Active learning usually used in the context of learning classi

24. ers
Task of learning classi

25. ers dierent from evaluating given classi

26. er
Active Evaluation of Classi

27. ers 3/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

28. Problem setup Accuracy estimation Results Summary References
Our goal
1 Handle arbitrary classi

29. er e.g. a user script
existing work assumes that C(x) is probabilistic and can output
well-calibrated Pr(yjx) values.
2 Provide interactive speed to user in loop even when D is very
large
even a single sequential scan on D may not be practical.
D can be accessed only via an index.
3 Require user to label as few additional instances as possible
Similar to active learning but dierent ...
Active learning usually used in the context of learning classi

30. ers
Task of learning classi

31. ers dierent from evaluating given classi

32. er
Active Evaluation of Classi

33. ers 3/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

34. Problem setup Accuracy estimation Results Summary References
Our goal
1 Handle arbitrary classi

35. er e.g. a user script
existing work assumes that C(x) is probabilistic and can output
well-calibrated Pr(yjx) values.
2 Provide interactive speed to user in loop even when D is very
large
even a single sequential scan on D may not be practical.
D can be accessed only via an index.
3 Require user to label as few additional instances as possible
Similar to active learning but dierent ...
Active learning usually used in the context of learning classi

36. ers
Task of learning classi

37. ers dierent from evaluating given classi

38. er
Active Evaluation of Classi

39. ers 3/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

40. Problem setup Accuracy estimation Results Summary References
Outline
The problem has two aspects
1 Accuracy estimation (What this talk is about)
Given a

41. xed L what is the best estimator of ?
How to do this scalably on large D?
2 Instance selection (Not covered in this talk, details in paper)
Selecting instances from D to be labeled by a human and adding
to L. Performed in a loop.
Active Evaluation of Classi

42. ers 4/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

43. Problem setup Accuracy estimation Results Summary References
Accuracy estimation
Simple averaged estimate : ^R = 1n
P
i2L ai is poor when L is
small or biased
A classical solution : strati

44. ed estimate
Stratify L and D into B buckets (L1;D1); : : : ; (LB;DB)
Measure accuracy ^b of Lb in each bucket b
Estimate weight wb as fraction of D instances in bucket b= jDbj
jDj
Strati

45. ed estimate ^S =
P
b wb ^b
Error of ^S Error of ^R if instances within a bucket are
homogeneous
Two challenges
1 Selecting a strati

46. cation strategy that puts instances with same
error in the same bucket
2 Finding wb scalably when D is large ) cannot stratify whole of D.
Active Evaluation of Classi

47. ers 5/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

48. Problem setup Accuracy estimation Results Summary References
Accuracy estimation
Simple averaged estimate : ^R = 1n
P
i2L ai is poor when L is
small or biased
A classical solution : strati

49. ed estimate
Stratify L and D into B buckets (L1;D1); : : : ; (LB;DB)
Measure accuracy ^b of Lb in each bucket b
Estimate weight wb as fraction of D instances in bucket b= jDbj
jDj
Strati

50. ed estimate ^S =
P
b wb ^b
Error of ^S Error of ^R if instances within a bucket are
homogeneous
Two challenges
1 Selecting a strati

51. cation strategy that puts instances with same
error in the same bucket
2 Finding wb scalably when D is large ) cannot stratify whole of D.
Active Evaluation of Classi

52. ers 5/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

53. Problem setup Accuracy estimation Results Summary References
Accuracy estimation
Simple averaged estimate : ^R = 1n
P
i2L ai is poor when L is
small or biased
A classical solution : strati

54. ed estimate
Stratify L and D into B buckets (L1;D1); : : : ; (LB;DB)
Measure accuracy ^b of Lb in each bucket b
Estimate weight wb as fraction of D instances in bucket b= jDbj
jDj
Strati

55. ed estimate ^S =
P
b wb ^b
Error of ^S Error of ^R if instances within a bucket are
homogeneous
Two challenges
1 Selecting a strati

56. cation strategy that puts instances with same
error in the same bucket
2 Finding wb scalably when D is large ) cannot stratify whole of D.
Active Evaluation of Classi

57. ers 5/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

58. Problem setup Accuracy estimation Results Summary References
Accuracy estimation
Simple averaged estimate : ^R = 1n
P
i2L ai is poor when L is
small or biased
A classical solution : strati

59. ed estimate
Stratify L and D into B buckets (L1;D1); : : : ; (LB;DB)
Measure accuracy ^b of Lb in each bucket b
Estimate weight wb as fraction of D instances in bucket b= jDbj
jDj
Strati

60. ed estimate ^S =
P
b wb ^b
Error of ^S Error of ^R if instances within a bucket are
homogeneous
Two challenges
1 Selecting a strati

61. cation strategy that puts instances with same
error in the same bucket
2 Finding wb scalably when D is large ) cannot stratify whole of D.
Active Evaluation of Classi

62. ers 5/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

63. Problem setup Accuracy estimation Results Summary References
Accuracy estimation
Simple averaged estimate : ^R = 1n
P
i2L ai is poor when L is
small or biased
A classical solution : strati

64. ed estimate
Stratify L and D into B buckets (L1;D1); : : : ; (LB;DB)
Measure accuracy ^b of Lb in each bucket b
Estimate weight wb as fraction of D instances in bucket b= jDbj
jDj
Strati

65. ed estimate ^S =
P
b wb ^b
Error of ^S Error of ^R if instances within a bucket are
homogeneous
Two challenges
1 Selecting a strati

66. cation strategy that puts instances with same
error in the same bucket
2 Finding wb scalably when D is large ) cannot stratify whole of D.
Active Evaluation of Classi

67. ers 5/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

68. Problem setup Accuracy estimation Results Summary References
Strati

69. cation strategy (h)
Existing approach : Bin Pr(yjx) values assumed to be provided
by a classi

70. er
Bennett and Carvalho Druck and McCallum
Not applicable since we wish to handle arbitrary classi

71. ers.
Our Proposal :
F(x; C) : A feature representation of the input x and the result of
deploying C on x
Learn a hash function h on features using the labeled data L
Strati

72. cation evolves as more labeled instances get added to L
(

73. xed in existing approaches)
Active Evaluation of Classi

74. ers 6/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

75. Problem setup Accuracy estimation Results Summary References
Strati

76. cation strategy (h)
Existing approach : Bin Pr(yjx) values assumed to be provided
by a classi

77. er
Bennett and Carvalho Druck and McCallum
Not applicable since we wish to handle arbitrary classi

78. ers.
Our Proposal :
F(x; C) : A feature representation of the input x and the result of
deploying C on x
Learn a hash function h on features using the labeled data L
Strati

79. cation evolves as more labeled instances get added to L
(

80. xed in existing approaches)
Active Evaluation of Classi

81. ers 6/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

82. Problem setup Accuracy estimation Results Summary References
Strati

83. cation strategy (h)
Existing approach : Bin Pr(yjx) values assumed to be provided
by a classi

84. er
Bennett and Carvalho Druck and McCallum
Not applicable since we wish to handle arbitrary classi

85. ers.
Our Proposal :
F(x; C) : A feature representation of the input x and the result of
deploying C on x
Learn a hash function h on features using the labeled data L
Strati

86. cation evolves as more labeled instances get added to L
(

87. xed in existing approaches)
Active Evaluation of Classi

88. ers 6/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

89. Problem setup Accuracy estimation Results Summary References
Strati

90. cation strategy (h)
Existing approach : Bin Pr(yjx) values assumed to be provided
by a classi

91. er
Bennett and Carvalho Druck and McCallum
Not applicable since we wish to handle arbitrary classi

92. ers.
Our Proposal :
F(x; C) : A feature representation of the input x and the result of
deploying C on x
Learn a hash function h on features using the labeled data L
Strati

93. cation evolves as more labeled instances get added to L
(

94. xed in existing approaches)
Active Evaluation of Classi

95. ers 6/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

96. Problem setup Accuracy estimation Results Summary References
Learning hyperplanes
Background
Hash function : concatenation of r bits
bit k = sign(wk :F(x)) i.e. the side of wk on which x lies
wk : parameters to be learnt
Learning problem : non-smooth and non-convex
Existing work : learning distance-preserving hash function
Learn wk parameters sequentially smooth the sign function
through various tricks
Our problem can be expressed as special case of learning
distance-preserving hash function
Exploit 0/1 nature of accuracy values rather than optimizing a
black-box distance measure
Allows a more ecient algorithm
Active Evaluation of Classi

97. ers 7/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

98. Problem setup Accuracy estimation Results Summary References
Learning hyperplanes
Background
Hash function : concatenation of r bits
bit k = sign(wk :F(x)) i.e. the side of wk on which x lies
wk : parameters to be learnt
Learning problem : non-smooth and non-convex
Existing work : learning distance-preserving hash function
Learn wk parameters sequentially smooth the sign function
through various tricks
Our problem can be expressed as special case of learning
distance-preserving hash function
Exploit 0/1 nature of accuracy values rather than optimizing a
black-box distance measure
Allows a more ecient algorithm
Active Evaluation of Classi

99. ers 7/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

100. Problem setup Accuracy estimation Results Summary References
Learning hyperplanes
Background
Hash function : concatenation of r bits
bit k = sign(wk :F(x)) i.e. the side of wk on which x lies
wk : parameters to be learnt
Learning problem : non-smooth and non-convex
Existing work : learning distance-preserving hash function
Learn wk parameters sequentially smooth the sign function
through various tricks
Our problem can be expressed as special case of learning
distance-preserving hash function
Exploit 0/1 nature of accuracy values rather than optimizing a
black-box distance measure
Allows a more ecient algorithm
Active Evaluation of Classi

101. ers 7/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

102. Problem setup Accuracy estimation Results Summary References
Learning hyperplanes
Background
Hash function : concatenation of r bits
bit k = sign(wk :F(x)) i.e. the side of wk on which x lies
wk : parameters to be learnt
Learning problem : non-smooth and non-convex
Existing work : learning distance-preserving hash function
Learn wk parameters sequentially smooth the sign function
through various tricks
Our problem can be expressed as special case of learning
distance-preserving hash function
Exploit 0/1 nature of accuracy values rather than optimizing a
black-box distance measure
Allows a more ecient algorithm
Active Evaluation of Classi

103. ers 7/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

104. Problem setup Accuracy estimation Results Summary References
Learning hyperplanes
Background
Hash function : concatenation of r bits
bit k = sign(wk :F(x)) i.e. the side of wk on which x lies
wk : parameters to be learnt
Learning problem : non-smooth and non-convex
Existing work : learning distance-preserving hash function
Learn wk parameters sequentially smooth the sign function
through various tricks
Our problem can be expressed as special case of learning
distance-preserving hash function
Exploit 0/1 nature of accuracy values rather than optimizing a
black-box distance measure
Allows a more ecient algorithm
Active Evaluation of Classi

105. ers 7/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

106. Problem setup Accuracy estimation Results Summary References
Learning hyperplanes
Our approach
Main step :

107. nd the best value of wk for a bit k, assuming
hyperplanes of other bits are

108. xed
For each bucket formed from remaining hyperplanes, arbitrarily
choose which side of hyperplane wk it wants to call +ve or -ve
Use logistic loss to

109. nd the optimal wk so that in each bucket,
wk correctly puts points on the right side
Can be solved optimally using a standard logistic classi

110. er
Use a EM-like iteration to re

111. ne the side chosen as positive in
each bucket until convergence
We see in our experiments that our method provides much better
results than existing algorithms
Active Evaluation of Classi

112. ers 8/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

113. Problem setup Accuracy estimation Results Summary References
Learning hyperplanes
Our approach
Main step :

114. nd the best value of wk for a bit k, assuming
hyperplanes of other bits are

115. xed
For each bucket formed from remaining hyperplanes, arbitrarily
choose which side of hyperplane wk it wants to call +ve or -ve
Use logistic loss to

116. nd the optimal wk so that in each bucket,
wk correctly puts points on the right side
Can be solved optimally using a standard logistic classi

117. er
Use a EM-like iteration to re

118. ne the side chosen as positive in
each bucket until convergence
We see in our experiments that our method provides much better
results than existing algorithms
Active Evaluation of Classi

119. ers 8/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

120. Problem setup Accuracy estimation Results Summary References
Learning hyperplanes
Our approach
Main step :

121. nd the best value of wk for a bit k, assuming
hyperplanes of other bits are

122. xed
For each bucket formed from remaining hyperplanes, arbitrarily
choose which side of hyperplane wk it wants to call +ve or -ve
Use logistic loss to

123. nd the optimal wk so that in each bucket,
wk correctly puts points on the right side
Can be solved optimally using a standard logistic classi

124. er
Use a EM-like iteration to re

125. ne the side chosen as positive in
each bucket until convergence
We see in our experiments that our method provides much better
results than existing algorithms
Active Evaluation of Classi

126. ers 8/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

127. Problem setup Accuracy estimation Results Summary References
Learning hyperplanes
Our approach
Main step :

128. nd the best value of wk for a bit k, assuming
hyperplanes of other bits are

129. xed
For each bucket formed from remaining hyperplanes, arbitrarily
choose which side of hyperplane wk it wants to call +ve or -ve
Use logistic loss to

130. nd the optimal wk so that in each bucket,
wk correctly puts points on the right side
Can be solved optimally using a standard logistic classi

131. er
Use a EM-like iteration to re

132. ne the side chosen as positive in
each bucket until convergence
We see in our experiments that our method provides much better
results than existing algorithms
Active Evaluation of Classi

133. ers 8/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

134. Problem setup Accuracy estimation Results Summary References
Learning hyperplanes
Our approach
Main step :

135. nd the best value of wk for a bit k, assuming
hyperplanes of other bits are

136. xed
For each bucket formed from remaining hyperplanes, arbitrarily
choose which side of hyperplane wk it wants to call +ve or -ve
Use logistic loss to

137. nd the optimal wk so that in each bucket,
wk correctly puts points on the right side
Can be solved optimally using a standard logistic classi

138. er
Use a EM-like iteration to re

139. ne the side chosen as positive in
each bucket until convergence
We see in our experiments that our method provides much better
results than existing algorithms
Active Evaluation of Classi

140. ers 8/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

141. Problem setup Accuracy estimation Results Summary References
Learning hyperplanes
Our approach
Main step :

142. nd the best value of wk for a bit k, assuming
hyperplanes of other bits are

143. xed
For each bucket formed from remaining hyperplanes, arbitrarily
choose which side of hyperplane wk it wants to call +ve or -ve
Use logistic loss to

144. nd the optimal wk so that in each bucket,
wk correctly puts points on the right side
Can be solved optimally using a standard logistic classi

145. er
Use a EM-like iteration to re

146. ne the side chosen as positive in
each bucket until convergence
We see in our experiments that our method provides much better
results than existing algorithms
Active Evaluation of Classi

147. ers 8/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

148. Problem setup Accuracy estimation Results Summary References
Estimating bucket weights without sequentially hashing D
^S =
P
b wb ^b = 1N
P
i2D ^h(fi )
Proposal sampling : ^Sq = 1N
1m
P
i2Q
^h(fi )
q(i)
Optimal q(i) / ^h(fi ) : impossible without assigning each i 2 D
to a bucket of h(:)
Allowed q(i ) are the ones which assign same probability to all
instances i within an index partition Du of D
Claim : Under above restriction, qu /
qP
b ^2
bp(bju) where
p(bju) is the fraction of i 2 Du with h(fi ) = b
p(bju) estimate : Initially depend on labeled data small static
sample. Re

149. ne estimate as more instances get sampled from Du
Active Evaluation of Classi

150. ers 9/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

151. Problem setup Accuracy estimation Results Summary References
Estimating bucket weights without sequentially hashing D
^S =
P
b wb ^b = 1N
P
i2D ^h(fi )
Proposal sampling : ^Sq = 1N
1m
P
i2Q
^h(fi )
q(i)
Optimal q(i) / ^h(fi ) : impossible without assigning each i 2 D
to a bucket of h(:)
Allowed q(i ) are the ones which assign same probability to all
instances i within an index partition Du of D
Claim : Under above restriction, qu /
qP
b ^2
bp(bju) where
p(bju) is the fraction of i 2 Du with h(fi ) = b
p(bju) estimate : Initially depend on labeled data small static
sample. Re

152. ne estimate as more instances get sampled from Du
Active Evaluation of Classi

153. ers 9/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

154. Problem setup Accuracy estimation Results Summary References
Estimating bucket weights without sequentially hashing D
^S =
P
b wb ^b = 1N
P
i2D ^h(fi )
Proposal sampling : ^Sq = 1N
1m
P
i2Q
^h(fi )
q(i)
Optimal q(i) / ^h(fi ) : impossible without assigning each i 2 D
to a bucket of h(:)
Allowed q(i ) are the ones which assign same probability to all
instances i within an index partition Du of D
Claim : Under above restriction, qu /
qP
b ^2
bp(bju) where
p(bju) is the fraction of i 2 Du with h(fi ) = b
p(bju) estimate : Initially depend on labeled data small static
sample. Re

155. ne estimate as more instances get sampled from Du
Active Evaluation of Classi

156. ers 9/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

157. Problem setup Accuracy estimation Results Summary References
Estimating bucket weights without sequentially hashing D
^S =
P
b wb ^b = 1N
P
i2D ^h(fi )
Proposal sampling : ^Sq = 1N
1m
P
i2Q
^h(fi )
q(i)
Optimal q(i) / ^h(fi ) : impossible without assigning each i 2 D
to a bucket of h(:)
Allowed q(i ) are the ones which assign same probability to all
instances i within an index partition Du of D
Claim : Under above restriction, qu /
qP
b ^2
bp(bju) where
p(bju) is the fraction of i 2 Du with h(fi ) = b
p(bju) estimate : Initially depend on labeled data small static
sample. Re

158. ne estimate as more instances get sampled from Du
Active Evaluation of Classi

159. ers 9/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

160. Problem setup Accuracy estimation Results Summary References
Estimating bucket weights without sequentially hashing D
^S =
P
b wb ^b = 1N
P
i2D ^h(fi )
Proposal sampling : ^Sq = 1N
1m
P
i2Q
^h(fi )
q(i)
Optimal q(i) / ^h(fi ) : impossible without assigning each i 2 D
to a bucket of h(:)
Allowed q(i ) are the ones which assign same probability to all
instances i within an index partition Du of D
Claim : Under above restriction, qu /
qP
b ^2
bp(bju) where
p(bju) is the fraction of i 2 Du with h(fi ) = b
p(bju) estimate : Initially depend on labeled data small static
sample. Re

161. ne estimate as more instances get sampled from Du
Active Evaluation of Classi

162. ers 9/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

163. Problem setup Accuracy estimation Results Summary References
Estimating bucket weights without sequentially hashing D
^S =
P
b wb ^b = 1N
P
i2D ^h(fi )
Proposal sampling : ^Sq = 1N
1m
P
i2Q
^h(fi )
q(i)
Optimal q(i) / ^h(fi ) : impossible without assigning each i 2 D
to a bucket of h(:)
Allowed q(i ) are the ones which assign same probability to all
instances i within an index partition Du of D
Claim : Under above restriction, qu /
qP
b ^2
bp(bju) where
p(bju) is the fraction of i 2 Du with h(fi ) = b
p(bju) estimate : Initially depend on labeled data small static
sample. Re

164. ne estimate as more instances get sampled from Du
Active Evaluation of Classi

165. ers 9/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

166. Problem setup Accuracy estimation Results Summary References
Estimating bucket weights without sequentially hashing D
^S =
P
b wb ^b = 1N
P
i2D ^h(fi )
Proposal sampling : ^Sq = 1N
1m
P
i2Q
^h(fi )
q(i)
Optimal q(i) / ^h(fi ) : impossible without assigning each i 2 D
to a bucket of h(:)
Allowed q(i ) are the ones which assign same probability to all
instances i within an index partition Du of D
Claim : Under above restriction, qu /
qP
b ^2
bp(bju) where
p(bju) is the fraction of i 2 Du with h(fi ) = b
p(bju) estimate : Initially depend on labeled data small static
sample. Re

167. ne estimate as more instances get sampled from Du
Active Evaluation of Classi

168. ers 9/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

169. Problem setup Accuracy estimation Results Summary References
Results
Summary of datasets used
TableAnnote : Annotate columns of Web tables to type nodes of
an ontology
Spam : Classifying web-pages as spam or not
DNA : Binary DNA classi

170. cation task
HomeGround, HomePredicted : Dataset of (entity, web-page)
instances and decide if web-page was a homepage for the entity
Dataset # Size Accuracy (%)
Features Seed(L) Unlabeled(D) Seed(L) True(D)
TableAnnote 42 541 11,954,983 56.4 16.5
Spam 1000 5000 350,000 86.4 93.2
DNA 800 100,000 50,000,000 72.2 77.9
HomeGround 66 514 1060 50.4 32.8
HomePredicted 66 8658 13,951,053 83.2 93.9
Active Evaluation of Classi

171. ers 10/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

172. Problem setup Accuracy estimation Results Summary References
Results
Comparison of estimation strategies on the TableAnnote dataset
Random Random Sampling
PropSample Sampling from proposal distribution (Sawade et al., 2010)
ScoreBins Strati

173. ed sampling with scores
Active Evaluation of Classi

174. ers 10/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

175. Problem setup Accuracy estimation Results Summary References
Results
Comparison of estimation strategies on remaining datasets
7%
6%
5%
4%
ScoreBins
Random
Slogistic
PropSample
Spam
3%
2%
1%
0%
7%
6%
5%
4%
3%
2%
1%
500 1000 1500 2000 2500
ScoreBins
Random
Slogistic
DNA
0%
PropSample
35000 35500 36000 36500 37000
30%
25%
20%
15%
ScoreBins
Random
Slogistic
PropSample
HomeGround
10%
5%
0%
12%
10%
500 1000 1500 2000 2500
8%
6%
ScoreBins
Random
Slogistic
PropSample
HomePredicted
4%
2%
0%
500 1500 2500 3500 4500
Figure: Absolute error (on the Y axis) of dierent estimation algorithms
against increasing number of labeled instances (on the X axis)
Active Evaluation of Classi

176. ers 10/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

177. Problem setup Accuracy estimation Results Summary References
Results
Comparison of dierent strati

178. cation methods on the TableAnnote dataset
TableAnnote
50%
45%
40%
35%
30%
25%
20%
Square error
NoStratify ScoreBins SeqProj_hash BRE_hash Slogistic_hash
15%
10%
5%
0%
2 5 7 2 5 7
250 541
Num bits
Train Size
NoStratify Simple averaging (no strati

179. cation)
ScoreBins Stratify using classi

180. er scores
SeqProj hash Learn hyperplanes using method in Wang et al.
BRE hash Learn hyperplanes using method in Kulis and Darrell
Active Evaluation of Classi

181. ers 10/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

182. Problem setup Accuracy estimation Results Summary References
Results
Comparison of dierent strati

183. cation methods on remaining datasets
Spam
9%
8%
7%
6%
5%
4%
Square error
NoStratify ScoreBins SeqProj_hash BRE_hash Slogistic_hash
3%
2%
1%
0%
2 5 7 2 5 7
1500 5000
Num bits
Train size
DNA
7%
6%
5%
4%
3%
Square error
NoStratify ScoreBins Slogistic_hash
2%
1%
0%
2 5 7 2 5 7 2 5 7
40000 50000 60000
Num bits
Train size
HomeGround
35%
30%
25%
20%
Square error
NoStratify ScoreBins SeqProj_hash BRE_hash Slogistic_hash
15%
10%
5%
0%
2 5 7 2 5 7
250 500
Num bits
Train size
HomePredicted
16%
14%
12%
10%
8%
6%
Square error
NoStratify ScoreBins SeqProj_hash BRE_hash Slogistic_hash
4%
2%
0%
2 5 7 2 5 7 2 5 7
1000 1500 4000
Num bits
Train size
Figure: Error of dierent strati

184. cation methods against increasing training
sizes and for dierent number of bits
Active Evaluation of Classi

185. ers 10/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

186. Problem setup Accuracy estimation Results Summary References
Results
Comparison of methods of sampling from indexed data for estimating bucket weights
1.2%
1.0%
0.8%
0.6%
0.4%
0.2%
0.0%
500
1500
2500
1000
2500
20000
1000
2500
20000
TableAnnote HomePredicted DNA
Data sample size
Difference with full scan
Our
Uniform
Figure: Comparing methods of sampling from indexed data for estimating
bucket weights
Active Evaluation of Classi

187. ers 10/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

188. Problem setup Accuracy estimation Results Summary References
Summary
1 Addressed the challenge of calibrating a classi

189. er's accuracy on
large unlabeled datasets given small amounts of labeled data and
a human labeler
2 Proposed a strati

190. ed sampling-based method for accuracy
estimation that provides better estimates than simple averaging
better selection of instances for labeling than random sampling
3 Between 15% and 62% relative reduction in error achieved
compared to existing approaches
4 Algorithm made scalable by proposing optimal sampling strategies
for accessing indexed unlabeled data directly
5 Close to optimal performance while reading three orders of
magnitude fewer instances on large datasets
Active Evaluation of Classi

191. ers 11/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

192. Problem setup Accuracy estimation Results Summary References
Summary
1 Addressed the challenge of calibrating a classi

193. er's accuracy on
large unlabeled datasets given small amounts of labeled data and
a human labeler
2 Proposed a strati

194. ed sampling-based method for accuracy
estimation that provides better estimates than simple averaging
better selection of instances for labeling than random sampling
3 Between 15% and 62% relative reduction in error achieved
compared to existing approaches
4 Algorithm made scalable by proposing optimal sampling strategies
for accessing indexed unlabeled data directly
5 Close to optimal performance while reading three orders of
magnitude fewer instances on large datasets
Active Evaluation of Classi

195. ers 11/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

196. Problem setup Accuracy estimation Results Summary References
Summary
1 Addressed the challenge of calibrating a classi

197. er's accuracy on
large unlabeled datasets given small amounts of labeled data and
a human labeler
2 Proposed a strati

198. ed sampling-based method for accuracy
estimation that provides better estimates than simple averaging
better selection of instances for labeling than random sampling
3 Between 15% and 62% relative reduction in error achieved
compared to existing approaches
4 Algorithm made scalable by proposing optimal sampling strategies
for accessing indexed unlabeled data directly
5 Close to optimal performance while reading three orders of
magnitude fewer instances on large datasets
Active Evaluation of Classi

199. ers 11/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

200. Problem setup Accuracy estimation Results Summary References
Summary
1 Addressed the challenge of calibrating a classi

201. er's accuracy on
large unlabeled datasets given small amounts of labeled data and
a human labeler
2 Proposed a strati

202. ed sampling-based method for accuracy
estimation that provides better estimates than simple averaging
better selection of instances for labeling than random sampling
3 Between 15% and 62% relative reduction in error achieved
compared to existing approaches
4 Algorithm made scalable by proposing optimal sampling strategies
for accessing indexed unlabeled data directly
5 Close to optimal performance while reading three orders of
magnitude fewer instances on large datasets
Active Evaluation of Classi

203. ers 11/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

204. Problem setup Accuracy estimation Results Summary References
Summary
1 Addressed the challenge of calibrating a classi

205. er's accuracy on
large unlabeled datasets given small amounts of labeled data and
a human labeler
2 Proposed a strati

206. ed sampling-based method for accuracy
estimation that provides better estimates than simple averaging
better selection of instances for labeling than random sampling
3 Between 15% and 62% relative reduction in error achieved
compared to existing approaches
4 Algorithm made scalable by proposing optimal sampling strategies
for accessing indexed unlabeled data directly
5 Close to optimal performance while reading three orders of
magnitude fewer instances on large datasets
Active Evaluation of Classi

207. ers 11/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

208. Problem setup Accuracy estimation Results Summary References
Thank You
Active Evaluation of Classi

209. ers 12/13 Namit Katariya, Arun Iyer, Sunita Sarawagi

210. Problem setup Accuracy estimation Results Summary References
References I
Bennett, P. N. and Carvalho, V. R. (2010). Online strati

211. ed sampling:
evaluating classi

212. ers at web-scale. In CIKM.
Druck, G. and McCallum, A. (2011). Toward interactive training and
evaluation. In CIKM.
Kulis, B. and Darrell, T. (2009). Learning to hash with binary
reconstructive embeddings. In NIPS.
Sawade, C., Landwehr, N., Bickel, S., and Scheer, T. (2010). Active
risk estimation. In ICML.
Wang, J., Kumar, S., and Chang, S. (2010). Sequential projection
learning for hashing with compact codes. In ICML.
Active Evaluation of Classi

213. ers 13/13 Namit Katariya, Arun Iyer, Sunita Sarawagi