Machine learning methods like neural networks and self-organizing maps can recognize CAPTCHAs with high accuracy using minimal preprocessing of images. Experiments showed neural networks outperformed self-organizing maps on CAPTCHAs with 4 or more letters. Accuracy decreased as the number of letters increased due to worse segmentation. Improving segmentation quality is key to maintaining high true positive rates. Future work involves better segmentation techniques and applying these methods to other problems like handwriting recognition.

1. Machine Learning Methods
for CAPTCHA Recognition
Rachel Shadoan
Zachery Tidwell, II
Constantine Priemski
Navya Chandana and Shakeeb

2. CAPTCHA
Completely Automated Public Turing Test to tell Computers and Humans Apart
Why are they interesting?
o Harder than normal text recognition
On par with handwriting recognition,
reading damaged text
o Techniques translate well to other problems
Facial recognition (Gonzaga, 2002)
Weed identification (Yang, 2000)
o Near infinite data sets
Easier to avoid over-fitting

3. Hypothesis
CAPTCHA recognition can be
accomplished to a high degree
of accuracy using machine
learning methods with minimal
preprocessing of inputs.

4. Methods
Learning Methods
o Feed-forward Neural
Nets
o Self-Organizing Maps
o K-Means
o Cluster Classification
Segmentation Methods
o Overlapping
o Whitespace
o K-Means
Tools
o JCaptcha
o Image Processing

5. JCaptcha
o Open-source CAPTCHA
generation software
o Highly configurable
Can produce CAPTCHAs of
many levels of difficulty
o Check it out at:
http://jcaptcha.sourceforge.net

6. Image Processing
Sparse Image
Represents Images as unbounded set of pixels
Each pixel is a value between 0 and 1 and a
coordinate pair
Center each image before turning into a matrix of
0s and 1s
Original After Transformation

7. As covered in class
Feed-Forward Neural Nets

8. Self-Organizing Maps
Training
Initialize N buckets to
random values
For each input
Find the bucket that is
“closest” to the input
Adjust the “closest”
bucket to more closely
match the input using
exponential average
Collection
For many inputs
Sort each input into
the bucket it most
closely matches
For each bucket and each
character
Calculate the
probability of that
character going into
that bucket.

9. K-Means
• Very similar to Self‐
Organizing Maps
(SOMs)
• Can use the same
classifying mechanism
as used for SOM

10. Overlapping Segmentation
• Divide image into
fixed number of
overlapping tiles of
the same size
• In our case, 20 x 20
pixels with a 50%
overlap
• Discard chunks
under a certain size
and chunks that are
all white
Note: This is a B with
part of it cut off, not
an E. Therein lies the
rub.

11. • Iterate through the
image from left to
right—segment
when a full column
of whitespace is
encountered
• Works perfectly for
well-spaced text
Whitespace Segmentation

12. K-Means Segmentation
• Performs better
than heuristic
segmentation on
closely-packed
inputs

13. Even‐width
K‐Means
Whitespace
Even‐width
K‐Means
Whitespace
Segmentation Comparison

14. Experiment 1
Machine Learning Method:
Self-Organizing Map
Topology
200 buckets, initialized randomly
Inputs:
3 letter CATPCHAs
Random fonts
Letters A-G
“Chunked” using overlapping segmentation

15. Experiment 1 Results
Buckets fell into three primary categories:
Distinguishable
letters
Chunks with halves
of two letters
Indistinguishable
noise

16. Experiment 1 Results

17. Experiment 2
ML Method:
Neural Net
Topology:
Fully connected
400 inputs
50 node hidden layer
7 outputs
Inputs:
Single letter CATPCHAs
Random fonts
Letters A-G
400 Nodes
50 Nodes
7 Nodes
Contains … ?
A: 0 or 1
B : 0 or 1
C: 0 or 1
D: 0 or 1
E: 0 or 1
F: 0 or 1
G: 0 or 1

18. Neural Net Learning Curve
Experiment 2 Results

19. Experiment 2 Results
Neural Net Accuracy vs. Size of Hidden Layer
Past a certain
number of nodes
in the hidden
layer, the
topology ceases
to have a huge
impact on
accuracy.

20. Experiment 3
ML Method:
Neural Net
Topology:
Fully connected
400 inputs
1000 node hidden layer
7 outputs
ML Method:
SOM
Topology:
500 buckets
Inputs:
4 letter CATPCHAs
Fandom fonts
Letters A-G

21. Experiment 3
Neural Net vs. SOM on CAPTCHAs Length 4, Letters A‐G

22. Experiment 4
ML Method:
Neural Net
Topology:
Fully connected
400 inputs
1000 node hidden layer
7 outputs
ML Method:
SOM
Topology:
500 buckets
Inputs:
4 letter CATPCHAs
Fandom fonts
Letters A-Z

23. Experiment 4
Neural Net vs. SOM on CAPTCHAs Length 4, Letters A‐Z

24. Experiment 5
ML Method:
Neural Net
Topology:
Fully connected
400 inputs
1000 node hidden layer
7 outputs
ML Method:
SOM
Topology:
500 buckets
Inputs:
5 letter CATPCHAs
Fandom fonts
Letters A-Z

25. Experiment 5
Neural Net vs. SOM on CAPTCHAs Length 5, Letters A-Z

26. What it all means
• Increasing number of characters
dramatically decreases total accuracy
because segmentation quality decreases
• True positive rate goes down when
segmentation quality decreases
• Hence, better segmentation is the key

27. Future Work
Improved Segmentation
o Wirescreen segmentation
o Ensemble techniques
Improved True Positive Rates with Current
System
o Ensemble techniques
New problems
o Handwriting recognition
o Bot net of doom

28. Questions?