This document summarizes Tesseract's approach to optical character recognition (OCR). It discusses Tesseract's use of segmentation, language models, and beam search to recognize text. Tesseract initially segments text but then refines the segmentation through chopping and combining as needed. It represents segmentation possibilities as a graph and integrates language models to search the graph and recognize words.

1. Tesseract Tutorial: DAS 2014 Tours France
6. Character
Segmentation, Language
Models and Beam Search
The heart of Tesseract
Ray Smith, Google Inc.

2. Tesseract Tutorial: DAS 2014 Tours France
Approaches to Segmentation
● Segment first using only geometry.
● Maximally chop, then combine with a beam
search. (Over-segmentation.)
● Sliding window to "avoid" segmentation
altogether.
● Tesseract: Chop only as needed, then
combine as needed.

3. Tesseract Tutorial: DAS 2014 Tours France
Over-Segmentation
● Aim is to maximize recall of chops with the compromise of
reduction in precision.

4. Tesseract Tutorial: DAS 2014 Tours France
Segmentation Graph
● Segmentation possibilities and classifier results form a directed
graph
l } I 1
m
u nh b
l } I 1 i l 1 e c r T
h b
i l 1 i l 1 i l 1 l } I 1
m
m
g 9 q
n u %

5. Tesseract Tutorial: DAS 2014 Tours France
Searching the Segmentation Graph
l } I 1
m
u nh b
l } I 1 i l 1 e c r T
h b
i l 1 i l 1 i l 1 l } I 1
m
m
g 9 q
n u
higher
%

6. Tesseract Tutorial: DAS 2014 Tours France
Searching the Segmentation Graph
l } I 1
m
u nh b
l } I 1 i l 1 e c r T
h b
i l 1 i l 1 i l 1 l } I 1
m
m
g 9 q
n u
l } I 1
m
u nh b
l } I 1 i l 1 e c r T
h b
i l 1 i l 1 i l 1 l } I 1
m
m
g 9 q
n u
higher
}uglier
%
%

7. Tesseract Tutorial: DAS 2014 Tours France
Integration of Language Models (General Methods)
● Implement Language Model as Finite State Machine.
● Search Language Model and Segmentation Graph in parallel.
● Combine “probabilities” in some sensible way.
● Hidden Markov Model methods are good example.

8. Tesseract Tutorial: DAS 2014 Tours France
Segmentation Free = Extreme Over-Segmentation
● Slide over the word/textline with a classifier/HMM.
● Beam search + shape model probs + language model probs solves
the segmentation internally.
● Really just mega-over-segmentation.

9. Tesseract Tutorial: DAS 2014 Tours France
Tesseract Segmentation Approach based on observations:
● Initial segmentation is often correct or close.
● Classifier generally doesn’t like incorrectly segmented
text.
● Over-segmentation often leads to poor results., eg m->iii

10. Tesseract Tutorial: DAS 2014 Tours France
Tesseract Segmentation Approach
Classify Initial Segmentation
Search Word: OK? Yes => Done
while any Bad Blob has any Chops available
Chop and classify pieces of Worst Choppable Blob
Search Word: OK? Yes => Done
while any fixable “Pain point”
Associate adjacent blobs and classify
Search Word: OK? Yes => Done

11. Tesseract Tutorial: DAS 2014 Tours France
Types of Pain Point
● Initial: Join each adjacent pair
● Ambiguity: Eg m/rn
● Path: Neighbors of blobs in the current best path

12. Tesseract Tutorial: DAS 2014 Tours France
Ratings MATRIX = Segmentation Graph
l } I 1
m
u nh b
l } I 1 i l 1 e c r T
h b
i l 1 i l 1 i l 1 l } I 1
m
m
g 9 q
n u
%
l}Ii
hb il1
un il1
m nu il1
g9q l}I1
% l}I1
hb il1
m ec
m rT
Each entry holds a BLOB_CHOICE_LIST providing classifier choices
with rating and certainty.

13. Tesseract Tutorial: DAS 2014 Tours France
Live Demo of Segmentation Search
Chopper demo
api/tesseract unlv/mag.3B/2/8022_028.3B.tif test1 inter segdemo chopdebug3
Col 2, line 6, word 1
Associator demo
api/tesseract unlv/doe3.3B/4/2214_007.3B.tif test1 inter segdemo
Col 2, line 8, word 2

14. Tesseract Tutorial: DAS 2014 Tours France
Evaluation of a WORD_CHOICE (no params-model)
Word Rating = word_factor ∑blob_choice->rating()
word_factor = segmentation
Condition base word_factor Add-ons
Frequent dawg word 1.0 Inconsistent case +0.1
Other dawg word 1.1 Inconsistent case +0.1
Non-dawg word 1.25 +0.01 for each
char over 3.
Inconsistent case +0.1
Inconsistent punc +0.2
Inconsistent chartype +0.3
Inconsistent script +0.5
Inconsisted char spacing +0.01
All except script +0.01 for each
additional occurrence.

15. Tesseract Tutorial: DAS 2014 Tours France
Evaluation of a WORD_CHOICE (with params-model)
Word Rating = word_factor ∑outline length
word_factor = weighted sum of word features:
● mean blob rating
● num inconsistent spaces
● num inconsistent char type
● num x-height inconsistencies
● num case inconsistencies
● word length (in type categories)

16. Tesseract Tutorial: DAS 2014 Tours France
Tesseract Word Recognizer

17. Tesseract Tutorial: DAS 2014 Tours France
Example of Chopping (unlv/mag.3B/2/8022_028.3B.tif
Col 2, line 6, word 1)
Word Distance Worst blob
Momm 212.2 7.7
Mommn 186.3 8.3
Momtfln178.0 9.2
Momtain 124.9 5.3
Mounm 184.0 7.7
Mountain 80.6 3.1 ACCEPT!

18. Tesseract Tutorial: DAS 2014 Tours France
Example of Combining (unlv/doe3.3B/4/2214_007.3B.tif,
col 2, line 8, word 2)
Word Distance
lilllit- 77.58
limit57.1
Emit 89.7
Unfit 95.4
Hulk 122.7
Bulk 136.8
Word Distance
HUM 120.6
MUM 127.0
BUM 134.7
1mm 112.8
Milk147.2
huh 137.1
fink 140.7
Emu 129.7
BMW 140.3

19. Tesseract Tutorial: DAS 2014 Tours France
Thanks for Listening!
Questions?