We have implemented and experimented with a lot of deep learning models for better OCR and simultaneously increasing image resolution.

1. Super Resolution
on Text Images
BTech Project - UG 2nd
Year
Indian Institute of Technology, Jodhpur

2. Project By
Nivedit Jain
(B18CSE039)
Student
Dr Gaurav Harit
Mentor/Guide
Sanskar Mani
(B18CSE048)
Student

3. 1Problem
2SRCNN
3Data Set Prep
4Our Implementations
5Metrics
6Use Cases
7Learning Outcomes
Overview

4. Problem Statement
Simultaneous Optimisation of Image Quality Improvement and
Text Content Extraction from scanned documents.

5. Problem Statement
Lower Resolution
Text Image (64*64) Higher Resolution
Text Image (128*128)
Focused towards
optimizing for OCR

6. SRCNN!

7. SRCNN?
● Super-Resolution Convolutional Neural Network (SRCNN)
● used for single image super resolution (SR)
Dong, Chao et al. “Image Super-Resolution Using Deep Convolutional Networks.” IEEE Transactions on Pattern Analysis and
Machine Intelligence 38 (2016): 295-307.

8. Formulation
● Upscale the low resolution image to the desired size using
bicubic interpolation.
● Apply CNN on the upscaled image in order to extract a high
dimensional vector using overlapping patches.

9. Formulation
● Apply CNN on the extracted high-dimensional vector in
order to non-linearly map the information to another
high-dimensional vector in order to prevent overfitting.
● Lastly we apply a final CNN on the higher-dimensional
vector in order to reconstruct the output image.

10. Loss Function
We use Mean-Squared error(MSE) as the loss function.

11. Bicubic
Interpolation
2D
Convolution
2D
Convolution
2D Convolution

12. Data Set
Preparation

13. Data Set Preparation
UNLV 1985 Distinct
Documents

14. Data Set Preparation
UNLV 1985 Distinct
Documents
Business
Letters
ReportsLegal
Documents
NewspaperMagazinesDepartment of
Energy

15. Randomly Select 50 Images from each
Data Set Preparation

16. Randomly Select 50 Images from each
Split 200/50/50 training/validate/test
Data Set Preparation

17. Cut Images to Obtain HR (128*128)

18. Cut Images to Obtain HR (128*128)
Manually Removed non useful images

19. Cut Images to Obtain HR (128*128)
Manually Removed non useful images
Apply Gaussian Blur (7*7 Kernel ,
𝜎=uniform(0.5,5)) and Downscale (64*64)

20. Cut Images to Obtain HR (128*128)
Manually Removed non useful images
Apply Gaussian Blur (7*7 Kernel ,
𝜎=uniform(0.5,5)) and Downscale (64*64)
HR

21. Cut Images to Obtain HR (128*128)
Manually Removed non useful images
Apply Gaussian Blur (7*7 Kernel ,
𝜎=uniform(0.5,5)) and Downscale (64*64)
HR
LR

22. Data Set Preparation
So our prepared dataset has
Test (HR-LR) Pairs : 10,929 Images
Train (HR-LR) Pairs : 42,602 Images
Validate (HR-LR) Pairs : 9659 Images

23. Our
Implementation

24. Function - g

25. Tesseract OCR
Tesseract is considered one
of the most accurate
open-source OCR engines
then available.*
https://en.wikipedia.org/wiki/Tesseract_(software) , https://tesseract-ocr.github.io/

26. OTSU Thresholding

27. OTSU Thresholding

28. Loss Function

29. Loss Function

30. Results

36. Other Experiments

37. Normal SRCNN

38. Results

44. SRCNN
Training Data :
(LR,Tesseracted)

45. Results

51. 2 Layer SRCNN
Training Data :
(LR,Tesseracted)

53. Results

59. Extra SRCNN
Training Data :
(LR,Tesseracted)

61. Results

67. SRCNN
Training Data :
(LR,Tesseract otsu)

68. Results

74. Evaluation
Matrices

75. Flow Metric

76. Flow Metric
Make a Bipartite Graph with Bounding Boxes
of Original and Predicted as nodes

77. Flow Metric
Make a Bipartite Graph with Bounding Boxes
of Original and Predicted as nodes
Add edges with percentage of intersection as
weights

78. Add Source and Sink, and connect with all
edges of weight 1

79. Add Source and Sink, and connect with all
edges of weight 1
Calculate the maximum flow

80. Bipartite Metric

81. Bipartite Metric
Make a Bipartite Graph with Bounding Boxes
of Original and Predicted as nodes

82. Bipartite Metric
Make a Bipartite Graph with Bounding Boxes
of Original and Predicted as nodes
Add edges with 1 as weight when the character matches and
also the intersection is greater than a threshold.

83. Add Source and Sink, and connect with all
edges of weight 1

84. Add Source and Sink, and connect with all
edges of weight 1
Calculate the maximum flow
This finds the maximum cardinality bipartite match
between the two B.B.

85. PSNR

86. PSNR

87. 85.32% 82.11%
Modified Loss Function
FlowBipartite PSNR
23.31db

88. 83.07% 80.55%
Normal SRCNN
FlowBipartite PSNR
28.94 db

89. 77.49% 74.04%
LR - Tesseract
FlowBipartite PSNR
21.80 db

90. 81.49% 78.82%
2-Layer SRCNN
FlowBipartite PSNR
24.28 db

91. 81.51% 78.91%
Extra SRCNN
FlowBipartite PSNR
22.63 db

92. 79.14% 75.76%
LR - Tesseract OTSU
FlowBipartite PSNR
20.41db

93. Metric Limitation

95. Some Use Cases
01Making OCR Better
02Digitization of Documents

96. Learning Outcomes
Basics of AI Python

97. Thank You!

99. Backup Slides