The document describes an algorithm for templateless marked element recognition in documents using computer vision. It discusses preprocessing steps like converting images to grayscale, blurring, and edge detection. It then describes detecting shapes like checkboxes and radio buttons using contour analysis and evaluating pixel thresholds to determine if elements are selected. Pseudocode provides details of the complete algorithm to detect and mark checked checkboxes and radio buttons on input images without predefined templates.

1. Templateless Marked Element Recognition
Identification using Computer Vision
Shivam, Chaurasia
shivam.chaurasia@tcs.com
891147
Tata Consultancy Services Pvt Ltd.
22 April, 2019
1 Introduction
OMR technology is used for collecting data from “fill-in-the-bubble” forms such
as educational tests, surveys, assessments, evaluations, and many other multiple
choice forms.
Computer vision is the automatic analysis of images and videos by computers
in order to gain some understanding of the world.Computer Vision is a rapidly
expanding area and it is becoming progressively easier for developers to make
use of this field due to the ready availability of high quality libraries (such as
OpenCV 2).
A checkmark field is an element on a machine-readable form (usually rect-
angular in shape and often called a “check box”) in which a mark should be
made.
This makes it clear that we need algorithms that are capable of finding both
square, circles and ellipses checkmarks.
To determine a element is selected we have used two level :
• Preprocessing: Color to Gray Scale Conversion of Image, Charachter Sep-
aration, Thinning using threshold, Edge detection
• Element Recognition: Based on shape detection algorithm in computer
vision we detect whether the region is checkbox or radio button.
• Pixel Threshold Evaluation: It calculates the percentage of fill in black
pixel to determine whether it is selected or not.
1

2. 2 Solution Approach
Input Image
Split in
Each
Individual
Blocks
Preprocess Get Bird Eye
identify
next shape
in each
block
evaluate
shape
parameter
is it
Radio?
is it
Marked?
update
color
update
color
is it
checkbox?
is checked
?
stop
yes
yes
yes
no
no
no
yes
2

3. 3 Algorithm Psuedo Code
Algorithm 1
Data: Input Image
Result: All Marked Checkboxes and Radio Button with Color
Initialize package;
LoadImage(Path) image with checkboxes and radio button
SplitGridImage(img, directorytosave) extract all bounding individual grid
region
while Each instance of grid do
PreprocessImage(img)
DetectContours(preprocessed image)
if DetectShape(contour) ← CHECKBOX then
PixelThresholdEvaluation(contours) finds filled in checkbox
instructions2
else
no checkbox found;
end
if DetectShape(contour) ← RADIO then
PixelThresholdEvaluation(contours) finds filled in radio button
else
no radio found;
end
end
3

4. (a) Input. (b) Section Detection. (c) Each Section
(d) Recognise and Mark
Figure 1: Complete Overview
4

5. 3.1 Individual Grid Image Extraction
(a) Input Image (b) Horizontal Lines Detected
Figure 2: Split Image in by finding horizontal and vertical lines using morpho-
logical operation
5

6. (a) Vertical Lines (b) Combined
(c) Shape Build For Crop-
ping Using Both Image
Figure 3: Final Splits
6

7. 3.2 Marked Element Detection Algorithm
Algorithm 2
Data: Input Image
Result: All Marked Checkboxes Colored
img ← LoadImage(Path) gray ← ConvertToGrayScale(img) blurred ←
GaussianBlur(gray)
edges ← CannyEdgeDetectoe(blurred, low, upper, sigma)
contours ← OpenCvGetContours(edged)
while Each contour instance do
peri ← arcLength(instance)
approx ← approxPolyDP(contour, sigma ∗ peri, closed)
(x, y, w, h) = cv2.boundingRect(approx)
area ← cv2.approxPolyDP(contour, sigma ∗ peri, closed)
circles ← HoughCircles(contour)
if circles = 0 then
Radio Detected
PixelThresholdEvaluation(contours) finds is it filled
drawContours(original image,[contour], plot all, color, thicknes)
else
no radio found;
end
if length(approx) ← 4 and aspectratio ≈ 1 and area in range then
CheckBox Detected
PixelThresholdEvaluation(contours) finds filled mark
drawContours(original image,[contour], plot all, color, thicknes)
else
no checkbox found;
end
end
(a) Input Image
(b) Horizontal Lines Detected
Figure 4: Detection And Mark
7

8. 4 Code
1 #IMPORTING ALL THE REQUIRED PACKAGE
2 import cv2 # version 4.1
3 import numpy as np
4 import copy
5 import numpy as np
6 import imutils
7 import matplotlib.pyplot as plt
8 import matplotlib.image as mpimg
9 import matplotlib
10 import glob, os
11
12 def sort_contours(cnts, method="left-to-right"):
13 '''
14 This function takes the opencv contours and sorts it in
15 descending or ascending order depends upon method parameter
16 '''
17 reverse = False
18 i = 0
19 if method == "right-to-left" or method == "bottom-to-top":
20 reverse = True
21 if method == "top-to-bottom" or method == "bottom-to-top":
22 i = 1
23 boundingBoxes = [cv2.boundingRect(c) for c in cnts]
24 (cnts, boundingBoxes) = zip(*sorted(zip(cnts, boundingBoxes),
25 key=lambda b:b[1][i], reverse=reverse))
26 return (cnts, boundingBoxes)
27
28
29 def checkbox_detect(image_path, method=''):
30 '''
31 Ths function take image path or image as an input
32 and detects checkbox in it
33
34 1. Read the image
35 2. Convert it into Grayscale
36 3. Blur the Image
37 4. Detect Edges using Canny edge detector
38 5. Detect all the contours
39 6. Identify the shape using area, threshold, aspect ratio,
40 contours closed/open
41 7. Draw boundaries on shapes found
42 '''
43
44 #A kernel of 5X5 matrix
45 kernel = np.ones((5,5), np.uint8)
8

9. 46 #1. Read the image: If the parameter method is path
47 # else use the image directly
48 if method=="path":
49 original_image = cv2.imread(image_path)
50 else:
51 original_image = image_path.copy()
52 image = original_image.copy()
53
54 #2. Convert it into Grayscale
55 gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
56 #3. Blur the Image
57 blurred = cv2.GaussianBlur(gray, (5, 5), 0)
58
59 #4. Detect Edges using Canny edge detector the lower and upper threshold
60 # boundaries are calculated using median and sigma
61 sigma = 0.33
62 v = np.median(blurred)
63 lower = int(max(0, (1.0 - sigma) * v))
64 upper = int(min(255, (1.0 + sigma) * v))
65 edged = cv2.Canny(blurred, lower, upper)
66
67 plt.imshow(edged)
68 plt.title('Edged Canny')
69 plt.show()
70
71 # 5. Detect all the contours and grab the values using imutilsgrab_contours
72 cnts = cv2.findContours(edged.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
73 cnts = imutils.grab_contours(cnts)
74
75 checkbox_contours = []
76 contour_image = edged.copy()
77
78 areas_pt = []
79 font = cv2.FONT_HERSHEY_SIMPLEX
80 # Loop over each and every contours for filtering the shape
81
82 for c in cnts:
83 # 6. Identify the shape using area, threshold, aspect ratio, contours closed/open
84 peri = cv2.arcLength(c, True)
85 approx = cv2.approxPolyDP(c, 0.035 * peri, True)
86 (x, y, w, h) = cv2.boundingRect(approx)
87 aspect_ratio = w / float(h)
88 area = cv2.contourArea(c)
89 areas_pt.append((len(approx), area, aspect_ratio))
90 if area>10.0 and area < 250.0 and (aspect_ratio >= 0.82 and aspect_ratio <= 1.2) :
91 # 7. Draw boundaries on shapes found
92 cv2.drawContours(original_image,[c], 0, (0,255,0), 3)
93 checkbox_contours.append(c)
9

10. 94
95 wid_six = [ (a,b,c) for a,b,c in areas_pt if a == 4]
96 print(areas_pt)
97
98 print(wid_six)
99 print(len(checkbox_contours))
100
101 plt.imshow(original_image)
102 plt.title('checkboxes_image')
103 plt.show()
104
105
106 def box_extraction(img_for_box_extraction_path, cropped_dir_path):
107 '''
108 This function takes the entire images split its into
109 individual grid element and stores the output in Cropped Folder
110 '''
111
112 img_wd_rect = cv2.imread(img_for_box_extraction_path)
113 temp = img_wd_rect.copy()
114 img=cv2.cvtColor(img_wd_rect.copy(),cv2.COLOR_BGR2GRAY)
115
116 # Thresholding the image
117 (thresh, img_bin) = cv2.threshold(img, 128, 255,
118 cv2.THRESH_BINARY | cv2.THRESH_OTSU)
119 img_bin = 255-img_bin # Invert the image
120 # Defining a kernel length
121 kernel_length = np.array(img).shape[1]//40
122
123 # A verticle kernel of (1 X kernel_length),
124 # which will detect all the verticle lines from the image.
125 verticle_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (1, kernel_length))
126
127 # A horizontal kernel of (kernel_length X 1),
128 # which will help to detect all the horizontal line from the image.
129 hori_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (kernel_length, 1))
130
131 # A kernel of (3 X 3) ones.
132 kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3))
133
134 # Morphological operation to detect verticle lines from an image
135 img_temp1 = cv2.erode(img_bin, verticle_kernel, iterations=3)
136 verticle_lines_img = cv2.dilate(img_temp1, verticle_kernel, iterations=3)
137 cv2.imwrite("verticle_lines.jpg",verticle_lines_img)
138
139 # Morphological operation to detect horizontal lines from an image
140 img_temp2 = cv2.erode(img_bin, hori_kernel, iterations=3)
141 horizontal_lines_img = cv2.dilate(img_temp2, hori_kernel, iterations=3)
10

11. 142 cv2.imwrite("horizontal_lines.jpg",horizontal_lines_img)
143
144 # Weighting parameters, this will decide the quantity of an image
145 # to be added to make a new image.
146 alpha = 0.5
147 beta = 1.0 - alpha
148
149 # This function helps to add two image with specific weight parameter
150 # to get a third image as summation of two image.
151 img_final_bin = cv2.addWeighted(verticle_lines_img, alpha,
152 horizontal_lines_img, beta, 0.0)
153 img_final_bin = cv2.erode(~img_final_bin, kernel, iterations=2)
154 (thresh, img_final_bin) = cv2.threshold(img_final_bin,
155 128, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)
156
157 # For Debugging
158 # Enable this line to see verticle and horizontal lines
159 # in the image which is used to find boxes
160 cv2.imwrite("img_final_bin.jpg",img_final_bin)
161 # Find contours for image, which will detect all the boxes
162 contours, hierarchy = cv2.findContours(
163 img_final_bin, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
164
165 # Sort all the contours by top to bottom.
166 (contours, boundingBoxes) = sort_contours(contours, method="top-to-bottom")
167 idx = 0
168
169 for c in contours:
170 # Returns the location and width,height for every contour
171 x, y, w, h = cv2.boundingRect(c)
172 # If the box height is greater then 20, widht is >80,
173 # then only save it as a box in "cropped/" folder.
174 if (w > 80 and h > 20) : # and w > 3*h
175 idx += 1
176 new_img = img[y:y+h, x:x+w]
177 cv2.imwrite(cropped_dir_path+str(idx) + '.png', new_img)
178 #cv2.drawContours(temp,[c],-1,(255,0,0),2)
179 cv2.rectangle(temp, (x, y), (x + w, y + h), (0, 0, 255), 2)
180
181
182 cv2.imwrite("intermediate.jpg",temp)
183 return img_wd_rect
184
185 # USAGE
186 #Already Available Single Instance
187 checkbox_detect('./images/13.png', method="path")
188 checkbox_detect('./images/15.png', method="path")
189 checkbox_detect('./images/25.png', method="path")
11

12. 190 checkbox_detect('./images/27.png', method="path")
191 checkbox_detect('./images/29.png', method="path")
192
193 # An complete grid image
194 op = box_extraction("./images/6.jpg", "./Cropped/")
195
196 folder = "./Cropped"
197 filenames = [name for name in os.listdir(folder) if name.endswith(".png")]
198
199 print(filenames)
200 for i in filenames:
201 checkbox_detect('./Cropped/'+i, method="path")
202
203
204 #CLEANUP - Deletion of files in cropped folder
205 for the_file in os.listdir(folder):
206 file_path = os.path.join(folder, the_file)
207 try:
208 if os.path.isfile(file_path):
209 os.unlink(file_path)
210
211 except Exception as e:
212 print(e)
5 Installation
1. Install Anaconda 3.7 - https://www.anaconda.com/distribution/
2. Anaconda comes with pre installed most of the packages.
3. Install Opencv 4.1.0
• Download Installer and note the location:
• https://opencv.org/releases/
• Then from installed locationopencvbuildpythoncv2python-3.7
copy all *.pyd files. And paste it in Anaconda3Libsite-packages
4. Install Dependencies from command line:
pip install imutils
pip install opencv-contrib-python
5. Execute the script:
python finalscript.py
12

13. 6 Commercial Similar Product
• IBM Datacap
13