3. Overview
• By 2021, the Canadian Radio Television and Telecommunication has mandated that all the
Canadian telephone companies must support next generation 911 voice and text messaging.
• Next generation text messaging includes provision for videos and Images.
• To achieve this, we develop an Image processing technique to next generation media to pre-
process information and showing the type of severity of the graphic image before Canadian
telecommunications operators receive it.
• The next generation 911 project has a list of challenges like Detection of presence or absence of
humans, Fire detection, Detection of image orientation and camera perspective, Automatic
detection of image blur and distortion in RGB images and finding the image scale of the unknown
image.
• In this project, I am working on part of problem which determines the scale of the image or field of
view of the unknown image.
4. Background
• The input image will be the Depth image which is estimated by self supervised monocular depth
estimation method which is an existing approach which leads to methodology.
• Following the estimation of the depth, the image is then analyzed to automatically find the image
scale by proposed methodology by using the libraries like OpenCV, Pillow (PIL), NumPy.
Input Depth Image
Raw Image
Proposed
method
Result
5. Flow Chart
Estimation of Dark
Pixel
• Dark
Pixel
Score
Finding the
Horizon
• Horizon
Score
Removing
Foreground Gradient
• Smooth
gradien
t Score
Object Pixels
• Number
of
Object
Pixels
Final Result
• Final
Score
Input
Image
6. Estimation of Dark Pixel
• The Dark pixel in this process represents the “Far Background” in the input image. They are not
informative directly but they may help to an understanding of the scale of the image.
• Far Background pixels also called Dark pixels should exist only in Wide field of view Images.
• Range function in OpenCV is used here to estimate the Dark pixel in the input image. The
estimation is determined by counting the dark pixels from the depth image by creating lower and
upper boundary of the dark pixels and counting all the pixels within the boundary.
• Score is determined by following expression,
• Score_stage1 = (Percentage of pixels in boundary/100) * 10
• Where Percentage = (pixel count in boundary/total pixel) *100
7. Estimation of Dark Pixel
Figure 1: Input image Figure 2: Dark Pixels or far background
Figure 3: Stage 1 score
Figure 4: Output image of stage 1
8. Finding the horizon in the image
• In this stage, the input image will be the output of the stage 1.
• The horizon in the image will be the region between the background and foreground.
• The horizon gives better understanding of image scale and orientation.
• Horizon is found by Otsu’s thresholding method.
• Horizon score is calculated by the following expression,
• average = (horizon_y1 + horizon_y2)/2
• percentage_of_horizon = (average/height)*100
• score_stage2 = (percentage_of_horizon/100) * 10
• Otsu’s thresholding in image processing is used to perform Automatic image thresholding.
• cv.threshold() function is used where cv.THRESH_OTSU is passed as extra flag.
• The threshold value can be chosen arbitrary. The algorithm then finds the optimal threshold value which
is returned as the output.
• The horizon is plotted in the region between the foreground and the background and the score of the
horizon is calculated.
9. Finding the horizon in the image
• After finding the horizon, the region above the horizon is removed. The resulting image will be the
input image to the next stage.
Figure 5: Input image Figure 6: output of Otsu’s thresholding Figure 7: plotting the Horizon
Figure 8: Score of stage 2 Figure 9: output image
10. Removing Foreground Gradient
• Aim of this stage is calculate the smooth gradient (foreground) and the Object pixels in the image.
Here the input image will be the output image of the stage 2.
• The parts of the foreground image that are un-interesting are removed.
• The physical ground is always recorded as the smooth gradient in an image and the gradient occurs
in a direction perpendicular to the horizon. Hence we can remove the pixels of the foreground with
this knowledge.
Figure 10: Input image of stage 3
Smooth
gradient
0
255 uneven
gradient
11. Removing Foreground Gradient
• Let us consider the input image in matrix form. The values in the first column decreases gradually
because of the smooth gradients and there are no objects in between.
Figure 11: NumPy object array of input image
12. Removing Foreground Gradient
• To remove the smooth gradients, a condition is passed. “ if the value keeps decreasing and the
difference between the lower pixel and upper pixel is less than or equal to the margin value, that
pixel is removed. This is done to all the columns in the matrix.
• If there is an object in between, the iteration is stopped for that particular column and it starts
from the next column. After the iteration is complete, the output image of the final stage looks like
this. Since the score of foreground gradient is 7.7 and object is 2.29, we can estimate that the
image has maximum of foreground and medium object.
Figure 13: Score of stage 3
Figure 12: output image of stage 3
13. Graphical Analysis
• This graph shows the analysis of intensity value of the left most column of the image.
• Values move up gradually without any deviation.
14. Graphical Analysis
• This graph shows the analysis of intensity value of right most column of the image.
• The gradient is smooth, as values move up gradually without any deviation
15. Graphical Analysis
• This graph shows the analysis of intensity value of the middle column of the image.
• The gradient is not smooth because there is object in this part of the image, hence there is
deviation in the graph.
Uneven gradient
16. Overall Score
• The overall score is calculated by the following expression
• C_1 * dark_score + C_2 * horizon_score + C_3 * gradient_score + C_4 * Object_pixels
• Where C_i is weighted constants
• The overall score will give you an estimate of the image scale or the field of view, where 0 being
narrow field of view and 10, wide field of view.
• In this case, the overall score will be
• 0.4 * 5.107 + 0.4 * 5.29 + 0.2 * 7.7 + 0.2 * 2.29 = 6.1568
• Since the score is above 5, we can estimate that the Image has wider field of view with medium
size object.
17. Case 2
Figure 13: score stage 1
Figure 14: score stage 2
Figure 15: score stage 3
Final score = 0.4*3.16+0.4*3.53+0.2*3.69+0.2*6.3 = 4.46
Since the score is below 5, the image scale can be estimated as narrow
field of view with large object.
18. Conclusion
• A simple approach to estimate the field of view of an unknown image
• With the information obtained from this approach, one could then use existing approaches which
are dependent on the field of view of the image.
