EnrichmentWeek Binus Computer Vision

1. COMPUTER VISION
GIA MUHAMMAD

2. Gia Muhammad
Direktur Teknologi, Software Engineering and Artificial Intelligence Specialist
Xingular.AI
PT. Xingular Inovasi Teknologi
https://xingular.ai
gia@xingular.ai
https://www.linkedin.com/in/giamuhammad/
https://github.com/giamuhammad
http://ipb.academia.edu/giamuhammad
+62819 0865 2920
Vendor Partnership

6. ARTIFICIAL INTELLIGENCE
Artificial Intelligence is the science and engineering of making machine have an intelligence (McCarthy, 2007)
Artificial
General
Intelligence
Artificial
Narrow
Intelligence
Artificial
Super
Intelligence
Technology Singularity

7. ARTIFICIAL INTELLIGENCE
Intelligence is about tolerance, it able to face uncertainty
Human also can make mistakes, yet progress in learning to avoid the
same mistake, so neither in AI…

8. COMPUTER VISION IN ARTIFICIAL
INTELLIGENCE Artificial
Intelligence
Natural Language
Processing
Computer/Machine
Vision
Object Recognition
Face Recognition
User Retention
Measurement
Security Access
and People
Tracking
Face and Body
Analytics
Audience
Measurement
Gesture
Recognition
Behavior
Recognition
Object Detection
and Tracking
Traffic Analytics
Environment/Scene
Perception
Optical Character
Recognition
ALPR
Automatic
Document Scanner
Barcode
1D Barcode
2D Barcode
Augmented Reality
Mixed Reality (XR)
Face Filter
Image Processing
Image Editing
Watermarking
Image
Compression
Robotics Speech Recognition

9. COMPUTER VISION IN ARTIFICIAL
INTELLIGENCE
Artificial
Intelligence
Non-
Knowledge
Based
Knowledge
Based

10. COMPUTER VISION IN ARTIFICIAL
INTELLIGENCE
Computer
Vision
Non-
Knowledge
Based
Image
Processing
Preprocessing
Image
Preparation
Image Feature
Visual
Perception
Pose Estimation View-
Geometry
Single View Multi View
Stereo Vision
Knowledge
Based
Computer
Vision
Non-
Knowledge
Based
Knowledge
Based
Object
Recognition
Machine
Learning
Non-Neural
Net
SVM
etc
Neural
Network
Deep Learning
CNN
RCNN

11. INTRODUCTION OF SELF-DRIVING CAR
Self-Driving car is a vehicle that is capable of sensing its environment and moving safely
with little or no human input (Hu; et al, 2020)

12. VEHICLE AUTOMATION (VA) LEVEL

13. The Human Monitors the Driving Environment The Automated System Monitors the Driving Environment
Source : SAE International and J3016
VEHICLE AUTOMATION (VA) LEVEL
“Artificial Intelligence is still not able to function properly in chaotic inner-city environments.”
(SingularityU The Netherlands)

14. HARDWARE (SENSOR) IN SELF-DRIVING
• Stereo Camera
• LiDAR
• GPS
• Radar
• Ultrasonic
• Wheel Encoder
• IMU (Inertial Measurement Unit) or INS (Inertial Navigation System)

15. STEREO CAMERA

16. DEVICE AND METHOD COMPARISON
LiDAR for Self-Driving Stereo Camera
Cost Expensive Proper
Range, Distance Measurement Extensive Limit, Depend on FOV and
Geometry Calculation
Precision to reconstruct object More Accurate, Sensitive Less Accurate
Method Laser Time-of-Flight, Structured Light,
Passive and Active Stereo

17. DEVICE AND METHOD COMPARISON
Laser Structured Light Time-of-Flight Passive Active
Method Triangulation
Precision Accurate Accurate Less Accurate, Depend
on 3D Reconstruction
Method
Less Accurate, Bad for
Dark Condition
Less Accurate, Depend
on 3D Reconstruction
Method/Algorithm
Speed < 1s About 2 seconds < 1s Depend on Computation
Time of Disparity Map
< 1s
Cost Very High High Medium Low. Just need more than
1 RGB Camera but need
extra compute chip to
run disparity map
Medium

18. LIDAR. LIGHT DETECTION AND RANGING
High Precision for Short and Long Range Object
Implementation
- Mapping for Remote Sensing
- Self-Driving Car
- 3D Printing from 3D Reconstruction
- Robot Navigation

19. STRUCTURED LIGHT
High Precision for Short Range Object
Implementation
- 3D Map Guides Robot in Industrial Factory
- 3D Printing from 3D Reconstruction

20. TIME OF FLIGHT
Low Precision for Short Range Object
(LiDAR like, using different laser, usually combined with RGB Camera Sensor)
Implementation
- 3D Printing from 3D Reconstruction
- 3D Object Detection
- Self-Driving Car
- Robot Navigation
- Camera Effect
- Game Control to Detect Skeleton (Microsoft Kinect)
RGB Camera Infrared Laser
Raw 3D
Reconstructed
AI-based
algorithm 3D
Reconstruction
refinement
Final Result
(High Precision)
3D Reconstructed Illustration of ToF Stereo Camera

21. LIDAR IN SELF-DRIVING CAR

22. WHY STEREO VISION AND AI
Elon Musk in Autonomy Day for Investors, 2019

23. WHY STEREO VISION AND AI
Less Precision 3D
Reconstructed
AI
(The ability to
handle
uncertainty)
Accurate 3D
Object
Recognition with
Low Cost Sensor

25. NON-KNOWLEDGE BASED AI
Search
Algorithm Blind
Breadth First
Search (BFS)
Depth First
Search
Informed A* Search

26. BREADTH FIRST SEARCH ALGORITHM
• Only decide first/init node
• List neighbour node first
• Using queue to find shortest path
Start(S) = ABCDEFG
Start(a) = bcdefgh
Start(D) = ?

27. BREADTH FIRST SEARCH ALGORITHM
Python Implementation
graph = {
'S' : ['A','B','C'],
'A' : ['D','S'],
'B' : ['E','S'],
'C' : ['F','S'],
'D' : ['G','A'],
'E' : ['G','B'],
'F' : ['G','C'],
'G' : ['D','E','F’]
}
visited = [] # List to keep track of visited nodes.
queue = [] #Initialize a queue
def bfs(visited, graph, node):
visited.append(node)
queue.append(node)
while queue:
s = queue.pop(0)
print (s, end = " ")
for neighbour in graph[s]:
if neighbour not in visited:
visited.append(neighbour)
queue.append(neighbour)
# Driver Code
bfs(visited, graph, 'S')
Open file 1-1_BFS

28. DEPTH FIRST SEARCH ALGORITHM
• Only decide first/init node
• List successor of node first
• Using stack to find shortest path
Start(S) = ADGEBFC
Start(C) = ?

29. DEPTH FIRST SEARCH ALGORITHM
Python Implementation graph = {
'S' : ['A','B','C'],
'A' : ['D','S'],
'B' : ['E','S'],
'C' : ['F','S'],
'D' : ['G','A'],
'E' : ['G','B'],
'F' : ['G','C'],
'G' : ['D', 'E','F']
}
visited = set() # Set to keep track of visited nodes.
def dfs(visited, graph, node):
if node not in visited:
print (node)
visited.add(node)
for neighbour in graph[node]:
dfs(visited, graph, neighbour)
# Driver Code
dfs(visited, graph, 'S')
Open file 1-2_DFS

30. A* ALGORITHM (INFORMED)
• Shortness Path Finding Algorithm
• Decide first/init and finish node
• Considering minimum cost
• F = g + h
where g is distance current node to
and the start node, h is estimated
distance from current node to end
node

31. WORKSHOP
A* ALGORITHM (INFORMED)
grid = [[0, 1, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0],
[0, 0, 0, 0, 1, 0]]
heuristic = [[9, 8, 7, 6, 5, 4],
[8, 7, 6, 5, 4, 3],
[7, 6, 5, 4, 3, 2],
[6, 5, 4, 3, 2, 1],
[5, 4, 3, 2, 1, 0]]
init = [0, 0]
goal = [len(grid)-1, len(grid[0])-1]
cost = 1
delta = [[-1, 0 ], # go up
[ 0, -1], # go left
[ 1, 0 ], # go down
[ 0, 1 ]] # go right
delta_name = ['^', '<', 'v', '>']
result = search(grid,init,goal,cost,heuristic)
for el in result:
print (el)
def search(grid,init,goal,cost,heuristic):
# ----------------------------------------
# modify the code below
# ----------------------------------------
closed = [[0 for col in range(len(grid[0]))] for row in range(len(grid))]
closed[init[0]][init[1]] = 1
expand = [[-1 for col in range(len(grid[0]))] for row in range(len(grid))]
action = [[-1 for col in range(len(grid[0]))] for row in range(len(grid))]
x = init[0]
y = init[1]
g = 0
f = g + heuristic[x][y]
open = [[f, g, x, y]]
found = False # flag that is set when search is complete
resign = False # flag set if we can't find expand
count = 0
while not found and not resign:
if len(open) == 0:
resign = True
return "Fail"
else:
open.sort()
open.reverse()
next = open.pop()
f = next[0]
g = next[1]
x = next[2]
y = next[3]
expand[x][y] = count
count += 1
if x == goal[0] and y == goal[1]:
found = True
else:
for i in range(len(delta)):
x2 = x + delta[i][0]
y2 = y + delta[i][1]
if x2 >= 0 and x2 < len(grid) and y2 >=0 and y2 < len(grid[0]):
if closed[x2][y2] == 0 and grid[x2][y2] == 0:
g2 = g + cost
f = g2 + heuristic[x2][y2]
open.append([f, g2, x2, y2])
closed[x2][y2] = 1
return expand
Open file 2-1_A_Asterisk

32. BFS/DFS AND A* FOR ROBOT NAVIGATION
WAREHOUSE CASE STUDY
Robot
Equipped with
Stereo
Camera or
LiDAR
Use BFS/DFS
to Scan
Environment
3D Map
Reconstructed
The Robot can
be navigated
using A*
pathfinding
and 3D Map
as parameter
When Robot
navigated,
Stereo
Camera/LiDAR
also active to
avoid obstacle
in current time

33. KNOWLEDGE-BASED AI
DATA SCIENCE Discovery
Domain Problem
Data Collection
Data
Preparation
Data Visualization (EDA)
Feature Engineering/Extraction
Data Cleansing
Up/Down Sampling
Model Planning
Regression
Classification
Clustering
Association
Model Building
Machine Learning Algorithm
Tunning Strategy
Model Evaluation
Operation Report of Model
Communication
Communicate the Insight to all
Stakeholder
Data
Science
Analytics Math,
Discrete Math,
Statistics
Programming
Domain/Business
Knowledge

34. DATA SCIENCE
DISCOVERY
Domain Problem Understanding
Example Problem
• Government need to know what happen in the current traffic
Example Solution
• Detect all vehicle type
Example Insight
• Personal car is the most vehicle that affect traffic jam in rush hour
Example Problem
• Government need to reduce traffic jam (Decision converted to Problem)
Example Solution
• Detect all plate number
Example Insight
• Odd and even plate number always balanced when traffic jam happen

35. DATA SCIENCE
DISCOVERY
EDA (Exploratory Data Analysis)

36. DATA SCIENCE
DATA PREPARATION
EDA (Exploratory Data Analysis)

37. DATA SCIENCE
DATA PREPARATION
Feature Engineering/Extraction
Why data need to be extracted?
- Among variables mostly have similar characteristic/feature
- Modelling should process data as knowledge to be learned but high dimensional data would make high
computational cost
- Raw knowledge would make modelling algorithm hard to find the pattern
What should we do?
- Find the pattern or feature manually
- EDA is one of tools to find the pattern

38. DATA SCIENCE
DATA PREPARATION
Feature Engineering/Extraction
Let’s we jump for a while to model planning and find the pattern
Example Case Study
- Fish needs to be classified (model planning) into 2 classes/categories
- What feature that possible to distinguishing the fishes?

39. DATA SCIENCE
DATA PREPARATION
Feature Engineering/Extraction

40. DATA SCIENCE
DATA PREPARATION
Feature Engineering/Extraction
PCA (Principal Component Analysis)
- Statistical method to find the feature with dimensionality reduction
- Transform first then reduce
Steps
1. Standardization (Data Normalization)
2. Find Covariance
3. Find Eigenvalue and Eigenvector from Covariance matrix (cov(X,Y) = A)
4. Sort and choose the Principal Components
5. Transform (Final = Normalized Data * Eigenvector)
Open file 3-1_FeatureExtract_PCA_Iris & 3-2_FeatureExtract_PCA_Cifar10

41. DATA SCIENCE
DATA PREPARATION
Feature Engineering/Extraction with Image Processing
• Edge Detection
• Line Detection
• Point Detection
• Convolution
Image Segmentation with Advanced Method
Image Segmentation with Basic Method

42. DATA SCIENCE
DATA PREPARATION
Feature Engineering/Extraction with Image Processing
Edge Detection
Open file 4-1_EdgeDetection | Please use web camera

43. DATA SCIENCE
DATA PREPARATION
Feature Engineering/Extraction with Image Processing
Line Detection
Open file 4-2_LineDetection | Please use web camera

44. DATA SCIENCE
DATA PREPARATION
Feature Engineering/Extraction with Image Processing
Line Detection
Open file 4-3_PointDetection | Please use web camera

45. DATA SCIENCE
DATA PREPARATION
Feature Engineering/Extraction with Image Processing
Morphology
Dilation
Erotion

46. DATA SCIENCE
DATA PREPARATION
Feature Engineering/Extraction with Image Processing
Image Convolution

47. DATA SCIENCE
MODEL PLANNING
Prediction Type
• Regression
• Classification
• Clustering
• Association

48. DATA SCIENCE
MODEL BUILDING
How to make a model?
Data (x)
f(x) = x^2
xi = (3, 5, 2, 1) yi = (9, 25, 4, 1)
Known Formula
E = mc^2
m = mass
c = speed of light
Energy
Known Formula
f(x) = ?
xi = (1, 3, 5, 7) yi = (1,3,5,7)
Unknown Formula
machine learning have an objective to find the formula
Classified Target (y)
f(x)
Blackbox
Formula
(model)

49. DATA SCIENCE
MODEL BUILDING
How to make a model?
Learning Process, Find Minimum Error with iterating method (comparing target data with the current model prediction)

50. DATA SCIENCE
MODEL BUILDING
Model Characteristic in Real-World Problem
- No Exact Formula
- Always have an error
- We never know what formula in math equation
Real World Problem

51. DATA SCIENCE
MODEL BUILDING
Minimum Error Trap

52. DATA SCIENCE
MODEL BUILDING
Over-fitting and Under-fitting
• Generalization is the key
• We need consider trade-off bias and error
• Model with the 100% accuracy need to be questioned. (it must be over-fitting)
• Sometime we should to stop learning process in local minima not global minima to avoid over-fitting

53. DATA SCIENCE
MODEL BUILDING
A sample formula for learning process
(Formula to find formula)

54. DATA SCIENCE
MODEL BUILDING
Data Splitting
• Split Data into training set and test set
• Using training set to get accuracy
• Using test set to get validation
• Data Split Strategy
• Hold-out (70:30, 80:20)
• Cross-validation (K-Fold)
Evaluation
• Confusion Matrix
• Accuracy
• Precision
• Recall
• Specificity
• F1 Score

55. DATA SCIENCE
MODEL BUILDING
What classifier type to separated the data?
• Linear?
• Non-Linear?
B
A

56. DATA SCIENCE
MODEL BUILDING
How Linear/Logistic Regression find best model
Open file
5-1_LinearRegression_Diabetes
5-2_LogisticRegression_Iris
5-3_LogisticRegression_PCA_Cifar10

57. DATA SCIENCE
MODEL BUILDING
Artificial Neural Network (ANN)
Neural Network
Artificial NN

58. DATA SCIENCE
MODEL BUILDING
Artificial Neural Network (ANN)
Perceptron Network
Able to create linear classifier

59. DATA SCIENCE
MODEL BUILDING
Artificial Neural Network (ANN)
Multilayer Perceptron Network
Able to create non-linear classifier
Easy to over-fitting
Create bias neuron to make model not overfit

60. DATA SCIENCE
MODEL BUILDING
How ANN Learn?
Back-propagate error
Using difference target with current prediction then consider it to update weight
Doing it iteratively until the error descend
Open file
5-4_ANN_PCA_Cifar10

61. DATA SCIENCE
MODEL BUILDING
Convolution
Neural Network

62. DATA SCIENCE
MODEL BUILDING
Deep Learning
Different Activation Function
More layer (Deep)
Different Backpropagation Algorithm

63. DATA SCIENCE
MODEL BUILDING
Activation Function and
Backpropagation algorithm
comparation

64. DATA SCIENCE
MODEL BUILDING
Deep Convolution Neural Network
Open file
5-4_DCNN_Cifar10

65. DATA SCIENCE
MODEL BUILDING
Deep Convolution Neural Network
Standardized DCNN Architecture for Object Recognition
ResNet (Residual Network) by AlexNet (Team
name) win the ImageNet 2012 competition
VGG b Simonyan and Zisserman

66. DATA SCIENCE
MODEL BUILDING

67. DATA SCIENCE
MODEL BUILDING
Deep Convolution Neural Network
Standarized DCNN Architecture for Object Recognition
Inception Architecture by Google

68. DATA SCIENCE
MODEL BUILDING
Deep Convolution Neural Network
Standardized DCNN Architecture for Object Recognition

69. DATA SCIENCE
MODEL BUILDING
Regional-Convolution Neural Network

70. COMPUTER VISION WITH DEEP LEARNING
IMPLEMENTATION
3D (Object) Regional-Convolution Neural Network for Self-Driving Car

71. COMPUTER VISION WITH DEEP LEARNING
IMPLEMENTATION

72. EDGE COMPUTING FOR AIOT