Thesis DEVELOPMENT OF AUTOMATIC TEACHING METHOD USING STEREO CAMERA FOR SCARA ROBOTS

1. VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY
HO CHI MINH UNIVERSITY OF TECHNOLOGY
FACULTY OF ELECTRICAL AND ELECTRONICS ENGINEERING
DEPT OF CONTROL ENGINEERING AND AUTOMATION
GRADUATION ESSAY
DEVELOPMENT OF AUTOMATIC TEACHING METHOD
USING STEREO CAMERA FOR SCARA ROBOTS
SVTH:
Phạm Phước Dũng 1710875
GVHD:
TS. Nguyễn Hoàng Giáp
26/12/2022

2. Content
Introduction
01
Hardware
System
Design
02
Software
Design
03
Experiments
and
Evaluation
04
Conclusion
and
Development
direction
05
26/12/2022 2

3. Introduction
System overview
01
26/12/2022 3

4. 01 Introduction
Reasons for choosing
the topic
Thesis objectives System overview
26/12/2022 4

5. 01 Introduction
1.1. Reasons for choosing the topic
26/12/2022 5

6. 1.1. Reasons for choosing the topic
26/12/2022 6
 The teaching-less method for robots is being strongly developed in the world because
of the advantages compared to the traditional teaching ones.
 The successful development of this topic will increase the diversity of robot applications
by solving problems that cannot be solved by traditional teaching methods.
Remote robot-assisted surgery
Robot welding

7. 01 Introduction
1.2. Thesis objectives
26/12/2022 7

8. 1.2. Thesis objectives
 Develop a teaching-less method using stereo camera for SCARA robot to perform the
pick-and-place feature with high precision.
 Build a camera - robot system which can operate based on designed algorithms and
principles.
26/12/2022 8
Pick-and-place feature with high precision Force Torque Sensor required

9. 01 Introduction
1.3. System overview
26/12/2022 9

10. 1.3. System overview
 Principle of system operation
26/12/2022 10

11. 1.3. System overview
 Hardware system structure
26/12/2022 11

12. 1.3. System overview
 Principle of component operation
26/12/2022 12

13. Hardware System Design
Design and build hardware system
02
26/12/2022 13

14. 02 Hardware System Design
Marker
 Passive marker
Stereo camera
 Stereo camera
mounted from 2 Basler
Dart daA1600-60um
 Module emits IR
 IR Filter
Hệ robot SCARA
 Robot SCARA Epson
EC251S
 DTP7H-coreCon robot
controller
26/12/2022 14

15. 02 Hardware System Design
2.1. Marker
26/12/2022 15

16. 2.1. Marker
26/12/2022 16
Robot Marker
 Passive marker
Setpoint Marker
Robot Marker Setpoint Marker
Length (mm) 60 98.2
Width (mm) 60 87
Height (mm) 75 22.7
Marker’s size

17. 02 Hardware System Design
2.2. Stereo camera
26/12/2022 17

18. 2.2. Stereo camera
Stereo camera mounted from 2 Basler Dart daA1600-60um
 Resolution: 2x(640x480) pixel.
 Max Frame: 45FPS.
 Accuracy: 0.25mm RMS
 Operating distance: 500mm to 1500mm
26/12/2022 18

19. 02 Hardware System Design
2.3. Robot SCARA
26/12/2022 19

20. 2.3. Robot SCARA
Robot SCARA EC251S DTP7H-coreCon robot controller
26/12/2022 20

21. Software Design
Build algorithms and principles for model
03
26/12/2022 21

22. 03 Software Design
Algorithm design
 Thread for position calculating
 Thread for end-effector's angle
updating
 Thread for communication
between PC and robot controller
 Convert distance
User interface design
 Main program
 Error evaluation experimental
program
Communicate with PC
using robot controller
 Setup communication protocol
on robot controller
26/12/2022 22

23. 03 Software Design
3.1. Algorithm design
26/12/2022 23

24. 3.1. Algorithm design
 Thread for position calculating
26/12/2022 24
𝛥𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 = (𝑥𝑠𝑒𝑡𝑝𝑜𝑖𝑛𝑡 − 𝑥𝑟𝑜𝑏𝑜𝑡)2+ (𝑦𝑠𝑒𝑡𝑝𝑜𝑖𝑛𝑡 − 𝑦𝑟𝑜𝑏𝑜𝑡)2+ (𝑧𝑠𝑒𝑡𝑝𝑜𝑖𝑛𝑡 − 𝑧𝑟𝑜𝑏𝑜𝑡)2

25. 3.1. Algorithm design
 Thread for end-effector's angle updating
26/12/2022 25

26. 3.1. Algorithm design
 Thread for end-effector's angle updating
26/12/2022 26
Rotation matrix obtained from Homogeneous matrix𝑐𝑎𝑚𝑒𝑟𝑎
𝑟𝑜𝑏𝑜𝑡𝑇𝑋𝑌𝑍 :
𝑐𝑎𝑚𝑒𝑟𝑎
𝑟𝑜𝑏𝑜𝑡
𝑅𝑋𝑌𝑍(, , ) =
𝑟11 𝑟12 𝑟13
𝑟21 𝑟22 𝑟23
𝑟31 𝑟23 𝑟33
So the pitch angle is:
 = arctan(−𝑟31, 𝑟32
2
+ 𝑟33
2
) (rad)
Calculation of rotation angle difference between two markers:
𝛥𝐴𝑛𝑔𝑙𝑒 =
180 ∗ (𝐻𝑜𝑚𝑒− 𝑆𝑒𝑡𝑝𝑜𝑖𝑛𝑡)
𝜋
(degree)

27. 3.1. Algorithm design
 Thread for communication between PC and robot controller
26/12/2022 27

28. 3.1. Algorithm design
 Thread for communication between PC and robot controller
26/12/2022 28
Time delay for robot moving to Setpoint position:
∆𝑇𝑀𝑜𝑣𝑖𝑛𝑔 𝑚𝑠 =
∆𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑚𝑚
𝑆𝑝𝑒𝑒𝑑
100
∗ 𝑅𝑜𝑏𝑜𝑡 𝑚𝑎𝑥𝑖𝑚𝑢𝑚 𝑚𝑜𝑣𝑖𝑛𝑔 𝑠𝑝𝑒𝑒𝑑
𝑚𝑚
𝑠
∗ 1000 + 500
- Robot maximum moving speed is set in robot controller to 1000 (mm/s).
- Speed ​​entered by the user, default will be 10 (%).
- Since the robot has time to accelerate and decelerate in each movement and this value cannot
be interfered with or measured, so in this project, I will use 500 (ms) to characterize this time
value.

29. 3.1. Algorithm design
 Convert distance using equivalent coordinate system
26/12/2022 29
Incremental value xRobot = - Incremental value xCamera
Incremental value yRobot = - Incremental value zCamera
Incremental value zRobot = Incremental value yCamera

30. 03 Software Design
3.2. User interface design
26/12/2022 30

31. 3.2. User interface design
 Main program
Module Tracker Module Pick Place App
26/12/2022 31

32. 3.2. User interface design
 Error evaluation experimental program
26/12/2022 32

33. 03 Software Design
3.3. Communicate with PC using robot controller
26/12/2022 33

34. 3.3. Communicate with PC using robot controller
 Config IP Address and Port on robot controller
26/12/2022 34
Programing on robot controller
Config IP Address

35. Experiments and Evaluation
Experimental results to verify the operability of model
04
26/12/2022 35

36. 04 Experiments and Evaluation
26/12/2022 36
Setup camera
Setup robot and experimental platform

37. 04 Experiments and Evaluation
26/12/2022 37
Setup experimental environment

38. 04 Experiments and Evaluation
Determining
repeatability
Checking
distance
correlation
Comparing
feedback
position
Evaluating the
operability of
model
26/12/2022 38

39. 04 Experiments and Evaluation
4.1. Determining repeatability
26/12/2022 39

40. 4.1. Determining repeatability
Mica panel with holes cut by laser
26/12/2022 40
Marker fit perfectly inside mica panel

41. 4.1. Determining repeatability
Marker’s position unchanged
26/12/2022 41
Capture marker’s position using software

42. 4.1. Determining repeatability
Deviation from the standard position (mm) Repeatability
(mm)
Min Max
1 0.012 0.088 0.045
2 0.016 0.273 0.078
3 0.022 0.116 0.051
Experiment results
26/12/2022 42

43. 04 Experiments and Evaluation
4.2. Checking distance correlation
26/12/2022 43

44. 4.2. Checking distance correlation
26/12/2022 44
Fixed marker to robot end-effector Get end-effector’s position
from robot controller Get marker’s position
from software

45. 4.2. Checking distance correlation
Experiment results
Distance difference between 2 points
determined by robot controller
𝒅𝒓 (mm)
Distance difference between 2
points determined by software
𝒅𝒄 (mm)
Error
(mm)
Mean Error
(mm)
RMS Error
(mm)
1 21.723 21.647 0.076
0.156 0.195
2 11.099 11.134 0.035
3 37.729 38.056 0.327
4 15.519 15.501 0.017
5 17.402 17.413 0.011
6 39.562 39.457 0.105
7 33.119 32.774 0.345
8 24.685 24.418 0.268
9 40.316 40.450 0.134
10 35.143 34.942 0.200
11 21.504 21.506 0.001
12 30.915 31.158 0.242
13 25.841 25.630 0.210
14 23.920 23.616 0.304
15 11.234 11.297 0.063
26/12/2022 45

46. 04 Experiments and Evaluation
4.3. Comparing feedback position
26/12/2022 46

47. 4.3. Comparing feedback position
Compare Setpoint’s position sent and Robot’s position received
26/12/2022 47
Graphing and calculating errors

48. 4.3. Comparing feedback position
Experiment results
 With 4 Setpoint unchanged position
26/12/2022 48

49. 4.3. Comparing feedback position
Experiment results
 With 4 Setpoint unchanged position
26/12/2022 49

50. 4.3. Comparing feedback position
Experiment results
 With 4 Setpoint unchanged position
26/12/2022 50

51. 4.3. Comparing feedback position
Experiment results
 With 4 Setpoint unchanged position
26/12/2022 51

52. 04 Experiments and Evaluation
4.4. Evaluating the operability of model
26/12/2022 52

53. 4.4. Evaluating the operability of model
 Video to demo the operability of model
26/12/2022 53

54. Conclusion and Development direction
Evaluate the completeness and feasibility of the model
05
26/12/2022 54

55. 05 Conclusion and Development direction
Conclusion Development direction
26/12/2022 55

56. 05 Conclusion and Development direction
5.1. Conclusion
26/12/2022 56

57. 5.1. Conclusion
 Achievements
 Complete the algorithm to calculate the displacement distance and update the rotation
angle of SCARA robot.
 Complete the system with Master and Slave stations to perform the pick-and-place
feature with SCARA robot.
 Complete the communication protocol between PC and SCARA robot controller.
 Complete user interface.
 Complete experiments to verify the operability of model.
26/12/2022 57

58. 5.1. Conclusion
 Difficulties
 Due to limitation on the marker’s surface which can reflect IR (3 out of 4), in case
SCARA robot rotate Robot Marker with the blind surface (the surface have no reflecting
point) toward camera, sorfware will unable to calculate and update the end-effector’s
rotation angle.
 Haven’t optimized the synchronization solution between the robot's moving time and
the camera’s capture time to update the robot's rotation angle.
26/12/2022 58

59. 05 Conclusion and Development direction
5.2. Development direction
26/12/2022 59

60. 5.2. Development direction
 Optimize the synchronization solution between the robot's moving time and the
camera’s capture time.
 Replace the other marker with no blind surface.
 Upgrade the robot system (controller, driver, robot) to minimize the mechanical error.
 Build a response system from Slave to Master to inform the current status of robot
(position, velocity, rotation,…).
 Build a controller mounted directly to camera in order to reduce system’s size.
26/12/2022 60

61. Thank you so much
for your interest and attention!

62. Principle of marker recognition
26/12/2022 62
3D reconstruction of
balls in DRB
: Comparing geometry consists of balls in
DRB
-> Center position, distance between
closer balls, etc.
Rom file data of
balls in DRB
(DRB)