2020aim

1. National Taiwan University
Graduate Institute of Electrical Engineering
Modular ROS Based Autonomous Mobile
Industrial Robot System for Automated Intelligent
Manufacturing Applications
Ren C. Luo, Shang Lun Lee, Yu Cheng Wen, and Chin Hao Hsu
Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan
Presenter: Shang Lun Lee
2020 IEEE/ASME International Conference on Advanced Intelligent Mechatronics
(AIM). Boston, USA. July 9, 2020
1

2. National Taiwan University
Graduate Institute of Electrical Engineering
 Background
 Objective
 Our Robotic Platform
 System Architecture
 SLAM & Navigation
 Manipulation
 Object Detection
 Grasp Pose Estimation
 Experiment
 Demonstration
 Conclusion
Outline
2

3. National Taiwan University
Graduate Institute of Electrical Engineering
Background (1/2)
3
 Mobile manipulator is a growing robotic machine such as service robot and AMR+cobot
 Small platform with 2~5kg arm payload v.s. large platform with 5~kg arm payload
 With mobility + dexterity + perception, they can do various tasks
-------- Service robot --------
Pr2 Pepper RB-1 Aeolus
-------- AMR + cobot--------
Robotnik+UR Omron+TM Clearpath+UR Kuka

4. National Taiwan University
Graduate Institute of Electrical Engineering
Background (2/2)
4
 Multiple modules need to be integrated and managed
 A flexible pipeline need to be constructed to complete specific scenario
Mobile robot Manipulator Sensors
Gripper Computers
Simultaneous Localization and Mapping (SLAM) Navigation
Manipulation
Grasp Pose Estimation
Object Detection
Text To Speech (TTS)
Automatic Speech Recognition (ASR)
Mobile Manipulator

5. National Taiwan University
Graduate Institute of Electrical Engineering
 Construct a system which can integrate and manage several functional modules
 Implement it into a consumer electronics packaging scenario with a general mobile
manipulation pipeline
Objective
5
Products
Components
Ｘ
Ｙ
Ｚ
Packaging Section
x
USB
Tripod
camera
Router
LiDAR
Camera
1.Grasping
2.Placing
3.Fetching
4.Stacking
Box
…….

6. National Taiwan University
Graduate Institute of Electrical Engineering
 We designed and developed an Autonomous Mobile Industrial Robot (AMIR)
composed of differential-drive AMR, 6 DOF industrial robot and 2 finger gripper
 With d435 RGB-D camera, 9DoF Razor IMU and two URG-04LX laser scanners
 The whole system is based on the Robot operating system (ROS) environment.
Our Robotic Platform
6

7. National Taiwan University
Graduate Institute of Electrical Engineering
System Architecture
7
 Our task-level architecture uses command central state machine for managing states
in a general mobile manipulation scenario based on SMACH
 The user command can be parsed to collecting and delivering
 Each function module is implemented into ROS node

8. National Taiwan University
Graduate Institute of Electrical Engineering 8

9. National Taiwan University
Graduate Institute of Electrical Engineering
SLAM & Navigation
9
 Once it gets command, the sub-state will switch between navigation and one of action
sub-states, and finally return to command state
 We use cartographer SLAM [W. Hess et al., 2016] for mapping and localization
 We use move_base for navigation
 Timed Elastic Band[C. Rösmann et al., 2017] as a local planner
 Dijkstra as a global planner
[Courtesy: http://wiki.ros.org/move_base]

10. National Taiwan University
Graduate Institute of Electrical Engineering
 We use Moveit! for motion planning of arm
 The general framework of the manipulation tasks is
composed of six sub-states
 PoseAttacking: move the camera above the
workspace to take a picture
 ObjectDetection: find the required object’s
bounding box and class
 PoseEstimation: estimate the grasping/stacking
pose of required object
 PickAndPlace: pick up the target object and place
to the destination
Manipulation
10

11. National Taiwan University
Graduate Institute of Electrical Engineering
Object Detection
11
 4 Classes: USB, Tripod, Box, Camera
 We use MobileNet-SSD [A. G. Howard et at., 2017][W. Liu et al., 2016]
for object detection implemented in TensorFlow
 It has less parameters and takes less time to
inference, while the performance can be maintained
[Courtesy: W. Liu et al., 2016]
Depthwise and pointwise
separable conv

12. National Taiwan University
Graduate Institute of Electrical Engineering
Grasp Pose Estimation
12
 We perform haf_grasping [D. Fischinger et al., 2015] for
point cloud based grasp pose estimation
 The filter is applied to segment the point cloud
into single object
cropped point cloud object point cloud
object grasp pose
haf_grasping

13. National Taiwan University
Graduate Institute of Electrical Engineering
 Background
 Objective
 System Architecture
 SLAM & Navigation
 Manipulation
 Object Detection
 Grasp Pose Estimation
 Experiment
 Demonstration
 Conclusion
Outline
13

14. National Taiwan University
Graduate Institute of Electrical Engineering
Experiment1 – Navigation (1/6)
14
 Our AMIR is 0.8m width and 1.2m length
 Autonomous navigating from A point in 304 room to B point in 302 room
 Avoiding obstacles and going through narrow door under prebuilt floor
map in 4 different cases.
2D occupancy grid map

15. National Taiwan University
Graduate Institute of Electrical Engineering
Experiment1 – Navigation (2/6)
15
 Case1: Empty corridor

16. National Taiwan University
Graduate Institute of Electrical Engineering
Experiment1 – Navigation (3/6)
16
 Case2: Corridor with a few obstacles

17. National Taiwan University
Graduate Institute of Electrical Engineering
Experiment1 – Navigation (4/6)
17
 Case3: Corridor with a few obstacles in another arrangement

18. National Taiwan University
Graduate Institute of Electrical Engineering
Experiment1 – Navigation (5/6)
18
 Case4: Corridor with a lot of obstacles

19. National Taiwan University
Graduate Institute of Electrical Engineering
Experiment1 – Navigation (6/6)
19

20. National Taiwan University
Graduate Institute of Electrical Engineering
Experiment2 - Object Detection (1/2)
20
 Training:
 With pretrained weights on MS COCO dataset
 Train on our dataset 320 images (4 class x 80 images)
 Testing:
 Test on our dataset 80 images (4 class x 20 images)
 If intersection over union (IOU) > 0.9: positive, else: negative
 Average Precision (AP) can be interpreted as an approximated area under
curve (AUC) of the Precision x Recall curve
𝐼𝑂𝑈 =
𝑎𝑟𝑒𝑎 𝑜𝑓 𝑜𝑣𝑒𝑟𝑙𝑎𝑝
𝑎𝑟𝑒𝑎 𝑜𝑓 𝑢𝑛𝑖𝑜𝑛
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =
𝑇𝑃
𝑇𝑃+𝐹𝑃
=
𝑇𝑃
𝑎𝑙𝑙 𝑑𝑒𝑡𝑒𝑐𝑡𝑖𝑜𝑛𝑠
𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑃
𝑇𝑃+𝐹𝑁
=
𝑇𝑃
𝑎𝑙𝑙 𝑔𝑟𝑜𝑢𝑛𝑑 𝑡𝑟𝑢𝑡ℎ

21. National Taiwan University
Graduate Institute of Electrical Engineering
 Computing precision & recall
as the confidence is changed
 𝐴𝑃𝑈𝑆𝐵 = 100.00%
 𝐴𝑃𝐵𝑜𝑥 = 97.61%
 𝐴𝑃𝑇𝑟𝑖𝑝𝑜𝑑 = 80.00%
 𝐴𝑃𝑐𝑎𝑚𝑒𝑟𝑎 = 92.33%
 𝑚𝐴𝑃 = 92.485%
Experiment2 - Object Detection (2/2)
21
There are FPs in low confidence
Confusing between tripod and camera
There are some FNs

22. National Taiwan University
Graduate Institute of Electrical Engineering
 The testing flow includes three stages, camera positioning, object detection and
grasp pose estimation, and object picking and placing
 The testing flow is executed 30 times.
 In the success case, the target can be lifted from the table and transported to the destination. As the grasp is
not robust enough, that is, the grasp pose is estimated at an unstable position, this will be regarded as a semi-
success one. If the grasp fails or the target object drops during the process, the trial is labeled as failure
Experiment3 - Grasping
22

23. National Taiwan University
Graduate Institute of Electrical Engineering
Demonstration
23

24. National Taiwan University
Graduate Institute of Electrical Engineering
 We propose a finite state machine based system to manage robot behaviors using
functional modules in an industrial collecting and delivering scenario.
 Our system integrates SLAM, navigation, manipulation, object detection and
grasping pose estimation together to complete assigned tasks, making the
automation scenario more maintainable and extendable.
 Also, we successfully demonstrate our system on the AMIR based on ROS in the
real world. Each part of system is experimentally examined and obtaining great
results.
Conclusion
24

25. National Taiwan University
Graduate Institute of Electrical Engineering 25
Thank you :)
Q &A