This document summarizes an approach for adapting a collaborative robot (cobot) workstation to human operators with different skills using a deep learning camera. The approach uses a camera and deep learning to recognize human operators and access their profile, which contains data like preferred speed. This profile is then used to adapt the robot's speed and behavior to the specific operator. The approach was implemented using an ABB YuMi cobot, Amazon DeepLens camera, AWS cloud services for facial recognition and data storage, and tested for accuracy and speed of recognition and adaptation. The results showed over 100% accuracy in operator recognition with average adaptation times of 12 seconds.

1. An approach for adapting a Cobot
Workstation to Human Operator
within a Deep Learning Camera
Contact Information
Tampere University
Engineering and Natural Sciences Faculty
Future Automation Systems and Technologies
Laboratory (FAST-Lab.)
P.O. Box 600,
FIN-33014 Tampere
Finland
Email: fast@tuni.fi
research.tuni.fi/fast
Conference Information
Conference: 17th IEEE International Conference
on Industrial Informatics (INDIN’19) 22-25 July
2019, Helsinki-Espoo, Finland
Title of the paper: Implementing a Human-Robot
Collaborative Assembly Workstation
Authors: Olatz De Miguel Lazaro, Wael M.
Mohammed, Borja Ramis Ferrer, Ronal Bejarano,
Jose L. Martinez Lastra
If you would like to recieve a reprint of the original paper, please contact us

2. An approach for adapting a Cobot
Workstation to Human Operator within
a Deep Learning Camera
17th IEEE International Conference on Industrial Informatics
INDIN 2019
22-25 July 2019, Helsinki-Espoo, Finland
Olatz De Miguel Lazaro, Wael M. Mohammed, Borja Ramis Ferrer, Ronal Bejarano,
Jose L. Martinez Lastra

3. Outline
• Introduction
• Objective
• Approach
• Implementation
• Experiment
• Conclusion
24.05.2019 3

4. Introduction
Collaborative
robots
Workers have
different
characteristics
Adaptation of
HRC
24.05.2019 4

5. Objective
Propose the adaptation of robots to the skills
of human operators in order to implement an
efficient, safe and comfortable synergy
between robots and humans working at the
same workspace.
Cobot image: http://www.manufacturinglounge.com/deciphering_industry_4_part2/
24.05.2019 5

6. Approach
Computer Vision  Deep Learning
Use a camera to recognize the human
operator that collaborates with the robot. The
user profile is processed and serves as an
input to a module in charge of adapting
specific features of the robot. In this manner,
the robot can adapt its speed according to the
skills of the user or stopping the process at
any time that the worked leaves unexpectedly
the workstation
24.05.2019 6

7. Approach
24.05.2019 7

8. Implementation
• ABB’s collaborative robot YuMi
• Amazon’s DeepLens video camera and development platform
o Integrated in AWS Cloud
• To cover the camera: 3D printed head
o Adapted from open source robot POPPY
Image: https://aws.amazon.com/deeplens/
24.05.2019 8

9. Implementation
AWS Lambda
Serverless computing service that runs code
and interacts with other AWS services
S3 Bucket
Cloud storage services.
Stores face images from video feed
AWS Rekognition
Facial recognition cloud service. Extracts
facial data from images and compares it to
faces in collection
DynamoDB
Database service.
Stores names of operators and data like
speed, height etc
Simple
Notification
Service (SNS)
Messaging service.
Sends message with operator’s data from
AWS Cloud to gateway
AWS Cloud
24.05.2019 9

10. Implementation
24.05.2019 10
REST
(POST)
WEBSocket
Web Server

11. Experiment
• Times (from face detection to movement of YuMi)
o Average: 12s
• Accuracy:
o Recognition of operators 100% accurate
• Limitation:
o AWS server sometimes malfunctions or exits Lambda before is finished
24.05.2019 11

12. Conclusion
• HRC requires adaptation of the process to the human operator
• DL provides a fast tool for adapting the tasks without changing the cobot’s code
• With adaptation  operations tailored to the operator
24.05.2019 12

13. Follow us
www.facebook.com/fast.laboratory
www.twitter.com/FAST_Lab
www.youtube.com/fastlaboratory
www.slideshare.net/fastlaboratory