1) AUPAIR is a personal home service robot developed by Seoul National University that won the RoboCup@Home League competition. 2) The RoboCup@Home competition involves various scenarios testing abilities like speech recognition, navigation, and assisting humans. 3) AUPAIR uses perception modules like object detection and activity recognition combined with action modules like following and guiding to solve tasks in the competition scenarios.

1. AUPAIR:
The Personal Home Service Robot that
Won the RoboCup@Home League
Beom-Jin Lee, Jinyoung Choi, Chung-Yeon Lee, Kyung-Wha Par
k, Sungjun Choi, Cheolho Han, Dong-Sig Han,
Christina Baek, Patrick Emaase and Byoung-Tak Zhang
Seoul National University, Seoul, Korea
Biointelligence Lab
Team Leader: Beom-Jin Lee

2. RoboCup

3. RoboCup
• Largest International Robot Competition!

4. Latest RoboCup
• 20th Annual Competition
• Total 5 Leagues
• RoboCupSoccer
• RoboCupIndustrial
• RoboCupRescue
• RoboCup@Home
• RoboCupJunior
• Total 17 Sub-leagues
• More than 3,500 dedicated scientists
• Developers from more than 40 countries

5. Latest RoboCup
• RoboCupSoccer
Middle Size LeagueSmall Size League

6. Latest RoboCup
• RoboCupSoccer
KidSize 40-90cm Four-robot team
TeenSize 80-140cm Three-robot team
AdultSize 130-180cm One-robot team

7. Latest RoboCup
• RoboCupSoccer
Simulation LeagueStandard Platform League

8. Latest RoboCup
• RoboCup@Home
Open Platform League Social Standard Platform League Domestic Standard Platform League

9. Latest RoboCup
• RoboCupIndustrial
RoboCup@WorkLogistics League

10. Latest RoboCup
• RoboCupRescue
Rescue Simulation LeagueRescue Robot League

11. RoboCup@Home

12. RoboCup@Home
• Second largest league in RoboCup
• Initiated the RoboCup@Home competitions in 2005
• Develop service and assistive robot technology for
autonomous service robots
Adaptive
Smart HRI
Wisspeintner, Thomas, et al. "RoboCup@
Home: Scientific competition and
benchmarking for domestic service
robots." Interaction Studies 10.3 (2009):
392-426.

13. RoboCup@Home
• RoboCup@Home Social Standard Platform League (SSPL)

14. RoboCup@Home SSPL Challenges
6 Scenarios + 2 Open Demonstration = 8 Challenges
1. Speech Recognition and Person Recognition
2. Cocktail Party
3. Help Me Carry
4. General Purpose Service Robot (GPSR)
5. Restaurant
6. Tour Guide
7. Enhanced Endurance General Purpose Service Robot (E2GPSR)
8. Open Challenge
9. Final Demonstration

15. RoboCup@Home SSPL Scenarios
Speech Recognition and Person Recognition
1. Play riddle game
2. Play bluff man game
Focus technique
Person, object, speech, activity
recognition
Q&A
Sound localization
12
people

16. RoboCup@Home SSPL Scenarios
Cocktail Party
1. Enter the party room
2. Find 3 standing or sitting guest
3. Go to barman and give order
4. Find missing beverage
5. Correct order
Focus technique
Person, object, speech, activity
recognition
Person description
1
2
3
4
5

17. RoboCup@Home SSPL Scenarios
Help Me Carry
1. Memorize operator and
follow
2. Save car waypoint and find
other person to help him
carry grocery
3. Guide person to car waypoint
Focus technique
Following, online mapping,
guiding
1
2
3

18. RoboCup@Home SSPL Scenarios
General Purpose Service Robot (GPSR)
1. Generate random command
from command generator
2. Solve the task and come back
to starting point
3. Repeat 3 times
Focus technique
Overall skill

19. RoboCup@Home SSPL Scenarios
Restaurant
1. No map
2. Check barman
3. Get order from a person
4. Go to barman and give order
5. Repeat 3-4
Focus technique
Online mapping

20. RoboCup@Home SSPL Scenarios
Tour Guide
1. Go out side
2. Find tourist who needs tour
3. Guide person to bench and
give introduction about
RoboCup
Focus technique
Online mapping
Q&A
1
2
3

21. RoboCup@Home SSPL Scenarios
Enhanced Endurance General Purpose Service Robot (E2GPSR)
Test special skills
1. Three in one (solve three
command within 5 minute)
2. Q&A with GPSR
Focus technique
Navigation
Q&A
Recognition skills

22. RoboCup@Home SSPL Total Solutions
• Person Recognition, Detection, Description, Identification
• Finding, Gender, Characteristics
• Activity Recognition
• Object Detection, Recognition, Distinction
• Navigation, Following, Guiding, SLAM, Collision Avoidance,
Approach
• Sound Localization
• Speech Recognition
• Q&A

23. Vision
Speech
Sensor
Environment
Learning
Online
Waypoint
Object
and
Person
Characteristic
Human
Re-
identification
Object
and
Person
Position
Action
Approach Guide
Reflex Following
Collision
Avoidance
Navigation
Perception
Object
Distinction
Speech
Recognition
Object
Detection
and
Recognition
Human Pose
Estimation
Obstacle
Recognition
Command
Parser
Recognition
Filter
Activity
Recognition
Scene
Description
Sound
Localization

24. RoboCup@Home SSPL
Implemented Modules (Perception)
Human Pose Estimation3Object Detection and Recognition2
[2] Redmon, et al., "YOLO9000: better, faster, stronger." arXiv preprint arXiv:1612.08242 (2016).
[3] Cao, et al. "Realtime multi-person 2d pose estimation using part affinity fields." arXiv preprint arXiv:1611.08050 (2016).

25. RoboCup@Home SSPL
Implemented Modules (Perception)
Human Re-identification4 Scene Description5
[4] Ahmed, et al., "An improved deep learning architecture for person re-identification." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.
[5] Johnson, et al., "Densecap: Fully convolutional localization networks for dense captioning." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.

26. RoboCup@Home SSPL
Implemented Modules (Perception)
Object Distinguishing Activity recognition

27. Object Detection and Recognition Person Re-identification Person Re-identification
RGB based Skeleton Scene Description

28. RoboCup@Home SSPL
Implemented Modules (Action)
Human Follower

29. RoboCup@Home SSPL
Implemented Modules (Action)
Reflex module
• Navigation
• Obstacle
Avoidance

30. RoboCup@Home SSPL
Implemented Modules (Action)
Approach

31. RoboCup@Home SSPL
Implemented Modules (Action)
Guide

32. Reflex Approach
Follow Follow Processing Screen

34. RoboCup@Home
Competitions

35. RoboCup@Home SSPL
Speech Recognition and Person Recognition

36. RoboCup@Home SSPL
Help Me Carry

37. RoboCup@Home SSPL
Cocktail Party

38. RoboCup@Home SSPL
General Purpose Service Robot (GPSR)

39. RoboCup@Home SSPL
Enhanced Endurance General Purpose Service Robot
(E2GPSR)

40. RoboCup@Home SSPL
Restaurant

41. RoboCup@Home SSPL
Tour guide

42. RoboCup@Home SSPL
Open Challenge

43. RoboCup@Home SSPL
Final Demonstration

44. RoboCup@Home
Scores

45. RoboCup@Home SSPL Day 1

46. RoboCup@Home SSPL Day 2

47. RoboCup@Home SSPL Day 3

48. RoboCup@Home SSPL Final!!!
1st Place!!!!!

49. Special Thanks
Kyung-Min Kim Jin-Hwa Kim
Kim, Kyung-Min, et al.
"DeepStory: Video Story
QA by Deep Embedded
Memory Networks." arXiv
preprint arXiv:1707.00836
(2017).)
Kim, Jin-Hwa, et al.
"Multimodal residual
learning for visual
qa." Advances in Neural
Information Processing
Systems. 2016.

50. RoboCup@Home SSPL
Dedicated Team Member

51. Thank You for Listening
Sponsorships

57. END