DRL & Robotics on ROS

1. Machine Learning & Robotics
DeepReinforcementLearning+ROS
Tommy Wu,
WCB IoT Solution Architect,
Microsoft

2. Today’s Machine Learning Categories

3. How to Play Games with Deep RL
An artificial agent for general Atari
game playing
• Learn to master 49 different
Atari games directly from
game screens
• Beat the best performing
learner from the same
domain in 43 games
• Excel human expert in 29
games

4. Let’s Play the Game with Reinforcement Learning
Single agent that plays both Space Invaders and Breakout on an ‘above human’ level
D:a9reinforcementLearningatari>python atari.py -g Breakout -r 'True'

5. What is the Reinforcement Learning
Create an agent that is capable of learning through trial
and error and ultimately solving the problem.
cart-pole system
Cartpole - The goal is to keep the cartpole
balanced by applying appropriate forces to
a pivot point.

6. Reinforcement Learning Key Factors
• RL is a general concept that can be simply described with
an agent that takes actions in an environment in order to
maximize its cumulative reward.
• Agents in RL algorithms are incentivized with punishments
for bad actions and rewards for good ones.

7. Deep Q-Learning (DQN)
• DQN is a RL technique that is aimed at choosing the best action
for given circumstances (observation). Each possible action for
each possible observation has its Q value, where ‘Q’ stands for
a quality of a given move.
• SARS (state, action, reward, state_next, terminal) and perform
Experience Replay.
• dqn_solver.remember(state, action, reward, state_next, terminal)
• dqn_solver.experience_replay()

8. OpenAI Gym toolkit for Reinforcement Leanring
• OpenAI Gym is a toolkit for developing and comparing reinforcement
learning algorithms.
• Gym open-source library - https://github.com/openai/gym
• A collection of environments to develop and test RL algorithms. It’s built on
a Markov chain model that is illustrated below.
Markov Chain

9. Simple Sample for DQN – Path On Right.
o-----T

10. Robotics ARM manipulation
Move a box by pushing it until it reaches the
desired goal in Simulation.
My experiment to
pickup the medicine
box

11. Robotics with Reinforcement Learning
The Robot Operating System (ROS) is a set of software libraries
and tools that help you build robot applications. From drivers to
state-of-the-art algorithms, and with powerful developer tools,
ROS has what you need for your next robotics project. And it's
all open source.

12. talker.py listener.cpp
ROS Node
ROS Core
/chatter
snoop.cpp
subscribe subscribe
publish
ROS Node ROS Node
Topic

13. Workspace Structure -

14. ROS packages for Autonomous System
• Gazebo - They provide the necessary interfaces
to simulate a robot in Gazebo using ROS
messages, services and dynamic reconfigure
• SLAM - The gmapping package provides laser-based
SLAM (Simultaneous Localization and Mapping), as a
ROS node. To create a 2-D occupancy grid map
• RVIZ - Rviz is a 3D visualizer for the Robot Operating
System (ROS) framework

15. Motion Detector with Camera in ROS
roslaunch motion_detector motion_detection.launch
rosrun rviz rviz

16. SLAM Navigation with Image Sensing (2D/3D)

17. Robotics Autonomous System Architecture on ROS – AI Care II
listen
robotcv
ROSCore
publish
/inference_objects
publish
/STT
Iot-central-app
listen
azure_cs_luis
(Speech)
/cmd_vel
Azure Custom
Vision
Action
Sets
Win10 ROS Melodic