This document discusses using deep reinforcement learning for real-time assembly planning in robot-based prefabricated construction. It proposes a framework using DRL to help with planning issues like flexibility, uncertainties, and obstacle avoidance. A DRL model is presented using states, actions, and rewards to learn optimal assembly strategies through simulation. Experimental case studies test the approach in simulations of increasing complexity. Results show DQN and DDQN outperform other algorithms, with future work focusing on more realistic simulations and improved scalability.

1. Deep Reinforcement Learning for
Real-Time Assembly Planning in
Robot-Based Prefabricated
Construction
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction
Presenter
Haider Ali
Usama Hassan (399898)

2. INTRODUCTION
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction
● Prefabricated Construction - Efficiency, Cost, Quality and Reduced
Waste.
● Despite intro of construction 4.0 - Traditional methods still dominate on-
site assembly processes.
● Challenges in Real time assembly planning methods:
❖ Difficult to model
❖ Safety and Quality control
❖ Obstacle Free assembly paths
❖ Construction scenario
❖ Uncertainties in assembly process
● Advances in DRL provides a solution to these optimization and control
problems.

3. LITERATURE REVIEW
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction
● BIM and IOT based: Management and monitoring of on-site assembly processes.
● Motion Planning Algorithms: Robot-based assembly process of hospitalization
facilities using prefab components and BIM based prototypes.
● Discrete Event Processing Models: Discrete event processing models integrated with
the BIM based tool to simulate the construction process.
● Geometric Reasoning based: A component-based automated assembly method for
prefabricated buildings
● DRL based Robotic Manipulator Control
● DRL Applications : Games, Transportation, Navigation, Healthcare, Constructions, etc.

4. PROPOSED METHODOLOGY
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction
● The framework of DRL for assembly
planning is depicted here.
● It contributes to the following issues:
❖ Flexible decision making between
on-site planning.
❖ Mitigate scenario complexities,
dynamics and uncertainties.
❖ A set of benchmark for assembly
planning in robot based
prefabricated construction.

5. DRL MODEL FOR ASSEMBLING PLANNING
States and Actions
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction
● Construction environment states represented as a tensor of shape
W * L * d, where W denotes width and L denotes length.
● Each cube's features and status encoded using a 6-tuple of integer values
within the tensor, represented by variable d.
● The tuple includes height of the component on the yard, number of transit
steps of the building component, height of the target position, etc
● Actions defined as operations on a component by a crane
● Six possible directions for executing an action: forward, backward, left,
right, up, and down

6. DRL MODEL FOR ASSEMBLING PLANNING
Rewards
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction
● Designing rewards as a critical factor for learning success and efficiency
● Transforming the time factor into the number of steps for assembly
completion in the simulation environment.
● Minimizing the number of component actions by providing a basic reward
for each step taken by the agent.
● Imposing a threshold on the maximum number of steps for individual
component construction to avoid over-exploration.
● Utilizing a progressive reward mechanism where the agent receives larger
rewards as more components are successfully built.

7. DRL MODEL FOR ASSEMBLING PLANNING
Rewards
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction
● The reward function is summarized where
Rc represents the rewards of a
component, i is the index and t is the total
steps of the component.
● The constant ci represents the serial no.
during assembly process and epsilon is the
step threshold for each component.

8. RECONFIGURABLE SIMULATOR FOR ASSEMBLY
OPERATIONS
Scenario Description
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction
● Prefabricated construction involves assembling prefab components
according to a construction plan, following three main steps: component
transfer, connection node positioning, and component connection.
● Robots are essential in facilitating prefab component assembly by
identifying component states and performing corresponding tasks.
● Component states, including initial, in transit, arrived, and assembled,
represent different stages of the assembly process.
● Simulator design considers safety requirements (avoiding collisions,
maintaining distance) and assembly requirements (order of assembly,
orientation adjustment). Six actions simulate component movement in 3D
space.

9. RECONFIGURABLE SIMULATOR FOR ASSEMBLY
OPERATIONS
Simulator Development
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction
● Developed a new simulation
environment for assembly planning
using DRL.
● Utilizes 3D grids and simplified BIM
model for prefabricated construction.
● Simulates construction tasks,
assembly processes, and evaluates
DRL performance.
● Developed with pygame and
pyOpenGL, using 1x1x1 cubes as
basic units.

10. TRANSFORMING THE SCENES INTO SIMULATION
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction

11. EXPERIMENTAL CASE STUDIES
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction
Four environment simulations were set up ranging from simple to more complex

12. RESULTS AND ANALYSIS
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction
MEAN REWARDS
± STANDARD ERRORS
OVER 15 RANDOM
SEEDS IN THE LAST 100
EPISODES ON ALL
ENVIRONMENTS.
THE BEST RESULTS
ARE IN BOLD

13. RESULTS AND ANALYSIS
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction
Comparison between RL
algorithms in all
environments with
different scenarios

14. CONCLUSION AND FUTURE WORK
Deep Reinforcement Learning for Real-Time Assembly Planning in Robot-Based Prefabricated Construction
● RL Integration: This paper explores the use of RL in construction planning, leveraging its
ability to learn optimal strategies through interaction with a simulation environment.
● Simplified Simulation Environment: A simplified environment is developed to simulate
robot-based construction processes, allowing for the evaluation of RL algorithms and their
performance in planning tasks.
● Performance Benchmarks: Four construction scenarios and environments are
considered, ranging from simple to complex, providing benchmarks to assess the
effectiveness of RL-based controllers. DQN and DDQN algorithms show superior
performance.
● Future Directions: Future work focuses on enhancing realism by incorporating 4D
planning software and real BIM models. Additionally, performance of DQN and DDQN
algorithms will be optimized using a short-sighted lean planner for improved scalability.