This document presents a proposed approach to automate organizing construction tools on a job site using a Fetch robot and image processing. The robot will take an image of the originally disorganized tools as input and use computer vision techniques to detect and relocate the tools to match the organized layout in the input image. The plan is to use ROS for robot control, OpenCV for image recognition, and OpenAI tools to train the robot to determine the optimal order for organizing objects. The overall goals are for the robot to be able to recognize surfaces, objects, and their locations in an image; grasp tools using its manipulator arm; and rearrange them to achieve the organized final state.

1. NEAT; A Fetch Robot for Organizing the
Construction job site
Hyeonae Jang
Computer and Information Technology
Purdue University
West Lafayette, IN, USA
jang145@purdue.edu
Oscar Wong Chong
Construction Management Technology
Purdue University
West Layayette, IN, USA
owongcho@purdue.edu
Mahdi Afkhamiaghda
Construction Management Technology
Purdue University
West Layayette, IN, USA
mafkhami@purdue.edu
Abstract—While the use of automation and robotics has been
increasing in the construction industry, there is a lack of research
on how these robots can help with tools organization in a
construction site. This paper presents an application specifically
designed to automate organizing construction tools and equip-
ment, and removing unwanted objects on a flat surface in a
construction site using a fetch robot and image processing. In
this study, the robot will sort a unorganized surface given the
original state image of the tools as input.
Index Terms—Construction, fetch Robot, Image Recognition
I. INTRODUCTION
Construction tools are necessary to any construction project
because it allows the workers to perform their designated
job on a daily basis. Due to the unstructured environment of
a construction job site. Many of these tools are often mis-
placed and/or unorganized. On average, each worker spends
90 minutes per day in a construction job site looking for
different documents, plans, job information, misplaced tools,
etc. Materials needs to be stored and organized so that they
would not pose any safety hazard to workers and employees
[1]. Reducing these types of hazards in construction sites
means lowering liabilities from injury and creating a more
accessible environment for everyone on site. A more organized
construction site is often a more productive one [2].
II. LITERATURE REVIEWS
A. Paper
According to numerous studies, the technical approach
for dealing with this issue is tracking the equipment using
technologies such as RFID tags [3], ultra wideband [4], and
GPS tracking [5]. However, these approaches while considered
costly, since all equipment and materials needs to be tagged
only shows the current position of equipment and does not help
with sorting the object. Decreasing or eliminating this time
period can increase work efficiency and eventually, save cost.
As [6] have discussed, while not having the power of industrial
robots to perform big manufacturing tasks, collaborative robots
can eliminate some of the current manual and mundane
construction tasks through an automated and relatively cheap
operation.
B. Fetch
Fetch Robot is autonomous mobile robot platform that has
a seven degree of freedom (DOF) manipulator arm, created
by Fetch Robotics, Inc [7] and it was designed to perform
tasks in human workplace. Although many robotics technology
are commercially available such as Mobile manipulator RB-
1 [8], Baxter [9], Ridgeback [10], amongst others, Fetch
Robot was selected for this experiment because of its mobility,
reachability (wide range of movements), and capability to
relocate objects.
III. PROPOSED APPROACH
The main idea of this project is to use Fetch research robot
to automatically organize construction tools on a flat surface
by providing the original state of the tool as input. Figure
1 illustrates an example of a situation where a messy desk
(before) is organized with stationeries (after) just as shown in
the given image(after).
Fig. 1. An example of situation of the project [11]
There is a high possibility of scaling up the project that
increases the importance of this project. Considering time
limitation, however, we decided to narrow down the scope
of the project having two main goals. The first goal of this
project is to detect objects to be reorganized based on a given
image. The second goal is to relocate the objects by a gripping
function of Fetch robot. To be more specific, the relocation
job includes placing objects shown in the given image on the
right location and removing unwanted substances which are

2. not shown in the given image. The following text describes
the software platforms required for the project.
A. ROS
The Robot Operating System (ROS) is a collection of tools,
libraries, and conventions that make up a flexible framework
for writing robot software [12]. The Fetch software system is
designed to work with ROS. All Fetch capabilities are thus
available via ROS messages, services or actions.
B. OpenCV
The OpenCV (Open Source Computer Vision Library) is
a library mainly used for computer vision, image processing,
and machine learning [13]. OpenCV enables Fetch robot to
have skills to recognizing flat surfaces and objects. The ability
to recognize flat surfaces allows Fetch robot to detect places
where objects are. This is the first step to searching for objects.
The ability to recognize different objects in the scene and
localize where they are from Fetch robot.
C. OpenAI
OpenAI is a non-profit artificial intelligence research or-
ganization. OpenAI has recently released simulated robotics
environments and used the environments to train models
which work on physical robots including Fetch Robot. One
of the expected difficulties of this project is the complexity
of training Fetch robot to decide the order of organizing
objects. We are planning to try the environments to allow
Fetch robot to understand and make a sequence of jobs making
use of OpenAI Gym, a toolkit for developing and comparing
reinforcement learning algorithms [14].
Once the core algorithm for relocating object is completed,
the overall scenario of this project could potentially include
the following goals by doing extended researches:
• Wireless communication between a user and Fetch to
assign jobs
• Localization to find the correct spot to organize
• Flat surfaces/object recognition
• Grasp and relocate objects
• Notification of job completion to the user
One other expected challenge is the dependency on the shape,
size, and texture of objects. This may cause the failure of
grasping objects. We are planning to conduct an experiment
on this. To figure out a possible limitation and breakthrough.
IV. PLAN
Before and during the spring break, we will be spending
most of our time on getting familiar with new software tools
as well as Fetch robot. We then are going to design a detailed
system of our program after considering feasibility of each
goal. Since we aims to implement our program based on a
physical body of Fetch robot not only in the simulator, we will
be keep developing and improving a prototype of the program
testing step by step for a couple of weeks. Finally, we will
finalize our project by analyzing and discussing the results.
Figure 2 illustrates an overall plan for the project. Before and
during the spring break, we will be spending most of our time
on getting familiar with new software tools as well as Fetch
robot. We then are going to design a detailed system of our
program after considering feasibility of each goal. Since we
aims to implement our program based on a physical body
of Fetch robot not only in the simulator, we will be keep
developing and improving a prototype of the program testing
step by step for a couple of weeks. Finally, we will finalize
our project by analyzing and discussing the results.
Fig. 2. Overall Plan for the project
—————————————————-
REFERENCES
[1] USBR.(October 2009).Standards for Material Han-
dling, Storage, and Disposal Received from
https://www.usbr.gov/ssle/safety/RSHS/sec11.pdf
[2] Panhandle Cleaning Restoration (April 14th, 2014—) Cleaning, Con-
struction . Received from https://www.panhandlecr.com/the-importance-
of-keeping-construction-sites-orderly/
[3] Team, Purdue ECT, ”Spot-R by Triax” (2018). ECT Fact Sheets. Paper
236. http://dx.doi.org/10.5703/1288284316632.
[4] Cheng, T., Venugopal, M., Teizer, J., Vela, P. A. (2011). Performance
evaluation of ultra wideband technology for construction resource loca-
tion tracking in harsh environments. Automation in Construction, 20(8),
1173-1184.
[5] Pradhananga, N., Teizer, J. (2013). Automatic spatio-temporal analysis
of construction site equipment operations using GPS data. Automation
in Construction, 29, 107-122. capitalized,” J. Name Stand. Abbrev., in
press.
[6] Afsari, K., Gupta, S., Afkhamiaghda, M., Lu, Z. Applications of
Collaborative Industrial Robots in Building Construction.
[7] Fetch Robotics.(n.d.). Retrieved March 1, 2019, from
https://fetchrobotics.com.
[8] Mobile manipulator RB-1.(n.d.). Retrieved March 1, 2019, from
https://www.robotnik.eu.
[9] Baxter.(n.d.). Retrieved March 1, 2019, from
https://www.rethinkrobotics.com.
[10] Ridgeback.(n.d.). Retrieved March 1, 2019, from
https://www.clearpathrobotics.com.
[11] The messy subject of clean desk policies (n.d.). Retrieved February 28,
2019, from https://paramountinteriors.com/blog.
[12] Robot Operating System Wikipedia. (n.d.). Retrieved February 28, 2019,
from https://en.wikipedia.org/wiki.
[13] Image Recognition and Object Detection SATYA
MALLICK Learnopencv.(n.d.). Retrieved February 28, 2019,
https://www.learnopencv.com.
[14] Ingredients for Robotics Research Openai.(n.d.). Retrieved February 28,
2019, from https://blog.openai.com.