NXTTour: An Open Source Robotic System Operated over the Internet

NXTTour: An Open Source Robotic System
Operated over the Internet
João Alves, Sancho Oliveira and Anders Christensen
Departamento de Ciências e Tecnologias da Informação
OSDOC’13, July 11, 2013, Lisbon, Portugal

Contents
1 Introduction
2 Related Work
3 System Prototype
4 Preliminary Experiments
5 Conclusion
J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 2 / 37

Introduction
Contents
1 Introduction
2 Related Work
3 System Prototype
5 Conclusion

Introduction
Introduction
To demonstrate a low-cost robot that can be tele-operated over
the internet using open source software.
Differential drive robot based on the LEGO Mindstorms
NXT and an Android smartphone
Compatible with the most commonly used modern internet
browsers
Compatible with several versions of Android

Introduction
Introduction
The work presented in this paper relies exclusively on open
source software.
Our system is a generic prototype of a tele-operated robot
Tele-operated robots have many potential uses since they
allow users to have a presence, to explore, and sometimes
even to perform actions remotely
Uses internet as the communication medium, so it can be
operated independently of the location of the user

Related Work
Contents
1 Introduction
2 Related Work
3 System Prototype
5 Conclusion

Related Work
Related Work
The work presented in this paper is related to autonomous, mobile robots and to
tele-operated robots.
To simplify the design of robotic controllers, Brooks [2] proposed the
subsumption architecture
The presence of video and mapped information can help in a navigation
tasks as Nielsen [6] concludes in his work
The design of a controller even for a simple task such as obstacle avoidance
can be non-trivial as Grechanovsky et al. [4] concludes in is study
The Stanford Cart was presented by Moravec [5] and uses the images from
the camera to plan its path and to avoid obstacles identiﬁed visually
Thrun et al. [7] built a robot for the Smithsonian Museum that was an
interactive guide
A robot that can be controlled over the internet was presented in Bianchini
et al. [1]

System Prototype
Contents
1 Introduction
2 Related Work
3 System Prototype
Overview
Technologies
Communications
The Robot
The NXTSlave Module
Mode of operation – SELF
Mode of operation – OPERATOR
Communication Protocol – NXT2Android
The NXTController Module
Web Server
Dynamic Registration
The NXTBrowser Module
Authentication Process
Implementation
5 Conclusion

System Prototype Overview
Overview
Several modules
Developed independently
Interfaces and simple communication protocols between
the modules
Extensible code base

System Prototype Technologies
Technologies
We resorted to a number of technologies in the design and
development of the system prototype:
A Lego NXT Mindstorms kit was used to build the robot
The leJOS1
ﬁrmware replacement was used for the
NXTSlave module
The Android Software Development Kit (SDK) was used for
the NXTController module
Standard technologies such as HTML, javascript and
Cascading Style Sheets (CSS) were used for NXTBrowser
module
1
http://lejos.sourceforge.net/

System Prototype Communications
Communications
The three modules communicate with each other through
different means of communication and using different protocols,
some standard, others implemented speciﬁcally for the system
prototype.

System Prototype The Robot
The Robot
Actuators:
rear end working in
reverse
Sensors:
Touch sensor
Color sensor
Lleft ultrasonic sensor
Right ultrasonic sensor
The robot, built with the LEGO Mindstorms NXT, is a differential
drive robot that uses two motors, each connected to the rear
wheels. The robot’s front wheel is a passive caster wheel. The
robot has an adaptable support for an Android smartphone.

System Prototype The NXTSlave Module
The NXTSlave Module
Runs on the LEGO Mindstorms NXT system
Translates the commands issued by the operator into actual
movement
Collects and sends sensory information back to the operator
Self-preservation mode that can subsume control
Implements autonomous obstacle avoidance and recovery
behaviors
Communicates with the control module, NXTController,
through Bluetooth.
written in Java language and uses the leJOS ﬁrmware

The NXTSlave Module
The logic to prevent collisions with obstacles and to recover
from dangerous situations such as when the robot is about to
drive over an edge is implemented in this module.
The robot has two modes of operation: ”SELF” and
”OPERATOR”. The mode determines how the robot is
controlled.

Mode of operation – SELF
This mode is the Self-preservation mode. The robot depending on
the readings of sensors, switches to this mode — SELF mode.
When an obstacle is sensed through one of the ultrasonic
sensors, the robot stops immediately and checks if the obstacle
is temporary or permanent.
If the obstacle is permanent, the robot performs an avoidance
maneuver: the robot moves straight backward, backward left,
or backward right depending on obstacle’s relative location.
If the touch sensor is activated, it means that the third passive
wheel no longer touches the ground and that the robot is about
to drive over an edge. In this situation, the robot performs a
movement backward in order to recover and avoid falling.

Mode of operation – OPERATOR
As long as there are no dangerous situations, the robot executes
the commands sent by the NXTController. The NXTSlave has a
set of preconﬁgured actions (move left, move right, move
forward and stop) that can be requested by the NXTController.

Communication Protocol – NXT2Android
Uses Bluetooth
NXTController receives the status of the sensors from
NXTSlave
Minimize bandwidth usage
Maximize the responsiveness of the system
Messages are composed of an header with an ID, followed
by a Byte indicating the message type, and then the
payload.
The information exchanged can be regarding the battery voltage
of NXT, sensory readings, the current control mode, and so on.
Messages are only sent to the NXTController when values
change.

System Prototype The NXTController Module
The NXTController Module
Runs on an Android
smartphone
Responsible for receiving the
operator’s commands and
sending them to the NXTSlave
Captures and transmits video
to the NXTBrowser
Forwards sensory information
and overall system status
information
User interface for the
conﬁguration of all parts of the
system
Establishes the connections to
the other modules

Web Server
The NXTController runs a web server based on NANOHTTPd2
enabling the module to communicate using the HyperText
Transfer Protocol (HTTP) protocol with NXTBrowser module.
A ﬁxed uniform resource locator (URL) based on a dynamic
registration system was implemented to allow the system to be
always accessible.
2
http://elonen.iki.fi/code/nanohttpd/

Dynamic Registration
Given the volatility of IP addresses that are often dynamically
assigned, we developed a registration system to facilitate a
simple connection process. When the NXTController web server
starts, it registers its newly assigned IP address on a server with
a ﬁxed name (address set in the options as ”address
publishing-site”). When the user navigates to the URL of the
server with a ﬁxed name (eg. http://jpralves.net/51),
the server will automatically redirect the user to the current
address of the NXTController. When the web server is disabled,
redirection is also disabled.

System Prototype The NXTBrowser Module
The NXTBrowser module runs in a browser:
User interface exposed to the operator
Receives context information from the robot
Sends commands to it
The NXTController module runs an embedded web server
to which NXTBrowser connects.

The NXTBrowser has an interface that provides the operator with sensory
information:
(a, b) obstacles proximity (color codes)
(c, d) measured distance
(e) touch sensor state
(f, g) tachometers of motors
(c) remote video
(h) information about the NXTSlave
(j) information about Android
(k) Four buttons of command
(l) operation mode
(m) video feed

Authentication Process
To prevent unauthorized access to the robot, the operator must
type in the correct password before a session can begin. The
embedded web server will only allow access to control of the
robot in the address that is the result of the SHA-1 hash [3] of
the password concatenated with a random key that is sent by it
(SALT).

Implementation
User interface is partly based on jQuery3
Followed a design that would allow the use in all kinds of
devices with at least a resolution of 1024 × 600 pixels
Has four buttons that allow the operator to interact with the
robot
Receives JSON messages from the server about state
information
3
http://www.jquery.com

Preliminary Experiments
Contents
1 Introduction
2 Related Work
3 System Prototype
NXTBrowser
NXTController
NXTSlave
NXTSlave - risk of falling down stairs
NXTSlave - obstacle detection and avoidance
5 Conclusion

A number of preliminary experiments were done to assess the
system prototype. A different set of experiments was designed
for each module.

Preliminary Experiments NXTBrowser
NXTBrowser
Since NXTBrowser runs in a browser and since no standard currently
exists for streaming video that works across all browsers, the video
feed in the system prototype is, in fact, implemented as a sequence of
JPEG images captured by the camera of the Android device onboard
the robot.
Tests were performed with four browsers representing the most
important rendering engines:
Engine Browser Version Site Video
Trident IE 9
Wekbit Chrome 22
Webkit Safari 6
Gecko Firefox 16
All the tested browsers successfully presented the HTML-based
user interface and the video feed.

Preliminary Experiments NXTController
NXTController
Extensive tests were made assess the compatibility of the
NXTController with different Android devices and OS versions.
Eight different Android handsets were tested and all of them
were able to run the NXTController module successfully. The
following table lists the Android devices tested and their
speciﬁcation:
Brand / Model Version CPU RAM Screen Weight
Google / Nexus S 4.1.2 1 GHz Cortex-A8 512MB 480 × 800 129g
Samsung / GT-S6500D 2.3.6 800 MHz Cortex-A5 512MB 320 × 480 105g
Samsung / GT-I9070 2.3.6 2× 1 GHz Cortex-A9 768MB 480 × 800 120g
HTC / One S 4.0.4 2× 1.5 GHz Krait 1GB 540 × 960 119g
TCT / Vodafone Smart II 2.3.7 832 MHz 512MB 320 × 480 120g
Huawei / U8510 IDEOS X3 2.3 600 MHz Qualcomm 256MB 320 × 480 104g
Huawei / U8815 Ascend G300 4.0.3 1 GHz Cortex-A5 512MB 480 × 800 140g
HTC / Desire C 4.0.3 600 MHz Cortex-A5 512MB 320 × 480 100g

Preliminary Experiments NXTSlave
NXTSlave
Preliminary functional experiments were preformed to test if the
robot was able to identify dangerous situations, and to test that
the NXTSlave module activates the proper programmed
behavior in case a dangerous situation is identiﬁed.

NXTSlave - risk of falling down stairs
If the robot approaches stairs straight on (with an angle of ≈
90◦
) it correctly triggers the autonomous reverse behavior to
avoid falling down the stairs. If the angle gets more acute, that
is, smaller than 25◦
or greater than 155◦
, the robot often falls
because one of the rear wheels falls over the edge before the
stairs are detected.

NXTSlave - obstacle detection and avoidance
The robot approached a wall at different angles – between
20◦
and 160◦
. The robot acted as expected when the wall was
approached from the left, right or front. During a different set of
experiments, we tested the performance of the obstacle
detection with objects of various sizes and shapes. We found
that robot did not always detect narrow obstacles, such as the
leg of a chair. In particular, when approaching a narrow obstacle
straight on, the robot often did not detect the obstacle because
of the orientation (at ±45◦
with respect to the front of the robot)
and the limited resolution of the ultrasonic sensors.

Conclusion
Contents
1 Introduction
2 Related Work
3 System Prototype
5 Conclusion
Future Works

Conclusion
Conclusion
We have presented a low-cost robot based on a LEGO
Mindstoms NXT kit and an Android smartphone that can be
tele-operated from a browser over the internet.
The robot accepts commands from the operator, such as
turn left, turn right, and move forward.
The onboard software constantly monitors the sensors to
assess the current situation.
Autonomous behaviors are triggered whenever a
dangerous situation is identiﬁed.
A remote operator thus cannot make the robot collide with
objects or damage the robot by driving it over an edge.

Conclusion
Conclusion
The GUI for the operator is web-based and it is compatible
with commonly used modern browsers.
The interface shows a real-time video feed from the robot
and the current sensor readings.
Our system is simple and straightforward to use without
prior training.
Autonomous behaviors that ensure the safety of the robot
and its environment combined with an easy to use interface
is essential for tele-operated robots to be adopted in
non-technological industries such as the real estate
business.
We have made extensive use of open source technologies
and we have made all source code for the system available
as open source software.

Conclusion
Conclusion
We have created a page on GitHub:
https://github.com/jpralves/tourrobot

Conclusion Future Works
Future Works
The code-based for our system prototype is modular and
extensible.
The robot’s low-level module, NXTSlave, is currently
implemented speciﬁcally for LEGO Mindstorms NXT, but it could
be ported to a different robotic platform with no changes in the
other modules. The use of other platforms, such as Arduino or
Raspberry Pi, would, for instance, allow for the construction of
more capable robotic systems.
In ongoing work, we are adapting the user interface so that the
system can be operated from a smartphone. We are also
studying autonomous behaviors that let the robot return to a
charging station when its battery level drops below a certain
threshold.

Thanks

D. Bianchini, M. B. Meneses, V. L. d. Marchi, and
J. Dobgenski.
Robô móvel controlado remotamente via Web.
7o Congresso Nacional de Iniciação Cientíﬁca -
CONIC-SEMESP 2007, (7):102–117, 2007.
R. Brooks.
A robust layered control system for a mobile robot.
Robotics and Automation, IEEE Journal of, 2(1):14–23,
1986.
D. Eastlake and T. Hansen.
RFC6234 - US Secure Hash Algorithms (SHA and
SHA-based HMAC and HKDF), 2011.
E. Grechanovsky and I. S. Pinsker.
An algorithm for moving a computer-controlled manipulator
while avoiding obstacles.

Proceedings of the 8th International Joint Conference on
Artiﬁcial Intelligence, pages 807–813, 1983.
H. P. Moravec.
The Stanford cart and the CMU rover.
Proceedings of the IEEE, 71(7):872–884, 1983.
C. W. Nielsen and M. A. Goodrich.
Comparing the usefulness of video and map information in
navigation tasks.
pages 95–101, New York, New York, USA, 2006. ACM, ACM
Press.
S. Thrun, M. Bennewitz, W. Burgard, A. B. Cremers,
F. Dellaert, D. Fox, D. Hahnel, C. Rosenberg, N. Roy,
J. Schulte, and D. Schulz.
MINERVA: a second-generation museum tour-guide robot.
volume 3, pages 1999–2005. IEEE, IEEE, 1999.

NXTTour: An Open Source Robotic System Operated over the Internet

More Related Content

What's hot

Similar to NXTTour: An Open Source Robotic System Operated over the Internet

More from Joao Alves

Recently uploaded

NXTTour: An Open Source Robotic System Operated over the Internet