Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
NXTTour: An Open Source Robotic System
Operated over the Internet
João Alves, Sancho Oliveira and Anders Christensen
Depar...
Contents
1 Introduction
2 Related Work
3 System Prototype
4 Preliminary Experiments
5 Conclusion
J. Alves, S. Oliveira, A....
Introduction
Contents
1 Introduction
2 Related Work
3 System Prototype
4 Preliminary Experiments
5 Conclusion
J. Alves, S....
Introduction
Introduction
To demonstrate a low-cost robot that can be tele-operated over
the internet using open source so...
Introduction
Introduction
The work presented in this paper relies exclusively on open
source software.
Our system is a gen...
Related Work
Contents
1 Introduction
2 Related Work
3 System Prototype
4 Preliminary Experiments
5 Conclusion
J. Alves, S....
Related Work
Related Work
The work presented in this paper is related to autonomous, mobile robots and to
tele-operated ro...
System Prototype
Contents
1 Introduction
2 Related Work
3 System Prototype
Overview
Technologies
Communications
The Robot
...
System Prototype Overview
Overview
Several modules
Developed independently
Interfaces and simple communication protocols b...
System Prototype Technologies
Technologies
We resorted to a number of technologies in the design and
development of the sy...
System Prototype Communications
Communications
The three modules communicate with each other through
different means of co...
System Prototype The Robot
The Robot
Actuators:
rear end working in
reverse
Sensors:
Touch sensor
Color sensor
Lleft ultra...
System Prototype The NXTSlave Module
The NXTSlave Module
Runs on the LEGO Mindstorms NXT system
Translates the commands is...
System Prototype The NXTSlave Module
The NXTSlave Module
The logic to prevent collisions with obstacles and to recover
fro...
System Prototype The NXTSlave Module
Mode of operation – SELF
This mode is the Self-preservation mode. The robot depending...
System Prototype The NXTSlave Module
Mode of operation – OPERATOR
As long as there are no dangerous situations, the robot ...
System Prototype The NXTSlave Module
Communication Protocol – NXT2Android
Uses Bluetooth
NXTController receives the status...
System Prototype The NXTController Module
The NXTController Module
Runs on an Android
smartphone
Responsible for receiving...
System Prototype The NXTController Module
Web Server
The NXTController runs a web server based on NANOHTTPd2
enabling the ...
System Prototype The NXTController Module
Dynamic Registration
Given the volatility of IP addresses that are often dynamic...
System Prototype The NXTBrowser Module
The NXTBrowser Module
The NXTBrowser module runs in a browser:
User interface expos...
System Prototype The NXTBrowser Module
The NXTBrowser Module
The NXTBrowser has an interface that provides the operator wi...
System Prototype The NXTBrowser Module
Authentication Process
To prevent unauthorized access to the robot, the operator mu...
System Prototype The NXTBrowser Module
Implementation
User interface is partly based on jQuery3
Followed a design that wou...
Preliminary Experiments
Contents
1 Introduction
2 Related Work
3 System Prototype
4 Preliminary Experiments
NXTBrowser
NXT...
Preliminary Experiments
Preliminary Experiments
A number of preliminary experiments were done to assess the
system prototy...
Preliminary Experiments NXTBrowser
NXTBrowser
Since NXTBrowser runs in a browser and since no standard currently
exists fo...
Preliminary Experiments NXTController
NXTController
Extensive tests were made assess the compatibility of the
NXTControlle...
Preliminary Experiments NXTSlave
NXTSlave
Preliminary functional experiments were preformed to test if the
robot was able ...
Preliminary Experiments NXTSlave
NXTSlave - risk of falling down stairs
If the robot approaches stairs straight on (with a...
Preliminary Experiments NXTSlave
NXTSlave - obstacle detection and avoidance
The robot approached a wall at different angl...
Conclusion
Contents
1 Introduction
2 Related Work
3 System Prototype
4 Preliminary Experiments
5 Conclusion
Future Works
J...
Conclusion
Conclusion
We have presented a low-cost robot based on a LEGO
Mindstoms NXT kit and an Android smartphone that ...
Conclusion
Conclusion
The GUI for the operator is web-based and it is compatible
with commonly used modern browsers.
The i...
Conclusion
Conclusion
We have created a page on GitHub:
https://github.com/jpralves/tourrobot
J. Alves, S. Oliveira, A. Ch...
Conclusion Future Works
Future Works
The code-based for our system prototype is modular and
extensible.
The robot’s low-le...
Conclusion Future Works
Thanks
J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 37 / 37
Conclusion Future Works
D. Bianchini, M. B. Meneses, V. L. d. Marchi, and
J. Dobgenski.
Robô móvel controlado remotamente ...
Conclusion Future Works
Proceedings of the 8th International Joint Conference on
Artificial Intelligence, pages 807–813, 19...
Upcoming SlideShare
Loading in …5
×

NXTTour: An Open Source Robotic System Operated over the Internet

701 views

Published on

The ubiquitous adoption of the internet means that as long as internet connections are available, communication can take place independently of geographic locations. In this paper, we demonstrate how the same can hold true for physical exploration of a remote location: we present a mobile robot that can be controlled over the internet from within a browser. Our robot executes commands issued by a remote operator, but at the same time ensures its own safety by autonomously avoiding obstacles such as walls and stairs. The operator receives a video feed and the readings of other sensors from the robot. Our system is built entirely using open source software, and we have made available all source code for our system and the schematics for the LEGO Mindstorms NXT robot.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

NXTTour: An Open Source Robotic System Operated over the Internet

  1. 1. NXTTour: An Open Source Robotic System Operated over the Internet João Alves, Sancho Oliveira and Anders Christensen Departamento de Ciências e Tecnologias da Informação OSDOC’13, July 11, 2013, Lisbon, Portugal
  2. 2. Contents 1 Introduction 2 Related Work 3 System Prototype 4 Preliminary Experiments 5 Conclusion J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 2 / 37
  3. 3. Introduction Contents 1 Introduction 2 Related Work 3 System Prototype 4 Preliminary Experiments 5 Conclusion J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 3 / 37
  4. 4. Introduction Introduction To demonstrate a low-cost robot that can be tele-operated over the internet using open source software. Differential drive robot based on the LEGO Mindstorms NXT and an Android smartphone Compatible with the most commonly used modern internet browsers Compatible with several versions of Android J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 4 / 37
  5. 5. Introduction Introduction The work presented in this paper relies exclusively on open source software. Our system is a generic prototype of a tele-operated robot Tele-operated robots have many potential uses since they allow users to have a presence, to explore, and sometimes even to perform actions remotely Uses internet as the communication medium, so it can be operated independently of the location of the user J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 5 / 37
  6. 6. Related Work Contents 1 Introduction 2 Related Work 3 System Prototype 4 Preliminary Experiments 5 Conclusion J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 6 / 37
  7. 7. Related Work Related Work The work presented in this paper is related to autonomous, mobile robots and to tele-operated robots. To simplify the design of robotic controllers, Brooks [2] proposed the subsumption architecture The presence of video and mapped information can help in a navigation tasks as Nielsen [6] concludes in his work The design of a controller even for a simple task such as obstacle avoidance can be non-trivial as Grechanovsky et al. [4] concludes in is study The Stanford Cart was presented by Moravec [5] and uses the images from the camera to plan its path and to avoid obstacles identified visually Thrun et al. [7] built a robot for the Smithsonian Museum that was an interactive guide A robot that can be controlled over the internet was presented in Bianchini et al. [1] J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 7 / 37
  8. 8. System Prototype Contents 1 Introduction 2 Related Work 3 System Prototype Overview Technologies Communications The Robot The NXTSlave Module Mode of operation – SELF Mode of operation – OPERATOR Communication Protocol – NXT2Android The NXTController Module Web Server Dynamic Registration The NXTBrowser Module Authentication Process Implementation 4 Preliminary Experiments 5 Conclusion J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 8 / 37
  9. 9. System Prototype Overview Overview Several modules Developed independently Interfaces and simple communication protocols between the modules Extensible code base J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 9 / 37
  10. 10. System Prototype Technologies Technologies We resorted to a number of technologies in the design and development of the system prototype: A Lego NXT Mindstorms kit was used to build the robot The leJOS1 firmware replacement was used for the NXTSlave module The Android Software Development Kit (SDK) was used for the NXTController module Standard technologies such as HTML, javascript and Cascading Style Sheets (CSS) were used for NXTBrowser module 1 http://lejos.sourceforge.net/ J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 10 / 37
  11. 11. System Prototype Communications Communications The three modules communicate with each other through different means of communication and using different protocols, some standard, others implemented specifically for the system prototype. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 11 / 37
  12. 12. System Prototype The Robot The Robot Actuators: rear end working in reverse Sensors: Touch sensor Color sensor Lleft ultrasonic sensor Right ultrasonic sensor The robot, built with the LEGO Mindstorms NXT, is a differential drive robot that uses two motors, each connected to the rear wheels. The robot’s front wheel is a passive caster wheel. The robot has an adaptable support for an Android smartphone. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 12 / 37
  13. 13. System Prototype The NXTSlave Module The NXTSlave Module Runs on the LEGO Mindstorms NXT system Translates the commands issued by the operator into actual movement Collects and sends sensory information back to the operator Self-preservation mode that can subsume control Implements autonomous obstacle avoidance and recovery behaviors Communicates with the control module, NXTController, through Bluetooth. written in Java language and uses the leJOS firmware J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 13 / 37
  14. 14. System Prototype The NXTSlave Module The NXTSlave Module The logic to prevent collisions with obstacles and to recover from dangerous situations such as when the robot is about to drive over an edge is implemented in this module. The robot has two modes of operation: ”SELF” and ”OPERATOR”. The mode determines how the robot is controlled. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 14 / 37
  15. 15. System Prototype The NXTSlave Module Mode of operation – SELF This mode is the Self-preservation mode. The robot depending on the readings of sensors, switches to this mode — SELF mode. When an obstacle is sensed through one of the ultrasonic sensors, the robot stops immediately and checks if the obstacle is temporary or permanent. If the obstacle is permanent, the robot performs an avoidance maneuver: the robot moves straight backward, backward left, or backward right depending on obstacle’s relative location. If the touch sensor is activated, it means that the third passive wheel no longer touches the ground and that the robot is about to drive over an edge. In this situation, the robot performs a movement backward in order to recover and avoid falling. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 15 / 37
  16. 16. System Prototype The NXTSlave Module Mode of operation – OPERATOR As long as there are no dangerous situations, the robot executes the commands sent by the NXTController. The NXTSlave has a set of preconfigured actions (move left, move right, move forward and stop) that can be requested by the NXTController. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 16 / 37
  17. 17. System Prototype The NXTSlave Module Communication Protocol – NXT2Android Uses Bluetooth NXTController receives the status of the sensors from NXTSlave Minimize bandwidth usage Maximize the responsiveness of the system Messages are composed of an header with an ID, followed by a Byte indicating the message type, and then the payload. The information exchanged can be regarding the battery voltage of NXT, sensory readings, the current control mode, and so on. Messages are only sent to the NXTController when values change. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 17 / 37
  18. 18. System Prototype The NXTController Module The NXTController Module Runs on an Android smartphone Responsible for receiving the operator’s commands and sending them to the NXTSlave Captures and transmits video to the NXTBrowser Forwards sensory information and overall system status information User interface for the configuration of all parts of the system Establishes the connections to the other modules J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 18 / 37
  19. 19. System Prototype The NXTController Module Web Server The NXTController runs a web server based on NANOHTTPd2 enabling the module to communicate using the HyperText Transfer Protocol (HTTP) protocol with NXTBrowser module. A fixed uniform resource locator (URL) based on a dynamic registration system was implemented to allow the system to be always accessible. 2 http://elonen.iki.fi/code/nanohttpd/ J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 19 / 37
  20. 20. System Prototype The NXTController Module Dynamic Registration Given the volatility of IP addresses that are often dynamically assigned, we developed a registration system to facilitate a simple connection process. When the NXTController web server starts, it registers its newly assigned IP address on a server with a fixed name (address set in the options as ”address publishing-site”). When the user navigates to the URL of the server with a fixed name (eg. http://jpralves.net/51), the server will automatically redirect the user to the current address of the NXTController. When the web server is disabled, redirection is also disabled. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 20 / 37
  21. 21. System Prototype The NXTBrowser Module The NXTBrowser Module The NXTBrowser module runs in a browser: User interface exposed to the operator Receives context information from the robot Sends commands to it The NXTController module runs an embedded web server to which NXTBrowser connects. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 21 / 37
  22. 22. System Prototype The NXTBrowser Module The NXTBrowser Module The NXTBrowser has an interface that provides the operator with sensory information: (a, b) obstacles proximity (color codes) (c, d) measured distance (e) touch sensor state (f, g) tachometers of motors (c) remote video (h) information about the NXTSlave (j) information about Android (k) Four buttons of command (l) operation mode (m) video feed J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 22 / 37
  23. 23. System Prototype The NXTBrowser Module Authentication Process To prevent unauthorized access to the robot, the operator must type in the correct password before a session can begin. The embedded web server will only allow access to control of the robot in the address that is the result of the SHA-1 hash [3] of the password concatenated with a random key that is sent by it (SALT). J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 23 / 37
  24. 24. System Prototype The NXTBrowser Module Implementation User interface is partly based on jQuery3 Followed a design that would allow the use in all kinds of devices with at least a resolution of 1024 × 600 pixels Has four buttons that allow the operator to interact with the robot Receives JSON messages from the server about state information 3 http://www.jquery.com J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 24 / 37
  25. 25. Preliminary Experiments Contents 1 Introduction 2 Related Work 3 System Prototype 4 Preliminary Experiments NXTBrowser NXTController NXTSlave NXTSlave - risk of falling down stairs NXTSlave - obstacle detection and avoidance 5 Conclusion J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 25 / 37
  26. 26. Preliminary Experiments Preliminary Experiments A number of preliminary experiments were done to assess the system prototype. A different set of experiments was designed for each module. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 26 / 37
  27. 27. Preliminary Experiments NXTBrowser NXTBrowser Since NXTBrowser runs in a browser and since no standard currently exists for streaming video that works across all browsers, the video feed in the system prototype is, in fact, implemented as a sequence of JPEG images captured by the camera of the Android device onboard the robot. Tests were performed with four browsers representing the most important rendering engines: Engine Browser Version Site Video Trident IE 9 Wekbit Chrome 22 Webkit Safari 6 Gecko Firefox 16 All the tested browsers successfully presented the HTML-based user interface and the video feed. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 27 / 37
  28. 28. Preliminary Experiments NXTController NXTController Extensive tests were made assess the compatibility of the NXTController with different Android devices and OS versions. Eight different Android handsets were tested and all of them were able to run the NXTController module successfully. The following table lists the Android devices tested and their specification: Brand / Model Version CPU RAM Screen Weight Google / Nexus S 4.1.2 1 GHz Cortex-A8 512MB 480 × 800 129g Samsung / GT-S6500D 2.3.6 800 MHz Cortex-A5 512MB 320 × 480 105g Samsung / GT-I9070 2.3.6 2× 1 GHz Cortex-A9 768MB 480 × 800 120g HTC / One S 4.0.4 2× 1.5 GHz Krait 1GB 540 × 960 119g TCT / Vodafone Smart II 2.3.7 832 MHz 512MB 320 × 480 120g Huawei / U8510 IDEOS X3 2.3 600 MHz Qualcomm 256MB 320 × 480 104g Huawei / U8815 Ascend G300 4.0.3 1 GHz Cortex-A5 512MB 480 × 800 140g HTC / Desire C 4.0.3 600 MHz Cortex-A5 512MB 320 × 480 100g J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 28 / 37
  29. 29. Preliminary Experiments NXTSlave NXTSlave Preliminary functional experiments were preformed to test if the robot was able to identify dangerous situations, and to test that the NXTSlave module activates the proper programmed behavior in case a dangerous situation is identified. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 29 / 37
  30. 30. Preliminary Experiments NXTSlave NXTSlave - risk of falling down stairs If the robot approaches stairs straight on (with an angle of ≈ 90◦ ) it correctly triggers the autonomous reverse behavior to avoid falling down the stairs. If the angle gets more acute, that is, smaller than 25◦ or greater than 155◦ , the robot often falls because one of the rear wheels falls over the edge before the stairs are detected. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 30 / 37
  31. 31. Preliminary Experiments NXTSlave NXTSlave - obstacle detection and avoidance The robot approached a wall at different angles – between 20◦ and 160◦ . The robot acted as expected when the wall was approached from the left, right or front. During a different set of experiments, we tested the performance of the obstacle detection with objects of various sizes and shapes. We found that robot did not always detect narrow obstacles, such as the leg of a chair. In particular, when approaching a narrow obstacle straight on, the robot often did not detect the obstacle because of the orientation (at ±45◦ with respect to the front of the robot) and the limited resolution of the ultrasonic sensors. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 31 / 37
  32. 32. Conclusion Contents 1 Introduction 2 Related Work 3 System Prototype 4 Preliminary Experiments 5 Conclusion Future Works J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 32 / 37
  33. 33. Conclusion Conclusion We have presented a low-cost robot based on a LEGO Mindstoms NXT kit and an Android smartphone that can be tele-operated from a browser over the internet. The robot accepts commands from the operator, such as turn left, turn right, and move forward. The onboard software constantly monitors the sensors to assess the current situation. Autonomous behaviors are triggered whenever a dangerous situation is identified. A remote operator thus cannot make the robot collide with objects or damage the robot by driving it over an edge. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 33 / 37
  34. 34. Conclusion Conclusion The GUI for the operator is web-based and it is compatible with commonly used modern browsers. The interface shows a real-time video feed from the robot and the current sensor readings. Our system is simple and straightforward to use without prior training. Autonomous behaviors that ensure the safety of the robot and its environment combined with an easy to use interface is essential for tele-operated robots to be adopted in non-technological industries such as the real estate business. We have made extensive use of open source technologies and we have made all source code for the system available as open source software. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 34 / 37
  35. 35. Conclusion Conclusion We have created a page on GitHub: https://github.com/jpralves/tourrobot J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 35 / 37
  36. 36. Conclusion Future Works Future Works The code-based for our system prototype is modular and extensible. The robot’s low-level module, NXTSlave, is currently implemented specifically for LEGO Mindstorms NXT, but it could be ported to a different robotic platform with no changes in the other modules. The use of other platforms, such as Arduino or Raspberry Pi, would, for instance, allow for the construction of more capable robotic systems. In ongoing work, we are adapting the user interface so that the system can be operated from a smartphone. We are also studying autonomous behaviors that let the robot return to a charging station when its battery level drops below a certain threshold. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 36 / 37
  37. 37. Conclusion Future Works Thanks J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 37 / 37
  38. 38. Conclusion Future Works D. Bianchini, M. B. Meneses, V. L. d. Marchi, and J. Dobgenski. Robô móvel controlado remotamente via Web. 7o Congresso Nacional de Iniciação Científica - CONIC-SEMESP 2007, (7):102–117, 2007. R. Brooks. A robust layered control system for a mobile robot. Robotics and Automation, IEEE Journal of, 2(1):14–23, 1986. D. Eastlake and T. Hansen. RFC6234 - US Secure Hash Algorithms (SHA and SHA-based HMAC and HKDF), 2011. E. Grechanovsky and I. S. Pinsker. An algorithm for moving a computer-controlled manipulator while avoiding obstacles. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 37 / 37
  39. 39. Conclusion Future Works Proceedings of the 8th International Joint Conference on Artificial Intelligence, pages 807–813, 1983. H. P. Moravec. The Stanford cart and the CMU rover. Proceedings of the IEEE, 71(7):872–884, 1983. C. W. Nielsen and M. A. Goodrich. Comparing the usefulness of video and map information in navigation tasks. pages 95–101, New York, New York, USA, 2006. ACM, ACM Press. S. Thrun, M. Bennewitz, W. Burgard, A. B. Cremers, F. Dellaert, D. Fox, D. Hahnel, C. Rosenberg, N. Roy, J. Schulte, and D. Schulz. MINERVA: a second-generation museum tour-guide robot. volume 3, pages 1999–2005. IEEE, IEEE, 1999. J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 37 / 37

×