Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Robot Operating Systen (ROS) on Hadoop (HDFS)

1,034 views

Published on

https://github.com/valtech/ros_hadoop - One PC/NAS can have a hard time with large Rosbag files. Hadoop provides distributed storage and parallel processing. InputFormat allows to process Rosbag files via Hadoop interfaces.

Published in: Technology
  • Be the first to comment

Robot Operating Systen (ROS) on Hadoop (HDFS)

  1. 1. 1 Robot Operating System (ROS) on Hadoop Working with large Rosbag files on HDFS
  2. 2. Working with large Rosbag files One PC/NAS can have a hard time with large Rosbag files Hadoop provides distributed storage and parallel processing Hadoop InputFormat allows to process Rosbag files via Hadoop interfaces: https://github.com/valtech/ros_hadoop 2
  3. 3. Hadoop InputFormat for ROS Benefits + Analyse ROS data with 3rd-party tools like Spark via Hadoop interface + Run parallel jobs on ROS data in Hadoop with MapReduce + Save time because data conversion not needed + Save storage because data duplication not needed 3
  4. 4. Search & Analysis + Hadoop InputFormat and Record Reader for Rosbag + Process Rosbag with Spark, Yarn, MapReduce, Hadoop Streaming API, … + Spark RDD are cached and optimized for analysis Large Rosbag Processing Engine Computer Network Storage Advanced Analytics RDD Record Reader RDD DataFrame, DataSet SQL, Spark APIs NumPy Ros Msg
  5. 5. Training & Evaluation + TensorFlow Record Reader + Protocol Buffers to serialize records + Save time because data conversion not needed + Save storage because data duplication not needed Training Engine Machine Learning Computer Network Storage Record Reader Ros msg Large Rosbag
  6. 6. Re-Simulation & Testing + Use Spark for preprocessing, transformation, cleansing, aggregation, time window selection before publish to ROS topics + Use Re-Simulation framework of choice to subscribe to the ROS topics Engine Re-Simulation with framework of choice Computer Network Storage Ros topic core subscribe publish Large Rosbag
  7. 7. Time Travel fold(left) t fold(right) reduce/ shuffle
  8. 8. Demo: To test locally use the Dockerfile 1. > git clone https://github.com/valtech/ros_hadoop 2. > docker build -t ros_hadoop:latest -f Dockerfile . 3. > docker run -it -v $(pwd):/root/ros_hadoop -p 8888:8888 ros_hadoop + HDFS 2.8.0 + Spark on Yarn 2.2.0 + ROS core Kinetic Kame + Tensorflow 1.0.1 + Keras 2.0.3 + Python 2.7.12 depends on ROS, Scala 2.11, Java 1.8
  9. 9. Hadoop InputFormat for ROS Apache License 2.0 Download https://github.com/valtech/ros_hadoop
  10. 10. thank you Jan Wiegelmann +49 (0)151/14 26 56 44 jan.wiegelmann@valtech.de

×