BTech thesis presentation of Aritra Sarkar at Indian Institute of Space Science and Technology

1. Elevation Mapping using Stereo Vision via
Heterogeneous Multi-agent Cloud Network
- B. Tech Project -
Aritra Sarkar
SC090B009
Siddharth Srivastava
SC09B129
Department of Avionics
Indian Institute of Space Science and Technology

2. Project Motivation
• Multi-agent system is the future of robotics
• Planetary explorations are significant in upcoming ISRO
missions
• The Internet is shifting to the cloud computing paradigm
• Emphasis on vision based sensors for augmented virtual
reality
• Flexibility offered in using electronics prototyping platform
for robotics
• This project has many applications in civilian emergency
response and technical operations

3. Swarm Robotics
• Rover organization types:
• Single rover system
• Centralized swarm system
• Leader-based swarm system
• Decentralized swarm system
• Metrics to evaluate future trends
• Design VfM
• Manufacturing VfM
• Hardware Robustness
• Software Robustness
• Scalability

4. Space Robotics
• Metrics for space applications
• Payload weight
• Testing VfM
• Exploration speed
• Energy tapping
Minimize weight ensuring robustness of a distributed system

5. Planetary Explorations
• Planetary exploration missions are upcoming Indian Space
Mission
1. “The mission to Mars will be launched on November, 2013
on-board PSLV-XL from Sriharikota with 5 payloads”
2. Future ISRO missions to have Rovers
3. NASA evaluating Swarm Rovers for its future missions

6. Cloud Computing
• The internet is shifting to the Cloud Computing paradigm
• Network access
• Resource pooling
• Scalability
• Metered service
• Services offered
• SaaS
• PaaS
• IaaS

7. Vision Sensors
• Emphasis on Vision based sensors for Augmented Virtual
Reality
1. Common Sensors in Robotics - IR, Ultrasounds, LDRs,
Accelerometers, Gyroscopes – gives values
2. Superiority of Image Sensor – Colour, Shape, Size,
Proximity, Motion tracking, Illumination
3. Requires intelligent processing and memory
4. Image sensor in a Swarm system – stereo vision, multi
direction vision
5. Most developed Augmented Reality sensor

8. Related Works
• Aerial photography for terrain mapping
• Multi-view 3D reconstruction of objects
• Visual odometry and SLAM for vehicular navigation
• Octree for 3D reconstruction of depth map
• Distributed multi-agent system designs
• LiDaR based navigation in robotics
• Cloud computing into Vehicular Ad-hoc Networks (VANET)
Key contribution of the Design
A multi-agent stereo vision enabled robotic network
designed for future planetary exploration missions

9. Project Attributes & Work Division
• Digital Elevation Mapping
• Stereovision and Camera Calibration
• Visual Odometric Tracking
• Heterogeneous Centralized Swarm
• Private Cloud Communication Setup
• Robotic (Embedded) Computer Network
• Hazard Detection and Control

10. Project Timeline
• January
• Literature survey
• Detailed project planning
• Hardware components procurement
• February
• Hardware assembly
• Software detailed planning
• Testing of small software modules
• March
• Exploring algorithm modifications
• Testing on KITTI dataset
• Camera calibration
• April
• Dataset in IIST campus and D4 corridor
• DEM and Odometric results

11. Basic System Design
• 2 Scout Rovers
• Stereo-camera rig and Wireless Device
• Cloud Interaction and All-terrain Movement
• Sends stereo images to skipper rover
• Execute movement command issued by the skipper
• 1 Skipper Rover
• Heavy-duty robot
• Builds 3D elevation maps from images sent by scout rovers
• Decides movement of itself and the Scout robots
• Serves as a Cloud storing the elevation model
and……
Virtual Reality System, Object Manipulation, Material
Characterization etc..

12. Program Flow

13. Program Flow

15. LEGO-CAD Model

17. Hardware
• Fiber glass sheet
• Motor holders
• Lego mount
• Nut bolts
• Wheels
• DC Motors
• Motor Drivers
• Arduino Duemilanove
• Alix 3d3 Wireless Routers
• Webcams
• USB Hub
• Laptop
Software
• Arduino IDE (Processing
language based)
• Voyage Linux
• OpenStack
• Mplayer
• FFmpeg
• Ubuntu 12.04LTS
• MATLAB 7.14.0.739 (R2012a)
Components

18. Control Module
• Arduino Development Platform
• Free-Duino (Indian low-cost Arduino clone) or Arduino Duemilanove “2009”
• ATmega 168 or ATmega 328, 16 MHz clock speed
• 14 digital I/O pins, 6 analog inputs
• 16KB flash memory
• On board programmer
• Processing based Arduino IDE
The open-source Arduino environment used to write code and upload it to the board. The
environment is written in Java, based mainly on Processing and avr-gcc

19. Sensor Module
• Web Camera INTEX IT-305WC with Night Vision
• Specifications:
Image Sensor : 1/7” High Quality CMOS sensor
Image Control : Saturation, Brightness, Contrast, Hue, Sharpness, Gamma, Edge Ration
Image Resolution : 640×480, 352×288, 320×240, 176×144, 160×120
Power Frequency : 50Hz, 60Hz
Exposure : Auto or Manual
Angle of View : 54 degree
Image Format : RGB24, 1420
Interface : USB2.0
Anti-Flicker : 50Hz, 60Hz or Outdoor
Frame Rate : Up to 30 fps
Power Consumption : 160mW typical
Focus Range : 4cm to infinity
Lens : Professional camera less with 16.0 Mega Pixels, F=2.0

20. Stereoscopic computer vision
• Stereopsis is the impression of depth that is perceived
with both eyes by binocular vision.
• Computer stereo vision is the extraction of 3D information
from digital images. By comparing information about a
scene from two views, 3D information can be extracted
from the relative positions of objects in the two panels.

21. Basics of Stereovision
• Simultaneous Localization and Mapping
• Unknown terrain, obstacle detection
• Epipolar Geometry
• Disparity Map and Occlusions
• Parallel camera Rig
• Wide baseline stereo mapping

22. Triangulation
• Image coordinates to World coordinates
Z = ( b * f ) / ( x1 - x2 )
X = x1 * Z / f
Y = y1 * Z / f
• b is the baseline of the camera
• d = (x1-x2) is the disparity

23. Baseline resolution
• Wider baseline – higher resolution
• Narrow baseline – higher stability

24. Odometric Techniques
• Geographical Position System
• Needs satellite system coverage
• Accuracy in centimeters
• Dead Reckoning - INS
• Error accumulates
• Should remain on always
• Encoders in wheels
• Wheel slip cannot be detected
• LiDaR – Light Detection and Ranging
• High power consumption
• Limited range

25. Visual Odometry
• Disparity Map at time ‘t’ and ‘t+1’
• Feature detection (find points) – Moravec or Harris Corner
• Feature matching (make correspondences) – SSD or SAD
• Feature tracking (remove outliners) – RANSAC or ModalBin
• Find pitch, roll, yaw

26. RANSAC vs. ModalBin
• RANSAC advantage - robust estimation for high outliners
• RANSAC disadvantage - no upper bound on the time
• Not suitable for our real-time constrain
• Histogram techniques for YPR used
• ModalBin method suggested

27. Camera setup
• Bot1 : Region Pixel Bounds in 320x240
• Region 1 = 94  Per 1 pixel
• Region 2 = 40  Per 2 pixel
• Region 3 = 25  Per 3 pixel
• Region 4 = 14  Per 4 pixel
• Region 5 = 09  Per 5 pixel

28. Camera calibration

29. Camera calibration

30. Camera calibration

31. Edge detection
• Sobel Filter
• Variance Filter

32. Calibration results

33. Calibration results

34. Rectifying Images
• Tree to sky recolouring
• Grayscale conversion
• Radial distortion correction
• Brightness and contrast equalization
• Rotation handling
• Aspect ratio, image size and resolution preservation
GB histogram
matching and
recolouring
Grayscale
conversion
Radial
distortion
correction
LR histogram
equalization
Rotation
correction
Aspect ratio
and Size
preservation
Rectified
stereo images
Stereo images
from camera

35. Tree to Sky Recolouring

36. Grayscale conversion
• Lightness method:
(max(R, G, B) + min(R, G, B)) / 2
• Average method:
(R + G + B) / 3
• Luminosity method:
0.2990 R + 0.5870 G + 0.1140 B (Rec 601 format for SDTV)
0.2126 R + 0.7152 G + 0.0722 B (Rec 709 format for HDTV)

37. Radial distortion correction
• Pinhole effect
• Barrel/Cushion effect
• 3rd order polynomial equation:
ax2 + bx + c
• [a b c] = [+0.5 +0.002 +0.000000]
• [a b c] = [+1.0 +0.000 - 0.000005]

38. Brightness and contrast equalization

39. Rotation, Aspect ratio and Size preservation
Aspect Ratio: 4:4
Aspect Ratio: 5:3Greyscale; Aspect Ratio: 4:3Rotation: +5 Rotation: -5
Pinhole Cushion
Original Image

40. Results – Disparity Map
Template Matching – MATLAB inbuilt Vision module – 27s Sobel Filter and Block Matching - MATLAB OpenCV – 0.01s

41. Coordinate Systems
• Landing Coordinate ( Global)
• Beacon Coordinate
• Local Coordinate
Move the Scouts
• Serial command (F, B, R, L, H) + time (in ms)
• Scout command: MATLAB -> Ubuntu Terminal -> Voyage
Terminal -> Arduino -> Motor Driver pins
• Skipper command: MATLAB -> Arduino -> Motor Driver
pins

42. Map Representation
• 2D occupancy grid
• 2.5D grid map
• 3D voxels vs. polygons
• Polygons useful when feature density varies
• Voxels more precise but high memory requirements
• Specifying material parameter of objects

43. Database
• Octree – tree with 8 children
• Actual data for query stored in leaf nodes
• 3D voxel data storage

44. The Cloud Structure
• OLSR based Private Cloud Network
• Terrain database stored in the Cloud
• All scouts has access to it
• Enhances scalable nature of the project

45. Swarm functions
• Disjoint mapping for faster exploration
• Collision avoidance
• Exploration buffer zone
• Hazard detection and recovery
• Navigation Hazard
• Communication Hazard

46. Communication hazard detection and recovery
• Hazard detection and recovery
• Random motion – unreliable for small swarm
• Spiral motion - 45% recovery rate
• Backtracking - 84% recovery rate

47. Results on
KITTI Dataset
• Match accuracy: avg. 200 points/step
• 1424 Time-step images with Ground-truth data
• Has GPS/INS, Laser sensor, Stereo-rig
Odometric Path after 50 step
i) DEM ii) Harris corner correspondence iii) Weighted shift in image plane iv) Shift in mapping plane (all for step 15)

48. Results in IIST
D4 Corridor 1st Floor
• 1000 image sequence
• Run-time: 5 minutes
• Colour images for Odometry
Harris corners and feature matching

49. Results in IIST
D4 Corridor 2nd Floor
• 277 image sequence
• Run-time: 4.5 minutes
• Colour images for Odometry
• Camera calibrated setup
• Markers placed for corners

50. Results in IIST
D4 Corridor 2nd Floor
• Ground Truth Length: 220 meters
• Odometric Length: 216 meters

51. Results in IIST
Outdoor environment
• 120 image sequence
• Run-time: 4 minutes
• Colour images for DEM
• Tree removal
• Contrast equalization
• Rugged terrain, Gaussian
smoothing

52. Results in IIST
Outdoor environment
• Ground Truth Length: 22 meters
• Odometric Length: 21.5 meters

53. Key contributions
• Modal Bin method in RT system
• Merging OLSR and Cloud system
• Indoor results odometry – 0.028% error
• Outdoor result odometry – 0.023% error
• Digital Elevation Map created successfully
• Rovers traversed in DEM without human intervention
• Multiple rover hazard scenario recovered - 84% accuracy
• Real time system with 5 second time step
• Overall exploration speed of 370 sq. meter per hour

54. Limitations
• Stereo setup not perfectly rigid – LEGO structures
• Camera FOVs and contrasts not identical
• Camera lens not perfectly spherical causing corner blurs at focus
• Wheels and DC Motor not suitable for all terrain and slope > 35 deg
• Blind region of 15cm in-front of rover
• Limited battery, no renewable recharge mechanism

55. Future scope
• Stereo cameras like Bumblebee from Point Grey
• Servo motors and Rhex all terrain wheels
• Local obstacle detection and avoidance with sensors
• Solar panel deployment
• Explore other camera setups like Catadioptric stereo imaging and
Triangular stereo cameras
• Graphical User Interface for the program
• Raspberry Pi microcomputer based system – ARM + Python instead
of AVR + MATLAB
• Develop skipper robot as portable space lab with more actuators
• Haptic control and virtual reality in skipper robot for space and other
applications

56. Other Application Domains
• Space Labs and Terraforming
• Civilian applications
• Fire fighting robots
• Unmanned mining operation
• Assistant system for differently-abled
• Military applications
• De-bombing operations
• Unmanned aircrafts/tanks

57. Acknowledgements
Our Project Guide:
• Dr. B.S. Manoj
Robotics Laboratory faculty in-charge:
• Dr. Deepak Mishra
• Dr. N. Selvaganesan
For helping us out in making the rover chassis:
• Manufacturing Lab IIST
For collaboration and result comparison:
• Mr. Abhishek Patil – Avionics – IIST
• Mr. William Baker - Luna Rovers – University of Vermont

59. References
[1] Gerhard Wei, “Adaptation and Learning in Multi-Agent Systems: Some Remarks and a
Bibliography” - Lecture Notes in Artificial Intelligence 1042 - Subseries of Lecture Notes in
Computer Science - “Adaptation and Learning in Multi-Agent Systems” - Proceeding of IJCAI’95
Workshop, August 1995
[2] Roger Y. Tsai, “A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision
Metrology Using Off-the-Shelf TV Cameras and Lenses”, IEEE Journal of Robotics and
Automation, Vol RA-3, No. 4, August 1987
[3] Ali, K.S., Vanelli, C.A., Biesiadecki, J.J., Maimone, M.W., Yang Cheng, San Martin, A.M.,
and Alexander, J.W. Oct. 2005,”Attitude and position estimation on the Mars exploration
rovers”, IEEE International Conference on Systems, Man and Cybernetics, 12-12, Oct. 2005,
California.
[4] P. Chris Leger, Whiteys Trebi-Ollennu, John R. Wright, Scott A. Maxwell, Robert G.
Bonitz,Jeffrey J. Biesiadecki, Frank R. Hartman, Brian K. Cooper, Eric T. Baumgartner, and Mark
W. Maimone, “Mars Exploration Rover Surface Operations: Driving Spirit at Gusev Crater”, Jet
Propulsion Laboratory, California Institute of Technology Pasadena, CA, USA
[5] Cheng, Y., Maimone, M.W. and Matthias, L. 2006,”Visual odometry on the Mars exploration
rovers”, IEEE Robotics & Automation Magazine, June, 2006.
[6] Kurt Konolige, Motilal Agrawal, and Joan Sol`a, “Large Scale Visual Odometry for Rough
Terrain”, SRI International 333 Ravenswood Ave Menlo Park, CA 94025

60. References
[7] Sergio A. Rodríguez F, Vincent Frémont, Philippe Bonnifait, “An Experiment of a 3D Real-Time
Robust Visual Odometry for Intelligent Vehicles”, Université de Technologie de Compiègne (UTC),
2CNRS Heudiasyc UMR 6599, France.
[8] Chris Harris & Mike Stephens, “A combined corner and edge detector”, Plessey Research
Roke Manor, United Kingdom,
[9] Hartley, R., and A. Zisserman, 2003, “Multiple View Geometry in Computer Vision”.
Cambridge University Press, Cambridge, UK, 2003.
[10] Andrea Fusiello, November, 1998, “Three dimensional vision for structure and motion
estimation”, Department of Mathematics & Informatics, University of Udine, Italy
[11] Arne Henrichsen, December 2000, “3D Reconstruction and Camera calibration from 2D
Images”, Master thesis at University of Cape Town 2000
[12] Hoga D. Setiawan, Mochamad Subchan, and Agus Budiyono - “Virtual Reality Simulation of
Fire Fighting Robot Dynamic and Motion” - Agus Budiyono, Bambang Riyanto, Endra Joelianto
(Eds.) - Intelligent Unmanned System : Theory and Applications - Studies in Computational
Intelligence 192 - Springer
[13] Amir A.F.Nassiraei and Kazuo Ishii - “How Does “Intelligent Mechanical Design Concept”
Help Us to Enhance Robot’s Function?” - Agus Budiyono, Bambang Riyanto, Endra Joelianto
(Eds.) - Intelligent Unmanned System: Theory and Applications - Studies in Computational
Intelligence 192 - Springer
[14] Johnny Holmström and Daniel Romero, “A Survey of Robotic Swarms”, Mälardalen
University, Västerås Sweden.

61. References
[15] Li Zhang, “Automatic Digital Surface Model (DSM) Generation from Linear Array Images”,
Swiss Federal Institute of Technology Zurich
[16] Allan Eisenman, Carl Christian Liebe, Mark W. Maimone, Mark A. Schwochert, Reg G.
Willson, “Mars exploration rover engineering cameras”, Jet Propulsion Laboratory, California
Institute of Technology, Pasadena, California
[17] Ivo Ihrke, “Digital elevation mapping using stereoscopic vision” – Royal Institute of
Technology, M. Tech 2001 thesis
[18] Jeffrey Biesiadecki, “The Athena SDM Rover - a Testbed for Mars Rover Mobility“, Mmark W.
Maimone, JPL
[19] Kurt Konolige, “Small Vision Systems: Hardware and Implementation”, Artificial Intelligence
Center, SRI International
[20] Don Murray Jim Little, “Using real-time stereo vision for mobile robot navigation”,
Computer Science Dept., University of British Columbia
[21] Florin Oniga and Sergiu Nedevschi, “Processing Dense Stereo Data Using Elevation Maps:
Road Surface, Traffic Isle and Obstacle Detection”, Technical University of Cluj–Napoca, Romania
[22] Ping-yuan Cui, Fu-zhan Yue, Hu-tao Cui, “Stereo vision based motion estimation for lunar
rover navigation”, Deep Space Exploration Research Center, Harbin Institute of Technology,
Harbin 150080, China
[23] Davide Scaramuzza and Friedrich Fraundorfer, “Visual Odometry”, IEEE Robotics &
Automation Magazine, December 2011

62. References
[24] www.swarmanoid.org
[25] www.k-team.com/mobile-robotics-products/khepera-iii
[26] https://decibel.ni.com/content/docs/DOC-13031
[27] www.lego.com/en-us/default.aspx
[28] www.arduino.cc/en/Main/arduinoBoardDuemilanove
[29] http://kiwi.cs.dal.ca/~dparks/CornerDetection/moravec.htm
[30] http://vision.ucsd.edu/~vrabaud/
[31] http://asrl.utias.utoronto.ca/datasets/3dmap/
[32] http://www.dis.uniroma1.it/~iocchi/stereo/triang.html
[33] http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example.html
[34] http://www.csse.uwa.edu.au/~pk/Research/MatlabFns/#robust
[35] http://www.dpreview.com/glossary/camera-system/sensor-sizes
[36] http://vasc.ri.cmu.edu/idb/html/stereo/index.html
[37] http://stereo.visionbib.com/
[38] http://www.electricstuff.co.uk/picvsavr.html
[39] http://cs.stanford.edu/group/roadrunner//old/index.html
[40] Karlsruhe Dataset: Stereo Video Sequences + rough GPS Poses
Dataset: 2009_09_08_drive_0010.zip
http://www.cvlibs.net/datasets/karlsruhe_sequences.html
[41] ATmega328 Datasheet
[42] Michael Margolis - “Arduino Cookbook”