A student project aims to build a system that converts visual input into audio signals to help the blind or visually impaired navigate. The project's objectives are to develop a prototype device that uses image processing and computer vision techniques to detect objects and hazards, and converts that visual information into audio cues. The system would integrate technologies like image segmentation, enhancement, and 3D modeling with an acoustic interface to describe a user's surroundings. A Pandaboard single-board computer with OpenCV is used to process images from a webcam in real-time and translate them into audio descriptions for visually impaired users.

1. The THIRD EYE
CORRELATING IMAGE-SOUND-VIDEO
Gunjan Gupta 10BEC112
Romil Shah 10BEC093
Sagar Patel 10BEC072
Mentor : Prof. Sachin Gajjar
1

2. Outline
 Motivation
 Technological Objective
 Present Scenario
 Bionics: Concept and Cyborg
 Our Innovation
 Approach (Pros and Cons)
 Applications
 Algorithms
 References
2

3. Wah Taj !! 3

4. Beautiful… 4

5. Again Nice to See!! 5

6. Oh, This bus is going to LAL DARWAJA 6

7. BUT CAN EVERYBODY SEE THIS
BEAUTY ?
CAN EVERYONE SAY THE BUS
NUMBER BY SEEING IT ?
NO!!
ALL PEOPLE DON’T HAVE SAME EYE-
SIGHT or EYE-VISION !
7

8. WHAT FOR THESE GUYS ?? 8

9. A project to build a system that
converts visual input into audio signals
which may lead to a practical product
to help the blind or visually impaired
navigate
9

10. Technological Objectives
 The main objective is to obtain a device for the whole Visually Impaired
(V.I.) community- totally blind, partially sighted, and people in a situation
of progressive loss of vision.
 To integrate the developments of image segmentation, image
enhancement, 3D Real World perception, audio representation of space
into a unique basic prototype with the purpose of assisting visually
impaired people in mobility in outdoor spaces.
 To develop new ways of interfacing with V. I. people- to develop an
acoustic interface (Ear-plugs for presentation) for the blinds.
 To develop a system able to calculate a person’s head orientation in a
static and dynamic situation.
10

11. Technological Objectives
 To integrate the 3D modelling system and the intelligent system for the
detection of risks and obstacles
 To integrate in a common platform all the data acquired from the
environment from different sources- artificial vision system, 3D Real World
Recognition System.
 To finally develop a prototype which can be commercialised
 During the project a simple device able to make a direct transformation
from the acquisition system into an acoustic representation will be created
as a final product
11

12. Gadgets/Technology for V.I. Person 12
 Brainport Vision Device
 Electric lollipop that allows blind people to 'see'
using their tongue
 Converts images captured by a tiny camera into
a series of electrical tingles, which can be felt on
the tongue
 Nerves then send these messages to the brain,
which turn the tingles back into pictures
 Costs around 6000 Pounds ( 5 Lakh INR)

13. Gadgets/Technology for V.I. Person
 Devices that alert you when a pot of liquid begins to boil
 Gadgets that make a sound when a cup or container you're pouring liquid
into is nearly full
 Knives with an adjustable guide to help you cut slices of even thickness
 Tactile watches and alarm clocks
13

14. Gadgets/Technology for V.I. Person 14

15. Cyborg / Bionics
 An organism “to which exogenous components have been added for the
purpose of adapting to new environments”.
 Recent meaning of the term bionics refers to merging organism and
machine.
 This approach results in a hybrid system combining biological and
engineering parts, which can also be referred as a cybernetic organism
(cyborg).
15

16. Natural-Artificial Feedback 16

17. Neil Harbisson: A Real Cyborg
17

18. What is Image Processing?
 In electrical engineering and computer science, image processing is any form of signal
processing for which the input is an image, such as a photograph or video frame; the
output of image processing may be either an image or, a set of characteristics or
parameters related to the image.
 Most image-processing techniques involve treating the image as a two-dimensional
signal and applying standard signal-processing techniques to it.
 In other words it is basically the transformation of data from a still or video camera
into either a decision or a new representation.
 All such transformations are done for achieving some particular goal. The input data
may be a live video feed, the decision may be that a face has been detected, and a new
representation may be conversion of a color image into a gray scale image.

19. Our Innovation
 Locating everyday items using Image-Video Processing.
 Detection of potential hazards (location and motion of objects, persons,
obstacles).
 Face Detection and Tagging.
 Text to Speech Conversion.
 Color Detection
 Detecting and analyzing in-front objects (living and non-living) and
perform operations on them.
19

20. Approach
 We are using OpenCV with Panda Board for this application.
 Interfacing OpenCV and Panda Board along with webcam to generate real
time images.
 Coding to find distance of objects or any obstacle and provide output as
an audio signal.
 Provide with inbuilt images and search for them. Coding to match the
images and provide the respective name of object as sound signal.
 Advance further to develop for general usage amongst people.
20

21. Advantage & Disadvantage
 Advantage
 Detection of known objects if they are lost.
 Can read newspaper, books, or rather any reading material.
 Can easily detect color and hence can be useful in various scenarios.
 Face Detection and Face tagging mode.(He can easily know person and can
also save new face)
 Most of the software are freeware, so cheap in cost.
 Disadvantage
 A fully-assembled device (Panda Board + Webcam) is to be carried for all the
operations.
 Bulkier in comparison to Mobile.
21

22. Hardware/Software
 The Panda Board is a low-power, low-cost single-board
computer development platform based on the Texas
Instruments OMAP4430 system on a chip (SoC). It is a community
supported development platform.
 The Panda Board ES is a newer version based on the OMAP4460 SoC, with
the CPU and GPU running at higher clock rates. Like its predecessor, it is a
community supported development platform. We are working with Panda
Board ES.
22

23. Pandaboard

24. Pandaboard Overview
 OMAP4 (Cortex-A9) CPU based open development platform.
 OMAP4430 Application processor
 1GB low-power DDR2
 Display HDMI v1.3 Connector (Type A) to drive HD displays, DVI-D Connector,
 3.5" audio in/out and HDMI Audio out
 Full size SD/MMC card
 Built in 802.11 & Bluetooth v2.1+EDR
 On board 10/100 Ethernet
 Expansion: 1xUSB OTG, 2xUSB HS host ports, General purpose expansion
header

25. What OS does Pandaboard support?
 Debian
 Ubuntu
 Android
 Fedora
 Sabayon
 Xubuntu
 Angstrom

26. OpenCV
 OpenCV grew out of an Intel Research initiative to advance CPU-intensive
applications.
 The intent behind OpenCV was to provide a platform that a student can
readily utilize for developing applications, instead of reinventing the basic
functions from scratch.

27. Why use OpenCV?
 It is free.
 It is fast.
 Good documentation in the form of documentation, tutorials, user groups,
forums etc.
 Lot of pre-built functions and algorithms to get a head start.
 There is an active development on interfaces for other languages like
Ruby, Python, Matlab etc.
 Pretty fast.

28. License
 OpenCV is released under a BSD license, it is free for both academic and
commercial use.
 There are only a few algorithms inside it that may have been patented and
those are not free.

29. KINECT

30. Kinect SENSOR
Photo Album
by Gunjan Gupta

31. BLOCK
DIAGRAM
31

32. Face Recognition 32

33. Object Detection
Dynamic (Moving)Stationary
33

34. Detection by Segmentation
34

35. Segmentation Algorithm
Segmentation
Discontinuity
Point Detection Line Detection Edge Detection
Similarity
Thresholding Region Growing
Region splitting
and merging
35

36. Using MATLAB
inbuilt function
Using our own
algorithm, we applied
linear differentiator. It
works fast and is more
efficient in
comparison to
MATLAB INBUILT
ALGO
36EDGE
DETECTION

37. Application
 This project is applied mainly in the commercial field of helping visually
impaired people with lower eyesight or completely BLIND
 Industrial applications can be devised and enhanced like robots and
machineries
 Concept can be used for Autonomous Cars
 For Security, Identifying and Tracking
37

38. Projects Implemented
1. Capturing Image using hand gesture and uploading it directly to Google Drive (Internet needed)
2. Face Detection and Person Identity Detection for running lifetime of prog
3. Optical Character Recognition
4. Cartoonifying Real Time Images
5. Colour Detection and Diff Sound for Diff Colour
6. Ball Detection
7. Head Pose Estimation
8. Augmented Reality based Position Change Tracking
9. Combined Project of OpenGL and OpenCV for Image Operation
10. Number Recognition in real-time
11. Text to Speech Synthesis using C++
12. Kinect with WINDOWS
13. Kinect with Ubuntu
14. Sound Marking

39. References
 A collaborative European project CASBliP, or Cognitive Aid System for
Blind People
 Andhajan Mandal, Ahmedabad, Gujarat, India
 Electric lollipop that allows blind people to 'see' using their tongue-
Brainport Vision Device
 OMAP4460 Pandaboard ES System Reference Manual
 Neil Harbisson-
http://www.ted.com/talks/neil_harbisson_i_listen_to_color.html
 AMTS-www.amts.co.in
39

40. THANK YOU