Sixth sense technology

SIXTH SENSE TECHNOLOGY
Vinay Sagar S.D
1BM16MCA52
MCA – 3rd Sem.
BMSCE
http://www.pranavmistry.com/projects/sixthsense/

 Introduction
 Components
 Working
 Gesture Detection Algorithm
 Applications
CONTENTS

INTRODUCTION
Sixth Sense is a wearable interface device that builds up physical
world around us with digital information and lets us use natural hand
gestures to interact with that information.
It was developed by Pranav, a PhD student in the Fluid Interfaces
Group at the MIT Media Lab.
Key Points
wearable interface device :- Pendant like mobile wearable device
Physical world around us with digital information :- Ex- Projector
Hand gestures :- Like Zoom-In, Zoom-Out

WHY THE NAME “SIXTH SENSE”
 It will sense your hand gestures.
 It will sense what object is in front of you.
 It will sense what you see.
 It will sense what information you need and project those
information on any surface.

Camera
Projector
Mobile Component
Colored Markers
COMPONENTS
The hardware components are coupled in a pendant like mobile
wearable device.

Camera
 Captures an object in view and tracks the user’s hand gestures
 It sends the data to smart phone
 It acts as a digital eye, connecting you to the world of digital
information

Projector
 The projector projects visual information enabling surfaces and physical
objects to be used as interfaces
 The project itself contains a battery inside, with 3 hours of battery life.
 A tiny LED projector displays data sent from the smart phone on any
surface in view–object, wall, or person.

Smart Phone
A Web-enabled smart phone in the user’s pocket processes the video
data
Other software searches the Web and interprets the hand gestures

Color Markers
It is at the tip of the user’s fingers .
 Marking the user’s fingers with red, yellow, green, and blue tape helps
the webcam recognize gestures
The movements and arrangements of these makers are interpreted into
gestures that act as interaction instructions.

HOW IT WORKS?
Images
Gestures

The plan of action
• Get images from the camera in a loop – Frames
https://www.youtube.com/watch?v=22TCiQ6_yTs
• Convert BGR image to HSV image
• Thresholding the Image (Binary Image)
• Scribbling the position of the Colored Markers
• Comparing the scribbled Image with pre-defined
gesture image

1. Get an image from the camera - Frames
• frame rate is the number of frames or images that are
projected or displayed per second.
• https://www.youtube.com/watch?v=22TCiQ6_yTs

2. We convert the RGB image into an HSV image.
• We'll be using the HSV colour space, instead of the more
common RGB colour space.
• HSV stands for HUE, SATURATION and VALUE.
• value range for HUE, SATURATION and VALUE are
respectively 0-179, 0-255 and 0-255.
• HUE represents the color,
• SATURATION represents the amount to which that respective
color is mixed with white.

• VALUE represents the amount to which that respective
color is mixed with black.
• Here the HUE is unique for that specific color distribution of
that object.
But SATURATION and VALUE may be vary according to the
lighting condition of that environment.
• In HSV, each colour is assigned a particular number (the
Hue- ex- 125).
• This gives us the advantage of having a single number (hue)
for the red color despite multiple shades of red(all the way
from dark redto a bright red).

• We have to supply the HSV Range of the color to be
Threshold
• After Thresholding, the Red will be white(1) and the rest will
be black(0)
3. Thresholding the Image
http://aishack.in/tutorials/tracking-colored-objects-opencv/

4. Scribbling the position of the marker
• This shows the movement of the colored marker.
• After capturing and thresholding of each frames, we will
mark the position of the color marker on a new image
specially created for this purpose.
• Then. We will join the previous position and the current
position with a line.
• This process continues until the color marker goes out of the
screen or a gesture is detected.

Scribbled Image

4. Comparing.
• In this stage we will compare the scribbled image with the
pre-defined gesture image.
• If they are similar, we invoke some function related to that
gesture, like @ symbol to open email application and
project through projector.

APPLICATIONS

Make a call
You can use the Sixth Sense to project
a keypad onto your hand, then use
that virtual keypad to make a call.
Call up a map
With the map application we can call
up the map of our choice and then
use thumbs and index fingers to
navigate the map

Check the time
Draw a circle on your wrist to get a
virtual watch that gives you the
correct time
Create multimedia reading
experiences
Sixth Sense can be programmed to
project related videos onto
newspaper articles you are reading

Drawing application
The drawing application lets the user
draw on any surface by tracking the
fingertip movements of the user’s index
finger
Zooming features
The user can zoom in or zoom out
using intuitive hand movements

Get product information
Sixth Sense uses image recognition
or marker technology to recognize
products we pick up, then feeds us
information on those products
Get book information
The system can project Amazon
ratings on that book, as well as reviews
and other relevant information

Get flight updates
The system will recognize your
boarding pass and let you know
whether your flight is on time and if
gate has changed.

Take pictures
If you fashion your index fingers and
thumbs into a square ("framing"
gesture), the system will snap a photo.
After taking the desired number of
photos, we can project them onto a
surface, and use gestures to sort
through the photos, and organize and
resize them.

REFERENCES
https://www.google.co.in/

Sixth sense technology

More Related Content

What's hot

Similar to Sixth sense technology

Recently uploaded

Sixth sense technology