3. INTRODUCTION
Sixth Sense is a wearable interface device that builds up physical
world around us with digital information and lets us use natural hand
gestures to interact with that information.
It was developed by Pranav, a PhD student in the Fluid Interfaces
Group at the MIT Media Lab.
Key Points
wearable interface device :- Pendant like mobile wearable device
Physical world around us with digital information :- Ex- Projector
Hand gestures :- Like Zoom-In, Zoom-Out
4. WHY THE NAME “SIXTH SENSE”
It will sense your hand gestures.
It will sense what object is in front of you.
It will sense what you see.
It will sense what information you need and project those
information on any surface.
6. Camera
Captures an object in view and tracks the user’s hand gestures
It sends the data to smart phone
It acts as a digital eye, connecting you to the world of digital
information
http://www.pranavmistry.com/projects/sixthsense/
7. Projector
The projector projects visual information enabling surfaces and physical
objects to be used as interfaces
The project itself contains a battery inside, with 3 hours of battery life.
A tiny LED projector displays data sent from the smart phone on any
surface in view–object, wall, or person.
http://www.pranavmistry.com/projects/sixthsense/
8. Smart Phone
A Web-enabled smart phone in the user’s pocket processes the video
data
Other software searches the Web and interprets the hand gestures
http://www.pranavmistry.com/projects/sixthsense/
9. Color Markers
It is at the tip of the user’s fingers .
Marking the user’s fingers with red, yellow, green, and blue tape helps
the webcam recognize gestures
The movements and arrangements of these makers are interpreted into
gestures that act as interaction instructions.
http://www.pranavmistry.com/projects/sixthsense/
11. The plan of action
• Get images from the camera in a loop – Frames
https://www.youtube.com/watch?v=22TCiQ6_yTs
• Convert BGR image to HSV image
• Thresholding the Image (Binary Image)
• Scribbling the position of the Colored Markers
• Comparing the scribbled Image with pre-defined
gesture image
12.
13. 1. Get an image from the camera - Frames
• frame rate is the number of frames or images that are
projected or displayed per second.
• https://www.youtube.com/watch?v=22TCiQ6_yTs
14. 2. We convert the RGB image into an HSV image.
• We'll be using the HSV colour space, instead of the more
common RGB colour space.
• HSV stands for HUE, SATURATION and VALUE.
• value range for HUE, SATURATION and VALUE are
respectively 0-179, 0-255 and 0-255.
• HUE represents the color,
• SATURATION represents the amount to which that respective
color is mixed with white.
15. • VALUE represents the amount to which that respective
color is mixed with black.
• Here the HUE is unique for that specific color distribution of
that object.
But SATURATION and VALUE may be vary according to the
lighting condition of that environment.
• In HSV, each colour is assigned a particular number (the
Hue- ex- 125).
• This gives us the advantage of having a single number (hue)
for the red color despite multiple shades of red(all the way
from dark redto a bright red).
16. • We have to supply the HSV Range of the color to be
Threshold
• After Thresholding, the Red will be white(1) and the rest will
be black(0)
3. Thresholding the Image
http://aishack.in/tutorials/tracking-colored-objects-opencv/
17. 4. Scribbling the position of the marker
• This shows the movement of the colored marker.
• After capturing and thresholding of each frames, we will
mark the position of the color marker on a new image
specially created for this purpose.
• Then. We will join the previous position and the current
position with a line.
• This process continues until the color marker goes out of the
screen or a gesture is detected.
19. 4. Comparing.
• In this stage we will compare the scribbled image with the
pre-defined gesture image.
• If they are similar, we invoke some function related to that
gesture, like @ symbol to open email application and
project through projector.
21. Make a call
You can use the Sixth Sense to project
a keypad onto your hand, then use
that virtual keypad to make a call.
Call up a map
With the map application we can call
up the map of our choice and then
use thumbs and index fingers to
navigate the map
http://www.pranavmistry.com/projects/sixthsense/
22. Check the time
Draw a circle on your wrist to get a
virtual watch that gives you the
correct time
Create multimedia reading
experiences
Sixth Sense can be programmed to
project related videos onto
newspaper articles you are reading
http://www.pranavmistry.com/projects/sixthsense/
23. Drawing application
The drawing application lets the user
draw on any surface by tracking the
fingertip movements of the user’s index
finger
Zooming features
The user can zoom in or zoom out
using intuitive hand movements
http://www.pranavmistry.com/projects/sixthsense/
24. Get product information
Sixth Sense uses image recognition
or marker technology to recognize
products we pick up, then feeds us
information on those products
Get book information
The system can project Amazon
ratings on that book, as well as reviews
and other relevant information
http://www.pranavmistry.com/projects/sixthsense/
25. Get flight updates
The system will recognize your
boarding pass and let you know
whether your flight is on time and if
gate has changed.
http://www.pranavmistry.com/projects/sixthsense/
26. Take pictures
If you fashion your index fingers and
thumbs into a square ("framing"
gesture), the system will snap a photo.
After taking the desired number of
photos, we can project them onto a
surface, and use gestures to sort
through the photos, and organize and
resize them.
http://www.pranavmistry.com/projects/sixthsense/