The document describes an artificial passenger system developed by IBM to monitor driver alertness on long solo drives. The system uses sensors like eye tracking and speech recognition to engage the driver in conversation. If signs of fatigue are detected, such as slowed responses, the system activates alarms to alert the driver. Its goal is to reduce accidents caused by drowsy driving and make long trips safer.

1. Presented By:
Vishnu Parsi
Guided By:

2. Introduction
• IBM (International business machines
corporation, NY) has developed a software that
holds a conversation with the driver to
determine whether the driver can respond
alertly enough, called “Artificial Passenger”.
• This was designed to make long solo journeys
safer and more bearable.

3. Why Such System?
• According to a national survey in UK and USA, it
is observed that the driver fatigue Annually
causes
• 10000 crashes
• 1500 deaths
• 7100 injuries
• Majority of off-road accidents observed,
were caused by eye closure of half and
even 2-3 seconds, where the normal
human eye blinks at 0.2-0.3 seconds.

4. What is an artificial
Passenger?
• Natural language e-companion.
• Sleep preventive device in cars to
overcome drowsiness.
• Life safety system.

5. What does it do?
• Detects alarm conditions through
sensors.
• Broadcasts pre-stored voice messages
over the speakers.
• Captures images of the driver.

6. Condition Sensor
Mobile Indicator
Device

7. About Artificial
Passenger
• The AP is an “Artificial Intelligence”
based companion that will be resident in
software and chips embedded in the
automobile dashboard.
• The system has a conversation planner
that holds a profile of you, including
details of your interests and profession.

8. • A microphone picks up your answer and
breaks it down into separate words with
speech-recognition software.
• A camera built into the dashboard also
tracks your lip movements to improve
the accuracy of speech recognition.
• A voice analyzer then looks for signs of
tiredness by checking to see if the
answer matches your profile. Slow
responses and lack of attention are
signs of fatigue.

9. • If you reply quickly and clearly, the system
judges you to be alert and tells the
conversation planner to continue the line of
questioning.
• If your response is slow or doesn’t make
sense, the voice analyzer assumes you are
dropping off and acts to get your attention.
• If driver displays signs of fatigue, the
artificial passenger might be programmed
to open all the windows, sound a buzzer,
increase background music volume, or
even spray the driver with ice water.

10. Devices Used in Artificial
Passenger
• Eye tracker
• Voice recognizer or speech recognizer
Natural language processor
• Driver analyzer
• Conversational planner
• Alarm
• Microphone
• Camera

11. Working Components
Eye-Tracker
• Collecting eye movement data requires
both hardware and software.
Hardware: Head-Mounted systems or
remote systems
• Both systems measure the corneal
reflection of the infrared LED, which
illuminates and generates a reflection off
the surface of the eye.

12. • This action causes the pupil to appear
as a bright disk in contrast to
surrounding iris and creates a small glint
underneath the pupil which is used for
calibration and tracking.
• Head mounted systems use multiple
data points to record eye movement and
measures pupil glint from multiple
angles.
• Remote systems ,in contrast, measure
the orientation of the eye relative to a
fixed unit such as camera.

13. Monitoring System

14. Software:
• Data collection and analysis is handled
by eye-tracking software.
• Software catalogs eye-tracking in one of
the two ways
• ERICA’s Eye Gaze software uses a
small red x to represent eye movement.
• In other, data is stored as a series of x/y
coordinates related to specific grid
points on the computer screen

15. Eye Tracker

16. Working Components
(Cotd..)
Automatic Speech Recognition
There are two ASRs used in the system:
• First one is “speaker independent” and
used for decoding voice signals of the
driver.
• Second one operates with voice car
media and decodes tapes, audio ,
telephones, mails etc.

17. Natural Language Processor
• Processes the decoded signals of voice
or text data provided from the ASR.
• Identifies related meanings from the
contents of the decoded messages.
• Produces variant of responses.
• This output goes to the driver analyzer
as an input.

18. Driver Analyzer
• Receives the textual data and voice
data.
• Measures the time response using a
clock.
• Time responses, conclusions about
driver’s alertness will be passed on to
the conversational planner.

19. Conversational Planner
• It is the heart of the system.
• Instructs the language generator to
produce the response.
• If the driver is in perfect condition CP
instructs the language generator to
continue the conversation otherwise it
will be instructed to change the
conversation.

20. The following is a typical scenario involving Artificial
Passenger. Imagine, driver “Joe” returning home
after late nights. His head starts to nod
AP Hey Joe, what did you get your daughter for her
birthday?
Joe It’s not her birthday!
AP You seem a little tired. Want to play a game?
Joe Yes.
AP You were a wiz at “Name that Tune” last time. I was
impressed. Want to try your hand at trivia?
Joe ok.
AP Pick a category: Hollywood Stars, Magic Moments
for Hall of Fame?
Joe Hall of Fame.

21. Working Components
(Cotd..)
Alarm
 If the CP (conversational planner) receives
information that the driver is about to fall
asleep an alarm system is activated.
Microphone
 For picking up words
and separate them by some internally
used software for conversation.

22. Working
Components(Cotd..)
Camera
 This will track the lip movements of the
driver.
 Used to improve the accuracy of the
speech recognition.

23. Showing the dashboard of the car where the whole artificial system is generally
attached.

24. Steps Involved in Tracking
 Automatically initialize lips and eyes
using color predicates and connected
components.
 Track lip corners using dark line
between lips and color predicate even
through large mouth movement like
yawning.

25. • Construct a bounding box of the head.
• Determine rotation using distances between
eye and lip feature points and sides of the
face.
• Determine eye blinking and eye closing
using the number and intensity of pixels in
the eye region.
• Determine driver vigilance level using all
acquired information.

26. Detecting Driver Vigilance
 Aiming a single camera at a head of the driver.
 Detecting frequency of up and down nodding and left
to right rotations of the head within a selected time
period with the camera.
 Determining frequency of eye blinking and eye
closing.
 Determining frequency of yawning of the driver within
the selected time period with the camera
 Generating an alarm signal in real time if the
frequency value of the up and down nodding, the left
to right rotations, the eye blinking, the eye closings,
the yawning exceeds a selected threshold value.

27. Application s
 Interface with Neighboring Cars -Determines if a
driver presents a high safety risk for ex. falling
asleep, tired, inexperienced or under the influence of
alcohol and signals the cars nearby to be careful of
the driver.
 Medical Application -The system can monitor a driver
and detect if they are sick ,for ex. having a stroke or
heart attack.
 In any problem it alerts the vehicles near by , so the
driver there can become alert.
 Opens and closes the doors and windows of the car
automatically.
 It is also used for the entertainment.
 Also used in cabins in airplanes, trains, boats etc

28. Future Implementation
 Will provide us with shortest time routing
based on road conditions changing
because of weather and traffic,
information about the cars on the route,
destination requirement (as-flight has
been delayed etc)

29. Conclusion
 Method for monitoring driver alertness
 Sufficient time to avert an accident.
 Successful implementation of Artificial
passenger would allow use of various
services in car like reading emails,
navigation, downloading music files,
voice games etc without compromising
on driver safe