The document describes an artificial passenger system that aims to prevent driver fatigue and accidents. The system uses sensors like eye trackers and cameras to monitor the driver for signs of drowsiness. It engages the driver in conversation to determine alertness. If drowsiness is detected, it takes actions like opening windows, sounding an alarm, or spraying water to wake the driver. The system analyzes speech and eye movements using various components like speech recognition and a conversational planner. Its goal is to keep drivers alert during long solo trips to make roads safer.

1. ARTIFICIALARTIFICIAL
PASSENGERPASSENGER(A Sleep Prevention Dialogue Based Car
System)
Prepared By
Jesalpura Riddhi
09-IT-14
Guided By
Mansi Parmar

2. IndexIndex
IntroductionIntroduction
Why such systemWhy such system
Artificial passengerArtificial passenger
What does it doWhat does it do
DevicesDevices
Working componentWorking component
AdvantagesAdvantages
 DisadvantagesDisadvantages
ApplicationApplication
Future enhancementFuture enhancement
ConclusionConclusion
BibliographyBibliography

3. IntroductionIntroduction
 IBM (International business machinesIBM (International business machines
corporation, NY) has developed a softwarecorporation, NY) has developed a software
that holds a conversation with the driver tothat holds a conversation with the driver to
determine whether the driver can responddetermine whether the driver can respond
alertly enough, called “alertly enough, called “Artificial PassengerArtificial Passenger”.”.
This was designed to makeThis was designed to make
long solo journeys safer andlong solo journeys safer and
more bearable.more bearable.

4. Why Such System?Why Such System?
 According to a national survey in UK and USA, itAccording to a national survey in UK and USA, it
is observed that the driver fatigue annuallyis observed that the driver fatigue annually
causescauses
• 10000 crashes10000 crashes
• 1500 deaths1500 deaths
• 7100 injuries7100 injuries
 Majority of off-road accidents observed, wereMajority of off-road accidents observed, were
caused by eye closure of half and even 2-3caused by eye closure of half and even 2-3
seconds, where the normal human eye blinks atseconds, where the normal human eye blinks at
0.2-0.3 seconds.0.2-0.3 seconds.

5. What is an Artificial Passenger?What is an Artificial Passenger?
 Natural language e-companion.Natural language e-companion.
 Sleep preventive device in cars toSleep preventive device in cars to
overcome drowsiness.overcome drowsiness.
 Life safety system.Life safety system.

6. What Does It Do?What Does It Do?
 Detects alarm conditions through sensors.Detects alarm conditions through sensors.
 Broadcasts pre-stored voice messagesBroadcasts pre-stored voice messages
over the speakers.over the speakers.
 Captures images of the driver.Captures images of the driver.

7. Condition SensorCondition Sensor
Mobile IndicatorMobile Indicator
DeviceDevice

8. About Artificial PassengerAbout Artificial Passenger
 The AP is anThe AP is an Artificial IntelligenceArtificial Intelligence basedbased
companion that will be resident in softwarecompanion that will be resident in software
and chips embedded in the automobileand chips embedded in the automobile
dashboard.dashboard.
 The system has a conversation plannerThe system has a conversation planner
that holds a profile of you, including detailsthat holds a profile of you, including details
of your interests and profession.of your interests and profession.

9.  A microphone picks up your answer andA microphone picks up your answer and
breaks it down into separate words withbreaks it down into separate words with
speech-recognition software.speech-recognition software.
 A camera built into the dashboard alsoA camera built into the dashboard also
tracks your lip movements to improve thetracks your lip movements to improve the
accuracy of speech recognition.accuracy of speech recognition.

10.  A voice analyzer then looks for signs ofA voice analyzer then looks for signs of
tiredness by checking to see if the answertiredness by checking to see if the answer
matches your profile. Slow responses andmatches your profile. Slow responses and
lack of attention are signs of fatigue.lack of attention are signs of fatigue.
 If you reply quickly and clearly, the systemIf you reply quickly and clearly, the system
judges you to be alert and tells thejudges you to be alert and tells the
conversation planner to continue the lineconversation planner to continue the line
of questioning.of questioning.

11.  If your response is slow or doesn’t makeIf your response is slow or doesn’t make
sense, the voice analyzer assumes yousense, the voice analyzer assumes you
are dropping off and acts to get yourare dropping off and acts to get your
attention.attention.
 If driver displays signs of fatigue, theIf driver displays signs of fatigue, the
artificial passenger might be programmedartificial passenger might be programmed
to open all the windows, sound a buzzer,to open all the windows, sound a buzzer,
increase background music volume, orincrease background music volume, or
even spray the driver with ice water.even spray the driver with ice water.

12. Devices Used in ArtificialDevices Used in Artificial
PassengerPassenger
 Eye trackerEye tracker
 Voice recognizer or speech recognizerVoice recognizer or speech recognizer
 Natural language processorNatural language processor
 Driver analyzerDriver analyzer
 Conversational plannerConversational planner
 AlarmAlarm
 MicrophoneMicrophone
 CameraCamera

13. Working ComponentsWorking Components
Eye-TrackerEye-Tracker
 Collecting eye movement data requires both hardware andCollecting eye movement data requires both hardware and
software.software.
Hardware:: Head-Mounted systems or remote systemsHardware:: Head-Mounted systems or remote systems
 Both systems measure the corneal reflection of the infrared LED,Both systems measure the corneal reflection of the infrared LED,
which illuminates and generates a reflection off the surface of thewhich illuminates and generates a reflection off the surface of the
eye.eye.
 This action causes the pupil to appear as a bright disk in contrast toThis action causes the pupil to appear as a bright disk in contrast to
surrounding iris and creates a small glint underneath the pupil whichsurrounding iris and creates a small glint underneath the pupil which
is used for calibration and tracking.is used for calibration and tracking.
 Head mounted systemsHead mounted systems use multiple data points to record eyeuse multiple data points to record eye
movement and measures pupil glint from multiple angles.movement and measures pupil glint from multiple angles.
 Remote systemsRemote systems ,in contrast, measure the orientation of the eye,in contrast, measure the orientation of the eye
relative to a fixed unit such as camera.relative to a fixed unit such as camera.

14. Monitoring SystemMonitoring System

15. Software:Software:
 Data collection and analysis is handledData collection and analysis is handled
by eye-tracking software.by eye-tracking software.
 Software catalogs eye-tracking in one ofSoftware catalogs eye-tracking in one of
the two waysthe two ways
1.1. ERICA’s Eye Gaze software uses aERICA’s Eye Gaze software uses a
small red x to represent eye movement.small red x to represent eye movement.
2.2. In other, data is stored as a series of x/yIn other, data is stored as a series of x/y
coordinates related to specific grid pointscoordinates related to specific grid points
on the computer screenon the computer screen

16. Eye TrackerEye Tracker

17. Working Components( cont…)Working Components( cont…)
Automatic Speech RecognitionAutomatic Speech Recognition
There are two ASRs used in the system:There are two ASRs used in the system:
 First one is “speaker independent” and used forFirst one is “speaker independent” and used for
decoding voice signals of the driver.decoding voice signals of the driver.
 Second one operates with voice car media andSecond one operates with voice car media and
decodes tapes, audio , telephones, mails etc.decodes tapes, audio , telephones, mails etc.

18. Working Components(Working Components(
contd…)contd…)
Natural Language ProcessorNatural Language Processor
 Processes the decoded signals of voice or text data providedProcesses the decoded signals of voice or text data provided
from the ASR.from the ASR.
 Identifies related meanings from the contents of the decodedIdentifies related meanings from the contents of the decoded
messages.messages.
 Produces variant of responses.Produces variant of responses.
 This output goes to the driver analyzer as an input.This output goes to the driver analyzer as an input.

19. Working Components(Working Components(
contd…)contd…)
Driver AnalyzerDriver Analyzer
 Receives the textual data and voice data.Receives the textual data and voice data.
 Measures the time response using a clock.Measures the time response using a clock.
 Time responses, conclusions about driver’sTime responses, conclusions about driver’s
alertness will be passed on to thealertness will be passed on to the
conversational planner.conversational planner.

20. Working Components(Working Components(
contd…)contd…)
Conversational PlannerConversational Planner
 It is the heart of the system.It is the heart of the system.
 Instructs the language generator to produce theInstructs the language generator to produce the
response.response.
 If the driver is in perfect condition CP instructs theIf the driver is in perfect condition CP instructs the
language generator to continue the conversationlanguage generator to continue the conversation
otherwise it will be instructed to change theotherwise it will be instructed to change the
conversation.conversation.

21. Working ComponentsWorking Components
(contd…)(contd…)
AlarmAlarm
 If the CP (conversational planner) receivesIf the CP (conversational planner) receives
information that the driver is about to fall asleepinformation that the driver is about to fall asleep
an alarm system is activated.an alarm system is activated.
MicrophoneMicrophone
 For picking up words and separateFor picking up words and separate
them by some internally used softwarethem by some internally used software
for conversation.for conversation.

22. Working ComponentsWorking Components
(contd…)(contd…)
CameraCamera
 This will track the lip movements of theThis will track the lip movements of the
driver.driver.
 Used to improve the accuracy of theUsed to improve the accuracy of the
speech recognition.speech recognition.

23. ..
Showing the dashboard of the car where the whole artificial system is generallyShowing the dashboard of the car where the whole artificial system is generally
attached.attached.

24. Steps Involved in TrackingSteps Involved in Tracking
 Automatically initialize lips and eyes usingAutomatically initialize lips and eyes using
color predicates and connectedcolor predicates and connected
components.components.
 Track lip corners using dark line betweenTrack lip corners using dark line between
lips and color predicate even throughlips and color predicate even through
large mouth movement like yawning.large mouth movement like yawning.

25.  Construct a bounding box of the head.Construct a bounding box of the head.
 Determine rotation using distancesDetermine rotation using distances
between eye and lip feature points andbetween eye and lip feature points and
sides of the face.sides of the face.
 Determine eye blinking and eye closingDetermine eye blinking and eye closing
using the number and intensity of pixels inusing the number and intensity of pixels in
the eye region.the eye region.
 Determine driver vigilance level using allDetermine driver vigilance level using all
acquired information.acquired information.

26. DetectingDetecting Driver VigilanceDriver Vigilance
 Aiming a single camera at a head of the driver.Aiming a single camera at a head of the driver.
 Detecting frequency of up and down nodding and left toDetecting frequency of up and down nodding and left to
right rotations of the head within a selected time periodright rotations of the head within a selected time period
with the camera.with the camera.
 Determining frequency of eye blinking and eye closing.Determining frequency of eye blinking and eye closing.
 Determining frequency of yawning of the driver within theDetermining frequency of yawning of the driver within the
selected time period with the cameraselected time period with the camera
 Generating an alarm signal in real time if the frequencyGenerating an alarm signal in real time if the frequency
value of the up and down nodding, the left to rightvalue of the up and down nodding, the left to right
rotations, the eye blinking, the eye closings, the yawningrotations, the eye blinking, the eye closings, the yawning
exceeds a selected threshold value.exceeds a selected threshold value.

27. AdvantagesAdvantages
 It is broadly used to prevent accident.It is broadly used to prevent accident.
 Artificial component establishes interfaceArtificial component establishes interface
with other drivers very easily.with other drivers very easily.
 If the driver gets a heart attack or drunk , itIf the driver gets a heart attack or drunk , it
will send signals to vehicles nearby aboutwill send signals to vehicles nearby about
this so driver there become alert.this so driver there become alert.

28. DisadvantagesDisadvantages
It is costly.
It can only be installed to high profiled

29. ApplicationsApplications
 Interface with Neighboring CarsInterface with Neighboring Cars --Determines if a driver presents aDetermines if a driver presents a
high safety risk for ex. falling asleep, tired, inexperienced or underhigh safety risk for ex. falling asleep, tired, inexperienced or under
the influence of alcohol and signals the cars nearby to be careful ofthe influence of alcohol and signals the cars nearby to be careful of
the driver.the driver.
 Medical ApplicationMedical Application -The system can monitor a driver and detect if-The system can monitor a driver and detect if
they are sick ,for ex. having a stroke or heart attack.they are sick ,for ex. having a stroke or heart attack.
 In any problem it alerts the vehicles near by , so the driver there canIn any problem it alerts the vehicles near by , so the driver there can
become alert.become alert.
 Opens and closes the doors and windows of the car automatically.Opens and closes the doors and windows of the car automatically.
 It is also used for the entertainment.It is also used for the entertainment.
 Also used in cabins in airplanes, trains, boats etcAlso used in cabins in airplanes, trains, boats etc

30. Future ImplementationFuture Implementation
 Will provide us with shortest time routingWill provide us with shortest time routing
based on road conditions changingbased on road conditions changing
because of weather and traffic, informationbecause of weather and traffic, information
about the cars on the route, destinationabout the cars on the route, destination
requirement (as-flight has been delayedrequirement (as-flight has been delayed
etc)etc)

31. ConclusionConclusion
Method for monitoring driver alertnessMethod for monitoring driver alertness
Sufficient time to avert an accident.Sufficient time to avert an accident.
Successful implementation of ArtificialSuccessful implementation of Artificial
passenger would allow use of variouspassenger would allow use of various
services in car like reading emails,services in car like reading emails,
navigation, downloading music files,navigation, downloading music files,
voice games etc without compromisingvoice games etc without compromising
on driver safetyon driver safety

32. BibliographyBibliography
 L R Bhal et al. “Performance of the IBML R Bhal et al. “Performance of the IBM
speech recognition systemspeech recognition system
 www.freepatentsonline.com/4682348.htmlwww.freepatentsonline.com/4682348.html
 www.slideshare.comwww.slideshare.com
 www.about.comwww.about.com
 www.answers.comwww.answers.com
 www.google.comwww.google.com
 www.wikipedia.comwww.wikipedia.com

33. Thank You…Thank You…