Human operator monitoring system

1. AgendaAgenda
 MotivationMotivation
 What isWhat is BlueEyeBlueEye technology ?technology ?
 What isWhat is BlueEyesBlueEyes ??
 System designingSystem designing
 System overviewSystem overview
 DAUDAU
 CSUCSU
 Data securityData security
 IBM researchIBM research
 ConclusionConclusion
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration

2. MotivationMotivation
 Human error - a frequent reason ofHuman error - a frequent reason of
catastrophes and ecological disasterscatastrophes and ecological disasters
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
-temporary indisposition-temporary indisposition
-weariness-weariness
-oversight-oversight

3. What is blue eyeWhat is blue eye
technology ?technology ?
 The BLUE EYES technologyThe BLUE EYES technology
aims at creating computationalaims at creating computational
machines that have perceptualmachines that have perceptual
and sensory ability like those ofand sensory ability like those of
human beings.human beings.

4. How can we makeHow can we make
computers “see” andcomputers “see” and
“feel” ?“feel” ?
 Blue eyes uses sensing technology toBlue eyes uses sensing technology to
identify a user’s actions and to extract keyidentify a user’s actions and to extract key
information.information.
 Information is then analyzed to determineInformation is then analyzed to determine
the user’s physical ,emotional ,orthe user’s physical ,emotional ,or
informational state.informational state.

5. What isWhat is BlueEyesBlueEyes ??
 The complex solution for human-operatorThe complex solution for human-operator
monitoring:monitoring:
– Visual attention monitoringVisual attention monitoring
– Physiological conditionPhysiological condition
– Operator’s position detectionOperator’s position detection
– Wireless data acquisition using bluetoothWireless data acquisition using bluetooth
– Real time user defined alarm triggeringReal time user defined alarm triggering
– Recorded data playbackRecorded data playback
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration

6.  Doesn’t predict nor interfere with operator’sDoesn’t predict nor interfere with operator’s
thoughtsthoughts
 Cannot force directly the operator to workCannot force directly the operator to work
What is BlueEyes not?What is BlueEyes not?

7. Applications ofApplications of BlueEyesBlueEyes
 Generic control roomsGeneric control rooms
((System can be applied in every working environmentSystem can be applied in every working environment
requiring permanent operator’s attention)requiring permanent operator’s attention)
– Power stationPower station
– Captain bridgeCaptain bridge
– Flight control centersFlight control centers
– Operating theatres – anesthesiologistsOperating theatres – anesthesiologists
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration

8. BlueEyesBlueEyes - benefits- benefits
 Prevention from dangerous incidentsPrevention from dangerous incidents
 Minimization ofMinimization of
– ecological consequencesecological consequences
– financial lossfinancial loss
– a threat to a human lifea threat to a human life
 The reconstructionThe reconstruction
of the course ofof the course of
operator’s workoperator’s work
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration

9. Implementation andImplementation and
engineering considerationsengineering considerations
-THREE groups of users-THREE groups of users
 OPERATOR -a person whose physiological parameters areOPERATOR -a person whose physiological parameters are
supervisedsupervised
- authorization in the system- authorization in the system
- receiving alerts- receiving alerts
 SUPERVISOR – a person responsible for analyzing operator’sSUPERVISOR – a person responsible for analyzing operator’s
condition and performance.condition and performance.
 SYSTEM ADMINISTRATORS- maintains the systemSYSTEM ADMINISTRATORS- maintains the system
deliver tools for addingdeliver tools for adding
- new operator’s to the database- new operator’s to the database
- defining alarm conditions- defining alarm conditions
- configuring logging tools- configuring logging tools
- creating new analyzer modules- creating new analyzer modules

10. DesigningDesigning
 A personal area network for linking allA personal area network for linking all
the operators and the supervisingthe operators and the supervising
systemsystem
 Two major unitsTwo major units
- DAU (data acquisition unit )DAU (data acquisition unit )
- CSU (central system unit )CSU (central system unit )

11. System overviewSystem overview
Central System UnitData Acquisition Unit
Atmel 89C52
microcontroller
Jazz
Multisensor
Connection
Manager
Data Analysis
Visualization
ModuleBluetooth Bluetooth
Bluetooth technology provides
means for creating a Personal
Area Network linking the
operators and the central
system.
Data
Logger
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
•Objectives
•System
overview
•DAU
•CSU

12. The eye movementThe eye movement
sensorsensor
 Off-shelf eye movement sensor – JazzOff-shelf eye movement sensor – Jazz
multisensormultisensor
 Supplies raw digital data regardingSupplies raw digital data regarding
- Eye positionEye position
- Level of blood oxygenationLevel of blood oxygenation
- Acceleration along horizontal and verticalAcceleration along horizontal and vertical
axesaxes
- Ambient light intensityAmbient light intensity
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration

13. Jazz multisensorJazz multisensor
Jazz MultisensorJazz Multisensor

14. JazzJazz

15. DAUDAU
The DAU consists of the following componentsThe DAU consists of the following components
 ATMEL 8952 microcontrollerATMEL 8952 microcontroller
 BLUE TOOTH MODULE – supports synchronous voice dataBLUE TOOTH MODULE – supports synchronous voice data
transmissiontransmission
 PCM CODEC –used to transmit operator’s voice and centralPCM CODEC –used to transmit operator’s voice and central
system sound feedbacksystem sound feedback
 UART –communication between bluetooth module andUART –communication between bluetooth module and
microcontroller (115200 bps)microcontroller (115200 bps)
 MAX232 –level shifterMAX232 –level shifter
 ALPHAUNUMERIC LCD displayALPHAUNUMERIC LCD display
 LED indicatorsLED indicators
 ID CARD interfaceID CARD interface

16. BlueEyes
Data Acquisition Unit
ID card
DAU - componentsDAU - components
MC 145483
PCM codec
ID card
interface
Simple
keyboard
Jazz Multisensor
Beeper
LCD display
LED indicators
Atmel 89C52
microcontroller
UART MAX232
UART
Bluetooth
module
microphone
earphone

17. Microcontroller & blueMicrocontroller & blue
tooth moduletooth module

18. EEPROMS & the PCBEEPROMS & the PCB

19. Implementation - DAUImplementation - DAU
Bluetooth module
PCM codec board
Batteries and
voltage level monitor
Microcontroller board
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
•DAU
•CSU

20. DAU - featuresDAU - features
 LightweightLightweight
 Runs on batteries - lowRuns on batteries - low
power consumptionpower consumption
 Easy to use - does notEasy to use - does not
disturb the operatordisturb the operator
workingworking
 ID cards for operatorID cards for operator
authorizationauthorization
 Voice transmission usingVoice transmission using
hardware PCM codechardware PCM codec
Data Acquisition Unit
Atmel 89C52
microcontroller
Jazz
Multisensor
Bluetooth
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
•Objectives
•System
overview
•DAU
•CSU

21. CSU - featuresCSU - features
Central System Unit
Connection
Manager
Data Analysis
Visualization
ModuleBluetooth
 ConnectionConnection
managementmanagement
 Data processingData processing
 VisualizationVisualization
 Data recordingData recording
 AccessAccess
verificationverification
 System maintenanceSystem maintenance
Connection
Manager
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
•Objectives
•System
overview
•DAU
•CSU
Data
Logger

22. CSU – componentsCSU – components
 CONNECTION MANAGER – main task to performCONNECTION MANAGER – main task to perform
low-level blue tooth communicationlow-level blue tooth communication
 DATA ANALYSIS MODULE – performs the analysisDATA ANALYSIS MODULE – performs the analysis
of the raw sensor data in order to obtain informationof the raw sensor data in order to obtain information
about operator’s physiological conditionabout operator’s physiological condition
 DATA LOGGER MODULE – provides support forDATA LOGGER MODULE – provides support for
storing the monitored data.storing the monitored data.
 VISULAIZATION MODULE – provides userVISULAIZATION MODULE – provides user
interface for the supervisorsinterface for the supervisors

23. CSU - VisualizationCSU - Visualization
ModuleModule•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
•Objectives
•System
overview
•DAU
•CSU
 Raw and processed data visualization using:Raw and processed data visualization using:
 audio and videoaudio and video
data playbackdata playback
– VU-metersVU-meters
– pie-chartspie-charts
– time seriestime series

24. Data securityData security
 Only registered mobile devicesOnly registered mobile devices
can connect to the systemcan connect to the system
 Bluetooth connection authenticationBluetooth connection authentication
 Bluetooth connection encryptionBluetooth connection encryption
 Access rights restrictionsAccess rights restrictions
 Personal and physiological data encryptionPersonal and physiological data encryption
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
•Objectives
•System
overview
•DAU
•CSU

25. IBM researchIBM research
 BLUE EYE – EMOTIONAL MOUSEBLUE EYE – EMOTIONAL MOUSE
sensors in the mouse ,sense thesensors in the mouse ,sense the
physiological attributes which are correlatedphysiological attributes which are correlated
to emotions using correlation modelto emotions using correlation model
-by simply touching the mouse ,the computer-by simply touching the mouse ,the computer
will be able to determine a person’swill be able to determine a person’s
emotional state.emotional state.
 BLUE EYE enabled TELEVISION – couldBLUE EYE enabled TELEVISION – could
become active when the user makes an eyebecome active when the user makes an eye
contact.contact.

26. ConclusionConclusion
 In the near future ,ordinary householdIn the near future ,ordinary household
devices- such as television ,devices- such as television ,
refrigerators ,ovens may be able to dorefrigerators ,ovens may be able to do
their jobs when we look at them andtheir jobs when we look at them and
speak to them.speak to them.
 Future applications of blue eyeFuture applications of blue eye
technology is limitlesstechnology is limitless

27. ReferencesReferences
 www.seminar4u.comwww.seminar4u.com
 www.howstuffswork.comwww.howstuffswork.com
 www.cs.put.poznan.comwww.cs.put.poznan.com
 www.whitepapers.comwww.whitepapers.com
 www.ibmresearchcenter.comwww.ibmresearchcenter.com
 www.almaden.ibm.comwww.almaden.ibm.com

28. Thank youThank you
BlueEyesBlueEyes
Human-OperatorHuman-Operator
Monitoring SystemMonitoring System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•System
demonstration
•Future
improvements
•Project work
summary