Blue eyes seminar report — Document Transcript 1. 1 ABSTRACT Human error is still one of the most frequent causes of catastrophes andecological disasters. The main reason is that the monitoring systems concern only thestate of the processes whereas human contribution to the overall performance of thesystem is left unsupervised. Since the control instruments are automated to a largeextent, a human – operator becomes a passive observer of the supervised system,which results in weariness and vigilance drop. This, he may not notice importantchanges of indications causing financial or ecological consequences and a threat tohuman life. It therefore is crucial to assure that the operator’s conscious brain isinvolved in an active system supervising over the whole work time period. It ispossible to measure indirectly the level of the operator’s conscious brain involvementusing eye motility analysis. Although there are capable sensors available on themarket, a complex solution enabling transformation, analysis and reasoning based onmeasured signals still does not exist. In large control rooms, wiring the operator to thecentral system is a serious limitation of his mobility and disables his operation.Utilization of wireless technology becomes essential. Blue Eyes is intended to be the complex solution for monitoring andrecording the operator’s conscious brain involvement as well as his Physiologicalcondition. This required designing a Personal Area Network linking all the Operatorsand the supervising system. As the operator using his sight and hearing senses thestate of the controlled system, the supervising system will look after his physiologicalcondition. AIM 1) To design smarter devices 2) To create devices with emotional intelligence 3) To create computational devices with perceptual abilities 2. 2 The idea of giving computers personality or, more accurately, emotionalintelligence" may seem creepy, but technologists say such machines would offerimportant advantages. De-spite their lightning speed and awesome powers of computation,todays PCs are essentially deaf, dumb, and blind. They cant see you, they cant hearyou, and they certainly dont care a whit how you feel. Every computer user knows thefrustration of nonsensical error messages, buggy software, and abrupt system crashes.We might berate the computer as if it were an unruly child, but, of course, themachine cant respond. "Its ironic that people feel like dummies in front of theircomputers, when in fact the computer is the dummy," says Rosalind Picard, acomputer science professor at the MIT Media Lab in Cambridge. A computer endowed with emotional intelligence, on the other hand,could recognize when its operator is feeling angry or frustrated and try to respond inan appropriate fashion. Such a computer might slow down or replay a tutorial programfor a confused student, or recognize when a designer is burned out and suggest he takea break. It could even play a recording of Beethovens "Moonlight Sonata" if it sensedanxiety or serve up a rousing Springsteen anthem if it detected lethargy. The possibleapplications of "emotion technology" extend far beyond the desktop. A car equippedwith an affective computing system could recognize when a driver is feeling drowsyand ad-vise her to pull over, or it might sense when a stressed-out motorist is about toexplode and warn him to slow down and cool off. These machines have got their own personality and this personalitydepends upon the moods of the user. INTRODUCTION BlueEyes system provides technical means for monitoring and recordingthe operator’s basic physiological parameters. The most important parameter issaccadic activity , which enables the system to monitor the status of the operator’s 1
3. 3visual attention along with head acceleration, which accompanies largedisplacementof the visual axis (saccades larger than 15 degrees). Complex industrialenvironmentcan create a danger of exposing the operator to toxic substances, which canaffect hiscardiac, circulatory and pulmonary systems. Thus, on the groundsofplethysmographic signal taken from the forehead skin surface, the systemcomputesheart beat rate and blood oxygenation. [Saccades are rapid eye jumps to newlocations within a visualenvironment assigned predominantly by the conscious attentionprocess.] The BlueEyes system checks above parameters against abnormal (e.g. alowlevel of blood oxygenation or a high pulse rate) or undesirable (e.g. a longerperiod oflowered visual attention) values and triggers user-defined alarms whennecessary. Quiteoften in an emergency situation operator speak to themselvesexpressing their surprise orstating verbally the problem. Therefore, the operator’svoice, physiological parametersand an overall view of the operating room arerecorded. This helps to reconstruct thecourse of operators’ work and provides data forlong-term analysis. BlueEyes consists of amobile measuring device and a centralanalytical system. The mobile device is integratedwith Bluetooth module providingwireless interface between sensors worn by the operatorand the central unit. ID cardsassigned to each of the operators and adequate user profileson the central unit sideprovide necessary data personalization so different people can usea single mobiledevice (called hereafter DAU – Data Acquisition Unit). The overallsystem diagram isshown in Figure 1. The tasks of the mobile Data Acquisition Unit are tomaintainBluetooth connections, to get information from the sensor and sending it overthewireless connection, to deliver the alarm messages sent from the Central SystemUnitto the operator and handle personalized ID cards. Central System Unit maintainstheother side of the Bluetooth connection, buffers incoming sensor data, performs on-linedata analysis, records the conclusions for further exploration andprovidesvisualization interface.4. 4 The task of the mobile Data Acquisition Unit are to maintain Bluetoothconnection, toget information from the sensor and sending it over the wirelessconnection ,to deliver thealarm messages sent from the Central System Unit to theoperator and handle personalizedID cards. Central System Unit maintains the otherside of the Bluetooth connection,buffers incoming sensor data, performs on-line dataanalysis, records the conclusion forfurther exploration and provides visualizationinterface.PERFORMANCEREQUIREMENTS The portable nature of the mobile unit results in a numberofperformance requirements. As the device is intended to run on batteries, lowpowerconsumption is the most important constraint. Moreover, it is necessary toassureproper timing while receiving and transmitting sensor signals. To make theoperationcomfortable the device should be lightweight and electrically safe. Finally theuse ofstandard and inexpensive IC’s will keep the price of the device at relatively lowlevel. The priority of the central unit is to provide real-time buffering andincoming sensorsignals and semi-real-time processing of the data, which requiresspeed-optimizes filteringand reasoning algorithms. Moreover, the design shouldassure the possibility ofdistributing the processing among two or more central unitnodes (e.g. to offload thedatabase system related tasks to a dedicated server).5. 5DESIGN METHODOLOGIES In creating the BlueEyes system a waterfall softwaredevelopmentmodel was used since it is suitable for unrepeatable and explorative projects.Duringthe course of the development UML standard notations were used. They
facilitatecommunication between team members, all the ideas are clearly expressed bymeansof various diagrams, which is a sound base for further development. The results ofthe functional design phase were documented on usecase diagrams. During the low-leveldesign stage the whole systems was divided intofive main modules. Each of them has anindependent, well-defined functionalinterface providing precise description of theservices offered to the other modules.All the interfaces are documented on UML class,interaction and state diagrams. Atthis point each of the modules can be assigned to a teammember, implemented andtested in parallel. The last stage of the project is the integratedsystem testing.INNOVATIVE IDEAS The unique feature of our system relies on thepossibility ofmonitoring the operator’s higher brain functions involved in the acquisitionof theinformation from the visual environment. The wireless link between the sensorswornby the operator and the supervising system offers new approach to systemoverallreliability and safety. This gives a possibility to design a supervising modulewhoserole is to assure the proper quality of the system performance. The newpossibilitiescan cover such areas as industry, transportation (by air, by road and by sea),militarycommand centers or operating theaters (anesthesiologists). IMPLEMANTATIONOF BLUE EYES TECHNOLOGYFUNCTIONAL DESIGN6. 6 During the functional design phase we used UML standarduse case notation, whichshows the functions the system offers to Operators particularusers. BlueEyes hasthree groups of users: System AdministratorsOperator is a person whose Supervisorsphysiological parameters are supervised. The operatorwears the DAU. The only functionsoffered to that user are authorization in the systemand receiving alarm alerts. Suchlimited functionality assures the device does notdisturb the work of the operator (Fig.2).Figure 2: Mobile Device UserAuthorization – the function is used when the operator’sduty starts. After insertinghis personal ID card into the mobile device and entering properPIN code the devicewill start listening for incoming Bluetooth connections. Once theconnection has beenestablished and authorization process has succeeded (the PIN code iscorrect) centralsystem starts monitoring the operator’s physiological parameters. Theauthorizationprocess shall be repeated after reinserting the ID card. It is not, however,required onreestablishing Bluetooth connection.Receiving alerts – the function suppliesthe operator with the information about themost important alerts regarding his or his co-workers’ condition and mobile devicestate (e.g. connection lost, battery low). Alarms aresignaled by using a beeper,7. 7earphone providing central system sound feedback and a small alphanumericLCDdisplay, which shows more detailed information.Supervisor is a person responsiblefor analyzing operators’ condition andperformance. The supervisor receives tools forinspecting present values of theparameters (On-line browsing) as well as browsing theresults of long-term analysis(Off-line browsing). During the on-line browsing it ispossible to watch a list of currentlyworking operators and the status of their mobiledevices. Selecting one of theoperators enables the supervisor to check the operator’scurrent physiologicalcondition (e.g. a pie chart showing active brain involvement) and ahistory of alarmsregarding the operator. All new incoming alerts are displayedimmediately so that thesupervisor is able to react fast. However, the presence of thehuman supervisor is notnecessary since the system is equipped with reasoning algorithmsand can triggeruser-defined actions (e.g. to inform the operator’s co-workers).
8. 8 During off-line browsing it is possible to reconstruct the course of theoperator’s dutywith all the physiological parameters, audio and video data. Acomprehensive dataanalysis can be performed enabling the supervisor to drawconclusions on operator’soverall performance and competency (e.g. for workingnight shifts).System Administratoris a user that maintains the system. The administrator deliverstools for adding newoperators to the database, defining alarm conditions, configuring logging tools andcreating new analyzer modules. While registering new operators the administrator entersappropriate data (anda photo if available) to the system database and programs hispersonal ID card. Defining alarm conditions – the function enables setting up user-definedalarm conditions by writing condition-action rules (e.g. low saccadic activityduring alonger period of time → inform operator’s co-workers, wake him up using thebeeperor playing appropriate sound and log the event in the database). Designing newanalyzer modules-based on earlier recorded data theadministrator can create new analyzermodule that can recognize other behaviors thanthose which are built-in the system. Thenew modules are created using decision treeinduction algorithm. The administratornames the new behavior to be recognized and9. 9points the data associated with it. The results received from the new modules canbeused in alarm conditions. Monitoring setup enables the administrator to choose theparameters tomonitor as well as the algorithms of the desired accuracy to computeparametervalues. Logger setup provides tools for selecting the parameters to berecorded.For audio data sampling frequency can be chosen. As regards the videosignal, a delaybetween storing consecutive frames can be set (e.g. one picture inevery twoseconds).Database maintenance – here the administrator can remove old or―uninteresting‖ datafrom the database. The ―uninteresting‖ data is suggested by the built-in reasoningsystem.DATA ACQUISITION UNIT (DAU) This section deals with thehardware part of the BlueEyes system with regard tothe physiological data sensor, theDAU hardware components and the microcontrollersoftware.PHYSIOLOGICAL DATASENSOR To provide the Data Acquisition Unit with necessary physiological data an off-shelf eye movement sensor – Jazz Multisensor is used. It supplies raw digitaldataregarding eye position, the level of blood oxygenation, acceleration alonghorizontaland vertical axes and ambient light intensity. Eye movement is measured usingdirectinfrared oculographic transducers. (The eye movement is sampled at 1 kHz, theotherparameters at 250 Hz. The sensor sends approximately 5.2 kB of data per second.)10. 10HARDWARE SPECIFICATION Microcontrollers (e.g. Atmel 8952microcontroller)can be used as the core of the Data Acquisition Unit since it is a well-established industrial standard and provides necessary functionalities(i.e. high speedserial port)at a low price. The Bluetooth module supports synchronous voice datatransmission .The codec reduces the microcontroller’s tasks and lessens the amount ofdata being sent over the UART. The Bluetooth module performs voice data compression,which results in smaller bandwidth utilization and better sound quality.11. 11 .Communication between the Bluetooth module and the microcontroller is carriedonusing standard UART interface. The speed of the UART is set to 115200 bps in ordertoassure that the entire sensor data is delivered in time to the central system. Thealphanumeric LCD display gives more information of incomingevents and helps theoperator enter PIN code. The LED indicators shows the result ofbuilt-in-self-test, powerlevel and the state of wireless connection. The simple keyboard is used to react to
incoming events and to enterPIN code while performing authorization procedure. The IDcard interface helpsconnect the operator’s personal identification card to the DAU. Afterinserting thecard authorization procedure starts. The operator’s unique identifier enablesthesupervising system to distinguish different operators.MICROCONTROLLERSOFTWARE SPECIFICATION12. 12 DAU software is written in assembler code, which assures the highestprogramefficiency and the lowest resource utilization. The DAU communicates withtheBluetooth module using Host Controller Interface (HCI) commands In the No ID cardstate a self-test is performed to check if the device isworking correctly. After the self-testpasses the sensor and Bluetooth module are resetand some initialization commands areissued(i.e. HCI_Reset,HCI_Ericsson_Set_UART_Baud_Rate etc.). Once the initializationhas beensuccessfully completed the device starts to check periodically for ID cardpresence by13. 13attempting to perform an I2C start condition. When the attempt succeeds andtheoperator’s identifier is read correctly the device enters User authorization state. In theUser authorization state the operator is prompted to enter hissecret PIN code. If the codematches the code read from the inserted ID card thedevice proceeds waiting for incomingBluetooth connections. On entering Waiting for connection state the DAU puts theBluetoothmodule in Inquiry and Page Scan mode. After the first connection requestappears, theDAU accepts it and enters Connection authentication state. In the Connectionauthentication state the DAU issues AuthenticationRequested HCI command. On LinkController’s Link_Key_Request the DAU sendsLink_Key_Negative_Reply in order toforce the Bluetooth module to generate the linkkey based on the supplied system accessPIN code. After a successful authenticationthe DAU enters the Data Processing state,otherwise it terminates the connection andenters the Waiting for connection state. Themain DAU operation takes place in the Data Processing state. Inthe state five main kindsof events are handled. Since the sensor data has to bedelivered on time to the centralsystem, data fetching is performed in high-priorityinterrupt handler procedure. Every 4msthe Jazz sensor raises the interrupt signalingthe data is ready for reading. The followingdata frame is used: Figure 6: Jazz Sensor frame format The preamble is used tosynchronize the beginning of the frame, EyeXrepresents the horizontal position of theeye, EyeY – vertical, AccX and AccY – the14. 14acceleration vectors along X and Y axes, PulsoOxy, Batt and Light –bloodoxygenation, voltage level and light intensity respectively. The figure belowshowsthe sensor communication timing. Figure 7: Jazz Sensor data fetching waveformThe received data is stored in an internal buffer; after the whole frameis completed theDAU encapsulates the data in an ACL frame and sends it over theBluetooth link. (Thefetching phase takes up approx. 192∝s (24 frames x 8∝s) and thesending phase takes at115200 bps approx. 2,8 ms, so the timing fits well in the 4mswindow.) In every stateremoving the ID card causes the device to enter the No IDcard state, terminating all theestablished connections. The second groups of events handled in the Data Processingstate aresystem messages and alerts. They are sent from the central system using theBluetoothlink. Since the communication also uses microcontrollers interrupt system theeventsare delivered instantly. The remaining time of the microcontroller is utilizedperforming LCDdisplay, checking the state of the buttons, ID card presence and batteryvoltage level.Depending on which button is pressed appropriate actions are launched. In
every stateremoving the ID card causes the device to enter the No ID card stateterminating allthe established connections. In the DAU there are two independent datasources-Jazz sensor andBluetooth Host Controller. Since they are both handled using theinterrupt system it isnecessary to decide which of the sources should have higher priority.Giving thesensor data the highest priority may result in losing some of the data sent bytheBluetooth module,as the transmission of the sensor data takes twice as much timeasreceiving one byte from UART. Missing one single byte sent from the Bluetoothcausesthe loss of control over the transmission. On the other hand, giving the15. 15Bluetooth the highest priority will make the DAU stop receiving the sensor datauntilthe Host Controller finishes its transmission. Central system alerts are the onlysignalsthat can appear during sensor data fetching after all the unimportant Bluetootheventshave been masked out. The best solution would be to make the centralunitsynchronize the alerts to be sent with the Bluetooth data reception. As thedeliveredoperating system is not a real-time system, the full synchronization is notpossible. As the Bluetooth module communicates asynchronously withthemicrocontroller there was a need of implementing a cyclic serial port buffer,featuringUART CTS/RTS flow control and a producer-consumer synchronizationmechanism.CENTRAL SYSTEM UNIT (CSU) CSU software is located on the deliveredComputer/System; in case oflarger resource demands the processing can be distributedamong a number of nodes.In this section we describe the four main CSU modules (seeFig. 1): ConnectionManager, Data Analysis, Data Logger and Visualization. The modulesexchange datausing specially designed single-producer multi consumer buffered thread-safe queues.Any number of consumer modules can register to receive the data suppliedby aproducer. Every single consumer can register at any number of producers,receivingtherefore different types of data. Naturally, every consumer may be a producerforother consumers. This approach enables high system scalability – new dataprocessingmodules (i.e. filters, data analyzers and loggers) can be easily added bysimplyregistering as a consumer.CONNECTION MANAGER Connection Manager’smain task is to perform low-level Bluetoothcommunication using Host.16. 16 Figure 8: Connection Manager Components Controller Interface commands. It isdesigned to cooperate with allavailable Bluetooth devices in order to support roaming.Additionally, ConnectionManager authorizes operators, manages their sessions,demultiplexes and buffers rawphysiological data. Figure 11 shows Connection ManagerArchitecture. Transport Layer Manager hides the details regarding actualBluetoothphysical transport interface (which can be either RS232 or UART or USBstandard)and provides uniform HCI command interface. Bluetooth Connection Manageris responsible for establishing andmaintaining connections using all available Bluetoothdevices. It periodically inquiresnew devices in an operating range and checks whetherthey are registered in thesystem database. Only with those devices the ConnectionManager will communicate.After establishing a connection an authentication procedureoccurs. The authenticationprocess is performed using system PIN code fetched from thedatabase. Once theconnection has been authenticated the mobile unit sends a data framecontaining theoperator’s identifier. Finally, the Connection Manager adds a SCO link(voiceconnection) and runs a new dedicated Operator Manager, which will manage thenewoperator’s session. Additionally, the Connection Manager maps the
operator’sidentifiers into the Bluetooth connections, so that when the operators roamaround the17. 17covered area a connection with an appropriate Bluetooth device is established andthedata stream is redirected accordingly. The data of each supervised operator is bufferedseparately in thededicated Operator Manager. At the startup it communicates with theOperator DataManager in order to get more detailed personal data. The most importantOperatorManager’s task is to buffer the incoming raw data and to split it into separatedatastreams related to each of the measured parameters. The raw data is sent to aLoggerModule, the split data streams are available for the other system modulesthroughproducer-consumer queues. Furthermore, the Operator Manager provides aninterfacefor sending alert messages to the related operator. Operator Data Managerprovides an interface to the operator databaseenabling the other modules to read or writepersonal data and system accessinformation.DATA ANALYSIS MODULE The moduleperforms the analysis of the raw sensor data in order toobtain information about theoperator’s physiological condition. The separatelyrunning Data Analysis Modulesupervises each of the working operators. The moduleconsists of a number of smalleranalyzers extracting different types of information.Each of the analyzers registers at theappropriate Operator Manager or anotheranalyzer as a data consumer and, acting as aproducer, provides the results of theanalysis. An analyzer can be either a simple signalfilter (e.g. Finite Input Response(FIR) filter) or a generic data extractor (e.g. signalvariance, saccade detector) or acustom detector module. As it is not able to predict all thesupervisors’ needs, thecustom modules are created by applying a supervised machinelearning algorithm to aset of earlier recorded examples containing the characteristicfeatures to berecognized. In the prototype we used an improved C4.5 decision treeinductionalgorithm. The computed features can be e.g. the operator’s position(standing,walking and lying) or whether his eyes are closed or opened.18. 18 As built-in analyzer modules we implemented a saccade detector,visual attentionlevel, blood oxygenation and pulse rate analyzers. The saccade detector registers as aneye movement and accelerometersignal variance data consumer and uses the data tosignal saccade occurrence. Sincesaccades are the fastest eye movements the algorithmcalculates eye movementvelocity and checks physiological Constraints. The algorithmhas two main steps: User adjustment step. The phase takes up 5 s. After bufferingapprox. 5 s of the signal differentiate it using three point central difference algorithm,which will give eye velocity time series. Sort the velocities by absolute value andcalculate upper 15% of the border velocity along both X – v0x and Y – v0y axes . As aresult v0x and On-line analyzer flow. Continuously v0y are cut-off velocities. calculateeye movement velocity using three point central difference algorithms. If the velocityexcess pre calculated v0 (both axes are considered separately) there is a possibility ofsaccade occurrence. Check the following conditions (if any of them is satisfied do notdetect a saccade): • the last saccade detection was less than 130 ms ago (physiologicalconstraint – the saccades will not occur more frequently) • the movement is nonlinear(physiological constraint) • compare the signal with accelerometer (rapid head movementmay force eye activity of comparable speed) • if the accelerometer signal is enormouslyuneven consider ignoring the signal due to possible sensor device movements. If none ofthe above conditions is satisfied – signal the saccade occurrence. The visual attention
level analyzer uses as input the results producedby the saccade detector. Low saccadicactivity (large delays between subsequent19. 19saccades) suggests lowered visual attention level (e.g. caused bythoughtfulness).Thus, we propose a simple algorithm that calculates the visual attentionlevel (Lva):Lva = 100/ts10, where ts10 denotes the time (in seconds) occupied by the lasttensaccades. Scientific research has proven  that during normal visualinformationintake the time between consecutive saccades should vary from 180 up to 350ms.this gives Lva at 28 up to 58 units. The values of Lva lower than 25 for a longerperiodof time should cause a warning condition. The following figure shows thesituationwhere the visual attention lowers for a few seconds. Figure 9: Saccadeoccurrence and Visual attention level20. 20 The Pulse rate analyzer registers for the oxyhemoglobin anddeoxyhemoglobinlevel data streams. Since both signals contain a strong sinusoidalcomponent related toheartbeat, the pulse rate can be calculated measuring the timedelay between subsequentextremes of one of the signals. We decided not to processonly one of the data streams –the algorithm is designed to choose dynamically one ofthem on the grounds of the signallevel. Unfortunately, the both signals are noised sothey must be filtered before furtherprocessing. We considered a number of differentalgorithms and decided to implementaverage value based smoothing. More detaileddiscussion is presented in section 3.3.5Tradeoffs and Optimization. The algorithmconsists in calculating an average signal valuein a window of 100 samples. In everystep the window is advanced 5 samples in order toreduce CPU load. This approachlowers the sampling rate from 250 Hz down to 50 Hz.However, since the Visualheartbeat frequency is at most 4 Hz the Nyquist conditionremains satisfied. Thefigures show the signal before (Fig. 10a) and after filtering (Fig10b). Figure: 10(a) Figure: 10(b) After filtering the signal the pulse calculation algorithmis applied. Thealgorithm chooses the point to be the next maximum if it satisfies threeconditions:points on the left and on the right have lower values, the previous extreme wasaminimum, and the time between the maximums is not too short(physiologicalconstraint). The new pulse value is calculated based on the distancebetween the newand the previous maximum detected. The algorithm gets the last 5calculated pulsevalues and discards 2 extreme values to average the rest. Finally, it doesthe same withthe minimums of the signal to obtain the second pulse rate value, whichgives the finalresult after averaging.21. 21 Additionally, we implemented a simple module that calculates averagebloodoxygenation level. Despite its simplicity the parameter is an important measureof theoperator’s physiological condition. The other signal features that are not recognized bythe built-inanalyzers can be extracted using custom modules created by Decision TreeInductionmodule. The custom module processes the generated decision tree, registersforneeded data streams and produces the desired output signal. Decision Tree Inductionmodule generates the decision trees, which arebinary trees with an attribute test in eachnode. The decision tree input data is anobject described by means of a set of attribute-value pairs. The algorithm is not able toprocess time series directly. The attributestherefore are average signal value, signalvariance and the strongest sinusoidalcomponents. As an output the decision treereturns the category the object belongs to. Inthe Decision Tree Induction module wemainly use C 4.5 algorithm , but also proposeour own modifications. Thealgorithm is a supervised learning from examples i.e. it
considers both attributes thatdescribe the case and a correct answer. The main idea is touse a divide-and-conquerapproach to split the initial set of examples into subsets using asimple rule (i-thattribute less than a value). Each division is based on entropy calculation– thedistribution with the lowest entropy is chosen. Additionally, we proposemanymodifications concerning some steps of the algorithm and further exploration ofthesystem. For each case to be classified C 4.5 traverses the tree until reaching theleafwhere appropriate category id is stored. To increase the hit ratio our system usesmoreadvanced procedure. For single analysis we develop a group of k trees (where kis aparameter), which we call a decision forest. Initial example set S is dividedrandomly intok+1 subsets S0 ... Sk. S0 is left to test the whole decision forest. Eachtree is inducedusing various modifications of the algorithm to provide results’independence. Each i-thtree is taught using SS0Si set (S without S0 and Si sets) andtested with Si that estimates asingle tree error ratio. Furthermore we extract all22. 22wrongly classified examples and calculate correlation matrix between each pair ofthetrees. In an exploring phase we use unequal voting rule – each tree has a voteofstrength of its reliability. Additionally, if two trees give the same answer their voteisweakened by the correlation ratio. Alarm Dispatcher Module is a very important part ofthe Data Analysismodule. It registers for the results of the data analysis, checks themwith regard to theuser-defined alarm conditions and launches appropriate actions whenneeded. Themodule is a producer of the alarm messages, so that they are accessible in theloggerand visualization modules.DATA LOGGER MODULE The module providessupport for storing the monitored data in order toenable the supervisor to reconstruct andanalyze the course of the operator’s duty. Themodule registers as a consumer of the datato be stored in the database. Each workingoperator’s data is recorded by a separateinstance of the Data Logger. Apart from theraw or processed physiological data, alertsand operator’s voice are stored. The rawdata is supplied by the related Operator Managermodule, whereas the Data Analysismodule delivers the processed data. The voice data isdelivered by a Voice DataAcquisition module. The module registers as an operator’svoice data consumer andoptionally processes the sound to be stored (i.e. reduces noise orremoves thefragments when the operator does not speak). The Logger’s task is to addappropriatetime stamps to enable the system to reconstruct the voice. Additionally, thereis a dedicated video data logger, which records thedata supplied by the Video DataAcquisition module (in the prototype we use JPEGcompression). The module is designedto handle one or more cameras using Video forWindows standard. The Data Logger isable to use any ODBC-compliant databasesystem. In the prototype we used MS SQLServer, which is a part of the Project Kit.VISUALIZATION MODULE23. 23 The module provides user interface for the supervisors. It enables themto watcheach of the working operator’s physiological condition along with a previewof selectedvideo source and his related sound stream. All the incoming alarmmessages are instantlysignaled to the supervisor. Moreover, the visualization modulecan be set in the off-linemode, where all the data is fetched from the database.Watching all the recordedphysiological parameters, alarms, video and audio data thesupervisor is able toreconstruct the course of the selected operator’s duty.TOOLS USED TO DEVELOPBLUE EYES In creating the hardware part of the DAU a development board wasbuilt,which enabled to mount, connect and test various peripheral devices cooperatingwith themicrocontroller. During the implementation of the DAU there was a need fora piece of
software to establish and test Bluetooth connections. Hence created a toolcalled BlueDentist. The tool provides support for controlling thecurrently connected Bluetoothdevice. Its functions are: Local device management(resetting, reading local BD_ADDR,putting in Inquiry/Page and Inquiry/Page scanmodes, reading the list of locally supportedfeatures and setting UART speed) andconnection management (receiving and displayingInquiry scan results, establishingACL links, adding SCO connections, performing linkauthorization procedure,sending test data packets and disconnecting).24. 24 Fig: Blue Dentist To test the possibilities and performance of the remaining partsof theProject Kit (computer, camera and database software) Blue Capture(Fig. 12) wascreated. The tool supports capturing video data from various sources(USB web-cam,industrial camera) and storing the data in the MS SQL Serverdatabase. Additionally, theapplication performs sound recording. After filtering andremoving insignificantfragments (i.e. silence) the audio data is stored in the database.Finally, the program playsthe recorded audiovisual stream. They used the software tomeasure database systemperformance and to optimize some of the SQL queries (e.g.we replaced correlated SQLqueries with cursor operations). Figure 12: BlueCapture Also a simple tool for recordingJazz Multisensory measurements wasintroduced. The program reads the data using aparallel port and writes it to a file. Toprogram the operator’s personal ID card we use astandard parallel port, as theEEPROMs and the port are both TTL-compliant. A simpledialog-based applicationhelps to accomplish the task. APPLICATIONS OF BLUE EYETECHNOLOGY25. 25MANUAL AND GAZE INPUT CASCADED (MAGIC) POINTINGThis workexplores a new direction in utilizing eye gaze for computer input. Gazetracking has longbeen considered as an alternative or potentially superior pointingmethod for computerinput. We believe that many fundamental limitations exist withtraditional gaze pointing.In particular, it is unnatural to overload a perceptual channelsuch as vision with a motorcontrol task. We therefore propose an alternativeapproach, dubbed MAGIC (Manual AndGaze Input Cascaded) pointing. With suchan approach, pointing appears to the user to bea manual task, used for finemanipulation and selection. However, a large portion of thecursormovement is eliminated by warping the cursor to the eye gaze area,whichencompasses the target. Two specific MAGIC pointing techniques,oneconservative and one liberal, were designed, analyzed, and implementedwith an eyetracker developed. They were then tested in a pilot study.This early stage explorationshowed that the MAGIC pointing techniquesmight offer many advantages, includingreduced physical effort and fatigueas compared to traditional manual pointing, greateraccuracy and naturalnessthan traditional gaze pointing, and possibly faster speed thanmanualpointing. The pros and cons of the two techniques are discussed in light ofbothperformance data and subjective reports.IMPLEMENTATIONThe MAGIC pointingprogram takes data from both themanual input device (of any type, such as a mouse) andthe eye trackingsystem running either on the same machine or on anothermachineconnected via serial port. Raw data from an eye tracker can not be directlyusedfor gaze-based interaction, due to noise from image processing, eyemovement jitters, andsamples taken during saccade (ballistic eyemovement) periods. We experimented withvarious filtering techniques andfound the most effective filter in our case is similar to thatdescribed. Thegoal of filter design in general is to make the best compromisebetweenpreserving signal bandwidth and eliminating unwanted noise. In the case of
26. 26eye tracking, as Jacob argued, eye information relevant to interaction lies inthefixations. The key is to select fixation points with minimal delay.Samples collectedduring a saccade are unwanted and should be avoided. Indesigning our algorithm forpicking points of fixation, we considered ourtracking system speed (30 Hz), and that theMAGIC pointing techniquesutilize gaze information only once for each new target,probablyimmediately after a saccade. Our filtering algorithm was designed to pickafixation with minimum delay by means of selecting two adjacent points overtwosamples.ARTIFICIAL INTELLIGENT SPEECH RECOGNITIONIt is important toconsider the environment in which thespeech recognition system has to work. Thegrammar used by the speakerand accepted by the system, noise level, noise type, positionof themicrophone, and speed and manner of the user’s speech are some factorsthat mayaffect the quality of speech recognition .When you dial thetelephone number of a bigcompany, you are likely to hear the sonorousvoice of a cultured lady who responds toyour call with great courtesysaying ―Welcome to company X. Please give me theextension number youwant‖. You pronounce the extension number, your name, and thename ofperson you want to contact. If the called person accepts the call, theconnection isgiven quickly. This is artificial intelligence where an automaticcall-handling system isused without employing any telephone operator.THE TECHNOLOGYArtificialintelligence (AI) involves two basic ideas. First,it involves studying the thought processesof human beings. Second, it dealswith representing those processes via machines (likecomputers, robots, etc).AI is behavior of a machine, which, if performed by a humanbeing, wouldbe called intelligent. It makes machines smarter and more useful, and islessexpensive than natural intelligence. Natural language processing (NLP)27. 27refers to artificial intelligence methods of communicating with a computerin anatural language like English. The main objective of a NLP program isto understandinput and initiate action. The input words are scanned andmatched against internallystored known words. Identification of a key wordcauses some action to be taken. In thisway, one can communicate with thecomputer in one’s language. No special commands orcomputer languageare required. There is no need to enter programs in a speciallanguageforcreating software.SPEECH RECOGNITIONThe user speaks to the computerthrough a microphone,which, in used; a simple system may contain a minimum of threefilters. Themore the number of filters used, the higher the probability ofaccuraterecognition. Presently, switched capacitor digital filters are used becausethesecan be custom-built in integrated circuit form. These are smaller and cheaperthan activefilters using operational amplifiers. The filter output isthen fed to the ADC to translatethe analogue signal into digital word. TheADC samples the filter outputs many times asecond. Each samplerepresents different amplitude of the signal .Evenly spaced verticallinesrepresent the amplitude of the audio filter output at the instant of sampling.Eachvalue is then converted to a binary number proportional to theamplitude of the sample. Acentral processor unit (CPU) controls the inputcircuits that are fed by the ADCS. A largeRAM (random access memory)stores all the digital values in a buffer area. This digitalinformation,representing the spoken word, is now accessed by the CPU to processitfurther. The normal speech has a frequency range of 200 Hz to 7 kHz.Recognizing atelephone call is more difficult as it has bandwidth limitationof 300 Hz to3.3 kHz.Asexplained earlier, the spoken words are processedby the filters and ADCs. The binary
representation of each of these wordsbecomes a template or standard, against which thefuture words arecompared. These templates are stored in the memory. Once the storing28. 28process is completed, the system can go into its active mode and is capableofidentifying spoken words. As each word is spoken, it is converted intobinary equivalentand stored in RAM. The computer then starts searchingand compares the binary inputpattern with the templates. t is to be notedthat even if the same speaker talks the sametext, there are always slightvariations in amplitude or loudness of the signal, pitch,frequencydifference, time gap, etc. Due to this reason, there is never a perfectmatchbetween the template and binary input word. The pattern matching processthereforeuses statistical techniques and is designed to look for the best fit.The values of binaryinput words are subtractedfrom the corresponding values in the templates. If both thevalues are same,the difference is zero and there is perfect match. If not, thesubtractionproduces some difference or error. The smaller the error, the better thematch.When the best match occurs, the word is identified and displayed onthe screen or used insome other manner. The search process takes aconsiderable amount of time, as the CPUhas to make many comparisonsbefore recognition occurs. This necessitates use of veryhigh-speedprocessors. A large RAM is also required as even though a spoken wordmaylast only a few hundred milliseconds, but the same is translated into manythousandsof digital words. It is important to note that alignment of wordsand templates are to bematched correctly in time, before computing thesimilarity score. This process, termed asdynamic time warping, recognizesthat different speakers pronounce the same words atdifferent speeds as wellas elongate different parts of the same word. This is important forthespeaker-independent recognizers.SIMPLE USER INTERST TRACKER (SUITOR)Computers would have been much more powerful,had they gained perceptual andsensory abilities of the living beings on theearth. What needs to be developed is anintimate relationship between thecomputer and the humans. And the Simple User InterestTracker (SUITOR)29. 29is a revolutionary approach in this direction. By observing the Webpage a netizenisbrowsing, the SUITOR can help by fetching more information at his desktop. Bysimplynoticing where the user’s eyes focus on the computer screen, the SUITOR canbe moreprecise in determining his topic of interest. It can even deliver relevantinformation to ahandheld device. The success lies in how much the suitor can beintimate to the user.IBMs BlueEyes research project began with a simplequestion, according to MyronFlickner, a manager in Almadens USERgroup: Can we exploit nonverbal cues to createmore effective userinterfaces? One such cue is gaze—the direction in which a personislooking. Flickner and his colleagues have created some new techniques fortracking apersons eyes and have incorporated this gaze-tracking technologyinto two prototypes.One, called SUITOR (Simple User Interest Tracker),fills a scrolling ticker on a computerscreen with information related to theusers current task. SUITOR knows where you arelooking, whatapplications you are running, and what Web pages you may be browsing."IfIm reading a Web page about IBM, for instance," says Paul Maglio, theAlmadencognitive scientist who invented SUITOR, "the system presentsthe latest stock price orbusiness news stories that could affect IBM. If I readthe headline off the ticker, it pops upthe story in a browser window. If I startto read the story, it adds related stories to theticker. Thats the whole idea ofan attentive system—one that attends to what you aredoing, typing, reading,so that it can attend to your information needs." CONCLUSION
The BlueEyes system is developed because of the need for areal-time monitoring systemfor a human operator. The approach is innovative since ithelps supervise the operator notthe process, as it is in presently available solutions.We hope the system in its commercialrelease will help avoid potential threatsresulting from human errors, such as weariness,oversight, tiredness or temporalindisposition. However, the prototype developed is agood estimation of thepossibilities of the final product. The use of a miniature CMOScamera integrated intothe eye movement sensor will enable the system to calculate thepoint of gaze and30. 30observe what the operator is actually looking at. Introducing voicerecognitionalgorithm will facilitate the communication between the operator and thecentralsystem and simplify authorization process. Despite considering in the report onlythe operators working in controlrooms, our solution may well be applied to everyday lifesituations. Assuming theoperator is a driver and the supervised process is car driving it ispossible to build asimpler embedded on-line system, which will only monitor consciousbraininvolvement and warn when necessary. As in this case the logging moduleisredundant, and the Bluetooth technology is becoming more and more popular,thecommercial implementation of such a system would be relatively inexpensive. Thefinal thing is to explain the name of our system.BlueEyes emphasizes the foundations ofthe project – Bluetoothtechnology and the movements of the eyes. Bluetooth providesreliable wirelesscommunication whereas the eye movements enable us to obtain a lot ofinterestingand important information.31. 31 REFERENCES1. Carpenter R. H. S., Movements of the eyes, 2nd edition, PionLimited, 1988,London.2. Bluetooth specification, version 1.0B, Bluetooth SIG, 1999.3.ROK 101 007 Bluetooth Module, Ericsson Microelectronics,2000.4. AT89C52 8-bitMicrocontroller Datasheet, Atmel.5. Intel Signal Processing Library –Reference Manual.