152 Berka et al. ognition memory test. Across tasks and participants, specific changes in the EEG were identified that were reliably associated with levels of cognitive workload. The EEG in- dexes were also shown to change as a function of training on the WCT and the learn- ing and memory task. Future applications of the system to augment cognition in mili- tary and industrial environments are discussed.1. INTRODUCTIONThe technical complexity and 24-hr schedule of contemporary industrial and mili-tary operations increasingly demand rapid skill acquisition and the ability to sus-tain high levels of performance for extended periods of time (Moore–Ede, 1993). Ef-ficient and reliable interfaces between human operators and machines are requiredto facilitate the integration and adoption of sophisticated technologies. With in-creased automation, the operator’s role is often simply to monitor, and maintainingvigilance becomes more difficult, with performance decrements increasing withtime-on-task (Parasuraman, Molloy, & Singh, 1993; Singh, Molloy, & Parasuraman,1993). The integration of physiological monitoring into the man–machine interfaceholds great promise both for real-time assessment of operator status and for pro- pyviding a means to allocate tasks between machines and humans based on the oper-ator status. Once meaningful real-time monitoring is achieved, intelligent feedback oor “closed-loop” systems can facilitate active intervention by the operator orthrough a third party, increasing safety, efficiency, and productivity (Parasuraman,Mouloua, & Molloy, 1996; Singh et al., 1993). o tCBahri, Deaton, Morrison, & Barnes, 1992; Parasuraman et al., 1993; Parasuraman, Research conducted over the past 40 years has established electroencephalogra- Nphy (EEG) and event-related potentials (ERPs) as the primary tools available to sci- oentists investigating neural indexes of cognition (Fabiani, Gratton, & Coles, 2000).Characteristic changes in the EEG and ERPs that reflect levels of alertness and Ddrowsiness, selective attention, workload, memory, and executive function havebeen identified (Akerstedt & Folkard, 1997; Fabiani et al., 2000; Gevins et al., 1996;Gevins et al., 1998; Gevins, Smith, McEvoy, & Yu, 1997; Hillyard, Hink, Schwent, &Picton, 1973; Polich, 2002; Polich & Herbst, 2000). EEG measures have been appliedin areas as diverse as human factors, lie detection, pharmacological research, andassessment of cognitive dysfunction caused by neurological disorders (Fabiani etal., 2000). EEG indexes of real-time cognitive state changes offer the capability ofproviding real-time input to adaptively automated systems (Parasuraman,Mouloua, & Hilburn, 1999). This article presents an integrated hardware and soft-ware solution for acquisition and real-time analysis of EEG to monitor indexes ofalertness, cognition, and memory. The system will translate the data into interpret-able feedback for state modification by the operator, an observer, or a machine. EEG has traditionally been confined to laboratory settings due to the technicalobstacles of recording high-quality data and the computational demands ofreal-time analysis. A reasonable expectation for success with ambulatory EEG ap-plications in operational environments is that the setup and acquisition of highquality recordings can be obtained by the user without technical assistance. A re-
Real-Time EEG Indexes of Cognition 153cently developed wireless EEG sensor headset achieves this goal by combiningbattery-powered hardware with a sensor placement system to provide a light-weight, easy-to-apply method to acquire and analyze six channels of high-qualityEEG (Figure 1). The EEG sensor headset requires no scalp preparation and pro-vides a comfortable and secure sensor-scalp interface for 12 to 24 hr of continu-ous use. The headset was designed with fixed sensor locations for three sizes(e.g., small, medium, and large). Sensor placement was determined using a data-base of over 225 participants so that each sensor is no more than one centimeterfrom the international 10 to 20 system coordinates. The workload studies de-scribed in this article required only three EEG channels. The remaining channelsare utilized to monitor other cognitive states including attention, learning, andmemory. Amplification, digitization, and radio frequency (RF) transmission of the signalsare accomplished with miniaturized electronics in a portable unit worn on thehead. The combination of amplification and digitization of the EEG close to the sen-sors and wireless transmission of the data facilitates the acquisition of high-qualitysignals even in high electromagnetic interference environments. Quantification of the EEG in real time, referred to as the B-Alert® system, isachieved using signal analysis techniques to identify and decontaminate fast and pyslow eye blinks, and identify and reject data points contaminated withelectromyography (EMG), amplifier saturation, or excursions due to movement ar- otifacts. Each 1-sec EEG epoch is then classified into one of four states of alertness:“high vigilance,” “low vigilance,” “relaxed wakefulness,” and “sleepy.” These four tCstates were empirically derived using EEG acquired from individuals participatingin sleep deprivation studies. The high and low vigilance states were modeled by ovarying the level of task engagement. Relaxed wakefulness is the state induced Nwhen participants are instructed to relax with eyes closed and is generally charac- oterized by predominance of EEG in the alpha frequency band (8–12 Hz.). Data forthe sleepy class were obtained using EEG samples acquired just subsequent tosleep onset. D The classification model utilizes discriminant function analysis derived from alarge normative database and is fitted to each individual’s unique EEG patterns FIGURE 1 Wireless EEG sensor headset: (a) Front view, (b) Rear view, and (c) Aug- mented-Cognition Technical Integration Experiment participant.
154 Berka et al.with data acquired from three baseline conditions. The B-Alert® model was devel-oped and evaluated for its capability to provide early warnings of the onset ofdrowsiness. The B-Alert® system was validated in sleep deprivation studies withperformance in a driving simulator (Levendowski, Berka, Olmstead, & Jarvik,1999), accuracy and reaction time during a psychomotor vigilance task, behavioralevidence as measured by cessation of finger tapping, visually scored observationsof facial signs of drowsiness (eye closures, head nods), and responses to a subjec-tive sleepiness questionnaire (Levendowski et al., 2001; Levendowski, Olmstead,Konstantinovic, Berka, & Westbrook, 2000). The B-Alert® model was independ-ently validated with visual inspection of the EEG signals and observations of facialsigns of drowsiness conducted by two board-certified sleep specialists(Levendowski et al., 2000; Mitler et al., 2002). The model was demonstrated to be ef-fective in characterizing excessive daytime drowsiness in patients with sleep apnea(Westbrook et al., 2002). Analysis of the B-Alert® indexes during 44 hr of sleep de-privation revealed that changes in the indexes could predict performance deficits(Mitler et al., 2002) and confirmed the previously reported observation (Balkin,2001; Doran, Van Dongen, & Dinges, 2001) that individuals differ in their vulnera-bility to sleep deprivation. Sleep deprivation studies (Levendowski et al., 2000;Mitler et al., 2002) revealed that highly engaging or difficult tasks induce higher pyoverall levels of vigilance as measured by B-Alert®, suggesting that the B-Alert® in-dexes may have utility in monitoring cognitive workload. o Other investigators have reported EEG measures of workload that reflected dif-ferences in task-related cognitive resource allocation, task mastery, and task over- tCload (Byrne & Parasuraman, 1996; Kramer, 1991; Pope, Bogart, & Bartolome, 1995;Prinzel, Freeman, Scerbo, Mikulka, & Pope, 2000; Sterman, 1995). The EEG vari- oables employed in these models to monitor workload included alpha suppression, Nincreased beta, increased frontal midline theta, and ratios such as beta–alpha plus otheta and alpha plus theta–beta. The B-Alert® model incorporates informationfrom the alpha, beta, and theta bands in assigning a class to each 1-sec of EEG. It Dwas hypothesized that increasing the workload would result in increasing levels ofhigh vigilance as measured by the percentage of B-Alert® high vigilance. The studies presented in this article were conducted to evaluate the efficacyof the sensor headset and B-Alert® system in monitoring mental workload dur-ing complex cognitive tasks for the Defense Advance Research Projects Agency(DARPA) Augmented Cognition (Aug-Cog) program. Figure 1c presents a par-ticipant during a typical Aug-Cog data acquisition session with the EEG sensorheadset, eye tracking and pupillometry (Marshall, St. John, Knust, & Binning,2003), and functional near infrared imaging headgear (Onaral, 2003). Three cog-nitive test paradigms were employed to evaluate workload measures. The War-ship Commander Task (WCT) simulated a multitasking Navy command andcontrol environment and allowed workload levels to be systematically manipu-lated (St. John, Kobus, & Morrison, 2002). A cognitive task with three levels ofdifficulty provided consistent sensory inputs and motor outputs while varyingworkload levels (Mathiak et al., 2002). Image Learning and Recognition Mem-ory Tests (Levendowski et al., 2002) were utilized to evaluate the B-Alert® in-dexes during learning and recognition memory.
Real-Time EEG Indexes of Cognition 1552. METHODS2.1. Acquisition Hardware SystemThe sensor headset acquires six channels of EEG or electrooculography (EOG) us-ing either a unipolar or bipolar montage. Data are sampled at 256 samples per sec-ond with a bandpass from 0.5 Hz and 65 Hz (at 3 dB attenuation) obtained digitallywith Sigma-Delta A/D converters. The RF link is frequency-modulated to transmitat a rate of 57 kBaud in the 915 MHz ISM band. When utilized in the bidirectionalmode, the firmware allows the host computer to initiate impedance monitoring ofthe sensors, select the transmission channel (so two or more headsets can be used inthe same room), and monitor battery power of the headset. Data are acquiredacross the RF link on a host computer via an RS232 interface. Data acquisition soft-ware stores the EEG data and synchronizes event codes generated by the cognitivetasks in the EEG record for ERP analysis. For all of the results presented later, thestandard hardware montage includes bipolar recordings from Fz to POz and Cz toPOz (required for the B-Alert® system), unipolar recordings from Fz, Cz, and POzreferenced to linked mastoids (available for ERP analysis), and a bipolar configura-tion for horizontal and vertical EOG (to confirm the accuracy of the eye blink iden- pytification and decontamination algorithm). o2.2. Artifact Identification and Decontamination, and Signal Processing o tCArtifacts automatically detected and decontaminated in the time-domain EEG sig-nal include 3, 5, or 7 data point spikes with amplitudes greater than 40 mV (caused Nby tapping or bumping of the sensors), amplifier saturation, and excursions that ooccur during the onset or recovery of saturations. For each of these artifacts, datapoints with 0 µ V are inserted, starting at the last zero crossing prior to and ending Dat the first zero crossing after the artifact. A 60 Hz notch filter is applied to all EEGdata. Three sets of filtered EEG data are then derived using a 0.5 Hz 256th orderhigh-pass FIR filter, a 4 Hz 640th order FIR high-pass filter, and a 7 Hz IIR low-passfilter. To obtain faster computations, both high-pass filters are realized by subtract-ing the output of the corresponding low-pass filter from the original signal. Identification of eye blinks in the EEG without the use of a reference EOG chan-nel is achieved by filtering the fast component of the FzPOz channel with a 7 Hz IIRlow-pass filter, applying cross-correlation analysis to the filtered signal using thepositive half of a 40 µ V 1.33 Hz sine wave as the target shape, and applying thresh-olds to the outputs from the cross-correlation analysis. Minima and maxima analy-sis in each direction from the point of maximum correlation is used to identify thedata points corresponding to the range between the start and end of each eye blink.Once eye blink ranges have been determined, the 0.5 Hz high-pass filtered EEG sig-nal from each channel is decontaminated by replacing the data points in the eyeblink region with the corresponding data after application of the 4 Hz filter. Decontaminated EEG is then segmented into overlapping 256 data-point win-dows called overlays. An epoch consists of three consecutive overlays. Fast-Fourier
156 Berka et al.transform is applied to each overlay of the decontaminated EEG signal multipliedby the Kaiser window (α = 6.0) to compute the power spectral densities (PSD). ThePSD values are adjusted to take into account zero values inserted for artifact-con-taminated data points. The PSD between 70 and 128 Hz is used to detect EMG arti-fact. Overlays with excessive EMG artifact (“EMG”) or with fewer than 128 datapoints (“missing data”) are rejected. The remaining overlays are then averaged toderive PSD for each epoch with a 50% overlapping window. Epochs with two ormore overlays with EMG or missing data are classified as invalid. For each channel,PSD values are derived for each 1-Hz bin (“bin”) from 3 Hz to 40 Hz and the totalPSD from 3 Hz to 40 Hz (“band”).2.3. Classification ModelThe four-class model was developed with data from three 5-min baseline condi-tions (i.e., baseline conditions = finger-tapping with eyes open [EO] and eyesclosed [EC] and three-choice psychomotor vigilance task [PVT]) and sleepy epochsselected from sleep-deprivation data using a database of 150 healthy participants.For each epoch from these four conditions, five variables were computed for each py1-Hz bin between 3 Hz and 40 Hz (5 variables × 38 bins) for each channel: thelogged PSD, the relative power compared to the total power between 3 Hz and 40 oHz, and the z scores for EO, EC, and PVT. The z scores were computed using themeans and standard deviations from each of the three baseline conditions. The tCidentification of fast blinks was also used as a predictive variable. A total of 381variables were available for each epoch based on a two-channel classification omodel (i.e., 190 variables each for FzPOz and CzPOz, plus fast-blink). N The variables from each artifact-free epoch for the four conditions were submit- oted to stepwise analysis to select those variables most predictive in a four-classmodel (high vigilance, low vigilance, relaxed wakefulness, and sleepy). A total of D19 variables were selected. The most predictive variables were as follows: (a) the zscore of the 10 Hz bin from CzPOz relative to the PVT task (partial r2 = 0.46), (b) thepresence of a fast blink (partial r2 = 0.09), and (c) the z score of 11 Hz from CzPOzrelative to eyes closed (partial r2 = 0.06). The r2 for each of the remaining 16 vari-ables was less than 0.03. The approach developed for the B-Alert® system was to utilize population datato establish the underlying model and then refine the discriminant function by ad-justing for individual differences in the EEG using data from the three baseline con-ditions. Although baseline data could be readily acquired for development of theclassification models for new individuals, sleep data could not be obtained a priori.Rather, the mean values of all variables from the three baseline conditions (190 vari-ables × two channels × three conditions) for all participants in the database weresubmitted to stepwise linear regressions to derive equations to predict thediscriminant function coefficients for the “sleepy” classification for the each of the19 predictive variables. Matrices were then derived using the aforementioned anal-yses to fit the four-class model to the individual and compute the probability of cor-
Real-Time EEG Indexes of Cognition 157rect classification into each of the four output classes on a second-by-second basisoffline or in real time.2.4. Warship Commander TaskThe WCT requires the user to monitor groups (“waves”) of incoming aircraft(“tracks”), to identify tracks as friendly or hostile, and to warn and then destroyhostile tracks. Each wave is 75 sec in duration with workload level manipulated byincreasing numbers of tracks per wave (6, 12, 18, or 24 tracks). The learning curvefor WCT is steep due to the complexity of task demand. Performance measurescomputed for each wave, including reaction times to identify, warn, and destroytracks and total game scores (as a percentage of possible points), correlate directlywith workload levels (number of tracks) and can be used as behavioral measures ofparticipant workload (St. John et al., 2002). Of a total of 15 “novices” evaluated in an initial study, one was eliminated fromthe study for fatigue and another for failing to reach proficiency on WCT, leaving atotal of 13 participants in the analysis. Novices were given 35 min of training toachieve sufficient proficiency to complete the basic protocol that included three pywaves of 6, 12, and 18 tracks. As part of the DARPA Augmented Cognition Techni-cal Integration Experiment (TIE), 10 participants were studied with additional lev- oels of WCT training, ranging from 1 hr to over 300 hr. This group of 10 participants tCcompleted multiple sessions with 3 waves of 6, 12, 18, and 24 tracks (presented inthe order 6, 18, 12, 24). o N2.5. Three-Level Cognitive Task o DOne of the limitations of the WCT in validating a cognitive workload measure wasthe fact that the number of stimuli and the amount of required motor activity (mouseand keypress) varied directly as a function of the “workload” manipulations. Toevaluate the B-Alert® EEG indexes without the sensory and motor confounds associ-ated with workload levels in the WCT, a three-level cognitive task, developed as partof the DARPAAugmented Cognition project (Mathiak et al., 2002), was evaluated in16 healthy participants. In this task, both the stimuli and motor demands are keptconstant during three levels of increasing task difficulty (easy, moderate, hard) byemploying the same number of stimuli and targets for each level of difficulty. Foreach level of difficulty, 250 trials of single integers between 1 and 8 are presented witha 1.6-sec Interstimulus Interval (ISI). The order of the digit presentation is identicalfor each of the levels of difficulty to maintain consistent visual inputs. For level 1(easy), participants are instructed to press the space bar with the index finger of bothhands only when they see the number 5. In level 2 (moderate), participants respondonly after any three consecutive even numbers and for level 3 (hard), they respondonly to a number the same as the number 2 trials earlier (2-back task). Performance
158 Berka et al.measures, including reaction times and percentage of correct responses, were com-puted to verify the levels of task difficulty.2.6. Image Learning and Recognition Memory TaskThe recognition memory task requires participants to memorize 20 images, andthen recognize those images when randomly interspersed among 80 new images ofthe same category (e.g. animals, food, travel, sports). The categories were selectedand evaluated for equivalence in test performance across categories. During thetraining period, each of the 20 images is presented sequentially twice for 1.25 secwith a 1.5-sec ISI. During the testing period, each of the 100 images is presented for100 msec. with a 2.1-sec ISI, while the participant identifies the training and newimages. Each participant completed two recognition memory sessions with twodifferent image categories.2.7. Participants pyA total of 45 fully-rested, healthy individuals (ages 18–50) participated in the WCTnovice (n = 13), three-level cognitive task (n = 16), Image Learning and Recognition oMemory Task (n = 19), and DARPA Augmented Cognition TIE (n = 10) studies.Thirteen participants took part in both the WCT novice and three-level cognitive tCtask All studies were conducted at the Advanced Brain Monitoring laboratory inCarlsbad, CA, between the hours of 9 a.m. and 2 p.m. with the exception of the oDARPA Augmented Cognition TIE, conducted at Pacific Science and Engineeringin San Diego, CA. N o3. RESULTS D3.1. WCT NovicesB-Alert® percentage high vigilance (HV) and WCT performance measures wereaveraged across the three waves for each of the workload levels (i.e., 6, 12, 18tracks) by and across participants. A repeated-measures analysis of variance(ANOVA) applied across the three WCT workload levels revealed an increasingpercentage of HV classifications as a result of increasing workload with a signifi-cant main effect for workload (F = 7.369, p < 0.005; Figure 2). Significance levelsreflect an adjustment to the degrees of freedom using the Greenhouse–Geisserprocedure to correct for violations of the sphericity assumption in repeated-mea-sures designs, when appropriate. Post hoc comparisons between the easy, moderate, and hard levels of the WCTrevealed significant differences for easy versus moderate (F = 5.095, p < 0.05) andeasy versus hard (F = 10.423, p < 0.01), with the moderate versus hard (F = 4.074, p <0.066) approaching significance. As expected, repeated measures ANOVAs across
Real-Time EEG Indexes of Cognition 159 FIGURE 2 Mean ± SE percentage of EEG epochs classified as high vigilance (% HV) dur- ing WCT in 13 novices for easy, moderate, and hard levels (6, 12, and 18 tracks, respectively).the three levels showed significant increases in the reaction time to identify tracks(WCT-Rtiff; F = 88.913, p < 0.001) and decreases in the game scores (WCT-%score; F= 117.78, p < 0.001) across the three levels (Figure 3). The Pearson product correla- pytions between HV and at least one of the WCT performance measures were r ≥ 0.85in the majority of participants. o3.2. WCT TIE o tC NRepeated measures ANOVAs across the four WCT workload levels revealed an in-creasing percentage of high vigilance classifications (Figure 4; F = 3.573, p < 0.001), osignificant increases in the reaction time to identify tracks (WCT-Rtiff; F = 27.485, p D< 0.001), and decreases in the game scores (WCT-%score; F = 14.585, p < 0.001)across the four levels of increasing workload with a significant main effect for FIGURE 3 Mean ± SE percentage game scores (%Score) and reaction time (RTiff) during WCT in 13 novices for easy, moderate, and hard levels (6, 12, and 18 tracks, respectively).
160 Berka et al. FIGURE 4 Mean ± SE percentage of epochs classified as high vigilance (% HV) dur- ing WCT in 10 Technical Integration Experiment participants for 6, 12, 18, and 24 tracks. o py o tC N o D FIGURE 5 Mean reaction time (RTiff) and percentage of epochs classified as high vigilance (% HV) across 12 waves in 10 Technical Integration Experiment participants during WCT.workload. Figure 5 presents the wave-by-wave relation between the mean B-Alert®indexes and mean WCT-RTiff across participants, illustrating that the B-Alert® in-dexes maintained the temporal characteristics of the WCT protocol. The correlationbetween B-Alert® indexes and the WCT-RTiff across waves and participants wassignificant but weak (r = .432, p < 0.01) as a result of within- and between-subjectvariability.3.3. Effects of increasing levels of training in WCTAn inspection of data from individual participants suggested that the percentageof HV classifications during WCT was influenced by the level of training. Figure 6illustrates this observation with data from four individual participants with in-
Real-Time EEG Indexes of Cognition 161 FIGURE 6 Percentage epochs classified as high vigilance (% HV) in 6, 12, 18, and 24 tracks for four participants with 0.6, 5, 10, and 200 hr of WCT training, respectively.creasing levels of WCT training: 35-min training, 5-hr training, 10-hr training, andover 40 hr of training. The workload effect is demonstrated by an increasing per-centage of high vigilance at each of the levels and the training effect is indicated by pyan overall decrease in the percentage of HV. These findings suggest that differencesin training among the TIE participants (ranging from 1 hr to 300 hr) might explain osome of the between-subject variability. To investigate this hypothesis, the 10 TIEparticipants were stratified into three groups based on amount of WCT training tC(i.e., 1–3 hr, n = 3; 6–8 hr, n = 4; and 40–300 hr, n = 3), group means were calculatedfor the percentage of HV, and an ANOVA was applied across workload levels by ogroup (see Figure 7). Despite the small sample size, the results revealed a signifi- N o D FIGURE 7 Mean ± SE percentage of epochs classified as high vigilance (% HV) in 6, 12, 18, and 24 tracks for three groups stratified based on WCT training: group 1 with 1 to 3 hr (n = 3), group 2 with 6 to 8 hr (n = 4), and group 3 with 40 to 300 hr (n = 3) of WCT training. Probabilities for the comparisons between workload levels are designated as *p < .05 and **p < .01.
162 Berka et al.cant interaction between level of expertise and workload level (number of tracks; F= 3.118, p < 0.01). Figure 7 illustrates that the B-Alert® percentage HV increases across WCT work-load levels for all three groups, however, the overall levels of HV across the fourworkload levels decreased dramatically across groups as a result of training. Thissuggests that as participants gain expertise, the level of vigilance is modulated tomeet task demands. For the participants with over 40 hr of training, there were nostatistically significant differences between the 12- to 18-track and 24-track condi-tions (Figure 7). Due to the small sample size (n = 3), it is premature to draw anyconclusions from these data.3.4. Three-Level Cognitive TaskThe mean percentage of B-Alert® HV classifications for each of the three difficultylevels is illustrated in Figure 8. Repeated measures ANOVAs across the three levelsof difficulty revealed a significant increase in the percentage of high vigilance clas-sifications as a result of increasing task difficulty (F = 21.962, p < 0.001). Compari-sons between the easy, moderate, and hard levels revealed significant differences pyfor easy versus moderate (F = 6.419, p < 0.05), easy versus hard (F = 24.608, p <0.001), and moderate versus hard (F = 27.321, p < 0.001). Repeated measures oANOVAs across the three difficulty levels revealed significant increases in reactiontime (F = 17.267, p < 0.001) and decreases in the percentage of correct responses (F = tC38.217, p < 0.001), confirming the actual increase in the levels of task difficulty (Fig-ure 9). The correlations between HV and at least one of the performance measures owere r ≥ 0.85 in the majority of participants. N o D FIGURE 8 Mean ± SE percentage of epochs classified as high vigilance (% HV) for three levels of cognitive task.
Real-Time EEG Indexes of Cognition 163 FIGURE 9 Mean ± SE for percentage of correct responses (% correct) and reaction time (RT) during easy, moderate, and hard difficulty levels of a three-level cognitive task.3.5. Image Learning and Recognition Memory Task o pyMean performance (percentage of correct responses) was computed for each of thetwo sessions. Mean percentages of B-Alert® HV classifications were computed for tCthe image memorization period and the recognition memory period for each of thetwo sessions of the memory test. Comparison of the performance results between othe two sessions revealed significant improvements in performance between ses-sion 1 and session 2 (Figure 10), suggesting a practice effect (t = –2.983, p < .01). Sim- Nilar practice-related results were obtained for the B-Alert® indexes with percentage oHV decreasing from session 1 to session 2 (t = 4.765, p < .001). In addition, signifi- D FIGURE 10 Mean ± SE for percentage of correct responses (% correct) during image learning and recognition memory test for two sessions (categories: animals and travel).
164 Berka et al. FIGURE 11 Mean ± SE percentage of epochs classified as high vigilance (% HV) for the image memorization period and the recognition memory period for the two sessions.cant within-session differences in the B-Alert® HV indexes were obtained withhigher levels of HV during the image memorization period when compared to therecognition memory period (Figure 11). This effect was enhanced for session 2 (t = py3.291, p < .005) in comparison to session 1 (t = 1.989, p = .06).4. DISCUSSION o o tCThe wireless sensor headset provided a reliable method for EEG acquisition andanalysis even within the constraints of the challenging TIE environment (see Figure N1c). Although previous investigators have reported high quality EEG acquisition in ooperational settings including airplane cockpits, long haul truck cabins, and trainoperator quarters (Caldwell, 1995; Kecklund & Akerstedt, 1993; Miller, 1995; DMitler, Miller, Lipsitz, Walsh, & Wylie, 1997; Sterman & Mann, 1995), the wirelesssensor headset represents a significant advance in technology for operational mon-itoring. The limited channel montage has now been demonstrated effective for thedetection of EEG indexes of alertness–drowsiness (Levendowski et al., 2001;Levendowski et al., 2000), attention- and memory-related ERPs (Levendowski etal., 2000; Mitler et al., 2002), and the quantification of cognitive workload duringthe WCT and the three-level cognitive task. The limited channel approach has alsobeen utilized by other investigators to achieve highly sensitive and reliable correla-tions between EEG and performance, including the ability to predict performanceon a second-by-second basis (Makeig & Inlow, 1993; Makeig & Jung, 1995, 1996;Sterman & Mann, 1995; Torsvall & Akerstedt, 1987). The B-Alert® system was designed to detect and predict vigilance decrements,and was validated in sleep deprivation studies using a variety of objective and sub-jective measures to confirm its sensitivity and reliability (Levendowski et al., 1999;Levendowski et al., 2001; Levendowski et al., 2000). In studies of more than 400healthy participants and sleep disorder patients, a number of tonic influences onthe B-Alert® indexes were observed including homeostatic and circadian effects,
Real-Time EEG Indexes of Cognition 165individual differences in vulnerability to sleep deprivation, and disease states suchas sleep and neurological disorders (Westbrook et al., 2002). Quantifiable changesin the B-Alert® indexes were predictive of fatigue-related decrements in accuracyand reaction time that occurred hours before the actual onset of sleep. B-Alert® in-dexes also identified lapses in alertness on a second-by-second basis that were pre-dictive of missed responses or errors on vigilance tests (Levendowski et al., 2001;Levendowski et al., 2000; Mitler et al., 2002). These studies also revealed a number of phasic influences on the B-Alert® in-dexes including the type of task the in which the participant was engaged, task dif-ficulty, and novelty (Levendowski et al., 2001; Levendowski et al., 2000; Mitler etal., 2002). Caffeine and nicotine were shown to affect the B-Alert® indexes (Berka etal., 2000). Although the B-Alert® model was designed for maximum sensitivity tothe vigilance and performance decrements resulting from the transition from wak-ing to drowsiness and sleep onset, these data suggested that the B-Alert® indexesmay have utility for monitoring workload.4.1. B-Alert® Indexes as Measures of Cognitive Workload pyCognitive workload has been conceptualized as the allocation of mental re-sources or effort required to maintain adequate performance on one or more otasks (Kahneman, 1973; Kramer, 1991; Wickens, 1984, 1992; Wickens & Holland,1984). Several investigators have reported EEG measures of workload and task tCdifficulty, and in a review of studies of EEG and workload, Sterman and Mann(1995; Sterman, 1995) concluded that EEG had the potential to be a valid and ob- ojective measure of mental workload. Studies of air traffic controllers (Brookings, NWilson, & Swain, 1996), airline pilots (Sterman & Mann, 1995; Sterman, Mann, & oKaiser, 1992;), drivers (Brookhuis & de Waard, 1993), and participants performingcognitive tasks (Smith, Gevins, Brown, Karnik, & Du, 2001), have related EEG Dshifts to changes in task complexity and task difficulty. The EEG variables com-monly employed by these investigators (e.g. changes in alpha, beta, and thetabands; Byrne & Parasuraman, 1996; Kramer, 1991; Pope et al., 1995; Prinzel et al.,2000; Sterman & Mann, 1995) are incorporated into the B-Alert® classificationmodel. The WCT developed for the DARPA project had several advantages for validat-ing workload indexes: The task was highly engaging with a series of multitaskingsubtests, the workload levels could be systematically varied, and the task resem-bled a navy command and control scenario (St. John et al., 2002). One problem withthe WCT was that the number of stimuli and the motor demands increased as afunction of increasing workload levels, leaving open the possibility that perceptualprocesses or motor activity were being quantified and not cognitive workload perse. Although changes in sensory processing and motor output are important ele-ments of increasing engagement in a task, it has been proposed that the neural re-sources involved in cognitive processes are relatively independent of those in-volved in the motor output (Donchin, Karis, Bashore, Coles, & Gratton, 1986;Gopher, 1992; Wickens, 1991).
166 Berka et al. The three-level cognitive task allowed a more precise definition of the validity ofthe B-Alert® indexes in the measurement of workload by keeping constant thenumber and type of stimuli as well as the motor demands while manipulating thecognitive requirements. The combined results from these two tasks confirm thatthe B-Alert® indexes are related to cognitive effort associated with task difficultyand not to the number of sensory inputs or the amount of motor output requiredfor the different levels. The image memory and recognition task offered a third paradigm for evaluatingthe B-Alert® indexes. The percentage of B-Alert® HV was significantly higher dur-ing the image memorization period when compared to the recognition memory pe-riod (Figure 11), presumably as a result of increased effort during the encoding pe-riod in comparison to the less demanding recognition period.4.2. Training EffectsAlthough the WCT experiments were not designed specifically to evaluate changesin EEG parameters as a function of training, the level of WCT expertise clearly hada substantial impact on the B-Alert® indexes, with overall high vigilance percent- pyages decreasing dramatically as a result of training. Thus, in the context of theWCT, the B-Alert® indexes reflect changes in task difficulty as well as the total ef- ofort expended by the operator. These results, in conjunction with performance dataand subjective reports, confirm that the cognitive, and by inference, neural, re- tCsources required for WCT decrease as a function of practice and are consistent withcurrent theories in cognitive skill acquisition (Fisk & Schneider, 1984; Schneider & oFisk, 1982; Schneider & Shiffrin, 1977). Similarly, brain imaging studies show a re- Nduction in the distribution and the magnitude of cortical activation as skill acquisi- otion progresses (Haier et al., 1992; Raichle et al., 1994). Further investigation is re-quired to determine whether the B-Alert® indexes can be used as predictive Dvariables to estimate, for example, the number of tracks that can be successfullyhandled by an operator at a specified time. Interestingly, the WCT “workload ef-fect” appeared to diminish after 40 to 300 hr of training. It would be of value forpractical applications to determine whether additional training would ultimatelyeliminate this distinction. The image learning and recognition memory tests provide additional supportfor the B-Alert® indexes reflecting practice effects that were correlated with im-proved performance (Figures 10 and 11). It is tempting to speculate that these dataimply that the B-Alert® indexes reflect practice-related changes in cognitive re-source allocation and that, as participants gain expertise, the level of “vigilance” asmeasured by B-Alert® is modulated to meet task demands. To support this conclu-sion, additional studies are required utilizing repeated measures testing on indi-viduals at regular intervals during the skill acquisition process. If the B-Alert® indexes related to skill acquisition can be replicated in a con-trolled experiment, they may have utility for monitoring progress during a varietyof computer-based training exercises. An intelligent brain-computer interfacecould be designed to assure that trainees move efficiently through various levels of
Real-Time EEG Indexes of Cognition 167expertise using the B-Alert® indexes to maintain an optimal pace of informationpresentation.5. CONCLUSIONThe wireless sensor headset and the B-Alert® EEG analysis software provide a ro-bust and reliable method for monitoring alertness and cognitive workload in oper-ational environments. The B-Alert indexes are sensitive to long-term and tran-sient fluctuations in the neural activity involved in alertness and cognitiveworkload. In future applications, the global B-Alert indexes could be applied incombination with other analytic techniques such as ERPs (Fabiani et al., 2000; Kok,2001; Levendowski et al., 2000) or event-related desynchronization (Klimesch,Russegger, Doppelmayr, & Pachinger, 1998) to capture a more detailed image of in-formation processing in the brain. Additional research is required to develop an intelligent interface that utilizesthe outputs from the B-Alert® to drive an adaptive automation system. The datasuggest that in addition to predicting performance decrements resulting from ex-cessive workload, the B-Alert® EEG indexes could have utility in optimizing the pyspeed and efficiency of computer-based training programs.REFERENCES o o tCAkerstedt, T., & Folkard, S. (1997). The three-process model of alertness and its extension to performance, sleep latency, and sleep length. Chronobiolology International, 14(2), 115–123. NBalkin, T. J. (2001, August). Sleep deprivation research at WRAIR. Paper presented at the DARPA Workshop, Las Vegas, NV. oBerka, C., Levendowski, D. J., Konstantinovic, Z. R., Olmstead, R. E., Davis, G., & Lumicao, D M. N. (2000, July). Detection of drowsiness with realtime analysis of the electroencephalogram (EEG). Paper presented at the National Institutes of Health SBIR Conference, Bethesda, MD.Brookhuis, K. A., & de Waard, D. (1993). The use of psychophysiology to assess driver status. Ergonomics, 36, 1099–1110.Brookings, J. B., Wilson, G. F., & Swain, C. R. (1996). Psychophysiological responses to changes in workload during simulated air traffic control. Biological Psychology, 42, 361–377.Byrne, E. A., & Parasuraman, R. (1996). Psychophysiology and adaptive automation. Biological Psychology, 42, 249–268.Caldwell, J. (1995). Assessing the impact of stressors on performance: Observations on levels of analyses. Biological Psychology, 40, 197–208.Donchin, E., Karis, D., Bashore, T. R., Coles, M. G., & Gratton, G. (1986). Cognitive psychophysiology and human information processing. In M. G. Coles, E. Donchin, & S. Porges (Eds.), Psychophysiology: Systems, processes and applications (pp. 244–266). New York: Guilford.Doran, S. M., Van Dongen, H. P., & Dinges, D. F. (2001). Sustained attention performance during sleep deprivation: Evidence of state instability. Archives Italiennes de Biologie, 139(3), 253–267.
168 Berka et al.Fabiani, M., Gratton, G., & Coles, M. G. (2000). Event-related brain potentials. In J. T. Caciooppo, L. G. Tassinary, & G. G. Berntson (Eds.), Handbook of psychophysiology (pp. 53–84). Cambridge, England: Cambridge University Press.Fisk, A. D., & Schneider, W. (1984). Memory as a function of attention, level of processing, and automatization. Journal of Experimental Psychology, Learning, Memory, and Cognition, 10, 181–197.Gevins, A., Smith, M. E., Le, J., Leong, H., Bennett, J., Martin, N., et al. (1996). High resolution evoked potential imaging of the cortical dynamics of human working memory. Electroen- cephalography and Clinical Neurophysiology, 98, 327–348.Gevins, A., Smith, M. E., Leong, H., McEvoy, L., Whitfield, S., Du, R., et al. (1998). Monitoring working memory load during computer-based tasks with EEG pattern recognition meth- ods. Human Factors, 40, 79–91.Gevins, A., Smith, M. E., McEvoy, L., & Yu, D. (1997). High-resolution EEG mapping of corti- cal activation related to working memory: Effects of task difficulty, type of processing, and practice. Cerebral Cortex, 7, 374–385.Gopher, D. (1992). The skill of attention control: Acquisition and execution of attention strat- egies. In D. E. Meyer & S. Kornblum (Eds.), Attention and performance XIV (pp. 299–322). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.Haier, R. J., Siegel, B. V., Jr., MacLachlan, A., Soderling, E., Lottenberg, S., & Buchsbaum, M. S. (1992). Regional glucose metabolic changes after learning a complex visuospatial/mo- tor task: A positron emission tomographic study. Brain Research, 570, 134–143. pyHillyard, S. A., Hink, R. F., Schwent, V. L., & Picton, T. W. (1973). Electrical signs of selective attention in the human brain. Science, 182, 177–180. oKahneman, D. (1973). Attention and effort. Englewood Cliffs, NJ: Prentice Hall.Kecklund, G., & Akerstedt, T. (1993). Sleepiness in long distance truck driving: An ambula- tC tory EEG study of night driving. Ergonomics, 36, 1007–1017.Klimesch, W., Russegger, H., Doppelmayr, M., & Pachinger, T. (1998). A method for the cal- o culation of induced band power: Implications for the significance of brain oscillations. N Electroencephalography and Clinical Neurophysiology, 108, 123–130.Kok, A. (2001). On the utility of P3 amplitude as a measure of processing capacity. Psychophysiology, 38, 557–577. o DKramer, A. F. (1991). Physiological metrics of mental workload: A review of recent progress. In D. L. Damos (Ed.), Multiple task performance (pp. 279–328). Washington, DC: Taylor & Francis.Levendowski, D. J., Berka, C., Olmstead, R. E., & Jarvik, M. (1999, October). Correlations be- tween EEG indices of alertness measures of performance and self-reported states while operating a driving simulator. Paper presented at the 29th annual meeting of the Society for Neurosci- ence, Miami, FL.Levendowski, D. J., Berka, C., Olmstead, R. E., Konstantinovic, Z. R., Davis, G., Lumicao, M. N., et al. (2001). Electroencephalographic indices predict future vulnerability to fatigue induced by sleep deprivation. Sleep, 24(Abstract Suppl.), A243–A244.Levendowski, D. J., Olmstead, R. E., Konstantinovic, Z. R., Berka, C., & Westbrook, P. R. (2000). Detection of electroencephalographic indices of drowsiness in realtime using a multi-level discriminant function analysis. Sleep, 23(Abstract Suppl. 2), A243–A244.Levendowski, D. J., Westbrook, P., Berka, C., Popovic, M. V., Pineda, J. A., Zavora, T. M., et al. (2002). Event-related potentials during a test of working memory differentiate sleep apnea patients from healthy subjects. Sleep, 25(Abstract Suppl.), A460–A461.Makeig, S. (1993). Lapses in alertness: Coherence of fluctuations in performance and EEG spectrum. Electroencephalography and Clinical Neurophysiology, 86, 23–35.
Real-Time EEG Indexes of Cognition 169Makeig, S., & Jung, T. P. (1995). Changes in alertness are a principal component of variance in the EEG spectrum. Neuroreport, 7, 213–216.Makeig, S., & Jung, T. P. (1996). Tonic, phasic, and transient EEG correlates of auditory awareness in drowsiness. Brain Research, Cognitive Brain Research, 4, 15–25.Mathiak, K., Hertrich, I., Rothe, S., Kincses, W. E., Lutzenberger, W., & Ackerman, H. (2002). Involuntary crossmodal enhancement of preattentive auditory processing is selective in space. Manuscript submitted for publication.Miller, J. C. (1995). Batch processing of 10,000 h of truck driver EEG data. Biological Psy- chology, 40, 209–222.Mitler, M. M., Miller, J. C., Lipsitz, J. J., Walsh, J. K., & Wylie, C. D. (1997). The sleep of long-haul truck drivers. New England Journal of Medicine, 337, 755–761.Mitler, M. M., Westbrook, P., Levendowski, D. J., Ensign, W. Y., Olmstead, R. E., Berka, C., et al. (2002). Validation of automated EEG quantification of alertness: Methods for early identification of individuals most susceptible to sleep deprivation. Sleep, 25(Abstract Suppl.), A147–A148.Moore–Ede, M. C. (1993). The twenty-four-hour society: Understanding human limits in a world that never stops. Reading, MA: Addison–Wesley.Parasuraman, R., Bahri, T., Deaton, J. E., Morrison, J. G., & Barnes, M. (1992). Theory and design of adaptive automation in adaptive systems (Progress Rep. No. NAWCADIWAR–92033–60). Warminster, PA: Naval Air Warfare Center, Aircraft Division.Parasuraman, R., Molloy, R., & Singh, I. L. (1993). Performance consequences of automation induced “complacency”. International Journal of Aviation Psychology, 3, 1–23. pyParasuraman, R., Mouloua, M., & Hilburn, B. (1999). Adaptive aiding and adaptive task allo- cation enhance human–machine interaction. In M. W. Scerbo & M. Mouloua (Eds.), Auto- o mation technology and human performance: Current research and trends (pp. 119–123). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. tCParasuraman, R., Mouloua, M., & Molloy, R. (1996). Effects of adaptive task allocation on monitoring of automated systems. Human Factors, 38, 665–679. oPolich, J. (2002). Neuropsychology of P3a and P3b: A theoretical overview. Advanced in N Electrophysiology in Clinical Practice and Research.Polich, J., & Herbst, K. L. (2000). P300 as a clinical assay: Rationale, evaluation, and findings. o International Journal of Psychophysiology, 38, 3–19. DPope, A. T., Bogart, E. H., & Bartolome, D. S. (1995). Biocybernetic system evaluates indices of operator engagement in automated task. Biological Psychology, 40, 187–195.Prinzel, L. J., Freeman, F. G., Scerbo, M. W., Mikulka, P. J., & Pope, A. T. (2000). A closed-loop system for examining psychophysiological measures for adaptive task allocation. International Journal of Aviation Psychology, 10, 393–410.Raichle, M. E., Fiez, J. A., Videen, T. O., MacLeod, A. M., Pardo, J. V., Fox, P. T., et al. (1994). Practice-related changes in human brain functional anatomy during nonmotor learning. Cerebral Cortex, 4, 8–26.Schneider, W., & Fisk, A. D. (1982). Concurrent automatic and controlled visual search: Can processing occur without cost? Journal of Experimental Psychology: Learning, Memory, and Cognition, 8, 261–278.Schneider, W., & Shiffrin, R. M. (1977). Controlled and automatic human information pro- cessing I: Detection, search, and attention. Psychological Reviews, 84, 1–66.Singh, I. L., Molloy, R., & Parasuraman, R. (1993). Automation-induced “complacency”: De- velopment of the complacency-potential rating scale. International Journal of Aviation Psy- chology, 3, 111–121.Smith, M. E., Gevins, A., Brown, H., Karnik, A., & Du, R. (2001). Monitoring task loading with multivariate EEG measures during complex forms of human–computer interaction. Human Factors, 43, 366–380.
170 Berka et al.St. John, M., Kobus, D. A., & Morrison, J. G. (2002). A multi-tasking environment for manipu- lating and measuring neural correlates of cognitive workload. In Proceedings of the 2002 IEEE 7th Conference on Human Factors and Power Plants (pp. 7.14–7.14). New York: IEEE.Sterman, M. B., & Mann, C. A. (1995). Concepts and applications of EEG analysis in aviation performance evaluation. Biological Psychology, 40, 115–130.Sterman, M. B., Mann, C. A., & Kaiser, D. A. (1992). Quantitative EEG patterns of differential in-flight workload (Abstract from Soar ’92). Sepulveda, VA: Sepulveda VA Medical Center.Torsvall, L., & Akerstedt, T. (1987). Sleepiness on the job: Continuously measured EEG changes in train drivers. Electroencephalography and Clinical Neurophysiology, 66, 502–511.Westbrook, P., Berka, C., Levendowski, D., Lumicao, M. N., Davis, G., Olmstead, R. E., et al. (2002). Biobehavioral quantification of alertness and memory in patients with sleep apnea. Sleep, 25(Abstract Suppl.), A49–A50.Wickens, C. D. (1984). Engineering psychology and human performance. Columbus, OH: Merrill.Wickens, C. D. (1991). Processing resources and attention. In D. L. Damos (Ed.), Multiple-task performance (pp. 1–34). London: Taylor & Francis.Wickens, C. D. (1992). Engineering psychology and human performance (2nd ed.). New York: HarperCollins.Wickens, C. D., & Holland, J. G. (1984). Engineering psychology and human performance (3rd ed.). Englewood Cliffs, NJ: Prentice Hall. o py o tC N o D