Activity and Emotion Recognition to Support Early Diagnosis of Psychiatric Diseases Pervasive Health Conference Tampere, 30 th January 2008

Paper Contributors David Tacconi, Oscar Mayora, CREATE-NET Paul Lukowicz, University of Passau Bert Arnrich, Cornelia Setz, Gerhard Tr ö ster, ETH Zurich Christian Haring, PSYCHIATRIC STATE HOSPITAL TIROL (PSHT) PSHT PSYCHIATRIC STATE HOSPITAL TIROL

David Tacconi, Oscar Mayora, CREATE-NET

Paul Lukowicz, University of Passau

Bert Arnrich, Cornelia Setz, Gerhard Tr ö ster, ETH Zurich

Christian Haring, PSYCHIATRIC STATE HOSPITAL TIROL (PSHT)

Outline Introduction Bipolar disorder Objectives and System Requirements Pervasive computing to support Bipolar Disorder diagnosis A proposed System Architecture Discussion and future work

Introduction

Bipolar disorder

Objectives and System Requirements

Pervasive computing to support Bipolar Disorder diagnosis

A proposed System Architecture

Discussion and future work

Introduction Global Burden of Disease Mental illness accounts for over 15% of the burden of diseases in established market economies (source World Health Organization, World Bank, Harvard University ) Disability Adjusted Life Years (DALYs): Measure the lost years of healthy life (premature death or disability) Depression: is the most common psychiatric disorder, accounting for 50.8 million DALYs or 10.7% of the global burden of disease It is ranked fourth among all causes of DALYs and is the leading nonfatal condition globally Bipolar disorder: account for another 14,1 million (3.0%) DALYs

Global Burden of Disease

Mental illness accounts for over 15% of the burden of diseases in established market economies (source World Health Organization, World Bank, Harvard University )

Disability Adjusted Life Years (DALYs):

Measure the lost years of healthy life (premature death or disability)

Depression:

is the most common psychiatric disorder, accounting for 50.8 million DALYs or 10.7% of the global burden of disease

It is ranked fourth among all causes of DALYs and is the leading nonfatal condition globally

Bipolar disorder:

account for another 14,1 million (3.0%) DALYs

Introduction Few technological solutions exist to aid people affected by mental illness. Obvious reasons are: people affected by mental illness are more likely to have problems dealing with complex technology providing behavioral assistance is much more difficult than providing physical assistance solutions require considerable amount of domain specific knowledge and can only be developed in close cooperation with psychiatrists

Few technological solutions exist to aid people affected by mental illness.

Obvious reasons are:

people affected by mental illness are more likely to have problems dealing with complex technology

providing behavioral assistance is much more difficult than providing physical assistance

solutions require considerable amount of domain specific knowledge and can only be developed in close cooperation with psychiatrists

Bipolar Disorder Bipolar disorder repeated relapses of mania and depression Recurrence rates are high at around 50% at one year and 70% at four years Treatments for Bipolar disorder: Pharmacotherapy is the main treatment nowadays Alternative: teach the patients to recognize and manage Early Warning Signs (EWS) Diagnosing through patient questionnaires: for depression the Hamilton Depression Scale (HAMD) for mania the Bech-Rafaelsen Mania Scale (BRMS) Both contain a series of questions related to patients’ activities and feelings

Bipolar disorder

repeated relapses of mania and depression

Recurrence rates are high at around 50% at one year and 70% at four years

Treatments for Bipolar disorder:

Pharmacotherapy is the main treatment nowadays

Alternative: teach the patients to recognize and manage Early Warning Signs (EWS)

Diagnosing through patient questionnaires:

for depression the Hamilton Depression Scale (HAMD)

for mania the Bech-Rafaelsen Mania Scale (BRMS)

Both contain a series of questions related to patients’ activities and feelings

Bipolar Disorder: the HAMD Depressed Mood Feelings of Guilty Suicide Insomnia (early) Insomnia (middle) Insomnia (late) Work and Activities Retardation: Psychomotor Agitation Anxiety (Psychological) Anxiety Somatic Somatic Symptoms (Gastrointestinal) Somatic Symptoms General General Symptoms Hypocondriasis Loss of Weight Insight Diurnal Variation Depersonalization and Derealization Paranoid Symptoms Evaluate between 0 and 4 to obtain the total score

Depressed Mood

Feelings of Guilty

Suicide

Insomnia (early)

Insomnia (middle)

Insomnia (late)

Work and Activities

Retardation: Psychomotor

Agitation

Anxiety (Psychological)

Anxiety Somatic

Somatic Symptoms (Gastrointestinal)

Somatic Symptoms General

General Symptoms

Hypocondriasis

Loss of Weight

Insight

Diurnal Variation

Depersonalization and Derealization

Paranoid Symptoms

Bipolar Disorder: the BRMS motor activity verbal activity flight of thoughts voice/noise level hostility/destructiveness mood and feelings of well-being self-esteem contact sleep (based on the average of the previous 3 nights) sexual interest and activity work level Evaluate between 0 and 4 to obtain the total score

motor activity

verbal activity

flight of thoughts

voice/noise level

hostility/destructiveness

mood and feelings of well-being

self-esteem

contact

sleep (based on the average of the previous 3 nights)

sexual interest and activity

work level

Objectives We identify Bipolar Disorder as a condition that can realistically benefit from behavioral monitoring We identify support in early detection of imminent transitions between normal, manic and depressed states as the specific contribution to therapy We identify specific behaviors that need to be detected by the proposed system, using the so called Hamilton Depression Scale ( HAMD ) and Bech-Rafaelsen Mania scale ( BRMS ), which are generally accepted tools in the diagnosis of depression and mania Based on literature study and previous work by the authors, we argue that detecting these specific behaviors is feasible We propose an appropriate system architecture based on existing devices and previous systems implemented by the authors groups

We identify Bipolar Disorder as a condition that can realistically benefit from behavioral monitoring

We identify support in early detection of imminent transitions between normal, manic and depressed states as the specific contribution to therapy

We identify specific behaviors that need to be detected by the proposed system, using the so called Hamilton Depression Scale ( HAMD ) and Bech-Rafaelsen Mania scale ( BRMS ), which are generally accepted tools in the diagnosis of depression and mania

Based on literature study and previous work by the authors, we argue that detecting these specific behaviors is feasible

We propose an appropriate system architecture based on existing devices and previous systems implemented by the authors groups

Pervasive Computing for diagnosis of BD From our experience and a literature study, we propose to look at the following parameters: HAMD: items 4,5,6 (insomnia) 7 (work and activities) 8 (psychomotoric retardation) 9 (agitation) 10, 11(anxiety) BRMS: items 1 (motoric activities) 2 (verbal activities) 4 (voice-noise level) 8 (contacts) 9 (sleep)

From our experience and a literature study, we propose to look at the following parameters:

HAMD:

items 4,5,6 (insomnia)

7 (work and activities)

8 (psychomotoric retardation)

9 (agitation)

10, 11(anxiety)

BRMS:

items 1 (motoric activities)

2 (verbal activities)

4 (voice-noise level)

8 (contacts)

9 (sleep)

Early Diagnosis 1/3 Insomnia and Sleep disorders ( HAMD 4-6, BRMS 9 ) ” Gold standard” (laboratory settings): polysomnographic monitoring of sleep time physiological parameters (e.g. respiration, heart rate variability) and sleep motion On-body sensors: unobtrusively embedded into biomedical clothes or mattresses allow to obtain preliminary diagnosis and to perform more frequent tests under real-life conditions Other solutions: thin film, dynamic quasi-piezoelectric sensors placed under the mattress capacitive pressure sensor mat would allow monitoring sleep motion

Insomnia and Sleep disorders ( HAMD 4-6, BRMS 9 )

” Gold standard” (laboratory settings):

polysomnographic monitoring of sleep time

physiological parameters (e.g. respiration, heart rate variability) and sleep motion

On-body sensors:

unobtrusively embedded into biomedical clothes or mattresses

allow to obtain preliminary diagnosis and to perform more frequent tests under real-life conditions

Other solutions:

thin film, dynamic quasi-piezoelectric sensors placed under the mattress

capacitive pressure sensor mat would allow monitoring sleep motion

Early Diagnosis 2/3 Verbal activities ( BRMS 2, 4 ), Contacts/Conversation ( BRMS 8 ) automatic speech character identification would allow to extract features describing contextual side information spoken messages convey besides information on characteristics as intonation, speaking rate or emotional state. emotion recognition can give to the therapists information about variation of the patient’s mental state.

Verbal activities ( BRMS 2, 4 ), Contacts/Conversation ( BRMS 8 )

automatic speech character identification

would allow to extract features describing contextual side information

spoken messages convey besides information on characteristics as intonation, speaking rate or emotional state.

emotion recognition can give to the therapists information about variation of the patient’s mental state.

Early Diagnosis 3/3 Activity Recognition ( HAMD 7,8,9,10,11, BRMS 1 ) Several past works on activity recognition: it remains unclear how most prior systems will perform under real life conditions Based on previous experience, we target systematic real life trials to: quantify his Work and Activities (HAMD 7 and BRMS 1) understand the Agitation (HAMD 9) and Anxiety (HAMD 10, 11) he experiences measure an eventual Psychomotoric Retardation (HAMD 8)

Activity Recognition ( HAMD 7,8,9,10,11, BRMS 1 )

Several past works on activity recognition:

it remains unclear how most prior systems will perform under real life conditions

Based on previous experience, we target systematic real life trials to:

quantify his Work and Activities (HAMD 7 and BRMS 1)

understand the Agitation (HAMD 9) and Anxiety (HAMD 10, 11) he experiences

measure an eventual Psychomotoric Retardation (HAMD 8)

System Architecture Constraints to be considered: patients are likely to reject pervasive computing technology in principle target devices should be as less obtrusive as possible patients cannot be asked to perform any training of devices, and this complicates things for emotion and activity recognition. Activity and emotion recognition is targeted to medium and long term behavior Higher errors in activity recognition are allowed doctors are more interested in average behaviors rather than in instantaneous activity pattern or emotions a patient is feeling in a given moment. Doctors are interested in behaviors that are repeated in time and that can be symptoms of disease’s relapse.

Constraints to be considered:

patients are likely to reject pervasive computing technology in principle

target devices should be as less obtrusive as possible

patients cannot be asked to perform any training of devices, and this complicates things for emotion and activity recognition.

Activity and emotion recognition is targeted to medium and long term behavior

Higher errors in activity recognition are allowed

doctors are more interested in average behaviors rather than in instantaneous activity pattern or emotions a patient is feeling in a given moment.

Doctors are interested in behaviors that are repeated in time and that can be symptoms of disease’s relapse.

System Architecture The User Interfaces module: present persuasive feedback to the users for motivating healthier patients’ behavior The User Model includes all patient’s characteristics, disease’s peculiarities and his preferences. Information stored in: User Profile ( UP ) Disease Description ( DD ) Patient Description ( PD ) The Context Acquisition module gathers data from Sensors and is driven by: Emotion Recognition Manager that selects sensors for emotion recognition Activity Recognition Manager that selects sensors for recognizing user’s activity User model manager gives proper inputs The Content Manager module is responsible: For uploading the data to the EMR through the Data Upload module For presenting information to the patient through the Feedback Manager module

The User Interfaces module:

present persuasive feedback to the users for motivating healthier patients’ behavior

The User Model includes all patient’s characteristics, disease’s peculiarities and his preferences. Information stored in:

User Profile ( UP )

Disease Description ( DD )

Patient Description ( PD )

The Context Acquisition module gathers data from Sensors and is driven by:

Emotion Recognition Manager that selects sensors for emotion recognition

Activity Recognition Manager that selects sensors for recognizing user’s activity

User model manager gives proper inputs

The Content Manager module is responsible:

For uploading the data to the EMR through the Data Upload module

For presenting information to the patient through the Feedback Manager module

Activity recognition

Emotion Recognition

Discussion and future work We have shown a feasibility study on applying existing pervasive computing techniques to support the early diagnosis of bipolar disorder We have formulated proper system requirements, showing then how current research and authors’ expertise can be leveraged for helping doctors and patients in recognizing early symptoms of depression and mania We have also defined a system architecture meeting such requirements Future work: Refine the P-cube platform implementation for the bipolar disorder scenario Integrate the implementation with indoor environment (e.g. Living Lab at Create-Net) including hand-over between outdoors and indoors Integrate the context recognition platforms we have in the context data block Continue in a close cooperation with doctors to define proper UI and medical records Test-bed trials to refine the system (in cooperation with doctors) Start working with patients for developing a first proof of concept of the proposed system

We have shown a feasibility study on applying existing pervasive computing techniques to support the early diagnosis of bipolar disorder

We have formulated proper system requirements, showing then how current research and authors’ expertise can be leveraged for helping doctors and patients in recognizing early symptoms of depression and mania

We have also defined a system architecture meeting such requirements

Future work:

Refine the P-cube platform implementation for the bipolar disorder scenario

Integrate the implementation with indoor environment (e.g. Living Lab at Create-Net) including hand-over between outdoors and indoors

Integrate the context recognition platforms we have in the context data block

Continue in a close cooperation with doctors to define proper UI and medical records

Test-bed trials to refine the system (in cooperation with doctors)

Start working with patients for developing a first proof of concept of the proposed system

Back-up slides: Mania Symptoms Increased energy, activity, and restlessness Excessively "high," overly good, euphoric mood Extreme irritability Racing thoughts and talking very fast, jumping from one idea to another Distractibility, can't concentrate well Little sleep needed Unrealistic beliefs in one's abilities and powers Poor judgment Spending sprees A lasting period of behavior that is different from usual Increased sexual drive Abuse of drugs, particularly cocaine, alcohol, and sleeping medications Provocative, intrusive, or aggressive behavior Denial that anything is wrong List taken from: http:// www.nimh.nih.gov/publicat/bipolar.cfm

Increased energy, activity, and restlessness

Excessively "high," overly good, euphoric mood

Extreme irritability

Racing thoughts and talking very fast, jumping from one idea to another

Distractibility, can't concentrate well

Little sleep needed

Unrealistic beliefs in one's abilities and powers

Poor judgment

Spending sprees

A lasting period of behavior that is different from usual

Increased sexual drive

Abuse of drugs, particularly cocaine, alcohol, and sleeping medications

Provocative, intrusive, or aggressive behavior

Denial that anything is wrong

List taken from: http:// www.nimh.nih.gov/publicat/bipolar.cfm

Back-up slides: Depression Symptoms Lasting sad, anxious, or empty mood Feelings of hopelessness or pessimism Feelings of guilt, worthlessness, or helplessness Loss of interest or pleasure in activities once enjoyed, including sex Decreased energy, a feeling of fatigue or of being "slowed down" Difficulty concentrating, remembering, making decisions Restlessness or irritability Sleeping too much, or can't sleep Change in appetite and/or unintended weight loss or gain Chronic pain or other persistent bodily symptoms that are not caused by physical illness or injury Thoughts of death or suicide, or suicide attempts iopList taken from: http:// www.nimh.nih.gov/publicat/bipolar.cfm

Lasting sad, anxious, or empty mood

Feelings of hopelessness or pessimism

Feelings of guilt, worthlessness, or helplessness

Loss of interest or pleasure in activities once enjoyed, including sex

Decreased energy, a feeling of fatigue or of being "slowed down"

Difficulty concentrating, remembering, making decisions

Restlessness or irritability

Sleeping too much, or can't sleep

Change in appetite and/or unintended weight loss or gain

Chronic pain or other persistent bodily symptoms that are not caused by physical illness or injury

Thoughts of death or suicide, or suicide attempts

iopList taken from: http:// www.nimh.nih.gov/publicat/bipolar.cfm

Existing devices: QBIC Belt Integrated Computer Key Features XScale CPU up to 400Mhz 256 MB SDRAM Low power Standard interfaces (USB, Serial, VGA, Bluetooth) Applications Data recording with on-line processing and communication

Key Features

XScale CPU up to 400Mhz

256 MB SDRAM

Low power

Standard interfaces (USB, Serial, VGA, Bluetooth)

Applications

Data recording with on-line processing and communication

Existing devices: Sensorbutton Sensors Accelerometers, light sensor, microphone, microprocessor Applications On-line context recognition Algorithms nearest-neighbor and C4.5 decision tree

Sensors

Accelerometers, light sensor, microphone, microprocessor

Applications

On-line context recognition

Algorithms

nearest-neighbor and C4.5 decision tree

Existing devices: Textile Pressure Sensor Pure textile, capacitive sensor Individually connected electrodes using silver coated yarn Localized detection of pressure distribution

Pure textile, capacitive sensor

Individually connected electrodes using silver coated yarn

Localized detection of pressure distribution