Information fusion and algorithm training framework objective in ICT4Life H2020 Project. Presented in IEEEHealthcom'16 within Project Alfred workshop in Munich (14-17 September 2016).

1. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
“Information fusion and algorithm training framework”
1st International Workshop on Open Platforms in the
Field of Independent Living and Active Ageing
Alejandro Sánchez-Rico de las Heras
@alejandrosr

2. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
Information is everywhere
Ambient
Sensors
Wearable
devices
Cameras
3D
Zenith
Normal
Behaviour
data
Data Analysis
(Algorithms)
Clinical
data
Profile
Disease information
Family history
Medication
Laboratory analysis
Medical imaging
…
Analyzed
data

3. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
Multimodal Fusion
Behaviour
data
Clinical
data
Profile
Disease information
Family history
Medication
…
Multimodal Fusion
Algorithm
sensor
analysis
Health Monitoring
Data
Feedback
Professional feedback
Patient evolution

4. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
ICT4Life Integrated Care Project
This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement Nº 690090.
@ICT4LifeEU

5. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
ICT4Life Overall Structure
This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement Nº 690090.
@ICT4LifeEU

6. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
ICT4Life Setup
This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement Nº 690090.
@ICT4LifeEU
Kinect
Binary
sensor

7. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
ICT4Life Framework
This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement Nº 690090.
ICT4Life Open API on Multimodal Fusion
• Open framework for platform flexibility to accommodate various sensors and information channels.
• Ready-to-use algorithms for re-training in new environments.
• Flexibility in amount and type of modalities.
@ICT4LifeEU
Abnormal Behaviour
Detection of abnormal human activities
using extensive training through:
• Video analysis algorithms.
• Ambient assisting living.
• Skeleton track data
Multimodal Fusion
• Detected abnormal behaviours.
• Clinical Data (EHR)
• Interactions with social networks
and gamified interfaces.
Decision Support module
• Personalization of interfaces
• Support caregivers and professionals.

8. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
Abnormal Behaviour
Detection of abnormal activities
• Set of activities to detect: Apathy, Freezing, …
• Set of data for algorithm training.
– Sensor data capture (Normal and abnormal conditions)
– Professional data feedback.
Skeleton information
3D Cameras
Spatial information
Zenith Cameras
Bracelet data
Binary sensors data
Training
Algorithm
Execution
Feedback
Sensor data
Classification
Learn from errors
Multiple sensor information
Sensor Calibration
Common Axis and Time system.
Initial training data
Output
Normal data (x2)
Abnormal data
Activities

9. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
Abnormal Behaviour
Open API Framework strategy
• Objective:
– Provide tools for training on new activities
– Be able to adapt platform to new disease scenarios.
• Needs flexibility on:
– Set of monitored activities.
– Algorithm training data for new activities.
– Output description and representation. (E.g. type of output: count, time, …)
– Final user feedback for algorithm improvement in time.
– Sensor data acquisition.
• Functionality:
– Activity register (Basic data, output representation)
– Training data load (Normal and abnormal data)
• E.g. Image captures, 3D skeleton data, …

10. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
Abnormal Behaviour
Activity Register Training data load
CRUD OF NEW ACTIVITIES
CREATION, READ, UPDATE, DELETE
Activity structure
{
code: num,
name: Activity name,
Type: count / time / ...
Scenarios: Scenario Ids, where analysis apply
Output: {
title: Output title of activity,
description: Output description of activity
}
}
Activity identification (code).
Activity data:
• Image captures.
• 3D camera data.
• Other sensor data.
• Calibration data for multisensor.
User assessment.
• Situation identification.
Process of user validation & feedback.
Validation

11. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
Multimodal Fusion
Open API Framework strategy
• Objective:
– Increase data structure flexibility and scalability considered on fusion analysis.
– Be able to adapt platform to new disease scenarios.
• Needs flexibility on:
– Set of data structures and associated patient data.
– Multimodal fusion rules.
• Adapt current rules to consider new data.
• Register new rules.
– Output description and representation.
– Final user feedback and patient evolution data for fusion improvement.
• Functionality:
– Adherence-Impact on existing standards (LOINC, SNOMED CT)
– Patient data for analysis.
– Rules update and creation.
– Efficient and easy integration of experts feedback on patient analysis and progress.

12. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
Multimodal Fusion
Electronic Health Record
(Input and Output)
Fusion output
Decision support tools or
other modules
Health Monitoring
(sensors data)
Data Pre-
processing
Fusion
techniques
Feature
Extraction
New Patient Data
Abnormal Behaviour
Information
Previous algorithm results
EHR added
sets
Adapted fusion rules
Feedback
Professional user input
Patient evolution
New Data
structures

13. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
Multimodal Fusion using late fusion
Probabilistic framework, where each low-level sub-subsystem outputs a
probability measure for each type of results.
𝑃 𝑖
𝑎𝑝𝑎𝑡ℎ𝑦 = 𝑤 𝑎 ∗ 𝑃 𝐴𝐵𝐷
𝑎𝑝𝑎𝑡ℎ𝑦 + 𝑤ℎ ∗ 𝑃 𝐻𝑀
𝑎𝑝𝑎𝑡ℎ𝑦 + 𝑤𝑒 ∗ 𝑃 𝐸𝐻𝑅
𝑎𝑝𝑎𝑡ℎ𝑦 + wn * PN( 𝑎𝑝𝑎𝑡ℎ𝑦)
Patient unique identifier
weights associated with each modality, based on knowledge or
learnt during training.
i
wa , wh
we, wn
*
Rule Creation and Adaptation
• Rule adaptation
• Weights matrix for data modalities.
• Probabilistic results on new data structures.
• Listener probabilistic functions.
• Rule creation / New infers results
• Add probabilistic functions for each data modality needed.
Fusion of each module output
(classification results)

14. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
Multimodal Fusion and standards
Data Structure Integration strategy
• Common terminology understanding.
• Integration with current data structures.
• Facilitate interoperability.
Data structure definition
• Data structure tree.
• Internal code.
• LOINC code match. (if possible)
• SNOMED on observation values. (if possible)
• E.g. HL7 ORU messages.
Data Observation Functionality
• Patient identifier.
• Data associated to previous registered structure.
• LOINC codes / Internal codes
• SNOMED CT Values / Internal values
• Time association if needed.
• To support fusion of coherent data.
Standards on health data terminology
Clinical terminology
Diseases.
Clinical findings.
Procedures.
Universal codes to identify
Tests
Observations

15. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
Development Community
ICT4Life targets to create a community for interested developers and
researchers in ICT and Health.
– Platform integration with other monitoring needs.
– Collaboration with other initiatives.
– Share integration ideas.
– ICT4Life in the crossroads of ICT, Medicine, Psychology.
This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement Nº 690090.
@ICT4LifeEU
Stay tuned
@ICT4LifeEU
www.ict4life.eu
info@ict4life.eu
Visit our website, join our newsletter and RRSS.

16. Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu
Thank you!
Alejandro Sánchez-Rico de las Heras
@alejandrosr