This document discusses research into human-in-loop cyber-physical systems using internet-of-things technologies. It describes a platform called RIPSAC that integrates sensor data from physical objects, infrastructure and humans to provide intelligent services. Example applications discussed include elderly monitoring, activity detection, and integrating data from various smart domains like transportation, healthcare and utilities. The document also outlines several research projects in areas like healthcare monitoring using mobile phones, new sensing techniques like 5-senses computing, distributed edge computing, and knowledge-driven analytics using semantic sensor descriptions.

1. Experience certainty.
Copyright © 2011 Tata Consultancy Services Limited
Human-in-loop Cyber-physical Systems
- Intelligent Infrastructure using Internet-of-Things
Arpan Pal
Head of Research
Innovation Lab, Kolkata

2. Research Theme

3. 3
Experience certainty.
Human-in-Loop Cyber-physical Systems
Humans
Physical
Objects and
Infrastructure
Computing
Infrastructure
Physical
Context
Discovery

4. 4
Experience certainty.
Signal
Processing
Internet-of-Things - towards Intelligent Infrastructure
Sense
Extract
Analyze
Respond
Learn
Monitor
Intelligent
Infra
@Home
@Building
@Vehicle
@Utility
@Mobile
@Store
@Road
“Intelligent” (Cyber) “Infrastructure” (Physical)
APPLICATION SERVICES
BACK-END PLATFORM
INTERNET
GATEWAY
Sense
Extract
Analyze
Respond
Communication
Computing

5. 5
Challenges and Solution Approach
Systems
• Making Sensing Unobtrusive and Pervasive
• Sensing the Human Context
 Cloud-based Sensor Data Analytics Platform
 Use of Mobile phones and Cameras as sensing devices
 New Sensors for human context – EEG, 5 senses computing
Signal Processing
• Preserving the Battery power of edge devices
• Extracting information from ambient-noise-corrupted sensor data
 Low-power computing algorithms
 Adaptive Signal Processing for Ambient Noise Removal
Communication
• Reducing the cost of Communication while preserving reliability and security
• Preserving the Privacy
 Secure, Lightweight yet Reliable communication protocols over Internet
 Quantifying Privacy vs. Utility measures
Computing
• Knowledge and Model Driven Analytics in Real-time
 Distributed Computing - Bringing Edge into the Grid
 Semantics driven Analytics - Reasoning and Semantic Sensor Web
 Model driven Analytics - Cyber-physical system modeling / Human behavior modeling

6. 6
Integrated Platform for Intelligent Infrastructure
People Feedback & Emotions
Social Media
Integrated Services
Sensors & IoT
Platform
Traditional Monitoring & Control
Systems
Citizen Data
Smart Integration Platform
Transportation Healthcare Electricity
WaterPublic Safety Tourism
Smart Integrated Services
Sense
Analyze
Extract
Respond
Intelligence
Smart Domain Services
Community
etc.
Sense: People Activity, Appliances, Vehicles , Road, Home/Bldg, Utility
Infrastructure
Detect gas leakage/water contamination :
mobilize rescue team, suggest optimum route
Divert Road Traffic in case of
Water Pipeline Burst
Correlate Electricity/Water
/Gas consumption patterns
Intelligent Integration Platform
Integrated Intelligent Services
RIPSAC – Real-time Integrated Platform Services & Analytics for Cyber-physical Systems

7. 7
Platform Architectural Layers
Internet
End Users &
Renderers
Administrators
Device Integration & Management Services
Analytics Services
Application Services
Storage
Messaging & Event Distribution Services
ApplicationServices
Presentation Services
Application Support Services
OS & Device Drivers
Sensor Signal Processing
Middleware
Middleware(Security/PrivacyandProtocols)
Edge Gateway
Sensors
Internet
Back-end

8. 8
Experience certainty.
Cyber Physical System – Research Theme
RIPSAC
Infrastructure
Simulation
New Sensing
Platforms,
5 Senses
Computing
Energy
Water
Transportation
Security
& Privacy
Distributed
Computing
Facilities
Data Integration
& Visualization
Safety
Buildings
Home
City
Vehicles
Non revenue
water Long term network
expansion
Renewable
sources
Energy
conservations
Intelligent
Infrastructures
Sensor Informatics
for Home
Sensor Informatics for
Vehicular Monitoring
Transport planning
Electricity Grid
Optimization
algorithms
Big data
Communications &
Sensor Informatics

9. 9
Experience certainty.
Human-in-Loop Cyber-physical System – Research Theme
RIPSAC
2D/3d Surveillance -
Kinect
Physiological
Sensing –
ECG, EEG, …
Physiology
Identity
Smartphone
Sensors –
GPS,
Accelerometer,
Magentometer,
Gyropscope,
Camera, Mic
Healthcare
• Elderly People Monitoring
• Chronic Disease People
Monitoring
Personalized learning
• Cognitive Load
Web and
Social Media
Smart
Meters
Wellness
• Fitness
• Stress, Fatigue
Organizational Behavior
• Personal Physical Network
Discovery
• Team Efficiency / Best Practice
Analysis
Retail
• Customer
Behavior Study
Driving
• Stress
• Fatigue

10. Research Projects – Ubiquitous Healthcare

11. 11
Intelligent Healthcare – Elderly / Chronic Patient Monitoring
ECG
Blood Pressure
Monitor
Pulse OxyMeter
Mobile phone
as medical
gateway
Patient
Records
Health Center / Home
Expert Doctor
• 2 papers in SMART 2012 and MobiHoc 2012
• Requirement from NUH (Peritonial Dialysis) and
SingHealth (Elderly Care)

12. 12
Intelligent Healthcare – Elderly / Chronic Patient Monitoring
Mobile phone
as medical
Sensor
Patient
Records
Health Center / Home
Expert Doctor
• Replace Medical Sensors with Mobile Phone Sensing
• Activity and Localization using Mobile Phones
• Activity and Identification using Kinect Camera
• Social Media as a Soft sensor
• Multimodal Fusion
Identification,
Localization and
Activity
Detection
• 1 paper in UbiComp 2013

13. 13
PPG based Pulse Measurement using Phone Camera
Subject1 Subject2 Subject3
Actual Detected Actual Detected Actual Detected
68 66 66 63 85 84
2.9% 4.5% 1.1%
Subject Actual Pulse
Rate
Detected Pulse Rate (20 sec
skip)
Detected Pulse Rate (automatic
detection)
PSNR (automatic
detection)
A 54 53.542 53.542 22.887
B 66 63.404 63.404 15.351
C 84 84.539 84.539 19.831
D 106 98.629 98.629 21.741
E 80 88.062 88.062 14.788
F 105 102.15 102.15 16.705
G 105 38.747 0 0
H 80 42.27 0 0
• 1 paper in Mobihoc 2013
• 1 patent filed TCS Fit4LIfe Pilot

14. 14
Blood Pressure and ECG Monitoring from PPG
Extract PPG
Features
BP Ground
Truth
Create BP Model
Extract PPG
Features
Predict BP levels
(SP, DP, PP)
Extract ECG
Features
Create ECG
Parameters’ Model
Predict ECG
Parameters
ECG Parameters
BP levels
(SP, DP, PP)
Training Phase
Testing Phase
Data set Pd Ps PP-diff < 15
Standard dataset with 14 PPG features 92.9% 74.7% 77.9%
TCS recorded dataset with 14 PPG features
+ height, weight, age
99.3% 82.7% 85.5%
BP Level Pd Ps
Very Low < 50 < 70
Low 50-65 70-100
Normal 65-90 100-
135
High 90-100 135-
160
Very High > 100 >160

15. 15
Mobile Phone based Activity Detection and Localization
Activity Detection
– Uses Accelerometer Data
– Gyroscope and Magnetometer for orientation
correction
– Step Count, Stride Length Estimation
– Walking, Brisk Walking, Running Classification
Indoor Localization
– Initial Referencing through GPS and
Magnetometer
– Inertial Navigation through accelerometer,
gyro, magnetometer
– Improved accuracy through Stride
Length, Kalman Filter / Particle Filter
– Tracking of non-smartphones via
Bluetooth
Location ID LOC I LOC II LOC III
Actual (ft) 2 4 6
Estimated
(ft)
1.95 4.16 6.27
% Error 2.5 4.12 4.4
• 1 paper in UbiComp 2013
• 3 patents filed
Activity Detection for TCS Fit4Life
Generalized indoor localization – Colleague Finder,
Retail, Emergency Evacuation

16. 16
Kinect Based Identification and Activity Detection
Human Identification
– Skeleton Model Based / Depth based
– External Stimulus based refinement
– Accuracy of 90% for less than 10 people
database
• 2D Camera with IR
depth sensor
• Excitation by IR light
pattern
• Directional Mic.
Activity Detection
– Enter/leave a room
– Sitting / Standing / Walking / Lying down
– Going to Bathroom
– Taking Medicine LIRIS human activities dataset
• 4 papers in Fuzz IEEE 2013 , CEC 2013, UbiComp 2013, SMC 2013
• 2 patents filed

17. 17
Social Media as a Soft-sensor for Healthcare
 Support Community Discovery
– Use hidden community detection by applying NLP on
posts to create the social graph to identify the
undeclared community for a given disease
 Disease onset discovery like dementia or Alzheimer's
disease or psychological disorders from social
network posts
– Search for patterns in posts to detect possible
symptoms to diseases
– E.g. - sentiment analysis on posts will give whether the
given post’s emotion is positive or negative. If the
emotions are cycling between positive and negative
extremes with some periodicity, probably the person
has bipolar disorder
• 1 paper in UbiComp 2012

18. Research Projects – New Sensing Techniques

19. 19
Experience certainty.
Bio-sensing for Cognitive Load
User Study
 Biased and Indirect
Biological Response using EEG
 Unbiased & more reliable method
• 2 papers in Fuzz IEEE 2013 , SMC 2013
• 2 patents filed
• Can be used for Usability Testing of GUI

20. 20
Experience certainty.
5 senses Computing - 3D Reconstruction with 2D images from
mobiles
• Low cost solution for 3D reconstruction from multiple 2D images captured from
mobile device.
• Derive the motion information from the inbuilt sensors of the mobile phone and then
aid in increasing the accuracy of the 3D reconstruction.
• Dynamic 3D reconstruction in outdoor environment which is not possible using a
Kinect.
• 3D reconstruction of homogeneous objects
• Future research focus on the multi-modal fusion of the 3D information and the other
sensors’ data for a given object or environment
Applications
• Agro-advisory Service
• Remote Diagnostics of Machines
• Remote Healthcare
• 1 paper in VISAPP 2012

21. Research Projects – Communication and
Computing

22. 22
Experience certainty.
Grid with an Edge
• CPU utilization better in Condor
• Condor performs better with more nodes
• Further advantages w.r.t Heterogeneity,
Versatility, Matchmaking & scheduling
• 3 papers in NGMAST 2012, PDPTA 2013, WiMob 2013
• 2 patents filed
• Flavio Bonomi et.al. MCC2012, Helsinki, Finland

23. 23
Experience certainty.
Communication Aspect- Replace HTTP
 Constrained Object Access Protocol (CoAP)
 Improved version of CoAP using dynamic network condition sensing
 Lightweight security protocols sensor authentication and data delivery
Use suitable lightweight application protocol between edge devices and core network
• 3 papers in ICDCN 2013, ICNC 2013, WPMC 2013
• 2 patents filed

24. 24
Experience certainty.
Knowledge-driven Analytics
Ontology and Description for Sensors
• Non-standard / Multi-standard sensor descriptions across multi-domains
• Bringing all sensor data in common semantic form
• Semantic Sensor Web (OGC-SWE)
• Will provide ease of sensor discovery
Ontology and Knowledge based Reasoning
• Reasoning over continuously flowing (streaming) facts
• Reasoning over streaming data is integrated with “background knowledge”
(static unchanging or slowly changing knowledge
• Spatio-temporal Queries
Semantic Analytics
• Analytics-as-a-Service
• Query language to specify processing workflow on Time series data
• 5 papers in ICST 2012 , ICNC 2013, ICUIMC 2013, ODBASE 2013, KES 2013
• 4 patents filed

25. 25
Experience certainty.
Privacy Preservation
• Information-theoretic approach for sensitivity analysis of sensor data
• Requirement based application utility measurement
• Balancing of Privacy vs. Utility
Privacy
Utility
• 1 paper in SecurIT 2012
• 2 patents filed
Privacy
Preservation
Tool

26. 26
Lab-at-a-Glance
Research Areas
• Sensor Signal Processing
• 2D/ 3D Image / Video Processing
• Protocols, Security and Privacy
• Parallel and Distributed Computing
• Stream Processing and Reasoning
• System Modeling and Identification
• Sematic Sensor Web
• Social Media Analytics
Academic Collaborations
• Singapore Management University (iCity Platform)
• Indian Statistical Institute (Protocol/Privacy/Security,
Image / Video Processing)
• IIT Kharagpur (Analytics, Semantic Processing)
• IIT Bombay (Energy and Utilities)
• Jadavpur University (Signal Processing)
Higher Studies
• PhD - 4
• Masters - 4
Total Researchers - 37

27. Thank You
arpan.pal@tcs.com