Today, I am going to talk about our work on the development of “Web-based platform for privacy protective avatar mediated distant-care”.
I will begin with the background of our work.
The number of households in which an elderly lives alone has increased. Social isolation of the elderly in such households has been acknowledged as a problem because it becomes difficult to notice the change of their health status.
Additionally, daily support and watching over the elderly, which used to be supported by the mutual aid of family members and local community, are now supported by human workers involved in nursing and welfare services, offered by local governments or private sectors. This is also recognized as one of the problems of this isolation, because it inevitably leads to the increase of social welfare expense.
Because of this isolation problem, various ICT distant-care services are provided. However, in the current state of popularization of ICT based care services, only simple and low-tech services are widely used, such as an emergency call with push button, locating services with wearable IC tags. Also, the care services that utilize advanced ICT device such as cameras and sensors are not widely used.
Additionally, the elderly and family members avoid such advanced ICT technologies because of the concern about privacy invasion.
it is said that privacy, the person’s right to control personal information revealed to others, should be opt-in especially in Europe.
also, opt-in policy is the property of having to choose explicitly to join or permit something.
When designing distant-care systems to observe the elderly, it is required to continuously observe the elderly for a long-time. For this reason, observations that are based on video capturing are often used. To protect one’s privacy in capturing video, many researchers developed technological solutions such as filters, automatic blurring and so on. However, such solutions are not enough to protect one’s privacy completely. For example, Filtering and automatic blurring cannot conceal one’s nudity.
Also, capturing video is not suited for opt-in because it is difficult to filter the information provided.
On the other hand, in sensor-based approach, it collects only the kinds of information that are permitted to be seen by the elderly, but it is difficult to understand the meanings of the value of sensor data.
Then, we proposed a solution that render an avatar of the elderly by using sensor data which is collected by motion capturing system. By doing so, it can be recognized elderly’s behavior without capturing video.
Therefore, our purpose is to propose a platform for distant-care system. In the platform, the elderly is observed as an avatar which is rendered by articular angle data of the Kinect. The system is privacy protective and suited for both opt-in and opt-out. Using this system, caregivers can observe the elderly real-time, and can be remote accessed via internet.
I will explain about the overview of our platform.
The system is composed of a three-sided client-server model: the server, caregiver side client, and elderly side client. The client of the elderly side consists of a KINECT sensor, programs for articular angle data acquisition and data processing, and a program to send the processed data to the server.
The server sends the data [click] received from the elderly side client [click] to the caregiver side program. The caregiver side program runs on a web browser. The program receives the articular angle data from the server and applies it to the avatar model data for rendering.
Generally, OpenGL, OpenGL ES for mobile, and DirectX are used in the development of applications using 3D Graphics. However, one of the problems in using these APIs is that we have to develop and to support applications compatible with each device and OS used. Another problem is the application has to be installed into the devices by users.
[click] Therefore, we used WebGL for rendering avatar animation.
The sensor data is sent from the elderly side client through the server to the caregiver side client, 15-30 times per second.
HTTP, a traditional communication protocol between a server and web-browser, requires a large amount of HTTP header in every request and response communication which leads to a significant delay for such kinds of communication. Additionally, the server cannot send the data actively to the web browser with this protocol.
On the other hand, WebSocket, a extended HTTP protocol, can realize real-time two-way communication between a server and web-browser. And also, WebSocket protocol is supported in almost all modern web-browsers.
[click] Therefore, we used WebSocket protocol for client-server communication.
Recently, various types of wearable sensors such as Apple Watch and Galaxy Gear have been developed. Using these sensors, we can continuously monitor one’s vital information such as blood pressure, heartbeat-rate, oxygen level, and so on.
The platform we developed can be customized by integrating these wearable devices to adapt to a huge variety of health-care situations.
This is an example device of sensor integration.
We prepared a sensing device including [click] a temperature and humidity sensor, and [click] heartbeat-rate sensor. These sensors are connected to a micro computer Arduino Uno.[click] and Arduino Uno is connected to a one-board computer, Raspberry Pi[click]. The Raspberry Pi is connected to a Wi-Fi adapter[click] and Battery.
The sensor data is sent from Arduino Uno to Raspberry Pi via serial communication, then to the server via Wi-Fi with WebSocket protocol. The sensor data is processed and formatted in JSON [click].
Let me show you an example of how our system works by using the platform.
This is a screen capture of caregiver side client running on the web browser. The avatar which reflects the elderly’s behavior is rendered on the left side. The location of sensed joints are depicted in the box at lower right.
The data of integrated sensors such as temperature, humidity, and heartbeat-rate, are shown at the green boxes at upper right. When the value of sensors goes beyond the set threshold, the green box turns red.
It is expected that delays will occur in the animation of the caregiver side client.
We explored the minimum bandwidth for the system to properly work without delays. In our test, we set the frequency of data acquisition/sending to 15frames per second, which is the lower limit for the animation to be recognized as one’s behavior.
The data amount of the articular angles was 6Kbytes a frame. And the total data amount of example sensing device was 50Bytes a frame. Both of these data amount are after processed to JSON format.
We measured the performance of the system ten seconds after communication was established.
We used two indexes: The average frame-rate, and the average time lapse after the server sent the data.
This graph shows the frame-rate and the time lapse when the bandwidth was controlled from 350Kbit/s to 850Kbit/s. The results shows that when the bandwidth was above 800Kbit/s, the animation was rendered smoothly without delay.
However, when the bandwidth was below 800Kbit/s, delay was seen and was accumulated with each passing moment. For example, when the bandwidth was 350Kbit/s, the 11 seconds delay was occurred.
From the result, assuming that mobile telecommunication bandwidths of late years are not under multiple Mbit/s, caregivers can observe the animation of the avatar without delay via mobile network.
Psychological burden such as how elderly react by having their behaviors tracked and showed in the form of virtual avatar of both being the observed and the caring by introducing our system into the nursing and personal care facility.
We held test demonstration at a nursing home for the elderly, we will continue doing so as additional research.
Development of Web-based
Platform for Privacy Protective
Avatar Mediated Distant-Care
YU KOBAYASHI, DAI HASEGAWA, SHINICHI SHIRAKAWA,
HIROSHI SAKUTA, EIJUN NAKAYAMA
AOYAMA GAKUIN UNIVERSITY, JAPAN
Social Requirements for the Elderly
Difficult to notice the change of
elderly’s health status
Falling down, Heart attack, …
Blood pressure, Posture, …
Needs of daily support and watching
over for elderly
Used to be supported by family
members and/or local community
Supporting business by human
Increase of social welfare expense
1980 1985 1990 1995 2000 2005 2010 2015 2020 2025 2030 2035
The number of households in which elderly who
lives alone in Japan*
*Cabinet Office, Government of Japan, ”Annual Report on the Aging Society”, 2015
Distributed ICT Care Services
Simple and Low-tech
(e.g. Emergency call services with push button devices)
Advanced ICT devices are not widely used
(e.g. cameras, sensors, …)
Trade-off problem of convenience in care services and privacy invasion
for Elderly and family members
The person’s right to control personal information revealed to
Should be opt-in (especially in Europe)
Of a selection, the property of having to choose explicitly to join or
Camera-based observations are often applied
No effective solutions to protect one’s privacy in capturing video
(e.g. Filtering and automatic blurring cannot conceal one’s nudity)
Capturing video is not suited for opt-in
(Difficult to filter the information)
No video capturing
Only collects the information which is permitted to be seen by the elderly
(Suited for opt-in)
Difficult to understand the meanings of the value of sensor data
Observe an avatar
Only collects the articular angle data by motion-capturing system (Kinect)
Observe an elderly as an avatar which is rendered by sensor data
Can be recognized elderly’s behavior
Propose a platform for distant-care system
Sensor-based observation (motion capturing with Kinect)
observe the avatar of an elderly
No video capturing required
Suited for both opt-in and opt-out
Real-time and remote accessible via Internet
Server Multi Devices
Caregiver SideServer SideElderly Side
Avatar Animation 8
OpenGL, OpenGL ES, DirectX
Need to develop
for each device or OS
Installing application needs
Difficult to support
Runs on modern browsers
PCs and Mobiles
A traditional communication protocol between a server and a web browser
Cannot send the data to web browser actively
Every request / response needs a relatively large amount of HTTP header
Only one HTTP header is needed (low-load)
Supported in almost all modern web browsers
Various types of wearable sensors have been developed
(e.g. blood pressure, heartbeat-rate, oxygen level)
Apple Watch Samsung Galaxy Gear
Example of Sensor Integration
Temperature / Humidity
Required Bandwidth Test
Data amount (15 FPS)
Articular angles: 6 Kbytes (JSON)
Sensors: 50 Bytes (JSON)
Measure the performance 10 seconds after communication
The average frame-rate
The average time lapse after the server sent the data (latency)
Result of Required Bandwidth Test
Above 800Kbit/s :
the animation was rendered smoothly
Below 800Kbit/s :
the delay was accumulated with each
(e.g. 350Kbit/s, 6FPS, 11s latency)
The animation of the avatar rendered
via mobile network.
200 400 600 800 1000 1200 1400
Conclusion and Future Works
We proposed a web-based platform for privacy-protected avatar mediated distant-care.
We measured the bandwidth required for avatar animation without delay.
We will evaluate the system’s …
Reveal how to detect any other particular health problems.
Test demonstration at a nursing home for elderly.
This research was partially supported by “Strategic Information and
Communications R&D Promotion Programme” (SCOPE) .