Homeostasis is nature’s engineering behind the most complex autonomic system that exists: the human body. Homeostasis is a self-regulating process by which biological systems tend to maintain stability while adjusting to conditions that are optimal for survival. Disruption in homeostasis results in malfunctioning of natural autonomic system causing chronic diseases. Chronic diseases have been the leading cause of death and human suffering in the last 50 years. They also have resulted in highest financial burden for individuals and countries. This can be corrected using external augmentation of the homeostasis loop. Recent progress in artificial pancreas for Type 1 Diabetes is a compelling example for such augmentation. In this paper we discuss emerging multimodal approaches for such augmentation in the context of chronical diseases. We show that multimodal sensing and fundamental technology developed by multimedia computing community may offer powerful augmentation of natural homeostasis to assist in management of chronic diseases.

1. Multimodal Augmented Homeostasis
Ramesh Jain
Department of Computer Science
UCI Future Health (https://futurehealth.ics.uci.edu/)
University of California, Irvine
jain@ics.uci.edu
With Several Collaborators

2. Multimodal Augmented Homeostasis: Agenda
• What is Multimodal Augmented Homeostasis?
• Health as a ‘Grand Challenge’ for Multimodal computing.
• Personal Health Navigator for Augmenting Homeostasis.
• Progress and Challenges.
2

3. Papers that are related to the ideas presented here.
1. Ramesh Jain, Laleh Jalali:
Objective Self. IEEE MultiMedia 21(4): 100-110 (2014)
2. N. Nag and R. Jain, “A navigational approach to health: Actionable guidance for improved quality of life,”
IEEE Comput., vol. 52, no. 4, pp. 12–20. April 2019.
3. Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, Ramesh Jain, “Food Computing”, ACM Computing
Surveys, 52(5):1-36, September, 2019
4. Weiqing Min, Shuqiang Jiang, and Ramesh Jain, ‘Food Recommendation: Framework, Existing Solutions,
and Challenges’, IEEE Trans on Multimedia, November 2019.
5. Nitish Nag, Vaibhav Pandey, Ramesh Jain:
Health Multimedia: Lifestyle Recommendations Based on Diverse Observations. ICMR 2017: 99-106
6. Hyungik Oh, Ramesh Jain:
From Multimedia Logs to Personal Chronicles. ACM Multimedia 2017: 881-889
7. Nitish Nag, Vaibhav Pandey, Preston J. Putzel, Hari Bhimaraju, Srikanth Krishnan, Ramesh Jain: Cross-
Modal Health State Estimation. ACM Multimedia 2018: 1993-2002
8. Hyungik Oh, Ramesh C. Jain:
Detecting Events of Daily Living Using Multimodal Data. CoRR abs/1905.09402 (2019)
9. Nitish Nag:
Health State Estimation. CoRR abs/2003.09312 (2020)
10. Ali Rostami, Vaibhav Pandey, Nitish Nag, Vesper Wang: Personal Food Model. ACM Multimedia 2020.
11. Laleh Jalali and Ramesh Jain: Event Mining for Explanatory Analysis (Book), Coming from ACM Books. 3

4. Augment
• “technologies that enhance human productivity or capability, or that
somehow add to the human body”
• to make greater, more numerous, larger, or more intense.
• make (something) greater by adding to it; increase.

5. Augmented Reality

6. Multimedia
• Essential for understanding real world
•No single mode provides complete
information in a real situation.
• Different modes must be combined for
understanding a situation.
• Different modes have complementary
information.

7. Multimodal is the future of Multimedia
• Multimedia
• GPS
• Heartrate
• Temperature
• Accelerometer
• …
7

8. Big Data is Multimedia Data
Multimedia
Realtime
Credibility

9. Dominant Applications of Multimedia
• Image Search
• Video on demand
• Video-conferencing
• Shopping
• Social Media
• The Best is yet to come – just now becoming popular.
9

10. The Most Important Application of Multimedia
Computing?
Health

11. BUT,
Healthcare is broken
in every country. Can Multimedia Community Help?

12. What is Homeostasis?
• Homeostasis is the state of steady internal, physical,
and chemical conditions maintained by living systems.
• Maintain optimal conditions for
• Temperature
• Glucose level
• pH
• Oxygen
• …
• Maintains the constancy of the internal environment in which the
cells of the body live and survive.
• Works every moment of life.

13. Important Turning Point in Health
In the mid-20th century, the primary causes of death
worldwide shifted from infections to chronic conditions.

14. The Trend in Global Health is Clear.

15. * Source: CDC
The Effect of Chronic Diseases, 2016*
7 of 10 Deaths
Due to chronic diseases
86% of Costs
Driven by treating chronic
diseases
Heart Disease Cancer Type 2 Diabetes Brain Diseases Lung Diseases Bone Diseases
We all are likely to face Chronic Diseases.

16. Why are Chronic Diseases so Common?
• All bodily functions deteriorate with age.
• When this dysfunction affects Homeostasis, the Onset of the Chronic
Disease starts.
• Usually by the time, this is detected, it is too late.
• P4 (predictive, preventive, personalized and participatory) is the
new Paradigm in Healthcare
16

17. Health Factors
This needs attention.

18. Homeostasis is Nature’s Engineering
Homeostasis: any self-regulating process by which biological
systems tend to maintain stability while adjusting to conditions
that are optimal for survival.
18

19. Cybernetics: Feedback revolutionizes system design
1. Sensors to detect State
2. Model – of the person
3. Controller: Logic and Inputs

20. Basic Systems Theory
Where x, u, and y are state, input, and output vectors;
A, B, C, and D are appropriate matrices for transformation.
Observability: Determine state using measurements.
Controllability: Get to desirable state.

21. Type 1 Diabetes
• Your pancreas doesn't make insulin or makes very little insulin.
• Treatment aims at maintaining normal blood sugar levels through
regular monitoring, insulin therapy, diet, and exercise.
21

22. Impact:
It has been a life-
saver for many
patients and their
families. ‘Artificial Pancreas’ for Type 1 Diabetes.
Cybernetics is now Used for Augmenting Homeostasis
Miracle for Type 1 Diabetes Patient

23. Why Are These Birds Huddling?
By Blalonde - Own work, CC BY-SA 4.0,
https://commons.wikimedia.org/w/index.php?curid=37172033

24. Augmenting Homeostasis: Want to help yourself!
24

25. Continuous Augmentation
25
An additional layer of homeostasis!!!

26. Augmented Homeostasis:
• Health State Estimation.
• A disease is just an undesirable region in health state space.
• Guidance: life-style, environment, and medication.
• Actuation
• Personal Model
• Goal State
• Close-loop perpetual operation
Self-regulating external process by which human systems achieve
health goals to maximize their quality of life.

27. Augmented Homeostasis Architecture
Controller
Continuous
Observations
Episodic
Observations
Personicle
Event Mining
Lifestyle
Actions,
Meds
Health
State
Person
Person
Model
Cloud
Goal
External
Factors
27
Control
Signals

28. Traditional Episodic Health Cycle
How can I
get more
attention?

29. Getting to a destination: 30 Years Ago.

30. Digitalization made getting directions easy.
Maps
Points of Interest
Starting Point
Destination
Mapping
System
Getting to a destination: 20 Years Ago.

31. Getting to a destination: 10 Years Ago.
Maps
Points of Interest
Current State : GPS
Current Traffic:
Waze
Destination
Cybernetic loop
Navigation
system
Knowing Current State is Magical.

32. Measure
Collect all data
Perpetual Health Guidance
Guidance
Estimate
(Health State)
Influence and
Compliance
How do we close
the loop?
Inspiration:
Navigation Systems.

33. When do people get best healthcare?
When they are about to die – in ICU.

34. Personal Health Navigator
•Guides you to better health.
•Uses real-time information to redirect you
around a disease.
•Provides step-by-step directions to get you to
your optimal health state, whether you’re a
professional athlete or a taxi driver.

35. Personal Health Navigator

36. Personal Health Navigator: Diabetes
Activity
Food
Medication
Glucose Level
Activity
Food
Medication
Glucose
Level in last
3 hours
Sensitivity to
different foods.
Sensitivity to
medication.
Activity Checks.
Glucose Level
Checks.

37. • Ping Wang and Ramesh Jain co-founded,
then joined by other people
• Licensed UCI technology
• A start-up company formed (Team of 7 now,
raised seed funding)
• App available on iPhone
Personal Diabetes Navigator
Dianavi is building Personal
Diabetes Navigator.
Their application is now
available on iPhone.
Disclaimer: I am a co-founder.

38. Sensors to Estimate Health State
• Two major classes:
• Low cost, continuous, unobtrusive (LCU) – Used in Wellness
• High cost, episodic, intrusive (HEI) – Used in Diagnostics
• Currently LCU is used for ‘Wellness’ and HEI for ‘Disease Control’.
• Though both are related and complementary, they are treated in
different silos.
• Both are to improve the quality of Life of a person.

39. Low Cost, Continuous, Unobtrusive Sensors (LCU)
Observing you all the time.

40. High Cost, Episodic, Intrusive (HEI)
Doctors use for better and precise diagnostics.

41. General and Personal Health State Space
General Health State Space Personal Health State Space

42. Heart
Failure
Olympic
Athlete
Coronary
Disease
High Blood
Pressure
Good Hiker
Healthy
Pregnancy
Health State: Multidimensional Space
Health state is what matters,
disease is just a subset of it.

43. Person :== Genome + Lifestyle + Exposome
‘Omics’
Personal
Model
Exposome
Personicle
AI + Learning
• Propensity to
diseases
• Rule Based
Health Behavior
• Your behavior
and health profile
Could be for
Whole Life
Recent Past
After a particular health Condition
(such as heart attack)

44. From data streams to Personal Chronicle (Personicle)

45. Add other personal streams as
NEEDED.
RAMESH JAIN@Imaging and Multimedia Analytics 45Jan 29, 2020

46. Personicle: Personal Chronicle
46
Personicle is a complete chronological log of person-
centered events in life (health, social, environmental)
captured objectively but augmented using subjective and
diagnostic information as needed.
Continuous
Observations
Episodic
Observations
Personicle
Person
• All continuous sensors work 24/7.
• Episodic data when suggested by
• Either continuous data tracking or
• By doctors.

47. Open Source Personicle
47
• Being built by Clearsense – Thanks, Charles and the team!

48. Pattern
Mining
High-level
Pattern
Formulation
Pattern Query
…
Data Streams Event Streams Semi-interval Event Sequences
Data-Driven Analysis
Hypothesis-Driven Analysis
While air pressure is high, pollution
starts increasing gradually, within T
time units asthma outbreak happens.
((pollution_inc_steadily ;ωT asthma_outbreak ) ||
airpressure_stayshigh)
Example:
Interactive Visualization
Interactive Event Mining: Correlation and Causality
Bottom Up
Top Down

49. Input to the System
• Food
• Exercise/activities
• Sleep
• Stress
• Environment
• Medicine
49

50. Food is the most important input.
50What I like to eat is not what my body likes!

51. Food Recommendation
•Who: Person.
•What: Food Items.
•When: Context. Food
Recommendation
System
Food
Items
Context
Personal
Model
List and
volumes
of Food
Items
and How
to get
them.

52. Food Logging is important application.
52
Information in a Food Log for Personal Food Model
Learn from Activity and sleep
Logging.
Can we develop a platform
for easier logging?

53. Building Food Model: Health
53

54. Two Challenges to Solve the Problem!
• Creating a detailed Food Chronicle
for a person.
• Creating a detailed World Food
Atlas.

55. Event Mining for Rule-Based Actionable Insights
Having difficulty sleeping well on several nights.
Warn him when he is at a Sichuan restaurant.
Personicle
Food Stream
Sleep
Spicy food and 2 glasses of wine result in sleepless
nights.

56. Homeostasis is essential for Life.
• Sensing for health state is multimodal.
• Multimedia community is expert in information extraction from
multimodal data streams.
• Augmented Homeostasis is a new grand challenge for our
community.
56

57. My dream: Empowering Personal Health
For every living person in the world.
Multimodal Computing will bring it.
We can make it happen.
Should we start? Now?

58. Thanks for your attention and Time
Time for Questions and Discussions
Continue after the session: jain@ics.uci.edu
58