Eat, Drink, and Enjoy
Ramesh Jain
(jain@ics.uci.edu)
With Several Collaborators

Human Evolution: Economist Dec 13, 2003.
Let’s accelerate the evolution.

Food is Most IMPORTANT in Life!
1. Social.
2. Religious.
3. Enjoyment.
4. Life.
5. Business
6. Agriculture
But has been mostly ignored by Computing community.
Time to Change.

My interest in Food:
I Love Eating!

My interest in Food:
Love Multimodal Culinary Event Experience

Visual

Smell

Taste

Tactile

My interest in Food:
‘merge’ my research passion with real life passion
1. Ruihan Xu, Luis Herranz, Shuqiang Jiang, Shuang Wang, Xinhang Song, Ramesh Jain:
Geolocalized Modeling for Dish Recognition. IEEE Transactions on Multimedia 17(8): 1187-1199 (2015)
2. Nitish Nag, Vaibhav Pandey, Hyungik Oh, Ramesh Jain:
Cybernetic Health. CoRR abs/1705.08514 (2017
3. Nitish Nag, Vaibhav Pandey, Ramesh Jain:
Health Multimedia: Lifestyle Recommendations Based on Diverse Observations. ICMR 2017: 99-106
4. Nitish Nag, Vaibhav Pandey, Abhisaar Sharma, Jonathan Lam, Runyi Wang, Ramesh Jain:
Pocket Dietitian: Automated Healthy Dish Recommendations by Location. ICIAP Workshops 2017: 444-452
5. Nitish Nag, Vaibhav Pandey, Ramesh Jain:
Health Multimedia: Lifestyle Recommendations Based on Diverse Observations. ICMR 2017: 99-106
6. Hyungik Oh, Jonathan Nguyen, Soundarya Soundararajan, Ramesh Jain:
Multimodal Food Journaling. HealthMedia@MM 2018: 39-47
7. N. Nag and R. Jain, “A navigational approach to health: Actionable guidance for improved quality of life,” IEEE Comput.,
vol. 52, no. 4, pp. 12–20. April 2019.
8. Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, Ramesh Jain, ACM Computing Surveys, 52(5):1-36, September,
2019
9. Weiqing Min, Shuqiang Jiang, Ramesh Jain:
Food Recommendation: Framework, Existing Solutions and Challenges. CoRR abs/1905.06269 (2019)
10. Hyungik Oh, Ramesh Jain:
Detecting Events of Daily Living Using Multimodal Data. CoRR abs/1905.09402 (2019)

What is the the most effective Medicine?
Zachary Zavislak for TIME

That is Old News!
Food Pyramid

I am in Nice, France!
• How do I get ‘there’: From my Hotel to Conference center?
• What do I eat and where?
• I have diabetes/allergy.
• I am a vegetarian.
• I am a vegetarian and have diabetes.
• I am a very picky eater.
• Have only 1 hour time.
• I am on tight budget.
Two problems I face!

Two Challenges to Solve the Problem!
• Creating a detailed Food Chronicle for a person.
• Creating a detailed World Food Atlas.

Food Recommendation
•What: Food Items.
•Who: Person.
•When: Context.
Food
Recommendation
System
Food
Items
Context
Personal
Model
List and
volumes
of Food
Items
and How
to get
them.

Challenges for Technology: Dish
Eating Well does not mean eating Tasteless.

Millions of Books on Food are Available!
Most non-fiction books in a bookstore are on Food.

Problem with Books: Old Technology.
• Books are not Actionable
• Books are not Personal
• Books are not Situational
• Do you still read paper books? Really?
People want Right Information, at Right Time, in
Right (Actionable) form.

Challenges for Technology: Dish/Food Items
• Ingredients
• Their source
• Nutritions
• Calories
• Taste
• Season and ingredients
• Cooking style
• Volume
• Individual items
• Cost

Society exists only as a mental concept;
in the real world there are only
individuals.
-- Oscar Wilde
Problem with Books: Personalization

An Individual is not an average of a Population
Longitudinal data for N-of-1 Studies.
An Average human has:
One breast, and
One testicle?????

Personal Factors for Food
• Religion
• Social
• Allergies
• Preferences
• Taste
• Price
• Health condition
• Health goals

How can we get this information for a person?
• Food preferences
• Likes and dislikes
• Limitations
• Social/Religious
• Health
• Recent History
• What has been eating
• Current situation
• Company
• Budget
Challenges for Technology: Person

Knowing a Person. Can we?
Businesses must understand their Customers.

Preferences
Environ-
mental
Allergies
Health
Social
And
Religious
How do I find all this?
Personal model is built using logs data.

Personal Chronicle: Personicle
Personicle is a complete chronological record of
person-centered events in life (health, social,
environmental) captured objectively but
augmented using subjective information as
needed.

Use some streams and
determine life events.
Add other personal streams as
NEEDED.

Most Important Lifestyle Factor:
Diary Food Database Scanning Barcode Taking a Picture
Current Techniques are Unsatisfactory.
Most Important Lifestyle Factor: Food Chronicle

Current Popular Approaches for Food Journal in
Multimedia
• FoodLog by Prof. Kiyo Aizawa.
• Many Food Recognition systems: Very active research
• Food Volume: Use one or more pictures
• Nutrition: Ingredient/Recipe recognition
Let’s practice Multimedia.
Use all relevant information
We are becoming camera-hammers.

Potential Solutions
• Triggering a food journaling process in a timely, proactive
manner.
• Improving the self-reporting while preserving high accuracy
• Other sources: electronic payments
• Automatic food logging
• New sensors

• Eating Moment Recognition
• Location
• Heart rate
• Speech Based Food Journaling
• Enhancement
• Payments
• Recall tips
Eating
One Approach: Food Channnel

Event Stream Mining for Rule-Based Actionable Insights
Spicy food and 2 glasses of wine result in sleepless nights.
Warn him when he is at an Sichuan restaurant.
Personicle
Food Stream
t1 t2 t3 t4 t5
Sleep
Representing different events in time band form.

Challenges for Technology: Context
• Location/availability
• Local regulations
• Seasonal/Environmental variations
• Personal immediate history
• Social Company
• Cost

Most Challenging Context
• Available dishes near a location
• Finding local variations in dishes and their effect on taste and
nutrition
Solution:
• Crawling and analyzing each dish on the menu of every restaurant.
• Understanding recipes used at home to analyze dishes.
• Think dish level rather than restaurant level

Should We Build Food Knowledge Graphs?
• Knowledge Graphs are the reason for efficient search – that we all
love.
• Why not creat Food Knowledge Graphs.
• Create World Food Atlas using Food Knowledge Graph.

Dish Name
Has
Ingredient
Culinary
Class
Calories
Nutrition
Has
Recipe
Recipe
Source
1000
Fat, …
Peanuts
French
Taste
Vector
Multi-
modal
aspects
Other
Names
Geo-
spatial

Understanding Tastes: FlavorNet
Bitter, salty, sour, astringent,
sweet, pungent (eg chili), and
umami.

Grand Challenges
•Creating a detailed Food Chronicle for a person.
•Creating a World Food Atlas.
To enjoy food, we must solve these two challenges
using Digital Technology.

Thanks.
jain@ics.uci.edu
Data is the Key.