The document discusses the author's interest in using technology to better understand food and provide personalized recommendations. It outlines two main challenges: 1) creating a detailed food chronicle for individuals to track their eating habits, preferences, allergies, and other personal factors, and 2) developing a world food atlas to understand different dishes and their ingredients, nutrition profiles, recipes, and regional variations. The author believes that solving these challenges through approaches like multimodal food journaling, event stream mining of personal data, and building food knowledge graphs can help people make better diet and lifestyle choices tailored to their individual needs.

1. Eat, Drink, and Enjoy
Ramesh Jain
(jain@ics.uci.edu)
With Several Collaborators

2. Human Evolution: Economist Dec 13, 2003.
Let’s accelerate the evolution.

3. Food is Most IMPORTANT in Life!
1. Social.
2. Religious.
3. Enjoyment.
4. Life.
5. Business
6. Agriculture
But has been mostly ignored by Computing community.
Time to Change.

4. My interest in Food:
I Love Eating!

5. My interest in Food:
Love Multimodal Culinary Event Experience

6. Visual

7. Smell

8. Taste

9. Tactile

10. My interest in Food:
‘merge’ my research passion with real life passion
1. Ruihan Xu, Luis Herranz, Shuqiang Jiang, Shuang Wang, Xinhang Song, Ramesh Jain:
Geolocalized Modeling for Dish Recognition. IEEE Transactions on Multimedia 17(8): 1187-1199 (2015)
2. Nitish Nag, Vaibhav Pandey, Hyungik Oh, Ramesh Jain:
Cybernetic Health. CoRR abs/1705.08514 (2017
3. Nitish Nag, Vaibhav Pandey, Ramesh Jain:
Health Multimedia: Lifestyle Recommendations Based on Diverse Observations. ICMR 2017: 99-106
4. Nitish Nag, Vaibhav Pandey, Abhisaar Sharma, Jonathan Lam, Runyi Wang, Ramesh Jain:
Pocket Dietitian: Automated Healthy Dish Recommendations by Location. ICIAP Workshops 2017: 444-452
5. Nitish Nag, Vaibhav Pandey, Ramesh Jain:
Health Multimedia: Lifestyle Recommendations Based on Diverse Observations. ICMR 2017: 99-106
6. Hyungik Oh, Jonathan Nguyen, Soundarya Soundararajan, Ramesh Jain:
Multimodal Food Journaling. HealthMedia@MM 2018: 39-47
7. N. Nag and R. Jain, “A navigational approach to health: Actionable guidance for improved quality of life,” IEEE Comput.,
vol. 52, no. 4, pp. 12–20. April 2019.
8. Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, Ramesh Jain, ACM Computing Surveys, 52(5):1-36, September,
2019
9. Weiqing Min, Shuqiang Jiang, Ramesh Jain:
Food Recommendation: Framework, Existing Solutions and Challenges. CoRR abs/1905.06269 (2019)
10. Hyungik Oh, Ramesh Jain:
Detecting Events of Daily Living Using Multimodal Data. CoRR abs/1905.09402 (2019)

11. What is the the most effective Medicine?
Zachary Zavislak for TIME

12. That is Old News!
Food Pyramid

14. I am in Nice, France!
• How do I get ‘there’: From my Hotel to Conference center?
• What do I eat and where?
• I have diabetes/allergy.
• I am a vegetarian.
• I am a vegetarian and have diabetes.
• I am a very picky eater.
• Have only 1 hour time.
• I am on tight budget.
Two problems I face!

15. Two Challenges to Solve the Problem!
• Creating a detailed Food Chronicle for a person.
• Creating a detailed World Food Atlas.

16. Food Recommendation
•What: Food Items.
•Who: Person.
•When: Context.
Food
Recommendation
System
Food
Items
Context
Personal
Model
List and
volumes
of Food
Items
and How
to get
them.

17. Challenges for Technology: Dish
Eating Well does not mean eating Tasteless.

18. Millions of Books on Food are Available!
Most non-fiction books in a bookstore are on Food.

19. Problem with Books: Old Technology.
• Books are not Actionable
• Books are not Personal
• Books are not Situational
• Do you still read paper books? Really?
People want Right Information, at Right Time, in
Right (Actionable) form.

20. Challenges for Technology: Dish/Food Items
• Ingredients
• Their source
• Nutritions
• Calories
• Taste
• Season and ingredients
• Cooking style
• Volume
• Individual items
• Cost

21. Society exists only as a mental concept;
in the real world there are only
individuals.
-- Oscar Wilde
Problem with Books: Personalization

22. An Individual is not an average of a Population
Longitudinal data for N-of-1 Studies.
An Average human has:
One breast, and
One testicle?????

23. Personal Factors for Food
• Religion
• Social
• Allergies
• Preferences
• Taste
• Price
• Health condition
• Health goals

24. How can we get this information for a person?
• Food preferences
• Likes and dislikes
• Limitations
• Social/Religious
• Health
• Recent History
• What has been eating
• Current situation
• Company
• Budget
Challenges for Technology: Person

25. Knowing a Person. Can we?
Businesses must understand their Customers.

26. Preferences
Environ-
mental
Allergies
Health
Social
And
Religious
How do I find all this?
Personal model is built using logs data.

27. Personal Chronicle: Personicle
Personicle is a complete chronological record of
person-centered events in life (health, social,
environmental) captured objectively but
augmented using subjective information as
needed.

28. Use some streams and
determine life events.
Add other personal streams as
NEEDED.

29. Most Important Lifestyle Factor:
Diary Food Database Scanning Barcode Taking a Picture
Current Techniques are Unsatisfactory.
Most Important Lifestyle Factor: Food Chronicle

30. Current Popular Approaches for Food Journal in
Multimedia
• FoodLog by Prof. Kiyo Aizawa.
• Many Food Recognition systems: Very active research
• Food Volume: Use one or more pictures
• Nutrition: Ingredient/Recipe recognition
Let’s practice Multimedia.
Use all relevant information
We are becoming camera-hammers.

31. Potential Solutions
• Triggering a food journaling process in a timely, proactive
manner.
• Improving the self-reporting while preserving high accuracy
• Other sources: electronic payments
• Automatic food logging
• New sensors

32. • Eating Moment Recognition
• Location
• Heart rate
• Speech Based Food Journaling
• Enhancement
• Payments
• Recall tips
Eating
One Approach: Food Channnel

33. Event Stream Mining for Rule-Based Actionable Insights
Spicy food and 2 glasses of wine result in sleepless nights.
Warn him when he is at an Sichuan restaurant.
Personicle
Food Stream
t1 t2 t3 t4 t5
Sleep
Representing different events in time band form.

34. Challenges for Technology: Context
• Location/availability
• Local regulations
• Seasonal/Environmental variations
• Personal immediate history
• Social Company
• Cost

35. Most Challenging Context
• Available dishes near a location
• Finding local variations in dishes and their effect on taste and
nutrition
Solution:
• Crawling and analyzing each dish on the menu of every restaurant.
• Understanding recipes used at home to analyze dishes.
• Think dish level rather than restaurant level

36. Should We Build Food Knowledge Graphs?
• Knowledge Graphs are the reason for efficient search – that we all
love.
• Why not creat Food Knowledge Graphs.
• Create World Food Atlas using Food Knowledge Graph.

37. Dish Name
Has
Ingredient
Culinary
Class
Calories
Nutrition
Has
Recipe
Recipe
Source
1000
Fat, …
Peanuts
French
Taste
Vector
Multi-
modal
aspects
Other
Names
Geo-
spatial

38. Understanding Tastes: FlavorNet
Bitter, salty, sour, astringent,
sweet, pungent (eg chili), and
umami.

39. Grand Challenges
•Creating a detailed Food Chronicle for a person.
•Creating a World Food Atlas.
To enjoy food, we must solve these two challenges
using Digital Technology.

40. Thanks.
jain@ics.uci.edu
Data is the Key.