I am in Nice, France!
• How do I get ‘there’: From my Hotel to Conference center?
• What do I eat and where?
• I have diabetes/allergy.
• I am a vegetarian.
• I am a vegetarian and have diabetes.
• I am a very picky eater.
• Have only 1 hour time.
• I am on tight budget.
Two problems I face!
Two Challenges to Solve the Problem!
• Creating a detailed Food Chronicle for a person.
• Creating a detailed World Food Atlas.
•What: Food Items.
Millions of Books on Food are Available!
Most non-fiction books in a bookstore are on Food.
Problem with Books: Old Technology.
• Books are not Actionable
• Books are not Personal
• Books are not Situational
• Do you still read paper books? Really?
People want Right Information, at Right Time, in
Right (Actionable) form.
Challenges for Technology: Dish/Food Items
• Their source
• Season and ingredients
• Cooking style
• Individual items
Society exists only as a mental concept;
in the real world there are only
-- Oscar Wilde
Problem with Books: Personalization
An Individual is not an average of a Population
Longitudinal data for N-of-1 Studies.
An Average human has:
One breast, and
Personal Factors for Food
• Health condition
• Health goals
How can we get this information for a person?
• Food preferences
• Likes and dislikes
• Recent History
• What has been eating
• Current situation
Challenges for Technology: Person
Personal Chronicle: Personicle
Personicle is a complete chronological record of
person-centered events in life (health, social,
environmental) captured objectively but
augmented using subjective information as
Most Important Lifestyle Factor:
Diary Food Database Scanning Barcode Taking a Picture
Current Techniques are Unsatisfactory.
Most Important Lifestyle Factor: Food Chronicle
Current Popular Approaches for Food Journal in
• FoodLog by Prof. Kiyo Aizawa.
• Many Food Recognition systems: Very active research
• Food Volume: Use one or more pictures
• Nutrition: Ingredient/Recipe recognition
Let’s practice Multimedia.
Use all relevant information
We are becoming camera-hammers.
• Triggering a food journaling process in a timely, proactive
• Improving the self-reporting while preserving high accuracy
• Other sources: electronic payments
• Automatic food logging
• New sensors
• Eating Moment Recognition
• Heart rate
• Speech Based Food Journaling
• Recall tips
One Approach: Food Channnel
Event Stream Mining for Rule-Based Actionable Insights
Spicy food and 2 glasses of wine result in sleepless nights.
Warn him when he is at an Sichuan restaurant.
t1 t2 t3 t4 t5
Representing different events in time band form.
Challenges for Technology: Context
• Local regulations
• Seasonal/Environmental variations
• Personal immediate history
• Social Company
Most Challenging Context
• Available dishes near a location
• Finding local variations in dishes and their effect on taste and
• Crawling and analyzing each dish on the menu of every restaurant.
• Understanding recipes used at home to analyze dishes.
• Think dish level rather than restaurant level
Should We Build Food Knowledge Graphs?
• Knowledge Graphs are the reason for efficient search – that we all
• Why not creat Food Knowledge Graphs.
• Create World Food Atlas using Food Knowledge Graph.