Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Eat, Drink, and Enjoy!

This was a talk I gave at MADIMA Workshop (ACM Multimedia 2019) in Nice, France, Oct 2019.

  • Login to see the comments

Eat, Drink, and Enjoy!

  1. 1. Eat, Drink, and Enjoy Ramesh Jain ( With Several Collaborators
  2. 2. Human Evolution: Economist Dec 13, 2003. Let’s accelerate the evolution.
  3. 3. Food is Most IMPORTANT in Life! 1. Social. 2. Religious. 3. Enjoyment. 4. Life. 5. Business 6. Agriculture But has been mostly ignored by Computing community. Time to Change.
  4. 4. My interest in Food: I Love Eating!
  5. 5. My interest in Food: Love Multimodal Culinary Event Experience
  6. 6. Visual
  7. 7. Smell
  8. 8. Taste
  9. 9. Tactile
  10. 10. My interest in Food: ‘merge’ my research passion with real life passion 1. Ruihan Xu, Luis Herranz, Shuqiang Jiang, Shuang Wang, Xinhang Song, Ramesh Jain: Geolocalized Modeling for Dish Recognition. IEEE Transactions on Multimedia 17(8): 1187-1199 (2015) 2. Nitish Nag, Vaibhav Pandey, Hyungik Oh, Ramesh Jain: Cybernetic Health. CoRR abs/1705.08514 (2017 3. Nitish Nag, Vaibhav Pandey, Ramesh Jain: Health Multimedia: Lifestyle Recommendations Based on Diverse Observations. ICMR 2017: 99-106 4. Nitish Nag, Vaibhav Pandey, Abhisaar Sharma, Jonathan Lam, Runyi Wang, Ramesh Jain: Pocket Dietitian: Automated Healthy Dish Recommendations by Location. ICIAP Workshops 2017: 444-452 5. Nitish Nag, Vaibhav Pandey, Ramesh Jain: Health Multimedia: Lifestyle Recommendations Based on Diverse Observations. ICMR 2017: 99-106 6. Hyungik Oh, Jonathan Nguyen, Soundarya Soundararajan, Ramesh Jain: Multimodal Food Journaling. HealthMedia@MM 2018: 39-47 7. N. Nag and R. Jain, “A navigational approach to health: Actionable guidance for improved quality of life,” IEEE Comput., vol. 52, no. 4, pp. 12–20. April 2019. 8. Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, Ramesh Jain, ACM Computing Surveys, 52(5):1-36, September, 2019 9. Weiqing Min, Shuqiang Jiang, Ramesh Jain: Food Recommendation: Framework, Existing Solutions and Challenges. CoRR abs/1905.06269 (2019) 10. Hyungik Oh, Ramesh Jain: Detecting Events of Daily Living Using Multimodal Data. CoRR abs/1905.09402 (2019)
  11. 11. What is the the most effective Medicine? Zachary Zavislak for TIME
  12. 12. That is Old News! Food Pyramid
  13. 13. I am in Nice, France! • How do I get ‘there’: From my Hotel to Conference center? • What do I eat and where? • I have diabetes/allergy. • I am a vegetarian. • I am a vegetarian and have diabetes. • I am a very picky eater. • Have only 1 hour time. • I am on tight budget. Two problems I face!
  14. 14. Two Challenges to Solve the Problem! • Creating a detailed Food Chronicle for a person. • Creating a detailed World Food Atlas.
  15. 15. Food Recommendation •What: Food Items. •Who: Person. •When: Context. Food Recommendation System Food Items Context Personal Model List and volumes of Food Items and How to get them.
  16. 16. Challenges for Technology: Dish Eating Well does not mean eating Tasteless.
  17. 17. Millions of Books on Food are Available! Most non-fiction books in a bookstore are on Food.
  18. 18. Problem with Books: Old Technology. • Books are not Actionable • Books are not Personal • Books are not Situational • Do you still read paper books? Really? People want Right Information, at Right Time, in Right (Actionable) form.
  19. 19. Challenges for Technology: Dish/Food Items • Ingredients • Their source • Nutritions • Calories • Taste • Season and ingredients • Cooking style • Volume • Individual items • Cost
  20. 20. Society exists only as a mental concept; in the real world there are only individuals. -- Oscar Wilde Problem with Books: Personalization
  21. 21. An Individual is not an average of a Population Longitudinal data for N-of-1 Studies. An Average human has: One breast, and One testicle?????
  22. 22. Personal Factors for Food • Religion • Social • Allergies • Preferences • Taste • Price • Health condition • Health goals
  23. 23. How can we get this information for a person? • Food preferences • Likes and dislikes • Limitations • Social/Religious • Health • Recent History • What has been eating • Current situation • Company • Budget Challenges for Technology: Person
  24. 24. Knowing a Person. Can we? Businesses must understand their Customers.
  25. 25. Preferences Environ- mental Allergies Health Social And Religious How do I find all this? Personal model is built using logs data.
  26. 26. Personal Chronicle: Personicle Personicle is a complete chronological record of person-centered events in life (health, social, environmental) captured objectively but augmented using subjective information as needed.
  27. 27. Use some streams and determine life events. Add other personal streams as NEEDED.
  28. 28. Most Important Lifestyle Factor: Diary Food Database Scanning Barcode Taking a Picture Current Techniques are Unsatisfactory. Most Important Lifestyle Factor: Food Chronicle
  29. 29. Current Popular Approaches for Food Journal in Multimedia • FoodLog by Prof. Kiyo Aizawa. • Many Food Recognition systems: Very active research • Food Volume: Use one or more pictures • Nutrition: Ingredient/Recipe recognition Let’s practice Multimedia. Use all relevant information We are becoming camera-hammers.
  30. 30. Potential Solutions • Triggering a food journaling process in a timely, proactive manner. • Improving the self-reporting while preserving high accuracy • Other sources: electronic payments • Automatic food logging • New sensors
  31. 31. • Eating Moment Recognition • Location • Heart rate • Speech Based Food Journaling • Enhancement • Payments • Recall tips Eating One Approach: Food Channnel
  32. 32. Event Stream Mining for Rule-Based Actionable Insights Spicy food and 2 glasses of wine result in sleepless nights. Warn him when he is at an Sichuan restaurant. Personicle Food Stream t1 t2 t3 t4 t5 Sleep Representing different events in time band form.
  33. 33. Challenges for Technology: Context • Location/availability • Local regulations • Seasonal/Environmental variations • Personal immediate history • Social Company • Cost
  34. 34. Most Challenging Context • Available dishes near a location • Finding local variations in dishes and their effect on taste and nutrition Solution: • Crawling and analyzing each dish on the menu of every restaurant. • Understanding recipes used at home to analyze dishes. • Think dish level rather than restaurant level
  35. 35. Should We Build Food Knowledge Graphs? • Knowledge Graphs are the reason for efficient search – that we all love. • Why not creat Food Knowledge Graphs. • Create World Food Atlas using Food Knowledge Graph.
  36. 36. Dish Name Has Ingredient Culinary Class Calories Nutrition Has Recipe Recipe Source 1000 Fat, … Peanuts French Taste Vector Multi- modal aspects Other Names Geo- spatial
  37. 37. Understanding Tastes: FlavorNet Bitter, salty, sour, astringent, sweet, pungent (eg chili), and umami.
  38. 38. Grand Challenges •Creating a detailed Food Chronicle for a person. •Creating a World Food Atlas. To enjoy food, we must solve these two challenges using Digital Technology.
  39. 39. Thanks. Data is the Key.