Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Lifelogging - The Early Years

13 views

Published on

Research progress with emerging forms of personal data

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Lifelogging - The Early Years

  1. 1. “Lifelogging - The Early Years” Associate Professor & Asst HoS at the School of Computing, Dublin City University, Ireland Principal co-Investigator at the Insight Centre for Data Analytics (@cathal - cathal@gmail.com) Cathal Gurrin Researcher Personal Data Analytics and e-health Educator Information Retrieval and Data Analytics Lifelogger A decade of visual lifelogging experience University of Kyoto - 15th June 2018 Progress with Emerging Forms of Personal Data
  2. 2. In 2006 I put on a wearable camera. 12 years and billions of data points later, I am still doing it….. WHY?
  3. 3. Guided by the idea that you can never capture what has already passed
  4. 4. 1928
 Bucky- Fuller 1980s
 Steve Mann 2004
 Williams (Sensecam) 2015
 Early Products 2006+
 Memory Studies 2004
 G. Bell
 (MyLifeBits) 1946
 Vannevar Bush
  5. 5. Human Ledger Life Experience The individual will have a human ledger & personal search engine for all life experience… activities, experiences, behaviours, information, biometrics… huge volumes of data captured passively.
  6. 6. OK.. so why should I care? MMM UMAP DATA ICMR SIGIR ECIR MM
  7. 7. What types of data? Driven by low-cost sensors, massive volumes of data are being created and years of data can fit on a $100 hard disk New data types that we are not used to working with…
  8. 8. QS Quantified Self
  9. 9. QS Quantified Self
  10. 10. Rich Sensing
  11. 11. Google Glass, Snapchat Spectacles, ion SnapCam, Narrative Clip, GoPro, Sony Xperia Eye, etc… capturing experiences not just biometrics… Visual Sensing
  12. 12. Writing Content Reading Content Device Interactions Loggerman: Privacy- aware
 HCI and information logging www.loggerman.org Z. Hinbarji, R. Albatal, N. O’Connor and C. Gurrin (2016) LoggerMan, a comprehensive logging and visualisation tool to capture computer usage. In: 22st International Conference on MultiMedia Modelling (MMM 2016), 4-6 Jan, 2016, Miami, FL Information Sensing
  13. 13. Using Sony Digital Paper or a digital pen (EchoPen, LiveScribe), you can create and annotate digital PDFs Writing Content Reading Content Information Sensing
  14. 14. 72 Heart Beats 12 GPS locations 12 Physical Activity Logs 2 images 450 keystrokes 0.07 Glucose readings And so on… Captured at different frequencies, with different error rates, and in a huge number of different modalities… 00:0200:0100:00 00:0500:0400:03 00:0800:0700:06 00:1100:1000:09
  15. 15. So… What do first generation human ledgers look like?
  16. 16. Visual Diary (DCU - 2006) A Doherty, C Ó Conaire, M Blighe,A.F. Smeaton, N.E. O’Connor (2008) Combining image descriptors to effectively retrieve events from visual lifelogs. In: MIR 2008 - ACM International Conference on Multimedia Information Retrieval, 30-31 October,Vancouver, Canada.
  17. 17. Life Abstraction (objects, people, products)
  18. 18. Linking Multiple Data Sources - images to stress levels
  19. 19. A simple search engine enhanced finding important events by 200% and made it 10 x faster for healthy individuals, when compared to an event-based browsing interface… A Doherty, K Pauly-Takacs, N Caprani, C Gurrin, C Moulin, N O'Connor and A.F. Smeaton (2012) Experiences of aiding autobiographical memory Using the SenseCam. Human–Computer Interaction, 27 (1-2). pp. 151-174. ISSN 0737-0024
  20. 20. KidsCam (Univ. Otago & DCU) extracted knowledge for ethnographic study (2015)… Wearable cameras on 200 school children - to understand exposure to fast-food marketing. ISBNPA 2017 Publication of the Year: "Children’s everyday exposure to food marketing: an objective analysis using wearable cameras", L. N. Signal, et al. International Journal of Behavioral Nutrition and Physical Activity. 201714:137
  21. 21. The Unanswered Questions What is a document? What are the use cases? How to index the content? What type of user queries will be make? How to know what is a good approach? …. and a lot more ….
  22. 22. 1 2 3 4 5 6 7 8 Aggregation / ViewSummary Human Activity Moment / short time period Single Reading Event / Scene Logical Sequence of Events Day / Experience Prior Work Data Retrieval Static Event Segmentation Summarisation
  23. 23. Extract Caffe Concepts Features 1000 low-level features/image Extract Microsoft’s Computer Vision API features - color - description - tagging - adult score for each image…… 251 275 310 387 1184 1205 Minute as base Unit of retrieval : select first image of every minute Euclidean distance vector to make event boundaries with large distance between features of adjacent images of every minute. Segmenting images into events Raw Image Stream of One Day A Basic (not-flexible) Approach to Event Segmentation… we need dynamic approaches Approaches for event segmentation of visual lifelog data. R Gupta & C Gurrin. Proc of MMM2018, Bangkok, Thailand, Feb 2018. Automatically Segmenting LifeLog Data into Events. A. R. Doherty & A. F. Smeaton. 2008 Ninth International Workshop on Image Analysis for Multimedia Interactive Services, Klagenfurt, 2008
  24. 24. The Unanswered Questions What is a document? What are the use cases? How to index the content? What type of user queries will be make? How to know what is a good approach? …. and a lot more ….
  25. 25. Quantified Self Personal Insights Data-driven Health Augmented Wellness Behaviour Change Enhanced Security Population-wide Analytics Augmented Community Augmenting Human Memory Nomenclators Augmented Memory Enhanced Productivity & Education Enhanced Interactions Rich Sharing and Reminiscing Augmented Cognition Many (Individual) Use Cases Health & Wellbeing Memory & Cognition
  26. 26. Quantified-Self Analysis Self-discovery Reflect Contextual Reminders Remind Sousveillance. Protection of me and bystanders Protect Find an item from the digital self Validate a memory Contextual support Answer Reminiscence Therapy Social applications Reminisce Digital Agents acting on our behalf, during life and after Represent The most interesting aspect (for me) is the potential for memory support, where the lifelog works in synergy with your own memory. Adapted from Abigail J. Sellen and Steve Whittaker. 2010. Beyond total capture: a constructive critique of lifelogging. Commun. ACM 53, 5 (May 2010), 70-77.
  27. 27. Focus on supporting knowledge acquisition and learning in the early years. 1. Knowledge Support From education to the workplace, providing information and insights to assist productivity and fitness. 2. Productivity Into old age, providing support for cognition and health to maintain independence and activity. 3. Health CHILD ADULT ELDERLY In reality, we can’t yet imagine the use-cases of human ledgers, but it could become a permanent companion assisting you throughout life. Constantly growing in size.
  28. 28. Given an information need, find relevant data to answer the request… Cross-modal Data Retrieval Find the moment in which the lifelogger was doing something of topical interest… Moment Recall / KIS Provide abstraction the data to answer a particular information need (e.g. health topics) Query-focused Abstraction Do all of the above across populations, and not just at the individual level. Cross-individual analysis & retrieval Find an event that fully answers an information need of the individual as a form of recall… Experience / Dynamic Event Retrieval Summarise the data to highlight important events and activities Summarisation Examine the Use-Cases as IR challenges
  29. 29. The Unanswered Questions What is a document? What are the use cases? How to index the content? What type of user queries will be make? How to know what is a good approach? …. and a lot more ….
  30. 30. Applying AI (machine/deep learning) can extract value from multimodal data … to find objects, activities, environments, brands… even open-source detectors work well.
  31. 31. Microsoft Cognitive Services Example Google and Microsoft provide online services
  32. 32. A lot of opportunity to merge different sensing modalities to develop human activity models… Kahneman et al.A survey method for characterizing daily life experience:The day reconstruction method. Science, 306(5702):1776–1780, 2004.
  33. 33. The Unanswered Questions What is a document? What are the use cases? How to index the content? What type of user queries will be make? How to know what is a good approach? …. and a lot more ….
  34. 34. TITLE: A380 DESCRIPTION: Find the moment(s) when I was taking a photo of an A380 NARRATIVE: To be considered relevant, the user must be seen to be taking a photo of an A380 airplane prior to boarding or after disembarking from an aircraft Known-Item Search Solution: Visual, Location, Activity
  35. 35. Real-time Known-Item Search <Topic duration="180"> <TopicID>LSC01</TopicID> <TopicType>development</TopicType> <Descriptions> <Description timestamp="0">In a coffee shop with my colleague in the afternoon called the Helix with at least one person in the background.</ Description> <Description timestamp="30">In a coffee shop with my colleague in the afternoon called the Helix with at least one person in the background and a plastic plant on my right side.</Description> <Description timestamp="60">In a coffee shop with my colleague in the afternoon called the Helix with at least one person in the background and a plastic plant on my right side. There are keys on the table in front of me and you can see the cafe sign on the left side. I walked to the cafe and it took less than two minutes to get there.</Description> <Description timestamp="90">In a coffee shop with my colleague in the afternoon called the Helix with at least one person in the background and a plastic plant on my right side. There are keys on the table in front of me and you can see the cafe sign on the left side. I walked to the cafe and it took less than two minutes to get there. My colleague in the foreground is wearing a white shirt and drinking coffee from a red paper cup.</ Description> <Description timestamp="120">In a coffee shop with my colleague in the afternoon called the Helix with at least one person in the background and a plastic plant on my right side. There are keys on the table in front of me and you can see the cafe sign on the left side. I walked to the cafe and it took less than two minutes to get there. My colleague in the foreground is wearing a white shirt and drinking coffee from a red paper cup. Immediately after having the coffee, I drive to the shop.</Description> <Description timestamp="150">In a coffee shop with my colleague in the afternoon called the Helix with at least one person in the background and a plastic plant on my right side. There are keys on the table in front of me and you can see the cafe sign on the left side. I walked to the cafe and it took less than two minutes to get there. My colleague in the foreground is wearing a white shirt and drinking coffee from a red paper cup. Immediately after having the coffee, I drive to the shop. It is a Monday.</Description> </Topic>
  36. 36. The Unanswered Questions What is a document? What are the use cases? How to index the content? What type of user queries will be make? How to know what is a good approach? …. and a lot more ….
  37. 37. Address the retrieval challenges… NTCIR Lifelog A Lifelog Search Challenge for Interactive Rertrieval LSC Build better annotation tools for visual and multimodal content ImageCLEF Three Main Vehicles
  38. 38. Challenge NTCIR Lifelog ImageClef LSC Ad-hoc Retrieval 12,13,14 Event Segmentation 13 Annotation 14 2017, 2018 Abstraction/ Summarisation 12,13,14 Interactive Retrieval 2018, 2019
  39. 39. NTCIR-Lifelog
  40. 40. NTCIR12-Lifelog Comparative Benchmarking Competition Five different teams with five different approaches…
  41. 41. NTCIR14-Lifelog3 • Rich lifelog data (60 days, 3 people), fully anonymised • Three sub-tasks: • LSAT - Lifelog Semantic Access Task • LIT - Lifelog Insights Task • LADT - Lifelog Activity Detection Task
  42. 42. MINUTE AS THE UNIT OF RETRIEVAL 24x7 heart rate, blood sugar, calorie burn, steps, skin temperature. Daily blood pressure. Weekly cholesterol measurements. Human Biometrics 2,000 images per day from the Autographer wearable camera. Accompanying concept annotations. Audio levels. Manual photos captured. Music listening Wearable Multimedia Physical activities (walking, running, transport, etc..), locations visited, food eaten,. Human Activity Onscreen reading, keystrokes on keyboard, mouse movements, computer activity, web pages viewed. Information Access 0201 0403
  43. 43. Sub-tasks • LSAT - Known-item search… think of it like a Google for the individual. • LIT - Lifelog Insights Subtask… health related insights based on biometrics and images. • LADT - Lifelog Activity Detection Subtask… find the activities from an ontology of life activities.
  44. 44. Lifelog Search Challenge
  45. 45. Lifelog Search Challenge Hopefully an annual task at ICMR conference. Interactive ad-hoc retrieval challenge in front of an audience. Six teams in the first challenge in 2018.
  46. 46. 2 51 6 4 3 DCU Ireland Klagenfurt Austria Charles U. Czech R. VNU Vietnam U Utrecht Netherlands UPC Spain
  47. 47. Winning team was a VR-System with visual and temporal search
  48. 48. ImageCLEF Lifelog
  49. 49. Lifelog Summarization Task Analyse the lifelog data and summarize them according to specific requirements. Lifelog Retrieval Task Analyse the lifelog data and according to several specific queries return the correct answers. ImageCLEF Lifelog Task (2017/2018)
  50. 50. Teams 66 registrations 21 signed copyright forms 19 submitted runs
  51. 51. Easy vs. Hard topics T8. Transporting Query: Summarize the moments when user u2 using public transportation. Description: Photos taken inside a car or a taxi are not relevant. Blurred or out of focus images are not relevant. Images that are covered are not relevant. 0.81
  52. 52. Easy vs. Hard topics T4. Working at home Query: Find the moment(s) in which user u1 was working at home. Description: To be consider to relevant, the user should be using computer for work, reviewing an article or taking some notes at home. Using computer for entertainment is not relevant. 0.08
  53. 53. The Unanswered Questions What is a document? What are the use cases? How to index the content? What type of user queries will be make? How to know what is a good approach? …. and a lot more ….
  54. 54. A variety of data, different timings, different accuracies, needing different tools. Data Processing Scalable & efficient indexing with contextual querying and no defined unit of retrieval. Search & Retrieval Use-cases need pervasive access and contextual querying. Anywhere, Anytime Develop fixed and ubiquitous capture & access methods for all stakeholders. User Experience The ethics of how to use rich personal data & doing so in a privacy-aware manner. Personal Data HUMAN COMPUTER INTERACTION MULTIMEDIA ANALYTICS ETHICS& PRIVACY PERVASIVE COMPUTING INFORMATION RETRIEVAL MEMORY MEMORY ETHINOGRAPHY Multidisciplinary Approach
  55. 55. Privacy Awareness - Automated Negative Face Blurring
 with real-time Policy-driven Access Restrictions (Ye et al. 2014) What about Privacy? The meaning of privacy changes across different jurisdictions Different demographics have different expectations
  56. 56. Storage Feature
 Extraction Professional Market Research Applications Semantic
 Enrichment User Software Analytics Engine Insight & Query Engine The Emergence of a Privacy-aware Ecosystem for Personal Media Storage, Analytics and Monetisation
  57. 57. In Summary: Individuals are beginning to capture our own human ledgers. These will be goal-driven with positive benefits for the individual. We need to understand what can be done to assist both researchers in developing tools and the life loggers to gain value from their content. We need to understand and find ways to… 02 Clean, segment and enrich the data, adding value. Enrich 03 Index the data in extensible and flexible indexing mechanisms Index 01 Multiple heterogenous data sources Gather 04 Support access for a wide variety of use- cases Use
  58. 58. ありがとうございました Associate Professor at the School of Computing, Dublin City University, Ireland Principal Investigator at the Insight Centre for Data Analytics (@cathal - cathal@gmail.com - http://about.me/cgurrin) Cathal Gurrin LifeLogging: Personal Big Data Cathal Gurrin,Alan F. Smeaton,Aiden R. Doherty Published: 16 June 2014 Do a google search and download the book from the DCU website.

×