SlideShare a Scribd company logo
1 of 77
When My Data Actually Becomes My
Data
Alan F. Smeaton
Dublin City University
@asmeaton
Who’s Alan Smeaton ?
- DCU Professor for +20 years
- Founding Director of Insight
- Decorated, highly-cited, worked
on 00’s research projects, loads of
PhD graduates
- Background in multimedia
analysis, indexing and search
- Known for his collaborations
• We have an adversarial relationship with our personal data
• Usually gathered by third parties and in theory we own our data, but
we don’t manage, get value from, or use it ourselves
• We read about abuses of it
• We can also gather data ourselves, about ourselves, for standalone
niche applications (sleep, steps, energy, screen usage, travel, etc.), or
we can do extreme forms of this called lifelogging
• Lets have a look at that niche application area, and some of the
extreme lifelogging
3
Digital Footprints
• Digital Footprints used to be the electronic evidence of a
computer user's activity (online, local, etc.), used for debugging
• Early search engines logged queries/clicks
• Clickthroughs were mined, turned into AdWords, pushed the
boundaries of ML, created Google
• Now a digital footprint is the electronic evidence of your
existence, since everything is now digital and so much is logged
by third parties anyway
Digital Footprints
• We’re aware we consciously and unconsciously leave digital
footprints – conscious ones from
• Web searches
• Website visits and cookies
• Internet connections
• Purchases
• Social media check-ins, posts and photos
• These are what we expect Google et al. to know about us …
like all marketing, they use this to segment their market (us) so
we get adverts (like me viewing YouTube videos with fingers
crossed)
• With rich data they segment their market into N=1 by turning
footprints into personality profiles
Digital Footprints
• So that’s our digital footprints
• Some we know about, its obvious, and we accept,
we even like it … the Faustian pact
• Some we don’t realise, we’re surprised at, but we’re
OK with
• Some is a bit creepy, maybe crossing a line, intrusive
• Our awareness varies hugely – vast majority don’t
realise, those that do know, don’t actually know it all
• Most of the time, 3rd parties gather data about us but
occasionally we gather data about ourselves, we gather it
• That’s called lifelogging
7
Depending on what you want
to measure, there is likely to
be a device (or an app)
Digitise as much as you
can of life experience…
for many reasons
(memory, health, etc.).
Lifelogging
Sense and analyse
factors of interest
through numbers to gain
knowledge
Using
knowledge for
self-
improvement
through
experimentatio
n
Quantified
Self
Biohackin
g
1928
Bucky Fuller
Archive
1980s
Steve Mann
2004
Williams
(Sensecam)
2010
Quantified-
Self
2006+
Memory
Studies
2004
G. Bell
(MyLifeBits)
1946
Vannevar
Bush
Richard Buckminster Fuller
• Interested in “a very accurate record of a human
being" … so he made himself his own case study ..
put everything in and created a very rigorous
record … the Dymaxion Chronofile…
• He documented his life, philosophy and ideas
scrupulously by a diary every 15 minutes, now on
display at the Stanford Library.
11
Richard Buckminster
"Bucky" Fuller was an
American architect,
systems theorist,
author, designer,
inventor and futurist.
Fuller was the second
World President of
Mensa from 1974 to
1983
With more than 140,000 papers and 1,700 hours of audio and video, all stretching to more than
1,400 linear feet of material, Fuller’s life might be the most documented life of all time. From 1917
to his death in 1983 he collected all documentation including (mail, newspaper clippings,
drawings, blueprints, models, and even bills.
The Dymaxion Chronofile has been at the Stanford University Libraries Department of Special
Collections since 1999. There you can pick any day of these years of his life and find out exactly
what he was doing nearly to the hour by flipping through a scrapbook.
Vannevar Bush (External Memory)
13
Vannevar Bush was an
American engineer,
inventor and science
administrator, who
during World War II
headed the U.S.
wartime military R&D
including initiation of the
Manhattan Project. "As
We May Think" has
turned out to be a
visionary and influential
essay.
https://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/
https://mediartinnovation.com/2014/06/06/vannevar-bush-memex-1945/
Vannevar Bush (External Memory)
Steve Mann’s Wearcam
• Steve Mann (University of Toronto) built a wearable
camera called Wearcam (wearcam.org). Steve would
wear the camera and it uploaded images to the WWW
for others to see.
• Mann has been referred to as the "father of wearable
computing", having created the first general-purpose
wearable computer. Mann has also been described as
"the world's first cyborg”.
15
Steven Mann is a
Canadian researcher
and inventor best
known for his work on
augmented reality,
computational
photography,
particularly wearable
computing and high
dynamic range
imaging.
2012
Steve Mann’s Wearcam
Gordon Bell
An early employee of Digital Equipment
Corporation (DEC), Bell designed several
PDP machines and later became Vice
President of Engineering, for VAX
computers
He is the experiment subject for the
MyLifeBits project, an experiment in life-
logging and an attempt to fulfill Vannevar
Bush's Memex.
17
Gordon Bell is an
American electrical
engineer, pioneer and
investor. He is the
founder of the
MyLifeBits project, an
experiment in life-
logging based on
Vannevar Bush's
Memex.
Lifelog
Life
Experience
The individual will have a lifelog / human
ledger for many aspects of life
experience… activities, experiences,
behaviours, information, biometrics… huge
volumes of data captured passively.
Gordon Bell – the lifelog philosophy
Cathal Gurrin
In 2006 he put on a wearable camera, he’s still wears one, every day all
day, 12 years and billions of data points later
His interest is in multimodal multimedia analysis, indexing and retrieval
Unstructured Lifelog Data
20
Wearable cameras to
capture our activities
21
A complete trace of the individual
Know their activities and interests
Know their habits
Know their interests and experiences
And know what they consume
Devices
Apple Watch & HealthKit
Wahoo Chest Strap
Fitbit Activity Tracker / Scales
Nokia Activity Tracker / Scales / BP
…
Aggregation Software
HealthKit
Microsoft Health
Gyrosco.pe
We’ll come
back to
this
Issues with Wearable Camera Images
Huge levels of redundancy
28
Issues with Wearable Camera Images
Vary in quality from unusable to photo-album
29
Issues with Wearable Camera Images
Don’t typically have
a salient object
Capture the hands
of the wearer
31
CATHAL’S LIFELOG
Autographer Panasonic 4K Google Glass
Moves App Withings BASIS Watch Reporter
RescueTime LoggerMan Camlapse
MyTracks OpenPaths
CameraPhoneInstagram
Media Lifelogging
Biometrics/
Activity
Lifelog
Information Access
SMS Backup
Youtube Log
VoiceRecorder
WebServices
Swarm
Location
Digital Paper PDF Anno. Web Pages
Consumption
e-book/mag
Last.fm
NarrativeClip
3
2
Volume Growth over a Decade
Which allows for the development of various types of interface.
3
4
Events with colour coded minutes… showing the dominant colours
3
5
Minute-by-
minute
segmentation
and
summarisation of
life activities
Segmentation of raw data into
units such as events or moments.
These can be enriched
automatically with metadata,
increasing their value.
Events are analogous to our episodic
memory and can be segmented based
on many forms of data.
Quantified Self
Personal Insights
Data-driven Health
Augmented
Wellness
Behaviour Change
Enhanced Security
Population-wide
Analytics
Augmented
Community
Augmenting Human
Memory
Nomenclators
Augmented
Memory
Enhanced Productivity
& Education
Enhanced Interactions
Rich Sharing and
Reminiscing
Augmented
Cognition
Some (Individual) Use Cases for Lifelogging
Health & Wellbeing Memory & Cognition
Quantified-Self
Analysis
Self-discovery
Reflect
Contextual
Reminders
Remind
Sousveillance.
Protection of me
and bystanders
Protect
Find an item from
the digital self
Validate a memory
Contextual support
Answer
Reminiscence
Therapy
Social applications
Reminisce
Digital Agents
acting on our
behalf, during life
and after
Represent
The most interesting aspect is the potential for memory support,
where the lifelog works in synergy with your own memory.
Adapted from Abigail J. Sellen and Steve Whittaker. 2010. Beyond total capture: a constructive critique
of lifelogging. Communications of the ACM. 53, 5 (May 2010), 70-77.
• Back to OTS or regular quantified self rather than
extreme lifelogging
• Lets have a deeper look at one form of lifelogging
… sleep
40
• Sleep is “active”, our brains do not shut down and
are almost as active, cataloging memories
• Sleep 5 stages – wake, relaxed wakefulness, light
sleep, deep sleep and REM sleep
• Starts from N1, goes through N2 to N3 (deep) and
then back up towards REM sleep, there is an
ordering
• If you sleep for 8 hours, you have 5 full cycles
Sleep Explained
• Each phase has characteristics
– REM – body paralysed, HR, RR increased,
body temperature drops, vivid dreams, brain
active, towards latter end of the night, memory
consolidation
– N1 – conscious of surroundings, hypnic jerks
– N2 – brief arousals, decreased HR, RR
– N3 or deep sleep – slowest HR, RR, difficult to
wake and then groggy
Sleep Explained
• Insufficient sleep makes you more
stupid, fatter, unhappier, poorer, sicker,
worse at sex, more grumpy, more likely
to get cancer, Alzheimer’s and to die in
a car crash !
• Recent years have focused on this,
we’re more aware because we
ourselves can now measure it
• [ Orthosomnia — a preoccupation with
perfecting one’s sleep data ]
Why is sleep important ?
• Sleep labs record EEG,
body and eye movement,
HR, HRV, RR, Oxygen
saturation, etc.
• They pool all these into a
polysomnograph for an
holistic overview, which
looks like …
Measuring Sleep (Properly)
1. Phone apps
2. Wrist-worn accelerometer devices
3. Movement radar or sonar
4. The Ōuru ring
Each of these uses a subset of sensors
Sensing Sleep – Our Options
Presented as a Hypnogram
• How ? They record ..
– Movement (in the bed, under pillow)
– Microphone (listening to your breathing)
– Sonar – inaudible frequency emitted and listened to, just like bats !
• There are many available, some freebies, some
paid … SleepScore, Sleep Cycle, Pillow, etc.
1. Smartphone Apps
• FitBit, Jawbone, Withings,
LARK, etc.
• These “just” do movement
but directly, so more
accurately than phone apps
2. Wrist-worn Accelerometers
• BiancaMed -> ResMed -> S+ was first to market
• Very low levels of transmitted radio-frequency
power acting as sonar
• Contactless, measures
motion, plus room
temperature, brightness
and noise level (from
phone, yes, phone is
listening as you sleep)
3. Sonar / Radar
4. The Ōura Ring
• Includes accelerometer,
gyroscope, temperature, and
heart rate
• HR sampled using IR spectrum
at 250 Hz so able to do much
more than wrist-worn
• Contactless battery charge lasts
7 days, data capacity 6 weeks
• Low power Bluetooth download
to phone
• Consumer grade wearables are not sleep labs,
they use a proxy for an orchestra of sensors !
• Each individual sensor will have errors
(movement, sweat, etc.)
• The algorithms to compute “sleep efficiency” are
opaque and proprietary
• I compared S+ with Ōura for me over 8 weeks and
…
Accuracy of sleep tracking ?
• 56 days, but S+ not continuous (travel, S+ is not
travel-friendly) – I missed 1, 1 and 7 days
• Correlation is (only) 0.147 (rises to 0.16 when
blanks eliminated)
• And because ranges may not be normalized, the
visualization reveals …
S+ and Ōura
S+ and Ōura - 56 days continuous – 0.147
So What ?
• Its all a bit … unsatisfactory
• I’m feeling a bit … let down … it shouldn’t be like this,
there’s something not right
• I’m actually feeling … trapped .. its my data, about
me, but I can’t fix this, for me, I can’t leverage benefit,
• Not everyone is like me though, and would want to do
this, but maybe you’d want somebody to do this for
you, like a wealth manager or stock broker
• There are some benefits at population level
56
Almost all vendors who allow us gather individual
data, get value from pooling anonymised data
• Fitbit – 150B hours of HR data, statistics from
millions of sleep nights
• 23a DNA testing – diseases, long-lost relatives
• Strava – global heatmap of runs/cycles
• Jaw – pinpointing earthquake epicentres !
Population-level Analytics from Pooled
Individual Data
Resting Heart Rate is an excellent indicator of overall health.
https://finance.yahoo.com/news/exclusive-fitbits-150-billion-hours-heart-data-reveals-secrets-human-health-133124215.html
Strava Global Heatmap
2014 – Northern Calif. 6.0M Earthquake
% people who woke up correlates with
distance from epicentre
Back to my problem of …
• Unsatisfactory … let down … trapped
63
• Open up about their algorithms and allow a
“normalisation” of sleep metrics for interoperability
and device/vendor independence
• Include “errorbars” in their outputs
• Allow exporting of our data, our original raw data,
with those error margins
Sleep Sensing Vendors ought to …
• Imagine a world where our data is managed for us by trusted
third parties analogous to how banks manage wealth
• They may pool it, like banks lend money
• They may share mine with organisations like social media
companies, educational institutions, entertainment companies,
etc. but only if I say so, and I might get paid, or I might get better
services
• I would be delighted rather than afraid, to gather data and have
data gathered about me for my benefit
• Most people would not micromanage data, most people don’t
micromanage wealth
• We have seen the data points that can be gathered
Utopia ? Naivete ?
• What’s the appetite for this ?
• Is there even a demand ?
• Are there green shoots against surveillance
capitalism ?
• DuckDuckGo, StartPage, SearX.me,
DisconnectSearch, MetaGer, Quant, all trace-free
search engines
Trace-free services
Devices
Apple Watch & HealthKit
Wahoo Chest Strap
Fitbit Activity Tracker / Scales
Nokia Activity Tracker / Scales / BP
…
Aggregation Software
HealthKit
Microsoft Health
Gyrosco.pe
Remember
this ?
6
9
• Consolidates health data from iPhone,
Apple Watch, and third-party apps
• Activity, Sleep, Mindfulness, and
Nutrition
• “You are in charge of your data”
• “The Health app lets you keep all your
health and fitness information under
your control and in one place on your
device. You decide which information is
placed in Health and which apps can
access your data through the
Health app”
dacadoo
dacadoo
dacadoo
dacadoo
dacadoo
dacadoo
• Revenue stream is
corporate
• Employer signs up
as a service to
employees,
employees use it,
employer gets
healthier (more
productive) staff
• I don’t see a spanner breaking up Surveillance
Capitalism, too traumatic
• I see a slow creep towards
– Surveillance Socialism (don’t like that)
– Surveillance Democracy (don’t like that either)
– Sousveillance
• Not happening overnight, generational
When My Data Actually Becomes My Data

More Related Content

Similar to When My Data Actually Becomes My Data

Lifelogging, egocentric vision and health: how a small wearable camera can he...
Lifelogging, egocentric vision and health: how a small wearable camera can he...Lifelogging, egocentric vision and health: how a small wearable camera can he...
Lifelogging, egocentric vision and health: how a small wearable camera can he...
Petia Radeva
 

Similar to When My Data Actually Becomes My Data (20)

Quantified Self: The what, the why and the brave new future
Quantified Self: The what, the why and the brave new futureQuantified Self: The what, the why and the brave new future
Quantified Self: The what, the why and the brave new future
 
Digital immortality Roadmap
Digital immortality RoadmapDigital immortality Roadmap
Digital immortality Roadmap
 
We will all have in Human Ledgers in the Future
We will all have in Human Ledgers in the FutureWe will all have in Human Ledgers in the Future
We will all have in Human Ledgers in the Future
 
Getting Started
Getting StartedGetting Started
Getting Started
 
Sensing devices by Masha Ioveva
Sensing devices by Masha IovevaSensing devices by Masha Ioveva
Sensing devices by Masha Ioveva
 
Making data
Making dataMaking data
Making data
 
Chapter3thornock2015
Chapter3thornock2015Chapter3thornock2015
Chapter3thornock2015
 
Machine Learning for Non-technical People
Machine Learning for Non-technical PeopleMachine Learning for Non-technical People
Machine Learning for Non-technical People
 
Acm achievement award acceptance 20101026
Acm achievement award acceptance 20101026Acm achievement award acceptance 20101026
Acm achievement award acceptance 20101026
 
Play, Pause, Rewind - The Era of Archived Lifetimes
Play, Pause, Rewind - The Era of Archived LifetimesPlay, Pause, Rewind - The Era of Archived Lifetimes
Play, Pause, Rewind - The Era of Archived Lifetimes
 
Lifelogging, egocentric vision and health: how a small wearable camera can he...
Lifelogging, egocentric vision and health: how a small wearable camera can he...Lifelogging, egocentric vision and health: how a small wearable camera can he...
Lifelogging, egocentric vision and health: how a small wearable camera can he...
 
Exploring the Future: Quantified Self and Learning
Exploring the Future: Quantified Self and LearningExploring the Future: Quantified Self and Learning
Exploring the Future: Quantified Self and Learning
 
Quantified-Self and Lifelogging Meets Internet of Things (IOT)
Quantified-Self and Lifelogging Meets Internet of Things (IOT)Quantified-Self and Lifelogging Meets Internet of Things (IOT)
Quantified-Self and Lifelogging Meets Internet of Things (IOT)
 
The Quantified Self and What it Means for Learning
The Quantified Self and What it Means for LearningThe Quantified Self and What it Means for Learning
The Quantified Self and What it Means for Learning
 
Big Data Better Life
Big Data Better LifeBig Data Better Life
Big Data Better Life
 
Quantified Self Revolution December 2013
Quantified Self Revolution December 2013Quantified Self Revolution December 2013
Quantified Self Revolution December 2013
 
Categories for data collection methods
Categories for data collection methodsCategories for data collection methods
Categories for data collection methods
 
Applications of Behavioural Economics to consumer insight
Applications of Behavioural Economics to consumer insightApplications of Behavioural Economics to consumer insight
Applications of Behavioural Economics to consumer insight
 
Forever Cyborgs – a long view on physical-digital interaction
Forever Cyborgs – a long view on physical-digital interactionForever Cyborgs – a long view on physical-digital interaction
Forever Cyborgs – a long view on physical-digital interaction
 
Effective Presenting with ‘Think, Feel, Do!’
Effective Presenting with ‘Think, Feel, Do!’Effective Presenting with ‘Think, Feel, Do!’
Effective Presenting with ‘Think, Feel, Do!’
 

Recently uploaded

Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
FIDO Alliance
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
Wonjun Hwang
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
FIDO Alliance
 

Recently uploaded (20)

Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
 
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxCyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 

When My Data Actually Becomes My Data

  • 1. When My Data Actually Becomes My Data Alan F. Smeaton Dublin City University @asmeaton
  • 2. Who’s Alan Smeaton ? - DCU Professor for +20 years - Founding Director of Insight - Decorated, highly-cited, worked on 00’s research projects, loads of PhD graduates - Background in multimedia analysis, indexing and search - Known for his collaborations
  • 3. • We have an adversarial relationship with our personal data • Usually gathered by third parties and in theory we own our data, but we don’t manage, get value from, or use it ourselves • We read about abuses of it • We can also gather data ourselves, about ourselves, for standalone niche applications (sleep, steps, energy, screen usage, travel, etc.), or we can do extreme forms of this called lifelogging • Lets have a look at that niche application area, and some of the extreme lifelogging 3
  • 4. Digital Footprints • Digital Footprints used to be the electronic evidence of a computer user's activity (online, local, etc.), used for debugging • Early search engines logged queries/clicks • Clickthroughs were mined, turned into AdWords, pushed the boundaries of ML, created Google • Now a digital footprint is the electronic evidence of your existence, since everything is now digital and so much is logged by third parties anyway
  • 5. Digital Footprints • We’re aware we consciously and unconsciously leave digital footprints – conscious ones from • Web searches • Website visits and cookies • Internet connections • Purchases • Social media check-ins, posts and photos • These are what we expect Google et al. to know about us … like all marketing, they use this to segment their market (us) so we get adverts (like me viewing YouTube videos with fingers crossed) • With rich data they segment their market into N=1 by turning footprints into personality profiles
  • 6. Digital Footprints • So that’s our digital footprints • Some we know about, its obvious, and we accept, we even like it … the Faustian pact • Some we don’t realise, we’re surprised at, but we’re OK with • Some is a bit creepy, maybe crossing a line, intrusive • Our awareness varies hugely – vast majority don’t realise, those that do know, don’t actually know it all
  • 7. • Most of the time, 3rd parties gather data about us but occasionally we gather data about ourselves, we gather it • That’s called lifelogging 7
  • 8. Depending on what you want to measure, there is likely to be a device (or an app)
  • 9. Digitise as much as you can of life experience… for many reasons (memory, health, etc.). Lifelogging Sense and analyse factors of interest through numbers to gain knowledge Using knowledge for self- improvement through experimentatio n Quantified Self Biohackin g
  • 11. Richard Buckminster Fuller • Interested in “a very accurate record of a human being" … so he made himself his own case study .. put everything in and created a very rigorous record … the Dymaxion Chronofile… • He documented his life, philosophy and ideas scrupulously by a diary every 15 minutes, now on display at the Stanford Library. 11 Richard Buckminster "Bucky" Fuller was an American architect, systems theorist, author, designer, inventor and futurist. Fuller was the second World President of Mensa from 1974 to 1983
  • 12. With more than 140,000 papers and 1,700 hours of audio and video, all stretching to more than 1,400 linear feet of material, Fuller’s life might be the most documented life of all time. From 1917 to his death in 1983 he collected all documentation including (mail, newspaper clippings, drawings, blueprints, models, and even bills. The Dymaxion Chronofile has been at the Stanford University Libraries Department of Special Collections since 1999. There you can pick any day of these years of his life and find out exactly what he was doing nearly to the hour by flipping through a scrapbook.
  • 13. Vannevar Bush (External Memory) 13 Vannevar Bush was an American engineer, inventor and science administrator, who during World War II headed the U.S. wartime military R&D including initiation of the Manhattan Project. "As We May Think" has turned out to be a visionary and influential essay. https://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/
  • 15. Steve Mann’s Wearcam • Steve Mann (University of Toronto) built a wearable camera called Wearcam (wearcam.org). Steve would wear the camera and it uploaded images to the WWW for others to see. • Mann has been referred to as the "father of wearable computing", having created the first general-purpose wearable computer. Mann has also been described as "the world's first cyborg”. 15 Steven Mann is a Canadian researcher and inventor best known for his work on augmented reality, computational photography, particularly wearable computing and high dynamic range imaging.
  • 17. Gordon Bell An early employee of Digital Equipment Corporation (DEC), Bell designed several PDP machines and later became Vice President of Engineering, for VAX computers He is the experiment subject for the MyLifeBits project, an experiment in life- logging and an attempt to fulfill Vannevar Bush's Memex. 17 Gordon Bell is an American electrical engineer, pioneer and investor. He is the founder of the MyLifeBits project, an experiment in life- logging based on Vannevar Bush's Memex.
  • 18. Lifelog Life Experience The individual will have a lifelog / human ledger for many aspects of life experience… activities, experiences, behaviours, information, biometrics… huge volumes of data captured passively. Gordon Bell – the lifelog philosophy
  • 19. Cathal Gurrin In 2006 he put on a wearable camera, he’s still wears one, every day all day, 12 years and billions of data points later His interest is in multimodal multimedia analysis, indexing and retrieval
  • 21. Wearable cameras to capture our activities 21
  • 22. A complete trace of the individual
  • 23. Know their activities and interests
  • 25. Know their interests and experiences
  • 26. And know what they consume
  • 27. Devices Apple Watch & HealthKit Wahoo Chest Strap Fitbit Activity Tracker / Scales Nokia Activity Tracker / Scales / BP … Aggregation Software HealthKit Microsoft Health Gyrosco.pe We’ll come back to this
  • 28. Issues with Wearable Camera Images Huge levels of redundancy 28
  • 29. Issues with Wearable Camera Images Vary in quality from unusable to photo-album 29
  • 30.
  • 31. Issues with Wearable Camera Images Don’t typically have a salient object Capture the hands of the wearer 31
  • 32. CATHAL’S LIFELOG Autographer Panasonic 4K Google Glass Moves App Withings BASIS Watch Reporter RescueTime LoggerMan Camlapse MyTracks OpenPaths CameraPhoneInstagram Media Lifelogging Biometrics/ Activity Lifelog Information Access SMS Backup Youtube Log VoiceRecorder WebServices Swarm Location Digital Paper PDF Anno. Web Pages Consumption e-book/mag Last.fm NarrativeClip 3 2
  • 33. Volume Growth over a Decade
  • 34. Which allows for the development of various types of interface. 3 4
  • 35. Events with colour coded minutes… showing the dominant colours 3 5
  • 37. Segmentation of raw data into units such as events or moments. These can be enriched automatically with metadata, increasing their value. Events are analogous to our episodic memory and can be segmented based on many forms of data.
  • 38. Quantified Self Personal Insights Data-driven Health Augmented Wellness Behaviour Change Enhanced Security Population-wide Analytics Augmented Community Augmenting Human Memory Nomenclators Augmented Memory Enhanced Productivity & Education Enhanced Interactions Rich Sharing and Reminiscing Augmented Cognition Some (Individual) Use Cases for Lifelogging Health & Wellbeing Memory & Cognition
  • 39. Quantified-Self Analysis Self-discovery Reflect Contextual Reminders Remind Sousveillance. Protection of me and bystanders Protect Find an item from the digital self Validate a memory Contextual support Answer Reminiscence Therapy Social applications Reminisce Digital Agents acting on our behalf, during life and after Represent The most interesting aspect is the potential for memory support, where the lifelog works in synergy with your own memory. Adapted from Abigail J. Sellen and Steve Whittaker. 2010. Beyond total capture: a constructive critique of lifelogging. Communications of the ACM. 53, 5 (May 2010), 70-77.
  • 40. • Back to OTS or regular quantified self rather than extreme lifelogging • Lets have a deeper look at one form of lifelogging … sleep 40
  • 41.
  • 42. • Sleep is “active”, our brains do not shut down and are almost as active, cataloging memories • Sleep 5 stages – wake, relaxed wakefulness, light sleep, deep sleep and REM sleep • Starts from N1, goes through N2 to N3 (deep) and then back up towards REM sleep, there is an ordering • If you sleep for 8 hours, you have 5 full cycles Sleep Explained
  • 43. • Each phase has characteristics – REM – body paralysed, HR, RR increased, body temperature drops, vivid dreams, brain active, towards latter end of the night, memory consolidation – N1 – conscious of surroundings, hypnic jerks – N2 – brief arousals, decreased HR, RR – N3 or deep sleep – slowest HR, RR, difficult to wake and then groggy Sleep Explained
  • 44. • Insufficient sleep makes you more stupid, fatter, unhappier, poorer, sicker, worse at sex, more grumpy, more likely to get cancer, Alzheimer’s and to die in a car crash ! • Recent years have focused on this, we’re more aware because we ourselves can now measure it • [ Orthosomnia — a preoccupation with perfecting one’s sleep data ] Why is sleep important ?
  • 45. • Sleep labs record EEG, body and eye movement, HR, HRV, RR, Oxygen saturation, etc. • They pool all these into a polysomnograph for an holistic overview, which looks like … Measuring Sleep (Properly)
  • 46.
  • 47. 1. Phone apps 2. Wrist-worn accelerometer devices 3. Movement radar or sonar 4. The Ōuru ring Each of these uses a subset of sensors Sensing Sleep – Our Options
  • 48. Presented as a Hypnogram
  • 49. • How ? They record .. – Movement (in the bed, under pillow) – Microphone (listening to your breathing) – Sonar – inaudible frequency emitted and listened to, just like bats ! • There are many available, some freebies, some paid … SleepScore, Sleep Cycle, Pillow, etc. 1. Smartphone Apps
  • 50. • FitBit, Jawbone, Withings, LARK, etc. • These “just” do movement but directly, so more accurately than phone apps 2. Wrist-worn Accelerometers
  • 51. • BiancaMed -> ResMed -> S+ was first to market • Very low levels of transmitted radio-frequency power acting as sonar • Contactless, measures motion, plus room temperature, brightness and noise level (from phone, yes, phone is listening as you sleep) 3. Sonar / Radar
  • 52. 4. The Ōura Ring • Includes accelerometer, gyroscope, temperature, and heart rate • HR sampled using IR spectrum at 250 Hz so able to do much more than wrist-worn • Contactless battery charge lasts 7 days, data capacity 6 weeks • Low power Bluetooth download to phone
  • 53. • Consumer grade wearables are not sleep labs, they use a proxy for an orchestra of sensors ! • Each individual sensor will have errors (movement, sweat, etc.) • The algorithms to compute “sleep efficiency” are opaque and proprietary • I compared S+ with Ōura for me over 8 weeks and … Accuracy of sleep tracking ?
  • 54. • 56 days, but S+ not continuous (travel, S+ is not travel-friendly) – I missed 1, 1 and 7 days • Correlation is (only) 0.147 (rises to 0.16 when blanks eliminated) • And because ranges may not be normalized, the visualization reveals … S+ and Ōura
  • 55. S+ and Ōura - 56 days continuous – 0.147
  • 56. So What ? • Its all a bit … unsatisfactory • I’m feeling a bit … let down … it shouldn’t be like this, there’s something not right • I’m actually feeling … trapped .. its my data, about me, but I can’t fix this, for me, I can’t leverage benefit, • Not everyone is like me though, and would want to do this, but maybe you’d want somebody to do this for you, like a wealth manager or stock broker • There are some benefits at population level 56
  • 57. Almost all vendors who allow us gather individual data, get value from pooling anonymised data • Fitbit – 150B hours of HR data, statistics from millions of sleep nights • 23a DNA testing – diseases, long-lost relatives • Strava – global heatmap of runs/cycles • Jaw – pinpointing earthquake epicentres ! Population-level Analytics from Pooled Individual Data
  • 58. Resting Heart Rate is an excellent indicator of overall health. https://finance.yahoo.com/news/exclusive-fitbits-150-billion-hours-heart-data-reveals-secrets-human-health-133124215.html
  • 60. 2014 – Northern Calif. 6.0M Earthquake % people who woke up correlates with distance from epicentre
  • 61.
  • 62.
  • 63. Back to my problem of … • Unsatisfactory … let down … trapped 63
  • 64. • Open up about their algorithms and allow a “normalisation” of sleep metrics for interoperability and device/vendor independence • Include “errorbars” in their outputs • Allow exporting of our data, our original raw data, with those error margins Sleep Sensing Vendors ought to …
  • 65. • Imagine a world where our data is managed for us by trusted third parties analogous to how banks manage wealth • They may pool it, like banks lend money • They may share mine with organisations like social media companies, educational institutions, entertainment companies, etc. but only if I say so, and I might get paid, or I might get better services • I would be delighted rather than afraid, to gather data and have data gathered about me for my benefit • Most people would not micromanage data, most people don’t micromanage wealth • We have seen the data points that can be gathered Utopia ? Naivete ?
  • 66. • What’s the appetite for this ? • Is there even a demand ? • Are there green shoots against surveillance capitalism ?
  • 67. • DuckDuckGo, StartPage, SearX.me, DisconnectSearch, MetaGer, Quant, all trace-free search engines Trace-free services
  • 68. Devices Apple Watch & HealthKit Wahoo Chest Strap Fitbit Activity Tracker / Scales Nokia Activity Tracker / Scales / BP … Aggregation Software HealthKit Microsoft Health Gyrosco.pe Remember this ?
  • 69. 6 9 • Consolidates health data from iPhone, Apple Watch, and third-party apps • Activity, Sleep, Mindfulness, and Nutrition • “You are in charge of your data” • “The Health app lets you keep all your health and fitness information under your control and in one place on your device. You decide which information is placed in Health and which apps can access your data through the Health app”
  • 75. dacadoo • Revenue stream is corporate • Employer signs up as a service to employees, employees use it, employer gets healthier (more productive) staff
  • 76. • I don’t see a spanner breaking up Surveillance Capitalism, too traumatic • I see a slow creep towards – Surveillance Socialism (don’t like that) – Surveillance Democracy (don’t like that either) – Sousveillance • Not happening overnight, generational

Editor's Notes

  1. Sunday Times article