Tried the life-logging by using artificial intelligence Neuraltalk2 and quantified the activity in a day.

1. Basic Study on Life-‐‑‒logging Video Capture
via Neuraltalk2
2016/1/15
-‐‑‒ Research Note -‐‑‒

2. About myself
• Motohiko Takeda（@kusojig / takeda@rabbitsdata.jp )
• Interests:
• Data Based Marketing
• Machine Learning, Data Mining
• Data Strategy Planning & Execusion
• Currently freelance consultant for data utilization
and sensor technologies. Mainly experienced consulting
projects for mobile, baking and advertising industries.

3. Motivation: to estimate the time for housekeeping
• Some households have conflict about burden
sharing about housekeeping.
• I spend a lot of time in kitchen…
• Thatʼ’s not true. I also help dishes!
• Oh it doesnʼ’t take much time. I pay more!
• OK, So why donʼ’t we analyze housekeeping cost
quantitatively by using technologies?

4. Approach: Computer Vision with AI (Neuraltalk2)
• Set web camera at the top of dinning room and take
photos for each 15 seconds.
• Caption the picture by Neuraltalk2
(Neuraltalk2 gives caption by describing the picture).
• Estimate housekeeping time by categorizing the caption
data.

5. Setup WEB camera around the celling and regulated by PC
Kitchen
Dinning
table
Refrigerator
Overview of room and camera
Camera keeping
• Camera connected with
laptop PC via USB
Camera
and
laptop
PC

6. Caption results were almost collect except detailed actions
• Neuraltalk2 recognized “a man/woman standing in a kitchen”
when somebody stands around the kitchen.
• However, the same capture were given when somebody is sitting
in the dinning table near the kitchen.
• It could not recognized more detailed action like opening the
refrigerator.

7. Estimated time of staying around the kitchen could explain
the amount of activity in a day
00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Estimated time of “man/woman standing in a kitchen” per hour
Prepare
for
tomorrow
meal
*
48min/hour
at
a
peak
Wake
up
Prepare
for
a
supper
Lunch
and
having
a
tea
• Largely measured the day activity around the kitchen.
Supper
* Stayed home for whole day

8. Issues: individual identification and intersection
• Issues 1: individual identifiation
• We have not implemented individual identification.
• For identifying spent time for each task,
implementation of individual identification including
detection of side-‐‑‒face is necessary.
* In case of husband/wife identification, only gender
identification might be enough.

9. • Issues 2: Intersection
• Pictures in house generally happens intersection
since room has small space compared with public
space.
• Attention towards the tilt of camera is needed.
Issues: individual identification and intersection

10. Conclusion and future tasks
• Even no customized AI program can identify the activity
time and patterns in the room largely.
• By implementing individual identification, time spent in
housekeeping could be identified in particular.
• By taking life-‐‑‒log in more detail, we can develop the
recommendation for daily life.
• This project is still on progress (Jan. 2016)

11. For more information…
• We have developing the research and development for
sensor technologies, machine learning and deep
learning.
• For more information, please feel free to contact:
takeda@rabbitsdata.jp