How To Monitor Human Behaviors:
Current Status & Our Approaches

October 24th, 2013
Pil Ho Kim, pilho.kim@unitn.it
Univers...
Current Status
• What did you feel when you read the title of this talk?
• Then let’s see what is going on now. We will re...
Google Now
• Objective: An Intelligent Personal Assistant (Know-it-all)
• http://www.google.com/landing/now/

Images © Goo...
Main Services (as of now)
#1. Manage your day

#2. Stay connected

#3. Be a local
4

Images © Google
Information Sources
• Gmail Cards (GC)
• Web History (WH)
• Synched Calendar (SC)
• Current Location (CL)
• Google Plus (G...
#1. Manage Your Day
Sources: Gmail Cards (GC), Web History (WH), Synched Calendar (SC), Current Location (CL), Google Plus...
#2. Stay Connected
Sources: Gmail Cards (GC), Web History (WH), Current Location (CL), Google Plus (G+), User Selection (U...
#3. Be A Local
Sources: Current Location (CL)

• Public transit (CL)
• Places (CL)
• Nearby attractions (CL)
• Nearby even...
Supported Platforms

Desktop supports are under way through Chrome notification:
source: https://support.google.com/mail/an...
User Interfacing
• Majority of natural language interfaces through
texts and also speech.

• Cards, cards and cards (Unifie...
What Others Think on Google Now?

Google Now for iOS review: straddling the creepy line
(May 2nd, 2013),The Telegraph

• A...
What Others Think on Google Now?

Review: Google Now useful as a supplement to Siri (4 May,
2013), USA Today

• Google Now...
What Others Think on Google Now?
Google’s Virtual Assistant Google Now Is Awesome - Here’s Why I Never
Want To Use It (22 ...
What Others Think on Google Now?

Google Now could be the injection of intelligence smart watches
need (16 Oct, 2013), the...
Development History
• Late 2011: Enhancing existing Google Voice Search
(Codename: Majel)

• June 27, 2012: Google Now app...
Trends on “Google Now”
Google Now
comes to iPhone

Images ©
Google Trend
Today Google Now Told Me..
Personal Review (I)
• Great example in merging heterogenous
sources to provide intelligent &
personalized services.

• Sho...
Personal Review (II)
• It looks clear that Google runs at least

multiple instances of real-time listeners
(CEP) for each ...
Moves
• Tracks the amount of exercise -- activity tracking app.
• http://www.moves-app.com
Features

Images ©
Moves-app.com
Our Approaches
+

Google Now

Moves

And keep you data under your hood !
Images © Google
& Moves-app.com
Big Picture

Data

Semantic

27
Big Picture

Sources

Data

Event
Processor

27

Semantic

Knowledge
Events & System
• We need to develop a system for event detection.
• We mean an event as a change from multiple sources.
•...
Implementation Strategies
• The amount and quality of information sources directly
affects the machine intelligence.

• Th...
Mining Data to Events
• Many methods exist like signal processing, statistical
analysis and probabilistic state modeling m...
Complex Event Processing
Video © Opher Etzion, IBM Research

CEP is a technology to help you process complex events.
So do...
Information Sources (as of now)
•
• Ambient Temperature (AT)
• Battery Status (BS)
• Light Level (LL)
• Gravity (GV)
• Gyr...
How to Detect Events?
Method #1. User Records

36
How to Detect Events?
Method #2. Manual Data Analysis

37
How to Detect Events?
Method #3. Automated Event Detection Using CEP

38
Esper?
http://www.espertech.com

•

An esper refers to an individual capable of telepathy and other similar
paranormal abi...
Esper Example (First Try)
Tracking user’s
indoor &
outdoor location

40
Esper Example (First Try)
Tracking user’s
indoor &
outdoor location

Too complex logic in the
query statement to output
an...
Current Supports:
Now You Can Write Much Less

41
Current Supports:
Now You Can Write Much Less
An abstract event class that encapsulate heterogenous
sources to trigger nex...
Thank You.

Any question?

43
References
Please refer following references for details.

• Opher Etzion, Modeling and semantics of events and

contexts,...
Upcoming SlideShare
Loading in …5
×

How To Monitor Human Behaviors: Current Status & Our Approaches

998 views

Published on

This talk will review the current status of human behavior monitoring methods to develop an intelligent software assistant for users’ daily lives. Google Now like existing Apps will be reviewed in comparison with our approaches. Technical aspects for system implementation will be thoroughly discussed to give a clear picture on the system including popular sensors, human activity event processing methods and behavior pattern mining approaches. Personal opinions and ideas will be freely discussed during the talk.

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
998
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
28
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

How To Monitor Human Behaviors: Current Status & Our Approaches

  1. 1. How To Monitor Human Behaviors: Current Status & Our Approaches October 24th, 2013 Pil Ho Kim, pilho.kim@unitn.it University of Trento, Italy 1
  2. 2. Current Status • What did you feel when you read the title of this talk? • Then let’s see what is going on now. We will review three software applications: • Google Now • Moves • Our work
  3. 3. Google Now • Objective: An Intelligent Personal Assistant (Know-it-all) • http://www.google.com/landing/now/ Images © Google
  4. 4. Main Services (as of now) #1. Manage your day #2. Stay connected #3. Be a local 4 Images © Google
  5. 5. Information Sources • Gmail Cards (GC) • Web History (WH) • Synched Calendar (SC) • Current Location (CL) • Google Plus (G+) • User Reminder (UR) • Current Time (CT) • “Remind me” Button (RB) • Google Offer Service (GO) • User Selection (US) • Event Booking (EB) • Google Finance Portfolio (GF) • Smart TV (ST) • Collaboration with 40 more companies (http:// www.google.com/landing/now/ integrations.html)
  6. 6. #1. Manage Your Day Sources: Gmail Cards (GC), Web History (WH), Synched Calendar (SC), Current Location (CL), Google Plus (G+), User Reminder (UR), Current Time (CT), “Remind me” button (RB), Google Offer service (GO) • Boarding pass (GC) • • • Activity summary (CL) • • Next appointment (SC, CL) • • Weather (CL) • • Traffic & Transit (CL, WH) • • Flights (GC) • • Hotels (GC) • • Restaurant reservations (GC, CL) • Events (GC, RB) Packages (GC) Friends’ birthday (G+) Your birthday (G+) Time reminders (UR, CT) Location reminders (UR, CT, CL) Event reminders (RB) Saved offers (GO) Car rentals (GC)
  7. 7. #2. Stay Connected Sources: Gmail Cards (GC), Web History (WH), Current Location (CL), Google Plus (G+), User Selection (US), Event Booking (EB), Google Finance Portfolio (GF), Smart TV (ST) • Zillow (Search nearby real estate listings) (WH, CL) • • Movies (WH, CL) • Fandango (Movie booking) Sports (US) (EB) • Concerts (WH, CL) • Stocks (WH, GF) • Public alerts (WH, CL) • Developing story & breaking news (WH) • Research topic (WH) • New albums (WH) • New books (WH) • New video games (WH) • New TV episodes (WH) • TV cards (ST) • Concert ticket (GC)
  8. 8. #3. Be A Local Sources: Current Location (CL) • Public transit (CL) • Places (CL) • Nearby attractions (CL) • Nearby events (CL) • Nearby photo spots (CL) • Translation (CL) • Currency (CL) • Time at home (CL)
  9. 9. Supported Platforms Desktop supports are under way through Chrome notification: source: https://support.google.com/mail/answer/1075549?hl=en Images © Google
  10. 10. User Interfacing • Majority of natural language interfaces through texts and also speech. • Cards, cards and cards (Unified representation of user’s personal events). • Sources from the user through interactions and monitoring in consideration of contexts from outer sources. • Notification methods: App and widget for mobile and notification bar for desktop
  11. 11. What Others Think on Google Now? Google Now for iOS review: straddling the creepy line (May 2nd, 2013),The Telegraph • As more services are added, it will keep track of more of your choices. These things are either creepy or cool, depending on your definition of privacy. • Unusually for Google it’s a beautiful, graceful interface that can present a lot of information in a concise, aesthetically pleasing format. • Features can be turned off with impressive granularity. Also the battery life is issued. Source: http://www.telegraph.co.uk/technology/mobile-app-reviews/10032788/Google-Now-foriOS-review-straddling-the-creepy-line.html
  12. 12. What Others Think on Google Now? Review: Google Now useful as a supplement to Siri (4 May, 2013), USA Today • Google Now’s power lies in giving you information you need to know before you have to ask. It works best as a supplement to Siri, rather than a replacement. • Google Now's judgments on what information I need to know can be sketchy at times. • Despite getting stray information at times, I find Google Now useful enough to leave it on. I could always customize the service by telling it never to give me stock quotes, for instance. Source: http://www.usatoday.com/story/tech/2013/05/04/google-now-siri/2129755/
  13. 13. What Others Think on Google Now? Google’s Virtual Assistant Google Now Is Awesome - Here’s Why I Never Want To Use It (22 June, 2013), Business Insider • Google Now puts Siri to shame – it's more accurate and far more versatile. • Trouble question:  It begs to be relied upon and integrated into your lifestyle in a major way. Even a casual Google Now user is still feeding all his data to it every time he uses the web. • The intrinsic nature of Google Now is that it's designed to be blindly relied upon, to be a "life aid." What happens when it goes kaput for some reason? All those things that you were used to not having to think about become problematic. • I'm in charge of my phone. My phone's not in charge of itself. Google Now nearly represents the phone coming to life, acting on its own and without your direction. It becomes slightly less your phone and you become a slightly less mindful. Source: http://www.businessinsider.com/the-case-against-google-now-2013-6
  14. 14. What Others Think on Google Now? Google Now could be the injection of intelligence smart watches need (16 Oct, 2013), theguardian • Google is reckoned to be working on a Google Now- focused smart watch codenamed ‘Gem’ which could finally make smart watches intelligent (Confirmed yesterday -- Source #2). • Industry experts reckon none of the current smart watches has truly "smart" functionality. And that leaves the door wide open for something that multiple sources are murmuring about: a Google Now-powered wearable. • Google started a project (Codename “Gem”) to bring Google Now to the smart watch. FYI, Google Glass already supports Google Now. Source #1: http://www.theguardian.com/technology/2013/oct/16/google-now-gem-smartwatch Source #2: http://online.wsj.com/news/articles/SB10001424052702304655104579165080029933904
  15. 15. Development History • Late 2011: Enhancing existing Google Voice Search (Codename: Majel) • June 27, 2012: Google Now appeared in Android 4.1 Jelly Bean Google I/O • October 29, 2012: UI changed to Cards with information pulled from Gmail account, user location, search history and more have been added since then. • March 21, 2013: iOS version of Google Now • May 15, 2013: Google I/O 2013 announced the upcoming release of Google Now on desktop platforms; the feature will be accessible only via Google Chrome or Google Chrome OS.
  16. 16. Trends on “Google Now” Google Now comes to iPhone Images © Google Trend
  17. 17. Today Google Now Told Me..
  18. 18. Personal Review (I) • Great example in merging heterogenous sources to provide intelligent & personalized services. • Shows how Google monitors personal behaviors and what it classifies. • The concept of Google Card adds strong flexibility in extending personalized services.
  19. 19. Personal Review (II) • It looks clear that Google runs at least multiple instances of real-time listeners (CEP) for each Google Now user. • Looks promising but not working well for me. Accuracy (in multiple aspects) could be an issue. This email was exchanged on Mar 2013. The content is in fact a customer service email asking about my old flight travel (11/10/2012 - 24/10/2012), not today (24/10/2013).
  20. 20. Moves • Tracks the amount of exercise -- activity tracking app. • http://www.moves-app.com
  21. 21. Features Images © Moves-app.com
  22. 22. Our Approaches + Google Now Moves And keep you data under your hood ! Images © Google & Moves-app.com
  23. 23. Big Picture Data Semantic 27
  24. 24. Big Picture Sources Data Event Processor 27 Semantic Knowledge
  25. 25. Events & System • We need to develop a system for event detection. • We mean an event as a change from multiple sources. • Personal events are so diverse and non-static. • So we do not define an event. • But our system provides a way for users (or developers) to define events of their own with good enough freedom. • Event streams can be modeled as a pattern to build up a personalized knowledge base (i.e. an ontology linking a word or words in semantic relations to low-level changes).
  26. 26. Implementation Strategies • The amount and quality of information sources directly affects the machine intelligence. • The challenges are then (1) a backend to collect enormous data from various sources and (2) methods to mine big data. • We have currently implemented (1), which will get better, and are working on (2).
  27. 27. Mining Data to Events • Many methods exist like signal processing, statistical analysis and probabilistic state modeling methods. • Why Complex Event Processing (CEP)? • Rich supports for temporal logics and complex controls. • Encapsulate above methods as event sources using, what are called, Platform Independent Model (PIM) and Computational Independent Model (CIM) that are under various testing for implementation. • Enable continuous learning rather than offline learning. Though, it should be noted that real-time pattern modeling needs significant efforts. 33
  28. 28. Complex Event Processing Video © Opher Etzion, IBM Research CEP is a technology to help you process complex events. So don’t be scared but start using it as a tool. 34 Source: http://www.slideshare.net/opher.etzion/event-semantics-and-modelpresented-at-the-event-based-multimedia-workshop-presentation-at-acmmultimedia-2013
  29. 29. Information Sources (as of now) • • Ambient Temperature (AT) • Battery Status (BS) • Light Level (LL) • Gravity (GV) • Gyroscope (GS) • Linear Acceleration (LA) • Orientation (OE) Accelerometer (AC) • Magnetic Field (MF) • Rotation Vector (RV) • GPS Location (GL) • Network Location (NL) • WIFI (WF) • Pressure (PL) • Relative Humidity (RH) We need more sources to be intelligent.
  30. 30. How to Detect Events? Method #1. User Records 36
  31. 31. How to Detect Events? Method #2. Manual Data Analysis 37
  32. 32. How to Detect Events? Method #3. Automated Event Detection Using CEP 38
  33. 33. Esper? http://www.espertech.com • An esper refers to an individual capable of telepathy and other similar paranormal abilities. • • Sorry. • Esper is an extensible middleware supporting: Esper is an open source event stream processing (ESP) and event correlation engine (CEP). • • • • • • Java, .Net, Map or XML events Runtime statement management API or configuration driven Plug-in SDK for functions, aggregations, views and pattern detection extensions Agile development and testing, integrates with Eclipse and JUnit Adapters: Adapters to any possible transport exist through integration with Apache Camel, ServiceMix and Mule ESB. EsperIO and Enterprise Edition provide additional adapters. 39
  34. 34. Esper Example (First Try) Tracking user’s indoor & outdoor location 40
  35. 35. Esper Example (First Try) Tracking user’s indoor & outdoor location Too complex logic in the query statement to output an event detection status Stream dependent domain knowledge Sensor-specific source selection 40
  36. 36. Current Supports: Now You Can Write Much Less 41
  37. 37. Current Supports: Now You Can Write Much Less An abstract event class that encapsulate heterogenous sources to trigger next actions (ex. pushing/pull notification, auto recording, pattern mining, etc.) Generalized status retrieval State changes are now modeled as a pattern. Domain knowledge becomes independent functions. An abstract event class that encapsulates heterogenous sources. 41
  38. 38. Thank You. Any question? 43
  39. 39. References Please refer following references for details. • Opher Etzion, Modeling and semantics of events and contexts, http://www.slideshare.net/opher.etzion/eventsemantics-and-model-presented-at-the-event-basedmultimedia-workshop-presentation-at-acmmultimedia-2013 • Google Now: http://www.google.com/landing/now/ • Moves: http://www.moves-app.com 44

×