This paper presents a context-aware mobile recommender system, codenamed Magitti. Magitti is unique in that it infers user activity from context and patterns of user behavior and, without its user having to issue a query, automatically generates recommendations for content matching. Extensive field studies of leisure time practices in an urban setting (Tokyo) motivated the idea, shaped the details of its design and provided data describing typical behavior patterns. The paper describes the fieldwork, user interface, system components and functionality, and an evaluation of the Magitti prototype.

1. Activity-Based Serendipitous Recommendations
with the Magitti Mobile Leisure Guide
System Codename: Magitti
Designed and Prototyped by PARC for
Dai Nippon Printing Co. Ltd.
Presenters
• Victoria Bellotti
• Bo Begole
• Ellen Isaacs
The Other Co-authors
Ed H. Chi, Nicolas Ducheneaut, Ji Fang,
Tracy King, Mark W. Newman, Kurt Partridge,
Bob Price, Paul Rasmussen, Michael Roberts,
Diane J. Schiano, Alan Walendowski

2. Overview
Recommendation Server
Infer Activity
Filter and Rank •Background
Model Preferences
Database Items
and motivating
Context: Time,
Location, etc.
fieldwork
Restaurants, stores,
events, etc.
Preferences:
Sushi, Bookstores,
etc.
•System design
Mobile
Device
Feedback
•Evaluation
Local Area
Consumer
2

3. About Dai Nippon Printing Co. Ltd.
• DNP is a world leader in printing technology
and solutions
• Affected by the shift from paper to Traditional
digital media Publishing
The Past: People carried magazines
The Present: Most Japanese use a mobile
phone to browse the Web
and read/write E-mail
• DNP asked PARC to develop core technology
for new, consumer-friendly digital media
• All design to be driven by real need
motivated a lot of work to identify:
• Best target users
Modern
• Best solution for their needs
Publishing
3

4. Contextual Publishing Concept Development
Discover Technology Fieldwork 1 Fieldwork 2 Finalized
Target Brainstorm Choose Confirm Concept
Users Best Idea and Refine Proposal
Assess many markets Share background Interviews, Evaluate design Future
domain info observations, mock-up in situ technology
and scenario analysis
feedback
Personas bring
customer to life
Develop scenarios
and obtain feedback
Leisure guide
concept
proposal,
“Magitti”
Brainstorm
design ideas Analyze results
Refine Refine design based
concept on user feedback
design
Choose the best
Young Adults Activity-Aware What to Build
4
at Leisure Leisure Guide

5. Many User Studies During Concept Development and Early
System Development
Features Content Interaction Venue database Activity type
Informing Design of Form-factor Functions style classification prediction
Identity Social factors Planning Coordination Leisure activity Leisure activity
Information in leisure Transportation venue types type timing &
Knowledge popularity probability
sources
of locale Technology use
Fashion Information Media Leisure activity Leisure activity type
desired use Leisure activity types type locations frequency
Correlating
Analysis: increasing abstraction Classifying Coding Counting
Observation 370 activity, time 3000 activity &
reminders Practices Needs Priorities Problems & location reports time reports
Survey Time Time
Data responses Diary
1000’s of Photos Notes 40 Transcripts 670 Responses 10 Transcripts Location entries
Observation In-depth Surveys Focus Activity Mobile-
interviews Groups Sampling phone
Diaries
Study Methods 5

6. From Fieldwork: Who Are the Users
• Japanese youth are especially receptive to new technology
• 19-25 year-olds spend 1.5 times more time in leisure activities
than 16-19 year-olds or 26-33 year-olds
• Less school and work pressure
• Ideal target for our design
• Still very, very busy
• School, jobs and little sleep
• Relaxation is a priority
• The system should do the work
• Want to know what others think
• Value opinions of real people
• Include end-user content
6

7. From Fieldwork: What do they Do?
• Outings often involve meeting friends
• Often at “halfway point” far from
homes
• Eager for local and localized info
• Unfamiliar with locations they visit
• Open to suggestions
• May not plan the main activity
• May not plan follow-on activities
• Motivation for Magitti 60
50
• A city-guide that assists in
exploration 40
30
Ratings of
“How well I know this neighborhood” 20
given by 170 young people stopped on the 10
streets in diverse neighborhoods in Tokyo 0
1 2 3 4 5 6
77
1 = Not at All 7 = Extremely Well

8. Overview
Recommendation Server
Infer Activity
Filter and Rank •Background
Model Preferences
Database Items
and motivating
Context: Time,
Location, etc.
fieldwork
Restaurants, stores,
events, etc.
Preferences:
Sushi, Bookstores,
etc.
•System design
Mobile
Device
•Evaluation
Local Area
Consumer
8

9. User Interface
Pie Menu Details
Map
9

10. Demo Video
http://www2.parc.com/csl/groups/ubicomp/videos/magitti_project_demonstration.wmv
10

11. Activity Information Utility
EAT Straits Cafe 0.77
Recommendable
Items EAT Fuki Sushi 0.64
Filtering
and SEE J. Gallery 0.60
Restaurant Reviews Ranking EAT Tamarine 0.57
Store Descriptions
Parks Descriptions DO Sam’s Salsa 0.39
Movie Listings
EAT Bistro Elan 0.38
Museum Events
Magazine Articles BUY Apple Store 0.33
… EAT Spalti 0.31

12. Context History
• Time • Prior population
• Location patterns
• Email analysis • User Queries
• Calendar analysis • User Locations
Eat 35%
What you Buy 20%
are doing See 25%
Do 10%
now Read 10%
Activity Information Utility
EAT Straits Cafe 0.77
Recommendable
Items EAT Fuki Sushi 0.64
Filtering
and SEE J. Gallery 0.60
Restaurant Reviews Ranking EAT Tamarine 0.57
Store Descriptions
Parks Descriptions DO Sam’s Salsa 0.39
Movie Listings
EAT Bistro Elan 0.38
Museum Events
Magazine Articles BUY Apple Store 0.33
… EAT Spalti 0.31

13. Personal Preferences Context History
• Explicit preferences • Time • Prior population
• Ratings of places • Location patterns
• Topics of documents read • Email analysis • User Queries
• Behavior; where/when/what • Calendar analysis • User Locations
Eat 35%
What you Buy 20%
What
are doing See 25%
you like Do 10%
now Read 10%
Activity Information Utility
EAT Straits Cafe 0.77
Recommendable
Items EAT Fuki Sushi 0.64
Filtering
and SEE J. Gallery 0.60
Restaurant Reviews Ranking EAT Tamarine 0.57
Store Descriptions
Parks Descriptions DO Sam’s Salsa 0.39
Movie Listings
EAT Bistro Elan 0.38
Museum Events
Magazine Articles BUY Apple Store 0.33
… EAT Spalti 0.31

14. Personal Preferences Context History
• Explicit preferences • Time • Prior population
• Ratings of places • Location patterns
• Topics of documents read • Email analysis • User Queries
• Behavior; where/when/what • Calendar analysis • User Locations
Eat 35%
What you Buy 20%
What
are doing See 25%
you like Do 10%
now Read 10%
Activity Information Utility
EAT Straits Cafe 0.77
Recommendable
Items EAT Fuki Sushi 0.64
Filtering
and SEE J. Gallery 0.60
Restaurant Reviews Ranking EAT Tamarine 0.57
Store Descriptions
Parks Descriptions DO Sam’s Salsa 0.39
Movie Listings
EAT Bistro Elan 0.38
Museum Events
Magazine Articles BUY Apple Store 0.33
… EAT Spalti 0.31

15. Predicting Activities from 100%
Mon-Thu
Sample Count (Total)
80
90%
70
80%
Population Priors
70% 60
NOT
60% 50 SEE
50% 40 DO
40% EAT OUT
30 SHOP
30%
Aggregate
20
When there is no user-specific
20%
10% 10
all data
0% 0
1 0
1 0
1 0
1 0
2 0
0
1 0
1 0
2 0
data, prior population data is used
0
0
0
0
:3
:0
:3
:0
:3
:0
:3
:0
:3
:0
:3
:0
:0
0
2
3
5
6
1
2
8
9
6
7
9
0
1
Time of Day
Friday
Mobile-phone Code each respondent’s 100%
Sample Count (Total)
20
90%
Diaries activities over 7-day week 80%
70% 15
NOT
60% SEE
50% 10 DO
40% EAT OUT
30% SHOP
5
20%
10%
0% 0
1 0
1 0
1 0
1 0
2 0
0
1 0
1 0
2 0
0
0
0
0
:3
:0
:3
:0
:3
:0
:3
:0
:3
:0
:3
:0
:0
0
2
3
5
6
1
2
8
9
6
7
9
0
1
Time of Day
Saturday
100% 25
Predict
Sample Count (Total)
Hourly activity report: 90%
• Who
80%
70%
20
probability
of each
NOT
• Where 60%
50%
15 SEE
DO
• When 40%
30%
10 EAT OUT
SHOP activity
• What 20% 5
type
10%
• Info used & desired 0% 0
1 0
1 0
1 0
1 0
2 0
0
1 0
1 0
2 0
0
0
0
0
:3
:0
:3
:0
:3
:0
:3
:0
:3
:0
:3
:0
:0
0
2
3
5
6
1
2
8
9
6
7
9
0
1
Time of Day
100%
Sunday
Sample Count (Total)
20
90%
80%
70% 15
NOT
60% SEE
50% 10 DO
40% EAT OUT
30% SHOP
5
20%
10%
15
0% 0
1 0
1 0
1 0
1 0
2 0
0
1 0
1 0
2 0
0
0
0
0
:3
:0
:3
:0
:3
:0
:3
:0
:3
:0
:3
:0
:0
0
2
3
5
6
1
2
8
9
6
7
9
0
1
Time of Day

16. Predicting Activities from Email/SMS
• How well do messages suggest activity?
• We examined a public set of 10,000 SMS messages from National University
of Singapore students, similar to the Magitti target demographic
• Approximately 11% of the messages contain information related to leisure
activities
tomorrow what time you be in school? think me and shuhui meeting in
school around 4. then duno still can see movie or not because
duno if a rest want meet for dinner.
• Keywords and linguistic structures are identified and sent to the activity
inference mechanism
ACTCAT=SEE, EAT :: ACTTIME=2007/05/26 16:00 ::
UNCERTAINTY=10 minutes :: TENSE=FUTURE
16

17. Learning Individual Patterns
Date/Time Location Address Venue Venue Activity
Name Type Class
Sun, 27 Jan 2008 37°26’39” 389 Ramona Evvia Restaurant EAT
11:57- 12:45 -122°9’38”
Tue, 29 Jan 2008 37°23’11” 545 Brickworks Cafe EAT
1:22 - 1:31 -122°9’02” Hamilton,
Wed, 30 Jan 2008 37°26’39” 143 Quarry Walgreens Store SHOP
11:57- 12:45 -122°9’18” Road
Fri, 1 Feb 2008 37°24’11” 854 Restoration Store SHOP
13:11 - 13:37 -122°9’00” University Hardware
… … … … …
Downtown
EAT Most
Likely
0 2 4 6 8 10 12 14 16 18 20 22 24
SHOP Most
Likely Time 
Individualized Shopping Center
pattern by region
Undetermined
0 2 4 6 8 10 12 14 16 18 20 22
17
24

18. Activity Inference Evaluation
Magitti Accuracy on Palo Alto Field Evaluation Data
100%
82%
77%
80%
62%
60%
40%
20%
0%
Baseline (EAT) Time and Place Priors * Priors + Learning†
* Time and Place Priors is significantly different than Baseline (Chi Square p=0.014, McNemar p=0.048).
† Priors + Learning is significantly different than Baseline (Chi Square p=0.0027, McNemar p=0.008).
18

19. Overview
Recommendation Server
Infer Activity
Filter and Rank •Background
Model Preferences
Database Items
and motivating
Context: Time,
Location, etc.
fieldwork
Restaurants, stores,
events, etc.
Preferences:
Sushi, Bookstores,
etc.
•System design
Mobile
Device
•Evaluation
Local Area
Consumer
19

20. Preliminary Field Evaluation
• 11 people, 32 outings (2.9 per person)
• Shadowed one outing per participant
• 60 places visited (1.9 per outing)
• 30 restaurants, 27 shops, 3 parks
• 16 outings accompanied by companion(s)
Using Magitti in a demo
20

21. Overall Usefulness
• Usefulness
• Average of 35.0 recommendation list pages viewed per outing
• People rated “helpfulness” 4.1 on 5-point scale (5 high)
• "Cool! I like that. I would never have found that place if it wasn't for
this.”
• "It makes life more interesting. It allows you to get out of your daily
routine, almost as if you’re going to a different city.”
• Serendipitous Discovery
• 53% of places visited were new to the participants
• On 67% of outings they went to at least one new place
• On 69% of outings, they noticed another new place to visit later
21

22. User Response
• Predicting User Activity
• People changed activity 5.1 times per outing
• “It’s very nice that it recommends things without you
having to do anything, but sometimes you want to ask
for specific things.”
• Even when Magitti got it right, they still sometimes
switched, apparently because they wanted all the
recommendations to be for that activity
• Social Use
• Five of eight users reported difficulty in sharing
experience with another person
• Magitti user seen as disconnected from others and/or
controlling the outing
22

23. Quality of Recommendations
• Recommendations rated 3.8 on 1-5 scale of "relevant
and of interest“
• "Most of the time, the list contained a mix of useful
and not so useful recommendations“
• Biggest factors to reduce confidence in
recommendations
• Not seeing a nearby place in the list
• Getting recommendations for places too far away
• Lack of transparency of reasons for recommendations
23

24. Replace Tedious Mobile Searching with
Personalized Recommendations
• Information and
suggestions based on
• Situation
• Past behavior
• Personal preferences
Stop searching!
Let information find you!
Victoria Bellotti, Bo Begole, Ed H. Chi, Nicolas Ducheneaut, Ji Fang, Ellen Isaacs,
Tracy King, Mark W. Newman, Kurt Partridge, Bob Price, Paul Rasmussen, Michael
Roberts, Diane J. Schiano, Alan Walendowski
Thanks also to: Ame Elliott and Dai Nippon Printing
24

25. Supplemental Slides
25

26. Predicting Astrid’s Grocery
Activities Hector’s Cafe
from
EAT BUY
Learned User
Patterns
Venue 50%
Likelihood: 50%
12:00 1:00
Weekly Behavior Patterns
Context History Monday Tuesda
Time Location Visit … … …
11:57- 12:45 37°26’39”
12:00 $ $
-122°9’38”
to 1:00 $$ $$
1:22 - 1:31 37°23’11” Grocery Grocery
-122°9’02” Cafe Cafe
… … … …
1:00 to … …