This document describes a study that aimed to establish a method for understanding user experiences in property searching through analyzing Twitter timelines. The researchers collected Twitter timelines of followers of a Japanese property search service account and used crowdsourcing microtasks to extract tweets related to property searching and analyze them based on a conventional property search process framework. Workers were asked to categorize timeline fragments as either related or unrelated to property searching. This allowed the researchers to build a corpus of property search behavior data derived from social media for analyzing user needs and experiences.

1. Mining
User
Experience
through
Crowdsourcing:
A
Property
Search
Behavior
Corpus
Derived
from
Microblogging
Timelines
Yoji
Kiyota
(NEXT
Co.,
Ltd,
Tokyo,
Japan)
Yasuyuki
Nirei,
Kosuke
Shinoda,
and
Satoshi
Kurihara
(Univ.
of
Electro-­‐CommunicaPons,
Tokyo,
Japan)
Hirohiko
Suwa
(NAIST,
Nara,
Japan)
DOCMAS/WEIN
2015
(WS1
of
WI-­‐IAT
2015)
6th
Dec.
2015
at
Singapore
Management
University
1

2. The
goals
of
this
study
• Establish
a
method
to
understand
various
behaviors
of
users
who
search
for
proper3es
(for
rent,
for
sales,
etc.)
• EsPmate
how
effecPve
is
microtask-­‐based
crowdsourcing
for
annota3ng
microblogging
3melines
with
user
experiences
2

3. HOME’S:
an
online
property
search
service
in
Japan
3

4. CharacterisPcs
of
property
search
(compared
with
other
products)
• taking
a
long
Pme
for
decision
– potenPal
needs
-­‐>
informaPon
gathering
-­‐>
contacPng
agents
-­‐>
property
preview
-­‐>
decision-­‐
making
and
contracPng
• user
needs
could
change
– trade-­‐offs
(price
vs.
condiPons)
– target
areas
– for
rent
or
for
sale?
– ...
→
understanding
user
needs
is
difficult!
4

5. ConvenPonal
approaches
for
understanding
user
needs
approaches
pros
cons
Analysis
of
user
behavior
logs
exhausPve
user
behavior
data
on
touch
points
(PCs,
smart
phones,
etc.)
is
available
behaviors
outside
the
available
touch
points
(e.g.
conversa3ons
with
agents,
families
and
friends)
have
major
impacts
on
user
experiences
QuesPonnair
es
users’
thoughts
and
senPments
can
be
gathered
unexpected
user
needs
and
unconscious
thoughts
and
sen3ments
cannot
be
obtained
Behavior
observaPon
suitable
for
idenPfying
needs
that
users
themselves
do
not
recognize
user
behaviors
on
property
search
services
change
through
search
processes
which
con3nue
from
weeks
to
several
years
5

6. Why
we
focused
on
Twiaer
Pmelines?
• Tweet
data
is
abundant
in
daily
user
behaviors,
including
acPons,
thoughts,
and
senPments
on
property
search
processes
• User
Pmelines
enable
us
to
trace
property
search
processes
of
specific
users,
which
conPnues
for
from
weeks
to
several
years.
6

7. A
snapshot
of
a
user
Pmeline
2010-­‐06-­‐14
19:16
Hmm.
We
have
just
moved
in
a
rented
house,
however,
I
get
rapidly
interested
in
buying
a
new
house!
I
feel
like
been
more
interested
by
previewing
properPes.
(48
tweets
are
omiaed)
2010-­‐07-­‐14
17:31
Now
I
come
to
the
decisive
moment
for
selecPng
a
new
house
for
buying.
Presently,
I
prefer
apartment
houses
to
single
houses,
because
single
houses
are
expensive.
But
I'm
indecisive,
and
I
cannot
decide
for
a
while...
(2
tweets
are
omiaed)
2010-­‐07-­‐14
18:17
@foo
Quite
so!
Now
we
are
currently
living
in
a
duplex
apartment,
I
am
worried
when
I
see
my
pregnant
wife
is
climbing
stairs
wheezily...
I
think
it
will
be
too
hard
over
seventy
years
old.
Finally,
we
would
feel
Presome
climbing
stairs,
and
spend
Pme
on
the
ground
floor.
7

8. The
issues
for
using
Twiaer
data
• User
Pmelines
also
have
a
lot
of
tweets
which
are
NOT
related
to
property
search
– How
to
extract
only
tweets
which
are
related
to
property
search?
• Tweet
analyses
based
on
a
convenPonal
framework
of
property
search
process
are
desirable
– potenPal
needs
-­‐>
informaPon
gathering
-­‐>
contacPng
agents
-­‐>
property
preview
-­‐>
decision-­‐
making
and
contracPng
8

9. Microtask-­‐based
crowdsourcing
9

10. The
overview
of
our
approach
10
(Task
1)
(Task
2)

11. Gathering
Twiaer
Pmelines
• Select
Pmelines
of
approx.
40,000
followers
of
@homes_kun
(a
mascot
character
of
HOME’S)
• Include
only
Pmelines
in
which
either
of
the
following
keywords
occur
– key
money
(礼金),
preview
(内見),
rent
(家賃)
• Exclude
Pmelines
of
which
over
25%
of
tweets
are
with
hyperlinks
– because
such
accounts
are
operated
by
real
estate
agents
→
86
user
3melines
were
extracted
11

12. Task
1:
disPnguish
Pmeline
fragments
related
to
property
search
behaviors
• Each
microtask
is
genarated
by
dividing
user
Pmelines
into
fragments
(at
most
five
tweets)
– 2,400
microtasking
ques3ons
were
generated
• Each
microtask
has
three
choices
• Each
microtask
is
requested
to
three
workers
(applying
the
majority
rule)
• A
task
set
consists
of
five
microtasks
– one
of
the
five
microtask
is
an
embedded
(dummy)
task
– workers
who
send
some
wrong
answers
to
the
embedded
task
were
eliminated
12

13. A
task
quesPon
of
Task
1
Q:
Judge
whether
the
tweet
user
want
to
search
properPes
or
not,
by
viewing
the
following
Pmeline
fragment.
a
Pmeline
fragment
(five
tweets)
he/she
is
searching
properPes.
he/she
is
NOT
searching
properPes.
I
don’t
know.
13

14. Task
1:
stats
• Task
size
– 2,400
microtasking
quesPons
– 396
workers
parPcipated
in
the
task
– all
the
microtasks
were
performed
in
2
hours
25
min.
– 18,000
JPY
(approx.
150
USD)
• 223
of
396
workers
correctly
answered
all
the
embedded
tasks,
and
secondly
105
workers
correctly
– the
answers
by
the
328
workers
were
finally
accepted
14

15. Task
2:
results
by
applying
the
majority
rule
User
Pmelines
which
have
either
of
the
286
fragments
are
extracted
as
the
candidates
for
Task
2
15

16. Task
2:
tagging
of
user
Pmelines
with
four
property
search
stages
• Choose
only
user
Pmelines
in
which
mulPple
fragments
within
six
months
were
categorized
by
the
majority
rule
– 67
user
Pmelines
were
chosen
• The
task
definiPon:
annotate
each
Pmeline
fragment
(at
most
ten
tweets)
into
five
categories
(four
property
stages
+
“no
stage”)
• Each
microtask
is
judged
by
the
majority
rule
16

17. Four
property
search
stages
S1
potenPal
needs
for
property
search
S2
gathering
of
property
informaPon
S3
contacPng
agents
and
previewing
properPes
S4
decision-­‐making
and
contracPng
17

18. Issues
of
annotaPon
• A
task
quesPon
with
five
choices
(four
stages
+
“no
stage”)
is
not
suitable
for
microtask-­‐based
crowdsourcing
– difficult
tasks
should
be
divided
into
combinaPons
of
easy
tasks
• Naïve
division
of
an
annotaPon
tasks
into
a
combinaPon
of
five
Yes/No
quesPons
extremely
increases
costs
18

19. A
task
flow
eliminaPng
#
of
quesPons
using
dependencies
between
stages
19
Whether
does
the
user
have
potenPal
needs
for
property
search?
Whether
is
the
user
gathering
property
informaPon?
Yes
Yes
Yes
“no stage”
S1 (potential needs)
S2 (gathering information)
S3 (contacting agents)
S4 (decision-making)
Yes
No
No
No
No
Whether
is
the
user
contacPng
agents
and
previewing
properPes
Whether
is
the
user
decide
to
move?
2400
fragments
32
fragments
51
fragments
47
fragments
14
fragments
196
fragments
132
fragments
68
fragments

20. Task
2:
combinaPon
of
stages
single
stage
(50
3melines)
mul3ple
stages
(17
3melines)
20

21. Major
user
behaviors
in
S1
behaviors
#
of
tagged
fragments
#
of
users
cohabitaPon
with
partners
3
3
college/university
graduaPon
1
1
changing
jobs
2
1
lease
expiraPon
of
rooms
1
1
21

22. Major
user
behaviors
in
S2
behaviors
#
of
tagged
fragments
#
of
users
work
trip
lengths
13
12
costs
(rents
and
prices)
20
17
locaPon
7
6
storage
3
3
menPons
of
property
searches
10
7
22

23. Major
user
behaviors
in
S3
behaviors
#
of
tagged
fragments
#
of
users
work
trip
lengths
7
7
costs
(rents
and
prices)
20
11
locaPon
6
3
public
security
3
3
menPons
of
property
searches
15
12
menPons
of
previewing
properPes
9
7
complicaPons
for
agents
4
3
23

24. Major
user
behaviors
in
S4
behaviors
#
of
tagged
fragments
#
of
users
menPons
of
decisions
of
new
houses
3
3
complicaPon
for
agents
3
4
24

25. Related
work
• Twiaer
as
a
social
sensor
– Dow
Jones
Industrial
Average
(Bollen
2011)
– stock
market
events
(Ruiz
2012)
– earthquake
reporPng
system
(Sakaki
2013)
– this
study
focuses
on
gaining
deeper
insights
for
user
experiences
• AnnotaPng
Twiaer
Pmelines
using
microtask-­‐
based
crowdsourcing
– named
enPPes
(person,
organizaPon,
etc.)
(Finin
2010)
– this
study
focuses
on
user
experiences
/
behaviors
25

26. Conclusion
• Mining
user
experiences
and
behaviors
of
property
search
by
applying
microtask-­‐based
crowdsourcing
to
Twiaer
Pmelines
– effecPve
for
tracing
long-­‐Pme
property
search
processes
• Future
work
– larger
experiments
– applying
to
other
domains
(cars,
insurances,
educaPons,
and
jobs)
26