Download to read offline
![Extent
of
mulCtasking
• 10.2
disLnct
sites,
2
visits
per
site.
Absence
Cme
• 50%
of
sites
are
revisited
aker
<
1min.
InterrupHon
of
a
task
• There
are
revisits
aker
long
breaks.
Performing
a
new
task
Online
mulLtasking
17
Online
MulCtasking
CharacterisLcs
0.00
0.25
0.50
0.75
1.00
10
2
10
1
10
0
10
1
10
2
Cumulativeprobability Absence time [min]
news (finance)
news (tech)
social media
mail
2.09
1.76
2.28
2.09
#Visits Absence
time [min]
3.85
3.95
4.47
6.86
Absence time: Time between two visits](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-17-320.jpg)
![Inter-‐site
engagement
35
Measuring
Engagement
Inter-‐site
engagement
metrics:
Network-‐level
Traffic
distribuCon
Flow
Extent
to
which
users
navigate
between
sites.
Density1
Diversity
of
inter-‐site
engagement.
Reciprocity2
Homogeneity
of
traffic
between
sites.
External
traffic
EntryDisparity
Variability
of
in-‐going
traffic
to
the
network.
ExitDisparity
Variability
of
out-‐going
traffic
from
the
network.
[1]
S.
Wasserman.
Social
network
analysis:
Methods
and
applicaHons,
1994.
[2]
T.
SquarHni,
F.
Picciolo,
F.
RuzzenenH,
and
D.
Garlaschelli.
Reciprocity
of
weighted
networks.
Nature:
ScienHfic
reports,
2013.](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-35-320.jpg)
![Inter-‐site
engagement
36
Measuring
Engagement
Inter-‐site
engagement
metrics:
Node-‐level
Traffic
distribuCon
PageRank1
Probability
that
a
user
will
visit
the
site.
Downstream
Probability
that
a
user
will
conLnue
browsing
to
other
sites.
External
traffic
EntryProb
Probability
that
a
user
enters
the
network
in
this
site.
ExitProb
Probability
that
a
user
leaves
the
network
in
this
site.
[1]
L.
Page,
S.
Brin,
R.
Motwani,
T.
Winograd.
The
pagerank
citaHon
ranking:
Bringing
order
to
the
web.
Technical
report,
Stanford
InfoLab,
1999.](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-36-320.jpg)
![• “In
a
world
full
of
choices
where
the
fleeCng
aPenCon
of
the
user
becomes
a
prime
resource,
it
is
essenLal
that
[...]
providers
do
not
just
design
[websites]
but
that
they
design
engaging
experiences.”
[A}ield].
• In
addiLon
to
uLlitarian
factors,
such
as
usability
and
usefulness,
we
must
consider
other
factors
of
interacLng
with
websites,
such
as
fun,
fulfillment,
play,
and
user
engagement.
Successful
websites
are
not
just
used,
they
are
engaged
with.
• In
order
to
design
engaging
websites,
it
is
crucial
to
understand
what
user
engagement
is
and
how
to
measure
it.
IntroducLon
60
MoCvaCon
Why
is
it
important
to
engage
users?](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-60-320.jpg)
![0-1 1-0.5 0.5
Kendall’s tau with p-value < 0.05
('-' insignificant correlations)
Site
engagement
64
EvaluaCon
CorrelaLons
between
engagement
metrics.
High
correlaCons
within
metric
groups.
Low
correlaCons
between
metric
groups.
[POP]#Users
[POP]#Visits
[POP]#Clicks
[ACT]PageViewsV
[ACT]DwellTimeV
[LOY]ActiveDays
[LOY]ReturnRate
#Users [POP] 0.82 0.75 - - 0.43 0.34
#Visits [POP] 0.82 0.85 - - 0.60 0.52
#Clicks [POP] 0.75 0.85 0.16 0.18 0.59 0.51
PageViewsV [ACT] - - 0.16 0.33 - -
DwellTimeV [ACT] - - 0.18 0.33 - -
ActiveDays [LOY] 0.43 0.60 0.59 - - 0.79
ReturnRate [LOY] 0.34 0.52 0.51 - - 0.79
0.69](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-64-320.jpg)
![Online
mulLtasking
67
MoCvaCon
Users
switch
between
sites,
to
perform
related
or
totally
unrelated
tasks.
Switching
between
tasks
(sites)
“…within-‐session
page
revisits
represent
the
most
common
form
of
revisitaLon,
covering
73,54%
of
all
revisits.”
[Herder]
Performing
tasks
(sites)
in
parallel
using
browser
tabs
“Most
of
our
parLcipants
switched
tabs
more
oken
than
they
used
the
back
bu;on.”
[Dubroy]
[Herder]
E.
Herder.
CharacterizaHons
of
user
web
revisit
behavior.
WWW
Workshop
ABIS,
2005.
[Dubroy]
P.
Dubroy,
R.
Balakrishnan.
A
study
of
tabbed
browsing
among
mozilla
firefox
users.
SIGCHI,
2010.](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-67-320.jpg)
![0-1 1-0.5 0.5
Spearman’s rho with p-value < 0.05
('-' insignificant correlations)
Online
mulLtasking
71
EvaluaCon
CorrelaLons
between
mulLtasking
and
acLvity
metrics.
[MT]SessVisits
[MT]SessSites
[MT]CumAct
[MT]AttShift
[MT]AttRange
[ACT]DwellTimeS
SessSites [MT] 0.42
CumAct [MT] 0.41 -
AttShift [MT] 0.09 - -
AttRange [MT] - - -0.38 0.27
DwellTimeS [ACT] 0.20 0.24 0.12 0.32 0.08
DwellTimeV [ACT] -0.40 - - 0.14 - 0.50
No
or
only
weak
correlaCons
between
the
metrics.
All
metrics
convey
different
aspects
about
users’
online
behaviour.](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-71-320.jpg)
![Inter-‐site
engagement
76
Inter-‐site
Engagement
Metrics
Density
describes
the
connecLvity
of
the
network.
Density
We
use
the
density
measure
of
[Wasserman]:
InterpretaCon
High
Density
à
High
inter-‐site
engagement
Users
navigate
between
many
different
sites
(inter-‐site
engagement
is
highly
diverse).
[Wasserman]
S.
Wasserman.
Social
network
analysis:
Methods
and
applicaHons,
1994.
Density =
# Edges
# Possible_ Edges
Density = 4/6 = 0.7
Flow = 2/6 = 0.3](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-76-320.jpg)
![Inter-‐site
engagement
77
Inter-‐site
Engagement
Metrics
Reciprocity
measures
the
homogeneity
of
traffic
between
two
sites.
Reciprocity
We
use
the
reciprocity
measure
of
[SquarLni]:
InterpretaCon
High
Reciprocity
à
High
inter-‐site
engagement
Users
navigate
between
two
sites
in
both
direcLons
(inter-‐site
engagement
is
highly
homogenious).
[SquarHni]
T.
SquarHni,
F.
Picciolo,
F.
RuzzenenH,
and
D.
Garlaschelli.
Reciprocity
of
weighted
networks.
Nature:
ScienHfic
reports,
2013.
#Clicks
between
node
i
and
j
wi, j
RP =
min[wi, j,wj,i ]
i<j∑
wi, ji≠j∑
1
10 5
20
1
Reciprocity = 15/50 = 0.3
Reciprocity = 2/37 = 0.05
10
10 5
20
5](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-77-320.jpg)
![Inter-‐site
engagement
78
Inter-‐site
Engagement
Metrics
Entry/ExitDisp
measures
how
the
traffic
to/from
the
network
is
distributed
over
the
sites.
Entry
disparity
and
exit
disparity
We
use
the
group
degree
measure
of
[Freeman]
and
adapt
it
as
follows:
InterpretaCon
High
Entry/ExitDisp
à
Low
inter-‐site
engagement
The
network
is
more
vulnerable
to
outages,
because
only
few
sites
are
used
to
enter
(leave)
the
network.
EntryDisp =
(gin
max − gin
i )
i∑
| N |• gin
ii∑
[Freeman]
L.
C
Freeman.
Centrality
in
social
networks
conceptual
clarificaHon.
Social
networks,
1979.
Number
of
visits
that
started
at
node
ni
(user
entered
the
network)
Maximum
value
of
gin
Number
of
nodes
| N |
gi
in
gin
max
EntryDisp = 20/3 40 = 0.17
20
10
10
40
5
5
EntryDisp = 70/3 50 = 0.47](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-78-320.jpg)
![Inter-‐site
engagement
79
EvaluaCon:
Network-‐level
CorrelaLons
between
inter-‐site
and
network
engagement
metrics.
[IS]Density
[IS]Reciprocity
[IS]EntryDisparity
[IS]ExitDisparity
[POP]#Sessions
[ACT]DwellTimeS
[ACT]#Sites
Flow [IS] - 0.15 0.23 0.30 - 0.35 0.65
Density [IS] 0.48 -0.61 -0.60 0.92 -0.45 -0.25
Reciprocity [IS] -0.38 -0.32 0.42 - 0.25
EntryDisparity [IS] 0.84 -0.54 0.33 -
ExitDisparity [IS] -0.55 0.38 0.20
0-1 1-0.5 0.5
Spearman’s rho with p-value < 0.01
('-' insignificant correlations)
Density
and
#Sessions
The
more
users
are
visiCng
the
network,
the
more
diverse
is
the
inter-‐
site
engagement.
Entry-‐
and
ExitDisparity
Volume
of
in-‐
and
out-‐
going
traffic
of
the
nodes
depend
on
each
other.
Flow
and
#Sites
The
more
sites
are
visited
during
a
session,
the
higher
the
flow
of
traffic.](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-79-320.jpg)
![Inter-‐site
engagement
80
EvaluaCon:
Node-‐level
CorrelaLons
between
inter-‐site
and
site
engagement
metrics.
[IS]Downstream
[IS]EntryProb
[IS]ExitProb
[POP]#Sessions
[ACT]DwellTimeS
[MT]#Visits
[MT]CumAct
PageRank [IS] 0.30 -0.08 -0.10 0.85 0.06 0.08 0.31
Downstream [IS] -0.27 -0.22 0.17 0.04 0.02 -0.02
EntryProb [IS] 0.79 0.12 -0.19 0.13 0.35
ExitProb [IS] 0.08 -0.18 0.18 0.32
0-1 1-0.5 0.5
Spearman’s rho with p-value < 0.01
('-' insignificant correlations)
PageRank
and
#Sessions
Popular
sites
in
the
provider
network,
are
also
visited
frequently
when
browsing
through
the
network.
Entry-‐
and
ExitProb
Nodes
that
are
used
to
enter
the
network
are
also
frequently
used
to
exit
the
network.](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-80-320.jpg)
![Wikipedia
90
Reading
PaPerns
Article
topic
Reading
behavior
ArticleViewsa
SessionArticlesa
Popularitya
ReadingTimea
CA - Percentage in topic
4,826 articles
11,579 behavior vectors
sportsperson
musician
media pers.
28%
26%
23%
CA
Exploration
artist/writer
historical fig.
polit./businessp.
43%
41%
37%
CA
5,278 articles
10,605 behavior vectors
Focus
3,876 articles
14,267 behavior vectors
historical fig.
criminal/victim
musican
42%
38%
38%
CA
Trending
5,684 articles
13,470 behavior vectors
media pers.
sportsperson
musician
27%
27%
19%
CA
Passing
28K [16K,51K]
11 [5,23]
7.7%
38K [21K,69K]
20 [9,41]
16.9%
26K [15K,45K]
10 [5,21]
10.5%
16K [10K,27K]
8 [3,18]
5.1%
ArtLen
#Edits
%HQA
#Edits - Number of edits
-1.0
0.5
-0.5
0.0
1.0
-1.0
0.5
-0.5
0.0
1.0
-1.0
0.5
-0.5
0.0
1.0
-1.0
0.5
-0.5
0.0
1.0
%HQA - Percentage of high quality articlesArtLen - Article length](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-90-320.jpg)
![94
Network
Effect
PaPern
Yahoo
Sites
change
their
popularity
in
the
same
way.
Ac>vity
(dwell
>me)
on
a
site
depends
more
on
the
site
itself,
but
there
are
some
nega>ve
dependencies.
Pattern
examples
41 patterns
Simple star-like
6 patterns
Complex star-like
1 pattern
Cluster-like
3.00 [3.00,4.00]
0.67 [0.00,0.89]
0 [0,0]
8.00 [7.00,18.00]
0.76 [0.56,0.84]
0 [0,0]
52
0.91
0.51
N
Recip
Trans
N - Number of nodes Recip - Reciprocity Trans - Transitivityservicefront page news providerleisure
(4) (5) (6)(1) (2) (3)](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-94-320.jpg)
![Online
news
98
Hyperlink
Performance
Number
of
Inline
Links
• <10
links
may
be
wasLng
an
opportunity
• 10-‐29
links
does
not
result
in
more
clicks
• >29
links
may
harm
the
user
experience
PosiCon
of
Inline
Links
• 30%
at
the
end,
16%
at
the
beginning,
46%
are
distributed
within
the
arLcle
text.
• Performance
of
links
located
at
the
beginning
of
the
text
is
very
low
(-‐28%)
• Best
performance
is
achieved
with
links
at
the
end
of
the
arLcle
text
(+35%)
Link popularity● Link performance
Position in article text
Linkpopularity [0.0,0.1[ [0.3,0.4[ [0.6,0.7[ [0.9,1.0]
10%
20%
30%
-0.2
0.0
0.2
Linkperformance
●●
●●
●●
●● ●● ●● ●● ●●
●●
●●
●●
●●
●● ●● ●● ●● ●● ●●
●●
●●
Number of inline links in article
Clicksperlink
0.0
0.2
0.4
0.6
[0,2] [9,11] [18,20] [27,29] [36,38]
Number of inline links in article
Numberofclicks
[0,2] [9,11] [18,20] [27,29] [36,38]
2.5
5.0
7.5](https://image.slidesharecdn.com/2015-02-fromsitetointer-siteuserengagementphddefense-150622162921-lva1-app6891/85/From-Site-to-Inter-site-User-Engagement-98-320.jpg)
Uploaded byJanette Lehmann
From Site to Inter-site User Engagement
The document discusses user engagement in websites, focusing on its measurement and the influence of online multitasking on user behavior. It emphasizes the need for new metrics to capture engagement better, accounting for inter-site relationships and multitasking patterns. Case studies, particularly involving Yahoo and Wikipedia, illustrate the complexities of measuring engagement both on and between sites.
From Site to Inter-site User Engagement
- 1. photo credit donsolo, CC BY-‐NC-‐SA 2.0 From Site to Inter-‐site User Engagement Jane;e Lehmann Barcelona, February 26, 2015 Advisors: Ricardo Baeza-‐Yates Co-‐Advisor: Mounia Lalmas
- 2. • User engagement is a quality of the user experience that emphasizes the posiLve aspects of interacLon with a website – in parLcular the fact of being capLvated by the website. • In-‐the-‐moment engagement Users stay on a website over a long Lme. • Long-‐term engagement Users come back frequently and over a long-‐term. IntroducLon 2 User Engagement DefiniLon Successful websites are not just used, they are engaged with.
- 3. User Engagement Measuring 3 IntroducLon Before we can design engaging websites, it is crucial that we are able to measure engagement. “If you can measure it, you can improve it.” Sir William Thomson Analysis/Planning Design Changes Measuring
- 4. Main Research Goals 4 IntroducLon Primary goal Can we define new engagement metrics that Measuring enhance our understanding of engagement? Secondary goal Can we idenLfy ways to influence engagement? Analysis/Planning Analysis/Planning Design Changes Measuring
- 5. IntroducLon 5 Analysis/ Planning Design Changes Measuring Online mulLtasking Inter-‐site engagement Site engagement Effect of providing off-‐site content Effect of hyperlinks
- 6. IntroducLon 6 Analysis/ Planning Design Changes Measuring Site engagement
- 7. Measuring Engagement InteracLon data 7 Site engagement Data Browsing events provided by Yahoo toolbar (client-‐side). Engagement Analysing the data using online behaviour metrics. Online session: Visit on Yahoo News
- 8. Site engagement 8 Measuring Engagement Online behaviour metrics K. Rodden, H. Hutchinson, X. Fu. Measuring the user experience on a large scale: User-‐centered metrics for web applicaHons. CHI, 2010. E. Peterson, J. Carrabis. Measuring the immeasurable: Visitor engagement. Web AnalyHcs DemysHfied, 2008. B. Haven, S. ViWal. Measuring engagement. Forrester Research, 2008. B. Weischedel and E. Huizingh. Website opHmizaHon with web metrics: A case study. Conference on Electronic commerce, 2006.
- 9. Site engagement 9 Measuring Engagement Online behaviour metrics Popularity #Users Number of users. #Visits Number of visits. #Clicks Number of clicks. AcCvity (within a visit) In-‐the-‐moment engagement PageViews Avg. number of page views per visit. DwellTime Avg. Lme on site per visit. Loyalty (across visits) Long-‐term engagement ReturnRate Number of Lmes a user visited the site. AcLveDays Number of days a user visited the site.
- 10. Site engagement 10 Measuring Engagement Differences in engagement ComScore, Alexa, GoogleAnalyHcs,… Shopping Users do not come frequently, but stay long Games Not many users, but they stay long News Users come frequently and stay long
- 11. Measuring Engagement Problem 11 Site engagement Isolated view: The metrics focus on engagement with a single site. RelaLonships to other sites are not considered.
- 12. IntroducLon 12 Analysis/ Planning Design Changes Measuring Online mulLtasking Site engagement
- 13. Online mulLtasking 13 MoCvaCon In-‐the-‐moment engagement ComScore, Alexa, GoogleAnalyHcs,… What web analyCcs think we do… 1 visit with 4 page views.
- 14. Online mulLtasking 14 MoCvaCon In-‐the-‐moment engagement ComScore, Alexa, GoogleAnalyHcs,… … and what we really do: 3 visit with on average 1.3 page views.
- 15. Online mulLtasking 15 MoCvaCon Online mulLtasking. Problem • Engagement metrics do not capture such behaviour. • Measuring acLvity on a site can lead to incorrect conclusions. Online mulCtasking Users visit several sites and switch between them during an online session, to perform related or totally unrelated tasks.
- 16. Research QuesCon 16 Online mulLtasking How can we measure engagement by accounLng for user mulLtasking behaviour? Analysis/Planning Design Changes Measuring
- 17. Extent of mulCtasking • 10.2 disLnct sites, 2 visits per site. Absence Cme • 50% of sites are revisited aker < 1min. InterrupHon of a task • There are revisits aker long breaks. Performing a new task Online mulLtasking 17 Online MulCtasking CharacterisLcs 0.00 0.25 0.50 0.75 1.00 10 2 10 1 10 0 10 1 10 2 Cumulativeprobability Absence time [min] news (finance) news (tech) social media mail 2.09 1.76 2.28 2.09 #Visits Absence time [min] 3.85 3.95 4.47 6.86 Absence time: Time between two visits
- 18. AcCvity paPerns • Four types: Decreasing, increasing, constant, complex. • Successive visits can belong together (i.e. to the same task). • Complex cases refer to no specific pa;ern or repeated pa;ern. Online mulLtasking 18 Online MulCtasking CharacterisLcs 1 2 3 4 ith visit on site 1 2 3 4 ith visit on site 1 2 3 4 ith visit on site 1 2 3 4 ith visit on site Proportionoftotal dwelltimeonsite 0.23 0.28 0.33 p-value = 0.09 m = -0.01 p-value = 0.07 m = -0.02 p-value = 0.79 m = 0.00 news (finance) sitesmail sites social media sites news (tech) sites decreasing attention increasing attention constant attention complex attention
- 19. Online mulLtasking 19 Measuring Engagement Online mulLtasking metrics Extent of mulCtasking SessSites Total number of sites accessed (#tasks). SessVisits Number of visits to site (site switching). Absence Cme CumAct Aggregates the dwell Lmes of the visits with accounLng for the Lme between the visits. AcCvity paPern A;Shik A;Range Describe the four cases of a;enLon shiks.
- 20. 20 CASE STUDY: MulCtasking PaPerns • ObjecCve: Analyse mulLtasking acLvity on sites; idenLfy mulLtasking pa;erns (clustering). • Metrics: Site DwellTime, MulLtasking metrics. • Data: July 2012, 2.5M users, 760 sites (shopping, news, search, etc.).
- 21. 21 Case Study: MulCtasking PaPerns Results No mulCtasking MulCtasking Quick Focused Rapid ConCnuous Recurring Checking weather Reading mails Following link to off-‐site content Purchasing an item Performing search Site DwellTime -‐-‐ ++ ++ ++ -‐-‐ Extent of mulCtasking -‐-‐ -‐-‐ ++ ++ ++ Absence Cme -‐-‐ ++ ++ ImplicaCons Provide interesHng off-‐ site content Shopping takes more than one visit Support user by finishing tasks quickly Online mulLtasking -- low value ++ high value
- 22. 22 Case Study: MulCtasking PaPerns Results No mulCtasking MulCtasking Quick Focused Rapid ConCnuous Recurring Checking weather Reading mails Following link to off-‐site content Purchasing an item Performing search Site DwellTime -‐-‐ ++ ++ ++ -‐-‐ Extent of mulCtasking -‐-‐ -‐-‐ ++ ++ ++ Absence Cme -‐-‐ ++ ++ AcCvity paPern Online mulLtasking De In CmCn 60% 0% De In CmCn 60% 0% De In CmCn 60% 0% Activity pattern: De – Decreasing In – Increasing Cn – Constant Cm - Complex -- low value ++ high value
- 23. 23 CASE STUDY: Wikipedia (on-‐site mulCtasking) • ObjecCve: Analyse reading acLvity on Wikipedia arLcles; idenLfy reading pa;erns (clustering). • Metrics: ArLcle DwellTime, #ArLcles in session, #Views to focal arLcle. • Data: Sep 2011 – Sep 2012, 500K users, 10K biography arLcles.
- 24. 24 Case Study: Wikipedia Approach Online mulLtasking Users’ reading behaviour on an Wikipedia arCcle ArLcle DwellTime How much Lme do users spend on an arLcle? #ArLcles in session Do users view also other arLcles during an online session? #Views on focal arLcle How oken do users view the arLcle?
- 25. 25 Case Study: Wikipedia Results No mulCtasking MulCtasking Focus ExploraCon Passing Focus is on focal arHcle Exploring topic around the focal arHcle Exploring topic and pass through the focal arHcle ArCcle DwellTime ++ -‐-‐ #ArCcles in session -‐-‐ ++ ++ #Views to focal arCcle ++ -‐-‐ ImplicaCons Content quality is important Links to addiHonal content are important ArHcles might need to be extended Online mulLtasking
- 26. On-‐site mulCtasking • MulLtasking between news arLcles of a provider. • MulLtasking between different tasks on a social media site (e.g. sharing, chapng, updaLng profile). • … Inter-‐site mulCtasking • MulLtasking when purchasing items online (comparing offers, product reviews, search, etc.) • … Online mulLtasking 26 Further Use Cases
- 27. Take Aways • AccounLng for mulLtasking leads to a be;er understanding on how users engage with sites. • Leaving a site does not necessarily entail less engagement, as users oken return to the site later on. Publications J. Lehmann, M. Lalmas, G. Dupret, and R. Baeza-Yates. Online multitasking and user engagement. CIKM 2013. J. Lehmann, C. Müller-Birn, D. Laniado, M. Lalmas, and A. Kaltenbrunner. Reader preferences and behavior on Wikipedia. HT 2014, Ted Nelson Newcomer Paper Award. J. Lehmann, C. Müller-Birn, D. Laniado, M. Lalmas, and A. Kaltenbrunner. What and how users read: Transforming reading behavior into valuable feedback for the Wikipedia community. Wikimania 2014. Online mulLtasking 27
- 28. IntroducLon 28 Analysis/ Planning Design Changes Measuring Online mulLtasking Inter-‐site engagement Site engagement
- 29. Inter-‐site engagement 29 MoCvaCon Large online service providers ComScore, Alexa, GoogleAnalyHcs,… Engagement Popularity: #Users, #Visits, … AcLvity: DwellTime, PageViews, … Loyalty: ReturnRate, AcLveDays, …
- 30. Inter-‐site engagement 30 MoCvaCon Large online service providers frontpage tv sports shopping autos search daLng jobs news shine groups maps local health answer weather games mail omg homes travel flickr finance Large online service providers (AOL, Google, Yahoo, etc.) have not only one site, but many sites. tumblr
- 31. Inter-‐site engagement 31 MoCvaCon Large online service providers frontpage tv sports shopping autos search daLng jobs news shine groups maps local health answer weather games mail omg homes travel flickr finance Providers want that users engage with many of their sites. tumblr
- 32. Inter-‐site engagement 32 MoCvaCon Online mulLtasking Problem • Engagement metrics do not measure engagement across sites. • How to adapt them is not obvious. Inter-‐site engagement Users visit sites that belong to the same network of sites.
- 33. Research QuesCon 33 Inter-‐site engagement How can we measure engagement by also considering the relaLonships between sites? Analysis/Planning Design Changes Measuring
- 34. Inter-‐site engagement 34 Traffic Networks Modelling We model sites (nodes) and user traffic (edges) between them as a network. Provider network G=(N, E, λ) N: Sites E: User traffic λ(e): Traffic volume (#Clicks) 4 clicks 2 clicks 50 clicks 10 clicks
- 35. Inter-‐site engagement 35 Measuring Engagement Inter-‐site engagement metrics: Network-‐level Traffic distribuCon Flow Extent to which users navigate between sites. Density1 Diversity of inter-‐site engagement. Reciprocity2 Homogeneity of traffic between sites. External traffic EntryDisparity Variability of in-‐going traffic to the network. ExitDisparity Variability of out-‐going traffic from the network. [1] S. Wasserman. Social network analysis: Methods and applicaHons, 1994. [2] T. SquarHni, F. Picciolo, F. RuzzenenH, and D. Garlaschelli. Reciprocity of weighted networks. Nature: ScienHfic reports, 2013.
- 36. Inter-‐site engagement 36 Measuring Engagement Inter-‐site engagement metrics: Node-‐level Traffic distribuCon PageRank1 Probability that a user will visit the site. Downstream Probability that a user will conLnue browsing to other sites. External traffic EntryProb Probability that a user enters the network in this site. ExitProb Probability that a user leaves the network in this site. [1] L. Page, S. Brin, R. Motwani, T. Winograd. The pagerank citaHon ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999.
- 37. 37 CASE STUDY: Yahoo Provider Networks • ObjecCve: Compare networks; characterise the sites in a network. • Metrics: Network DwellTime, Site DwellTime, Inter-‐ site engagement metrics. • Data: February 2014, 3.2M clicks/network, 4 country-‐based networks, 31 sites per network.
- 38. 38 Case Study: Yahoo Comparing provider networks Network 1 Network 2 Network 3 Network 4 High engaging Users engage quickly with many sites Users engage to a subset of sites Low engaging Network DwellTime ++ -‐-‐ ++ -‐-‐ Traffic DistribuCon ++ ++ Flow ++ Density -‐-‐ -‐-‐ Entry Disparity ++ -‐-‐ ++ ImplicaCons The network is performing well. This should be looked into. MoHvate users to visit other sites. This should be looked into. Inter-‐site engagement -- low value ++ high value
- 39. 39 Case Study: Yahoo Sites within a provider network Traffic Hub Supporter Focused Engagement Shared Engagement Search, front pages Support, services Leisure, support News, leisure Site DwellTime -‐-‐ -‐-‐ ++ ++ Traffic DistribuCon ++ -‐-‐ -‐-‐ ++ Entry Probability ++ -‐-‐ ++ -‐-‐ ImplicaCons The sites forward traffic to other sites. Users visit sites for specific needs and support. MoHvate users to visit other sites. The sites are performing well. Inter-‐site engagement -- low value ++ high value
- 40. Comparing networks • Device, Lme, upstream traffic, user. • SimulaLons (effect of adding/removing sites). • … Network types • Network of pages (e.g. compare language-‐based Wikipedia networks) • Network of sites from different providers (e.g. shopping sites, news providers) • … Inter-‐site engagement 40 Further Use Cases
- 41. Take Aways • Inter-‐site engagement allows for a more comprehensive look at user engagement by also considering the relaLonships between sites. • Deeply engaged users do not only engage with one site, but with many sites in a network. Publications J. Lehmann, M. Lalmas, and R. Baeza- Yates. Measuring Inter-Site Engagement. Handbook of Statistics, Elsevier, 2015. To appear. J. Lehmann, M. Lalmas, R. Baeza-Yates, and E. Yom-Tov. Networked User Engagement. ACM Workshop on User engagement optimization at CIKM, 2013. J. Lehmann, M. Lalmas, and R. Baeza- Yates. Temporal Variations in Networked User Engagement. TNETS Satellite at ECCS, 2013. Some of the metrics were employed to characterise online news reading across news sites: J. Lehmann, C. Castillo, M. Lalmas, and R. Baeza-Yates. Story-Focused Reading in Online News. Submitted for publication. Inter-‐site engagement 41
- 42. IntroducLon 42 Analysis/ Planning Design Changes Measuring Online mulLtasking Inter-‐site engagement Site engagement Effect of providing off-‐site content
- 43. 43 CASE STUDY: Online News • Hypothesis: It may be beneficial (long-‐term) to enLce users to leave a site by offering interesLng off-‐site content. • Data: October 2013, 57K users, 50 news sites, 26K news arLcles.
- 44. Types of reading sessions No click Did not follow a related link. Off-‐site click Followed a related link to content on another site. Effect on engagement Short-‐term Dwell Lme per reading session. Long-‐term Probability that user starts next reading session within the next 12h. 44 Case Study: Online News Related off-‐site content Approach Effect of providing off-‐site content
- 45. Providing links to related off-‐site content has a no short-‐term effect, but a posiCve long-‐term effect. 45 Case Study: Online News Results Effect of providing off-‐site content News provider Dwelltimepersession News provider p(absence12h) No Click Off-site click
- 46. IntroducLon 46 Analysis/ Planning Design Changes Measuring Online mulLtasking Inter-‐site engagement Site engagement Effect of providing off-‐site content Effect of hyperlinks
- 47. 47 CASE STUDY: Yahoo Provider Network • Hypothesis: We can use hyperlinks to influence inter-‐site engagement in a provider network. • Data: February 2014, 235M clicks, Yahoo US network, 73 sites.
- 48. Hyperlink vs. traffic network On-‐site Links/Traffic to pages within the same site. Inter-‐site Links/Traffic to pages to other sites in the network. External Links/Traffic to somewhere else on the Web. 48 Case Study: Yahoo Approach frontpage sports search news shine groups answer weather mail omg homes flickr Effect of hyperlinks
- 49. Hyperlinks can be used to influence site and inter-‐site engagement in a provider network. However, both types of engagement influence each other. 49 Case Study: Yahoo Results Effect of hyperlinks Traffic On-site Inter-site External Hyperlinks On-site Inter-site External 0.54 -0.40 - -0.45 0.50 - -0.38 - 0.39
- 50. IntroducLon 50 Analysis/ Planning Design Changes Measuring Online mulLtasking Inter-‐site engagement Site engagement Effect of providing off-‐site content Effect of hyperlinks
- 51. Two new perspecHves for measuring engagement which consider the relaLonships between sites. Online mulCtasking Accounts for user mulLtasking behaviour. Inter-‐site engagement Accounts for the traffic between sites. ContribuLons and future work 51 Main ContribuCons Measuring engagement Analysis/ Planning Design Changes Measuring
- 52. AccounLng for the new perspecLves when influencing engagement. Online news Providing related off-‐site content influences long-‐term engagement. Provider network Hyperlinks affect site and inter-‐site engagement, but both influence each other. ContribuLons and future work 52 Main ContribuCons Analysis/Planning Analysis/ Planning Design Changes Measuring
- 53. Wikipedia Providing informaLon about readers’ engagement to the editor community. Yahoo Using inter-‐site engagement metrics to make informed decisions about design changes (hyperlinks). Spiegel Online Measuring and improving engagement by providing interesLng off-‐site content. ContribuLons and future work 53 What next? Ongoing and future work Analysis/ Planning Design Changes Measuring
- 54. photo credit donsolo, CC BY-‐NC-‐SA 2.0 Thank you! Jane;e Lehmann Barcelona, February 26, 2015 lehmannj@acm.org Acknowledgements Ricardo Baeza-‐Yates Mounia Lalmas Claudia Müller-‐Birn Carlos CasLllo David Laniado Andreas Kaltenbrunner Elad Yom-‐Tov Georges Dupret Guy Shaked Fabrizio Silvestri Gabriele Tolomei Ethan Zuckerman John Agapiou Andy Haines Diego Sáez-‐Trumper Hemant Purohit Noora Al Emadi Mohammed El-‐Haddad Nasir Khan
- 55. • Mounia Lalmasand Janette Lehmann. “Models of User Engagement”. In H. L. O’Brien and M. Lalmas (Eds.), Why Engagement Matters: Cross-disciplinary Perspectives and Innovations on User Engagement with Digital Media. Springer, 2015, in progress. • Janette Lehmann, Mounia Lalmas, Elad Yom-Tov, and Georges Dupret. “Models of user engagement.” International Conference on User Modeling, Adaptation, and Personalization (UMAP 2012), pp. 164-175, Montreal, Canada, July, 2012. • Janette Lehmann, Mounia Lalmas, Georges Dupret, and Ricardo Baeza-Yates. “Online multitasking and user engagement.” ACM International Conference on Information and Knowledge Management (CIKM 2013), pp. 519-528, San Francisco, United States, October, 2013. • Janette Lehmann, Mounia Lalmas, and Ricardo Baeza-Yates. “Measuring Inter-Site Engagement.”. In V. Govindaraju, V. V. Raghavan, and C. R. Rao (Eds.), Handbook of Statistics, Elsevier, 2015. • Janette Lehmann, Mounia Lalmas, Ricardo Baeza-Yates, and Elad Yom-Tov. “Networked User Engagement.”, ACM Workshop on User engagement optimization at CIKM, pp. 7-10, San Francisco, United States, October, 2013. • Janette Lehmann, Mounia Lalmas, and Ricardo Baeza-Yates. “Temporal Variations in Networked User Engagement.”, TNETS Satellite at European Conference on Complex Systems (ECCS), Barcelona, Spain, September, 2013. • Mounia Lalmas, Janette Lehmann, Guy Shaked, Fabrizio Silvestri, and Gabriele Tolomei. “Measuring Post-click User Experience with Mobile Native Advertising on Streams.”, submitted for publication. • Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. “Reader preferences and behavior on Wikipedia.”, ACM International Conference on Hypertext and Social Media (HT 2014), pp. 88-97, Santiago, Chile, September, 2014, Ted Nelson Newcomer Paper Award. • Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. “What and how users read: Transforming reading behavior into valuable feedback for the Wikipedia community.”, Presentation at Wikimania, London, UK, August, 2014. • Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ricardo Baeza-Yates. “Story-Focused Reading in Online News.”, submitted for publication. • Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ethan Zuckerman. “Transient News Crowds in Social Media.” International AAAI Conference on Weblogs and Social Media (ICWSM 2013), Boston, USA, July, 2013. • Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ethan Zuckerman. “Finding News Curators in Twitter.” ACM International Conference on World Wide Web Companion (WWW 2013 Companion), 863-870, Rio de Janeiro, Brazil, May, 2013. 55 PublicaCons
- 56. User engagement • MouniaLalmas, Heather L O’Brien, and Elad Yom-Tov. Measuring user engagement. Synthesis Lectures on Sample Series #1. Morgan and cLaypool publishers, 2014. • Heather L O’Brien and Elaine G Toms. What is user engagement? a conceptual framework for defining user engagement with technology. American Society for Information Science and Technology (ASIS&T), 59(6):938–955, 2008. • Simon Attfield, Gabriella Kazai, Mounia Lalmas, and Benjamin Piwowarski. Towards a science of user engagement (position paper). In Proc. Workshop on User Modelling for Web Applications, WSDM, 2011. • Kerry Rodden, Hilary Hutchinson, and Xin Fu. Measuring the user experience on a large scale: user-centered metrics for web applications. In Proc. Conference on Human Factors in Computing Systems, CHI, pages 2395–2398. ACM, 2010. Online behaviour metrics • Brian Haven and Suresh Vittal. Measuring engagement. Forrester Research, 2008. • Eric T Peterson and Joseph Carrabis. Measuring the immeasurable: Visitor engagement. Web Analytics Demystified, 2008. • Kerry Rodden, Hilary Hutchinson, and Xin Fu. Measuring the user experience on a large scale: user-centered metrics for web applications. In Proc. Conference on Human Factors in Computing Systems, CHI, pages 2395–2398. ACM, 2010. • Georges Dupret and Mounia Lalmas. Absence time and user engagement: evaluating ranking functions. In Proc. Conference on Web Search and Data Mining, WSDM, pages 173–182. ACM, 2013. • Randolph E Bucklin and Catarina Sismeiro. A model of web site browsing behavior estimated on clickstream data. Journal of Marketing Research, 40(3):249–267, 2003. • Birgit Weischedel and Eelko KRE Huizingh. Website optimization with web metrics: a case study. In Proc. Conference on Electronic commerce: The new e-commerce: innovations for conquering current barriers, obstacles and limitations to conducting successful business on the internet, pages 463–470. ACM, 2006. • Peifeng Yin, Ping Luo, Wang-Chien Lee, and Min Wang. Silence is also evidence: interpreting dwell time for recommendation from psychological perspective. In Proc. Conference on Knowledge Discovery and Data Mining, SIGKDD, pages 989–997. ACM, 2013. 56 Selected References
- 57. Online multitasking • QingWang and Huiyou Chang. Multitasking bar: prototype and evaluation of introducing the task concept into a browser. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 103–112. ACM, 2010. • Hartmut Obendorf, Harald Weinreich, Eelco Herder, and Matthias Mayer. Web page revisitation revisited: implications of a long-term click-stream study of browser usage. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 597–606. ACM, 2007. • Jeff Huang and Ryen W White. Parallel browsing behavior on the web. In Proc. Conference on Hypertext and Hypermedia, HT, pages 13–18. ACM, 2010. • Patrick Dubroy and Ravin Balakrishnan. A study of tabbed browsing among mozilla firefox users. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 673–682. ACM, 2010. Inter-site engagement • Mark EJ Newman. The structure and function of complex networks. SIAM review, 45(2):167–256, 2003. 76, 77, 165 • Anna Chmiel, Kamila Kowalska, and Janusz A Hołyst. Scaling of human behavior during portal browsing. • Mark R Meiss, Filippo Menczer, Santo Fortunato, Alessandro Flammini, and Alessandro Vespignani. Ranking web sites with real user traffic. In Proc. Conference on Web Search and Data Mining, WSDM, pages 65–76. ACM, 2008. • Young-Hoon Park and Peter S Fader. Modeling browsing behavior at multiple websites. Marketing Science, 23(3):280–303, 2004. • Qiqi Jiang, Chuan-Hoo Tan, and Kwok-Kee Wei. Cross-website navigation behavior and purchase commitment: A pluralistic field research. In Proc. Pacific Asia Conference on Information Systems, PACIS, 2012. • Kevin Koidl, Owen Conlan, and Vincent Wade. Cross-site personalization: assisting users in addressing information needs that span independently hosted websites. In Proc. Conference on Hypertext and Hypermedia, HT, pages 66–76. ACM, 2014. • The PEW Research Center. Understanding the participatory news consumer. http://www.pewinternet.org/~/media/Files/ Reports/ 2010/PIP_Understanding_the_Participatory_News_Consumer. pdf, 2010. • Jason MT Roos. Hyper-Media Search and Consumption. PhD thesis, Duke University, 2012. 57 Selected References
- 58. Link economy • JosephTurow and Lokman Tsui. The hyperlinked society. The University of Michigan Press, 2008. • Juliette De Maeyer. Hyperlinks and journalism: where do they connect? In Proc. Future of Journalism Conference, 2011. • Chrysanthos Dellarocas, Zsolt Katona, and William Rand. Media, aggregators, and the link economy: Strategic hyperlink formation in content networks. Management Science, 59(10):2360–2379, 2013. • Jason MT Roos. Hyper-Media Search and Consumption. PhD thesis, Duke University, 2012. • Chrysanthos Dellarocas, Zsolt Katona, and William Rand. Media, aggregators, and the link economy: Strategic hyperlink formation in content networks. Management Science, 59(10):2360–2379, 2013. • Hakan Ceylan, Ioannis Arapakis, Pinar Donmez, and Mounia Lalmas. Automatically embedding newsworthy links to articles. In Proc. Conference on Information and Knowledge Management, CIKM, pages 1502–1506. ACM, 2012. Recommendation • Richard McCreadie, Craig Macdonald, and Iadh Ounis. News vertical search: when and what to display to users. In Proc. Conference on Research and Development in Information Retrieval, SIGIR, pages 253–262. ACM, 2013. • Samuel Ieong, Mohammad Mahdian, and Sergei Vassilvitskii. Advertising in a stream. In Proc. Conference on World Wide Web, WWW, pages 29–38. ACM, 2014. • Eric Sodomka, Sébastien Lahaie, and Dustin Hillard. A predictive model for advertiser value-per-click in sponsored search. In Proc. Conference on Information and Knowledge Management, CIKM, pages 1179–1190. ACM, 2013. • Narongsak Thongpapanl and Abdul Rehman Ashraf. Enhancing online performance through website content and personalization. Journal of Computer Information Systems, 52(1):3, 2011. • Jian Wang and Yi Zhang. Utilizing marginal net utility for recommendation in e-commerce. In Proc. Conference on Research and Development in Information Retrieval, SIGIR, pages 1003–1012. ACM, 2011. • Joshua Porter. Designing for the social web. Peachpit Press, 2010. 58 Selected References
- 59.
- 60. • “In a world full of choices where the fleeCng aPenCon of the user becomes a prime resource, it is essenLal that [...] providers do not just design [websites] but that they design engaging experiences.” [A}ield]. • In addiLon to uLlitarian factors, such as usability and usefulness, we must consider other factors of interacLng with websites, such as fun, fulfillment, play, and user engagement. Successful websites are not just used, they are engaged with. • In order to design engaging websites, it is crucial to understand what user engagement is and how to measure it. IntroducLon 60 MoCvaCon Why is it important to engage users?
- 61. Methodology InteracLon data, online sessions and site visits. 61 IntroducLon t0 t1 t2 t3 t4 t5 t6 t7 session end session start time Online session Browsing activity on Wikipedia https://ie-mg42.mail.yahoo.com http://en.wikipedia.org/wiki/Freddie˙Mercury http://www.bbc.com/news/uk-29149115 http://www.bbc.com/news/uk-england-nottinghamshire-29643802 http://en.wikipedia.org/wiki/Star Wars http://en.wikipedia.org/wiki/Yoda http://en.wikipedia.org/wiki/Albert˙Einstein https://www.facebook.com/janette.lehmann.5 t0 t1 t2 t3 t4 t5 t6 t7 bc0 bc0 bc0 bc0 bc0 bc0 bc0 bc0 BCookie Timestamp URL - - - http://www.bbc.com/news/uk-29149115 - http://en.wikipedia.org/wiki/Star Wars http://en.wikipedia.org/wiki/Yoda - ReferrerURL Interaction data Page view on Wikipedia Page view on other site
- 62. IntroducLon 62 Thesis structure Metrics that account for site popularity, activity and loyalty Advertising Chapter 7 Site engagement How users experience ads on desktop and mobile devices? Does ad quality affect the engagement with the publisher? How can we identify high quality ads? Site engagement Chapter 4 Multitasking Chapter 5 Inter-site engagement Chapter 6 Metrics that account for traffic between sites Metrics that account for user multitasking behaviour (III+IV)Applications(II)Fund. Wikipedia Chapter 8 Site engagement and multitasking How users read articles in Wikipedia? Does the activity of editors align with the engagement of readers? How can readers be valuable for editors? Yahoo Chapter 9 Inter-site engagement How users engage with a provider network of sites? Does the hyperlink structure affect site and inter-site engagement? Online news Chapter 10+11 Inter-site engagement How users read stories across news providers? Do hyperlinks to related content influence provider engagement? How can we automatically detect related content? Characterising user engagement Comparing site characteristics and user engagement Applications to impact user engagement
- 63.
- 64. 0-1 1-0.5 0.5 Kendall’stau with p-value < 0.05 ('-' insignificant correlations) Site engagement 64 EvaluaCon CorrelaLons between engagement metrics. High correlaCons within metric groups. Low correlaCons between metric groups. [POP]#Users [POP]#Visits [POP]#Clicks [ACT]PageViewsV [ACT]DwellTimeV [LOY]ActiveDays [LOY]ReturnRate #Users [POP] 0.82 0.75 - - 0.43 0.34 #Visits [POP] 0.82 0.85 - - 0.60 0.52 #Clicks [POP] 0.75 0.85 0.16 0.18 0.59 0.51 PageViewsV [ACT] - - 0.16 0.33 - - DwellTimeV [ACT] - - 0.18 0.33 - - ActiveDays [LOY] 0.43 0.60 0.59 - - 0.79 ReturnRate [LOY] 0.34 0.52 0.51 - - 0.79 0.69
- 65. Site engagement 65 PaPerns of Site Engagement Engagement depends on the site at hand. Games Not many users, but they stay long Search Users come frequently, but do not stay long Social media Users come frequently and stay long Shopping Users do not come frequently, but stay long News Users come frequently and stay long Service Users do not come frequently, but stay long
- 66.
- 67. Online mulLtasking 67 MoCvaCon Users switch between sites, to perform related or totally unrelated tasks. Switching between tasks (sites) “…within-‐session page revisits represent the most common form of revisitaLon, covering 73,54% of all revisits.” [Herder] Performing tasks (sites) in parallel using browser tabs “Most of our parLcipants switched tabs more oken than they used the back bu;on.” [Dubroy] [Herder] E. Herder. CharacterizaHons of user web revisit behavior. WWW Workshop ABIS, 2005. [Dubroy] P. Dubroy, R. Balakrishnan. A study of tabbed browsing among mozilla firefox users. SIGCHI, 2010.
- 68. Online mulLtasking 68 Data Dataset and site categories. Cat. Subcat. %Sites Description news 22.1% news 5.79% news (soc.) 5.13% society news (sport) 2.63% news (enter.) 2.24% music, movies, tv, etc. news 1.97% news (life) 1.58% health, housing, etc. news (tech) 1.58% technology news (weather) 1.18% service 15.5% service 7.63% translators, banks, etc. mail 3.95% maps 3.03% organisation 0.92% bookmarks, calendar, etc. search 15.3% search 12.63% search (special) 1.58% search for lyrics, jobs, etc. directory 1.05% sharing 9.6% blogging 3.55% knowledge 3.55% collaborative creation and collection of content sharing 2.50% sharing of videos, etc. navi 9.3% front page 6.58% front page (p.) 1.84% personalised front pages sitemap 0.92% leisure 8.7% adult 2.76% games 1.97% social media 1.97% dating 1.05% entertainment 0.92% sites with music, tv, etc.support 8.7% support 1.58% sites that provide products and support for them download 7.11% downloading software shopping 7.9% shopping 4.34% auctions 2.11% comparison 1.45% sites to compare prices of products settings 2.9% login 1.71% site settings 1.18% pr e setting, site personalisation InteracCon data • July 2012 • 2.5M users • 785M page views NavigaCon model • We defined a new navigaLon model (see paper for details) Site categories • 760 sites from 70 countries/ regions • 11 categories • 33 subcategories
- 69. Online mulLtasking 69 MulCtasking Metrics CumAct accounts for the acLvity between site visits. CumulaCve acCvity The metric is defined as follows: InterpretaCon High CumAct à High engagement If users return aker short Lme, they return to conLnue with same task. If users return aker longer Lme, they return to perform a new task – a sign of loyalty. CumActk = log10 (v1 + ivi k •vi i=2 n ∑ ) Browsing acLvity during the ith visit Browsing acLvity between the (i-‐1)th and ith visit Rescaling factor for ivi k = 3 vi ivi 1 4 3 10 3 CumAct = log10 (3+13 •4+103 •3) = 3.48 Site visit
- 70. Online mulLtasking 70 MulCtasking Metrics AWRange and AWShik describe changes between the visits. APenCon shie and range The metrics is defined as follows: InterpretaCon AWShik models the shik of a;enLon, and AWRange models the fluctuaLons in the browsing acLvity. AttShiftn = invn − minInvn | maxInvn |− | minInvn | AttRangen = σ (Vn ) µ(Vn ) Variance in the visit acLvity Average of the visit acLvity Number of visits in session ModificaLon of the “Inversion number” n = 4 σ µi Inv 0 >0 -‐1 constant decreasing 0 constant complex +1 constant increasing AWenHon range AWenHon shik
- 71. 0-1 1-0.5 0.5 Spearman’srho with p-value < 0.05 ('-' insignificant correlations) Online mulLtasking 71 EvaluaCon CorrelaLons between mulLtasking and acLvity metrics. [MT]SessVisits [MT]SessSites [MT]CumAct [MT]AttShift [MT]AttRange [ACT]DwellTimeS SessSites [MT] 0.42 CumAct [MT] 0.41 - AttShift [MT] 0.09 - - AttRange [MT] - - -0.38 0.27 DwellTimeS [ACT] 0.20 0.24 0.12 0.32 0.08 DwellTimeV [ACT] -0.40 - - 0.14 - 0.50 No or only weak correlaCons between the metrics. All metrics convey different aspects about users’ online behaviour.
- 72. Online mulLtasking 72 MulCtasking PaPerns Cluster centers, site categories and acLvity pa;erns. CategoriesMultitasking DwellTimeV CumAct SessVisitsDwellTimeS sitemap site settings news (wheather) download +75% +73% +69% +67% PD 139 sites Quick task Continuous multitasking SessSitesBars from left to right: 111 sites auctions shopping adult dating +79% +71% +71% +62% PD -1.0 1.0 0.0 -1.0 1.0 0.0 PD - Probability difference Activity Activity pattern: De - Decreasing In - Increasing Cn - Constant Cm - Complex De In CmCn De In CmCn 60% 0% 147 sites Recurring task search front page (p.) front page organisation +77% +62% +57% +24% PD -1.0 1.0 0.0 De In CmCn 0.6 0.0 137 sites Focused task news (tech) news (life) support mail +66% +66% +65% +64% PD -1.0 1.0 0.0 -1.0 1.0 0.0 De In CmCn 60% 0% 60% 0% 142 sites Rapid multitasking news (enter.) knowledge comparison service +64% +63% +62% +59% PD -1.0 1.0 0.0 -1.0 1.0 0.0 De In CmCn 60% 0% Single-task-oriented browsing Multitask-oriented browsing
- 73.
- 74. Inter-‐site engagement 74 Data Dataset, network and site categories. InteracCon data • August 2013 to July 2014 • 53M sessions Provider network G=(N, E, λ) N: 155 Yahoo sites from five countries E: User traffic λ(e): Traffic volume (#Clicks) Site categories • 155 sites from 5 countries • 5 categories Cat. %Sites Description 35% 19% 13% 23% 10% news service leisure provider front page mail, calendar, etc. social media, games, etc. account settings, help, etc. front pages, site maps servicefront page news providerleisure
- 75. Inter-‐site engagement 75 Inter-‐site Engagement Metrics Flow accounts for the extent users navigate between sites. Traffic Flow The metric is defined as follows: InterpretaCon High Flow à High inter-‐site engagement Users navigate oken between the sites of the network. Flow = wi, ji, j∑ vii∑ #Clicks between node i and j #Visits on node i wi, j vi Flow = 30/60 = 0.5 10 5 20 20 20 10 5 1 1 20 20 20 1 1 Flow = 4/60 = 0.07
- 76. Inter-‐site engagement 76 Inter-‐site Engagement Metrics Density describes the connecLvity of the network. Density We use the density measure of [Wasserman]: InterpretaCon High Density à High inter-‐site engagement Users navigate between many different sites (inter-‐site engagement is highly diverse). [Wasserman] S. Wasserman. Social network analysis: Methods and applicaHons, 1994. Density = # Edges # Possible_ Edges Density = 4/6 = 0.7 Flow = 2/6 = 0.3
- 77. Inter-‐site engagement 77 Inter-‐site Engagement Metrics Reciprocity measures the homogeneity of traffic between two sites. Reciprocity We use the reciprocity measure of [SquarLni]: InterpretaCon High Reciprocity à High inter-‐site engagement Users navigate between two sites in both direcLons (inter-‐site engagement is highly homogenious). [SquarHni] T. SquarHni, F. Picciolo, F. RuzzenenH, and D. Garlaschelli. Reciprocity of weighted networks. Nature: ScienHfic reports, 2013. #Clicks between node i and j wi, j RP = min[wi, j,wj,i ] i<j∑ wi, ji≠j∑ 1 10 5 20 1 Reciprocity = 15/50 = 0.3 Reciprocity = 2/37 = 0.05 10 10 5 20 5
- 78. Inter-‐site engagement 78 Inter-‐site Engagement Metrics Entry/ExitDisp measures how the traffic to/from the network is distributed over the sites. Entry disparity and exit disparity We use the group degree measure of [Freeman] and adapt it as follows: InterpretaCon High Entry/ExitDisp à Low inter-‐site engagement The network is more vulnerable to outages, because only few sites are used to enter (leave) the network. EntryDisp = (gin max − gin i ) i∑ | N |• gin ii∑ [Freeman] L. C Freeman. Centrality in social networks conceptual clarificaHon. Social networks, 1979. Number of visits that started at node ni (user entered the network) Maximum value of gin Number of nodes | N | gi in gin max EntryDisp = 20/3 40 = 0.17 20 10 10 40 5 5 EntryDisp = 70/3 50 = 0.47
- 79. Inter-‐site engagement 79 EvaluaCon: Network-‐level CorrelaLons between inter-‐site and network engagement metrics. [IS]Density [IS]Reciprocity [IS]EntryDisparity [IS]ExitDisparity [POP]#Sessions [ACT]DwellTimeS [ACT]#Sites Flow [IS] - 0.15 0.23 0.30 - 0.35 0.65 Density [IS] 0.48 -0.61 -0.60 0.92 -0.45 -0.25 Reciprocity [IS] -0.38 -0.32 0.42 - 0.25 EntryDisparity [IS] 0.84 -0.54 0.33 - ExitDisparity [IS] -0.55 0.38 0.20 0-1 1-0.5 0.5 Spearman’s rho with p-value < 0.01 ('-' insignificant correlations) Density and #Sessions The more users are visiCng the network, the more diverse is the inter-‐ site engagement. Entry-‐ and ExitDisparity Volume of in-‐ and out-‐ going traffic of the nodes depend on each other. Flow and #Sites The more sites are visited during a session, the higher the flow of traffic.
- 80. Inter-‐site engagement 80 EvaluaCon: Node-‐level CorrelaLons between inter-‐site and site engagement metrics. [IS]Downstream [IS]EntryProb [IS]ExitProb [POP]#Sessions [ACT]DwellTimeS [MT]#Visits [MT]CumAct PageRank [IS] 0.30 -0.08 -0.10 0.85 0.06 0.08 0.31 Downstream [IS] -0.27 -0.22 0.17 0.04 0.02 -0.02 EntryProb [IS] 0.79 0.12 -0.19 0.13 0.35 ExitProb [IS] 0.08 -0.18 0.18 0.32 0-1 1-0.5 0.5 Spearman’s rho with p-value < 0.01 ('-' insignificant correlations) PageRank and #Sessions Popular sites in the provider network, are also visited frequently when browsing through the network. Entry-‐ and ExitProb Nodes that are used to enter the network are also frequently used to exit the network.
- 81. Inter-‐site engagement 81 Comparing Provider Networks Country2 Country1 Country3 Country4 Country5 Flow Reciprocity EntryDisparityDensity DwellTimeBars from left to right: -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0
- 82. Inter-‐site engagement 82 PaPerns of Inter-‐site Engagement CategoriesEngagement PageRank EntryProb DwellTimeDownstream PD - Probability difference 46 sites Focused eng. front page service provider leisure news +80% +63% -100% -100% -100% PD 23 sites Traffic hub 46 sites Supporter 40 sites Shared eng. CumActBars from left to right: -1.0 1.0 0.0 provider service news front page leisure +31% +19% -2% -10% -100% PD leisure provider service news front page +67% +48% -21% -94% -100% PD news leisure provider front page service +63% -61% -100% -100% -100% PD -1.0 1.0 0.0 -1.0 1.0 0.0 -1.0 1.0 0.0
- 83.
- 84. NaLve AdverLsing 84 Effect on User Engagement 0% 200% 400% 600% short ad clicks long ad clicks adclickdifference short ad clicks long ad clicks clicksperdaydifference PosiLve experience has a strong effect on users clicking on ads again, and a small effect on user engagement with the stream.
- 85. NaLve AdverLsing 85 Mobile vs. Desktop Ad post-‐click experience between mobile and desktop differs. For dwell Lme we obtain rho = 0.50; this value is even smaller for bounce rate with rho = 0.23. 0.00 0.05 0.10 0.15 dwell time difference p(dwelltimedifference) higher on mobilehigher on desktop 0.00 0.05 0.10 0.15 bounce rate difference p(bounceratedifference) higher on mobilehigher on desktop
- 86. NaLve AdverLsing 86 Mobile OpCmised Landing Pages Dwell Cme: The distribuLon is very similar for both groups. Bounce rate: Decreases by 6.9% (median decreases by 30.4%) for Opt landing pages but increases by 13.4% (median decreases by 11.5%) for Npt landing pages. not mobile optimized mobile optimized 0.0 0.1 0.2 0.3 dwell time difference p(dwelltimedifference) higher on mobilehigher on desktop mobile opt. not mobile opt. 0.0 0.1 0.2 bounce rate difference p(bounceratedifference) higher on mobilehigher on desktop mobile opt. not mobile opt.
- 87.
- 88. Wikipedia 88 Wikipedia Research Literature review by Okoli et al.: The people’s encyclopedia under the gaze of the sages: A systemaLc review of scholarly research on wikipedia.
- 89. Wikipedia 89 Reading Preferences Popularitylow high ArticleLengthshortlong borderline casesII I III IV Jeanne Tsai Douglas Adams Luis Palomino Anne Stears Peter Ehrlich Alec Mango Stephen D. Lovejoy 1st Dalai Lama Dexter Jackson (safety) Katie Green Brittany Borman Anthony Anenih Ronnie Bird Jan Anderson (scientist) Fitch Robertson Sean Bennett For 4.2% (group IV) of the articles editing activity is low, but reading activity is high.!
- 90. Wikipedia 90 Reading PaPerns Article topic Reading behavior ArticleViewsa SessionArticlesa Popularitya ReadingTimea CA - Percentage in topic 4,826 articles 11,579 behavior vectors sportsperson musician media pers. 28% 26% 23% CA Exploration artist/writer historical fig. polit./businessp. 43% 41% 37% CA 5,278 articles 10,605 behavior vectors Focus 3,876 articles 14,267 behavior vectors historical fig. criminal/victim musican 42% 38% 38% CA Trending 5,684 articles 13,470 behavior vectors media pers. sportsperson musician 27% 27% 19% CA Passing 28K [16K,51K] 11 [5,23] 7.7% 38K [21K,69K] 20 [9,41] 16.9% 26K [15K,45K] 10 [5,21] 10.5% 16K [10K,27K] 8 [3,18] 5.1% ArtLen #Edits %HQA #Edits - Number of edits -1.0 0.5 -0.5 0.0 1.0 -1.0 0.5 -0.5 0.0 1.0 -1.0 0.5 -0.5 0.0 1.0 -1.0 0.5 -0.5 0.0 1.0 %HQA - Percentage of high quality articlesArtLen - Article length
- 91. Wikipedia 91 Reading PaPerns over Time Stability • 30% of the arLcles are popular in 1 month • 10% are popular over the whole 13-‐months • Almost all arLcles have one reading pa;ern half of their life Lme TransiCons • TransiLons are temporary – arLcles move temporarily to another cluster • High reciprocity – similar number of transiLons in both direcLons • “Focus” cluster is isolated -‐ ArLcles in that cluster are the most stable ones • Strong connecLon between the “Passing”, “ExploraLon”, and “Trending” clusters – many arLcles adopt all three pa;erns
- 92.
- 93. 93 Upstream Traffic TeleportaCon Social media / News Search / Ext-‐Yahoo Users engage (quickly) to many sites. Users conHnue with same acHvity inside the provider network. Users visit site they are interested in, perform a quick task, and leave. Network DwellTime -‐-‐ ++ -‐-‐ Traffic DistribuCon ++ -‐-‐ -‐-‐ Entry Disparity -‐-‐ Yahoo Users engage differently depending on where they are coming from.
- 94. 94 Network Effect PaPern Yahoo Sites change their popularity in the same way. Ac>vity (dwell >me) on a site depends more on the site itself, but there are some nega>ve dependencies. Pattern examples 41 patterns Simple star-like 6 patterns Complex star-like 1 pattern Cluster-like 3.00 [3.00,4.00] 0.67 [0.00,0.89] 0 [0,0] 8.00 [7.00,18.00] 0.76 [0.56,0.84] 0 [0,0] 52 0.91 0.51 N Recip Trans N - Number of nodes Recip - Reciprocity Trans - Transitivityservicefront page news providerleisure (4) (5) (6)(1) (2) (3)
- 95. 95 Hyperlink Performance Yahoo 0% 25% 50% 75% 100% Onsitelinks front page providerservice news leisure Intersitelinks front page providerservice news leisure 0% 20% 40% 60% 80% Externallinks front page providerservice news leisure 20% 40% 60% (a) PageRank and downstream. Traffic PageRank Downstream Hyperlinks PageRank 0.54 - Downstream - - (b) On-site, inter-site, and external. Traffic On-site Inter-site External Hyperlinks On-site 0.54 -0.45 -0.38 Inter-site -0.40 0.50 - External - - 0.39
- 96.
- 97. Online news 97 Focused versus Non-‐focused Sessions Internal Non-focused sessionsFocused sessions● (b) Duration (d) p(focused session) (a) %Sessions (f) Flow 25 15 5 60% 20% 0.6 0.2 0.2 0.1 2 3 4 5 6 7 7 2 3 4 5 6 7 7 2 3 4 5 6 7 7 #Articles #Articles #Articles ● ● ● ● ● ● ● (c) #Providers 2.5 2.0 1.5 ● ● ● ● ● ● ● ●● ●● ●● ●● ●● ●● ●● (e) EntryDisparity 0.5 0.3 0.1 ● ● ● ● ● ● ● When users focus on a news story, they spend more >me reading the ar>cles and the inter-‐site engagement between providers is higher.
- 98. Online news 98 Hyperlink Performance Number of Inline Links • <10 links may be wasLng an opportunity • 10-‐29 links does not result in more clicks • >29 links may harm the user experience PosiCon of Inline Links • 30% at the end, 16% at the beginning, 46% are distributed within the arLcle text. • Performance of links located at the beginning of the text is very low (-‐28%) • Best performance is achieved with links at the end of the arLcle text (+35%) Link popularity● Link performance Position in article text Linkpopularity [0.0,0.1[ [0.3,0.4[ [0.6,0.7[ [0.9,1.0] 10% 20% 30% -0.2 0.0 0.2 Linkperformance ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● Number of inline links in article Clicksperlink 0.0 0.2 0.4 0.6 [0,2] [9,11] [18,20] [27,29] [36,38] Number of inline links in article Numberofclicks [0,2] [9,11] [18,20] [27,29] [36,38] 2.5 5.0 7.5
- 99. Online news 99 Effect on User Engagement Internal Focused Short-‐term: Only 3 (out of 50) providers have their corresponding average dwell Lme lower for the story-‐focused provider sessions. The average increase in dwell Lme from non-‐story-‐focused to story-‐focused provider sessions is 50%. Long-‐term: For 78% of the providers, we find that there are more users that return earlier aker they have a story-‐focused provider session. Internal News provider Dwelltimepersession Non-focused Focused Ext-focused News provider p(absence12h) Non-focused Focused Ext-focused
- 100. Online news 100 Effect on User Engagement External Focused Short-‐term: We do not observe an effect on the dwell Lme (neither posiLve nor negaLve). The average increase is only 5.5%, and based on the K-‐S test we cannot confirm that the distribuLons are different (p-‐ value=0.36). Long-‐term: For 70% of these news sites, the probability that users return within the following 12 hours increases (the average increase is 76%). External News provider Dwelltimepersession Non-focused Focused Ext-focused News provider p(absence12h) Non-focused Focused Ext-focused
- 101. Online news 101 Discovering Story-‐related Content in TwiPer