1. Krist Wongsuphasawat /@kristw
visualization
at Twitter
logs
&

2. Krist Wongsuphasawat /@kristw

3. Krist Wongsuphasawat /@kristw
Computer Engineer
Bangkok, Thailand
Chulalongkorn University

4. Krist Wongsuphasawat /@kristw
Computer Engineer
Bangkok, Thailand
Programming + Soccer

5. Krist Wongsuphasawat /@kristw
Computer Engineer
Bangkok, Thailand
Programming + Soccer

6. Krist Wongsuphasawat /@kristw
Computer Engineer
Bangkok, Thailand
Programming + Soccer

7. Krist Wongsuphasawat /@kristw
Computer Engineer
Bangkok, Thailand
M.S. in Computer Science
Univ. of Maryland

8. Krist Wongsuphasawat /@kristw
Computer Engineer
Bangkok, Thailand
PhD in Computer Science
Univ. of Maryland
Information Visualization

9. Krist Wongsuphasawat /@kristw
Computer Engineer
Bangkok, Thailand
PhD in Computer Science
Univ. of Maryland
Information Visualization
IBM
Microsoft

10. Krist Wongsuphasawat /@kristw
Computer Engineer
Bangkok, Thailand
PhD in Computer Science
Univ. of Maryland
Information Visualization
IBM
Microsoft
Data Visualization Scientist
Twitter

11. internal tools
public-facing

12. public-facing

13. interactive.twitter.com

14. internal tools

15. Krist Wongsuphasawat & Jimmy Lin
@kristw
Using visualizations
to monitor changes and harvest insights
from log data at Twitter
@lintool
IEEE VAST 2014

16. Logging user activities
& data analysis

17. UsersUseTwitter

18. UsersUse
Product Managers
Curious
Twitter

19. UsersUse
Curious
Engineers
Log data
in Hadoop
Write Twitter
Instrument
Product Managers

20. What are being logged?
tweet
activities

21. What are being logged?
tweet from home timeline on twitter.com
tweet from search page on iPhone
activities

22. What are being logged?
tweet from home timeline on twitter.com
tweet from search page on iPhone
sign up
log in
retweet
etc.
activities

23. Organize?

24. log event a.k.a. “client event”
[Lee et al. 2012]

25. log event a.k.a. “client event”
client : page : section : component : element : action
web : home : timeline : tweet_box : button : tweet
1) User ID
2) Timestamp
3) Event name
4) Event detail
[Lee et al. 2012]

26. Log data

27. UsersUse
Curious
Engineers
Log data
in Hadoop
Twitter
Instrument
Write
Product Managers
bigger than
Tweet data

28. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Ask
Twitter
Instrument
Write
Product Managers

29. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Find
Ask
Twitter
Instrument
Write
Product Managers

30. Log data

31. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Find, Clean
Ask
Twitter
Instrument
Write
Product Managers

32. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Find, Clean
Ask
Monitor
Twitter
Instrument
Write
Product Managers

33. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Find, Clean, Analyze
Ask
Monitor
Twitter
Instrument
Write
Product Managers

34. Log data
EngineersData Scientists
Usersin Hadoop
Find, Clean, Analyze
Use
Monitor
Ask
Curious
1 2
Twitter
Instrument
Write
Product Managers

35. Part I
Find & Monitor
Client Events

36. Motivation

37. Log data
in Hadoop
Engineers & Data Scientists
billions of rows

38. Log data
in Hadoop
Aggregate
10,000+ event types
date client page section comp. elem. action count
20141011 web home home - - impression 100
20141011 web home wtf - - click 20
Engineers & Data Scientists
Client event collection

39. Log data
in Hadoop
Aggregate
10,000+ event types
date client page section comp. elem. action count
20141011 web home home - - impression 100
20141011 web home wtf - - click 20
Engineers & Data Scientists
Client event collection
(Who-to-Follow)

40. Log data
in Hadoop
Aggregate
Client event collection
Engineers & Data Scientists

41. Log data
in Hadoop
Aggregate
Find
client page section component element action
Search
Client event collection
Engineers & Data Scientists

42. Log data
in Hadoop
Aggregate
Find
client page section component element action
Search
Client event collection
Engineers & Data Scientists

43. section?
component?
element?

44. client page section component element action
Search
Find
Log data
in Hadoop
Aggregate
web home * * impression*
Client event collection
Engineers & Data Scientists

45. client page section component element action
Search
Find
Query
Return
Log data
in Hadoop
Results
web : home : home : - : - : impression
web : home : wtf : - : - : impression
Aggregate
web home * * impression*
Client event collection
Engineers & Data Scientists

46. client page section component element action
Search
Find
Query
Return
Log data
in Hadoop
Results
web : home : home : - : - : impression
web : home : wtf : - : - : impression
Aggregate
search can be better
Client event collection
Engineers & Data Scientists

47. client page section component element action
Search
Find
Query
Return
Log data
in Hadoop
Results
web : home : home : - : - : impression
web : home : wtf : - : - : impression
Aggregate
10,000+ event types
search can be better
Client event collection
Engineers & Data Scientists

48. client page section component element action
Search
Find
Query
Return
Log data
in Hadoop
Results
web : home : home : - : - : impression
web : home : wtf : - : - : impression
Aggregate
search can be better
10,000+ event types
not everybody knows
What are all sections under web:home?
Client event collection
Engineers & Data Scientists

49. client page section component element action
Search
Find
Query
Return
Log data
in Hadoop
Results
web : home : home : - : - : impression
Aggregate
search can be better
one graph / event
10,000+ event types
not everybody knows
What are all sections under web:home?
Client event collection
Engineers & Data Scientists

50. client page section component element action
Search
Find
Query
Return
Log data
in Hadoop
Results
web : home : home : - : - : impression
Aggregate
search can be better
one graph / event
x 10,000
10,000+ event types
not everybody knows
What are all sections under web:home?
Client event collection
Engineers & Data Scientists

51. !
• Search for client events
• Explore client event collection
• Monitor changes
Goals

52. • Session analysis
!
• Monitor network logs, not user activity logs
Related work
[Lam et al. 2007, Shen et al. 2013]
[Ghoniem et al. 2013]

53. Design

54. Client event collection
Engineers & Data Scientists

55. See
Client event collection
Engineers & Data Scientists

56. See
Interactions
search box => filter
Client event collection
narrow down
Engineers & Data Scientists

57. See
How to visualize?
narrow down
Client event collection
Engineers & Data Scientists
Interactions
search box => filter

58. See
How to visualize?
narrow down
Client event collection
Engineers & Data Scientists
client : page : section : component : element : actionInteractions
search box => filter

59. Client event hierarchy
iphone home -
- - impression
tweet tweet click
iphone:home:-:-:-:impression
iphone:home:-:tweet:tweet:click

60. Detect changes
iphone home -
- - impression
tweet tweet click
iphone home -
- - impression
tweet tweet click
TODAY
7 DAYS AGO
compared to

61. Calculate changes
+5% +5% +5%
+10% +10% +10%
-5% -5% -5%
DIFF

62. Display changes
iphone home -
- - impression
tweet tweet click
Map of the Market [Wattenberg 1999], StemView [Guerra-Gomez et al. 2013]

63. Display changes
home -
- - impression
tweet tweet click
iphone

64. Demo
Scribe Radar

65. Twitter for Banana

67. • Since Dec 2013
• 500 unique users, 10 users / day
!
• No training
Deployment

68. Users: PMs, Data Scientists, Engineers
• Search
• Monitor
• See effects after major product launch
Use cases
more information in the paper

69. Part II
Analysis

70. Count page visits
banana : home : - : - : - : impression
home page

71. Funnel
home page
profile page

72. Funnel analysis
banana : home : - : - : - : impression
banana : profile : - : - : - : impression
1 jobhome page
profile page
1 hour

73. Funnel analysis
banana : home : - : - : - : impression
banana : profile : - : - : - : impression banana : search : - : - : - : impression
home page
profile page search page
2 jobs
2 hours

74. Funnel analysis
banana : home : - : - : - : impression
banana : profile : - : - : - : impression banana : search : - : - : - : impression
home page
profile page search page
Specify all funnels manually!
n jobs
n hours

75. Goal
banana : home : - : - : - : impression
… ……
1 job => all funnels, visualized
home page

76. • Visualize an overview of event sequences
!
Related work
[Wongsuphasawat et al. 2011, Monroe et al. 2013, …]

77. • Visualize an overview of event sequences
!
• Big data? eBay checkout sequences
!
One funnel at a time
Checkout > Payment > Confirm > Success
Related work
[Wongsuphasawat et al. 2011, Monroe et al. 2013, …]
[Shen et al. 2013]

78. LifeFlow
[CHI2011]
!
(simplified)

79. User sessions
Session#1
A
B
start
end
Session#4
start
end
A
Session#2
B
start
end
A
Session#3
C
start
end
A

80. Aggregate
4 sessions
A
BB C
start
end endend
A A
end
A

81. Aggregate
A
BB C
start
end endend
end
4 sessions

82. Aggregate
C
start
end endend
end
A
B
4 sessions

83. Aggregate
C
start
end endend
end
A
B
4 sessions

84. Aggregate
C
start
end endend
A
B end
4 sessions

85. Aggregate
C
start
endend
A
B end
4 sessions

86. Aggregate
C
start
endend
A
B end
4 sessions

87. Aggregate
start
endend
A
CB end
4 sessions

88. Aggregate
4,000,000 sessions
endend
A
CB end
start

89. try with “sample” data
(~millions sessions, 10,000+ event types)
!
original paper
(100,000 sessions, ~10 event types)

90. not meaningful
!
small slice of data
but huge file

91. How to make it work?

92. # of unique sequences

93. 1. Reduce event types
Reduce # of unique sequences

94. 1. Reduce event types
Reduce # of unique sequences
10,000 types select
tweet
sign up
log out

95. 1. Reduce event types
Reduce # of unique sequences
10,000 types select
tweet
sign up
log out

96. 1. Reduce event types
Reduce # of unique sequences
10,000 types select merge
tweet from home timeline
tweet from search page
tweet …
= tweet

97. 1. Reduce event types
2. Reduce sequence length
Reduce # of unique sequences

98. 1. Reduce event types
2. Reduce sequence length
Reduce # of unique sequences
session
1000 events

99. 1. Reduce event types
2. Reduce sequence length
Reduce # of unique sequences
session
10 events after (window size & direction)
1000 events
visit home page (alignment)

100. 1. Reduce event types
2. Reduce sequence length
Reduce # of unique sequences
Ask users for input}

101. 1. Reduce event types
2. Reduce sequence length
3. More aggregation on Hadoop
Reduce # of unique sequences
Ask users for input}

102. Collapse events
Sequence
ABBBCCCC
ABBCC
ABC
ABCCCC
ABCD
ABCCCD
ABCCE
ABCDF
ABCDG
ABCDH
e.g.
tweet, tweet, tweet, … = tweet

103. Sequence
ABC
ABC
ABC
ABC
ABCD
ABCD
ABCE
ABCDF
ABCDG
ABCDH
Collapse events

104. Group & Count
Sequence
ABC
ABCD
ABCE
ABCDF
ABCDG
ABCDH
…
Count
2000
80
20
1
1
1
…

105. Group & Count
Sequence
ABC
ABCD
ABCE
ABCDF
ABCDG
ABCDH
ABCDI
ABCDJK
ABCDJL
Count
2000
80
20
1
1
1
1
1
1
rare sequences
(count < threshold)

106. Truncate
Sequence
ABC
ABCD
ABCE
ABCDx
ABCDx
ABCDx
ABCDx
ABCDJx
ABCDJx
Count
2000
80
20
1
1
1
1
1
1
Replace last event with x (…)

107. Sequence
ABC
ABCD
ABCE
ABCDx
ABCDJx
Count
2000
80
20
4
2
Group & Count

108. Truncate more
Sequence
ABC
ABCD
ABCE
ABCDx
ABCDx
Count
2000
80
20
4
2

109. Group & Count
Sequence
ABC
ABCD
ABCE
ABCDx
Count
2000
80
20
6

110. 1. Define set of events
2. Pick alignment, direction and window size
3. Run Hadoop job (with more aggregation)
4. Wait for it… (2+ hrs)
5. Visualize
Final process
~100,000 patterns (10MB)
gazillion patterns (TBs)

111. Demo
Flying Sessions

112. • Since Jan 2013
• Fewer users, but more in-depth ad-hoc analysis
• Initial meeting to provide support
Deployment

113. • What did users do when they visit Twitter? (in demo)
• Where did users give up in the sign up process?
• more in the paper
Case studies

114. • Large-scale User Activity Logs + Visual Analytics
Conclusions & Future work

115. • Large-scale User Activity Logs + Visual Analytics
• Find, Monitor & Explore
+ Anomaly detection & automatic alert
• Funnel Analysis
+ More interactivity & data / reduce wait time
• Used in day-to-day operations at Twitter
Conclusions & Future work

116. Conclusions & Future work
Challenge
big data
small data
visualize & interact
• Large-scale User Activity Logs + Visual Analytics
• Find, Monitor & Explore
+ Anomaly detection & automatic alert
• Funnel Analysis
+ More interactivity & data / reduce wait time
• Used in day-to-day operations at Twitter
aggregate
& sacrifice

117. • Large-scale User Activity Logs + Visual Analytics
• Find, Monitor & Explore
+ Anomaly detection & automatic alert
• Funnel Analysis
+ More interactivity & data / reduce wait time
• Used in day-to-day operations at Twitter
• Generalize to smaller systems
Conclusions & Future work
Challenge
big data
small data
visualize & interact
aggregate
& sacrifice

118. • Data Scientists & Engineers @Twitter — Linus Lee, Chuang Liu
• Feedback from reviewers, Ben Shneiderman & Catherine Plaisant
Acknowledgement

119. • Large-scale User Activity Logs + Visual Analytics
• Find, Monitor & Explore
+ Anomaly detection & automatic alert
• Funnel Analysis
+ More interactivity & data / reduce wait time
• Used in day-to-day operations at Twitter
• Generalize to smaller systems
Conclusions & Future work
Challenge
big data
small data
visualize & interact
kristw@twitter.com / @kristw
aggregate
& sacrifice

120. One more thing …

141. Questions?

142. Thank you