Logs & Visualizations at Twitter

1. Krist Wongsuphasawat /@kristw visualization at Twitter logs &

2. Krist Wongsuphasawat /@kristw

3. Krist Wongsuphasawat /@kristw Computer Engineer Bangkok, Thailand Chulalongkorn University

4. Krist Wongsuphasawat /@kristw Computer Engineer Bangkok, Thailand Programming + Soccer

7. Krist Wongsuphasawat /@kristw Computer Engineer Bangkok, Thailand M.S. in Computer Science Univ. of Maryland

8. Krist Wongsuphasawat /@kristw Computer Engineer Bangkok, Thailand PhD in Computer Science Univ. of Maryland Information Visualization

9. Krist Wongsuphasawat /@kristw Computer Engineer Bangkok, Thailand PhD in Computer Science Univ. of Maryland Information Visualization IBM Microsoft

10. Krist Wongsuphasawat /@kristw Computer Engineer Bangkok, Thailand PhD in Computer Science Univ. of Maryland Information Visualization IBM Microsoft Data Visualization Scientist Twitter

11. internal tools public-facing

12. public-facing

13. interactive.twitter.com

14. internal tools

15. Krist Wongsuphasawat & Jimmy Lin @kristw Using visualizations to monitor changes and harvest insights from log data at Twitter @lintool IEEE VAST 2014

16. Logging user activities & data analysis

17. UsersUseTwitter

18. UsersUse Product Managers Curious Twitter

19. UsersUse Curious Engineers Log data in Hadoop Write Twitter Instrument Product Managers

20. What are being logged? tweet activities

21. What are being logged? tweet from home timeline on twitter.com tweet from search page on iPhone activities

22. What are being logged? tweet from home timeline on twitter.com tweet from search page on iPhone sign up log in retweet etc. activities

23. Organize?

24. log event a.k.a. “client event” [Lee et al. 2012]

25. log event a.k.a. “client event” client : page : section : component : element : action web : home : timeline : tweet_box : button : tweet 1) User ID 2) Timestamp 3) Event name 4) Event detail [Lee et al. 2012]

26. Log data

27. UsersUse Curious Engineers Log data in Hadoop Twitter Instrument Write Product Managers bigger than Tweet data

28. UsersUse Curious Engineers Log data in Hadoop Data Scientists Ask Twitter Instrument Write Product Managers

29. UsersUse Curious Engineers Log data in Hadoop Data Scientists Find Ask Twitter Instrument Write Product Managers

30. Log data

31. UsersUse Curious Engineers Log data in Hadoop Data Scientists Find, Clean Ask Twitter Instrument Write Product Managers

32. UsersUse Curious Engineers Log data in Hadoop Data Scientists Find, Clean Ask Monitor Twitter Instrument Write Product Managers

33. UsersUse Curious Engineers Log data in Hadoop Data Scientists Find, Clean, Analyze Ask Monitor Twitter Instrument Write Product Managers

34. Log data EngineersData Scientists Usersin Hadoop Find, Clean, Analyze Use Monitor Ask Curious 1 2 Twitter Instrument Write Product Managers

35. Part I Find & Monitor Client Events

36. Motivation

37. Log data in Hadoop Engineers & Data Scientists billions of rows

38. Log data in Hadoop Aggregate 10,000+ event types date client page section comp. elem. action count 20141011 web home home - - impression 100 20141011 web home wtf - - click 20 Engineers & Data Scientists Client event collection

39. Log data in Hadoop Aggregate 10,000+ event types date client page section comp. elem. action count 20141011 web home home - - impression 100 20141011 web home wtf - - click 20 Engineers & Data Scientists Client event collection (Who-to-Follow)

40. Log data in Hadoop Aggregate Client event collection Engineers & Data Scientists

41. Log data in Hadoop Aggregate Find client page section component element action Search Client event collection Engineers & Data Scientists

42. Log data in Hadoop Aggregate Find client page section component element action Search Client event collection Engineers & Data Scientists

43. section? component? element?

44. client page section component element action Search Find Log data in Hadoop Aggregate web home * * impression* Client event collection Engineers & Data Scientists

45. client page section component element action Search Find Query Return Log data in Hadoop Results web : home : home : - : - : impression web : home : wtf : - : - : impression Aggregate web home * * impression* Client event collection Engineers & Data Scientists

46. client page section component element action Search Find Query Return Log data in Hadoop Results web : home : home : - : - : impression web : home : wtf : - : - : impression Aggregate search can be better Client event collection Engineers & Data Scientists

47. client page section component element action Search Find Query Return Log data in Hadoop Results web : home : home : - : - : impression web : home : wtf : - : - : impression Aggregate 10,000+ event types search can be better Client event collection Engineers & Data Scientists

48. client page section component element action Search Find Query Return Log data in Hadoop Results web : home : home : - : - : impression web : home : wtf : - : - : impression Aggregate search can be better 10,000+ event types not everybody knows What are all sections under web:home? Client event collection Engineers & Data Scientists

49. client page section component element action Search Find Query Return Log data in Hadoop Results web : home : home : - : - : impression Aggregate search can be better one graph / event 10,000+ event types not everybody knows What are all sections under web:home? Client event collection Engineers & Data Scientists

50. client page section component element action Search Find Query Return Log data in Hadoop Results web : home : home : - : - : impression Aggregate search can be better one graph / event x 10,000 10,000+ event types not everybody knows What are all sections under web:home? Client event collection Engineers & Data Scientists

51. ! • Search for client events • Explore client event collection • Monitor changes Goals

52. • Session analysis ! • Monitor network logs, not user activity logs Related work [Lam et al. 2007, Shen et al. 2013] [Ghoniem et al. 2013]

53. Design

54. Client event collection Engineers & Data Scientists

55. See Client event collection Engineers & Data Scientists

56. See Interactions search box => ﬁlter Client event collection narrow down Engineers & Data Scientists

57. See How to visualize? narrow down Client event collection Engineers & Data Scientists Interactions search box => ﬁlter

58. See How to visualize? narrow down Client event collection Engineers & Data Scientists client : page : section : component : element : actionInteractions search box => ﬁlter

59. Client event hierarchy iphone home - - - impression tweet tweet click iphone:home:-:-:-:impression iphone:home:-:tweet:tweet:click

60. Detect changes iphone home - - - impression tweet tweet click iphone home - - - impression tweet tweet click TODAY 7 DAYS AGO compared to

61. Calculate changes +5% +5% +5% +10% +10% +10% -5% -5% -5% DIFF

62. Display changes iphone home - - - impression tweet tweet click Map of the Market [Wattenberg 1999], StemView [Guerra-Gomez et al. 2013]

63. Display changes home - - - impression tweet tweet click iphone

64. Demo Scribe Radar

65. Twitter for Banana

67. • Since Dec 2013 • 500 unique users, 10 users / day ! • No training Deployment

68. Users: PMs, Data Scientists, Engineers • Search • Monitor • See effects after major product launch Use cases more information in the paper

69. Part II Analysis

70. Count page visits banana : home : - : - : - : impression home page

71. Funnel home page proﬁle page

72. Funnel analysis banana : home : - : - : - : impression banana : proﬁle : - : - : - : impression 1 jobhome page proﬁle page 1 hour

73. Funnel analysis banana : home : - : - : - : impression banana : proﬁle : - : - : - : impression banana : search : - : - : - : impression home page proﬁle page search page 2 jobs 2 hours

74. Funnel analysis banana : home : - : - : - : impression banana : proﬁle : - : - : - : impression banana : search : - : - : - : impression home page proﬁle page search page Specify all funnels manually! n jobs n hours

75. Goal banana : home : - : - : - : impression … …… 1 job => all funnels, visualized home page

76. • Visualize an overview of event sequences ! Related work [Wongsuphasawat et al. 2011, Monroe et al. 2013, …]

77. • Visualize an overview of event sequences ! • Big data? eBay checkout sequences ! One funnel at a time  Checkout > Payment > Conﬁrm > Success Related work [Wongsuphasawat et al. 2011, Monroe et al. 2013, …] [Shen et al. 2013]

78. LifeFlow [CHI2011] ! (simpliﬁed)

79. User sessions Session#1 A B start end Session#4 start end A Session#2 B start end A Session#3 C start end A

80. Aggregate 4 sessions A BB C start end endend A A end A

81. Aggregate A BB C start end endend end 4 sessions

82. Aggregate C start end endend end A B 4 sessions

83. Aggregate C start end endend end A B 4 sessions

84. Aggregate C start end endend A B end 4 sessions

85. Aggregate C start endend A B end 4 sessions

86. Aggregate C start endend A B end 4 sessions

87. Aggregate start endend A CB end 4 sessions

88. Aggregate 4,000,000 sessions endend A CB end start

89. try with “sample” data (~millions sessions, 10,000+ event types) ! original paper (100,000 sessions, ~10 event types)

90. not meaningful ! small slice of data but huge ﬁle

91. How to make it work?

92. # of unique sequences

93. 1. Reduce event types Reduce # of unique sequences

94. 1. Reduce event types Reduce # of unique sequences 10,000 types select tweet sign up log out

95. 1. Reduce event types Reduce # of unique sequences 10,000 types select tweet sign up log out

96. 1. Reduce event types Reduce # of unique sequences 10,000 types select merge tweet from home timeline tweet from search page tweet … = tweet

97. 1. Reduce event types 2. Reduce sequence length Reduce # of unique sequences

98. 1. Reduce event types 2. Reduce sequence length Reduce # of unique sequences session 1000 events

99. 1. Reduce event types 2. Reduce sequence length Reduce # of unique sequences session 10 events after (window size & direction) 1000 events visit home page (alignment)

100. 1. Reduce event types 2. Reduce sequence length Reduce # of unique sequences Ask users for input}

101. 1. Reduce event types 2. Reduce sequence length 3. More aggregation on Hadoop Reduce # of unique sequences Ask users for input}

102. Collapse events Sequence ABBBCCCC ABBCC ABC ABCCCC ABCD ABCCCD ABCCE ABCDF ABCDG ABCDH e.g. tweet, tweet, tweet, … = tweet

103. Sequence ABC ABC ABC ABC ABCD ABCD ABCE ABCDF ABCDG ABCDH Collapse events

104. Group & Count Sequence ABC ABCD ABCE ABCDF ABCDG ABCDH … Count 2000 80 20 1 1 1 …

105. Group & Count Sequence ABC ABCD ABCE ABCDF ABCDG ABCDH ABCDI ABCDJK ABCDJL Count 2000 80 20 1 1 1 1 1 1 rare sequences (count < threshold)

106. Truncate Sequence ABC ABCD ABCE ABCDx ABCDx ABCDx ABCDx ABCDJx ABCDJx Count 2000 80 20 1 1 1 1 1 1 Replace last event with x (…)

107. Sequence ABC ABCD ABCE ABCDx ABCDJx Count 2000 80 20 4 2 Group & Count

108. Truncate more Sequence ABC ABCD ABCE ABCDx ABCDx Count 2000 80 20 4 2

109. Group & Count Sequence ABC ABCD ABCE ABCDx Count 2000 80 20 6

110. 1. Deﬁne set of events 2. Pick alignment, direction and window size 3. Run Hadoop job (with more aggregation) 4. Wait for it… (2+ hrs) 5. Visualize Final process ~100,000 patterns (10MB) gazillion patterns (TBs)

111. Demo Flying Sessions

112. • Since Jan 2013 • Fewer users, but more in-depth ad-hoc analysis • Initial meeting to provide support Deployment

113. • What did users do when they visit Twitter? (in demo) • Where did users give up in the sign up process? • more in the paper Case studies

114. • Large-scale User Activity Logs + Visual Analytics Conclusions & Future work

115. • Large-scale User Activity Logs + Visual Analytics • Find, Monitor & Explore + Anomaly detection & automatic alert • Funnel Analysis + More interactivity & data / reduce wait time • Used in day-to-day operations at Twitter Conclusions & Future work

116. Conclusions & Future work Challenge big data small data visualize & interact • Large-scale User Activity Logs + Visual Analytics • Find, Monitor & Explore + Anomaly detection & automatic alert • Funnel Analysis + More interactivity & data / reduce wait time • Used in day-to-day operations at Twitter aggregate & sacriﬁce

117. • Large-scale User Activity Logs + Visual Analytics • Find, Monitor & Explore + Anomaly detection & automatic alert • Funnel Analysis + More interactivity & data / reduce wait time • Used in day-to-day operations at Twitter • Generalize to smaller systems Conclusions & Future work Challenge big data small data visualize & interact aggregate & sacriﬁce

118. • Data Scientists & Engineers @Twitter — Linus Lee, Chuang Liu • Feedback from reviewers, Ben Shneiderman & Catherine Plaisant Acknowledgement

119. • Large-scale User Activity Logs + Visual Analytics • Find, Monitor & Explore + Anomaly detection & automatic alert • Funnel Analysis + More interactivity & data / reduce wait time • Used in day-to-day operations at Twitter • Generalize to smaller systems Conclusions & Future work Challenge big data small data visualize & interact kristw@twitter.com / @kristw aggregate & sacriﬁce

120. One more thing …

141. Questions?

142. Thank you

Logs & Visualizations at Twitter

Logs & Visualizations at Twitter

Recommended

More Related Content

What's hot

What's hot(10)

Viewers also liked

Viewers also liked(20)

Similar to Logs & Visualizations at Twitter

Similar to Logs & Visualizations at Twitter(20)

More from Krist Wongsuphasawat

More from Krist Wongsuphasawat(20)

Recently uploaded

Recently uploaded(20)

Logs & Visualizations at Twitter