Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1Q3kAZc. Vaclav Petricek discusses how to train models as well as architect and build a scalable system powered by Storm, Hadoop, Spark, Spring Boot and Vowpal Wabbit that meets SLAs measured in tens of milliseconds. Filmed at qconnewyork.com. Vaclav Petricek is a Director of Data Science at Ticketmaster and Live Nation Entertainment in Hollywood, prior to that he lead Machine Learning at Santa Monica-based eHarmony. He has been responsible for building real-time recommendation systems, optimization and large machine learning as well as people analytics.

1. Fans just wanna have fun!
@petricek

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/storm-hadoop-spark

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. Every year:
hundreds of millions tickets sold
worth Billions of Dollars

9. !
Tickets to the Fans!
(not scalpers)!
!
Recommendations!
!
Who are my fans? !

10. Quality evaluation
Data collection
Recommendation Strategies

12. Javascript request URL:http://pixelserver:8080/pixel?
evt=page_load
imp=e4beb325-310e-493a-88cd-3da709ef509a
impp=2d60a5ee-ac02-46df-8d0e-fbbe976361ef
bid=bj6LvMJg-NnIOXp7tScUAHp-vcLsATCZoDvp08idDaIZo-4Iok1ppkDXZvA3RUWwoSV9IiRB
pid=lILEaTVwqW8iwByWQzh0Iuk3w4kxXAhBzoFx2-PBdBriuqOKeRousDZgIgrVZQ
mid=lILEaTVwqW8iwByWQzh0Iuk3w4kxXAhBzoFx2-PBdBriuqOKeRousDZgIgrVZQ
sid=lS9oSQ72XqeVxPX0axzEjwFMJQ5RfAbYOdA5G_IssvgmOl4fiQHQ5gO0Lg_FDQNZ5hirrvm
pgt=Artist%3A+On+Sale%3A+List
dmn=TM_US
aid=1687536
mkt=35
tsu=1433631937802
ref=http%3A%2F%2Fwww.ticketmaster.com%2F
pgn=1
tml=tm_homeA_b_10001_1
fbt=not_authorized
Of Monsters and Men
1px
1px“pixel”

13. http://pixelserver/api/pixel?…

14. Number of requests over time
something big went on sale
~10 x

15. Quality evaluation
Data collection
Recommendation Strategies

16. Ticket Alert

19. ~100m
users
tens of thousands
artists

20. ~100m
users
thousands
of artists
Availability
Recommendability

22. Online
Recommendations

23. Top Sellers
Offline aggregate over many days

24. Hot right now
30min
sliding window
count rank “hot right now”filter

25. After viewing
fans eventually bought

26. Personalized
recommendations

27. 100m
users
tens of thousands
artists
Recommender!
Service

28. Service endpoints
Public endpoints

29. Ticket!
Availability !
Service
Event Feed
Recommender!
Service
Pixel!
Service
100ms SLA
async
Storm

30. $ docker-compose up!
:-)

31. Quality

32. Expert feedback

34. Availability

35. A/B testing

36. Stealth testing

37. hdfs bolttee bolt
Recommender!
service
replay!
bolt
hdfs bolt

38. recB
recA
(live)
clickA
impA
CTR
recA
0
0.25
0.5
0.75
1
0
0.25
0.5
0.75
1
recB

39. Quality evaluation
Data collection
Recommendation Strategies

40. Contextual bandits
Exploration v Exploitation
More context
(user, artist, venue, event metadata)
Next steps

41. !
Tickets to the Fans!
(not scalpers)!
!
Recommendations!
!
Who are my fans? !

44. $$$
$$$
$
$

45. Counting unique users
O(N*log(N))
O(N)
HyperLogLog
O(N)

46. HLL
$$$HLL
+
+
+
reduce
HLL
HLL
HLL
HLL
HLL
HLL
HLL
HLL
HLL
HLL
HLL
HLL
map
HLL

47. Unique female users
——————————
Unique users total
Unique users with family
——————————
Unique users total
Unique high income users
———————————
Unique users total
$$$HLL
=
HLL
HLL
HLL
HLL
=
=
HLL

48. SF Philharmonic
Phish
Meshuggagh Backstreet Boys

49. Chvrchs 15% Asian K. Michelle 57% African American
Ramon Ayala 92% Hispanic Tom Petty 96% Caucasian

50. Millennials Grits and Biscuits
Gen X Depeche Mode
Baby Boomers Chicago Musical

52. Tens of thousands
models
Next

53. !
Tickets to the Fans!
(not scalpers)!
!
Recommendations!
!
Who are my fans? !

55. 0
500
1000
Price

56. |+

58. vowpal wabbit
JNI
Storm

59. vw
vw
vw
vw
?

60. In-cart conversion
4 x

61. Arms Race
Next steps

62. !
Tickets to the Fans!
(not scalpers)!
!
Recommendations!
!
Who are my fans? !

63. Decades of
granular and
longitudinal data
Now what?

64. Recommend
the best Seat
for you

65. linkedin.com/in/petricek

66. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/storm-
hadoop-spark