This document discusses Kobo's use of big data analytics for ebooks and ereaders. It describes how Kobo uses technologies like Hadoop, Storm, and Solix to process, store, and analyze streaming data from ebooks. Kobo's big data team analyzes this data to power search and recommendations functions, perform content analysis tasks like related items and adult content filtering, and extract metadata from books to link to online information. Kobo's optimization of webpage layouts also utilizes big data approaches to test configurations and maximize user engagement.

1. BIG DATA FOR EBOOKS AND EREADERS
INMAR GIVONI, PHD
VP, BIG DATA, KOBO
IGIVONI@KOBO.COM

2. A BIT ABOUT KOBO

3. A LOT OF BITS OF KOBO DATA

4. A LOT OF BITS OF KOBO DATA

5. TECHNOLOGY/TOOLS USED BY KOBO BIG DATA
•Processing/Streaming: Hadoop, Storm, Flume
•Storage/streaming: Hive, sql, redis, couchbase
•Search: Solr+plugins
•Languages: Python, Java, C++, R

6. ABOUT KOBO’S BIG DATA TEAM

7. BIG DATA IS LIKE TEENAGE SEX: EVERYONE TALKS ABOUT IT, NOBODY REALLY KNOWS HOW TO DO IT, EVERYONE THINKS EVERYONE ELSE IS DOING IT, SO EVERYONE CLAIMS THEY ARE DOING IT...
DAN ARIELY

8. SEARCH

9. RECOMMENDATIONS

10. RELATED ITEMS THROUGH CONTENT ANALYSIS

12. SIMILARITY ANALYSIS

13. ADULT CONTENT CLASSIFICATION

14. BEYOND THE BOOK

15. •FIND INTERESTING THINGS IN A BOOK
•RELATE THEM TO INTERESTING CONTENT
FROM THE INTERNET

16. 1. Key-term detection
2. Disambiguation
3. Filtering

17. KEY-TERM EXTRACTION
•Approach: TF-IDF top terms
•Only to Wikipedia terms
•Greedy up-to-5-gram matching

18. DISAMBIGUATING KEY-TERMS
•Problem : Disambiguation of item to Wikipedia articles mapping
•Solution : choose Wikipedia articles that make sense collectively [1]
[1] Local and Global Algorithms for Disambiguation to Wikipedia, ACL 2011

19. Gandalf
Hobbit
Bilbo Baggins

20. Gandalf
Hobbit
Bilbo Baggins

21. Gandalf
Hobbit
Bilbo Baggins

22. WIKIPEDIA TERM SIMILARITY
•Using Google Similarity Distance

23. IN MORE TECHNICAL TERMS…
•Max-weight k-clique in a k-partite graph problem.
•We used a greedy approach

24. SOME SCREEN SHOTS

25. SOME SCREEN SHOTS

26. SOME SCREEN SHOTS

27. PAGE LAYOUT OPTIMIZATION

28. •MANY WAYS OF SHOWING CONTENT ON THE KOBO WEBPAGE
•FIND BEST LAYOUT

29. MAIN PROBLEMS
•Automatic generation of layouts
•Finding optimal layout

30. LAYOUT GENERATION
•Local search
•Start from a handful of expert generated layouts
•Use widget statistics to make informed swap/exchange steps

31. OPTIMIZATION FRAMEWORK – EXPLORE/EXPLOIT
•Several actions an agent can take
•Agent takes action, and observes payoff

32. •Pick a policy for choosing action with good trade-off
•If you always take the best one so far, may be missing on better options
•If you always explore new things you losses accumulate
OPTIMIZATION FRAMEWORK – EXPLORE/EXPLOIT

33. MULTI-ARMED BANDIT MODEL
•Maximize total expected reward
•Bayesian framework (Agarwal et al [1])
•Context Sensitive Variant (Li et al [2])
[1] Explore/Exploit schemes for web content optimization, Yahoo Research, ICDM ‘09
[2] A contextual-bandit approach to personalized news article recommendation, Yahoo Research, WWW ‘10

34. FEATURE VECTOR CONSTRUCTION
•User information:
•Purchase history, by genre, by price sensitivity, freshness
•Browsing history
•Geo
•Extensions to
•Optimize for profit (not CTR)
•Account for different widget categories

35. UPCOMING PROJECTS

36. THANK YOU!
QUESTIONS?