Identifying Returning Users with Spark
•
2 likes•648 views
Identifying returning users cross-device it's an important requirement in order to provide an optimal experience to our customers. In this talk we'll see how Spark enables us to do it both in near real-time using Spark Streaming and in batch mode using GraphX. Slides for the Spark London Meetup: http://www.meetup.com/Spark-London/events/233326443/
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Identifying Returning Users with Spark
- 2. Simply Business • Largest UK business insurer provider • More than 370,000 policy holders • Using BML, tech and data to disrupt the business insurance market
- 3. Data ’n’ Analytics • 4 Data Engineers • 3 Business Intelligence Developers • 1 Data Scientist • 1 Director of Data Science • And hiring! :-)
- 4. Our Infrastructure Event Enrichment (Spark Streaming) Event Collector (REST Endpoint) Kinesis MongoDB Kinesis Redshi> Downstream NRT (Spark Streaming)
- 6. The Challenge Accurately identify users over time as they interact with us
- 7. Why? Knowledge To understand how our visitors behave: • They have long buying cycles • They use multiple devices • They engage with us through multiple channels
- 8. Why? Cost Analysis To know how much it costs to acquire new customers: • They visit us from different marketing channels • Each of them have different costs • We remunerate our partners depending on that
- 9. Why? Personalization To adapt their visit while they interact with us: • We can trigger offers depending on the channel they come from • We can prefill fields if they’re recognized • We can change the look and feel of the website
- 10. Cookies? Their work great when you want to identify people coming back to your website. Challenges: • Different devices • Cleared cookies • Private sessions • What about telephone calls…?
- 11. Use More IDs! We can use other IDs provided by visitors to link sessions: • Login details • Email addresses • Telephone numbers • Credit card numbers • Fingerprints • …
- 13. How many visitors do we have here? Example Timestamp Event Cookie ID Email 1 page_view 111 2 details_submiFed 111 a@a.com 3 page_view 555 4 page_view 888 5 policy_sold 555 a@a.com 6 details_submiFed 888 b@b.com
- 14. How many visitors do we have here? Only 2 Example Timestamp Event Cookie ID Email 1 page_view 111 2 details_submiFed 111 a@a.com 3 page_view 555 4 page_view 888 5 policy_sold 555 a@a.com 6 details_submiFed 888 b@b.com
- 15. Example: Streaming Timestamp Event Cookie ID Email 1 page_view 111 2 details_submiFed 111 a@a.com 3 page_view 555 4 page_view 888 5 policy_sold 555 a@a.com 6 details_submiFed 888 b@b.com Visitor A: 111
- 16. Example: Streaming Timestamp Event Cookie ID Email 1 page_view 111 2 details_submiFed 111 a@a.com 3 page_view 555 4 page_view 888 5 policy_sold 555 a@a.com 6 details_submiFed 888 b@b.com Visitor A: 111, a@a.com
- 17. Example: Streaming Timestamp Event Cookie ID Email 1 page_view 111 2 details_submiFed 111 a@a.com 3 page_view 555 4 page_view 888 5 policy_sold 555 a@a.com 6 details_submiFed 888 b@b.com Visitor A: 111, a@a.com Visitor B: 555
- 18. Example: Streaming Timestamp Event Cookie ID Email 1 page_view 111 2 details_submiFed 111 a@a.com 3 page_view 555 4 page_view 888 5 policy_sold 555 a@a.com 6 details_submiFed 888 b@b.com Visitor A: 111, a@a.com Visitor B: 555 Visitor C: 888
- 19. Example: Streaming Timestamp Event Cookie ID Email 1 page_view 111 2 details_submiFed 111 a@a.com 3 page_view 555 4 page_view 888 5 policy_sold 555 a@a.com 6 details_submiFed 888 b@b.com Visitor A: 111, 555, a@a.com Visitor C: 888 Visitors A and B were merged!
- 20. Example: Streaming Timestamp Event Cookie ID Email 1 page_view 111 2 details_submiFed 111 a@a.com 3 page_view 555 4 page_view 888 5 policy_sold 555 a@a.com 6 details_submiFed 888 b@b.com Visitor A: 111, 555, a@a.com Visitor C: 888, b@b.com
- 21. Streaming Algorithm for event in incoming_events: prev_visitors = [get_visitor(id) for id in ids(event)] if len(prev_visitors) == 0: # First time that we see that visitor create_visitor(event) elif len(prev_visitors) == 1: # Visitor coming back using a known ID update_visitor(prev_visitors[0], event) else: # Event IDs were thought to belong to different visitors merge_and_update_visitor(prev_visitors, event) Store mapping: event_ids [1..N] ---> [1] visitor
- 22. Batch Solution
- 23. Batch We also need to solve this problem in batch mode because every now and then we reprocess all our data. We cannot do it with Spark Streaming because: • Kinesis only stores up to 7 days of data • It would take too long • We risk overwhelming our MongoDB
- 24. Example: Batch Timestamp Event Cookie ID Email 1 page_view 111 2 details_submiFed 111 a@a.com 3 page_view 555 4 page_view 888 5 policy_sold 555 a@a.com 6 details_submiFed 888 b@b.com
- 25. Batch Algorithm • Find the connected components in the graph • Already implemented in GraphX! • Vertices: visitor IDs • Edges: pairs of IDs occurring in the same event
- 26. Example: Batch Timestamp Event Cookie ID Email 1 page_view 111 2 details_submiFed 111 a@a.com 3 page_view 555 4 page_view 888 5 policy_sold 555 a@a.com 6 details_submiFed 888 b@b.com 111 a@a.com 555 888 b@b.com
- 27. GraphX: Issues • There is no Python API for GraphX, and Graphframes performance is worse than GraphX’s • GraphX requires vertices to have numeric IDs • You’ll have to create a mapping between the real IDs and the numeric IDs • Perform the connected components calculation • Map back the numeric IDs to the real IDs • Documentation is not as good as other Spark projects
- 28. Sum Up
- 29. NRT: Spark Streaming Batch: GraphX Latency Seconds Minutes/Hours Throughput Low High Good for Online Analysis Development Your own Algorithm available Sum Up
- 30. Beware That two events share an ID does not necessarily mean that were generated by the same person. Watch out for: • Clashing IDs • Fake IDs: test@test.com, 07123456789, etc. • Shared devices: couples, public computers, etc. • People using your system on behalf of someone else • Bugs Analyze your data first!