This document summarizes the architecture of Allegro's recommendation system. It uses a lambda architecture with Apache Spark, Kafka, Cassandra and Elasticsearch. Over 78 million user events are processed daily to identify 18 million interesting items and group them into 210,000 meta items and 740,000 clusters. Collaborative filtering with Apache Mahout identifies similar items using implicit feedback to address cold start problems. Infrastructure is deployed on the cloud with monitoring, logging and automated testing.

1. Kornel Kiełczewski" Grupa Allegro"
"
26.02.2015"
Marcin Cylke"

2. 1400 req/s"
25 req/s"
Context – recommendations in numbers"
t"
req/s"

3. Context – recommendations in numbers"
• 78’000’000 user events each day"
• 18’000’000 interesting items"
"

4. Meta Items and Clusters"
• 210’000 meta items"
• 740’000 clusters"
"

5. Collaborative filtering"
• Apache Mahout"
• Implicit feedback"
• What is similar to D?"

6. Collaborative filtering"
• Apache Mahout"
• Implicit feedback"
• C is similar to D"

7. Collaborative filtering"
• Apache Mahout"
• Implicit feedback"
• Cold start problem"

8. Important assumptions"
• No Transactions"
• Eventual Consistency"

9. Scala Programming Language"

10. Lambda Architecture"

11. Lambda Architecture - Speed Layer"
• Kafka"
• Spark Streaming"
• Cassandra"
"
• ElasticSearch"

12. Lambda Architecture - Batch Layer"
• Spark"
• Hive"
• Mahout"
• HDFS"
• Scheduling – luigi"

13. Lambda Architecture - Serving Layer"
• REST API"
• ElasticSearch"
• Cassandra"
• Business Logic"

14. Infrastructure"
• continuous delivery in the cloud"
• monitoring"
• logging: kibana"
• time series: graphite + tessera"
• custom dashboards"
• cabot + pagerduty"

15. • Cassandra – fear the antipattern"
• ElasticSearch - partitions"
• Choose your NoSQL wisely"
• Jepsen"
Issues"

16. Issues"
• Kerberos authentication"

17. Issues"
• Quality measures"
• A/B testing"
• Useful metrics and
automated checks"

18. Summary"
Lambda Architecture seems like an
attractive solution"

20. Thank you."