Daniel Zohar @ Memrise will describe his journey with event data pipelines.

2. • CTO at
Daniel Zohar
• A world leading language learning app
• Making a positive impact on millions of
people’s lives!

3. Data Pipelines @ Memrise
Past, Present & Future

4. Let’s talk about data-pipelines

5. Let’s talk about data-pipelines
(analytics)

6. Let’s talk about data-pipelines
a data pipeline is the software that consolidates data from
multiple sources and makes it available to be used
strategically

7. Let’s talk about data-pipelines

8. Let’s talk about data-pipelines

9. Let’s talk about data-pipelines

10. Data Pipelines at Memrise

11. Data Pipelines at Memrise
• Millions of user sessions a day
• ~90MM events per day (avg. 1,000/s,
max. 2000/s)
• ~330GB of raw data

12. Data Pipelines at Memrise
< 2014

14. Data Pipelines at Memrise
2014< 2014

15. Data Pipelines at Memrise 2014

16. Data Pipeline

17. Data Pipeline

18. Workflow Orchestration

19. Reporting

20. • ETL processes were written in MapReduce (Hive) -
not easy to work with and add new processes.
• UI was custom made. Every report required
planning, developer time and a deploy.
• Some reports running on production DB
Pain points

21. No access to raw data
Pain points

22. Data Pipelines at Memrise
20152014< 2014

23. Snowplow to the rescue!

24. Snowplow to the rescue!
Own your data!

25. Snowplow to the rescue!
Own your data!
and much more…

26. Data Pipelines at Memrise 2015

27. Data Pipeline
KINESIS

28. Data Analysis

29. Data Analysis

30. Workflow Orchestration

31. Workflow Orchestration

32. Workflow Orchestration

33. Workflow Orchestration

34. Data Pipelines at Memrise
201620152014< 2014

35. Realtime Monitoring
“Monitor everything” mentality (with alerts)

36. Integrate Back Into the Product

37. Integrate Back Into the Product

38. Integrate Back Into the Product

39. Thank You!

40. 🚀
• Data Engineer
• Platform Lead
• Platform Engineer
Memrise is
hiring!
memrise.com/jobs