AppsFlyer presents their real-time solution using MemSQL at the Reversim Summit!

1. Journey to the
Real-Time Analytics
in Extreme Growth
yulia@appsflyer.com

2. Real Time Dashboard
• User acquisition
• 8B events daily

3. Data is Mutable

4. Previous solution - Toku (Mongo)
KAFKA
Toku
writers Toku master
Toku slaves
Dashboard

5. Toku Problems
• Failures on weekly basis
• Bad modeling
• No recovery

6. Requirements
• RealTime
• More events (more data)
• More dimensions (MUCH MORE DATA !!!)
• Stability
• Faster

7. Dashboard - DB abstraction level
KAFKA
Toku
writers Toku master
Toku slaves
Dashboard
Middleware
(Vishnu)

8. We tried...

10. https://www.meetup.com/Druid-Israel/events/232075974/

11. What did we gain?
• Flexible middleware
• Batch daily process - first step to recovery
• Developers Paradise

12. Down to Earth

13. MemSQL
In Memory Scalable DB

14. Current Solution - MemSQL

15. MemSQL Architecture
KAFKA
MemSQL
writers
Memsql
Cluster
Dashboard
Middleware
(Vishnu)
MemSQL
writers
Memsql
Cluster (Slave)

16. Recovery
KAFKA(24h)
MemSQL
writers
Master
Memsql
Cluster
Dashboard
Middleware
(Vishnu)
Yesterday
snapshot
Recovery
Memsql
Cluster
MemSQL writers -
only current day

17. Mem SQL - Quick Win
• Fast
• Recoverable
• Possibility to return to 0 point
• Ability to add new features
• More Data (X30)

18. Show me the numbers
• Data - 100 GB x 2 clusters
• Query Latency - 1-3 seconds
• Machines x 2 clusters
– 2 aggregators - m4.4xlarge
– 4 leaves - r3.4xlarge
• Cost reduction $20K less
than toku monthly

19. Good Enough Approach
• More data - more money
• Less money - less data

20. Current - Architecture
KAFKA
writers -
only new
data
Memsql
Rowstore
Cluster
1-2 weeks
Dashboard
Middleware
(Vishnu)Daily Batch
process
S3
files
Memsql
Columnstore
History
Cluster
Daily

21. “Premature optimization is
a root of all evil”
Donald Knuth

22. Yulia@appsflyer.com

23. appsflyer.com/jobs

24. http://www.shutterstock.com/pic.mhtml?utm_campaign=ClipartLogo&irgwc=1&tpl=46764-50655&id=154723511&language=en&utm_medi
um=Affiliate&utm_source=46764
http://www.samatters.com/wp-content/uploads/2015/07/round-peg.jpg
http://marsmedia.info/en/blog/cassandra.png
http://www.zdnet.de/wp-content/uploads/2013/10/mongodb-logo.jpg
https://chris.lu/upload/images/redis.png
https://upload.wikimedia.org/wikipedia/en/b/ba/Druid_MasterLogo_Full_Color_Small.png
https://www.leftronic.com/wp-content/uploads/2015/04/Amazonredshift_220x110.png