We’ve been running a mix of Redshift, EMR, Glue and Athena for data processing and analysis for several years, and have learnt a lot along the way. In this talk, I’ll cover some of the highlights and lowlights of running data warehouses on AWS, and where we’ve found the right tool for each task. See the full talk on here: https://youtu.be/RsPWfAGVGF8

1. MANCHESTER

2. Lessons learnt - Building a data lake with Redshift,
EMR, and Athena
Ewan Leith | 2nd April 2019
@EwanToo on Twitter

3. What do RealityMine do?
Opt-In Market Research
Data Processing and Reporting

4. Where we started

5. Migrating to Redshift
• Fast or Dense Nodes
• Fixed ratio of CPU to storage
• Table distribution & Indexes
• Supports Complex SQL
• Daily Data Loads

6. We built an analytics platform

7. Interactive Query Builder

8. Dynamic Filtering

9. Great graphing capabilities

10. It didn’t work…

11. Struggled with concurrent queries

12. High Latency

13. Slow to scale up clusters

14. We moved on

15. What we needed next
• Faster Data
• More CPU power, and On-Demand
Capacity
• Cost-Effective, if possible!

16. Apache Spark
• “Big Data” Processing Engine
• AWS Elastic MapReduce (EMR)
• Low operational effort
• Spot Instances can be very cheap

17. Data Lake v0.1
• On-demand reporting clusters
• Store everything on S3 in
Parquet file format
• Hourly Data Loads to Redshift

18. AWS Athena
• Based on Presto
• “Serverless” Data Query
Engine
• Reads files from S3
• Extremely Fast

19. Data Lake v1
• Athena querying Parquet on
S3
• Fast, but hard to query
complex data structures

20. AWS Redshift Spectrum
• Redshift SQL Engine
• Scale-Out Queries
• Reads from S3 and local disk
• Extremely Fast

21. Data Lake v2
• AWS Glue - Automated
Extraction & Schema
Management
• Linked with Athena and
Redshift Spectrum

22. We’re building an analytics platform
It’s working this time!

23. What’s Next?
• Redshift Concurrency Scaling
• Athena enhancements
• Data deletion needs work

24. Conclusions
• AWS Glue makes life better
• Apache Spark on EMR for raw
CPU power
• Redshift and Athena both work
great

25. Thanks for Listening!
Any Questions?
@EwanToo on Twitter