Presentation from the Data Lake Monsters meetup devoted to building big data solutions in the cloud

1. Data Lake
Make data pleasant to swim in

2. Data Warehouse
App/Module 1
Ideal world
App/Module 2
App/Module 3
Operational
storages
One version of the Truth
Reporting
oriented
Data Mart 1
Data Mart 2
Data Mart 3

3. Real world

4. From engineering point of view
• Multiple independent apps/modules (data stores)
• Data is not consistent across apps
• Heterogeneous storage types
• Different historical data storage policies
• Each module has separate team which owns it

5. From business point of view
• Same domain
• Same customer base (at least partially)
• We have all the necessary data to provide more sophisticated
reporting

6. That results in requests like
• Can I build a couple
of custom reports
for sales demo?
• We need more
visibility across the
board
• Hospital X is our
major customer, lets
give them integral
insight

7. Oh, that is simple!
We need MORE ETLs

8. I like it, but…

9. Major issues with ETL-based approach
• Long feedback loop for customer-facing development (demo, custom
reports, on-demand requests)
• Consumes a lot of engineering resources
• Code-base growth
• Unnecessary load on data storages in production
• High cost

10. Seems that we need generic data source
• Should contain all of the data
• SQL-like interface
• Fast enough for on-demand analysis
• Cheap

11. Solution options – Data Warehouse
Pros
• Storage with predefined
structure
• One version of truth
• Contains all the data
• SQL-based interface
Cons
• Long time to implement
• High cost
• Requires new development to
support new data

12. Solution options – Data lake
BLOB Store SQL-based Engine

13. Solution options – Data lake
Pros
• Fast to implement
• Cheap
• Contains all the data and more
• SQL-based interface
• Easy to add new data source
Cons
• Semi-structured data
• Can’t be used as a source of truth
• Data consistency is poor

14. Product
Domain: Healthcare
Product: Provide deep insight into patient experience
Project goal:
1. Gather patients feedback
2. Provide deep insight into feedback by means of reporting

15. DI v3
Product pipeline
Sampling
Data ingestion
Data Mart v3
Data Mart v2
Data Mart v1
Sampling Outreach Feedback Reporting
DI v2
DI v1
Outreach v3
Outreach v2
Outreach v1
Feedback

16. Processing pipeline
Active versions Storage Stack Throughput/DB Size
Data Ingestion 3 MySQL
MS SQL 2012 R2
~2000K/day
~100 Gb
Sampling 1 MS SQL 2005
MS SQL 2008
~2000K/day
Outreach 3 MS SQL 2008
MS SQL 2012 R2
DynamoDB
~300K/day
~600 Gb
Feedback 1 MS SQL 2012 R2
DynamoDB
~60K/day
~100 Gb
Reporting 3 MS SQL 2012 R2
Vertica 7.x
Full refresh daily
~300 Gb MS SQL
~300 Gb Vertica

17. Here comes The Data Lake

18. Stage 1 – Fill the lake

19. Stage 1 – Config

20. Stage 1 - Results
Pros
• Took about a month to implement
by a team of 2 people
• Data is there
• Cheap
• Fast enough
• Analysts team can start playing
with it
Cons
• Data quality
• Data is up to yesterday
• Overall pipeline performance

21. Stage 2 – Make it pleasant to swim in
Operational storages One version of the Truth Reporting oriented

22. Stage 2 - Results
Cons
• Data is up to yesterday
• Overall pipeline performance
Pros
• Implemented by Analysts team
with support of 1 engineer
• Data quality is there
• Doesn’t require engineering
involvement to produce new data
sets

25. Applications
• Integral view of the platform
• Exploration playground for analytics team
• Great basis for DM implementation
• Debug and troubleshooting

26. What next?
• Integration with NoSQL storages
• Incremental load
• Streaming directly to DL

27. Summary
• DL core is a BLOB storage + SQL-based engine
• DL will not resolve data quality issues automatically
• DL is not a replacement for Data Warehouse/Data
Mart
• There should be a strong use case
• It provides a lot of possibilities for exploratory use
cases
• Gives you cheap and performant enough playground
to build new reports
• Easily extendable with new data