Snowflakes in the Cloud Real world experience on a new approach for Big Data
•
1 like•191 views
Session 15. Speaker: - http://www.devfestdc.org/speakers/robert-fehrmann/ Session: - http://www.devfestdc.org/sessions/snowflakes-in-the-cloud-a-real-world-experience-on-a-new-approach-for-big-data/
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Snowflakes in the Cloud Real world experience on a new approach for Big Data
Similar to Snowflakes in the Cloud Real world experience on a new approach for Big Data (20)
More from DevFest DC
More from DevFest DC (12)
Recently uploaded
Recently uploaded (20)
Snowflakes in the Cloud Real world experience on a new approach for Big Data
- 1. Snowflakes in the Cloud Real world experience on a new approach for Big Data Robert Fehrmann Principal Architect @ Snagajob
- 3. About Me ● Master Degree in Computer Science from “Technische Universitaet Braunschweig” ● 25 years building the data tier for applications in different verticals ● Evangelist for polyglot data environments ● Community involvement (MongoDB User Groups / DevOps)
- 4. Agenda ● The Snagajob Story: ○ How did we end up in the Big Data World using Snowflake ● Gotcha’s ○ Interesting stories on using Snowflake
- 5. Funnel Analysis 750 000 postings every day 600,000 unique visitors X% find the posting interesting Y% apply for the posting (candidate) Z% Using Analytics to understand the funnel - Geographical Analysis - Customer Analysis - Historical Analysis - Industry Analysis - Click through rate & abandoning the search - What makes a Posting Interesting, ...
- 6. Event Collection Framework V1 Web WebWeb Message Bus LB Tracking Service Tracking Service Flume Flume Flume Hadoop Hue Impala Report Console SQL-DW Looker Vertica
- 7. Evolution 201620142012 “We want to be a cloud based company” Peter Harris, CEO 2015 Search Continues For a true cloud solution till …. Data warehouse & platform software ( on premise) Vertica Data Warehouse Hadoop Vertica Data Warehouse Move to Cloud Doesn’t solve all problems Hadoop
- 8. Goals for Next Generation Solution ● Horizontal Scalability ● PaaS ● Stability ● Ease of Use ● Can’t be more expensive
- 9. Architecture
- 10. Event Collection Framework V2 Web WebWeb Message Bus LB Tracking Service Tracking Service FiveTran Salesforce Netsuite Kenisis Snowflake Looker Snowflake Portal AdHoc Spark MongoDB
- 12. Results: Better Use of Resources
- 13. Gotcha #1 ● Problem: ○ Funnel Analysis got slower over time ● Base Metrics ○ 15 Billion rows ○ Analysis on monthly dataset: 2 minutes per run ○ 3 medium clusters in DW during business hours
- 14. Take 1 10 min !!!
- 15. Take 1 14 Billion row hmmm,...
- 16. Take 2
- 17. Take 2 3 min !
- 18. Take 2 Yeah ,... Hmm ,...
- 19. Take 3
- 20. Take 3 7 secs !!!
- 21. Take 3
- 25. SCD Type 2 / Mapping / View View Mapping Table N-Key S-Key .. N-Key-1 S-Key-2 N-Key-2 S-Key-5 Fact Table S-Key N-Key ... S-Key-1 N-Key-1 S-Key-2 N-Key-1 S-Key-3 N-Key-2 S-Key-4 N-Key-2 S-Key-5 N-Key-3
- 26. What’s going on: Case 2 Snowflake
- 27. Other Features ● Undrop (DB, Table, Schema) no restore required ● Clone (DB, Table, Schema) (metadata only operation) ● Native JSON Parsing (as well as CSV, AVRO, XML, Parquet) ● Automatic Encryption of Data ● Automatic Query Optimization (no tuning) ● All Data in one place (single source of truth)