Downloaded 165 times
Uploaded byMatillion
Master the Multi-Clustered Data Warehouse - Snowflake
The document discusses the challenges faced by data professionals and emphasizes the need for a technology that effectively connects the diverse data ecosystem known as the 'data nation.' It highlights how Snowflake, in collaboration with Matillion, offers cloud-native solutions for data integration, analytics, and live data sharing while minimizing management efforts and costs. The presentation showcases real-world examples of organizations benefiting from Snowflake’s capabilities, particularly in scalability and performance.
More Related Content
Similar to Master the Multi-Clustered Data Warehouse - Snowflake
Master the Multi-Clustered Data Warehouse - Snowflake
- 1. 1© 2018 SnowflakeComputing Inc. All Rights Reserved. Y O U R D A T A , N O L I M I T S Master the Multi-Clustered Data Warehouse Alex Letterese, Account Executive Matillion
- 2. 2© 2018 SnowflakeComputing Inc. All Rights Reserved. Our presenters Alex Letterese Account Executive Craig Collier Global Solutions Architect Kalyan Arangam Solution Architect
- 3. 3© 2018 SnowflakeComputing Inc. All Rights Reserved. Uniting the Data Nation
- 4. © 2018 SnowflakeComputing Inc. All Rights Reserved. DataPeople Applications What is Data Nation?
- 5. © 2018 SnowflakeComputing Inc. All Rights Reserved. The people in data nation are divided People who analyze data People who store and curate data People who secure the data People who have to get new data People who want to upgrade the data platform The people who want to share the data People who need new data immediately People who pay for the data platform
- 6. © 2018 SnowflakeComputing Inc. All Rights Reserved. Their data is disparate Structured data Semi- structured data Data mart for sales Data mart for ops Corporate data warehouse Customer data Partner data Internal data Streaming data Static data
- 7. © 2018 SnowflakeComputing Inc. All Rights Reserved. The apps are the source of many problems Concurrency problems when too many people use the database Database that can only support a defined amount of structured data Unsecure and challenging methods for sharing data Manual, maintenance intensive tools Expensive up-front database purchases Specialized skills needed for different tools
- 8. © 2018 SnowflakeComputing Inc. All Rights Reserved. There are fresh challenges on the horizon Need it all in real time for up to the minute insight Expanding to new markets with different regulations Applications in cloud and on-prem
- 9. © 2018 SnowflakeComputing Inc. All Rights Reserved. Data People Applications WE NEED A TECHNOLOGY THAT CONNECTS THE DATA NATION GLOBALLY
- 10. © 2017 SnowflakeComputing Inc. All Rights Reserved. Data People Applications THE CLOUD IS A PERFECT CANVAS TO MAKE THOSE CONNECTIONS
- 11. Snowflake + Matillionhelp Unite the Data Nation Data Integration Business Intelligence & Analytics Data Warehouse Enterprise apps Data Sources Corporate Web Mobile IoT
- 12. © 2018 SnowflakeComputing Inc. All Rights Reserved. Complete SQL Database Minimal Management All of Your Data All of Your Users How can Snowflake help Unite the Data Nation? Pay Only for What You Use Live Data Sharing
- 13. © 2018 SnowflakeComputing Inc. All Rights Reserved. Complete SQL Data Warehouse STANDARD SQL Full DML (read/write/modify), analytic SQL BROAD ECOSYSTEM Leverage the skills and tools you already have – BI, ETL… AVAILABILITY Automatic data replication, online recovery, high availability and geo redundancy SECURITY Automatic encryption, third-party certification
- 14. © 2018 SnowflakeComputing Inc. All Rights Reserved. 14 Minimal Management NO Infrastructure NO Tuning NO Optimization NO Indexing NO Storage worries NO Vacuuming NO Partitioning NO Required sorting NO Workload mgmt. NO Manual backups It Just Works!
- 15. © 2018 SnowflakeComputing Inc. All Rights Reserved. asdfAll your data Structured Semi-structuredCSV JSON Avro XML Parquet
- 16. © 2018 SnowflakeComputing Inc. All Rights Reserved. asdf Data Science SQL Analysts ETL & Processing BI & Analytics Tools 16 All Your Users Database Services
- 17. © 2018 SnowflakeComputing Inc. All Rights Reserved. asdf Traditional databases are inflexible Snowflake uses the cloud to enable elasticity Usage varies Pay for only what you use with no overprovisioning Eliminate overbuy Instant Elasticity No need for capacity planning, make capacity decisions on the fly 17 Only pay for what you use
- 18. © 2018 SnowflakeComputing Inc. All Rights Reserved. 18 Seamless data sharing Data Provider Snowflake Account Data to Share Share Live Data Instantly Grant Secure Row Level Access Use One Share for Many NO FTP or EDI NO Duplicate Files or Storage NO Additional Charge Secure Data Share Data Science WH Consumer 2 Snowflake Account Wants provider data for data scientists Consumer 1 Snowflake Account SQL Analysts WH Wants provider data for SQL analysts Live Data Sharing
- 19. © 2018 SnowflakeComputing Inc. All Rights Reserved. Database Services Data Science (L) Data Transformation (M) SQL Queries (S) How is this possible? A Brand New Architecture Easy-to-use Service with No Management Unlimited Compute Clusters to Serve Every Use Flexible Cloud Storage For All Kinds of Data Managemen t Optimization Securit y Availabilit y Transaction s Metadata BI/Reporting (Multi-Cluster M) Structured Semi-Structured 19 Infinite Concurrency
- 20. © 2017 SnowflakeComputing Inc. All Rights Reserved. Uniting your data nation with Snowflake and Matillion? +
- 21. © 2018 SnowflakeComputing Inc. All Rights Reserved. Matillion ETL for Snowflake • Cloud native Data integration tool built specifically for AWS and Snowflake • Push-down ELT architecture • Intuitive browser-based UX • Retail like acquisition through AWS Marketplace – pay by hour About Matillion
- 22. Customer - UpsideTravel Matt Boegner Data Engineer, Upside Travel Scenario Upside ingests over 3TB of raw flight/hotel inventory on a daily basis, in addition to website, mobile app, and marketing data streams. A lot of data and a lot of data processing! Negative Consequences Existing data warehouse couldn’t keep up Degraded performance due to concurrent loading Expensive to maintain After Matillion ETL for Snowflake made their migration to Snowflake simple and quick, and Upside was able to actualize the benefits of Snowflake within 2 weeks, at production workloads. “Use Snowflake if you have a cloud data warehouse; use Matillion if you use Snowflake.”
- 23. Snowflake Flight/ Hotel data Websitedata Mobile data Marketing data Looker – BI & Visualization Better performance, mitigation of concurrency issues and improved scalability • Migration to production live in under 2 weeks • Scalability (up and down) >> Cost Efficiency (Transform applied directly)
- 24. Matillion ETL forSnowflake • 5 minute launch process on Marketplace • 14 day Free Trial • From $1.37 per hr How to engage • SA team to support PoCs, demos and training • Support Portal • Tutorials & Customer references on YouTube How to Engage
- 25.
- 26. Use bulk loadingto get the data to the cloud Leverage the scalable compute layer to do the bulk of the data processing Isolate load and transform jobs from queries to prevent resource contention Focus on organizing the data to support business users © 2018 Snowflake | All Rights Reserved. MASTER THE MULTI-CLUSTERED DATA WAREHOUSE Best Practices
- 27. 27© 2018 SnowflakeComputing Inc. All Rights Reserved. Q & A
- 28. © 2017 SnowflakeComputing Inc. All Rights Reserved. How will you unite your data nation with Snowflake and Matillion? Ask us how to start today. + #BetterTogether #UniteTheDataNation
Editor's Notes
- #3 <Alex> Introduce our speakers – pass it over to them to do brief intros.
- #5 It’s all of us Everyone who cares about working with data Better answers to drive the business forward Whether you are small or big, data nation is for you
- #6 Data is critical, but it’s a challenging space. Every org struggling to get value from data. Divide: People who are responsible for the data, and the people who are using the data Irordinate amount of time wrangling data
- #7 Different types It’s all over the place and difficult to bring together Data isn’t just about whats in front of you
- #9 Connecting different applications GDPR and other regulations Things that used to take 24 hours, now we need under a minute or less
- #11 Effectively infinite resources - Serve every use case with unbounded compute Connect all of your apps with limitless connectivity Unite your data in one place with infinite storage
- #13 Basically skip this slide
- #14 Familiar! The language of analytics. Name other products Not a query tool ETL can work with Snowflake, BI works with Snowflake -The difference is that it SCALES.
- #15 A huge difference: built as a cloud saas service You load the data, you run the queries, and the rest we take care for you Hard to grasp… But it just works!
- #16 Many different types of data Complete picture requires multiple variations of data Different environments (SQL Hadoop) Transformation… requires effort Machine generated data has structure! Has a schema. -Heirarchical, repetitive aptterns -Just changing… -Analytics team unaware -Can you find that schema? We find it for you. Virtual column, access with SQL with dot notation
- #17 Anyone has the rights can gain insight Make decisions and share Ability to allow all of your end users when they want it -Always a food chain – who is on top? -Don’t touch the data warehouse!
- #18 Monday morning dashboard rush Thousands of nodes at once With Snowflake you only pay for what you use. Compute is separate from storage so you pay for things separately. Snowflake is a fully elastic cloud data warehouse—scale up and down at any time. The cost of storage is comparable to the price of Amazon S3—with a one-year contract your cost of storage is only $23 / compressed TB In other words, you can effectively store data in the world’s most powerful data warehouse for the same price as Amazon S3.
- #19 Leverage data as a business asset Companies need data to gain insight into their own company -Drive revenue -Competitive insight -Closer partnerships Snowflake enables any data stored in Snowflake, Securely share with anyone -Your account or other Snowflake customers Your data, you have complete ownership, pay for storage Enable other organizations access to the data through secure view When they run queries, they do that in their own account We don’t participate
- #20 That’s not possible! Multi cluster shared data At the heart of the architecture is the data itself -Blobs within S3 (storage engine) Immutability is a tremendous advantage. When you load its loaded into S3 as a set of files -As the data is brought in, we extract metadata and put it into columns and micropartions for pruning -Don’t declare partition keys -Tens of millions of partitions To do work, you need a virtual warehouse. -Cluster of compute, EC2 nodes on AWS -They do all the processing -Different sizes -Multi-cluster – more than one cluster at a time. Concurrency. -Servcices, the layer that makes this all possible When you login online, you authenticate and it passes through services When you issue a query, we parse your query, perform the right prune, and efficiently scan the data for you When a multi statement transaction starts, every other query will see the data as it was before it began… until the query is done then they see the updated version It all works because S3 is immutable!