A 30 day plan to start ending your data struggle with Snowflake
•Download as PPTX, PDF•
2 likes•2,687 views
This document outlines a 30-day plan to address common data struggles around loading, integrating, analyzing, and collaborating on data using Snowflake's data platform. It describes setting up a team, defining goals and scope, loading sample data, testing and deploying business logic transformations, creating warehouses for business intelligence tools, and connecting BI tools to the data. The goal is that after 30 days, teams will be collaborating more effectively, able to easily load and combine different data sources, have accurate business logic implemented, and gain more insights from their data.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to A 30 day plan to start ending your data struggle with Snowflake
Similar to A 30 day plan to start ending your data struggle with Snowflake (20)
Recently uploaded
Recently uploaded (20)
A 30 day plan to start ending your data struggle with Snowflake
- 1. 1© 2017 Snowflake Computing Inc. All Rights Reserved. Y O U R D A T A , N O L I M I T S A 30 Day Plan to End Your Struggle for Data
- 2. Common data struggles © 2016 Snowflake Computing Inc. All Rights Reserved. 2 AnalyticsData IntegrationData Loading Collaboration
- 3. © 2016 Snowflake Computing Inc. All Rights Reserved. 3 Data Loading
- 4. © 2017 Snowflake Computing Inc. All Rights Reserved. 4 Struggle to Load Data Resource Contention Capacity Planning Preparing disparate data to load “Where can I connect to that new JSON web log data?” -BI Team – Have to flatten to store semi-structured (or use noSQL) – Storage and compute are limited – Architecture forces linear compute capacity
- 5. © 2017 Snowflake Computing Inc. All Rights Reserved. 5 Tackle loading challenges with Snowflake Contention Capacity Disparate data – Variant column type supports semi-structured – No more flattening (unless you want to) – Built on the cloud (S3, EC2) – Scale data and compute to load any data – Unlimited virtual warehouses allow independent compute – Isolate loading and other tasks
- 6. © 2016 Snowflake Computing Inc. All Rights Reserved. 6 Data Integration
- 7. © 2017 Snowflake Computing Inc. All Rights Reserved. 7 Struggle to Integrate Data Making sense of data in silos Editing and transforming data Support evolving business logic and disparate use cases – Hard to transform different datasets while in different silos/formats – noSQL tools complex, not all data stores ACID complaint – Contention an issue while transforming – No way to easily experiment with and add business logic – Different people have different use cases “Are the updated KPI’s in the sensor data tables?” - Data scientist
- 8. © 2017 Snowflake Computing Inc. All Rights Reserved. 8 Improve data integration with Snowflake Silos Editing and transforming Business logic – Native storage for semi-structured, ANSI standard SQL and dot notation to use it – Combine all of your data fluidly – ACID compliant with virtual data warehouses – Edit, transform, insert, delete, however or whenever you want – Zero-copy cloning – Rapidly iterate, test and promote business logic for multiple people
- 9. © 2016 Snowflake Computing Inc. All Rights Reserved. 9 Data analytics
- 10. © 2017 Snowflake Computing Inc. All Rights Reserved. 10 Struggle to Analyze Data Queues Delays – Analysts are always the end of the resource priority queue – Even with unlimited access, database is non-performant “How come the dashboard isn’t working?” - Sales director
- 11. © 2017 Snowflake Computing Inc. All Rights Reserved. 11 Analyzing Efficiently with Snowflake Queues Delays – Independent virtual warehouses – Scale up, down or out to serve analytics use cases – Autoscaling and multi-cluster warehouses – Automatically match compute to even massive demand
- 12. © 2016 Snowflake Computing Inc. All Rights Reserved. 12 Collaboration
- 13. © 2017 Snowflake Computing Inc. All Rights Reserved. 13 Struggle to Collaborate Incessant fixing Siloed teams – Fixing loading, integration and analytics struggles burns time – Conflicts from those struggles reduce morale – Technical and business teams often not working together (physically or otherwise) “I’m so buried under this queue I can’t make the BI standup” - IT team member “I could ask IT for an updated table, but I’m not sure who was working on it.” - BI team member
- 14. © 2017 Snowflake Computing Inc. All Rights Reserved. 14 Start Collaborating with Snowflake Fixing Siloed teams – Address the other struggles as referenced – Free more time for collaboration and discussion – With new time, start new discussions around data – Build updates and additions into a scheduled meet-up
- 15. © 2016 Snowflake Computing Inc. All Rights Reserved. 15 A 30-day Plan to Start Ending Your Struggle with Snowflake
- 16. © 2016 Snowflake Computing Inc. All Rights Reserved. 16 Start from the beginning – what’s the analytics goal? 1. Define the team 2. Discuss blocking issues and a place to start 3. Define the scope 4. Define success criteria 5. Try Snowflake On-Demand 6. Plan status updates going forward Start Ending Your Data Struggle – Week 1
- 17. © 2017 Snowflake Computing Inc. All Rights Reserved. 17 Start Ending Your Data Struggle – Week 2 1. Find data to load 2. Create a Warehouse 3. Load data – Work within defined scope, agree as a team – Use data that’s new, challenging, or semi-structured – Will need this to load data – Create a database and a table – Stage your data – Load from stage to database
- 18. © 2017 Snowflake Computing Inc. All Rights Reserved. 18 Start Ending Your Data Struggle – Week 3 1. Test and deploy business logic 2. Optional: Create Integration WH 3. Optional: Plan ongoing loading and transform – Discuss metrics, KPIs, transformations to add – Use zero-copy cloning to test and then promote – Isolate integration and transformation – Use zero-copy cloning to test iterations safely and promote
- 19. © 2017 Snowflake Computing Inc. All Rights Reserved. 19 Start Ending Your Data Struggle – Week 4 1. Create Warehouses for BI 2. Create analytics users 3. Connect your BI to Snowflake – Avoid queues with isolated compute resources – Optionally, set up auto-scaling – Spread the value of the data – Use this as an opportunity to share and discuss – Use Tableau, Looker, etc. to query your data live – Consider publishing dashboards with live connect
- 20. © 2017 Snowflake Computing Inc. All Rights Reserved. 20 After 30 days you should see improvements 1. Your team should be talking and collaborating more 2. You should be able to easily load and combine data 3. You should have accurate business logic in your data 4. You should be finding more insight TRY SNOWFLAKE FOR FREE
Editor's Notes
- Data loading – struggle to load, store and manage data Data integration – struggle to unify and integrate disparate data sources Analytics – Struggle to analyze data quickly and effectively Collaboration – Because your spending so much time on the other three problems, its difficult to get everyone on the same page, to work together to find insight in your data
- Preparing disparate data to load The struggle to load data begins with the need to prepare disparate datasets to load. Many organizations are dealing with a host of new semi-structured data in formats like JSON and Avro that require flattening to load into a relational database. Or, they choose to store semi-structured data separate from relational data in a NoSQL store, creating silos. Capacity planning Finding space for data can be another enormous challenge. Large numbers of complex datasets can quickly snowball into a storage capacity issue on fixed size on-premises or cloud data platforms. Resource contention Loading large datasets also requires significant compute capacity. Many data warehouses are already strained under normal business workloads, and the compute needed for loading forces those other processes to be pushed back in the priority queue. All of these problems lead to difficult conversations about whose data or use case is most important. One project might need funding for an open source, semi-structured data store. Another wants to expand the on-premises data warehouse. One team wants to load clickstream data, and another needs finance data. Prioritizing completely different needs can be a minefield that leads to a host of struggles within and between teams.
- Tackle loading challenges with Snowflake Snowflake addresses each loading challenge with simplicity. Semi-structured data can be loaded natively alongside structured data, and queried together in one location. Because Snowflake’s built on the cloud, you can store as much data as you want with no need to prioritize different datasets. Best of all, you can create independent compute resources, called virtual warehouses, for each of your use cases, negating the need for queues.
- Making sense of data in silos With data scattered across NoSQL data lakes, cloud applications, and data warehouses (not to mention flat files and CSVs), organizations are struggling to combine and analyze their data in one cohesive picture. Editing and transforming data Every system that stores data has it’s challenges, but many organizations are finding it particularly hard to analyze and understand data in NoSQL systems like Hadoop. Semi-structured open source data stores require a large amount of custom configuration, uncommon skillets, and transformation to successfully combine with other business data. They also rarely support edit, update and insert commands that are essential to data modeling and transformation. Supporting evolving business logic and disparate use cases It’s hard for the business to drive evolutions in business logic within the database when it takes arduous manual process to test and update. Often, entire databases need to be physically copied in order to test a simple change to a table or derived field, which can be extremely expensive and time consuming. Because different people within the organization have different data needs, a “single source of the truth” is often too ungainly and impractical for most organizations to maintain and use. All of these problems make it difficult to generate a refined view of what the data actually says. Differing methods of transforming data arise, with competing factions struggling to promote their own methods of working with, storing and querying data. People from throughout the business wonder where they can find the “right” version of their metrics and KPIs.
- Improve data integration with Snowflake Snowflake makes data integration straightforward. You can load all of your data, in almost any structured or semi-structured format, so you can avoid data silos. Transforming is made easier with ANSI standard SQL and dot notation for semi-structured data. Inserts, deletes and other common operations are fully supported. You can even rapidly test and update with zero-copy cloning, driving faster iteration in business logic.
- Queues Analytics users are always at the bottom of the resource priority queue. It’s not always designed to be that way, but if ETL, as a simplified example, needs to run for 45 minutes every hour, then there’s little time left over for the analytics team to access and iterate on the database. Delays Through the eyes of an analyst, nothing ever works fast enough. But, often disappointing performance isn’t for lack of trying. Many data warehouses require hours and hours of painstaking optimization, tuning, indexing, sorting, and vacuuming from a dedicated data engineer. To add to the pain, often one optimization will lead to deoptimization in another area The struggle to analyze data is one of the most visible. Report consumers complain that the BI tool isn’t working fast enough. The BI team points their finger to the data engineers. But, at the end of the day, antiquated database technology is the real culprit.
- Analyzing efficiently with Snowflake Snowflake addresses efficient analytics in two ways. As we saw before, independent virtual warehouses can help with concurrent queries, allowing ETL and BI to run side by side at the same time. Large or variable analytics workloads within a single warehouse can be dealt with using mutli-cluster warehouses, and even autoscaling to automatically match your compute resources to need.
- The struggle to load, integrate and analyze data leads to a fourth struggle that’s often the worst. Collaboration.
- Incessant fixing If the organization spends all its time endlessly solving loading, integration, and analytics struggles, it’s impossible to break away and think at a higher level about what needs to be accomplished. Data is a constant flash point of disagreement, rather than a rallying point for collaboration. Siloed teams Historically, there’s been a dividing line between technical, IT implementers, and less-technical business side consumers. This was partly driven by technology, but reinforced by organizational structures that don’t favor cross team collaboration. The lack of collaboration is the end result of the struggles, and the most frustrating of them all. How can two disparate types of people, on two different teams (or multiple different teams) effectively work together when they are completely buried under the weight of their antiquated data platform.
- Analyzing efficiently with Snowflake As we noted previously, Snowflake can help to solve loading, integration and analytics struggles, freeing time for collaboration and higher level planning. Working together with Snowflake, the dividing line between IT and BI becomes less important. IT can lead the business with technology and empower the BI team to analyze data. On the same token, with more accessible technology in the form of Snowflake, BI teams can take an active role in the curation and modeling of data that has historically rested solely on IT’s shoulders.
- Week one is all about the team. It’s time to bring everyone around the same table to figure out the best way to move forward with your data. Keep your conversation focused on an achievable goal: trying to get an important dataset into Snowflake for analysis. Discuss blocking issues, but be sure to define them in terms of technology, rather than people. Once you’ve got a plan to get around any blocking issues, set up Snowflake On-Demand for free and make a plan to bring the team together for status updates in the weeks to follow. Pro tip: Think big. Every new Snowflake On-Demand customer gets $400 in free credits to play around with, more than enough to load and store a massive dataset. One Snowflake customer performance tested Snowflake against a $10,000,000 on premise database with only $100. It was 100x faster.
- Week 2 is when the practical, real life work begins. Pick up where you left off with your team, and discuss the right data to load into Snowflake. Clearly define the scope within that dataset, so you settle on a dataset that is large enough to be useful but also flexible enough to get out of it’s current location within the week. Once you’ve got your data, it’s time to create a warehouse, database and tables to load your data into. Pro tip: Remember to stay open minded about semi-structured data too - in fact, that might be the best dataset to get started with in Snowflake. Store semi-structured data in nested form within the Variant type column, and transform with dot notation using standard SQL statements.
- By week 3, you should have data loaded and perhaps you’ve already started querying and using it. If not, now is a good time to start. Make sure to take note of the business logic (in the form of calculations, derived fields, KPIs, etc) that it would make sense to add. Work with the team to futher define this logic, and experiment with zero-copy cloning to test transformations to your production data from the safety of a cloned database. When you’ve got your business logic added, look to add an additional warehouse for ongoing loading and transformation needs. Pro tip: The value of Snowflake increases exponentially with the number of related data sources you are able to load and integrate. In other words, sales data from Salesforce is more than twice as interesting when people are able to combine it with account based web interaction data from Google Analytics.
- As week 4 rolls around, it’s time to spread the value of your data as widely as possible. Add users to Snowflake, along with roles and permissions to match. Create auto-scaling warehouses for the BI, analytics and reporting teams to enable everyone to access data without contention. Connect Snowflake to your BI tool to begin creating the visualizations and dashboards that will power the insight you need. Pro tip: Many organizations that have traditionally relied on extracts or in-memory data are using Snowflake as a live-connection within their BI tool. Experiment and take advantage of the speed and flexibility that Snowflake can give your team.
- After 30 days, you should see some significant improvements. Your team should be talking about your data and collaborating more. You should be able to easily load and combine the data that matters to your business. There should be useful business logic within the data you loaded into Snowflake, and plans to test and expand even more. Your BI and analytics should be performing quickly on the data you’ve loaded, generating further interest in your overall plans for your data platform. The most important change you should see after this 30 day plan is within your relationships. The struggles that defined your loading, integration, analytics and collaboration should have given way to a new but promising spirit of mutual ownership. Next steps The next steps are up to you, but they look a lot like the first 30 day plan in elongated form. Continue the discussion. Load more data. Expand the number of users and groups that can access and benefit from the data that you’ve loaded in Snowflake. It’s also important to continually share and elevate the success and experiences you’ve had ending the struggle for data within your organization. Show executives and leaders the value of your data, and the time that you’ve put into perfecting it for analysis. Lastly, make sure to share your experiences outside of your own organization. Speak at conferences and events so you can synthesize what you’ve learned and spread the benefit of your experience to people that are still struggling with their data.