Dive Into Data Lakes

•Download as PPTX, PDF•

0 likes•421 views

The document is a presentation on data lakes given by James Johnson and Paul Johnson. It discusses what a data lake is, how it compares to a data warehouse, best practices for data lake architecture, and demonstrations of Matillion ETL tools for loading data into Amazon Redshift, BigQuery and Snowflake. The agenda includes defining a data lake, comparing data lakes to data warehouses, demonstrating Matillion with various data sources, and how to engage with Matillion for trials or support.

Software

1
www.matillion.com
© 2017 Matillion. All rights reserved.
Presented by:
Copyright © 2017. All rights reserved. Matillion, trademarks, registered trademarks or
service marks are property of their respective owners. 10/10/2017
Dive into Data Lakes
James Johnson and Paul Johnson
Wednesday, April 25th, 11am EDT

2
www.matillion.com
© 2017 Matillion. All rights reserved.
Who we are
• AWS Advanced Technology Partner and Big Data
Competency Holder
• Google Cloud Platform Partner
• Matillion Products
- Matillion ETL for Amazon Redshift
- Matillion ETL for Snowflake
- Matillion ETL for BigQuery
• Over 60 5* reviews on the AWS Marketplace

3
www.matillion.com
© 2017 Matillion. All rights reserved.
Agenda
• What is a Data Lake?
• Data Lakes vs. Data Warehouses
• Data Lake Best Practice & Reference Architecture
• Matillion Demo
- Amazon Redshift Spectrum (ELT using S3)
- BigQuery (ELT using Google Cloud Storage)
- Snowflake (ELT with JSON files)
• How to Engage

4
www.matillion.com
© 2017 Matillion. All rights reserved.
What is a Data Lake?
Data MartData Lake

5
www.matillion.com
© 2017 Matillion. All rights reserved.
Characteristics of a Data Lake
• Data is stored in native format
• Stored forever
• Flexible access
• Schema-on-Read
• Data Lake is not an S3/Cloud
Storage bucket
• It’s not a dumping zone
• Free-for-all access

6
www.matillion.com
© 2017 Matillion. All rights reserved.
Data Lake vs. Data Warehouse
Complimentary or Contradictory?

7
www.matillion.com
© 2017 Matillion. All rights reserved.
Data Lake
• Data is stored in native format
• Store forever
• Flexible access to raw data
• Schema-on-Read
• Seperate storage & compute
• Data requires transformation
• Expensive to store large volumes
• Transformed before loading (ETL)
• Schema-on-Write
• Tightly coupled storage & compute
Traditional On-Premise Data
Warehouse

8
www.matillion.com
© 2017 Matillion. All rights reserved.
ETL Pipeline

9
www.matillion.com
© 2017 Matillion. All rights reserved.
Data Lake
• Data is stored in native format
• Store forever
• Flexible access to raw data
• Schema-on-Read
• Seperate storage & compute
• Support for semi-structured data
• Store forever
• Transformed after loading (ELT)
• Schema-on-Write & Schema-on-Read
• Seperate storage & compute
Modern Cloud Data Warehouse

10
www.matillion.com
© 2017 Matillion. All rights reserved.
ELT Pipeline

11
www.matillion.com
© 2017 Matillion. All rights reserved.
Data
Sources
ERP
CRM
Logs
Devices
Ingest
Files
Streams
Records
Presentation
Layer
Layer
Data Scientist
Data Analysts
Business
Users
Applications
Transactions
Data Lake
Process
Automation
Landing Zone
Trusted Zone
Consumption Zone
Refined for specific use.
Aggregated,
Denormalized etc.
Sandbox
Exploratory Analytics
Governance Tools
Master Data
Reference Data
Immutable
log
Metadata
& Lineage
Data
Quality
Data
Catalog Security

12
www.matillion.com
© 2017 Matillion. All rights reserved.
Data Lake Architecture Best Practices
Decouple storage & compute
• Data -> Store -> Process -> Store -> Analyse -> Answers
• Allows huge volumes of data to be stored without paying for compute
• Allows replacement of processing technology (e.g. Spark)
Use the right tool for the job
• Data structure, latency, throughput, access patterns
Log-centric design patterns
• Immutable logs, materialized views
• Delete nothing.
• Rebuild your data at a point in time
Leverage serverless or managed services
• Scalable/elastic, available, reliable, secure, no/low admin

13
www.matillion.com
© 2017 Matillion. All rights reserved.
Demonstration

14
www.matillion.com
© 2017 Matillion. All rights reserved.
• Available only on AWS Marketplace/Cloud
Launcher
• 14 day free trial and test drives
• SA team to support PoCs, demos and training
• Customer reference videos on YouTube
How to Engage

15
www.matillion.com
© 2017 Matillion. All rights reserved.

16
www.matillion.com
© 2017 Matillion. All rights reserved.
Presented by:
Copyright © 2017. All rights reserved. Matillion, trademarks, registered trademarks
or service marks are property of their respective owners. 7/23/2018
Thank You
James Johnson and Paul Johnson

Parallel processing, columnar structure, and petabyte scaling make Amazon Redshift a powerful data warehousing platform. Get the full value of your company's analytics with Amazon Redshift and Matillion ETL. - Get an overview of the latest Amazon Redshift features - Find out how massive amounts of data can be loaded in a matter of minutes with Matilion ETL - Learn best practices for cloud data warehousing with Amazon Redshift Matillion ETL is the fastest, easiest, and most efficient way to leverage these benefits and achieve the full potential of your business.

ELT vs. ETL - How they’re different and why it matters

Matillion

ELT is a fundamentally better way to load and transform your data. It’s faster. It’s more efficient. And Matillion’s browser-based interface makes it easier than ever to work with your data. You’re using data to improve your world: shouldn’t the tools you use return the favor? In this webinar: - Explore the differences between ELT and ETL - Learn why ELT is a better, more modern process - Discover the latest trends in ELT and how they apply to your business - Find out how Matillion ETL makes loading large amounts of data easier

Simplifying Your Journey to the Cloud: The Benefits of a Cloud-Based Data War...

Matillion

As companies grow, so does the volume of their data. Without the proper solutions in place to quickly store, measure and analyze that data, its usefulness quickly declines. See our latest webinar to learn about how companies are increasingly turning towards cloud-based data warehousing to derive more value out of their data and apply their findings to make smarter business decisions. The webinar covers core topics including: - The benefits of using Snowflake’s unique architecture for interacting with data. - How Matillion can help you quickly load and transform your data to maximize its value. - Expert advice on how to apply data warehousing and ETL best practices. Watch the full webinar: https://youtu.be/mIOm3j431OQ

Get Savvy with Snowflake

Matillion

In this webinar you'll learn how to quickly and easily improve your business using Snowflake and Matillion ETL for Snowflake. Webinar presented by Solution Architects Craig Collier (Snowflake) adn Kalyan Arangam (Matillion). In this webinar: - Learn to optimize Snowflake and leverage Matillion ETL for Snowflake - Discover tips and tricks to improve performance - Get invaluable insights from data warehousing pros

Kickstart your data strategy for 2018: Getting started with Amazon Redshift

Matillion

Start planning your data strategy for 2018 with our webinar on Amazon Redshift. Whether you are a seasoned Amazon Redshift user, or just getting started, our webinar slides will show you all the tips and tricks to quickly and easily get moving and start working with your data in Amazon Redshift. Learn how Amazon Redshift can help you quickly and effortlessly analyze your data to make smart business decisions, and how Matillion ETL can help you save time and money by getting your data into Amazon Redshift in just a few minutes. In this webinar learn: - An overview of petabyte scale data warehouses, the architecture, and use cases - An introduction to Amazon Redshift’s parallel processing, columnar, and scaled out architecture - Learn how to configure your data warehouse cluster, optimize your scheme, and quickly load your data - An overview of all the latest features of Amazon Redshift - How Matillion ETL works with Amazon Redshift and can help you load massive amounts of data in minutes

Snowflake Company Presentation

AndrewJiang18

The document discusses Snowflake, a cloud data warehouse company. Snowflake addresses the problem of efficiently storing and accessing large amounts of user data. It provides an easy to use cloud platform as an alternative to expensive in-house servers. Snowflake's business model involves clients renting storage and computation power on a pay-per-usage basis. Though it has high costs, Snowflake has seen rapid growth and raised over $1.4 billion from investors. Its competitive advantages include an architecture built specifically for the cloud and a focus on speed, ease of use and cost effectiveness.

Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...

Amazon Web Services

Snowflake is a cloud-based data warehouse that is built for the cloud. It was founded in 2012 and has raised $1 billion in funding. Snowflake's architecture separates storage, compute, and metadata services, allowing it to offer unlimited scalability, multiple clusters that can access shared data with no downtime, and full transactional consistency across the system. Snowflake has over 2000 customers including large enterprises that use it for analytics, data science, and sharing large volumes of data securely.

Google BigQuery Best Practices

Matillion

In this webinar you'll learn about the best practices for Google BigQuery—and how Matillion ETL makes loading your data faster and easier. Find out from our experts how to leverage one of the largest, fastest, and most capable cloud data warehouses to improve your business and save money. In this webinar: - Discover how to work fast and efficiently with Google BigQuery - Find out the best ways to monitor and control costs - Learn to leverage Matillion ETL and optimize Google BigQuery - Get tips and tricks for better performance

DataStax has an aggressive product strategy to bring together multiple workloads and multiple data modeling options into a single data platform with DSE 5.0. In this session, we will share the DataStax vision of a multi-model database platform that can provide full coverage for your applications in a single product that includes modeling options for OLTP, OLAP, and Search with support for business models that can be Tabular (CQL), Document (JSON), or Graph (highly-connected relationship-based).

Master the Multi-Clustered Data Warehouse - Snowflake

Matillion

Snowflake is one of the most powerful, efficient data warehouses on the market today—and we joined forces with the Snowflake team to show you how it works! In this webinar: - Learn how to optimize Snowflake - Hear insider tips and tricks on how to improve performance - Get expert insights from Craig Collier, Technical Architect from Snowflake, and Kalyan Arangam, Solution Architect from Matillion - Find out how leading brands like Converse, Duo Security, and Pets at Home use Snowflake and Matillion ETL to make data-driven decisions - Discover how Matillion ETL and Snowflake work together to modernize your data world - Learn how to utilize the impressive scalability of Snowflake and Matillion

Altis Webinar: Use Cases For The Modern Data Platform

Altis Consulting

This document discusses use cases for a modern data platform. It begins by outlining the agenda, then provides background on Altis, the consulting firm. The document defines a modern data platform and explains how it differs from traditional setups. It discusses three approaches for selecting initial use cases: lift and shift with a twist, hitting a roadmap milestone, and supporting an organizational strategy. Examples are provided of where each approach has worked and struggled. The document covers design patterns, managing costs, and identifying success criteria for use cases.

Continuous Data Replication into Cloud Storage with Oracle GoldenGate

Michael Rainey

Continuous flow. Streaming. Near real-time. These are all terms used to identify the business’s need for quick access to data. It’s a common request, even if the data must flow from on-premises to the cloud. Oracle GoldenGate is the data replication solution built for fast data. In this session, we’ll look at how GoldenGate can be configured to extract transactions from the Oracle database and load them into a cloud object store, such as Amazon S3. There are many different use cases for this type of continuous load of data into the cloud. We’ll explore these solutions and the various tools that can be used to access and analyze the data from the cloud object store, leaving attendees with ideas for implementing a full source-to-cloud data replication solution. Presented at ITOUG Tech Days 2019

Snowflake + Power BI: Cloud Analytics for Everyone

Angel Abundez

This document discusses architectures for using Snowflake and Power BI together. It begins by describing the benefits of each technology. It then outlines several architectural scenarios for connecting Snowflake to Power BI, including using a Power BI gateway, without a gateway, and connecting to Analysis Services. The document also provides examples of usage scenarios and developer best practices. It concludes with a section on data governance considerations for architectures with and without a Power BI gateway.

SLC Snowflake User Group - Mar 12, 2020

Nathan Skousen

Data Sharing with Snowflake

Snowflake Computing

Every day, businesses across a wide variety of industries share data to support insights that drive efficiency and new business opportunities. However, existing methods for sharing data involve great effort on the part of data providers to share data, and involve great effort on the part of data customers to make use of that data. However, existing approaches to data sharing (such as e-mail, FTP, EDI, and APIs) have significant overhead and friction. For one, legacy approaches such as e-mail and FTP were never intended to support the big data volumes of today. Other data sharing methods also involve enormous effort. All of these methods require not only that the data be extracted, copied, transformed, and loaded, but also that related schemas and metadata must be transported as well. This creates a burden on data providers to deconstruct and stage data sets. This burden and effort is mirrored for the data recipient, who must reconstruct the data. As a result, companies are handicapped in their ability to fully realize the value in their data assets. Snowflake Data Sharing allows companies to grant instant access to ready-to-use data to any number of partners or data customers without any data movement, copying, or complex pipelines. Using Snowflake Data Sharing, companies can derive new insights and value from data much more quickly and with significantly less effort than current data sharing methods. As a result, companies now have a new approach and a powerful new tool to get the full value out of their data assets.

Snowflake: The Good, the Bad, and the Ugly

Tyler Wishnoff

Elastic Data Warehousing

Snowflake Computing

The document discusses elastic data warehousing using Snowflake's cloud-based data warehouse as a service. Traditional data warehousing and NoSQL solutions are costly and complex to manage. Snowflake provides a fully managed elastic cloud data warehouse that can scale instantly. It allows consolidating all data in one place and enables fast analytics on diverse data sources at massive scale, without the infrastructure complexity or management overhead of other solutions. Customers have realized significantly faster analytics, lower costs, and the ability to easily add new workloads compared to their previous data platforms.

Snowflake Overview

Snowflake Computing

Unleashing the Power of your Data

Itai Yaffe

Orit Alul (Sr. Solutions Architect) @ AWS: As data is growing at an exponential rate, we are interested not only in being able to analyze the past or present but also in predicting the future! In this session, Orit will talk about the power of data combined with machine learning. Building a highly scalable and flexible data architecture in the cloud to collect, process, and analyze data, in order to get timely insights and react quickly to new information. In addition, Orit will present best practices, performance and optimization tips for building a Data Lake in the cloud.

Dataiku & Snowflake Meetup Berlin 2020

Harald Erb

From the Data Work Out event: Performant and scalable Data Science with Dataiku DSS and Snowflake Managing the whole process of setting up a machine learning environment from end-to-end becomes significantly easier when using cloud-based technologies. The ability to provision infrastructure on demand (IaaS) solves the problem of manually requesting virtual machines. It also provides immediate access to compute resources whenever they are needed. But that still leaves the administrative overhead of managing the ML software and the platform to store and manage the data. A fully managed end-to-end machine learning platform like Dataiku Data Science Studio (DSS) that enables data scientists, machine learning experts, and even business users to quickly build, train and host machine learning models at scale, needs to access data from many different sources and can also access data provided by Snowflake. Storing data in Snowflake has three significant advantages: a single source of truth, shorten the data preparation cycle, scale as you go.

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

Cathrine Wilhelmsen

Data Warehouse - Incremental Migration to the Cloud

Michael Rainey

A data warehouse (DW) migration is no small undertaking, especially when moving from on-premises to the cloud. A typical data warehouse has numerous data sources connecting and loading data into the DW, ETL tools and data integration scripts performing transformations, and reporting, advanced analytics, or ad-hoc query tools accessing the data for insights and analysis. That’s a lot to coordinate and the data warehouse cannot be migrated all at once. Using a data replication technology such as Oracle GoldenGate, the data warehouse migration can be performed incrementally by keeping the data in-sync between the original DW and the new, cloud DW. This session will dive into the steps necessary for this incremental migration approach and walk through a customer use case scenario, leaving attendees with an understanding of how to perform a data warehouse migration to the cloud. Presented at RMOUG Training Days 2019

AWS Summit Singapore 2019 | Snowflake: Your Data. No Limits

AWS Summits

This document discusses Snowflake, a cloud data platform. It describes Snowflake's mission to enable organizations to be data-driven. It outlines problems with traditional data architectures like complexity, limited scalability, inability to consolidate data, and rigid costs. Snowflake's solution is a cloud-native data warehouse delivered as a service that offers instant elasticity, end-to-end security, and the ability to query structured and semi-structured data using SQL. Key benefits of Snowflake include supporting any scale of data, users and workloads; paying only for resources used; and providing simplicity, scalability, flexibility and elasticity.

Optimize the performance, cost, and value of databases.pptx

IDERA Software

Smartsheet’s Transition to Snowflake and Databricks: The Why and Immediate Im...

Databricks

Altis AWS Snowflake Practice

SamanthaSwain7

Migrating your traditional Data Warehouse to a Modern Data Lake

Amazon Web Services

In this session, we discuss the latest features of Amazon Redshift and Redshift Spectrum, and take a deep dive into its architecture and inner workings. We share many of the recent availability, performance, and management enhancements and how they improve your end user experience. You also hear from 21st Century Fox, who presents a case study of their fast migration from an on-premises data warehouse to Amazon Redshift. Learn how they are expanding their data warehouse to a data lake that encompasses multiple data sources and data formats. This architecture helps them tie together siloed business units and get actionable 360-degree insights across their consumer base.

AWS Storage State of the Union

Amazon Web Services

2017 OpenWorld Keynote for Data Integration

Jeffrey T. Pollock

What's hot

PSSUG Nov 2012: Big Data with SQL Server

Mark Kromer

DataStax: Datastax Enterprise - The Multi-Model Platform

DataStax Academy

Master the Multi-Clustered Data Warehouse - Snowflake

Matillion

Altis Webinar: Use Cases For The Modern Data Platform

Altis Consulting

Continuous Data Replication into Cloud Storage with Oracle GoldenGate

Michael Rainey

Snowflake + Power BI: Cloud Analytics for Everyone

Angel Abundez

SLC Snowflake User Group - Mar 12, 2020

Nathan Skousen

Data Sharing with Snowflake

Snowflake Computing

Snowflake: The Good, the Bad, and the Ugly

Tyler Wishnoff

Elastic Data Warehousing

Snowflake Computing

Snowflake Overview

Snowflake Computing

Unleashing the Power of your Data

Itai Yaffe

Dataiku & Snowflake Meetup Berlin 2020

Harald Erb

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

Cathrine Wilhelmsen

Data Warehouse - Incremental Migration to the Cloud

Michael Rainey

AWS Summit Singapore 2019 | Snowflake: Your Data. No Limits

AWS Summits

Optimize the performance, cost, and value of databases.pptx

IDERA Software

Smartsheet’s Transition to Snowflake and Databricks: The Why and Immediate Im...

Databricks

Altis AWS Snowflake Practice

SamanthaSwain7

What's hot (19)

PSSUG Nov 2012: Big Data with SQL Server

DataStax: Datastax Enterprise - The Multi-Model Platform

Master the Multi-Clustered Data Warehouse - Snowflake

Altis Webinar: Use Cases For The Modern Data Platform

Continuous Data Replication into Cloud Storage with Oracle GoldenGate

Snowflake + Power BI: Cloud Analytics for Everyone

SLC Snowflake User Group - Mar 12, 2020

Data Sharing with Snowflake

Snowflake: The Good, the Bad, and the Ugly

Elastic Data Warehousing

Snowflake Overview

Unleashing the Power of your Data

Dataiku & Snowflake Meetup Berlin 2020

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

Data Warehouse - Incremental Migration to the Cloud

AWS Summit Singapore 2019 | Snowflake: Your Data. No Limits

Optimize the performance, cost, and value of databases.pptx

Smartsheet’s Transition to Snowflake and Databricks: The Why and Immediate Im...

Altis AWS Snowflake Practice

Similar to Dive Into Data Lakes

Migrating your traditional Data Warehouse to a Modern Data Lake

Amazon Web Services

AWS Storage State of the Union

Amazon Web Services

2017 OpenWorld Keynote for Data Integration

Jeffrey T. Pollock

Lets Talk Google BigQuery

Matillion

Google BigQuery is one of the largest, fastest, and most capable cloud data warehouses on the market. In this webinar, we review BigQuery best practices and show you how Matillion ETL can help you get the most out of the platform to gain a competitive edge. In this webinar: - Discover how to work quickly and efficiently with Google BigQuery - Find out the best ways to monitor and control costs - Hear tips and tricks for loading and transforming massive amounts of data in BigQuery with Matillion ETL - Get expert advice on improving your performance in BigQuery for quicker data analysis - Learn how to optimize BigQuery for your marketing analytics needs

Using Data Lakes

Amazon Web Services

Innovation Track AWS Cloud Experience Argentina - Data Lakes & Analytics en AWS

Amazon Web Services LATAM

Data lakes allow organizations to store all types of data in a centralized repository at scale. AWS Lake Formation makes it easy to build secure data lakes by automatically registering and cleaning data, enforcing access permissions, and enabling analytics. Data stored in data lakes can be analyzed using services like Amazon Athena, Redshift, and EMR depending on the type of analysis and latency required.

Unlocking big data with Hadoop + MySQL

Ricky Setyawan

Applying AWS Purpose-Built Database Strategy - SRV307 - Toronto AWS Summit

Amazon Web Services

In this session, we dive deep into applying the "AWS Purpose-Built Database Strategy" to determine which databases to use for which components of your application. Learn how to evaluate a new workload for the best managed database option based on specific application needs related to data shape, data size at limit, computational requirements, programmability, throughput and latency needs, etc. This session explains the ideal use cases for relational and non-relational database services, including Amazon Aurora, Amazon DynamoDB, Amazon ElastiCache for Redis, Amazon Neptune, and Amazon Redshift.

Using Data Lakes

Amazon Web Services

Data Lakehouse, Data Mesh, and Data Fabric (r1)

James Serra

Best Practices For Building and Operating A Managed Data Lake - StampedeCon 2016

StampedeCon

The document discusses using a data lake approach with EMC Isilon storage to address various business use cases. It describes how the solution provides shared storage for multiple workloads through multi-protocol support, enables data protection and isolation of client data, and allows testing applications across Hadoop distributions through a common platform. Examples are given of how this approach supports an enterprise data hub, data warehouse offloading, data integration, and enrichment services.

Using data lakes to quench your analytics fire - AWS Summit Cape Town 2018

Amazon Web Services

Speaker: Shafreen Sayyed, AWS Level: 200 Traditional data storage and analytic tools no longer provide the agility and flexibility required to deliver relevant business insights. We are seeing more and more organisations shift to a data lake solution. This approach allows you to store massive amounts of data in a central location so its readily available to be categorized, processed, analyzed, and consumed by diverse organizational groups. In this session, we’ll assemble a data lake using services such as Amazon S3, Amazon Kinesis, Amazon Athena, Amazon EMR, AWS Glue and integration with Amazon Redshift Spectrum.

Building-a-Modern-Data-Platform-in-the-Cloud.pdf

Amazon Web Services

Modern data is massive, quickly evolving, unstructured, and increasingly hard to catalog and understand from multiple consumers and applications. This session will guide you though the best practices for designing a robust data architecture, highlightning the benefits and typical challenges of data lakes and data warehouses. We will build a scalable solution based on managed services such as Amazon Athena, AWS Glue, and AWS Lake Formation.

Intelligent Integration OOW2017 - Jeff Pollock

Jeffrey T. Pollock

Building a modern data platform in the cloud. AWS DevDay Nordics

javier ramirez

Amazon Redshift with Full 360 Inc.

Amazon Web Services

Full 360 is a cloud consulting firm that provides big data, API/UX, and cloud operations services. They helped a customer migrate their data from Netezza to Redshift, building a structured data lake and optimizing queries for equivalent or better performance. Lessons from the project included data standardization, tuning techniques like encoding and sort keys, and creating reusable ingestion processes. The migration reduced license costs and improved operational flexibility.

Using AWS Purpose-Built Databases to Modernize your Applications

Amazon Web Services

As you look to modernizing your applications, you will need to consider your database options to meet the new application requirements. AWS offers a series of purpose-built databases that include relational, key value, document, graph and cache use cases to help you deliver new and enhanced functionalities. In this webinar session, we share the different modern application architectures, and how to combine different database services to meet your requirements. Understand how to modernize your relational databases through easy upgrades with Amazon Relational Database Service and learn how to migrate from one database to another with AWS Database Migration Service and AWS Schema Conversion Tool. Speaker: Blair Layton, Business Development Manager, Amazon Web Services

Using Data Lakes: Data Analytics Week SF

Amazon Web Services

Data Analytics Week at the San Francisco Loft Using Data Lakes A data lake can be used as a source for both structured and unstructured data - but how? We'll look at using open standards including Spark and Presto with Amazon EMR, Amazon Redshift Spectrum and Amazon Athena to process and understand data. Speakers: John Mallory - Principal Business Development Manager Storage (Object), AWS Hemant Borole - Sr. Big Data Consultant, AWS

The Marriage of the Data Lake and the Data Warehouse and Why You Need Both

Adaryl "Bob" Wakefield, MBA

In the past few years, the term "data lake" has leaked into our lexicon. But what exactly IS a data lake? Some IT managers confuse data lakes with data warehouses. Some people think data lakes replace data warehouses. Both of these conclusions are false. Their is room in your data architecture for both data lakes and data warehouses. They both have different use cases and those use cases can be complementary. Todd Reichmuth, Solutions Engineer with Snowflake Computing, has spent the past 18 years in the world of Data Warehousing and Big Data. He spent that time at Netezza and then later at IBM Data. Earlier in 2018 making the jump to the cloud at Snowflake Computing. Mike Myer, Sales Director with Snowflake Computing, has spent the past 6 years in the world of Security and looking to drive awareness to better Data Warehousing and Big Data solutions available! Was previously at local tech companies FireMon and Lockpath and decided to join Snowflake due to the disruptive technology that's truly helping folks in the Big Data world on a day to day basis.

Consolidate your data marts for fast, flexible analytics 5.24.18

Cloudera, Inc.

In this webinar, Cloudera and AtScale will showcase: How a company can modernize their analytic architecture to deliver flexibility and agility to more end-users. How using AtScale’s Universal Semantic layer can end the data chaos and allow business users to use the data in the modern platform. Highlight the performance of AtScale and Cloudera’s analytic database with newly completed TPC-DS standard benchmarking. Best practices for migrating from legacy appliances.

Similar to Dive Into Data Lakes (20)

Migrating your traditional Data Warehouse to a Modern Data Lake

AWS Storage State of the Union

2017 OpenWorld Keynote for Data Integration

Lets Talk Google BigQuery

Using Data Lakes

Innovation Track AWS Cloud Experience Argentina - Data Lakes & Analytics en AWS

Unlocking big data with Hadoop + MySQL

Applying AWS Purpose-Built Database Strategy - SRV307 - Toronto AWS Summit

Using Data Lakes

Data Lakehouse, Data Mesh, and Data Fabric (r1)

Best Practices For Building and Operating A Managed Data Lake - StampedeCon 2016

Using data lakes to quench your analytics fire - AWS Summit Cape Town 2018

Building-a-Modern-Data-Platform-in-the-Cloud.pdf

Intelligent Integration OOW2017 - Jeff Pollock

Building a modern data platform in the cloud. AWS DevDay Nordics

Amazon Redshift with Full 360 Inc.

Using AWS Purpose-Built Databases to Modernize your Applications

Using Data Lakes: Data Analytics Week SF

The Marriage of the Data Lake and the Data Warehouse and Why You Need Both

Consolidate your data marts for fast, flexible analytics 5.24.18

More from Matillion

ELT is Better. Here's Why.

Matillion

The demand for speed in data loading and transformation has never been greater—companies that can’t keep up will be left behind. Traditional ETL is being outpaced by cloud-native ELT tools which are both faster and more efficient. In this webinar, we examine the differences between ETL and ELT, and explain what they mean for your data and your business. - Learn what makes ELT different from traditional ETL - See how a modern ELT architecture outperforms ETL - Explore how the latest trends in ELT technology affect your business - Discover how Matillion’s ELT solutions can help you quickly load and transform massive amounts of data

Pick a Winner: How to Choose a Data Warehouse

Matillion

This document summarizes a webinar about choosing a cloud-based data warehouse. It discusses evaluating different data warehouse technologies like MPP vs SMP and columnar vs row-oriented storage. It also introduces Matillion, a company that provides ETL tools for loading and transforming data in Amazon Redshift, Snowflake and Google BigQuery data warehouses. The webinar agenda includes discussing when a data warehouse is needed, criteria for evaluating options, different technologies, and a demonstration of Matillion's software.

Using ELT to load 1 Billion Rows of Data in 15 Minutes

Matillion

Matillion ETL, which uses push-down ELT, is the fastest way to build end-to-end, production-ready big data stacks in Amazon Redshift, Google BigQuery, and Snowflake. Watch us create a finished fact table in fifteen minutes using one billion rows of data from multiple source systems—live. Watch the webinar to - Learn how to process big data faster with Matillion ETL. - Learn how to leverage Matillion ETL for Amazon Redshift, - Google BigQuery, and Snowflake - Find out how Matillion ETL makes loading and preparing massive amounts of data easier and more efficient - Discover how brands like GE, Converse, and Docusign use Matillion ETL to deliver low-cost, high-powered data solutions

How to Choose a Data Warehouse

Matillion

Everyone is moving their data to the cloud - but with all the different choices for a cloud based data warehouse, how do you know which one to choose? How do you know which warehouse will be the best and most flexible for your needs? In this webinar learn: - Why and what is a data warehouse - do you even need one? - The best criteria when evaluating which data warehouse to choose - What problems a data warehouse solves - What is “Big Data” and how does this provide business value - How Matillion can help you work with your data in your data warehouse of choice

Using Google Cloud for Marketing Analytics: How the7stars, the UK’s largest i...

Matillion

the7Stars, the leading UK Digital Marketing agency, has global clients ranging from Nintendo to Suzuki to Iceland. With growing data volumes, the7Stars faced the challenge of centralizing all their customers’ marketing data for quick and easy analysis. In this joint webinar, you will hear about how the7Stars are using Google BigQuery as their data warehouse collating data from many different sources, allowing them to grow their business and attract new customers. the7Stars is also using Matillion ETL to combine the data from different sources and load it all into BigQuery enabling agile and responsive market analysis giving their clients a competitive edge, while saving time and money. In this webinar learn: - the7Stars’ data journey for maximizing value - Google BigQuery, BigQuery Data Transfer Service and best practices for marketing analytics - How to collect data from different sources and streamline transformations and queries in Google BigQuery with Matillion ETL - Benefits being actualized by 7 Stars, such as saving time/money and growing their customer base Watch the full webinar: https://youtu.be/8VEHf_wAXao

Webinar | Accessing Your Data Lake Assets from Amazon Redshift Spectrum

Matillion

In the third webinar in our Amazon Redshift Spectrum Series, learn more about how you can accessing your data lake assets from Amazon Redshift Spectrum. Spectrum allows you to use the analytic capabilities of Amazon Redshift beyond the data which is in your data warehouse to query large amounts of semistructured and structured data in your “data lake,” without having to load or transform it into Amazon Redshift. In this webinar learn: - How to launch Amazon Redshift On-Demand - About Amazon Redshift Spectrum from Greg Khairallah (AWS) - More about Matillion's Spectrum features - Through demonstration, how to accessing your data lake assets from Amazon Redshift Spectrum Watch the full webinar: https://youtu.be/mOKWovh5l4g

Webinar | Getting Started With Amazon Redshift Spectrum

Matillion

In the first webinar in our Amazon Redshift Spectrum Series, learn more about getting started with Spectrum. Spectrum allows you to use the analytic capabilities of Amazon Redshift beyond the data which is in your data warehouse to query large amounts of semistructured and structured data in your “data lake,” without having to load or transform it into Amazon Redshift. In this webinar learn: - What is Amazon Redshift Spectrum? - How to set up AWS Security for Amazon Redshift Spectrum - What are external schemas and external tables - How to query S3 data with Redshift SQL via Amazon Redshift Spectrum using Matillion ETL

Getting Started With Amazon Redshift

Matillion

More from Matillion (8)

ELT is Better. Here's Why.

Pick a Winner: How to Choose a Data Warehouse

Using ELT to load 1 Billion Rows of Data in 15 Minutes

How to Choose a Data Warehouse

Using Google Cloud for Marketing Analytics: How the7stars, the UK’s largest i...

Webinar | Accessing Your Data Lake Assets from Amazon Redshift Spectrum

Webinar | Getting Started With Amazon Redshift Spectrum

Getting Started With Amazon Redshift

Recently uploaded

原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样

mz5nrf0n

原版一模一样【微信：741003700 】【美国纽约州立大学奥尔巴尼分校毕业证学位证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

Vitthal Shirke Java Microservices Resume.pdf

Vitthal Shirke

KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD

rodomar2

Oracle 23c New Features For DBAs and Developers.pptx

Remote DBA Services

Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris

Neo4j

Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris

Neo4j

socradar-q1-2024-aviation-industry-report.pdf

SOCRadar

SOCRadar's Aviation Industry Q1 Incident Report is out now! The aviation industry has always been a prime target for cybercriminals due to its critical infrastructure and high stakes. In the first quarter of 2024, the sector faced an alarming surge in cybersecurity threats, revealing its vulnerabilities and the relentless sophistication of cyber attackers. SOCRadar’s Aviation Industry, Quarterly Incident Report, provides an in-depth analysis of these threats, detected and examined through our extensive monitoring of hacker forums, Telegram channels, and dark web platforms.

Artificia Intellicence and XPath Extension Functions

Octavian Nadolu

GraphSummit Paris - The art of the possible with Graph Technology

Neo4j

A Study of Variable-Role-based Feature Enrichment in Neural Models of Code

Aftab Hussain

Understanding variable roles in code has been found to be helpful by students in learning programming -- could variable roles help deep neural models in performing coding tasks? We do an exploratory study. - These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia

Transform Your Communication with Cloud-Based IVR Solutions

TheSMSPoint

Discover the power of Cloud-Based IVR Solutions to streamline communication processes. Embrace scalability and cost-efficiency while enhancing customer experiences with features like automated call routing and voice recognition. Accessible from anywhere, these solutions integrate seamlessly with existing systems, providing real-time analytics for continuous improvement. Revolutionize your communication strategy today with Cloud-Based IVR Solutions. Learn more at: https://thesmspoint.com/channel/cloud-telephony

Microservice Teams - How the cloud changes the way we work

Sven Peters

A lot of technical challenges and complexity come with building a cloud-native and distributed architecture. The way we develop backend software has fundamentally changed in the last ten years. Managing a microservices architecture demands a lot of us to ensure observability and operational resiliency. But did you also change the way you run your development teams? Sven will talk about Atlassian’s journey from a monolith to a multi-tenanted architecture and how it affected the way the engineering teams work. You will learn how we shifted to service ownership, moved to more autonomous teams (and its challenges), and established platform and enablement teams.

openEuler Case Study - The Journey to Supply Chain Security

Shane Coughlan

LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM

lorraineandreiamcidl

Atelier - Innover avec l’IA Générative et les graphes de connaissances

Neo4j

Atelier - Innover avec l’IA Générative et les graphes de connaissances Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement. Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf

timtebeek1

DDS-Security 1.2 - What's New? Stronger security for long-running systems

Gerardo Pardo-Castellote

8 Best Automated Android App Testing Tool and Framework in 2024.pdf

kalichargn70th171

How to write a program in any programming language

Rakesh Kumar R

Recently uploaded (20)

原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样

Essentials of Automations: The Art of Triggers and Actions in FME

Vitthal Shirke Java Microservices Resume.pdf

KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD

Oracle 23c New Features For DBAs and Developers.pptx

Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris

socradar-q1-2024-aviation-industry-report.pdf

Artificia Intellicence and XPath Extension Functions

GraphSummit Paris - The art of the possible with Graph Technology

A Study of Variable-Role-based Feature Enrichment in Neural Models of Code

Transform Your Communication with Cloud-Based IVR Solutions

Microservice Teams - How the cloud changes the way we work

openEuler Case Study - The Journey to Supply Chain Security

LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM

Atelier - Innover avec l’IA Générative et les graphes de connaissances

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf

DDS-Security 1.2 - What's New? Stronger security for long-running systems

8 Best Automated Android App Testing Tool and Framework in 2024.pdf

How to write a program in any programming language

Dive Into Data Lakes

1. 1 www.matillion.com © 2017 Matillion. All rights reserved. Presented by: Copyright © 2017. All rights reserved. Matillion, trademarks, registered trademarks or service marks are property of their respective owners. 10/10/2017 Dive into Data Lakes James Johnson and Paul Johnson Wednesday, April 25th, 11am EDT

2. 2 www.matillion.com © 2017 Matillion. All rights reserved. Who we are • AWS Advanced Technology Partner and Big Data Competency Holder • Google Cloud Platform Partner • Matillion Products - Matillion ETL for Amazon Redshift - Matillion ETL for Snowflake - Matillion ETL for BigQuery • Over 60 5* reviews on the AWS Marketplace

3. 3 www.matillion.com © 2017 Matillion. All rights reserved. Agenda • What is a Data Lake? • Data Lakes vs. Data Warehouses • Data Lake Best Practice & Reference Architecture • Matillion Demo - Amazon Redshift Spectrum (ELT using S3) - BigQuery (ELT using Google Cloud Storage) - Snowflake (ELT with JSON files) • How to Engage

5. 5 www.matillion.com © 2017 Matillion. All rights reserved. Characteristics of a Data Lake • Data is stored in native format • Stored forever • Flexible access • Schema-on-Read • Data Lake is not an S3/Cloud Storage bucket • It’s not a dumping zone • Free-for-all access

7. 7 www.matillion.com © 2017 Matillion. All rights reserved. Data Lake • Data is stored in native format • Store forever • Flexible access to raw data • Schema-on-Read • Seperate storage & compute • Data requires transformation • Expensive to store large volumes • Transformed before loading (ETL) • Schema-on-Write • Tightly coupled storage & compute Traditional On-Premise Data Warehouse

9. 9 www.matillion.com © 2017 Matillion. All rights reserved. Data Lake • Data is stored in native format • Store forever • Flexible access to raw data • Schema-on-Read • Seperate storage & compute • Support for semi-structured data • Store forever • Transformed after loading (ELT) • Schema-on-Write & Schema-on-Read • Seperate storage & compute Modern Cloud Data Warehouse

11. 11 www.matillion.com © 2017 Matillion. All rights reserved. Data Sources ERP CRM Logs Devices Ingest Files Streams Records Presentation Layer Layer Data Scientist Data Analysts Business Users Applications Transactions Data Lake Process Automation Landing Zone Trusted Zone Consumption Zone Refined for specific use. Aggregated, Denormalized etc. Sandbox Exploratory Analytics Governance Tools Master Data Reference Data Immutable log Metadata & Lineage Data Quality Data Catalog Security

12. 12 www.matillion.com © 2017 Matillion. All rights reserved. Data Lake Architecture Best Practices Decouple storage & compute • Data -> Store -> Process -> Store -> Analyse -> Answers • Allows huge volumes of data to be stored without paying for compute • Allows replacement of processing technology (e.g. Spark) Use the right tool for the job • Data structure, latency, throughput, access patterns Log-centric design patterns • Immutable logs, materialized views • Delete nothing. • Rebuild your data at a point in time Leverage serverless or managed services • Scalable/elastic, available, reliable, secure, no/low admin

14. 14 www.matillion.com © 2017 Matillion. All rights reserved. • Available only on AWS Marketplace/Cloud Launcher • 14 day free trial and test drives • SA team to support PoCs, demos and training • Customer reference videos on YouTube How to Engage

16. 16 www.matillion.com © 2017 Matillion. All rights reserved. Presented by: Copyright © 2017. All rights reserved. Matillion, trademarks, registered trademarks or service marks are property of their respective owners. 7/23/2018 Thank You James Johnson and Paul Johnson

Dive Into Data Lakes

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Dive Into Data Lakes

Similar to Dive Into Data Lakes (20)

More from Matillion

More from Matillion (8)

Recently uploaded

Recently uploaded (20)

Dive Into Data Lakes