A data warehouse (DW) migration is no small undertaking, especially when moving from on-premises to the cloud. A typical data warehouse has numerous data sources connecting and loading data into the DW, ETL tools and data integration scripts performing transformations, and reporting, advanced analytics, or ad-hoc query tools accessing the data for insights and analysis. That’s a lot to coordinate and the data warehouse cannot be migrated all at once. Using a data replication technology such as Oracle GoldenGate, the data warehouse migration can be performed incrementally by keeping the data in-sync between the original DW and the new, cloud DW. This session will dive into the steps necessary for this incremental migration approach and walk through a customer use case scenario, leaving attendees with an understanding of how to perform a data warehouse migration to the cloud. Presented at RMOUG Training Days 2019

1. © 2019 Snowflake Computing Inc. All Rights Reserved
DATA WAREHOUSE:
INCREMENTAL MIGRATION
TO THE CLOUD
Michael Rainey | RMOUG Training Days
February 2019

2. © 2019 Snowflake Computing Inc. All Rights Reserved
ABOUT ME
Michael Rainey
Senior Solutions Architect, Professional Services at Snowflake Computing
Oracle ACE Director
Twitter: @mRainey
Email: michael.rainey@snowflake.com

3. © 2019 Snowflake Computing Inc. All Rights Reserved
3 YEARS IN STEALTH + 3 YEARS GA
1000+ employees
Over 2000 customers
today
Over $850M in venture
funding from leading
investors
First customers
2014, general
availability 2015
Founded 2012 by
industry veterans
with over 120
database patents
Queries processed in
Snowflake per day:
100 million
Largest single
table:
68 trillion rows
Largest number of
tables single DB:
200,000
Single customer
most data:
> 40PB
Single customer
most users:
> 10,000
Fun facts:

4. © 2019 Snowflake Computing Inc. All Rights Reserved
CLOUD DATA WAREHOUSING FOR DUMMIES
Grab a copy at
the Snowflake
booth!

5. © 2019 Snowflake Computing Inc. All Rights Reserved
WHAT IS DATA WAREHOUSING?

6. © 2018 Snowflake Computing Inc. All Rights Reserved
THE EVOLUTION OF DATA PLATFORMS
Data warehousing has evolved over four decades
1980’s 1990’s 2000’s 2010’s
Relational
database
Data
warehouse
appliance
Data
warehouse
& platform
software
Built-for-the cloud
data warehouse

7. © 2019 Snowflake Computing Inc. All Rights Reserved
DATA WAREHOUSE DEPLOYMENT OPTIONS
Three categories of data warehousing options
Packaged Software Appliance A true SaaS data
warehouse
• Customer installs -
purchases hardware
separately
• Need IT expertise to build
and manage the data
warehouse
• Data warehouse is hosted
on hardware
• Need to specify disk space
and compute resources
• Pay for compute and
storage as you go
• Zero management
required

8. © 2019 Snowflake Computing Inc. All Rights Reserved
TRADITIONAL SOLUTIONS CANNOT SCALE
Decentralized, local storage
Single cluster
Shared-nothing
Shared storage
Single cluster
Shared-disk

9. © 2019 Snowflake Computing Inc. All Rights Reserved
Source: Kerry Osborne’s keynote at RMOUG Training Days 2019
WHERE WE SPEND TIME

10. © 2019 Snowflake Computing Inc. All Rights Reserved
WHAT’S NEEDED? A CLOUD DATA WAREHOUSE
Current technologies are insufficient
Easy
Management
All Data in
One System
Skilled staff misutilized on
database management;
difficult to add new users,
workloads or data
Data locked in disparate
systems for different data
types
High upfront and ongoing
costs, along with
complexity, prohibit
scaling
Many query tools and
noSQL systems require
specialized knowledge
Scale
On-Demand
Use Standard
Skills

11. © 2019 Snowflake Computing Inc. All Rights Reserved
WE’VE DECIDED TO GO CLOUD...NOW WHAT?
EDW
Data Sources
Data Lake
or Hadoop
Datamarts
ETL
BI / Analytics
OLTP
databases
Enterprise
applications
Web apps
Third-party
Other

12. © 2019 Snowflake Computing Inc. All Rights Reserved
WE’VE DECIDED TO GO CLOUD...NOW WHAT?
Logical Datamarts
BI / Analytics
EDW
Data Lake
ETL or ELT
OLTP
databases
Enterprise
applications
Web apps
Third-party
Other
Data Sources

13. © 2019 Snowflake Computing Inc. All Rights Reserved
What to think about:
● SQL syntax - differences and gaps
● Technologies and tools involved
○ ETL/ELT tool
○ BI, analytics, reporting tools
○ Excel, direct query access, etc.
● Security! Users and roles, data security at rest/in flight
● Planning administration and operations changes
○ How to monitor the DW
○ Updated roles / responsibilities
● Downtime - keep it to a minimum (as in, zero downtime)
● Training on the new cloud data warehouse platform
○ SQL syntax differences
○ Connecting to the DW
GETTING READY

14. © 2019 Snowflake Computing Inc. All Rights Reserved
MIGRATION APPROACH - LIFT AND SHIFT

15. © 2019 Snowflake Computing Inc. All Rights Reserved
MIGRATION APPROACH - DATA VIRTUALIZATION

16. © 2019 Snowflake Computing Inc. All Rights Reserved
MIGRATION APPROACH - INCREMENTAL
USING CONTINUOUS REPLICATION

17. © 2019 Snowflake Computing Inc. All Rights Reserved
Move the data and begin taking realizing benefits of the cloud data warehouse immediately
● Connect business intelligence/reporting/analytics tools first
● Gain champions of the cloud data warehouse
Applications continue to work against the legacy data warehouse
● ETL still runs as-is - migrate/rebuild at your own pace
● Additional sources such as master data management (MDM), streaming, etc can be migrated based
on priority - not all at once
Minimize migration risk
● ETL rebuilding / modifications
WHY REPLICATION FOR MIGRATION?

18. © 2019 Snowflake Computing Inc. All Rights Reserved
REPLICATION OPTION - ORACLE GOLDENGATE

19. © 2019 Snowflake Computing Inc. All Rights Reserved
CONTINUOUS REPLICATION TO CLOUD DW

20. © 2019 Snowflake Computing Inc. All Rights Reserved
CONTINUOUS REPLICATION TO CLOUD DW
Snowflake
Database
External
S3
Snowpipe Service
Server-less
Loader
S3 notification
File data

21. © 2019 Snowflake Computing Inc. All Rights Reserved
Begin data replication from source
● Build up transactions during initial load time
Perform initial load of data from current DW to cloud DW
● Capture SCN or equivalent as-of initial load
Begin replication to target after initial load SCN
● Transactions will “catch up” and sync
Repoint reporting / analytics tools at cloud data warehouse
● Begin with focus on single application or subject area
● Work through initial challenges (SQL syntax, data type conversions, etc)
Migrate ETL processes
● Work through migration/modification in DEV environment
Once ETL ready, switch off data replication and switch on migrated ETL processes
INCREMENTAL MIGRATION USING DATA
REPLICATION EXAMPLE STEPS

22. © 2019 Snowflake Computing Inc. All Rights Reserved
Cost!
Availability of data replication software
● Does the company have a license already?
● Which tools that can be used for micro-batch loading?
Latency requirements
● For example, if ETL batch process runs twice daily, can the cloud
data warehouse be updated 15 minutes later?
Project timeline
Not a one-size fits all approach
CONSIDERATIONS

23. © 2019 Snowflake Computing Inc. All Rights Reserved
Oracle GoldenGate docs:
https://docs.oracle.com/goldengate/c1221/gg-winux/index.html
GoldenGate for Big Data docs:
https://docs.oracle.com/goldengate/bd123210/gg-bd/index.html
Oracle Data Integration blog:
https://blogs.oracle.com/dataintegration/data-integration
Continuous Data Replication into Snowflake with Oracle Goldengate blog post:
https://www.snowflake.com/blog/continuous-data-replication-into-snowflake-with-oracle-goldengate/
Replicating Data to Oracle Autonomous Data Warehouse Cloud:
https://docs.oracle.com/goldengate/c1230/gg-winux/GGODB/replicating-data-oracle-autonomous-data-war
ehouse-cloud.htm
MORE INFORMATION

24. THANK YOU
© 2019 Snowflake Computing Inc. All Rights Reserved

25. © 2019 Snowflake Computing Inc. All Rights Reserved
DISCOVER THE PERFORMANCE, CONCURRENCY,
AND SIMPLICITY OF SNOWFLAKE
As easy as 1-2-3!
01 Visit Snowflake.com
02 Click “Try for Free”
03 Sign up & register
Snowflake is the only data warehouse built for the cloud.
You can automatically scale compute up, out, or
down—independent of storage. Plus, you have the power
of a complete SQL database, with zero management, that
can grow with you to support all of your data and all of
your users. With Snowflake On Demand™, pay only for
what you use.
Sign up and receive
$400 worth of free
usage for 30 days!