Database Migrations Science Ever Growing

© 2016 IDERA, Inc. All rights reserved.
Proprietary and confidential.
THE EVER GROWING SCIENCE OF
DATABASE MIGRATIONS
Presented by Bert Scalzo, PhD, MBA & Oracle ACE
May 16th, 2018

BOOKS BY AUTHOR
Fall 2017

© 2018 IDERA, Inc. All rights reserved. 3
SESSION DESCRIPTION
Many information technology professionals may not recognize it, but the bulk of
their work has been and continues to be nothing more than database
migrations. In the old days to share files across systems, then to move files into
relational databases, then to load into data warehouses, and finally now we're
moving to NoSQL and the cloud.
In the presentation we'll delve into the ever growing and increasingly complex
world of database migrations. Some of these considerations include:
▪ What issues must be planned for and overcome
▪ What problems are likely to occur
▪ What types of tools exist

BACKGROUND & TARGET AUDIENCE
▪ Over 30 years I’ve spent time working at both commercial and government
institutions in various roles from entry level QA tester all the way up to up to
Director of IT
▪ Toughest role has been as lead architect/DBA for mission critical production
systems with extremely strict support responsibilities
▪ I’ve also spent time working for various software development companies
where programmers who don’t have production support responsibilities but
nonetheless build tools for those people who support mission critical systems
▪ This topic will concentrate on traditional business application developers

HOW MUCH TIME CODING
I’ve seen many numbers
quoted, but the common
belief seems to be roughly
25% of the time

OF THAT 25% - MOST IS SIMPLY MOVING DATA
▪ Several studies have put forward that most IT professionals spend the
vast majority of their time simply moving data (as high as 80% - 85%)
▪ My 30 years experience finds no possible reason to doubt those findings
▪ Thus we might be actually better off being paid by the gigabytes of data
that we move rather than salary 
It does make me wonder why I had to learn
so many advanced data types and complex
algorithms when much of my work has been
as a simple data mule …

“
Sisyphus was punished for his self-
aggrandizing craftiness and deceitfulness by
being forced to roll an immense boulder up a
hill only for it to roll down when it nears the top,
repeating this action for eternity

“
Programmers are punished for their self-
aggrandizing craftiness and deceitfulness
by being forced to move immense amounts
of data from source to target, repeating this
action for eternity

THE STATES OF MATTER
▪ Remember that data is any company’s most
important asset, so working with that asset is
both natural and to be expected
▪ Data like matter has states, it’s either bound
(i.e. at rest) or liberated (i.e. in motion)
▪ DBAs handle data at rest while programmers
put data in motion via apps or data migrations
▪ Migrating data may not be sexy, but it’s what
application developers are asked to do most
of the time (i.e. I need a copy of that …)
Liberated
Bound

EXAMPLES OF TRADITIONAL DATA MIGRATION
▪ Traditional OLTP “data feeds” from one system to another
▪ Population of data warehouse directly from source systems (ETL)
▪ Population of a centralized Operational Data Store (ODS)
▪ Population of data warehouse (DW) indirectly from central ODS
▪ Population of data marts from either source systems, ODS or DW
▪ Data extraction from source systems into cubes for business analysis
▪ Data extraction for delivery to business users (e.g. CSV to import Excel)

THE MODERN WORLD OF DATA MIGRATIONS
▪ Data engineers construct massive big data reservoirs sometimes referred
to as “data pools” for investigation by both data scientists & data analysts
(often NoSQL) as businesses embrace data analytics & data mining
▪ Consolidating databases to save on licensing costs
▪ Moving from expensive database platform to an open source alternative
▪ Moving portions of databases into the cloud
▪ Moving entire databases into the cloud
▪ Combos of all the above

“
Tantalus was made to stand in a pool of
water beneath a fruit tree with low
branches, with the fruit ever eluding his
grasp, and the water always receding
before he could take a drink.

“
Analysts are made to stand in a pool of
data beneath the tree of knowledge with
low branches, with the facts ever eluding
their grasp, and information always
receding before they can claim success.

THE EVER GROWING DATA DEMAND
▪ Businesses are addicted to information since they see it as an edge
▪ Technology improvements have lowered costs to keep historical data
▪ Data mining, data analytics and data science all added fuel to this fire
▪ The cloud makes all this quicker and cheaper to deploy …

THE NATURE OF DATA GROWTH

That’s 180 billion terabytes of data!!!

“
I feel self-doubt whether I'm doing
something hard or easy.
Sigourney Weaver

“
Drones can be useful tools, and I am all
about useful tools. One of my mottos is
'the right tool for the right job.'
Martha Stewart

CUSTOM CODE
▪ If source and target same then have programmers write custom code in
database language (e.g. T-SQL or PL/SQL) to move data
▪ If source and target different then have programmers write custom code in
3GL or 4GL scripting language (e.g. PowerShell or PERL) to move data
▪ Even if you know the systems and the data, the time required can be
prohibitive but management often makes the mistake that this will be
cheaper than buying tools
▪ Performance often poor since multithreaded, parallelized code generally
not easy to write nor debug (and few are good at it anyhow)

EXPORT / IMPORT – DATA (WILL SHOW IN DEMO)
▪ If source and target same then often options exist to use export/import
(with proprietary file format) or backup/restore as a data movement tool
▪ If source and target different then sometimes export offers the ability to
dump data either as SQL statements or as raw data (e.g. CSV file) but
then target ideally should offer high speed data loader (e.g. SQL/Loader)
▪ Sometimes these tools offer parallel capabilities but make sure to know
the caveats such as using more space due to “holes” in the data from
parallel inserts not filling entire pages / blocks
▪ High disk space cost since must have three copies of data existing at the
same time (i.e. source, target and exported files)

EXPORT / IMPORT – SQL (WILL SHOW IN DEMO)
▪ Often viewed as a “safe & easy” choice since SQL INSERT command is
generally fairly uniform across databases with few proprietary additions
▪ Need one of two capabilities: generated SQL INSERT contains COMMIT
every N statements or a SQL command line tool with auto commit option
▪ Generally neither parallelized nor multithreaded (i.e. slow option)
▪ Difficulties with highly complex data type conversion rules for date, time,
datetime and single/double quoted strings needing escaped
▪ High disk space cost since must have three copies of data existing at the
same time (i.e. source, target and exported files)

EXTRACT TRANSFORM LOAD (ETL)
▪ ETL tools are quite popular and there are many open source tools (free)
▪ ETL tools were designed for and are great at performing highly complex
transformations, however simple tasks are often made more tedious by
the overhead of such a robust and powerful design
▪ ETL tools are HUGE, often a couple gigabytes for just the client GUI and
also requiring deployment of a server agent on “close vicinity” servers
▪ ETL tool learning curves are significant and unless used daily often must
be relearned when needed
▪ ETL tools may not best leverage database platform specific capabilities

BASIC DATA MOVERS (WILL SHOW IN DEMO)
▪ Good choice when all you need to do is to move data from point A to B
▪ Often offer good parallel and multithreaded capabilities for speed
▪ Handles all the highly complex data type conversion rules for date, time,
▪ Client based tools only good for relatively small databases (PC CPU and
memory plus network bottlenecks)
▪ Some tools offer ability for client to run on server or an agent on server
▪ There are some free tools but they generally require paying to scale up

REPLICATION BASED
▪ This is a somewhat newer and novel approach that is gaining traction
▪ Replication tools offering cross database support can be used for either
synchronous or asynchronous replication as the data movement engine
▪ Often offer excellent parallel and multithreaded capabilities for speed
▪ Handles all the highly complex data type conversion rules for date, time,
▪ Generally handled by privileged users like DBAs, not app developers
▪ Solution can be quite expensive unless you already have replication tool

CLOUD SOLUTION – DATABASE VENDOR – SQL SERVER
▪ SQL Server 2016 introduced the “stretch database” concept (to Azure)
▪ Relatively simple syntax to define rules for what portions of the data are
placed in the cloud (note that this is not just one time but rather ongoing)
▪ SQL Server query optimizer makes this data split totally transparent to all
applications (so cloud adoption can easily be rolled out incrementally)
▪ There are some limitations (e.g. data types, index types, table types, etc.)
▪ Can be expensive: space + compute + DSU (a database stretch unit
represents the power of the query and is quantified by your workload
objectives: how fast rows are written, read and computed against)

CLOUD SOLUTION – DATABASE VENDOR – ORACLE
▪ Oracle 12c R2 multitenant offers “relocate PDB” concept (to Oracle Cloud)
▪ Can be done online!!!
▪ Simple drag and drop operation to perform!!!
▪ Bidirectional – so can incrementally deploy or revert as needed
▪ May perform quicker than one would imagine – sometimes fastest option
▪ Require source database is a PDB (i.e. pluggable database container)
▪ Cost – requires enterprise edition with multitenant option (next page)

CLOUD VENDOR SOLUTION – EXAMPLE AWS
▪ Import/Export service (mail your portable storage devices to Amazon)
▪ AWS Snowball Appliance (create snowball job, AWS sends portable
storage device, run snowball client to copy the data, ship back the
portable storage device to AWS, data automatically copied to your S3)
▪ AWS Glue (ETL tool for migrating cloud to cloud)
▪ AWS Kinesis Data Firehose (easiest way to load streaming loud data into
cloud data stores and cloud analytics tools)
▪ There are more and Amazon is constantly creating new offerings …

CLOUD VENDOR SOLUTION – EXAMPLE AZURE
▪ Import/Export service (mail your portable storage devices to Microsoft)
▪ Azure Data Factory (fully managed cloud-based data integration service)
▪ Azure Polybase (fastest possible loading of Azure SQL Data Warehouse
leveraging the entire Massively Parallel Processing (MPP) architecture)
• PolyBase with T-SQL
• PolyBase with SSIS
• PolyBase with Azure Data Factory (ADF)
• PolyBase with Azure DataBricks
▪ There are more and Microsoft is constantly creating new offerings …

DEMO

Database Migrations Science Ever Growing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Database Migrations Science Ever Growing

Similar to Database Migrations Science Ever Growing (20)

More from IDERA Software

More from IDERA Software (20)

Recently uploaded

Recently uploaded (20)

Database Migrations Science Ever Growing