Deep dive session - how to achieve database freedom

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How to Achieve Database
Freedom

Agenda
• Database Market Changes
• Application Architecture Changes
• Increasing Cloud Adoption
• Database Freedom – How are customers migrating today?
• Adopting a Database Freedom Strategy
• Migrating to AWS
• Managing Database Migration Projects
• Hints and Tips from Real Customer Examples
• Discussion!

Traditional Database Architecture
App/Web Tier
Client Tier
RDBMS
one database for
all workloads

Traditional Database Architecture
key-value access
complex queries
transactions
analytics
App/Web Tier
Client Tier
RDBMS

Data categories and common use
cases
Relational Key-value Document In-memory Graph
Referential
integrity, ACID
transactions,
schema-
on-write
Low-latency,
key lookups
with high
throughput and
fast ingestion
of data
Indexing and
storing
documents
with support
for query on
any attribute
Microseconds
latency, key-
based queries,
and specialized
data structures
Creating and
navigating
data relations
easily and
quickly
Lift and shift,
EMR, CRM,
finance
Real-time bidding,
shopping cart,
social
Content
management,
personalization,
mobile
Leaderboards,
real-time
analytics,
caching
Fraud detection,
social
networking,
recommendation
engine
Search
Indexing and
searching
semistructured
logs and data
Product
catalog, help
and FAQs,
full text
Time-series Ledger
Collect, store,
and process
data sequenced
by time
IoT
applications,
event tracking
Complete,
immutable, and
verifiable history
of all changes to
application data
Systems
of record,
supply chain,
health care,
registrations,
financial

AWS: Purpose-built databases
Relational Key-value Document In-memory Graph Search
Amazon
DynamoDB
Amazon
Neptune
Amazon
RDS
Aurora CommercialCommunity
Amazon
ElastiCache
Amazon
Elasticsearch
Service
Amazon
DocumentDB
Time-series Ledger
Amazon
Timestream
Amazon
Quantum
Ledger
Database

Microservices at Amazon
Service-Oriented Architecture
(SOA)
Single-purpose
Connect only through APIs
Connect over HTTPS
“Microservices”

Monolithic vs. SOA vs. Microservices
SOA
Coarse-grained
Microservices
Fine-grained
Monolithic
Single Unit

Microservices…
Eliminates any long-term commitment to a technology stack
Polyglot ecosystem
Polyglot persistence
• Decompose Databases
• Database per microservice pattern
Allows easy use of Canary and Blue-Green deployments

Early AWS DB Market – Use Cases
Web/CMS
Gaming
E-Commerce
Phone Apps

AWS Capability has Grown Substantially
Web/CMS
Gaming
E-Commerce
Phone Apps
• Database Migration Service
• Aurora MySQL-compatible edition
• Aurora PostgreSQL-compatible edition
• Performance Insights
• RDS SQL Server: Win. Auth, .BAK, EE-LI, 2016, 16 TB
• RDS Oracle: RCU, SE2 LI, OLS, OEM, Spatial and Multimedia
• 2017: Stop/Start, R4, RI Flexibility
• 2018: X1, X1e, Optimize CPU
• 2019: Global Databases, Aurora MySQL Multi-Master

Business Critical Workloads
Web/CMS
Gaming
E-Commerce
Phone Apps
ERP
CRM
BPM
SCM
HCM
Payroll
Billing
Retail / POS
Accounting
Document Automation
Workflow Management
Legal Matter Management
Risk Management
Enterprise DW
BI
Health ManagementMarketing
Customer Experience
Enterprise Workloads Are Now Migrating

Non-Addressable (Extreme scale, data sovereignty issues)
Addressable Workloads
Addressable Workloads
Web/CMS
Gaming
E-Commerce
Phone Apps

Forrester Wave: Database-
As-A-Service, Q2 2019
The Forrester Wave™ is copyrighted by Forrester Research, Inc. Forrester and Forrester Wave™ are trademarks of Forrester Research, Inc. The Forrester Wave™ is a graphical representation of Forrester's call on a market and is plotted using a
detailed spreadsheet with exposed scores, weightings, and comments. Forrester does not endorse any vendor, product, or service depicted in the Forrester Wave. Information is based on best available resources. Opinions reflect judgment at the
time and are subject to change.
“Amazon has the largest
adoption and broadest range of
options. Amazon continues to
have the largest adoption of
DBaaS, including supporting
some of the largest and most
complex deployments.”
analyst
report
s
ANALYST REPORT

Are you operating with old world databases?
AUDIT
Very expensive Proprietary Lock-in
Punitive
licensing
You’ve got mail

Amazon.com Case Studies

Globe Telecom

AWS database migration partners

Change: People, Processes and Policies
• Migrating to new databases is not just a technology change, but an
organizational change
• People fear change and will require assurances, training and guidance
• On-premises processes need to be modified for the cloud or new
processes developed
• Policies and standards reflect old thinking
• Oracle RAC for tier 1 systems
• SQL Server for .NET applications
• Enterprise Edition by default
• Exceptions for Open Source databases

Executive Sponsorship
• Individual project migrations can be done ad-hoc for validation
• Adopting a strategy requires executive sponsorship to drive change
• Individual organizations inside IT will resist change
• Clear objectives and goals need to be set with regular cadence to the
sponsor
• Key to removing obstacles

Key Steps to Enabling a Database Freedom
Strategy• Policy change to force open source software for new projects with
commercial databases needing SVP (or higher) approval
• Gain confidence and skills from new projects, identify skill gaps
• Evaluate database migration for projects that require hardware or tech
refresh, e.g. major ISV application upgrade, Unix to Linux
• Change policies and processes based on lessons learned
• Engage a licensing partner for licensing strategy review
• Evaluate application and database landscape for migration strategy
• Engage ISVs to determine their database strategy
• Adopt training plans to fill skill gaps and engage specialist partners
• Create specialist teams to assist with migrations (CoE)

What are DMS and SCT?
Our goal: Allow customers the freedom to choose the best data platform
for their needs #DBFreedom
AWS Schema Conversion Tool (SCT) converts your commercial
database and data warehouse schemas to open-source engines or
AWS-native services, such as Amazon Aurora and Redshift
AWS Database Migration Service (DMS) easily and
securely migrates and/or replicate your databases and data
warehouses to AWS

DMS Momentum – Automation and Acceleration

DMS Momentum – Automation and
Acceleration

Our Customers

Use DMS to securely continuously
(CDC) replicate mission-critical data
from various sources to Redshift. Use
the API to fully automate the process.
DMS reduced the time it took to
migrate over 35 million users to Aurora
MySQL by 40%.
Migrated data from on-premises Oracle
RAC to RDS for PostgreSQL. Today
replicates data to Redshift. Use SCT for
database modernization and DMS to split
the data and keep all systems in full sync.
Verizon is targeting a mass migration off
of Oracle to PostgreSQL. One project
expects a potential savings of $1.75M.
Trimble Leveraged SCT to determine an
appropriate target engine. Reduced
infrastructure costs to less than one-
fourth of their privately hosted
infrastructure
The US Department
of Veterans Affairs used
DMS to modernize and securely migrate
a 20 years Oracle database to RDS with
no downtime.
DMS Customer Experience

When to use DMS and SCT?
Modernize Migrate Replicate

AWS Schema Conversion Tool (SCT)
• Modernize your data warehouse
Convert your Oracle, SQL Server, Netezza,
Greenplum, Vertica or Teradata to
Amazon Redshift
Modernize
Amazon
Aurora
Amazon
Redshift
• Modernize your database
Convert your Oracle, SQL Server, or Db2 LUW to
PostgreSQL, MySQL, or Amazon Aurora
MySQL
PostgreSQL

AWS Database Migration Service (DMS)
Migrate
• Migrate business-critical applications
• Migrate data warehouse to Redshift
• Upgrade to a minor version
• Consolidate shards into Aurora
• Archive old data
• Migrate from NoSQL to SQL, SQL to
NoSQL or NoSQL to NoSQL
Amazon RDS
Amazon
Redshift
Amazon
Aurora
Amazon
DynamoDB
Amazon S3

AWS Database Migration Service (DMS)
Replicate
• Create cross region Read Replicas
• Run your analytics in the cloud
• Populate your data lake

AWS SCT Product Highlights
• Assessment of migration
compatibility of source
databases with open-source
database engines – RDS for
MySQL, RDS for PostgreSQL,
and Amazon Aurora
• Recommends best target
engine
• Provides details level of efforts
to complete migration
Assessment Report
• Attempts to convert all schema
and code objects to the target
engine, including stored
procedures and functions
• Scans and converts embedded
SQL statements in app code
• Generates a report with
recommendations
Converts Schema and
Code
• Extracts data through local
migration agents
• Files are loaded to an Amazon
S3 bucket and to Amazon
Redshift
• Netezza
• Vertica
• Greenplum
• Teradata
• Oracle
• SQL Server
Extracts and Migrates
DW to Amazon Redshift

• Designed for Solutions Architects, Partners and Consultants
• Assess workloads by evaluating proprietary features, complexity, technology, size, etc.
• Recommends a migration strategy and migration tools
• Gives migration engineers clear and actionable information
• Integrated with AWS SCT and DMS
• Migration Complexity Assessments
• Workload Migration Strategy
• Migration Tools
Workload Qualification Framework (WQF)
WQF classifies OLTP and DW workloads into five categories:
Category 1 ODBC/JBDC workloads
Category 2 Light, proprietary feature workloads
Category 3 Heavy, proprietary feature workloads
Category 4 Engine-specific workloads
Category 5 Non-portable, high-risk or lift-and-shift
Assesses and classifies OLTP and OLAP workloads to determine the ease of
migration, staff-hour consumption, and inform on appropriate target AWS Services

AWS DMS Product Highlights
• Checks migration task settings
prior to launch
• Alerts for potential issues, such
as unsupported data types
• Prevent unnecessary, time
consuming runtime migration
failures
Pre-migration
Assessment
• Validates that all data selected
for migration migrated properly
• Includes both stages of full load
and CDC
• Applies for both homogeneous
and heterogeneous migrations
Data Validation
Use AWS Snowball & AWS DMS to
migrate data to AWS
• Migrate large databases (over
5TB)
• Migrate many databases at once
• Migrate over slow network
• Push vs. Pull
Snowball Integration

AWS DMS – Endpoint Support Expansion
Amazon
Redshift
Amazon
DynamoDB
Amazon S3
Amazon S3
Amazon
Aurora
Amazon
Aurora
Oracle SQL Server Netezza
Greenplum Vertica Teradata
AWS Snowball
Edge
MongoDB Cassandra
Amazon
Elasticsearch
Amazon KinesisAmazon
DocumentDB

Old World to AWS Migration Playbooks
• Topic-by-topic overview of how to migrate databases and
data warehouses to AWS services
• Covers all proprietary features and the different database
objects
• Migration best practices
• Oracle to Aurora PostgreSQL
• Oracle to Aurora MySQL
• SQL Server to Aurora MySQL
• SQL Server to Aurora PostgreSQL
• Oracle Data warehouses to Redshift
SCT DMS Playbook
Schema Data Best Practices

When to use DMS?
Migrate
• Migrate business-critical applications
• Migrate data warehouse to Redshift
• Upgrade to a minor version
• Consolidate shards into Aurora
• Archive old data
• Migrate from NoSQL to SQL, SQL to
NoSQL or NoSQL to NoSQL
Targets:
Amazon Dynamo
DB
Amazon Redshift
Amazon S3
Amazon Aurora
Amazon S3
Sources:
• Create cross-regions Read Replicas
• Run your analytics in the cloud
• Propagate your data lake
Replicate

Database migration process
Step 1: Convert or Copy your Schema
Source DB or DW
AWS SCT
Native Tool
Destination DB or DW
Step 2: Move your data
Source DB or DW
AWS SCT
Destination DB or DW
AWS DMS

Customer
premises
Application users
AWS
Internet
VPN
Start a replication instance
Connect to source and target
databases
Select tables, schemas, or
databases
 Let AWS DMS create tables,
load data, and keep them in
sync
 Switch applications over to
the target at your convenience
Keep your apps running during the migration
AWS
DMS

Resources available to customers - DMS
Getting Started Guide: Review technical
documentation.
Features and benefits: Highlights DMS
features.
Pricing: Prices for replication instances,
storage, and data transfer.
Support: Post your questions to our
Support forum.
AWS SDK: Java-based API for creating
and managing data migration tasks.
AWS Command Line Interface: Start
and stop replication tasks with simple
commands.

Resources available to customers -
SCT
User Guide: Review technical docs at
aws.amazon.com/documentation/SchemaConversionTool
/
or choose the Download button.
Download area: Get installation files for the
Schema Conversion Tool.
Support forums: Ask questions and review
how-to guides.
https://forums.aws.amazon.com/forum.jspa?forumID=208
.

Application Assessment
Business driver and intended ROI?
Migration sponsor (business owner, C-level)?
ISV application? Does the ISV support the target?
Maintenance window for the migration?
Design documentation?
Original developers/DBAs still available?

Database Assessment
How many database objects (tables, triggers, SPs, users, etc.)?
How much data?
Complexity of the SPs and triggers?
Proprietary DB features?
Non-standard or custom data types?
Character set conversions?
Time zone or UTC?
User authentication method?
Licensing mechanism (cores, users, ULA etc.)

Application Technical Assessment
Database Access:
SQL statements throughout the code?
Calls to a data abstraction layer?
API calls?
ANSI SQL used where possible?
SQL complexity, e.g. analytics with many joins or simple CRUD?
Number of lines of SQL code?
Application access, e.g. LDAP, DB Users, etc.

Building a Migration Team
Application architect/developer: Application expert who can identify what
components are important, complex, redundant, etc.
Source DBA: Knows the database design, schema, features used and
what must be migrated to the target.
Target DBA: An expert in the target database to help map features from the
source DB with the Source DBA.
AWS Solution Architect: Determines the correct target architecture in
AWS and is familiar with DMS/SCT.
Application/Database Developers: Customer and/or partner resources to
migrate the stored procedures, triggers and application code.

Hiring and Developing Talent
New skills are needed for the target DB and often AWS if migrating from on-
premises
Develop training plans for existing employees
Hire in required skills if necessary
Retrain, redeploy or make people redundant whose skills are no longer
relevant

Pilot/POC
Choose a reasonably complex module/component to migrate to validate
your assumptions from the Assessment.
You should:
Obtain more accurate migration assessments
Determine what can be automated
Learn how the migration tools behave (limitations, bugs, improvements
needed)
Learn what skills are missing from your team

Database migration – multi phase process
Phase Description Automation Effort (%)
1 Assessment SCT 2
2 Database Schema Conversion SCT 14
3 Application Conversion/Remediation SCT 25
4 Scripts Conversion SCT 7
5 Integration with 3rd party applications 3
6 Data Migration DMS 4
7 Functional testing of the entire system 29
8 Performance tuning SCT 2
9 Integration and deployment 7
10 Training and knowledge 2
11 Documentation and version control 2
12 Post production support 3

Hints and Tips – General Guidance
• Database migrations must be managed as a proper IT project.
• Application code, integration points, functional and performance
testing, resourcing, support, etc.
• Things will go wrong!
• Data types, query performance, networking bandwidth, networking
devices (firewall and optimizers), bugs in source databases.
• Test all scenarios, e.g. don’t assume because it worked in dev/test that
you can just execute in production. Always plan tests in production too.
• Use native migration methods e.g. DataPump, .BAK, Mysqldump,
Dataguard.
• Ask AWS for help through the Forum, Support and your account team.

Optimize at all Points!

Hints and Tips – Post Data Migration
• Data validation
• Indexes, stored procedures, triggers, sequences
• Fail back plan
• Cutover plan: big bang, CDC + testing, parallel run through load balancer
• Operations: Backups, HA/DR, performance monitoring and alerts
• Continuous replication: consider DMS with MAZ option

Hints and Tips – Real Customer Issues
• Network optimizer reduced DMS throughput
• Network firewall dropped long running DB connections
• Network bandwidth doesn’t allow for peak or batch operation throughput
• Data type mapping from Oracle to PostgreSQL causes performance problems
• Sequences need to be updated at the end of CDC
• Source database not patched and didn’t support CDC of partitioned tables on DR
• Customer used DMS without using SCT!
• AWS accounts with incorrect IAM permissions
• Incorrect security group and other access related issues
• Didn’t read the documentation!
• T2 instance ran out of CPU credits and migration fell behind

Thank you!

Deep dive session - how to achieve database freedom

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Deep dive session - how to achieve database freedom

Similar to Deep dive session - how to achieve database freedom (20)

Recently uploaded

Recently uploaded (20)

Deep dive session - how to achieve database freedom