With Open Source database management systems getting mature, they are evolving as a target system for database migration. This talk focuses on migration scenarios and strategies, proposing an elaborate migration workflow. Arguments for and against migrating to Open Source databases are discussed. Lower cost and the advantages of Open Source code and community are reasons for deciding pro migration. Furthermore, Open Source migration candidates will have to provide enterprise-level features. I present five Open Source databases: Firebird, Ingres, MaxDB, MySQL and PostgreSQL, and compare them with respect to features, performance and data warehousing capabilities. Subsequently the database migration workflow is proposed: A process model including all steps on the path from the source system to the target system. It is composed of four activities: analysis, design, implementation and testing. These are applied to the database system’s assets: database management system software, schema, data, client SQL and infrastructure. The steps are ordered to provide for a manageable workflow with special regard to their interdependencies.

1. Migration to Open Source
Databases
Jutta Horstmann
www.osdbmigration.org

2. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 2
whoami
● Unix/Linux sysadmin
● DBA, developer (Informix)
● DB developer (Oracle)
● Web stuff (MySQL, PostgreSQL)
● Claim to Fame: OpenUsability.org
● Comp Sci Diploma Thesis:
Migration to Open Source Databases

3. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 3
Agenda
● What's this?
● Why migrate?
● Where to migrate?
● How to migrate?
● What to migrate?
➔ About
➔ Pros and Cons
➔ Open Source Databases
➔ Workflow: Activities
➔ Workflow: Assets

4. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 4
Migration?
"The process of changing from the use of one platform,
environment, IT system, etc., to another, esp. in such a
way as to avoid interruptions in service."
(Oxford English Dictionary)

5. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 5
Migration: Objectives
The target system:
● same functionality
● extendable
● incorporate all data
● modern hardware,
software, architecture
The migration workflow:
● Minimize risk
● Stay on budget
● Deliver in due time
● Minimize downtime

6. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 6
OSDB Migration:Pros and Cons
YesNo
● Cost? Time? Effort?
● Lack of roadmap
● Licensing
● ISV Support
● Maintenance
● Accountability
● Features
● Features?
● TCO
● Open Source
– Code
– Community
● Standards
● Security
● Independence

7. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 7
Where to migrate?
Firebird MaxDB Ingres
MySQLPostgreSQL

8. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 8

9. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 9
Enterprise-Level?
Requirements:
● Advanced data integrity mechanisms
● Advanced database objects
● Advanced SQL features
● Advanced features, tools, support for
– Performance and scalability
– (High) Availability
– Security

10. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 10
OSDB Features
Firebird
1.5
Ingres
r3
MySQL
4.1
MaxDB
7.5
PostgreSQL
8.0
Advanced
Indexing
GIS support
MVCC
Two phase
commit
UD Functions
UD data types
Updatable Views
MyISAM
MyISAM
InnoDB
v.5.0v.7.6
v.5.0
v.8.1v.5.0

11. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 11
Performance / Scalability
Firebird
1.5
Ingres
r3
MySQL
4.1
MaxDB
7.5
PostgreSQL
8.0
Tablespaces
Table
Partitioning
Parallelization
Built-In
Clustering
Built-In
Load Balancing
v.8.1

12. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 12
How to migrate?
Database Migration Workflow
● Activity-Centered?
● Asset-Centered?
● Proposal: Mixed-Model Workflow
● Walk-Through

13. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 13
Activity-Centered Workflow
Analysis
Data
SQL
Schema
Infra-
structure
DBMS
Data
SQL
Schema
DBMS
Data
SQL
Schema
DBMS
Data
SQL
Schema
DBMS
Design Implementation Test
Source System Source vs. Target
System
Target System Target System
Infra-
structure
Infra-
structure
Infra-
structure

14. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 14
Asset-Centered Workflow
Analysis
Desig
n
DBMS
Analysis
Design
Implementation
Test
Analysis
Desig
n
Analysis
Design
Implementation
Test
Analysis
Desig
n
Analysis
Design
Implementation
Test
Analysis
Desig
n
Analysis
Design
Implementation
Test
Analysis
Desig
n
Analysis
Design
Implementation
Test
Schema Data SQL
Infra-
structure
Source
System
Source
vs. Target
System
Target
System
Target
System

15. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 15
Mixed-Model Workflow
1. Analyze the whole source system (all assets).
2. Migrate the DBMS software (design, implement, test).
3. Migrate the schema(s) (design, implement).
4. Migrate test data (design, implement, test).
5. Test the migrated schema(s) with the test data.
6. Migrate the client SQL (design, implement, test).
7. Migrate the database system's infrastructure (d, i, t).
8. Cut-over: Full data migration.
9. Final testing and evaluation.

16. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 16
Analyze the whole system
Analysis
Data
SQL
Schema
Infra-
structure
DBMS ● How is the software configured?
● Complexity, quality of database design?
● How much data? Which condition?
● SQL isolated? Standard compliant?
● Tools, Policies, Tuning ...?

17. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 17
Where to look? Who to ask?
?
Application
Data
Processes
Documentation
Data Dictionary
History
Data Flows
Data Dictionary
Developers, Admins, Users

18. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 18
Migrate the DBMS software
1. Choose Open Source DBMS
2. Devise mappings for configurations
3. Install & configure new software
4. Validate the working of the DBMS by
using test databases and tools
Design
Implemen-
tation
Test
DBMS

19. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 19
Migrate the schema(s)
Design
Implemen-
tation
Schema
Test
1. Provide abstraction (logical, conceptual?)
2. Devise mappings to target system
3. Implement by loading converted schema
into target system
4.Validate the schema's correctness based
on loading and querying test data.

20. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 20
Migrate Test Data
1. Correction and “cleansing” of “dirty data”
2. Devise conversions and mappings
3. Transfer some test data:
a) Extraction
b)Transformation
c) Load
4. Test: Errors at load? Query data!
Design
Implement
Test Data
Test
Data

21. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 21
Migrate Client SQL
Database connectivity
● Change ODBC/JDBC driver
● Switch client
● Client source available: Migrate!
1. Provide abstraction (Standard SQL) OR
2. Convert directly to target SQL syntax
3. Test statements against schema and
test data
Design
Implemen-
tation
Test
SQL

22. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 22
Migrate Infrastructure
Design
Infra-
structure
Test
● Tools: Administration, Development,
Design
● Change ODBC/JDBC driver
● Switch client
● Administrative Tasks
– Gather policies and jobs
– Implement them in the target system way
Implementation

23. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 23
Full Data Migration (Cut-Over)
1. Strategy:
a) Cold Turkey?
b) Chicken Little?
c) Butterfly?
2. Data Transfer:
Extraction, Transformation, Load
3. Test: Errors at load? Query data!
Design
Test
Data
Implementation

24. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 24
“Cold Turkey”
● Take source system offline
● Extract data
● Transform data
● Load data into target system
● Take target system online
Problems:
● How long offline? How much data?
● Fallback?

25. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 25
“Chicken Little”
● Source and target system operate in parallel
● “Gateway” coordinates and maps queries
● Incremental data migration
Problem:
● Complex implementation
● Update consistency?
Gateway

26. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 26
“Butterfly”
● Only source system online
● Data gets frozen read-only
● Store manipulation results in “Temp Store 0”
● Transfer frozen data
● Freeze TS0, store manipulation results in TS1
● Transfer frozen data ... and so on
● “Data Access Allocator coordinates and maps
queries
● Problem: Complex implementation!

27. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 27
“Butterfly”
Data Access Allocator
Temp
Store 1 TS 2 TS n
Read-only
TS n+1
Data
Transformer

28. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 28
Summary: Problems
● Different SQL implementations
● User-defined data types/functions
● Stored Procedures
● Proprietary interfaces
● Flawed schema design
● Differences in Data Types
● “Dirty Data”
● Reserved Words

29. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 29
Help! - Migration Tools
● Automatization
● Integrity violation detection
● Re-Use
● Knowledge on source and target system
● Documentation
● Code Generation
● Script Scheduling
● Validation
● Speed!

30. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 30
Migration Tools, Examples
● Closed Source:
– SQLWays (Ispirer). Only Windows.
● Source: any, Target: MySQL, PostgreSQL
– ProgressionDB (Versora). Linux, Windows.
● MSSQL -> MySQL, PostgreSQL, Ingres
● Open Source:
– Ingres Mio. $ Challenge (CA): shift2ingres
– MySQL Migration Toolkit (MySQL AB)

31. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 31
Summary
Migration to Open Source Databases ?
● Several Open Source options
● Migration is an ambitious project
● Workflow and results depending on state
of source system
● Automatize the process, where possible!

32. Jutta Horstmann, OpenDB Con, Nov 8 2005. http://www.osdbmigration.org 32
Questions?