PGConf.ASIA 2019 Bali - Foreign Data Wrappers - Etsuro Fujita & Tatsuro Yamada

Copyright © NTT COMWARE CORPORATION 2019Copyright © NTT COMWARE CORPORATION 2019
PGConf.Asia 2019
NTT Comware
Tatsuro Yamada
Foreign Data Wrappers:
A Powerful Technology for Data Integration
Part 2

Copyright © NTT COMWARE CORPORATION 2019
Who I am?
 Tatsuro Yamada
⁃ From Tokyo, Japan
 Work
- for NTT Comware as a Database engineer
- Oracle_fdw committer
- Organizers of PGConf.Asia
- pg_plan_advsr author
 Interest
- Craft beer, Jazz samba, Camping
- Automated Plan tuning

Agenda
1. Oracle_fdw under the food
2. Tips
3. Migration Tool: ora_migrator
4. Conclusion

1.1. What is oracle_fdw
1.2. Use cases
1.3. Characteristics
5

1.1. What is oracle_fdw
Feature
– oracle_fdw can wrap an oracle table as a PostgreSQL
foreign table and use it as a data source!
– PostgreSQL can access Oracle tables via foreign tables
How it works
– Reference to the foreign table in a query executed on
PostgreSQL is converted to a query to Oracle to fetch the
data from the Oracle table
– It converts the result obtained from Oracle into a format
that PostgreSQL’s executor can use
– It’s kind of an Oracle client tool
6

1.2. Use cases
 Data migration
– You can migrate data from oracle to PostgreSQL
 Data linkage
– You can link data between oracle and PostgreSQL
 Monitor
– You can monitor Oracle from PostgreSQL
7
PostgreSQL OracleQuery

1.3. Characteristics
 Long term support
 Advanced features
 High support level
8

Long term support
Oracle_fdw has a Long support policy
– Postgres_fdw supports from 9.4 on
– Oracle_fdw supports from 9.1 on
Note
Supported Oracle version is based on Oracle Instant client
9
9.1 9.2 9.3 9.4 9.5 9.6 10 11
https://www.postgresql.org/support/versioning/
Postgres_fdw
Oracle_fdw

Advanced features
oracle_fw is able to push-down many clauses ※1
10
Supported
Push-down
postgres_fdw oracle_fdw Others
※2
Where   
Sort   ×
Join  ※3 ×
Aggregate  × ×
※1 In SELECT statement
※2 Almost all fdws can’t but a few fdws can
※3 There is a restriction Only 2-way join is possible
Is there any
Workarouds?

High support level
 Here is a recent interaction (edited for clarity) on a ML,
where a user testified to the level of support they received
from the oracle_fdw maintainers!
11
https://www.postgresql.org/message-id/flat/CAP-dhMr2oFoLZWO3wSARPmgXoMqcOkVz6av99icmBQCjot-MCg%40mail.gmail.com
Can you please expand on "support was premium”?
Sure. I meant to say all our questions were answered and bug that
we reported was fixed. All responses were very fast and high
quality.
Hello. We recently used it to "archive" an Oracle to Postgres 9.5
on Windows and we also use to monitor our Oracle 11g from
Postgres.
In both cases, the developer support was premium.
Can anyone here comment on whether oracle_fdw is safe to
use in production?

2. Tips

2. Tips
2.1. Using View as Foreign Table
2.2. How to check Execution Plan on remote server
2.3. ORA-08177
13

2.1. Using View as Foreign Table
If you are in any of the situations listed below,
it's better to define a view in Oracle using the
tuned query and point your foreign table to it.
– Can't push down certain features of a query
– The execution plan in Oracle is poor
e.g. in Oracle
 Aggregate
create view aggs as select count(*) from oratab;
 Complicated Joins
create view joins as select ... t1 join t2 on … join t3 on …
 Control execution plan
create view ind_hoge as select /*+ index(hoge index1) */ ...
14

2.2. How to check Execution Plan
on remote server
Use Explain (analyze, verbose), Luke!
QUERY PLAN
--------------------------------------------------------------------------------
Foreign Scan on public.agg_hoge (cost=10000.00..20000.00
rows=1000 width=4)
(actual time=1.161..1.189
rows=1 loops=1)
Output: count
Oracle query: SELECT /*768ff6bb452f1e26c5081f4214768bdd*/
r1."COUNT" FROM "AGG" r1
Oracle plan: SELECT STATEMENT
Oracle plan: VIEW AGG
Oracle plan: SORT AGGREGATE
Oracle plan: TABLE ACCESS FULL ORATAB
15

2.3. ORA-08177
Can't serialize access for this transaction
1. SERIALIZABLE transaction isolation level
Concurrent DMLs via oracle_fdw can lead to this error because
oracle_fdw uses SERIALIZABLE transaction isolation level.
2. Deferred segment creation
In Oracle 11.2 or above, the first INSERT on a newly created table
can lead to this error unless you created the table with “SEGMENT
CREATION IMMEDIATE”.
e.g.
CREATE TABLE scott.test (
id NUMBER(5) PRIMARY KEY,
g MDSYS.SDO_GEOMETRY
) SEGMENT CREATION IMMEDIATE;
16

3.1. About ora_migrator
3.2. How to use
18

3.1. About ora_migrator
Ora_migrator is a migration tool using
oracle_fdw and focuses on schema migration
 The following objects are migrated:
- Table/View
- Index
- Sequence
- Function
- Trigger
- Constraint
Note
– Migrating Stored procedure and Sql on business applications
is not supported
– May need to modify views, functions, and triggers by hand
19

3.2. How to use
1. Install
make install
2. Run Prepare func and Create Staging schemas
e.g. Oracle_migrate_prepare(server => ‘oracle’,
only_schemas => ‘{SCOTT}’)
3. Correct DDL that causes an error on Staging schema
e.g. UPDATE pgsql_stage.views
SET definition = '...'
WHERE schema = 'scott' AND view_name = ‘hoge_oratab';
4. Run other funcs sequentially
oracle_migrate_mkforeign()
oracle_migrate_tables()
oracle_migrate_constraints()
oracle_migrate_views()
...
oracle_migrate_finish()
20
Please check:
“New ways to migrate from Oracle”
by Laurenz Albe, Cybertec
https://p2d2.cz/files/migrate.pdf
Preconditions:
Installed oracle_fdw
and
Created Foreign server

4. Conclusion

4. Conclusion
22
 Allows you to access Oracle from PostgreSQL
 Long-term support
 Advanced features
 High support Level
2. Tips
 Know the view workaround when push-down doesn’t work
 Explain (analyze, verbose)
 Beware of Transaction Isolation level and
Deferred segment creation
 Allows Schema Conversion easily

Thanks!
Terima kasih!
tatsuro.yamada.tf@nttcom.co.jp

Any questions?

References

References
 Source code on Github
 oracle_fdw
https://github.com/laurenz/oracle_fdw
 ora_migrator
https://github.com/cybertec-postgresql/ora_migrator
 Ora2pg
https://github.com/darold/ora2pg
 Orafce
https://github.com/orafce/orafce
 pg_plan_advsr
https://github.com/ossc-db/pg_plan_advsr
 Useful documents for migration
– PostgreSQL wiki
https://wiki.postgresql.org/wiki/Converting_from_other_Databases_to_PostgreSQL
– MS Azure
https://datamigration.microsoft.com/
26

Oracle and Java are registered trademarks of Oracle and/or its affiliates.
Other names may be trademarks of their respective owners.
27

PGConf.ASIA 2019 Bali - Foreign Data Wrappers - Etsuro Fujita & Tatsuro Yamada

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to PGConf.ASIA 2019 Bali - Foreign Data Wrappers - Etsuro Fujita & Tatsuro Yamada

Similar to PGConf.ASIA 2019 Bali - Foreign Data Wrappers - Etsuro Fujita & Tatsuro Yamada (20)

More from Equnix Business Solutions

More from Equnix Business Solutions (20)

Recently uploaded

Recently uploaded (20)

PGConf.ASIA 2019 Bali - Foreign Data Wrappers - Etsuro Fujita & Tatsuro Yamada