PostgreSQL Foreign Data wrappers Connectivity to multiple data sources

1. PostgreSQL
Foreign Data wrappers
Connectivity to multiple
data sources
05/06/2018 – V1.0 – Frédéric Bamiere

2. What are Foreign Data Wrappers ?
2
 External data sources look like local tables
 SQL databases :
 MySQL - MariaDB
 Oracle
 SQL Server
 And … PostgreSQL…
 NOSQL:
 CouchDB,
 Redis,
 MongoDB
 Files
 LDAP
 Web Services

3. Why FDW ?
3
 Let the database do the work
 Avoid ETL interfaces and file transfer
 Security

4. Source Dalibo : wiki.postgresql.org/images/7/7e/Conferences-write_a_foreign_data_wrapper_in_15_minutes-presentation.pdf
4
FDW picture

5. How FDW ?
5
Foreign database
Client’s foreign DB
PostgreSQL foreign data
wrapper for foreign DB
User/password
Server
Port
Database
Nothing on this side !
Wrapper object
definitions

6. How ?
6
 In 2003, a new specification called SQL/MED ("SQL Management of External Data")
was added to the SQL standard. It is a standardized way of handling access to remote
objects from SQL databases.
 Generic SQL Database Wrappers :
 ODBC : CartoDB, PostgreSQL Foreign-data Wrapper for ODBC
 JDBC: ?
 Specific SQL Database Wrappers :
 PostgreSQL : Native
 Oracle : Oracle_fdw
 MariaDB - MySQL : mysql_fdw
 SQL Server : tds_fdw
 Others :
 MongoDB : mongo_fdw

7. 7
Install :
• The extension
mysql_fdw
• Mysql Client
10.2.9-MariaDB
https://github.com/EnterpriseDB/mysql_fdw

8. CREATE FOREIGN SERVER
8
-- To be executed as postgres on the meetup database
CREATE EXTENSION IF NOT EXISTS mysql_fdw;
DROP SCHEMA IF EXISTS mariadb_root;
CREATE SCHEMA mariadb_root;
grant all on schema mariadb_root to meetup_owner;
DROP SERVER IF EXISTS foreign_mariadb CASCADE;
CREATE SERVER foreign_mariadb
FOREIGN DATA WRAPPER mysql_fdw
OPTIONS (host 'dctpbdda02.int.tdc',
port '3306');
CREATE USER MAPPING FOR meetup_owner
SERVER foreign_mariadb
OPTIONS (username 'root', password 'new-password');
GRANT USAGE ON FOREIGN SERVER foreign_mariadb TO meetup_owner;
-- end of script

9. CREATE FOREIGN TABLE
9
-- To be executed as meetup_owner on the meetup database
DROP FOREIGN TABLE tiers;
--CREATE FOREIGN TABLE mariadb_root.tiers (
CREATE FOREIGN TABLE tiers (
idtiers int8,
type_tiers varchar(3),
numero_tiers varchar(20),
raison_sociale varchar(100),
user_creation varchar(20),
date_creation timestamp,
user_maj varchar(20),
date_maj timestamp,
idtraitement int8)
SERVER foreign_mariadb
OPTIONS (dbname 'dev_cdp_paiement', table_name 'tiers');
-- it is possible to define a subset of a table.

10. IMPORT FOREIGN SCHEMA
10
-- To be executed as meetup_owner on the meetup database
DROP FOREIGN TABLE tiers;
-- to be executed as meetup_owner
IMPORT FOREIGN SCHEMA "dev_cdp_paiement"
FROM SERVER foreign_mariadb
INTO mariadb_root;

11. 11
Foreign database
PostgreSQL V10.4

12. 12
Foreign database
Install :
• The extension oracle_fdw
• Oracle Client
V12.1
https://github.com/laurenz/oracle_fdw

13. Heterogeneous JOINS
13

14. Heterogeneous JOINS !!
14

15. Test your functionalities !
15
 All foreign data wrappers do no support same functionality
 Update
 Joins between foreign tables limitations (see each fdw documentation)
 Data types conversion
 Charset difference between source and target
 Example: MariaDB : Charset conversion error: Re creating meetup database in utf8 : it works !
 Null values
 INSERT/UPDATE/DELETE : It depends on the FDW capabilities

16. Documentation
16
https://pgxn.org/tag/foreign%20data%20wrapper/

17. Documentation
17