Foreign data wrappers in PostgreSQL allow data from external data stores like MySQL, Redis, and CSV files to be accessed using SQL. Wrappers implement the SQL/MED specification and are developed as PostgreSQL extensions. This allows data from these sources to be queried, analyzed, transformed, and indexed using PostgreSQL features. The presentation demonstrated creating foreign servers, user mappings, and tables to integrate yard inventory from CSV, online inventory from Redis, and sales from MySQL into a single PostgreSQL database.

1. PostgreSQL 9.5
Foreign Data Wrappers
Pittsburgh PostgreSQL Users Group
2016 April 20
Nicholas Kiraly
github.com/nkiraly
Twitter @NicholasKiraly
kiraly.nicholas@gmail.com

2. Introductions
Systems Integration Engineer
Open Source / PostgreSQL / FreeBSD Advocate
To play along with today’s examples, vagrant up with
https://github.com/nkiraly/koadstation/tree/master/dbdevu14
Comments and additions welcome during this presentation.
SQL and other code examples can be found at
https://github.com/nkiraly/jhe-demo/tree/master/

3. What are Foreign Data Wrappers?
Implementation of SQL/MED specification - “SQL Management of External Data”
Read and write to remote data stores such as MySQL, Redis, RSS, LDAP, HDFS, et
cetera.
These wrappers enable the database developer to serve, analyze, ETL, and index data
from these remote stores with all the advantages of PostgreSQL types, partial indexes,
PL/pgSQL functions, et al.
For more details, see https://wiki.postgresql.org/wiki/Foreign_data_wrappers

4. How do Foreign Data Wrappers work?
Implemented as
native C Postgres extension
C Skeleton Blackhole https://bitbucket.org/adunstan/blackhole_fdw
wrapper extension language such as
Python with Multicorn http://multicorn.org/
Ruby with Holycorn https://github.com/franckverrot/holycorn
Providing a postgresql extension that exports FDW handler and validator functions.
Then you can define foreign tables with CREATE FOREIGN TABLE .. SERVER ;

5. Use cases for Foreign Data Wrappers
ETL source or target
Backup source or target
Normalization / denormalization of data for use by analytics
GPU accelerated sequential scan - PGStrom is a scan provider utilizing CUDA for
parallel scans of large sequences with complex qualifiers
Legacy or varied systems integration platform for analytics or next generation
development
Utilization of large external or real time data like OpenStreetMaps PBF or USGS
seismic data API.

6. Creating a Foreign Data Wrapper
Example in C - Blackhole
https://bitbucket.org/adunstan/blackhole_fdw/src/fe87f38a8dda40cba43b67a6cd20834b7bf0fe9d/src
/blackhole_fdw.c
Python Platform - Multicorn
http://multicorn.org/implementing-an-fdw/

7. Declarative
SQL Examples
Install Extension
Foreign Server
User Mapping
Foreign Table
with postgres_fdw

8. Install / Enable Postgres FDW
CREATE EXTENSION postgres_fdw;

9. Defining a foreign Postgres Server
CREATE SERVER foreign_server
FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host '10.22.33.45', port '5432', dbname 'foreign_db');

10. Defining a foreign User Mapping
CREATE USER MAPPING FOR local_user
SERVER foreign_server
OPTIONS (user 'foreign_user', password 'password');
CREATE USER MAPPING FOR PUBLIC
SERVER foreign_server
OPTIONS (user 'foreign_readonlyuser', password 'password');

11. Defining a foreign Postgres Table
CREATE FOREIGN TABLE foreign_users (
user_id integer NOT NULL,
user_name data text,
user_profile text
)
SERVER foreign_server
OPTIONS (schema_name 'some_schema', table_name 'some_table');

12. Systems
Integration
Demo
Storage Systems in Place
Yard Inventory - CSV
Yard Sales - CSV
Online Inventory - Redis 3.0
Online Sales - MySQL 5.7
for
Janey Heavy Equipment

13. Yard Inventory / Sales- CSV

14. Online Inventory - Redis 3.0

15. Online Sales - MySQL 5.7

16. PostgreSQL 9.5
Foreign Data Wrappers
Questions ? Answers !
Feedbacks $#!7$%
SELECT
implementation,
knowledge,
take_aways
FROM foreign_server
WHERE
dat_foreign_data_tho
IS NOT NULL
Nicholas Kiraly
github.com/nkiraly
Twitter @NicholasKiraly
kiraly.nicholas@gmail.com

17. References
FDW Overview https://wiki.postgresql.org/wiki/Foreign_data_wrappers
CSV FDW http://www.postgresql.org/docs/9.5/static/file-fdw.html
Redis FDW https://github.com/nahanni/rw_redis_fdw
MySQL FDW https://github.com/EnterpriseDB/mysql_fdw
PostgreSQL http://www.postgresql.org/docs/current/static/postgres-fdw.html