Unlocking Proprietary Datawith PostgreSQL ForeignData WrappersPat PattersonPrincipal Developer Evangelistppatterson@salesf...
Agenda Foreign Data Wrappers Writing FDW’s in C Multicorn Database.com FDW for PostgreSQL FDW in action
Why Foreign Data Wrappers? External data sources look like local tables!   – Other SQL database      • MySQL, Oracle, SQL...
Why Foreign Data Wrappers? Make the database do the work  – SELECT syntax     • DISTINCT, ORDER BY etc  – Functions     •...
Foreign Data Wrappers 2003 - SQL Management of External Data (SQL/MED) 2011 – PostgreSQL 9.1 implementation   – Read-onl...
FDW’s in PostgreSQL ‘Compiled language’ (C) interface Implement a set of callbacks  typedefstructFdwRoutine  {  NodeTagt...
FDW’s in PostgreSQL Much work!     • CouchDBFDW     • https://github.com/ZhengYang/couchdb_fdw/     • couchdb_fdw.c> 1700...
Multicorn http://multicorn.org/ PostgreSQL 9.1+ extension Python framework for FDW’s Implement two methods…
Multicornfrom multicorn import ForeignDataWrapperclass ConstantForeignDataWrapper(ForeignDataWrapper):    def __init__(sel...
Database.com FDW for PostgreSQL OAuth login to Database.com / Force.com   – Refresh on token expiry Force.com REST API  ...
Demo
Conclusion Foreign Data Wrappers make the whole world look like  tables! Writing FDW’s in C is hard!   – Or, at least, t...
Resources http://wiki.postgresql.org/wiki/SQL/MED http://wiki.postgresql.org/wiki/Foreign_data_wrappers http://multicor...
Unlocking Proprietary Data with PostgreSQL Foreign Data Wrappers
Upcoming SlideShare
Loading in...5
×

Unlocking Proprietary Data with PostgreSQL Foreign Data Wrappers

3,688

Published on

PostgreSQL 9.1 introduced ‘Foreign Data Wrappers’ (FDW) – a partial implementation of the SQL/MED standard for handling access to remote data sources. FDW allows PostgreSQL to expose remote data as foreign tables which then behave similarly to native PostgreSQL tables, in particular, allowing remote data to be queried with SQL statements.

This session provides an overview of Foreign Data Wrappers, looks at the native interface for writing FDWs in C, and contrasts this with Multicorn, an open source framework that allows FDWs to be developed in Python. We will show a real-world Python FDW that retrieves business data from salesforce.com, with a sample client application that demonstrates how foreign data can be combined with data held in native PostgreSQL tables using a simple SQL JOIN.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,688
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
28
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Unlocking Proprietary Data with PostgreSQL Foreign Data Wrappers

  1. 1. Unlocking Proprietary Datawith PostgreSQL ForeignData WrappersPat PattersonPrincipal Developer Evangelistppatterson@salesforce.com@metadaddy
  2. 2. Agenda Foreign Data Wrappers Writing FDW’s in C Multicorn Database.com FDW for PostgreSQL FDW in action
  3. 3. Why Foreign Data Wrappers? External data sources look like local tables! – Other SQL database • MySQL, Oracle, SQL Server, etc – NoSQLdatabase • CouchDB, Redis, etc – File – LDAP – Web services • Twitter!
  4. 4. Why Foreign Data Wrappers? Make the database do the work – SELECT syntax • DISTINCT, ORDER BY etc – Functions • COUNT(), MIN(), MAX() etc – JOIN external data to internal tables – Use standard apps, libraries for data analysis, reporting
  5. 5. Foreign Data Wrappers 2003 - SQL Management of External Data (SQL/MED) 2011 – PostgreSQL 9.1 implementation – Read-only – SELECT-clause optimization – WHERE-clause push-down • Minimize data requested from external source Future Improvements – JOIN push-down • Where two foreign tables are in the same server – Support cursors
  6. 6. FDW’s in PostgreSQL ‘Compiled language’ (C) interface Implement a set of callbacks typedefstructFdwRoutine { NodeTagtype; /* These functions are required. */ GetForeignRelSize_functionGetForeignRelSize; GetForeignPaths_functionGetForeignPaths; GetForeignPlan_functionGetForeignPlan; ExplainForeignScan_functionExplainForeignScan; BeginForeignScan_functionBeginForeignScan; IterateForeignScan_functionIterateForeignScan; ReScanForeignScan_functionReScanForeignScan; EndForeignScan_functionEndForeignScan; /* These functions are optional. */ AnalyzeForeignTable_functionAnalyzeForeignTable; } FdwRoutine;
  7. 7. FDW’s in PostgreSQL Much work! • CouchDBFDW • https://github.com/ZhengYang/couchdb_fdw/ • couchdb_fdw.c> 1700 LoC
  8. 8. Multicorn http://multicorn.org/ PostgreSQL 9.1+ extension Python framework for FDW’s Implement two methods…
  9. 9. Multicornfrom multicorn import ForeignDataWrapperclass ConstantForeignDataWrapper(ForeignDataWrapper): def __init__(self, options, columns):super(ConstantForeignDataWrapper,self).__init__(options, columns)self.columns = columns def execute(self, quals, columns): for index in range(20): line = {} for column_name in self.columns:line[column_name] = %s %s % (column_name, index) yield line
  10. 10. Database.com FDW for PostgreSQL OAuth login to Database.com / Force.com – Refresh on token expiry Force.com REST API – SOQL query • SELECT firstname, lastname FROM Contact Request thread puts records in Queue, execute() method gets them from Queue JSON parsing – skip embedded metadat < 250 lines code
  11. 11. Demo
  12. 12. Conclusion Foreign Data Wrappers make the whole world look like tables! Writing FDW’s in C is hard! – Or, at least, time consuming! Writing FDW’s in Python via Multicorn is easy! – Or, at least, quick! Try it for yourself!
  13. 13. Resources http://wiki.postgresql.org/wiki/SQL/MED http://wiki.postgresql.org/wiki/Foreign_data_wrappers http://multicorn.org/ https://github.com/metadaddy-sfdc/Database.com- FDW-for-PostgreSQL
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×