Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Making Postgres Central in Your Data Center

1,025 views

Published on

Postgres has the unique ability to act as a powerful data aggregator or information hub in many IT centers bringing together data from different databases and in different formats.
This presentation reviews Postgres' extensibility, foreign data wrappers, and ability to work with structured relational and unstructured NoSQL-like information such as documents and key-value data.

The Postgres capabilities are unrivaled in enabling a complete view of customers or businesses, analyzing disparate data together, and breaking down data silos within the enterprise.

If you would like to listen to the recording please visit EnterpriseDB > Resources > Webcasts > Ondemand Webcasts.

To speak to someone about EnterpriseDB's solutions and services please email sales@enterprisedb.com.

Published in: Technology
  • A professional essay writing services can alleviate your stress in writing a successful paper and take the pressure off you to hand it in on time. Check out, please HelpWriting.net
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Making Postgres Central in Your Data Center

  1. 1. Making Postgres Central in Your Data Center BRUCE MOMJIAN January, 2015 This talk explores why Postgres is uniquely capable of functioning as a central database in enterprises. Title concept from Josh Berkus Creative Commons Attribution License http://momjian.us/presentations 1 / 44
  2. 2. PostgreSQL the database… ◮ Open Source Object Relational DBMS since 1996 ◮ Distributed under the PostgreSQL License ◮ Similar technical heritage as Oracle, SQL Server & DB2 ◮ However, a strong adherence to standards (ANSI-SQL 2008) ◮ Highly extensible and adaptable design ◮ Languages, indexing, data types, etc. ◮ E.g. PostGIS, JSONB, SQL/MED ◮ Extensive use throughout the world for applications and organizations of all types ◮ Bundled into Red Hat Enterprise Linux, Ubuntu, CentOS and Amazon Linux Making Postgres Central in Your Data Center 2 / 44
  3. 3. PostgreSQL the community… ◮ Independent community led by a Core Team of six ◮ Large, active and vibrant community ◮ www.postgresql.org ◮ Downloads, Mailing lists, Documentation ◮ Sponsors sampler: ◮ Google, Red Hat, VMWare, Skype, Salesforce, HP and EnterpriseDB ◮ http://www.postgresql.org/community/ Making Postgres Central in Your Data Center 3 / 44
  4. 4. EnterpriseDB the company… ◮ The worldwide leader of Postgres based products and services founded in 2004 ◮ Customers include 50 of the Fortune 500 and 98 of the Forbes Global 2000 ◮ Enterprise offerings: ◮ PostgreSQL Support, Services and Training ◮ Postgres Plus Advanced Server with Oracle Compatibility ◮ Tools for Monitoring, Replication, HA, Backup & Recovery Community ◮ Citizenship ◮ Contributor of key features: Materialized Views, JSON, & more ◮ Nine community members on staff Making Postgres Central in Your Data Center 4 / 44
  5. 5. EnterpriseDB the company… Making Postgres Central in Your Data Center 5 / 44
  6. 6. Outline 1. Object-Relational (extensibility) 2. NoSQL 3. Data analytics 4. Foreign data wrappers (database federation) 5. Central role Making Postgres Central in Your Data Center 6 / 44
  7. 7. 1. Object-Relational (Extensibility) Object-relational databases like Postgres support support classes and inheritance, but most importantly, they define database functionality as objects that can be easily manipulated. http://en.wikipedia.org/wiki/Object-relational_database Making Postgres Central in Your Data Center 7 / 44
  8. 8. How Is this Accomplished? starelid staattnum staop pg_statistic oprleft oprright oprresult oprcom oprnegate oprlsortop oprrsortop oprcode oprrest oprjoin pg_operator typrelid typelem typinput typoutput typbasetype pg_type prolang prorettype pg_proc pg_rewrite ev_class datlastsysoid pg_database tgfoid tgrelid pg_trigger inhrelid pg_inherits inhparent pg_language pg_namespacepg_depend pg_shadow pg_aggregate aggfinalfn aggtransfn aggfnoid aggtranstype castsource casttarget pg_cast castfunc pg_description pg_constraint contypid pg_conversion conproc amopopr amopclaid pg_attribute indexrelid attnum amopclaid atttypid indrelid pg_attrdef pg_group adrelid pg_index adnum pg_am pg_amop amgettuple reltoastidxid aminsert reltoastrelid amcostestimate amproc ambeginscan pg_amproc amrescan relfilenode amendscan relam ammarkpos reltype amrestrpos pg_class ambuild opcdeftype ambulkdelete pg_opclass attrelid http://www.postgresql.org/docs/current/static/catalogs.html Making Postgres Central in Your Data Center 8 / 44
  9. 9. Example: ISBN Data Type CREATE EXTENSION isn; dT List of data types Schema | Name | Description --------+--------+-------------------------------------------------- public | ean13 | International European Article Number (EAN13) public | isbn | International Standard Book Number (ISBN) public | isbn13 | International Standard Book Number 13 (ISBN13) public | ismn | International Standard Music Number (ISMN) public | ismn13 | International Standard Music Number 13 (ISMN13) public | issn | International Standard Serial Number (ISSN) public | issn13 | International Standard Serial Number 13 (ISSN13) public | upc | Universal Product Code (UPC) http://www.postgresql.org/docs/current/static/isn.html Making Postgres Central in Your Data Center 9 / 44
  10. 10. ISBN Behaves Just Like Built-In Types dTS … pg_catalog | integer | -2 billion to 2 billion integer, 4-byte storage … public | isbn | International Standard Book Number (ISBN) Making Postgres Central in Your Data Center 10 / 44
  11. 11. The System Catalog Entry for INTEGER SELECT * FROM pg_type WHERE typname = ’int4’; -[ RECORD 1 ]--+--------- typname | int4 typnamespace | 11 typowner | 10 typlen | 4 typbyval | t typtype | b typcategory | N typispreferred | f typisdefined | t typdelim | , typrelid | 0 typelem | 0 typarray | 1007 typinput | int4in typoutput | int4out typreceive | int4recv typsend | int4send typmodin | - typmodout | - typanalyze | - typalign | i typstorage | p typnotnull | f typbasetype | 0 Making Postgres Central in Your Data Center 11 / 44
  12. 12. The System Catalog Entry for ISBN SELECT * FROM pg_type WHERE typname = ’isbn’; -[ RECORD 1 ]--+--------------- typname | isbn typnamespace | 2200 typowner | 10 typlen | 8 typbyval | t typtype | b typcategory | U typispreferred | f typisdefined | t typdelim | , typrelid | 0 typelem | 0 typarray | 16405 typinput | isbn_in typoutput | public.isn_out typreceive | - typsend | - typmodin | - typmodout | - typanalyze | - typalign | d typstorage | p typnotnull | f typbasetype | 0 Making Postgres Central in Your Data Center 12 / 44
  13. 13. Not Just Data Types, Languages CREATE EXTENSION plpythonu; dL List of languages Name | Owner | Trusted | Description -----------+----------+---------+------------------------------------------ plpgsql | postgres | t | PL/pgSQL procedural language plpythonu | postgres | f | PL/PythonU untrusted procedural language http://www.postgresql.org/docs/current/static/plpython.html Making Postgres Central in Your Data Center 13 / 44
  14. 14. Available Languages ◮ PL/Java ◮ PL/Perl ◮ PL/pgSQL (like PL/SQL) ◮ PL/PHP ◮ PL/Python. ◮ PL/R (like SPSS) ◮ PL/Ruby ◮ PL/Scheme ◮ PL/sh ◮ PL/Tcl ◮ SPI (C) http://www.postgresql.org/docs/current/static/external-pl.html Making Postgres Central in Your Data Center 14 / 44
  15. 15. Specialized Indexing Methods ◮ BTree ◮ Hash ◮ GiST (generalized search tree) ◮ SP-GiST (space-partitioned GiST) ◮ GIN (generalized inverted index) http://www.postgresql.org/docs/current/static/indexam.html Making Postgres Central in Your Data Center 15 / 44
  16. 16. Index Types Are Defined in the System Catalogs Too SELECT amname FROM pg_am; amname -------- btree hash gist gin spgist http://www.postgresql.org/docs/current/static/catalog-pg-am.html Making Postgres Central in Your Data Center 16 / 44
  17. 17. Operators Have Similar Flexibility Operators are function calls with left and right arguments of specified types: doS Schema | Name | Left arg type | Right arg type | Result type | Description … pg_catalog | + | integer | integer | integer | add dfS Schema | Name | Result data type | Argument data types | Type … pg_catalog | int4pl | integer | integer, integer | normal Making Postgres Central in Your Data Center 17 / 44
  18. 18. Other Extensibility ◮ Casts are defined in pg_cast, int4(float8) ◮ Aggregates are defined in pg_aggregate, sum(int4) Making Postgres Central in Your Data Center 18 / 44
  19. 19. Externally Developed Plug-Ins ◮ PostGIS (Geographical Information System) ◮ PL/v8 (server-side JavaScript) ◮ experimentation, e.g. full text search was originally externally developed Making Postgres Central in Your Data Center 19 / 44
  20. 20. Offshoots of Postgres ◮ AsterDB ◮ Greenplum ◮ Informix ◮ Netezza ◮ ParAccel ◮ Postgres XC ◮ Redshift (Amazon) ◮ Truviso ◮ Vertica ◮ Yahoo! Everest https://wiki.postgresql.org/wiki/PostgreSQL_derived_databases http://de.slideshare.net/pgconf/elephant-roads-a-tour-of-postgres-forks Making Postgres Central in Your Data Center 20 / 44
  21. 21. Offshoots of Postgres https://raw.github.com/daamien/artwork/master/inkscape/PostgreSQL_timeline/timeline_postgresql.png Making Postgres Central in Your Data Center 21 / 44
  22. 22. Plug-In Is Not a Bad Word Many databases treat extensions as special cases, with serious limitations. Postgres built-ins use the same API as extensions, so ll extensions operate just like built-in functionality. Making Postgres Central in Your Data Center 22 / 44
  23. 23. Extensions and Built-In Facilities Behave the Same Extensions PL/R ISN PostGIS Postgres System Tables int4 btree sum() PL/pgSQL Making Postgres Central in Your Data Center 23 / 44
  24. 24. 2. NoSQL SQL Making Postgres Central in Your Data Center 24 / 44
  25. 25. NoSQL Types There is no single NoSQL technology. They all take different approaches and have different features and drawbacks: ◮ Key-Value stores, e.g. Redis ◮ Document databases, e.g. MongoDB (JSON) ◮ Columnar stores: Cassandra ◮ Graph databases: Neo4j Making Postgres Central in Your Data Center 25 / 44
  26. 26. Why NoSQL Exists Generally, NoSQL is optimized for: ◮ Fast querying ◮ Auto-sharding ◮ Flexible schemas Making Postgres Central in Your Data Center 26 / 44
  27. 27. NoSQL Sacrifices ◮ A powerful query language ◮ A sophisticated query optimizer ◮ Data normalization ◮ Joins ◮ Referential integrity ◮ Durability Making Postgres Central in Your Data Center 27 / 44
  28. 28. Are These Drawbacks Worth the Cost? ◮ Difficult Reporting Data must be brought to the client for analysis, e.g. no aggregates or data analysis functions. Schema-less data requires complex client-side knowledge for processing ◮ Complex Application Design Without powerful query language and query optimizer, the client software is responsible for efficiently accessing data and for data consistency ◮ Durability Administrators are responsible for data retention Making Postgres Central in Your Data Center 28 / 44
  29. 29. When Should NoSQL Be Used? ◮ Massive write scaling is required, more than a single server can provide ◮ Only simple data access pattern is required ◮ Additional resource allocation for development is acceptable ◮ Strong data retention or transactional guarantees are not required ◮ Unstructured duplicate data that greatly benefits from column compression Making Postgres Central in Your Data Center 29 / 44
  30. 30. When Should Relational Storage Be Used? ◮ Easy administration ◮ Variable workloads and reporting ◮ Simplified application development ◮ Strong data retention Making Postgres Central in Your Data Center 30 / 44
  31. 31. The Best of Both Worlds: Postgres Postgres has many NoSQL features without the drawbacks: ◮ Schema-less data types, with sophisticated indexing support ◮ Transactional schema changes with rapid additional and removal of columns ◮ Durability by default, but controllable per-table or per-transaction ◮ Postgres XC and PL/Proxy allow auto-sharding for write scaling Making Postgres Central in Your Data Center 31 / 44
  32. 32. Schema-Less Data: JSON CREATE TABLE customer (id SERIAL, data JSON); INSERT INTO customer VALUES (DEFAULT, ’{"name" : "Bill", "age" : 21}’); SELECT data->’name’ FROM customer WHERE (data->’age’)::text = ’21’; ?column? ---------- "Bill" Making Postgres Central in Your Data Center 32 / 44
  33. 33. Easy Relational Schema Changes ALTER TABLE customer ADD COLUMN status CHAR(1); BEGIN WORK; ALTER TABLE customer ADD COLUMN debt_limit NUMERIC(10,2); ALTER TABLE customer ADD COLUMN creation_date TIMESTAMP WITH TIME ZONE; ALTER TABLE customer RENAME TO cust; COMMIT; Making Postgres Central in Your Data Center 33 / 44
  34. 34. 3. Data Analytics ◮ Aggregates ◮ Optimizer ◮ Server-side languages, e.g. PL/R ◮ Window functions ◮ Bitmap heap scans ◮ Tablespaces ◮ Data partitioning ◮ Materialized views ◮ Common (recursive) table expressions ◮ Min/Max indexes (coming in 2015) Data warehouse-specific solutions are required for parallel operations across servers. http://www.slideshare.net/PGExperts/really-big-elephants-postgresql-dw-15833438 http://wiki.postgresql.org/images/3/38/PGDay2009-EN-Datawarehousing_with_PostgreSQL.pdf Making Postgres Central in Your Data Center 34 / 44
  35. 35. Read-Only Slaves for Analytics Network Data WarehouseMaster Server /pg_xlog/pg_xlog Making Postgres Central in Your Data Center 35 / 44
  36. 36. 4. Foreign Data Wrappers (Database Federation) Foreign data wrappers (SQL MED) allow queries to read and write data to foreign data sources. Foreign database support includes: ◮ CouchDB ◮ Informix ◮ MongoDB ◮ MySQL ◮ Neo4j ◮ Oracle ◮ Postgres ◮ Redis The transfer of joins, aggregates, and sorts to foreign servers is not yet implemented. http://www.postgresql.org/docs/current/static/ddl-foreign-data.html http://wiki.postgresql.org/wiki/Foreign_data_wrappers Making Postgres Central in Your Data Center 36 / 44
  37. 37. Foreign Data Wrappers to Interfaces ◮ JDBC ◮ LDAP ◮ ODBC Making Postgres Central in Your Data Center 37 / 44
  38. 38. Foreign Data Wrappers to Non-Traditional Data Sources ◮ Files ◮ HTTP ◮ AWS S3 ◮ Twitter Making Postgres Central in Your Data Center 38 / 44
  39. 39. Foreign Data Wrapper Example CREATE SERVER postgres_fdw_test FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host ’localhost’, dbname ’fdw_test’); CREATE USER MAPPING FOR PUBLIC SERVER postgres_fdw_test OPTIONS (password ’’); CREATE FOREIGN TABLE other_world (greeting TEXT) SERVER postgres_fdw_test OPTIONS (table_name ’world’); det List of foreign tables Schema | Table | Server --------+-------------+------------------- public | other_world | postgres_fdw_test (1 row) Foreign Postgres server name in red; foreign table name in blue Making Postgres Central in Your Data Center 39 / 44
  40. 40. Read and Read/Write Data Sources Postgres ora_tab tw_tab mon_tab MongoDB Twitter Oracle Making Postgres Central in Your Data Center 40 / 44
  41. 41. 5. Postgres Centrality Postgres can rightly take a central place in the data center with its: ◮ Object-Relation flexibility and extensibility ◮ NoSQL-like workloads ◮ Powerful data analytics capabilities ◮ Access to foreign data sources No other database has all of these key components. Making Postgres Central in Your Data Center 41 / 44
  42. 42. Postgres’s Central Role Extensions NoSQL Postgres Warehouse Data Foreign Data Wrappers Window Functions Data Paritioning Bitmap Scans Sharding Oracle Twitter MongoDB Easy DDL JSON PL/R ISN PostGIS Making Postgres Central in Your Data Center 42 / 44
  43. 43. Additional Resources… ◮ Postgres Downloads: ◮ www.enterprisedb.com/downloads ◮ Product and Services information: ◮ info@enterprisedb.com Making Postgres Central in Your Data Center 43 / 44
  44. 44. Conclusion http://momjian.us/presentations http://flickr.com/photos/vpickering/3617513255 Making Postgres Central in Your Data Center 44 / 44

×