Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Postgres в основе вашего дата-центра, Bruce Momjian (EnterpriseDB)

1,052 views

Published on

Доклад Брюса Момджяна на HighLoad++ 2014.

Published in: Internet
  • Be the first to comment

  • Be the first to like this

Postgres в основе вашего дата-центра, Bruce Momjian (EnterpriseDB)

  1. 1. Making Postgres Central in Your Data Center BRUCE MOMJIAN October, 2014 This talk explores why Postgres is uniquely capable of functioning as a central database in enterprises. Title concept from Josh Berkus Creative Commons Attribution License http://momjian.us/presentations 1 / 38
  2. 2. Outline 1. Object-Relational (extensibility) 2. NoSQL 3. Data analytics 4. Foreign data wrappers (database federation) 5. Central role Making Postgres Central in Your Data Center 2 / 38
  3. 3. 1. Object-Relational (Extensibility) Object-relational databases like Postgres support support classes and inheritance, but most importantly, they define database functionality as objects that can be easily manipulated. http://en.wikipedia.org/wiki/Object-relational_database Making Postgres Central in Your Data Center 3 / 38
  4. 4. How Is this Accomplished? pg_trigger tgrelid pg_proc prolang prorettype pg_type pg_class pg_opclass opcdeftype reltype amrestrpos pg_statistic starelid staattnum staop pg_aggregate aggfnoid aggtransfn aggfinalfn aggtranstype pg_operator oprleft oprright oprresult oprcom oprnegate oprlsortop oprrsortop oprcode oprrest oprjoin typrelid typelem typinput typoutput typbasetype pg_database datlastsysoid pg_language pg_rewrite ev_class tgfoid pg_inherits inhrelid inhparent pg_cast castsource casttarget castfunc relam relfilenode pg_depend pg_namespace pg_shadow ambeginscan amrescan amendscan ammarkpos ambuild ambulkdelete pg_description pg_constraint contypid pg_conversion conproc pg_amproc amproc amopclaid amopopr pg_attribute indexrelid attnum amopclaid atttypid indrelid pg_attrdef pg_group adrelid pg_index adnum pg_am pg_amop amgettuple reltoastidxid aminsert reltoastrelid amcostestimate attrelid http://www.postgresql.org/docs/current/static/catalogs.html Making Postgres Central in Your Data Center 4 / 38
  5. 5. Example: ISBN Data Type CREATE EXTENSION isn; dT List of data types Schema | Name | Description --------+--------+-------------------------------------------------- public | ean13 | International European Article Number (EAN13) public | isbn | International Standard Book Number (ISBN) public | isbn13 | International Standard Book Number 13 (ISBN13) public | ismn | International Standard Music Number (ISMN) public | ismn13 | International Standard Music Number 13 (ISMN13) public | issn | International Standard Serial Number (ISSN) public | issn13 | International Standard Serial Number 13 (ISSN13) public | upc | Universal Product Code (UPC) http://www.postgresql.org/docs/current/static/isn.html Making Postgres Central in Your Data Center 5 / 38
  6. 6. ISBN Behaves Just Like Built-In Types dTS … pg_catalog | integer | -2 billion to 2 billion integer, 4-byte storage … public | isbn | International Standard Book Number (ISBN) Making Postgres Central in Your Data Center 6 / 38
  7. 7. The System Catalog Entry for INTEGER SELECT * FROM pg_type WHERE typname = ’int4’; -[ RECORD 1 ]--+--------- typname | int4 typnamespace | 11 typowner | 10 typlen | 4 typbyval | t typtype | b typcategory | N typispreferred | f typisdefined | t typdelim | , typrelid | 0 typelem | 0 typarray | 1007 typinput | int4in typoutput | int4out typreceive | int4recv typsend | int4send typmodin | - typmodout | - typanalyze | - typalign | i typstorage | p typnotnull | f Making Postgres Central in Your Data Center 7 / 38
  8. 8. The System Catalog Entry for ISBN SELECT * FROM pg_type WHERE typname = ’isbn’; -[ RECORD 1 ]--+--------------- typname | isbn typnamespace | 2200 typowner | 10 typlen | 8 typbyval | t typtype | b typcategory | U typispreferred | f typisdefined | t typdelim | , typrelid | 0 typelem | 0 typarray | 16405 typinput | isbn_in typoutput | public.isn_out typreceive | - typsend | - typmodin | - typmodout | - typanalyze | - typalign | d typstorage | p typnotnull | f Making Postgres Central in Your Data Center 8 / 38
  9. 9. Not Just Data Types, Languages CREATE EXTENSION plpythonu; dL List of languages Name | Owner | Trusted | Description -----------+----------+---------+------------------------------------------ plpgsql | postgres | t | PL/pgSQL procedural language plpythonu | postgres | f | PL/PythonU untrusted procedural language http://www.postgresql.org/docs/current/static/plpython.html Making Postgres Central in Your Data Center 9 / 38
  10. 10. Available Languages I PL/Java I PL/Perl I PL/pgSQL (like PL/SQL) I PL/PHP I PL/Python. I PL/R (like SPSS) I PL/Ruby I PL/Scheme I PL/sh I PL/Tcl I SPI (C) http://www.postgresql.org/docs/current/static/external-pl.html Making Postgres Central in Your Data Center 10 / 38
  11. 11. Specialized Indexing Methods I BTree I Hash I GiST (generalized search tree) I SP-GiST (space-partitioned GiST) I GIN (generalized inverted index) http://www.postgresql.org/docs/current/static/indexam.html Making Postgres Central in Your Data Center 11 / 38
  12. 12. Index Types Are Defined in the System Catalogs Too SELECT amname FROM pg_am; amname -------- btree hash gist gin spgist http://www.postgresql.org/docs/current/static/catalog-pg-am.html Making Postgres Central in Your Data Center 12 / 38
  13. 13. Operators Have Similar Flexibility Operators are function calls with left and right arguments of specified types: doS Schema | Name | Left arg type | Right arg type | Result type | Description … pg_catalog | + | integer | integer | integer | add dfS Schema | Name | Result data type | Argument data types | Type … pg_catalog | int4pl | integer | integer, integer | normal Making Postgres Central in Your Data Center 13 / 38
  14. 14. Other Extensibility I Casts are defined in pg_cast, int4(float8) I Aggregates are defined in pg_aggregate, sum(int4) Making Postgres Central in Your Data Center 14 / 38
  15. 15. Externally Developed Plug-Ins I PostGIS (Geographical Information System) I PL/v8 (server-side JavaScript) I experimentation, e.g. full text search was originally externally developed Making Postgres Central in Your Data Center 15 / 38
  16. 16. Offshoots of Postgres I AsterDB I Greenplum I Informix I Netezza I ParAccel I Postgres XC I Redshift (Amazon) I Truviso I Vertica I Yahoo! Everest https://wiki.postgresql.org/wiki/PostgreSQL_derived_databases http://de.slideshare.net/pgconf/elephant-roads-a-tour-of-postgres-forks Making Postgres Central in Your Data Center 16 / 38
  17. 17. Offshoots of Postgres https://raw.github.com/daamien/artwork/master/inkscape/PostgreSQL_timeline/timeline_postgresql.png Making Postgres Central in Your Data Center 17 / 38
  18. 18. Plug-In Is Not a Bad Word Many databases treat extensions as special cases, with serious limitations. Postgres built-ins use the same API as extensions, so ll extensions operate just like built-in functionality. Making Postgres Central in Your Data Center 18 / 38
  19. 19. Extensions and Built-In Facilities Behave the Same Extensions PL/R ISN PostGIS Postgres System Tables int4 sum() btree PL/pgSQL Making Postgres Central in Your Data Center 19 / 38
  20. 20. 2. NoSQL SQL Making Postgres Central in Your Data Center 20 / 38
  21. 21. NoSQL Types There is no single NoSQL technology. They all take different approaches and have different features and drawbacks: I Key-Value stores, e.g. Redis I Document databases, e.g. MongoDB (JSON) I Columnar stores: Cassandra I Graph databases: Neo4j Making Postgres Central in Your Data Center 21 / 38
  22. 22. Why NoSQL Exists Generally, NoSQL provides fast querying, auto-sharding, and flexible schemas by avoiding: I A powerful query language I A sophisticated query optimizer I Data normalization I Joins I Referential integrity I Durability Making Postgres Central in Your Data Center 22 / 38
  23. 23. Are These Drawbacks Worth the Cost? I Difficult Reporting Data must be brought to the client for analysis, e.g. no aggregates or data analysis functions. Schema-less data requires complex client-side knowledge for processing I Complex Application Design Without powerful query language and query optimizer, the client software is responsible for efficiently accessing data and for data consistency I Durability Administrators are responsible for data retention Making Postgres Central in Your Data Center 23 / 38
  24. 24. When Should NoSQL Be Used? I Massive write scaling is required, more than a single server can provide I Only simple data access pattern is required I Additional resources allocation for development is acceptable I Strong data retention or transactional guarantees are not required I Unstructured duplicate data that greatly benefits from column compression Making Postgres Central in Your Data Center 24 / 38
  25. 25. When Should Relational Storage Be Used? I Easy administration I Variable workloads and reporting I Simplified application development I Strong data retention Making Postgres Central in Your Data Center 25 / 38
  26. 26. The Best of Both Worlds: Postgres Postgres has many NoSQL features without the drawbacks: I Schema-less data types, with sophisticated indexing support I Transactional schema changes with rapid additional and removal of columns I Durability by default, but controllable per-table or per-transaction I Postgres XC and PL/Proxy allow auto-sharding for write scaling Making Postgres Central in Your Data Center 26 / 38
  27. 27. Schema-Less Data: JSON CREATE TABLE customer (id SERIAL, data JSON); INSERT INTO customer VALUES (DEFAULT, ’{"name" : "Bill", "age" : 21}’); SELECT data->’name’ FROM customer WHERE (data->’age’)::text = ’21’; ?column? ---------- "Bill" Making Postgres Central in Your Data Center 27 / 38
  28. 28. Easy Relational Schema Changes ALTER TABLE customer ADD COLUMN status CHAR(1); BEGIN WORK; ALTER TABLE customer ADD COLUMN debt_limit NUMERIC(10,2); ALTER TABLE customer ADD COLUMN creation_date TIMESTAMP WITH TIME ZONE; ALTER TABLE customer RENAME TO cust; COMMIT; Making Postgres Central in Your Data Center 28 / 38
  29. 29. 3. Data Analytics I Aggregates I Optimizer I Server-side languages, e.g. PL/R I Window functions I Bitmap heap scans I Tablespaces I Data partitioning I Materialized views I Common (recursive) table expressions I Min/Max indexes (coming in 2015) Data warehouse-specific solutions are required for parallel operations across servers. http://www.slideshare.net/PGExperts/really-big-elephants-postgresql-dw-15833438 http://wiki.postgresql.org/images/3/38/PGDay2009-EN-Datawarehousing_with_PostgreSQL.pdf Making Postgres Central in Your Data Center 29 / 38
  30. 30. Read-Only Slaves for Analytics Master Server Data Warehouse Network /pg_xlog /pg_xlog Making Postgres Central in Your Data Center 30 / 38
  31. 31. 4. Foreign Data Wrappers (Database Federation) Foreign data wrappers (SQL MED) allow queries to read and write data to foreign data sources. Foreign database support includes: I CouchDB I Informix I MongoDB I MySQL I Neo4j I Oracle I Postgres I Redis The transfer of joins, aggregates, and sorts to foreign servers is not yet implemented. http://www.postgresql.org/docs/current/static/ddl-foreign-data.html http://wiki.postgresql.org/wiki/Foreign_data_wrappers Making Postgres Central in Your Data Center 31 / 38
  32. 32. Foreign Data Wrappers to Interfaces I JDBC I LDAP I ODBC Making Postgres Central in Your Data Center 32 / 38
  33. 33. Foreign Data Wrappers to Non-Traditional Data Sources I Files I HTTP I AWS S3 I Twitter Making Postgres Central in Your Data Center 33 / 38
  34. 34. Foreign Data Wrapper Example CREATE SERVER postgres_fdw_test FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host ’localhost’, dbname ’fdw_test’); CREATE USER MAPPING FOR PUBLIC SERVER postgres_fdw_test OPTIONS (password ’’); CREATE FOREIGN TABLE other_world (greeting TEXT) SERVER postgres_fdw_test OPTIONS (table_name ’world’); det List of foreign tables Schema | Table | Server --------+-------------+------------------- public | other_world | postgres_fdw_test (1 row) Foreign Postgres server name in red; foreign table name in blue Making Postgres Central in Your Data Center 34 / 38
  35. 35. Read and Read/Write Data Sources Postgres ora_tab mon_tab tw_tab MongoDB Twitter Oracle Making Postgres Central in Your Data Center 35 / 38
  36. 36. 5. Postgres Centrality Postgres can rightly take a central place in the data center with its: I Object-Relation flexibility and extensibility I NoSQL-like workloads I Powerful data analytics capabilities I Access to foreign data sources No other database has all of these key components. Making Postgres Central in Your Data Center 36 / 38
  37. 37. Postgres’s Central Role Extensions NoSQL Postgres Oracle Foreign Data Wrappers Data Warehouse MongoDB Twitter Window Functions Data Paritioning Bitmap Scans Sharding PL/R JSON Easy DDL ISN PostGIS Making Postgres Central in Your Data Center 37 / 38
  38. 38. Conclusion http://momjian.us/presentations http://flickr.com/photos/vpickering/3617513255 Making Postgres Central in Your Data Center 38 / 38

×