Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Let's turn your PostgreSQL into columnar store with cstore_fdw


Published on

Talk on cstore_fdw PostgreSQL extension developed by Citus Data, presented at PgConf EU 2015 in Vienna.

Published in: Data & Analytics
  • Login to see the comments

Let's turn your PostgreSQL into columnar store with cstore_fdw

  1. 1. Let’s turn your PostgreSQL into columnar store with cstore_fdw Jan Holčapek PgConf EU 2015 Vienna
  2. 2. What’s cstore_fdw? Foreign data wrapper allowing for storing and reading data in a columnar fashion.
  3. 3. Trivia: foreign data wrapper Extensibility feature of PostgreSQL introduced in 9.1. Means for accessing data outside the database. Plenty of them Two of them officially supported file_fdw postgres_fdw
  4. 4. Trivia: columnar storage Row-oriented store: data for a particular row are stored consecutively on a disk. Column-oriented store: data for a particular column are stored consecutively on a disk. col 1 col 2 ... ... ... ... col N-1 col N col 1 col 2 ... ... ... ... col N-1 col N
  5. 5. Naïve example Table 100 columns 10 million rows 1 billion values Sum of a single column Row store Read 1 billion values to use only 10 million of them Column store Read 10 million values actually needed
  6. 6. Benefits of columnar storage Less I/O Projections Better compression and/or encoding Column-level aggregates
  7. 7. What’s cstore_fdw again? Foreign data wrapper allowing for storing data to and reading data from the CStore file format.
  8. 8. CStore file format Derived from ORC format Optimized Row Columnar employed in Apache Hive Row stripes, row blocks, skip lists col 1 col 2 ... col N-1 col N val 1 val 5 val 9 val 2 val 6 val 10 val 3 val 7 ... val 4 val 8
  9. 9. How to get it Download and compile manually git clone cd cstore_fdw make sudo make install edit postgresql.conf add cstore_fdw to shared_preload_libraries restart postgresql Wait until there’s a Fedora package dnf install postgresql-cstore_fdw
  10. 10. How to get it working As a superuser CREATE EXTENSION cstore_fdw; GRANT USAGE ON FOREIGN DATA WRAPPER TO public; As a regular user CREATE SERVER cstore_server FOREIGN DATA WRAPPER cstore_fdw; CREATE FOREIGN TABLE foo (...) SERVER cstore_server OPTIONS ( filename ’/tmp/foo’, stripe_row_count 150000, block_row_count 10000, compression ’pglz’ -- or ’none’ );
  11. 11. What we can do with cstore Version 1.3 released in July 2015 Create foreign table Alter foreign table Add or drop columns Load data using COPY Load data using INSERT INTO … SELECT Unload data using COPY TO
  12. 12. What we cannot do with cstore Plain insert Update Upsert Delete data Truncate
  13. 13. Conclusion Not a good match for OLTP workload Great match for OLAP workload
  14. 14. Demo! create native and cstore tables, load it with data compare loading times and disk utilization simple sum with date limit sum with group by with date limit slightly more complex query
  15. 15. That’s all folks! Give cstore_fdw a try. Thank you! Let Citus Data guys know how you like it or what you are missing. Thank you! Submit feedback on my talk. Thank you!