• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Postgres-XC as a Key Value Store Compared To MongoDB
 

Postgres-XC as a Key Value Store Compared To MongoDB

on

  • 9,017 views

This presentation discusses how Postgres-XC can be used as a PostgreSQL-based key-value store using features like hstore and JSON. It also compares performance to MongoDB for a read workload

This presentation discusses how Postgres-XC can be used as a PostgreSQL-based key-value store using features like hstore and JSON. It also compares performance to MongoDB for a read workload

Statistics

Views

Total Views
9,017
Views on SlideShare
8,987
Embed Views
30

Actions

Likes
12
Downloads
0
Comments
1

3 Embeds 30

https://twitter.com 28
https://web.tweetdeck.com 1
http://webmail.tatung.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • I have updated this presentation to provide better context of the measurements and more measurement results. More measurements coming in the future
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Postgres-XC as a Key Value Store Compared To MongoDB Postgres-XC as a Key Value Store Compared To MongoDB Presentation Transcript

    • A Postgres-XCDistributed Key-Value StoreMason SharpApril 15, 2013CC License: Attribution-NonCommercial-ShareAlike
    • Who Am I?Mason Sharp●Original architect of Stado / GridSQL●One of original architects of Postgres-XC●Former architect at EnterpriseDB●Co-organizer of NYC PostgreSQL User Group●Co-founder and CTO of
    • Agenda●Why use a key-value store?●PostgreSQL features●XML●hstore●JSON●Postgres-XC Overview●Measurements: MongoDB versus Postgres-XC
    • Agenda●Why use a key-value store?●PostgreSQL features●XML●hstore●JSON●Postgres-XC Overview●Measurements: MongoDB versus Postgres-XC
    • Why Use a Key-Value Store?●Document oriented vs. row oriented●Unstructured data●Semi-structured data●Self-describing / schema-less●Uses Tags●Dynamic attributes for different objects●Dwight Merriman, CEO 10gen (paraphrasing):●“Some customers use MongoDB just for the schema-less features. They dont need the scalability andrun on one single server” (!)●“Easier for developers” (...)
    • Why Use a Key-Value Store? (2)●Key-value makes for an easy distributed store●Multiple servers●In-memory●No complicated schema changes●But PostgreSQLs ALTER TABLE exclusive locksmay be brief●Need to be “web-scale”●Perception that it scales better●What if it no longer fits in memory?●A series of unfortunate anecdotes
    • PostgreSQLDocument Store Capabilities
    • XML●--with-libxml at build time●Native data type●CREATE TABLE foo (myid int, data xml)●ValidationINSERT INTO foo VALUES (2, <aaa);ERROR: invalid XML contentDetail: line 1: Couldnt find end of Start Tagaaa line 1●Xpath●Mapping & Export functions
    • hstore●Contrib module●CREATE EXTENSION hstore●Key/value pairs●Data type
    • hstoreCREATE TABLE foo (myid int, hdata hstore);INSERT INTO foo VALUES (10,"name"=>"fred", "department"=>"IT");
    • hstoreSELECT hdata->name FROM foo WHERE id = 10;?column?----------fred(1 row)# Extract all department values where it is an attributeSELECT hdata->departmentFROM fooWHERE hdata ? department;
    • Hstore Manipulation●Concatenatea=>b, c=>d::hstore || c=>x, d=>q::hstore"a"=>"b", "c"=>"x", "d"=>"q"●Delete elementdelete(a=>1,b=>2,b)"a"=>"1"
    • hstore# Get a list of unique keysSELECT DISTINCT (each(hdata)).keyFROM foo
    • hstore - Indexes●Btree index only helps with =●Gin and gist indexes will help with operators●@> left operand contains right●? contains key●?& contains all keys in array●?| contains at least one key in array●Can create index on custom function●Extract a particular key value
    • JSON●JavaScript Object Notation●PostgreSQL 9.2 basic support●array_to_json●row_to_jsonNote: Postgres-XC 1.0.2 based on PostgreSQL9.1, will be based on 9.2 soon
    • JSON – looking ahead toPostgreSQL 9.3●PostgreSQL 9.3●json_agg●hstore_to_json●hstore_to_json_loose●… and much morehttp://www.postgresql.org/docs/devel/static/functions-json.html
    • Composite TypeCREATE TYPE address AS (street TEXT,city TEXT,state TEXT,zip CHAR(10));CREATE TABLE customer (full_name TEXT,mail_address address);
    • row_to_jsontest1=# select row_to_json(customer) fromcustomer;{"full_name":"Joe Lee","mail_address": {"street":"100 Broad Street","city":"Red Bank","state":"NJ","zip":"07701 "}}
    • 19●PostgreSQL-based database clusterSame API to Apps as PostgreSQL• Same drivers●Symmetric Multi-headed ClusterNo master, no slave• Not just PostgreSQL replication.• Application can read/write to any coordinator serverConsistent database view to all the transactions• Complete ACID property to all the transactions in the cluster●Scales both for Write and Read
    • Sep 20, 2012 Postgres-XC 20
    • Sep 20, 2012 Postgres-XC 21Postgres-XC ClusterCoordinatorData NodePG-XC ServerCoordinatorData NodeCoordinatorData NodeCoordinatorData Node・・・・・Communication amongPG-XC serversAdd PG-XC servers asneededGlobal TransactionManagerApplication can connect to any server to have the same database view and service.GTMPG-XC Server PG-XC Server PG-XC Server
    • Coordinator Overview●Based on PostgreSQL●Accepts connections from clients●Parses and plans requests●Interacts with Global Transaction Manager●Uses pooler for Data Node connections●Sends down XIDs and snapshots to Data Nodes●Collects results and returns to client●Uses two phase commit if necessary22
    • Data Node Overview●Based on PostgreSQL●Where user created data is actually stored●Coordinators (not clients) connects to DataNodes●Accepts XID and snapshots from Coordinator●The rest is fairly similar to vanilla PostgreSQL23
    • Sep 20, 2012 Postgres-XC 24Global Transaction ManagerCluster nodesGTMXIDSnapshotTimestampSequence values
    • GTM Overview●Issues Transaction IDs (XIDs)●Issues Snapshots●Issues Timestamps●Issues Sequences●Based on PostgreSQL procarray code●Multi-threaded25
    • GTM Proxy●Runs on other nodes●Groups requests together●Reduces number of connections to GTM●Reduces traffic to GTM26
    • Sep 20, 2012 Postgres-XC 27Summary● Coordinator● Visible to apps● SQL analysis, planning, execution● Connection pooling● Datanode (or simply “NODE”)● Actual database store● Local SQL execution● GTM (Global Transaction Manager)● Provides consistent database view to transactions– GXID (Global Transaction ID)– Snapshot (List of active transactions)– Other global values such as SEQUENCE● GTM Proxy, integrates server-local transaction requirement for performancePostgres-XC core, based uponvanilla PostgreSQLShare same binaryMay want to colocateDifferent binaries
    • MongoDB vs Postgres-XCPerformance Comparison●Three data nodes (16GB RAM each)●Postgres-XC also used a coordinator●Adds latency●Out-of-the-box default configuration●No replicas
    • Insert Comparison – single thread●0 – 1M Rows●MongoDB: 7m 06s●Postgres-XC: 131m 1s●Postgres-XC COPY: 43s●10M – 20M Rows●MongoDB: 64m 48●Postgres-XC: 354m 56sGTM in XC adds a lot of latency hurtingsingle-threaded performance
    • Read Comparison(shorter is better)1 2 3 4 5 6 7 8 9 1000.511.522.5MongoDBPostgres-XCRows (millions)Time(seconds)
    • Update Comparison – single thread50 GB, single thread●1000 Updates by partitioned key●MongoDB: 43s●Postgres-XC: 1m 6s●1000 Updates by indexed non-partitioned key●MongoDB: 7m 55s●Postgres-XC: 1m 54sNon-partitioned index-based faster in XC
    • Update Concurrency on Key
    • Possible Future Tests●Insert,Select concurrency test (important)●Mixed workload●Measure in-memory and not in-memory●Impact of replicas for availability●MongoDB replicas●Postgres-XC streaming replication●Have seen about 15% perf drop for two sync slaves●MongoDB Write-Concern durability settings (tryjournaled)●Hstore
    • Other PostgreSQL Results?●Christophe Pettus:wiki.postgresql.org/images/b/b4/Pg-as-nosql-pgday-fosdem-2013.pdf●Single laptop-based tests, but interesting●
    • Summary●PostgreSQL has schema-less functionality built-in and can act as a key-value store●Postgres-XC can scale this out horizontally tomultiple servers●MongoDB performs much better for lowconcurrency for inserts●In XC, use COPY or multiple threads to populate●Postgres-XC performs better for non-partitionedindexed access●Postgres-XC can perform about the same toMongoDB for reads
    • Summary (2)If Postgres-XC generally performs similarly toMongoDB, why not use XC and●Stick with ACID●Feel secure with PostgreSQL maturity●Leverage PostgreSQL features and community
    • Thank YouMason Sharpmason@stormdb.com@mason_db
    • Content Attribution●Postgres-XC Development Group●Koichi Suzuki●Michael Paquier●Ashutosh Bapat●Pavan Deolasee●Christophe Pettus●Mason Sharp●...