Federating Queries Using postgres fdw
john melesky
Rentrak, Inc
September 17, 2013
Who Am I?
A long-time programmer, working with PostgreSQL in the
cloud
Who Am I?
A long-time programmer, working with PostgreSQL in the
cloud my butt
Who Am I?
A long-time programmer, working with PostgreSQL in the
cloud my butt
Now, a DBA, working with PostgreSQL on real...
Who Am I?
A long-time programmer, working with PostgreSQL in the
cloud my butt
Now, a DBA, working with PostgreSQL on real...
Who Am I?
A long-time programmer, working with PostgreSQL in the
cloud my butt
Now, a DBA, working with PostgreSQL on real...
PostgreSQL inheritance partitioning
create table transactions (
id serial,
user_id bigint,
time_utc timestamp,
int_value b...
PostgreSQL inheritance partitioning
create table transactions (
id serial,
user_id bigint,
time_utc timestamp,
int_value b...
Old-school partitioning
create view transactions as (
select * from transactions_201301
union all
select * from transactio...
Why don’t we still use this?
Why don’t we still use this?
1. No insert triggers on views
Why don’t we still use this?
1. No insert triggers on views
2. No ”inherit indexes” without additional misdirection
Why don’t we still use this?
1. No insert triggers on views
2. No ”inherit indexes” without additional misdirection
3. Bas...
Postgres Foreign Data Wrapper
-- just once
create extension postgres_fdw;
-- once per data node
create server node0 foreig...
Federating, Old-school
create view transactions as (
select * from transactions_node0
union all
select * from transactions...
Querying
primary=# explain select count(*) from transactions;
QUERY PLAN
-------------------------------------------------...
Querying
primary=# explain verbose select count(*) from transactions;
QUERY PLAN
-----------------------------------------...
Querying
primary=# select count(*) from transactions;
count
---------
1095336
(1 row)
Time: 3035.054 ms
Round-robin
primary
node 0
(id % 4 = 0)
node 1
(id % 4 = 1)
node 2
(id % 4 = 2)
node 3
(id % 4 = 3)
Round-robin
primary=# create foreign table transactions_node0 (
primary(# id serial,
primary(# user_id bigint,
primary(# t...
Domain-based (aka ”sharding”)
primary
node 0
(customer = 'bigone')
node 1
(customer in ('bigtwo', 'bigthree')
node 2
(cust...
Range-based
primary
node 0
(date between '2013-01-01' and '2013-01-31')
node 1
(date between ...)
node 2
(date between ......
Table-based
primary
node 0
(users table(s))
node 1
(transactions table)
node 2
(session tables)
Multi-head
primary1
node0 node1node2 node3
primary2
Multi-head
primary1
node0 node1node2 node3
primary2primary3 primary4
Demo time
Limitations: Network traffic
Limitations: Network traffic
primary=# select count(*) from transactions_local;
count
---------
1095336
(1 row)
Time: 209.09...
Limitations: Network traffic
primary=# select count(*) from transactions_local;
count
---------
1095336
(1 row)
Time: 209.09...
Limitations: Dumb queries
Limitations: Dumb queries
primary=# explain verbose select count(*) from transactions;
QUERY PLAN
------------------------...
Limitations: Dumb queries
primary=# explain verbose select count(*) from transactions;
QUERY PLAN
------------------------...
Limitations: Dumb queries
select type, count(*)
from users
group by type
order by 2 desc;
Limitations: Joins
Limitations: Joins
select count(*)
from transactions t, users u
where t.user_id = u.id
and u.type = ’mistaken’;
Limitations: Keys
Limitations: Keys
’Nuff said
Limitations: Constraint exclusion
Remember this?
ERROR: constraints are not supported on foreign tables
LINE 6: check ((id...
Limitations: Single-threaded executer
Limitations: Single-threaded executer
How many nodes do you have?
Limitations: Single-threaded executer
How many nodes do you have?
Do you know what they’re doing?
Strategies
Large working set, small nodes
Node-level partitioning
Heavy distributed processing
Multi-head
Strategy: Large working set, small nodes
Strategy: Large working set, small nodes
Your working set is larger than one node’s RAM
Strategy: Large working set, small nodes
Your working set is larger than one node’s RAM
... but you have lots of nodes
Strategy: Large working set, small nodes
Your working set is larger than one node’s RAM
... but you have lots of nodes
(an...
Strategy: Large working set, small nodes
Your working set is larger than one node’s RAM
... but you have lots of nodes
(an...
Strategy: Node-level partitioning
Like parititioning, but with a separate node per partition group!
Strategy: Node-level partitioning
Like parititioning, but with a separate node per partition group!
As a total strategy, t...
Heavy distributed processing
Heavy distributed processing
Take advantage of lots of CPUs
Heavy distributed processing
Take advantage of lots of CPUs
Works well when you have node-discrete workloads
Heavy distributed processing
Take advantage of lots of CPUs
Works well when you have node-discrete workloads
Lock manageme...
Heavy distributed processing
Take advantage of lots of CPUs
Works well when you have node-discrete workloads
Lock manageme...
Multi-headed
Multi-headed
Like replication, but with no overhead or delay!
Multi-headed
Like replication, but with no overhead or delay!
Also, no storage overhead!
Multi-headed
Like replication, but with no overhead or delay!
Also, no storage overhead!
Might work well with the distribu...
Multi-headed
Like replication, but with no overhead or delay!
Also, no storage overhead!
Might work well with the distribu...
Pan-Strategy Advice
Pan-Strategy Advice
Think very carefully about what tables should live where
Pan-Strategy Advice
Think very carefully about what tables should live where
Think very carefully about tuning settings (e...
Pan-Strategy Advice
Think very carefully about what tables should live where
Think very carefully about tuning settings (e...
Pan-Strategy Advice
Think very carefully about what tables should live where
Think very carefully about tuning settings (e...
Pan-Strategy Advice
Think very carefully about what tables should live where
Think very carefully about tuning settings (e...
Thanks!
Questions?
Any questions?
Questions?
Any questions?
John, do you use this approach for your databases?
Questions?
Any questions?
John, do you use this approach for your databases?
Why not?
Thanks!
Thanks!
Plug: Stephen Frost has another postgres fdw talk tomorrow
Thanks!
Plug: Stephen Frost has another postgres fdw talk tomorrow
Also: Rentrak is hiring: programmers, sysadmins, and de...
Federating Queries Using postgres fdw
Introduction
Who am I?
Partitioning
PostgreSQL inheritance partitioning
Old-school p...
Upcoming SlideShare
Loading in …5
×

John Melesky - Federating Queries Using Postgres FDW @ Postgres Open

3,473 views

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,473
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
34
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

John Melesky - Federating Queries Using Postgres FDW @ Postgres Open

  1. 1. Federating Queries Using postgres fdw john melesky Rentrak, Inc September 17, 2013
  2. 2. Who Am I? A long-time programmer, working with PostgreSQL in the cloud
  3. 3. Who Am I? A long-time programmer, working with PostgreSQL in the cloud my butt
  4. 4. Who Am I? A long-time programmer, working with PostgreSQL in the cloud my butt Now, a DBA, working with PostgreSQL on real machines with real disks
  5. 5. Who Am I? A long-time programmer, working with PostgreSQL in the cloud my butt Now, a DBA, working with PostgreSQL on real machines VMWare with real disks
  6. 6. Who Am I? A long-time programmer, working with PostgreSQL in the cloud my butt Now, a DBA, working with PostgreSQL on real machines VMWare with real disks NetApps
  7. 7. PostgreSQL inheritance partitioning create table transactions ( id serial, user_id bigint, time_utc timestamp, int_value bigint, txt_value text, primary key (id) ); create table transactions_201306 ( like transactions including indexes, check (time_utc >= ’2013-06-01’ and time_utc < ’2013-07-01’) ) inherits (transactions);
  8. 8. PostgreSQL inheritance partitioning create table transactions ( id serial, user_id bigint, time_utc timestamp, int_value bigint, txt_value text, primary key (id) ); create table transactions_201306 ( like transactions including indexes, check (time_utc >= ’2013-06-01’ and time_utc < ’2013-07-01’) ) inherits (transactions); You know this already
  9. 9. Old-school partitioning create view transactions as ( select * from transactions_201301 union all select * from transactions_201302 union all select * from transactions_201303 union all select * from transactions_201304 union all ... );
  10. 10. Why don’t we still use this?
  11. 11. Why don’t we still use this? 1. No insert triggers on views
  12. 12. Why don’t we still use this? 1. No insert triggers on views 2. No ”inherit indexes” without additional misdirection
  13. 13. Why don’t we still use this? 1. No insert triggers on views 2. No ”inherit indexes” without additional misdirection 3. Basically, we have a better option with inheritence partitioning
  14. 14. Postgres Foreign Data Wrapper -- just once create extension postgres_fdw; -- once per data node create server node0 foreign data wrapper postgres_fdw options (connection stuff); create user mapping for app_user server node0; -- once per table per node create foreign table transactions_node0 (table definition) server node0 options (table_name ’transactions’);
  15. 15. Federating, Old-school create view transactions as ( select * from transactions_node0 union all select * from transactions_node1 union all select * from transactions_node2 union all select * from transactions_node3 union all ... );
  16. 16. Querying primary=# explain select count(*) from transactions; QUERY PLAN --------------------------------------------------------- Aggregate (cost=1767.38..1767.39 rows=1 width=0) -> Append (cost=100.00..1699.12 rows=27304 width=0) -> Foreign Scan on transactions_node0 (cost=100.00..212.39 rows=3413 width=0) -> Foreign Scan on transactions_node1 (cost=100.00..212.39 rows=3413 width=0) -> Foreign Scan on transactions_node2 (cost=100.00..212.39 rows=3413 width=0) -> Foreign Scan on transactions_node3 (cost=100.00..212.39 rows=3413 width=0) -> Foreign Scan on transactions_node4 (cost=100.00..212.39 rows=3413 width=0) ... (10 rows) Time: 1.226 ms
  17. 17. Querying primary=# explain verbose select count(*) from transactions; QUERY PLAN ----------------------------------------------------------------- Aggregate (cost=1767.38..1767.39 rows=1 width=0) Output: count(*) -> Append (cost=100.00..1699.12 rows=27304 width=0) -> Foreign Scan on public.transactions_node0 (cost=100.00..212.39 rows=3413 width=0) Remote SQL: SELECT NULL FROM public.transactions -> Foreign Scan on public.transactions_node1 (cost=100.00..212.39 rows=3413 width=0) Remote SQL: SELECT NULL FROM public.transactions -> Foreign Scan on public.transactions_node2 (cost=100.00..212.39 rows=3413 width=0) Remote SQL: SELECT NULL FROM public.transactions ... (19 rows) Time: 1.273 ms
  18. 18. Querying primary=# select count(*) from transactions; count --------- 1095336 (1 row) Time: 3035.054 ms
  19. 19. Round-robin primary node 0 (id % 4 = 0) node 1 (id % 4 = 1) node 2 (id % 4 = 2) node 3 (id % 4 = 3)
  20. 20. Round-robin primary=# create foreign table transactions_node0 ( primary(# id serial, primary(# user_id bigint, primary(# time_utc timestamp, primary(# int_value bigint, primary(# txt_value text, primary(# check ((id % 8) = 0) primary(# ) server node0 primary(# options (table_name ’transactions’); ERROR: constraints are not supported on foreign tables LINE 6: check ((id % 8) = 0)) server node0 ...
  21. 21. Domain-based (aka ”sharding”) primary node 0 (customer = 'bigone') node 1 (customer in ('bigtwo', 'bigthree') node 2 (customer in (...)) node 3 (customer in (...))
  22. 22. Range-based primary node 0 (date between '2013-01-01' and '2013-01-31') node 1 (date between ...) node 2 (date between ...) node 3 date between ...)
  23. 23. Table-based primary node 0 (users table(s)) node 1 (transactions table) node 2 (session tables)
  24. 24. Multi-head primary1 node0 node1node2 node3 primary2
  25. 25. Multi-head primary1 node0 node1node2 node3 primary2primary3 primary4
  26. 26. Demo time
  27. 27. Limitations: Network traffic
  28. 28. Limitations: Network traffic primary=# select count(*) from transactions_local; count --------- 1095336 (1 row) Time: 209.097 ms
  29. 29. Limitations: Network traffic primary=# select count(*) from transactions_local; count --------- 1095336 (1 row) Time: 209.097 ms primary=# select count(*) from transactions_primary; count --------- 1095336 (1 row) Time: 2867.385 ms
  30. 30. Limitations: Dumb queries
  31. 31. Limitations: Dumb queries primary=# explain verbose select count(*) from transactions; QUERY PLAN ----------------------------------------------------------------- Aggregate (cost=1767.38..1767.39 rows=1 width=0) Output: count(*) -> Append (cost=100.00..1699.12 rows=27304 width=0) -> Foreign Scan on public.transactions_node0 (cost=100.00..212.39 rows=3413 width=0) Remote SQL: SELECT NULL FROM public.transactions ...
  32. 32. Limitations: Dumb queries primary=# explain verbose select count(*) from transactions; QUERY PLAN ----------------------------------------------------------------- Aggregate (cost=1767.38..1767.39 rows=1 width=0) Output: count(*) -> Append (cost=100.00..1699.12 rows=27304 width=0) -> Foreign Scan on public.transactions_node0 (cost=100.00..212.39 rows=3413 width=0) Remote SQL: SELECT NULL FROM public.transactions ... primary=# explain verbose select avg(int_value) from transactions; QUERY PLAN -------------------------------------------------------------------------- Aggregate (cost=1545.60..1545.61 rows=1 width=8) Output: avg(transactions_node0.int_value) -> Append (cost=100.00..1494.40 rows=20480 width=8) -> Foreign Scan on public.transactions_node0 (cost=100.00..186.80 rows=2560 width=8) Output: transactions_node0.int_value Remote SQL: SELECT int_value FROM public.transactions ...
  33. 33. Limitations: Dumb queries select type, count(*) from users group by type order by 2 desc;
  34. 34. Limitations: Joins
  35. 35. Limitations: Joins select count(*) from transactions t, users u where t.user_id = u.id and u.type = ’mistaken’;
  36. 36. Limitations: Keys
  37. 37. Limitations: Keys ’Nuff said
  38. 38. Limitations: Constraint exclusion Remember this? ERROR: constraints are not supported on foreign tables LINE 6: check ((id % 8) = 0)) server node0 ....
  39. 39. Limitations: Single-threaded executer
  40. 40. Limitations: Single-threaded executer How many nodes do you have?
  41. 41. Limitations: Single-threaded executer How many nodes do you have? Do you know what they’re doing?
  42. 42. Strategies Large working set, small nodes Node-level partitioning Heavy distributed processing Multi-head
  43. 43. Strategy: Large working set, small nodes
  44. 44. Strategy: Large working set, small nodes Your working set is larger than one node’s RAM
  45. 45. Strategy: Large working set, small nodes Your working set is larger than one node’s RAM ... but you have lots of nodes
  46. 46. Strategy: Large working set, small nodes Your working set is larger than one node’s RAM ... but you have lots of nodes (and network is faster than disk)
  47. 47. Strategy: Large working set, small nodes Your working set is larger than one node’s RAM ... but you have lots of nodes (and network is faster than disk) This might be worth looking into if you’re on AWS, but please, please test it first
  48. 48. Strategy: Node-level partitioning Like parititioning, but with a separate node per partition group!
  49. 49. Strategy: Node-level partitioning Like parititioning, but with a separate node per partition group! As a total strategy, this is probably not worthwhile. However, it can work with a fast ”current data” node combining with slower ”archived data” nodes.
  50. 50. Heavy distributed processing
  51. 51. Heavy distributed processing Take advantage of lots of CPUs
  52. 52. Heavy distributed processing Take advantage of lots of CPUs Works well when you have node-discrete workloads
  53. 53. Heavy distributed processing Take advantage of lots of CPUs Works well when you have node-discrete workloads Lock management can become a bit hairier
  54. 54. Heavy distributed processing Take advantage of lots of CPUs Works well when you have node-discrete workloads Lock management can become a bit hairier This might actually be a useful use case
  55. 55. Multi-headed
  56. 56. Multi-headed Like replication, but with no overhead or delay!
  57. 57. Multi-headed Like replication, but with no overhead or delay! Also, no storage overhead!
  58. 58. Multi-headed Like replication, but with no overhead or delay! Also, no storage overhead! Might work well with the distributed processing setup
  59. 59. Multi-headed Like replication, but with no overhead or delay! Also, no storage overhead! Might work well with the distributed processing setup In fact, given the overhead that lands on the head node, it might be necessary for a working FDW federation setup
  60. 60. Pan-Strategy Advice
  61. 61. Pan-Strategy Advice Think very carefully about what tables should live where
  62. 62. Pan-Strategy Advice Think very carefully about what tables should live where Think very carefully about tuning settings (especially on your head node) work mem shared buffers temp buffers
  63. 63. Pan-Strategy Advice Think very carefully about what tables should live where Think very carefully about tuning settings (especially on your head node) work mem shared buffers temp buffers Think very carefully about how many data nodes you want
  64. 64. Pan-Strategy Advice Think very carefully about what tables should live where Think very carefully about tuning settings (especially on your head node) work mem shared buffers temp buffers Think very carefully about how many data nodes you want Think very carefully about network vs. disk vs. dumb-query costs
  65. 65. Pan-Strategy Advice Think very carefully about what tables should live where Think very carefully about tuning settings (especially on your head node) work mem shared buffers temp buffers Think very carefully about how many data nodes you want Think very carefully about network vs. disk vs. dumb-query costs Think very carefully!
  66. 66. Thanks!
  67. 67. Questions? Any questions?
  68. 68. Questions? Any questions? John, do you use this approach for your databases?
  69. 69. Questions? Any questions? John, do you use this approach for your databases? Why not?
  70. 70. Thanks!
  71. 71. Thanks! Plug: Stephen Frost has another postgres fdw talk tomorrow
  72. 72. Thanks! Plug: Stephen Frost has another postgres fdw talk tomorrow Also: Rentrak is hiring: programmers, sysadmins, and devops
  73. 73. Federating Queries Using postgres fdw Introduction Who am I? Partitioning PostgreSQL inheritance partitioning Old-school partitioning Federating Queries Federation Strategies Overview Trial and Error Demo Limitations Strategies Wrap-up

×