Josh Berkus
  LCA 2013
Write                Cascading
      Read          Performance              Replication
    Scalability
                               Index-only
                                 Scans
    Start


 Other
Features
                                                     JSON
                     DDL
                  Migrations
                               Range Types
                                               SP-GiST
 Admin Tools


    New Feature Grand Prix
START YOUR ENGINES!
Read Scalability
●   fast path sorting
●   reduced overhead frequent table
    locks
●   prproc shared memory
    optimization
●   laxy vxid locks
●   heap hot search buffer
Read Scalability



          350,000 q.p.s.
Write Performance
●   better group commit
●   reduce WalInsertLock contention
●   batch tuple copy
●   reduced checkpoint activity
Write Scalability




          +50% more
          writes/second
Parallel Bulk Load



        10-15 parallel streams
        up to 3X load speed
Index-Only Scans
            Col
3ms    A    Cos
                      table books
                   Title
       C
       D
            Dea
            Dem            300ms
                   Will There Be Faith?
                   State of Wonder
                   Sphere 18c Flr
 A     H    Hea    Dead Eyes
 H     I    Het    The Greatest Music
 N                 Stories Never Told
            Rea
 S     N           Skeleton Canyon
            Res
 U     R           The Long Snapper
            Ska    Super Immunity
       S    Sut    The Barbary Pirates
       T           Heat
            Tab
                   Confessions of a
            The
       U           Prairie Bitch
       W    Uvh    Talking Pictures
            Woz    Reggie Jackson
“count(*) is slow in
      postgres”
Index-Only Scans
            Col
3ms    A    Cos
                               30ms
       C    Dea
       D    Dem

 A     H    Hea
 H     I    Het
 N          Rea
 S     N
            Res
 U     R
            Ska
       S    Sut
       T
            Tab
            The
       U
       W    Uvh   Visibility
            Woz     Map
Cascading Replication


            walreciever
walsender




            walreciever




            walreciever
Cascading Replication
Sydney           North Carolina


                           walsender
     walsender                         walreciever




                   walreciever




                                       walreciever
Replication Improvements
●   pg_receivexlog
    an xlog stream reader
●   standby-only backup
    no more snapshotting the master
●   recv vs. write modes
    for synchronous replication
JSON
●   JSON data type
●   array-to-json, row-to-json
    functions
●   get query results as JSON!
PL/v8
PL/v8
●   Based on Google's v8 JS engine
●   write fast stored procedures in
    JavaScript
●   create ad-hoc attribute indexes
    on JSON data
A data type and PL made for each
other!
create or replace function
fib(n int) returns int as $$
  function fib(n) {
    return n<2 ? n : fib(n-1)
        + fib(n-2)
  }
  return fib(n)
$$ LANGUAGE plv8 IMMUTABLE
STRICT;
pl/coffee
PL/coffee
=# CREATE OR REPLACE FUNCTION
public.fibonacci(n integer)
 RETURNS integer LANGUAGE plcoffee
IMMUTABLE STRICT
AS $function$
fibonacci = (x)->
  return 0 if x == 0
  return 1 if x == 1
  return fibonacci(x-1) + fibonacci(x-2)
return fibonacci n
$function$;
CREATE FUNCTION
Indexing
7.1: Generalized Search Tree
      (GiST)
8.1: Generalized Inverted Index
      (GIN)
9.1: K-Nearest Neighbor
      (KNN)
9.2: Space-GiST
SP-GiST
●   Based on “Space-Partitioning
    Trees”
●   Faster to read,
    faster to update
    than GiST
    ●   Will cause
        PostGIS 2.1!
SP-GiST
knn=# create index pt_gist_idx on geo
using gist(point);
CREATE INDEX
Time: 36672.283 ms
Size: 153,124,864
knn=# create index pt_spgist_idx on geo
using spgist(point);
CREATE INDEX
Time: 12805.530 ms ~ 3 times faster !
Size: 153,788,416 ~ the same size
Range Types
●   Temporal range:
    [ 2012-04-10, 2012-04-12 )
●   Alpha Index:
    ( Abbe, Babel ]
●   Linear distance:
    ( 375.453, 374.441 )
DDL pit stop
runtime DDL made easy
●   DROP INDEX CONCURRENTLY
●   NOT VALID CHECK constraints
●   less rewriting for ALTER TABLE
●   ALTER IF EXISTS
●   RENAME domains, FDWs
●   vacuum skips locked pages
Instrumentation
better monitoring
●   new pg_stat_statements
●   more autovacuum logging
●   track_io_timing
●   deadlock, temp file counters
●   checkpoint timing in
    pg_stat_bgwriter
better EXPLAIN
●   track buffers dirtied/written
●   report filtered-out rows
●   no-TIMING option
Other Features
●   lower CPU wakeups (power
    saving)
●   more sepgsql security
●   optimizer improvements
●   pg_hba.conf fixes
●   XML improvements
●   exportable snapshots
Other Features
●   more pg_dump/pg_restore
    options
●   more psql features
●   more pg_upgrade fixes,
    improvements
●   dropped support for antiques
    ●   dgux, nextstep, sunos4, svr4, ultrix4,
        univel, bsdi
Finish Line!
The Next Race: 9.3
9.3 Features (so far)
●   More JSON goodness!
    ●   accessor methods
    ●   constructor methods
●   Writable Foreign Data Wrappers
●   Event Triggers
    ●   write a trigger on anything
●   Updatable VIEWs
9.3 Features (so far)
●   create your own background
    workers
●   LATERAL JOINs
●   Streaming-only Remastering
●   Improved FK locking
●   pg_stat_plans
SHMMAX
SHMALL
More Information
●   Josh Berkus: josh@pgexperts.com
    ●   www.databasesoup.com
●   Upcoming Events
    ●   Melbourne pgDay, Monday Feb 5
        –   2013.pgday.org.au
    ●   PyPgDay at PyCon 2013
    ●   pgCon, Ottawa, May 21-25
        Text and graphics of this talk are licensed Creative Commons Share-Alike, except where otherwise
        noted. Most racing photos in the presentation are stock photos which may only be reproduced and
        reused as a part of this presenstation. Graphs courtesy Robert Haas and Heikki Linnakangas.
        SP-Gist examples from Oleg Bartunov, PL/v8 and PL/Coffee examples courtesy of Will Leinweber.

92 grand prix_2013

  • 1.
    Josh Berkus LCA 2013
  • 3.
    Write Cascading Read Performance Replication Scalability Index-only Scans Start Other Features JSON DDL Migrations Range Types SP-GiST Admin Tools New Feature Grand Prix
  • 4.
  • 5.
    Read Scalability ● fast path sorting ● reduced overhead frequent table locks ● prproc shared memory optimization ● laxy vxid locks ● heap hot search buffer
  • 6.
    Read Scalability 350,000 q.p.s.
  • 7.
    Write Performance ● better group commit ● reduce WalInsertLock contention ● batch tuple copy ● reduced checkpoint activity
  • 8.
    Write Scalability +50% more writes/second
  • 9.
    Parallel Bulk Load 10-15 parallel streams up to 3X load speed
  • 10.
    Index-Only Scans Col 3ms A Cos table books Title C D Dea Dem 300ms Will There Be Faith? State of Wonder Sphere 18c Flr A H Hea Dead Eyes H I Het The Greatest Music N Stories Never Told Rea S N Skeleton Canyon Res U R The Long Snapper Ska Super Immunity S Sut The Barbary Pirates T Heat Tab Confessions of a The U Prairie Bitch W Uvh Talking Pictures Woz Reggie Jackson
  • 11.
    “count(*) is slowin postgres”
  • 12.
    Index-Only Scans Col 3ms A Cos 30ms C Dea D Dem A H Hea H I Het N Rea S N Res U R Ska S Sut T Tab The U W Uvh Visibility Woz Map
  • 13.
    Cascading Replication walreciever walsender walreciever walreciever
  • 14.
    Cascading Replication Sydney North Carolina walsender walsender walreciever walreciever walreciever
  • 15.
    Replication Improvements ● pg_receivexlog an xlog stream reader ● standby-only backup no more snapshotting the master ● recv vs. write modes for synchronous replication
  • 16.
    JSON ● JSON data type ● array-to-json, row-to-json functions ● get query results as JSON!
  • 17.
  • 18.
    PL/v8 ● Based on Google's v8 JS engine ● write fast stored procedures in JavaScript ● create ad-hoc attribute indexes on JSON data A data type and PL made for each other!
  • 19.
    create or replacefunction fib(n int) returns int as $$ function fib(n) { return n<2 ? n : fib(n-1) + fib(n-2) } return fib(n) $$ LANGUAGE plv8 IMMUTABLE STRICT;
  • 20.
  • 21.
    PL/coffee =# CREATE ORREPLACE FUNCTION public.fibonacci(n integer) RETURNS integer LANGUAGE plcoffee IMMUTABLE STRICT AS $function$ fibonacci = (x)-> return 0 if x == 0 return 1 if x == 1 return fibonacci(x-1) + fibonacci(x-2) return fibonacci n $function$; CREATE FUNCTION
  • 22.
    Indexing 7.1: Generalized SearchTree (GiST) 8.1: Generalized Inverted Index (GIN) 9.1: K-Nearest Neighbor (KNN)
  • 23.
  • 24.
    SP-GiST ● Based on “Space-Partitioning Trees” ● Faster to read, faster to update than GiST ● Will cause PostGIS 2.1!
  • 25.
    SP-GiST knn=# create indexpt_gist_idx on geo using gist(point); CREATE INDEX Time: 36672.283 ms Size: 153,124,864 knn=# create index pt_spgist_idx on geo using spgist(point); CREATE INDEX Time: 12805.530 ms ~ 3 times faster ! Size: 153,788,416 ~ the same size
  • 26.
    Range Types ● Temporal range: [ 2012-04-10, 2012-04-12 ) ● Alpha Index: ( Abbe, Babel ] ● Linear distance: ( 375.453, 374.441 )
  • 27.
  • 28.
    runtime DDL madeeasy ● DROP INDEX CONCURRENTLY ● NOT VALID CHECK constraints ● less rewriting for ALTER TABLE ● ALTER IF EXISTS ● RENAME domains, FDWs ● vacuum skips locked pages
  • 29.
  • 30.
    better monitoring ● new pg_stat_statements ● more autovacuum logging ● track_io_timing ● deadlock, temp file counters ● checkpoint timing in pg_stat_bgwriter
  • 31.
    better EXPLAIN ● track buffers dirtied/written ● report filtered-out rows ● no-TIMING option
  • 32.
    Other Features ● lower CPU wakeups (power saving) ● more sepgsql security ● optimizer improvements ● pg_hba.conf fixes ● XML improvements ● exportable snapshots
  • 33.
    Other Features ● more pg_dump/pg_restore options ● more psql features ● more pg_upgrade fixes, improvements ● dropped support for antiques ● dgux, nextstep, sunos4, svr4, ultrix4, univel, bsdi
  • 34.
  • 35.
  • 36.
    9.3 Features (sofar) ● More JSON goodness! ● accessor methods ● constructor methods ● Writable Foreign Data Wrappers ● Event Triggers ● write a trigger on anything ● Updatable VIEWs
  • 37.
    9.3 Features (sofar) ● create your own background workers ● LATERAL JOINs ● Streaming-only Remastering ● Improved FK locking ● pg_stat_plans
  • 38.
  • 39.
    More Information ● Josh Berkus: josh@pgexperts.com ● www.databasesoup.com ● Upcoming Events ● Melbourne pgDay, Monday Feb 5 – 2013.pgday.org.au ● PyPgDay at PyCon 2013 ● pgCon, Ottawa, May 21-25 Text and graphics of this talk are licensed Creative Commons Share-Alike, except where otherwise noted. Most racing photos in the presentation are stock photos which may only be reproduced and reused as a part of this presenstation. Graphs courtesy Robert Haas and Heikki Linnakangas. SP-Gist examples from Oleg Bartunov, PL/v8 and PL/Coffee examples courtesy of Will Leinweber.