PL/Perl New Features    in PostgreSQL 9.0      Tim Bunce - Dec 2010     Creative Commons BY-NC-SA 3.0
PL/Perl Changes                 ‣ USER                 ‣ INTERNAL                 ‣ DBA                 ‣ NYTPROF         ...
PL/Perl Changes   ‣ USER   ‣ INTERNAL   ‣ DBA   ‣ NYTPROF   ‣ FUTURE
New Builtins
quote_... quote_literal( "foo"            ) ==> "foo" quote_literal( "dont "carp"" ) ==> "dont "carp"" quote_literal( ""  ...
quote_...(undef) quote_literal( undef   ) ==> undefquote_nullable( undef   ) ==> "NULL"   quote_ident( undef   ) ==> """" ...
{en,de}code_byteaencode_bytea( "foo"            ) ==> "x666f6f"decode_bytea( "x666f6f"      ) ==> "foo"decode_bytea( "1461...
looks_like_numberlooks_like_number( 1           ) ==> 1looks_like_number( 0           ) ==> 1looks_like_number( "+7.2e-9" ...
encode_array_*encode_array_literal( ["foo","bar"] )   ==> "{"foo", "bar"}"encode_array_constructor( ["foo","bar"] )   ==> ...
encode_array_*encode_array_literal(       "foo"   ) ==> "foo"encode_array_constructor(   "foo"   ) ==> "foo"encode_array_l...
Trusted require/use• require/use work for already loaded modulesuse strict;            # old way: BEGIN { strict->import()...
CONTEXT: ...• PL/Perl tracks the context of log messages  - before:      WARNING: ...some warning from perl code...  - now...
DO ... LANGUAGE ...;• Arbitrary chunks of code can be executed  directly from psql, or client apps, via DO   -   Thanks to...
Other Changes• Using $a and $b in sort blocks now works!• eval { ... } and eval "..." now work!• END blocks are now run at...
PL/Perl Changes   ‣ USER   ‣ INTERNAL   ‣ DBA   ‣ NYTPROF   ‣ FUTURE
INTERNAL• The Safe module is no longer used for plperl  - Now faster, simpler, and more secure• Validates return values ar...
PL/Perl Changes   ‣ USER   ‣ INTERNAL   ‣ DBA   ‣ NYTPROF   ‣ FUTURE
New plperl.* Config• Specify perl code to run during initialization:  plperl.on_init            = ...perl code...  plperl.o...
Interpreter Lifecycle
Birth1. Perl interpreter created2. Options from PERL5OPT env var are processed3. PL/Perl support bootstrap code is execute...
Adolescence6. Interpreter is specialised for plperl (if that’s used first)    •   Modules loaded: strict, warnings, feature...
Siblings• If plplerlu code is run later    - then a new interpreter is created    - similarly if plperlu is run first and p...
Death• Finally, when the session ends:   - Access to the database is disabled   - END blocks, if any, are run (if exiting ...
plperl.on_init• Handy to set global perl configuation  plperl.on_init=use lib qw(/myapp); use ...;  plperl.on_init=require ...
PL/Perl Best Practice• Include explicit use statements in functions  For plperlu thatll actually load if needed  For plper...
plperl.on_plperl_init• Originally intended for things like  -   PGOPTIONS="-c plperl.on_plperl_init=..."  - to enable debu...
PL/Perl Changes   ‣ USER   ‣ INTERNAL   ‣ DBA   ‣ NYTPROF   ‣ FUTURE
Devel::NYTProfPerl Source Code ProfilerPostgreSQL::PLPerl::NYTProf
Enabling NYTProf• Via postgres.conf:  plperl.on_init=use PostgreSQL::PLPerl::NYTProf• Via environment variable:  PERL5OPT=...
Reporting from NYTProf• Writes per-backend data files:  $PGDATA/nytprof.out.$pid• To generate a report:  nytprofhtml --file...
~ Demo ~              Of plperl.on_init in postgresql.conf            And use of PostgreSQL::PLPerl::NYTProf          Scre...
PL/Perl Changes   ‣ USER   ‣ INTERNAL   ‣ DBA   ‣ NYTPROF   ‣ FUTURE
Future Possibilities•   Optimize trigger handling overheads•   Remove overheads of threaded perls•   $array_ref = decode_a...
Questions?Tim.Bunce@pobox.comhttp://blog.timbunce.org @timbunce on twitter
http://xkcd.com/519/
Upcoming SlideShare
Loading in...5
×

PL/Perl - New Features in PostgreSQL 9.0 201012

6,148

Published on

Slides of my talk given at PGDay.EU in Stuttgart Germany in December 2010.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
6,148
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
11
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "PL/Perl - New Features in PostgreSQL 9.0 201012"

  1. 1. PL/Perl New Features in PostgreSQL 9.0 Tim Bunce - Dec 2010 Creative Commons BY-NC-SA 3.0
  2. 2. PL/Perl Changes ‣ USER ‣ INTERNAL ‣ DBA ‣ NYTPROF ‣ FUTURESome features are in recent 8.x.y due to back-porting of security changes
  3. 3. PL/Perl Changes ‣ USER ‣ INTERNAL ‣ DBA ‣ NYTPROF ‣ FUTURE
  4. 4. New Builtins
  5. 5. quote_... quote_literal( "foo" ) ==> "foo" quote_literal( "dont "carp"" ) ==> "dont "carp"" quote_literal( "" ) ==> ""quote_nullable( "foo" ) ==> "foo"quote_nullable( "dont "carp"" ) ==> "dont "carp""quote_nullable( "" ) ==> ""quote_ident( "foo" ) ==> "foo"quote_ident( "dont "carp"" ) ==> ""dont ""carp""""quote_ident( "" ) ==> """"
  6. 6. quote_...(undef) quote_literal( undef ) ==> undefquote_nullable( undef ) ==> "NULL" quote_ident( undef ) ==> """" (warn)
  7. 7. {en,de}code_byteaencode_bytea( "foo" ) ==> "x666f6f"decode_bytea( "x666f6f" ) ==> "foo"decode_bytea( "146157157") ==> "foo"encode_bytea( "x{263a}" ) ==> "xe298ba" UTF8decode_bytea( "xe298ba" ) ==> "342230272" Not UTF8encode_bytea( "" ) ==> "x"decode_bytea( "x" ) ==> ""decode_bytea( "" ) ==> ""encode_bytea( undef ) ==> "x" (warn)decode_bytea( undef ) ==> "" (warn)
  8. 8. looks_like_numberlooks_like_number( 1 ) ==> 1looks_like_number( 0 ) ==> 1looks_like_number( "+7.2e-9" ) ==> 1looks_like_number( "foo" ) ==> 0looks_like_number( "" ) ==> 0looks_like_number( undef ) ==> undeflooks_like_number( " 4 " ) ==> 1looks_like_number( "5plus" ) ==> 0 (but 5plus+0=5)
  9. 9. encode_array_*encode_array_literal( ["foo","bar"] ) ==> "{"foo", "bar"}"encode_array_constructor( ["foo","bar"] ) ==> "ARRAY[foo, bar]"encode_array_literal( [1,[2,[undef]]] ) ==> "{"1", {"2", {NULL}}}"encode_array_constructor( [1,[2,[undef]]] ) ==> "ARRAY[1, ARRAY[2, ARRAY[NULL]]]"
  10. 10. encode_array_*encode_array_literal( "foo" ) ==> "foo"encode_array_constructor( "foo" ) ==> "foo"encode_array_literal( undef ) ==> undefencode_array_constructor( undef ) ==> "NULL"
  11. 11. Trusted require/use• require/use work for already loaded modulesuse strict; # old way: BEGIN { strict->import(); }• extra pre-loaded modulesuse warnings;use Carp;use feature qw(say); # for perl 5.10 or lateruse utf8; # if server_encoding is utf8
  12. 12. CONTEXT: ...• PL/Perl tracks the context of log messages - before: WARNING: ...some warning from perl code... - now: WARNING: ...some warning from perl code... CONTEXT: PL/Perl function "..." - Thanks to Alexey Klyukin.
  13. 13. DO ... LANGUAGE ...;• Arbitrary chunks of code can be executed directly from psql, or client apps, via DO - Thanks to Petr Jelinek, Joshua Tolley, Hannu Valtonen• No need to create and run a stored procedure each time: DO $$ spi_exec("... $_ ...") for a..z; $$ language plperl;
  14. 14. Other Changes• Using $a and $b in sort blocks now works!• eval { ... } and eval "..." now work!• END blocks are now run at end of session - they cant (currently) access the database• Warnings from perl are now WARNINGs - they used to be NOTICE
  15. 15. PL/Perl Changes ‣ USER ‣ INTERNAL ‣ DBA ‣ NYTPROF ‣ FUTURE
  16. 16. INTERNAL• The Safe module is no longer used for plperl - Now faster, simpler, and more secure• Validates return values are in server encoding - ERROR: invalid byte sequence for encoding - Thanks to Andrew Dunstan• Internal code refactoring and cleanup
  17. 17. PL/Perl Changes ‣ USER ‣ INTERNAL ‣ DBA ‣ NYTPROF ‣ FUTURE
  18. 18. New plperl.* Config• Specify perl code to run during initialization: plperl.on_init = ...perl code... plperl.on_plperlu_init = ...perl code... plperl.on_plperl_init = ...perl code...• Can only be set by superuser or postgres.conf• Code cant access the database
  19. 19. Interpreter Lifecycle
  20. 20. Birth1. Perl interpreter created2. Options from PERL5OPT env var are processed3. PL/Perl support bootstrap code is executed4. plperl.on_init code runs (unrestricted)Above steps may happen in postmaster process at startup,if plperl is loaded via shared_preload_libraries.Otherwise they happen at first use.No access to database.
  21. 21. Adolescence6. Interpreter is specialised for plperl (if that’s used first) • Modules loaded: strict, warnings, features, Carp • Unsafe perl ops are restricted (require, open etc.) • DynaLoader package is deleted • plperl.on_plperl_init code runs (restricted)7. Database access is enabled8. Perl interpreter is made available for use9. Executes whatever action called it into existence
  22. 22. Siblings• If plplerlu code is run later - then a new interpreter is created - similarly if plperlu is run first and plperl run later• If plplerl with a different security context is run - then a new interpreter is created for the ROLE - That’s a recent security fix: http://wiki.postgresql.org/wiki/20101005securityreleaseNote impact on shared_preload_libraries in these cases
  23. 23. Death• Finally, when the session ends: - Access to the database is disabled - END blocks, if any, are run (if exiting cleanly)
  24. 24. plperl.on_init• Handy to set global perl configuation plperl.on_init=use lib qw(/myapp); use ...; plperl.on_init=require "plperloninit.pl"; Effectively defines ‘approved’ modules for plperl• SECURITY RISK! Only load modules youre happy for plperl code to use. Also check any other modules loaded as dependencies! Use Devel::TraceLoad to see whats actually loaded: PERL5OPT=-MDevel::Trace=summary pg_ctl ...
  25. 25. PL/Perl Best Practice• Include explicit use statements in functions For plperlu thatll actually load if needed For plperl itll check that module is loaded - so youll get an immediate clear failure if not - (e.g., on a replica with old postgres.conf file)
  26. 26. plperl.on_plperl_init• Originally intended for things like - PGOPTIONS="-c plperl.on_plperl_init=..." - to enable debug or profiling for a session• But...• Can only be set by superuser or postgres.conf - due to SECURITY DEFINER risk at the time - that’s now been patched (20101005, CVE-2010-3433) - so this restriction may be removed in future
  27. 27. PL/Perl Changes ‣ USER ‣ INTERNAL ‣ DBA ‣ NYTPROF ‣ FUTURE
  28. 28. Devel::NYTProfPerl Source Code ProfilerPostgreSQL::PLPerl::NYTProf
  29. 29. Enabling NYTProf• Via postgres.conf: plperl.on_init=use PostgreSQL::PLPerl::NYTProf• Via environment variable: PERL5OPT=-MPostgreSQL::PLPerl::NYTProf pg_ctl ...• Is immediately active for all connections.• To enable on demand for one connection: NYTPROF=start=no PERL5OPT=... pg_ctl ... DO DB::enable_profile LANGUAGE plperl;
  30. 30. Reporting from NYTProf• Writes per-backend data files: $PGDATA/nytprof.out.$pid• To generate a report: nytprofhtml --file=$PGDATA/nytprof.out.4321 --open
  31. 31. ~ Demo ~ Of plperl.on_init in postgresql.conf And use of PostgreSQL::PLPerl::NYTProf Screencast: http://timbunce.blip.tv/file/3691795/Video: http://www.fosslc.org/drupal/content/plperl-new-features-90
  32. 32. PL/Perl Changes ‣ USER ‣ INTERNAL ‣ DBA ‣ NYTPROF ‣ FUTURE
  33. 33. Future Possibilities• Optimize trigger handling overheads• Remove overheads of threaded perls• $array_ref = decode_array_literal(‘{...}’)• $hash_ref = decode_hstore_literal(‘x=>42’)• Array params as array refs (with overloading)• Rewrite encode_array_literal in C.
  34. 34. Questions?Tim.Bunce@pobox.comhttp://blog.timbunce.org @timbunce on twitter
  35. 35. http://xkcd.com/519/
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×