0
My	
  experience	
  with	
  embedding	
  
PostgreSQL	
  
Postgres	
  Conf	
  EU,	
  Dublin,	
  Ireland	
  
October	
  2013...
About	
  me	
  
•  Sr	
  Manager,	
  VMware	
  	
  	
  Data	
  
–  Leads	
  Postgres	
  team	
  at	
  VMware	
  
–  Blog	
...
Agenda	
  
•  Business	
  reasons	
  for	
  embedding	
  database	
  
•  Technical	
  architecture	
  journey	
  

© 2013 ...
The	
  IniXal	
  Problem	
  
•  >30	
  	
  #	
  products	
  and	
  product	
  components	
  
•  >20	
  	
  	
  requires	
 ...
SQL	
  Server	
  Express	
  
•  Currently	
  shipped	
  for	
  some	
  Windows	
  based	
  
products	
  
–  However	
  lim...
Proposal:	
  PostgreSQL?	
  
•  PostgreSQL	
  is	
  free	
  and	
  open	
  sourced	
  
•  Long	
  history	
  of	
  stabili...
IniXal	
  External	
  Feedback	
  
•  Actually	
  got	
  pushback	
  from	
  customers,	
  partners	
  
–  PostgreSQL	
  n...
IniXal	
  Internal	
  Feedback	
  	
  
•  DB	
  Support	
  Matrix	
  complexity	
  increases	
  
–  Every	
  new	
  DB	
  ...
Plan:	
  Embedded	
  Default	
  DB	
  
•  Embed	
  PostgreSQL	
  as	
  part	
  of	
  the	
  product	
  
•  Make	
  it	
  t...
Ground	
  Work:	
  MigraXon	
  
•  Products	
  used	
  various	
  databases	
  
–  SQL	
  Server	
  
–  Oracle	
  	
  
–  ...
MigraXon	
  
•  Leveraged	
  contractors	
  to	
  migrate	
  stored	
  
procedures	
  
•  Performance	
  bake-­‐off	
  test...
Example:	
  vCenter	
  Server	
  Appliance	
  

© 2013 VMware, Inc. All rights reserved
Plan:	
  Backup	
  &	
  Recovery	
  
•  DB	
  not	
  the	
  only	
  thing	
  to	
  be	
  backed	
  up	
  
•  ConfiguraXon	
...
IniXal	
  Embedded	
  DB	
  Deployments	
  
•  Each	
  Product	
  team	
  is	
  different	
  
–  	
  has	
  different	
  lev...
Early	
  Results	
  
•  YES	
  !	
  Customer	
  started	
  seeing	
  benefit	
  
•  However	
  
–  Experience	
  of	
  Post...
Some	
  encounters	
  
•  Portable	
  binaries	
  –	
  Shrink	
  wrapped	
  sokware	
  
(not	
  compiled	
  on	
  same	
  ...
MulX-­‐Plaborm	
  Requirement	
  
•  As	
  company	
  grew,	
  acquisiXons	
  grew	
  
•  Some	
  products	
  only	
  work...
SoluXon:	
  Embedded	
  DBaaS	
  
•  Provide	
  End	
  to	
  End	
  Experience	
  with	
  
PostgreSQL	
  

© 2013 VMware, ...
InstallaXon	
  
•  Extend	
  installaXon	
  to	
  setup	
  a	
  service	
  
•  Configurable	
  to	
  set	
  up	
  as	
  int...
InstallaXon	
  
•  RPM	
  	
  installaXon	
  
–  Dependent	
  OSS	
  Libraries	
  to	
  make	
  RPM	
  portable	
  
–  Ser...
Installer	
  on	
  Windows	
  
• 
• 
• 
• 

PostgreSQL	
  Installer	
  on	
  Windows	
  
Service	
  setup	
  on	
  Windows...
Backup	
  &	
  Recovery	
  
•  While	
  other	
  backups	
  strategy	
  are	
  available	
  
–  Nothing	
  beats	
  standa...
Upgrades	
  
•  Seamless	
  upgrade	
  to	
  new	
  version	
  
–  RPM	
  Upgrades	
  needs	
  to	
  be	
  transparent	
  ...
Upgrades	
  
•  Problems	
  sXll	
  not	
  completely	
  resolved	
  
–  Reduce	
  downXme	
  for	
  major	
  release	
  u...
Performance	
  
•  Default	
  PostgreSQL	
  semng	
  are	
  conservaXve	
  
•  Most	
  deployments	
  use	
  default	
  va...
Performance	
  related	
  tools	
  
•  Provided	
  tools	
  to	
  understand	
  what’s	
  
happening	
  with	
  Postgres	
...
Example:	
  pg_top	
  

© 2013 VMware, Inc. All rights reserved
Performance	
  related	
  future	
  work	
  
•  ExisXng	
  tools	
  sXll	
  meant	
  for	
  expert	
  users	
  
•  Lot	
  ...
Availability	
  
•  Created	
  a	
  watchdog	
  for	
  PostgreSQL	
  to	
  restart	
  
PostgreSQL	
  if	
  database	
  can...
Related	
  Problems:	
  Crashes	
  
•  PostgreSQL	
  started	
  throwing	
  errors	
  with	
  “Out	
  
of	
  Memory”	
  er...
Supportability	
  
•  When	
  customers	
  hit	
  a	
  problem	
  
–  Hard	
  to	
  get	
  access	
  to	
  the	
  system	
...
Usability	
  
•  Experts	
  love	
  command	
  line	
  uXliXes	
  
•  Novices	
  love	
  GUI,	
  WebUI	
  
•  It	
  needs	...
Pluggable	
  UI	
  
•  Matching	
  VMware’s	
  GUI	
  Plaborm	
  

© 2013 VMware, Inc. All rights reserved
Embedded	
  DBaaS	
  Architecture	
  
VMware	
  
Product	
  

	
  

Service	
  	
  

	
  

Configurator	
  

	
  

WebUI	
 ...
Virtual	
  Appliances	
  

© 2013 VMware, Inc. All rights reserved
Virtual	
  Appliances	
  

© 2013 VMware, Inc. All rights reserved
ReplicaXon	
  between	
  vApps	
  
•  Provide	
  simple	
  uXliXes	
  to	
  quickly	
  
–  archive_command	
  
–  create_r...
In	
  Closing	
  
•  Free	
  Sokware	
  is	
  nice	
  
•  Experience	
  –	
  What	
  business	
  users	
  care	
  
–  User...
Thanks.	
  
QuesXons?	
  
Download	
  VMware	
  vFabric	
  Postgres	
  	
  
hKp://vmware.com/go/download-­‐vfabric-­‐postg...
Upcoming SlideShare
Loading in...5
×

My experience with embedding PostgreSQL

6,222

Published on

At my current company, we embed PostgreSQL based technologies in various applications shipped as shrink-wrapped software. In this session we talk about the experience of embedding PostgreSQL where it is not directly exposed to end-user and the issues encountered on how they were resolved.

We will talk about business reasons,technical architecture of deployments, upgrades, security processes on how to work with embedded PostgreSQL databases.

Published in: Technology

Transcript of " My experience with embedding PostgreSQL"

  1. 1. My  experience  with  embedding   PostgreSQL   Postgres  Conf  EU,  Dublin,  Ireland   October  2013     Jignesh  Shah,     Sr  Manager,  VMware   © 2013 VMware, Inc. All rights reserved
  2. 2. About  me   •  Sr  Manager,  VMware      Data   –  Leads  Postgres  team  at  VMware   –  Blog  at  hKp://jkshah.blogspot.com     •  Previously  Staff  Engineer,  Sun  Microsystems   –  Team  Member  of  the  first  published  SpecJAppServer   2004  benchmark  with  PostgreSQL   –  Performance  of  PostgreSQL  on  Solaris/Sun  systems   •  Working  with  PostgreSQL  circa  2005   © 2013 VMware, Inc. All rights reserved
  3. 3. Agenda   •  Business  reasons  for  embedding  database   •  Technical  architecture  journey   © 2013 VMware, Inc. All rights reserved
  4. 4. The  IniXal  Problem   •  >30    #  products  and  product  components   •  >20      requires    DB   •  Customers  have  to  get  their  own  DB  license     •  Increases  the  Total  Cost  of  Deployment   –  As  database  load  increases,  CPU  cores  required   increases,  so  does  the  total  license  cost  of  DB   © 2013 VMware, Inc. All rights reserved
  5. 5. SQL  Server  Express   •  Currently  shipped  for  some  Windows  based   products   –  However  limits  not  enough     –  Max  10GB  database  usage   •  Standard  and  Enterprise  EdiXons  get   expensive  preKy  quickly   •  Not  Available  on  Linux     © 2013 VMware, Inc. All rights reserved
  6. 6. Proposal:  PostgreSQL?   •  PostgreSQL  is  free  and  open  sourced   •  Long  history  of  stability   •  Friendly  BSD  License   •  Plus  I  am  at  a  Postgres  Conference  J   © 2013 VMware, Inc. All rights reserved
  7. 7. IniXal  External  Feedback   •  Actually  got  pushback  from  customers,  partners   –  PostgreSQL  not  part  of  their  database  standards   –  No  in-­‐house  PostgreSQL  DBA  skills   –  Lack  of  integrated  tools  ,  monitor,  ease-­‐of-­‐use   –  Large  enterprises  have  to  convince  their  managed   hosXng  partners  to  support  the  database  also   •  AKendees  here  are  going  to  be  educated  about  it     –  but  we  are  worried  about  the  rest  of  the  world   © 2013 VMware, Inc. All rights reserved
  8. 8. IniXal  Internal  Feedback     •  DB  Support  Matrix  complexity  increases   –  Every  new  DB  support  increases  N  plaborm  test   –  Large  scale  tests  are  serialized  due  to  resources   © 2013 VMware, Inc. All rights reserved
  9. 9. Plan:  Embedded  Default  DB   •  Embed  PostgreSQL  as  part  of  the  product   •  Make  it  the  default  choice  opXon  for  most  deployment   •  Do  not  expose  database  directly  to  endusers   •  ApplicaXon  will  manage  the  DB   •  Support  it  for  producXon  deployment   •  Customer  can  sXll  select  to  use  an  external  database  of   their  choice  or  save  DB  License  cost  with  the   embedded  one     © 2013 VMware, Inc. All rights reserved  
  10. 10. Ground  Work:  MigraXon   •  Products  used  various  databases   –  SQL  Server   –  Oracle     –  MySQL   •  Few  had  about  1500  stored  procedures   •  Some  used  exoXc  features   •  For  example    MySQL  used  InMemory  Engine,  InnodB   engine,  MyISAM  engine  in  the  same  workload   © 2013 VMware, Inc. All rights reserved
  11. 11. MigraXon   •  Leveraged  contractors  to  migrate  stored   procedures   •  Performance  bake-­‐off  tests  to  convince   product  teams   •  Integrated  into  the  build  infrastructure  to   ease  pulling  components   © 2013 VMware, Inc. All rights reserved
  12. 12. Example:  vCenter  Server  Appliance   © 2013 VMware, Inc. All rights reserved
  13. 13. Plan:  Backup  &  Recovery   •  DB  not  the  only  thing  to  be  backed  up   •  ConfiguraXon  files   •  ApplicaXon  Status   •  All  data  to  be  backed  up  went  into  a  virtual  disk  backed   up  with  vADP   •  pg_dump  /  pg_restore  used  in  few  cases     © 2013 VMware, Inc. All rights reserved
  14. 14. IniXal  Embedded  DB  Deployments   •  Each  Product  team  is  different   –   has  different  levels  of  DB  Development  skills   •  Each  architect  comes  up  with  their  own  design     –  Postgres  Service  designed  (Start/Stop)   –  own  DB  Layout  design   –  own  DB  Security  design   –  Some  Tuned  their  DB   –  Many  did  not   © 2013 VMware, Inc. All rights reserved
  15. 15. Early  Results   •  YES  !  Customer  started  seeing  benefit   •  However   –  Experience  of  PostgreSQL  different  on  each  product   –  Random  security  holes  idenXfied  due  to  lack  of   knowledge  of  PostgreSQL  nuances   –  Performance  configuraXon  was  an  issue   –  Upgrade  to  new  versions  of  PostgreSQL  was  an   akerthought  (What  no  in-­‐place  upgrade?)   –  Legal  work  required  for  OSS  increased   –  Resulted  in  PostgreSQL  version  sXckiness   © 2013 VMware, Inc. All rights reserved
  16. 16. Some  encounters   •  Portable  binaries  –  Shrink  wrapped  sokware   (not  compiled  on  same  systems)   –  OSS  Libraries  used  for  compiling  PostgreSQL  have   to  be  shipped     –  Manage  security  patches  for  OSS  Libraries   •  Need  Standard  drivers  like  ODBC,  JDBC   © 2013 VMware, Inc. All rights reserved
  17. 17. MulX-­‐Plaborm  Requirement   •  As  company  grew,  acquisiXons  grew   •  Some  products  only  worked  on  Windows   •  Few  used  exclusively  .NET  framework   © 2013 VMware, Inc. All rights reserved
  18. 18. SoluXon:  Embedded  DBaaS   •  Provide  End  to  End  Experience  with   PostgreSQL   © 2013 VMware, Inc. All rights reserved
  19. 19. InstallaXon   •  Extend  installaXon  to  setup  a  service   •  Configurable  to  set  up  as  internal  service  or   external  service  (mostly  embedded)   •  Security  handled  as  part  of  service  setup  to   follow  best  pracXcesfor  embedded   •  Security  based  on  access  and  minimum  roles   defined   © 2013 VMware, Inc. All rights reserved
  20. 20. InstallaXon   •  RPM    installaXon   –  Dependent  OSS  Libraries  to  make  RPM  portable   –  Server  and  client  RPMs   –  Service  Setup  configurator  to  setup  a  default  service   •  •  •  •  User  setup   initdb  setup   pg_hba  setup     Password  handling   –  Provide  tools  to  quickly  setup  DB  Owner,  DB  for   various  users  of  the  service   © 2013 VMware, Inc. All rights reserved
  21. 21. Installer  on  Windows   •  •  •  •  PostgreSQL  Installer  on  Windows   Service  setup  on  Windows   ODBC  Support     Npgsql  Support  for  .NET  Plaborms   © 2013 VMware, Inc. All rights reserved
  22. 22. Backup  &  Recovery   •  While  other  backups  strategy  are  available   –  Nothing  beats  standard  database  backups   –  Design  cron  jobs  for  criXcal  data     –  Design  Archive  On  Demand  for  semng  up   replicaXon  quickly   –  Most  customer  do  not  want  to  hack  with  backup/ restore  scripts.     –  Should  work  out  of  the  box   © 2013 VMware, Inc. All rights reserved
  23. 23. Upgrades   •  Seamless  upgrade  to  new  version   –  RPM  Upgrades  needs  to  be  transparent  for  minor   versions   –  RPM  upgrades  for  major  version  needs  to  migrate   data  from  the  older  version  in  the  default  service   © 2013 VMware, Inc. All rights reserved
  24. 24. Upgrades   •  Problems  sXll  not  completely  resolved   –  Reduce  downXme  for  major  release  upgrades   –  Generally  migraXon  since  in-­‐place  upgrade  does   not  provide  ways  to  roll  back  the  upgrade   © 2013 VMware, Inc. All rights reserved
  25. 25. Performance   •  Default  PostgreSQL  semng  are  conservaXve   •  Most  deployments  use  default  values   •  Started  doing  auto-­‐tuning   –  Modified  PostgreSQL  core  engine  with  intelligence   –  For  checkpoint  segments,    shared_memory   •  Added  tool  to  auto  tune  postgresql.conf     –  Based  on  memory  share  for  DB  Service   –  Product  workload  dependent     © 2013 VMware, Inc. All rights reserved
  26. 26. Performance  related  tools   •  Provided  tools  to  understand  what’s   happening  with  Postgres   –  pg_top  with  changes   •  Changes  already  contributed  back  to  community   © 2013 VMware, Inc. All rights reserved
  27. 27. Example:  pg_top   © 2013 VMware, Inc. All rights reserved
  28. 28. Performance  related  future  work   •  ExisXng  tools  sXll  meant  for  expert  users   •  Lot  of  demand  for  simple  performance  reports   •  Looking  into  pg_statspack   © 2013 VMware, Inc. All rights reserved
  29. 29. Availability   •  Created  a  watchdog  for  PostgreSQL  to  restart   PostgreSQL  if  database  cannot  be  accessed  for   long   •  If  restart  fails  then  it  can  throw  alerts   •  Also  created  plugins  for  monitoring  the   service   © 2013 VMware, Inc. All rights reserved
  30. 30. Related  Problems:  Crashes   •  PostgreSQL  started  throwing  errors  with  “Out   of  Memory”  errors  in  stress  tests     •  MulXple  applicaXons  can  fight  for  memory   •  Made  Performance  semngs  less  aggressively   © 2013 VMware, Inc. All rights reserved
  31. 31. Supportability   •  When  customers  hit  a  problem   –  Hard  to  get  access  to  the  system   •  Created  support  package  tool  to  collect   relevant  informaXon   –  Collect  config,  status  in  a  tar  file  for  analysis   © 2013 VMware, Inc. All rights reserved
  32. 32. Usability   •  Experts  love  command  line  uXliXes   •  Novices  love  GUI,  WebUI   •  It  needs  to  match  the  other  GUIs  VMware  ship   © 2013 VMware, Inc. All rights reserved
  33. 33. Pluggable  UI   •  Matching  VMware’s  GUI  Plaborm   © 2013 VMware, Inc. All rights reserved
  34. 34. Embedded  DBaaS  Architecture   VMware   Product     Service       Configurator     WebUI       Pg_top         JDBC     PostgreSQL     ODBC       Linux  Based  VM   Watchdog   Perf-­‐Tuner   Monitoring   Support   package  tool   vSphere  Plaborm   © 2013 VMware, Inc. All rights reserved
  35. 35. Virtual  Appliances   © 2013 VMware, Inc. All rights reserved
  36. 36. Virtual  Appliances   © 2013 VMware, Inc. All rights reserved
  37. 37. ReplicaXon  between  vApps   •  Provide  simple  uXliXes  to  quickly   –  archive_command   –  create_replicaXon_user   –  run_as_replica   –  show_replica_status   –  promote_replica_to_primary   © 2013 VMware, Inc. All rights reserved
  38. 38. In  Closing   •  Free  Sokware  is  nice   •  Experience  –  What  business  users  care   –  User  Experience   –  DBA  Experience   –  Support  Experience   –  etc   •  Improve  the  Experience  with  the  technology   © 2013 VMware, Inc. All rights reserved
  39. 39. Thanks.   QuesXons?   Download  VMware  vFabric  Postgres     hKp://vmware.com/go/download-­‐vfabric-­‐postgres   Follow us on twitter: @vPostgres Data Platform Blog: http://blogs.vmware.com/dataplatforms © 2013 VMware, Inc. All rights reserved
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×