Understanding PostgreSQL LWLocks                  Jignesh Shah           Staff Engineer, VMware Inc              PgCon 201...
About MySelf§  Joined	  VMware	  in	  	  2010	  	       q  	  PostgreSQL	  performance	  on	  vSphere	  §  Previously	 ...
Contentv What	  are	  PostgreSQL	  LWLocks	  ?	  v Understanding	  modes	  of	  LWLock	  v LWLocks	  API	  v Architect...
What are PostgreSQL LWLocks?§  LWLocks	  –	  Light	  Weight	  Locks	  of	  PostgreSQL	  §  Primary	  Intent	       q  M...
Understanding Modes of LWLocksExclusiveLock	  § 	   For	  a	  parcular	  lockid	  	  there	  can	  be	  only	  1	  exclus...
LWLocks API§  LWLockAssign(void)	         §  Get	  a	  dynamic	  LWLock	  number	  (mostly	  at	  module	  load)	  §  L...
Architecture Flow of LWLocksLWLockCondionalAcquire	  	  •  Easy	  lock	  it	  if	  available	  else	  return	  false	  (No...
Wait Queue List•  Wait	  Queue	  is	  protected	  by	  a	  mutex	  which	  is	  accessed	  using	  	   SpinLocks	  •  When...
Lock Acquisition Internals	  	     Spins	  on	  	  ‘lock’	  mutex	                     Sleep	  on	  	  its	  process	     ...
Lock Release Internals	  	   Spins	  on	  	  ‘lock’	  mutex	  	  	   Release	  Lock	  	  	                                ...
LWLocks Wait Queue List FIFO Example•  Consider	  something	  80-­‐20	  Shared	  /Exclusive	  rao	  	   	  	   	   S S S S...
Observations about LWLocks Wait Queue List•  All	  Operaons	  on	  Wait	  Queue	  List	  are	  serialized	  	  	        • ...
LWLocks – Defined in lwlock.htypedef enum LWLockId{    BufFreelistLock,          ShmemIndexLock,       OidGenLock,    	   ...
MonitoringLWLOCK_STAT	  •  Special	  build	  with	  LWLOCK_STAT	  defined	  	  LOCK_DEBUG	  	  •  Puts	  debug	  messages	 ...
Dynamic Tracing for LWLocks•      postgresql-­‐lwlock-­‐wait-­‐start(lockid,mode)	  • 	     Postgresql-­‐lwlock-­‐wait-­‐d...
Example Monitoring using Systemtap(DBT2)       LOCKNAME   LWID   M W/A       COUNT      SUM-TIME(us) MAX-TIME(us) AVG-TIME...
Top Locks - WALWriteLock•  Protects	  	  writes	  on	  WAL	  •  Acquired	  when	  WAL	  records	  flushed	  to	  disk	  	  ...
Top Locks - WALInsertLock•  Protects	  WAL	  Buffers	  •  Increasing	  wal	  buffers	  may	  help	  though	  to	  only	  cer...
Top Locks - ProcArrayLock•  Protects	  ProcArray	  structure	  •  It	  used	  to	  be	  that	  every	  transacon	  actuall...
Top Locks - SInvalidReadLock•  Protects	  sinval	  array	  	  •  Readers	  take	  “Shared”	  SInvalReadLock	  	  •  SIClea...
Top Locks - CLogControlLock•  Protects	  CLogControl	  structure	  •  Generally	  not	  a	  problem	  	  	  •  If	  it	  s...
Example Monitoring using Systemtap(Sysbench simple read)     LOCKNAME   LWID   M W/A       COUNT     SUM-TIME(us) MAX-TIME...
Top Locks - LockMgrLocks•  Protects	  relaons	  •  Sets	  of	  about	  16	  Lock	  Parons	  by	  default	  to	  handle	  a...
Other Locks - BufMappingLocks•  Protects	  regions	  of	  Buffers	  •  Sets	  of	  about	  16	  Regions	  of	  Buffers	  by	...
Problems with LWLock•  Overall	  system	  gravitates	  to	  certain	  top	  locks	  •  Single	  mutex	  lock	  protects	  ...
Questions / More Informationv Email:	  jshah@vmware.com	  v Learn	  more	  about	  PostgreSQL	      q  hIp://www.postgr...
Upcoming SlideShare
Loading in...5
×

Understanding PostgreSQL LW Locks

1,428

Published on

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,428
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
29
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Understanding PostgreSQL LW Locks

  1. 1. Understanding PostgreSQL LWLocks Jignesh Shah Staff Engineer, VMware Inc PgCon 2011 - Ottawa
  2. 2. About MySelf§  Joined  VMware  in    2010     q   PostgreSQL  performance  on  vSphere  §  Previously  at  Sun  Microsystems  from  2000-­‐2010   q  Database  Performance  on  Solaris/Sun  Systems  §  Work  with    PostgreSQL  Performance  Community     q  Scaling,  BoIlenecks  using  various  workloads  §  My  Blog:  hIp://jkshah.blogspot.com         © 2011 VMware Inc 2
  3. 3. Contentv What  are  PostgreSQL  LWLocks  ?  v Understanding  modes  of  LWLock  v LWLocks  API  v Architecture  of  the  LWLocks  Framework    v   Internals  of  LWLock  Wait  List    v Ways  to  monitors  the  events  related  to  LWLocks  v Top  LWLocks.               © 2011 VMware Inc 3
  4. 4. What are PostgreSQL LWLocks?§  LWLocks  –  Light  Weight  Locks  of  PostgreSQL  §  Primary  Intent   q  Mutually  Exclusive  access  to  shared  memory  structures  §  Offers  Shared  and  Exclusive  mode  §  Not  to  be  confused  with  the  PostgreSQL  Locks    §  PostgreSQL  Locks  depends  on  LWLocks  to  protect  its  shared   state  §  Uses  SpinLockAcquire/SpinLockRelease  and   PGSemaphoreLock/PGSemaphoreUnlock   © 2011 VMware Inc 4
  5. 5. Understanding Modes of LWLocksExclusiveLock  §    For  a  parcular  lockid    there  can  be  only  1  exclusivelock  held     (no  shared  locks  and  no  other  exclusive  lock  for  the  lockid)      SharedLock  •  For  a  parcular  lockid  there  can  be  1  or  more  SharedLock  held   (no  exclusive  lock  for  the  lockid)           © 2011 VMware Inc 5
  6. 6. LWLocks API§  LWLockAssign(void)   §  Get  a  dynamic  LWLock  number  (mostly  at  module  load)  §  LWLockAcquire(LWLockId  lockid,  LWLockMode  mode)   §  Waits  ll  Lock  is  acquired    in  EXCLUSIVE  or  SHARED  mode  §  LWLockCondionalAcquire(LWLockId  lockid,  LWLockMode  mode)   §  Returns  False  if  it  not  available  (Non-­‐blocking)  §  LWLockRelease(LWLockId  lockid)   §  Releases  the  lock  §  LWLockReleaseAll  (void)   §  Used  aaer  an  ERROR  to  cleanup  §  LWLockHeldByMe  (LWLockId  lockid)   §  Debug  support  test       © 2011 VMware Inc 6
  7. 7. Architecture Flow of LWLocksLWLockCondionalAcquire    •  Easy  lock  it  if  available  else  return  false  (Non-­‐blocking)    LWLockAcquire    •  Lock    if  available  or  else  put  me  in    wait  queue  (FIFO)  and    got     to  sleep  on  a  semaphore  (and  loop)  •  Only  return  when  lock  is  available  (Blocking)  LWLockRelease  •  Release  the  Lock  and  if  there  are  no  more  shared  locks   remaining  then  wake  up  the  next  process  waing  in  the  wait   queue         © 2011 VMware Inc 7
  8. 8. Wait Queue List•  Wait  Queue  is  protected  by  a  mutex  which  is  accessed  using     SpinLocks  •  When  the  lock  is  released  and  there  are  no  more  shared  locks     pending  then  the  process  releasing  the  lock  will  wake  the  next     waiter  as  follows   •  If  the  next  process  in  wait  queue  is  waing  for  an   EXCLUSIVE  mode  ,  only  that  process  is  removed  from   the  queue  and  wakes  them  up   •  Or  since  the  next  process  is  waing  for  SHARED  mode,  it   will  try  to    remove    as  many  consecuve  SHARED  mode   processes  and  wake  them  up   © 2011 VMware Inc 8
  9. 9. Lock Acquisition Internals     Spins  on    ‘lock’  mutex   Sleep  on    its  process   wait  semaphore       Lock   •  If  Not  Add  to  Wait  Queue  List     Available   •  Release  ‘lock’  mutex     Release    ‘lock’  mutex   © 2011 VMware Inc 9
  10. 10. Lock Release Internals     Spins  on    ‘lock’  mutex       Release  Lock       •  Wake  Up    process  next  in  FIFO     Shared   •  If  it  is  Shared  wake  up  all   sequenal  shared  waiters Locks  ?   Release    ‘lock’  mutex   © 2011 VMware Inc 10
  11. 11. LWLocks Wait Queue List FIFO Example•  Consider  something  80-­‐20  Shared  /Exclusive  rao           S S S S E S E S S S S     It  will  wake  up  first  5  shared  locks  then  wait  ll  all  of  them   releases  the  lock  before  waking  up  the  process  asking   for  Exclusive   It  will  do  only  exclusive  one  then  and  one  it  releases  the   lock,  wakes  up  the  next  1  shared  one     © 2011 VMware Inc 11
  12. 12. Observations about LWLocks Wait Queue List•  All  Operaons  on  Wait  Queue  List  are  serialized       •  Not  scalable  on  SMP  architecture  •  Currently  only  FIFO  supported      •  No  restricon  on  shared  wakeups         © 2011 VMware Inc 12
  13. 13. LWLocks – Defined in lwlock.htypedef enum LWLockId{ BufFreelistLock, ShmemIndexLock, OidGenLock,   XidGenLock, SInvalWriteLock, ProcArrayLock, WALInsertLock, SInvalReadLock, WALWriteLock, ControlFileLock, CheckpointLock, CLogControlLock,   SubtransControlLock, MultiXactGenLock, MultiXactOffsetControlLock, MultiXactMemberControlLock, RelCacheInitLock, BgWriterCommLock,   TwoPhaseStateLock, TablespaceCreateLock, BtreeVacuumLock, AddinShmemInitLock, AutovacuumLock, AutovacuumScheduleLock, SyncScanLock, RelationMappingLock, AsyncCtlLock, AsyncQueueLock, SerializableXactHashLock, SerializableFinishedListLock, SerializablePredicateLockListLock, OldSerXidLock, SyncRepLock, /* Individual lock IDs end here */ FirstBufMappingLock, FirstLockMgrLock = FirstBufMappingLock + NUM_BUFFER_PARTITIONS, /*16 */ FirstPredicateLockMgrLock = FirstLockMgrLock + NUM_LOCK_PARTITIONS, /* must be last except for MaxDynamicLWLock: */ NumFixedLWLocks = FirstPredicateLockMgrLock + NUM_PREDICATELOCK_PARTITIONS, MaxDynamicLWLock = 1000000000} LWLockId; © 2011 VMware Inc 13
  14. 14. MonitoringLWLOCK_STAT  •  Special  build  with  LWLOCK_STAT  defined    LOCK_DEBUG    •  Puts  debug  messages  in  system  alert  log    DYNAMIC  TRACING  (DTrace,  SystemTap)    •  Recommended  •  Useful  to  find  hot  locks  •  Not  for  producon  use,  because  it  can  make  the  server   unstable   © 2011 VMware Inc 14
  15. 15. Dynamic Tracing for LWLocks•  postgresql-­‐lwlock-­‐wait-­‐start(lockid,mode)  •    Postgresql-­‐lwlock-­‐wait-­‐done(lockid,mode)  •    Postgresql-­‐lwlock-­‐acquire(lockid,mode)  •    Postgresql-­‐lwlock-­‐condacquire(lockid,mode)  •  Postgresql-­‐lwlock-­‐condacquire-­‐fail(lockid,mode)  •  Postgresql-­‐lwlock-­‐release(lockid)   © 2011 VMware Inc 15
  16. 16. Example Monitoring using Systemtap(DBT2) LOCKNAME LWID M W/A COUNT SUM-TIME(us) MAX-TIME(us) AVG-TIME(us) WALInsertLock 7 Ex W 14013 2746505 3955 195 WALWriteLock 8 Ex W 10006 25508653 286749 2549 LockMgrLock 55 Ex W 2035 203429 3323 99 LockMgrLock 45 Ex W 932 54297 2860 58 LockMgrLock 54 Ex W 673 24362 1062 36 ProcArrayLock 4 Ex W 515 15907 666 30 ProcArrayLock 4 Sh W 176 6064 97 34 LockMgrLock 56 Ex W 171 5826 376 34CLogControlLock 11 Sh W 111 22490 6127 202 LockMgrLock 57 Ex W 101 5524 1326 54 LockMgrLock 59 Ex W 79 2883 347 36CLogControlLock 11 Ex W 58 8543 4439 147 LockMgrLock 49 Ex W 57 1848 76 32 LockMgrLock 47 Ex W 57 3166 1468 55 © 2011 VMware Inc 16
  17. 17. Top Locks - WALWriteLock•  Protects    writes  on  WAL  •  Acquired  when  WAL  records  flushed  to  disk    •  Acquired  when  WAL  Log  switch  occurs    •  Improve  the  underlying  storage  of  pg_xlog    •  Synchronous_commit=off  helps  indirectly  (  Do  not  wait  for   flush  to  the  disk)  •  Full_page_writes=off  also  helps  reduce  the  stress  (but  not   recommended  since  resiliency  goes  down)   © 2011 VMware Inc 17
  18. 18. Top Locks - WALInsertLock•  Protects  WAL  Buffers  •  Increasing  wal  buffers  may  help  though  to  only  certain  extent    •  Snychnrous_commit  off  will  lead  to  increased  pressure  on  this       lock  (  Not  a  bad  thing  )  •  However  eventually  not  much  can  be  done  once  it  gets  to  a   big  problem  without  new  commits  •  Full_page_writes=off  certainly  helps  (again  not  recommended   since  it  reduces  the  resiliency  of  the  database  from  write   errors)   © 2011 VMware Inc 18
  19. 19. Top Locks - ProcArrayLock•  Protects  ProcArray  structure  •  It  used  to  be  that  every  transacon  actually  acquired  this  lock       in  exclusive  mode  before  commit  causing  it  to  be  a  top  lock  •  Fixed  in  9.0       © 2011 VMware Inc 19
  20. 20. Top Locks - SInvalidReadLock•  Protects  sinval  array    •  Readers  take  “Shared”  SInvalReadLock    •  SICleanupQueue  and  other  array-­‐wide  updates  take       “Excluslive”  SInvalReadLock  to  lock  out  all  readers  •  Long  wait  mes  to  acquire  SInvalidReadLock    generally  results   when  the    Shared  Buffer  pool  is  being  stressed    •  Increase  shared_buffers  in  postgresql.conf  corresponding  to   acve  data  size     © 2011 VMware Inc 20
  21. 21. Top Locks - CLogControlLock•  Protects  CLogControl  structure  •  Generally  not  a  problem      •  If  it  shows  on  the  top  lists  check       •  $PGDATA/pg_clog  should  be  on  buffered  file  system   © 2011 VMware Inc 21
  22. 22. Example Monitoring using Systemtap(Sysbench simple read) LOCKNAME LWID M W/A COUNT SUM-TIME(us) MAX-TIME(us) AVG-TIME(us) LockMgrLock 45 Ex W 85343 469682510 13152 5503 LockMgrLock 57 Ex W 57547 30903727 8313 537 LockMgrLock 44 Ex W 390 34061 1670 87 LockMgrLock 59 Ex W 375 41570 2032 110 LockMgrLock 56 Ex W 361 39685 1889 109 LockMgrLock 47 Ex W 344 24548 1564 71 LockMgrLock 54 Ex W 335 67770 2319 202 LockMgrLock 50 Ex W 325 44213 1690 136 LockMgrLock 49 Ex W 325 39280 1475 120 LockMgrLock 55 Ex W 323 39448 1584 122 LockMgrLock 48 Ex W 323 26982 1669 83 © 2011 VMware Inc 22
  23. 23. Top Locks - LockMgrLocks•  Protects  relaons  •  Sets  of  about  16  Lock  Parons  by  default  to  handle  all       relaons  •  Each  relaon  is  part  of  only  one  paron  (irrespecve  of  size)       © 2011 VMware Inc 23
  24. 24. Other Locks - BufMappingLocks•  Protects  regions  of  Buffers  •  Sets  of  about  16  Regions  of  Buffers  by  default  to  handle  the       whole  Bufferpool  •  Only  taken  shared  access       © 2011 VMware Inc 24
  25. 25. Problems with LWLock•  Overall  system  gravitates  to  certain  top  locks  •  Single  mutex  lock  protects  adding  and  depleng  the  wait       queue       •  Not  SMP  Scalable   •  Chance  for  opmizaon  out  there   •  Performance  limited  to  serialized  rate  the  locks  are   processed     © 2011 VMware Inc 25
  26. 26. Questions / More Informationv Email:  jshah@vmware.com  v Learn  more  about  PostgreSQL   q  hIp://www.postgresql.org  v Blog:  hIp://jkshah.blogspot.com       © 2011 VMware Inc 26
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×