Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

History of database monitoring

1,570 views

Published on

History of database monitoring

Published in: Software
  • Be the first to comment

History of database monitoring

  1. 1. Midnight January 28, 1986 Lives are on the line History of Database Monitoring My own experiences http://kylehailey.com Kylelf@gmail.com
  2. 2. History … 1988 6 1989 1990 1991 1992 7 1993 1994 1995 1996 1997 8 1998 1999 2000 2001 9 2002 2003 10 2005 10.2 2006 2008 200X 2016
  3. 3. History … 1988 6 1989 Utlbstat/utlestat 1990 joined 1991 1992 7 1993 1994 1995 1996 1997 8 1998 1999 2000 2001 9 2002 2003 10 2005 10.2 2006 2008 200X 2016
  4. 4. History … 1988 6 1989 Utlbstat/utlestat 1990 joined 1991 1992 7 1993 1994 1995 1996 1997 8 1998 1999 2000 2001 9 2002 2003 10 2005 10.2 2006 2008 200X 2016
  5. 5. History … 1988 6 1989 Utlbstat/utlestat 1990 joined 1991 1992 7 1993 1994 1995 1996 1997 8 1998 1999 2000 2001 9 2002 2003 10 2005 10.2 2006 2008 200X 2016
  6. 6. History … 1988 6 1989 Utlbstat/utlestat 1990 joined 1991 1992 7 1993 1994 1995 1996 1997 8 1998 1999 2000 2001 9 2002 2003 10 2005 10.2 2006 2008 200X 2016
  7. 7. History … 1988 6 1989 Utlbstat/utlestat 1990 joined 1991 1992 7 Waits 1993 Patrol vs M2 1994 1995 1996 1997 8 1998 1999 2000 2001 9 2002 2003 10 2005 10.2 2006 2008 200X 2016 M2
  8. 8. History … 1988 6 1989 Utlbstat/utlestat 1990 joined 1991 1992 7 Waits Waits 1993 Patrol vs M2 1994 Tcl/Tk Europe Car 1995 Tcl/Tk waits 1996 1997 8 1998 1999 2000 2001 9 2002 2003 10 2005 10.2 2006 2008 200X 2016
  9. 9. History … 1988 6 1989 Utlbstat/utlestat 1990 joined 1991 1992 7 Waits Waits 1993 Patrol vs M2 1994 Tcl/Tk Europe Car 1995 Tcl/Tk waits 1996 1997 8 1998 1999 2000 2001 9 2002 2003 10 2005 10.2 2006 2008 200X 2016
  10. 10. History … 1988 6 1989 Utlbstat/utlestat 1990 joined 1991 1992 7 Waits Waits 1993 Patrol vs M2 1994 Tcl/Tk Europe Car 1995 Tcl/Tk waits 1996 1997 8 1998 Statspack 8.1.6 top waits 1999 Spotlight 2000 2001 9 2002 2003 10 2005 10.2 2006 2008 200X 2016
  11. 11. History … 1988 6 1989 Utlbstat/utlestat 1990 joined 1991 1992 7 Waits Waits 1993 Patrol vs M2 1994 Tcl/Tk Europe Car 1995 Tcl/Tk waits 1996 1997 8 1998 Statspack 8.1.6 top waits 1999 Spotlight 2000 Statspack 9iR2 top events 2001 9 2002 design OEM 10 2003 10 2005 10.2 2006 2008 200X 2016
  12. 12. History … 1988 6 1989 Utlbstat/utlestat 1990 joined 1991 1992 7 Waits Waits 1993 Patrol vs M2 1994 Tcl/Tk Europe Car 1995 Tcl/Tk waits 1996 1997 8 1998 Statspack 8.1.6 top waits 1999 Spotlight 2000 Statspack 9iR2 top events 2001 9 2002 design OEM 10 2003 10 OEM 10 2005 10.2 Top Activity 2006 2008 200X 2016
  13. 13. History … 1988 6 1989 Utlbstat/utlestat 1990 joined 1991 1992 7 Waits Waits 1993 Patrol vs M2 1994 Tcl/Tk Europe Car 1995 Tcl/Tk waits 1996 1997 8 1998 Statspack 8.1.6 top waits 1999 Spotlight 2000 Statspack 9iR2 top events 2001 9 2002 design OEM 10 2003 10 OEM 10 2005 10.2 Top Activity 2006 Ashmon 2008 DB Optimizer 200X YaaMs Delphix 2016
  14. 14. History … 1988 6 1989 Utlbstat/utlestat 1990 joined 1991 1992 7 Waits Waits 1993 Patrol vs M2 1994 Tcl/Tk Europe Car 1995 Tcl/Tk waits 1996 1997 8 1998 Statspack 8.1.6 top waits 1999 Spotlight 2000 Statspack 9iR2 top events 2001 9 2002 design OEM 10 2003 10 OEM 10 2005 10.2 Top Activity 2006 Ashmon 2008 DB Optimizer 200X YaaMs Delphix 2016 Amazon
  15. 15. Computer Performance 1. Interactive • Opaque: Static, idiosyncrasies Vs • Fun : Graphic , informative 1. Sampling • Counters and ratios Vs • Waits and sampling 1. Graphics • Spaghetti on the wall Vs • Intelligence in the interface
  16. 16. Computer Performance 1. Interactive • Opaque: Static, idiosyncrasies Vs • Fun : Graphic , informative 1. Sampling • Counters and ratios Vs • Waits and sampling 1. Graphics • Spaghetti on the wall Vs • Intelligence in the interface
  17. 17. Computer Performance 1. Interactive • Opaque: Static, idiosyncrasies Vs • Fun : Graphic , informative 1. Sampling • Counters and ratios Vs • Waits and sampling 1. Graphics • Spaghetti on the wall Vs • Intelligence in the interface
  18. 18. Theme: Let’s make performance tuning a video game
  19. 19. The journey of simplicity: Designing an Interface
  20. 20. The journey of simplicity 1. Seems simple “When you start looking at a problem and it seems really simple, you don’t really understand the complexity of the problem.” – Steve Jobs
  21. 21. The journey of simplicity 1. Seems simple 2. Realize it’s complex
  22. 22. The journey of simplicity 1. Seems simple 2. It’s complex 3. Create complex solution “Then you get into the problem, and you see that it’s really complicated, and you come up with all these convoluted solutions. That’s sort of the middle, and that’s where most people stop.” – Steve Jobs
  23. 23. The journey of simplicity 1. Seems simple 2. It’s complex 3. Complex solution 4. Complex solution is bad
  24. 24. The journey of simplicity 1. Seems simple 2. It’s complex 3. Complex solution 4. Complex is bad 5. Simple powerful is hard “But the really great person will keep on going and find the key, the underlying principle of the problem — and come up with an elegant, really beautiful solution that works.” – Steve Jobs
  25. 25. Simple can be harder than complex. You have to work hard to get your thinking clean to make it simple.
  26. 26. Prototype & Iterate (Cary Millsap)
  27. 27. Thought: images are running the world
  28. 28. Computers (&DBs)- can be “black boxes”
  29. 29. How do you get in?
  30. 30. How do you get in?
  31. 31. Wordstar
  32. 32. Text Friction
  33. 33. You log in, then what?
  34. 34. 1984 OK, UI can change everything
  35. 35. Computer performance
  36. 36. What’s inside? OR
  37. 37. How do you make good tools ?
  38. 38. Prototype and Iterate
  39. 39. Make it a video game !
  40. 40. Utlbstat/Utlestat … •Intrusive •Overwhelming •Ratios & Averages rem $Header: utlbstat.sql 26-feb-96.19:20:51 gpongrac Exp Rem Copyright (c) 1988, 1996 by Oracle Corporation Rem NAME REM UTLBSTAT.SQL Rem MODIFIED Rem khailey 03/15/99 - add current user fields to Rem jloaiza 10/14/95 - add tablespace size Rem jloaiza 09/19/95 - add waitstat Rem jloaiza 09/04/95 - add per second and backgro Rem drady 09/09/93 - merge changes from branch Rem drady 03/22/93 - merge changes from branch Rem drady 08/24/93 - bug 173918 Rem drady 03/04/93 - fix bug 152986 Rem glumpkin 11/16/92 - Renamed from UTLSTATB.SQL Rem glumpkin 10/19/92 - Renamed from BSTAT.SQL Rem jloaiza 01/07/92 - rework for version 7 Rem mroberts 08/16/91 - fix view for v7 Rem rlim 04/29/91 - change char to varc Rem Laursen 01/01/91 - V6 to V7 merge Rem Loaiza 04/04/89 - fix run dates to minutes inst Rem Martin 02/22/89 - Creation Rem Jloaiza 02/23/89 - changed table names, added da Rem insert into stats$begin_event select * from v$system_even insert into stats$begin_roll select * from v$rollstat; insert into stats$begin_file select * from stats$file_vie insert into stats$begin_dc select * from v$rowcache; insert into stats$begin_stats select * from v$sysstat; insert into stats$begin_lib select * from v$librarycache; insert into stats$begin_latch select * from v$latch;
  41. 41. 1993 First Monitors - Patrol
  42. 42. 1993 Patrol on Dec 8400
  43. 43. 1993
  44. 44. Roger Saunders M2 •Light Weight •
  45. 45. Roger Saunders M2 •Light Weight •Direct Memory Access •
  46. 46. Roger Saunders M2 •Light Weight •Direct Memory Access •Sampling
  47. 47. 1994 Light weight graphic : Tcl/Tk + M2
  48. 48. 1995 Tcl/Tk dynamic
  49. 49. Monitor Everything Approach
  50. 50. 1995 Monitor Everything Approach
  51. 51. Monitor Everything Approach
  52. 52. Monitor Everything Approach
  53. 53. Wait Events Became the Focus Graphs created dynamically Easy to destroy
  54. 54. Wait Events Became the Focus Graphs created dynamically Easy to destroy
  55. 55. Improvements •Scale Graphs Equally Eliminate: •Background waits •Idle Waits •Extraneous waits •Wait counts
  56. 56. log_file_switch_completion : increase log file size http://oraperf.sourceforge.net/seminar/ex3_test_2.html Centi- seconds
  57. 57. free buffer waits : increase db_block_buffers
  58. 58. log file sync : log file -> raw device
  59. 59. db file sequential read : increase db_block_buffers
  60. 60. write complete wait free buffer waits : increase db_block_buffers db file sequential reads
  61. 61. Final
  62. 62. Tuning catproc.sql on version 7 Compulsive Tuning Disorder Missing CPU usage to put into perspective
  63. 63. 1998 8.1.6 Statspack •Kyle wanted M2 & Graphics •Connie wanted statspack •boss : neither ! •Connie did statspack  •I left 
  64. 64. 1998 8.1.6 Statspack •Kyle wanted M2 & Graphics •Connie wanted statspack •boss : neither ! •Connie did statspack  •I left 
  65. 65. 1998 8.1.6 Statspack •Kyle wanted M2 & Graphics •Connie wanted statspack •boss : neither ! •Connie did statspack  •I left 
  66. 66. 1998 8.1.6 Statspack •Kyle wanted M2 & Graphics •Connie wanted statspack •boss : neither ! •Connie did statspack  •I left  •In the mean time ….
  67. 67. 1
  68. 68. 2
  69. 69. 3
  70. 70. 4
  71. 71. 5
  72. 72. 6
  73. 73. 7
  74. 74. 8
  75. 75. 9
  76. 76. 10
  77. 77. 11
  78. 78. 1999 Spotlight
  79. 79. Without Reading a Manual •Handspring’s site crashed Nov 25, 1999 •Biggest Sales Day of the Year •Library cache latch contention •No DBAs •Downloaded Quest’s Spotlight •Installed and Identified problem with Minutes •Solution in code fix
  80. 80. Spotlight – Stacked Waits !
  81. 81. Ratio of waits to CPU • How do you get CPU? • CPU stats? Centi- seconds
  82. 82. Session count CPU stacked No wait groups
  83. 83. EM v10 Proposed Perf Page (v4.1)
  84. 84. Avoid Scrolling and Hiding Data
  85. 85. OEM
  86. 86. Sampling vs Counters •Given a wait bottleneck •Which User •Which SQL •What object / file / block •Not feasible with counters •Easy and cheap with Sampling : Multi-dimensional
  87. 87. Before ASH:Before ASH: • Sessions v$sesstat, v$session_event • # sessions x (# wait events + statistics) • Example (150 x (800+200) = 150,000 ) • SQL v$sql • Could be 10000s • Takes out latches that compete with other sql executions • Objects V$segstat • Could be 1000s of objects • Files v$filestat Expensive !
  88. 88. Multi-DimensionalMulti-Dimensional Top Consumers Session User Object Module.Action Program Service Client Wait X Top Resources • CPU • Waits • Event (800*) • I/O • File • Block • Time Top SQL SQL ID Plan Child# X And Aggregated over any time Period
  89. 89. Multi-dimensionalMulti-dimensional 25 34 36 38 45 63 65 8725 34 36 38 45 63 65 87 F1qcyh20550cfF1qcyh20550cf fj6gjgsshtxyxfj6gjgsshtxyx 0cjsxw5ndqdbc0cjsxw5ndqdbc 8t8as9usk11qw8t8as9usk11qw dr1rkrznhh95bdr1rkrznhh95b 10dkqv3kr8xa510dkqv3kr8xa5 38zhkf4jdyff438zhkf4jdyff4 298wmz1kxjs1m298wmz1kxjs1m C PU C PUEnq: TX – row lock contention Enq: TX – row lock contention SQ L*N et break/reset to client SQ L*N et break/reset to client db file scattered read db file scattered read db file sequential read db file sequential read IO Application SQLSQL Sessions W aits W aits ServiceService Scott SystemUserUser ProgramProgram Sys Sqlplus Toad GL OE Package Procedure Plan Child #
  90. 90. Sampling vs Waits
  91. 91. Statistic Lag Time Copyright 2006 Kyle Hailey CountersCounters SamplesSamples Slight LagsSlight Lags
  92. 92. If you are not tuning for time, you are wasting time Max CPU (yard stick) Top ActivityTop Activity SQLSQL SessionsSessions LOADLOAD
  93. 93. Cambrian Explosion :YaaMs
  94. 94. Confio Ignite
  95. 95. Dell Foglight
  96. 96. Lab128
  97. 97. D.side
  98. 98. W-ASH
  99. 99. Ashviewer
  100. 100. emlite
  101. 101. ASHmon
  102. 102. MyOra
  103. 103. Mumbai
  104. 104. Precise
  105. 105. Lighty
  106. 106. Example Problem How so you communicated quantive data ?
  107. 107. Midnight January 28, 1986 Lives are on the line Thanks to Edward Tufte Night before the Flight Jan 27,1986
  108. 108. Estimated launch temperature 29º
  109. 109. 13 Pages Faxed
  110. 110. 13 Pages Faxed 3 different types of names
  111. 111. Damage (in overwhelming detail) but No Temperatures 13 Pages Faxed
  112. 112. 13 Pages Faxed Missing Data for 5 erosion damage flights Blow by Damage Test engines fired horizontally
  113. 113. 13 Pages Faxed Shows “blow by”, not more important “erosion” Damage at hottest and coldest launches * (of the flights shown) Next day’s flight
  114. 114. 13 Pages Faxed Predict Temperature Recommendation
  115. 115. 55 65 7560 70 80 1 Original Engineering data 2 3 ““damages atdamages at the hottestthe hottest and coldestand coldest Temperature”Temperature” Would you launch?
  116. 116. Congressional Hearings Evidence No Damage Legend Damage hard to read
  117. 117. Congressional Hearings Evidence Temperature correlation difficult
  118. 118. 55 65 7560 70 80 1 Original Data 2 3
  119. 119. Clearer 1. Y-Axis amount of damage (not number of damage) 55 65 7560 70 80 4 8 12
  120. 120. 1. Y-Axis amount of damage (not number of damage) 2. Include successes * 55 65 7560 70 80 4 8 12 Clearer * Only external temperatures were known not the temperature of the solid rocket boosters Be accurate enough
  121. 121. 1. Y-Axis amount of damage (not number of damage) 2. Include successes 3. Mark Differences 55 65 7560 70 80 4 8 12 Clearer
  122. 122. 1. Y-Axis amount of damage (not number of damage) 2. Include successes 3. Mark Differences 4. Normalize same temp 55 65 7560 70 80 4 8 12 Clearer
  123. 123. 1. Y-Axis amount of damage (not number of damage) 2. Include successes 3. Mark Differences 4. Normalize same temp 55 65 7560 70 80 4 8 12 Clearer Damage on every flight below 65 No damage on every flight above 75
  124. 124. 1. Y-Axis amount of damage (not number of damage) 2. Include successes 3. Mark Differences 4. Normalize same temp 55 65 7560 70 80 4 8 12 Clearer Known World
  125. 125. 1. Y-Axis amount of damage (not number of damage) 2. Include successes 3. Mark Differences 4. Normalize same temp 5. Scale known vs unknown 55 65 7560 70 80 4 8 12 4 8 12 30 40 5035 45 XX Clearer
  126. 126. Difficult NASA Engineers Fail Congressional Investigators Fail Data Visualization is Difficult But … Lack of Clarity can be devastating
  127. 127. Visualization can be powerful
  128. 128. “If I can't picture it, I can't understand it” Anscombe's Quartet I II III IV x y x y x y x y 10 8.04 10 9.14 10 7.46 8 6.58 8 6.95 8 8.14 8 6.77 8 5.76 13 7.58 13 8.74 13 12.74 8 7.71 9 8.81 9 8.77 9 7.11 8 8.84 11 8.33 11 9.26 11 7.81 8 8.47 14 9.96 14 8.1 14 8.84 8 7.04 6 7.24 6 6.13 6 6.08 8 5.25 4 4.26 4 3.1 4 5.39 19 12.5 12 10.84 12 9.13 12 8.15 8 5.56 7 4.82 7 7.26 7 6.42 8 7.91 5 5.68 5 4.74 5 5.73 8 6.89 Average 9 7.5 9 7.5 9 7.5 9 7.5 Standard Deviation 3.31 2.03 3.31 2.03 3.31 2.03 3.31 2.03 Linear Regression 1.33 1.33 1.33 1.33 - Albert Einstein- Albert Einstein
  129. 129. Graphics for Anscombe’s Quartet
  130. 130. Counties in US  > 3000 Counties  > 50 pages “The humans … are exceptionally good at parsing visual information.” Knowledge representation in cognitive science. Westbury, C. & Wilensky, U. (1998)
  131. 131. Visualizations can also obfuscate
  132. 132. Pretty Picture
  133. 133. Spaghetti at the wall
  134. 134. Spaghetti at the wall II
  135. 135. Amazon Cloudwatch
  136. 136. Imagine Trying to Drive your Car And is updated once and hourAnd is updated once and hour Or would you like it toOr would you like it to look …look … Would you want your dashboard to look like :Would you want your dashboard to look like :
  137. 137. If you are not tuning for time, you are wasting time
  138. 138. When Developers sayWhen Developers say The Database is slowThe Database is slow
  139. 139. AAS ~= 0AAS ~= 0
  140. 140. Do You Want? Engineering Data?Engineering Data?
  141. 141. Pretty PicturesPretty Pictures Do You Want?
  142. 142. Clean and ClearClean and Clear ? ? ? ?? ? ? ? ? ?? ? Do You Want?
  143. 143. Summary •Textual statistics – difficult to parse •Pretty pictures misleading •Goal clear graphics powerful Kylelf@gmail.com http://kylehailey.com
  144. 144. Summary •Textual statistics – difficult to parse •Pretty pictures misleading •Goal clear graphics powerful Simple can be harder than complex. You have to work hard to get your thinking clean to make it simple. Prototype & Iterate Kylelf@gmail.com http://kylehailey.com
  145. 145. •END

×