Patrick Gates and Peter Schroeder
How to Build a Personal DBA Lab to Keep
Up with Ever Changing Demands
3/19/2014 2www.datavail.com
Our Virtual Box End State Configuration
Oracle Base machine
• DV_BASE_OEL_6.3
DNS server
• DV_...
3/19/2014 3www.datavail.com
3/19/2014 4www.datavail.com
What is Virtual Box?
• VirtualBox is a general-purpose full virtualizer for x86 hardware,
targ...
3/19/2014 5www.datavail.com
Why we used VirtualBox
• It’s Free!!
• Supports Multiple Operating Systems
• Ability to setup ...
3/19/2014 6www.datavail.com
How we did it
We did not reinvent the wheel building this lab
We used lots of good blogs and s...
3/19/2014 7www.datavail.com
How we did it - Continued
• Enables you to get deeper understanding of how things actually wor...
3/19/2014 8www.datavail.com
Setup a BASE machine
• Build everything into base machine so you don’t have redo it
every time...
3/19/2014 9www.datavail.com
3/19/2014 10www.datavail.com
3/19/2014 11www.datavail.com
3/19/2014 12www.datavail.com
3/19/2014 13www.datavail.com
3/19/2014 14www.datavail.com
3/19/2014 15www.datavail.com
3/19/2014 16www.datavail.com
3/19/2014 17www.datavail.com
3/19/2014 18www.datavail.com
3/19/2014 19www.datavail.com
3/19/2014 20www.datavail.com
3/19/2014 21www.datavail.com
3/19/2014 22www.datavail.com
3/19/2014 23www.datavail.com
3/19/2014 24www.datavail.com
3/19/2014 25www.datavail.com
3/19/2014 26www.datavail.com
3/19/2014 27www.datavail.com
3/19/2014 28www.datavail.com
3/19/2014 29www.datavail.com
3/19/2014 30www.datavail.com
3/19/2014 31www.datavail.com
3/19/2014 32www.datavail.com
3/19/2014 33www.datavail.com
3/19/2014 34www.datavail.com
3/19/2014 35www.datavail.com
3/19/2014 36www.datavail.com
3/19/2014 37www.datavail.com
3/19/2014 38www.datavail.com
3/19/2014 39www.datavail.com
3/19/2014 40www.datavail.com
3/19/2014 41www.datavail.com
3/19/2014 42www.datavail.com
3/19/2014 43www.datavail.com
3/19/2014 44www.datavail.com
3/19/2014 45www.datavail.com
3/19/2014 46www.datavail.com
3/19/2014 47www.datavail.com
3/19/2014 48www.datavail.com
3/19/2014 49www.datavail.com
3/19/2014 50www.datavail.com
3/19/2014 51www.datavail.com
3/19/2014 52www.datavail.com
3/19/2014 53www.datavail.com
3/19/2014 54www.datavail.com
3/19/2014 55www.datavail.com
3/19/2014 56www.datavail.com
3/19/2014 57www.datavail.com
3/19/2014 58www.datavail.com
3/19/2014 59www.datavail.com
3/19/2014 60www.datavail.com
3/19/2014 61www.datavail.com
3/19/2014 62www.datavail.com
3/19/2014 63www.datavail.com
3/19/2014 64www.datavail.com
3/19/2014 65www.datavail.com
3/19/2014 66www.datavail.com
3/19/2014 67www.datavail.com
Cloning the BASE machine
Why clone the BASE machine?
• OS is already loaded
• Oracle OS user ...
3/19/2014 68www.datavail.com
Clone Base machine for a DNS server
Why create a DNS Server?
• Mimics a production environmen...
3/19/2014 69www.datavail.com
3/19/2014 70www.datavail.com
3/19/2014 71www.datavail.com
3/19/2014 72www.datavail.com
3/19/2014 73www.datavail.com
3/19/2014 74www.datavail.com
3/19/2014 75www.datavail.com
3/19/2014 76www.datavail.com
3/19/2014 77www.datavail.com
3/19/2014 78www.datavail.com
3/19/2014 79www.datavail.com
3/19/2014 80www.datavail.com
3/19/2014 81www.datavail.com
3/19/2014 82www.datavail.com
3/19/2014 83www.datavail.com
3/19/2014 84www.datavail.com
3/19/2014 85www.datavail.com
3/19/2014 86www.datavail.com
3/19/2014 87www.datavail.com
3/19/2014 88www.datavail.com
3/19/2014 89www.datavail.com
3/19/2014 90www.datavail.com
3/19/2014 91www.datavail.com
3/19/2014 92www.datavail.com
3/19/2014 93www.datavail.com
3/19/2014 94www.datavail.com
3/19/2014 95www.datavail.com
3/19/2014 96www.datavail.com
3/19/2014 97www.datavail.com
3/19/2014 98www.datavail.com
3/19/2014 99www.datavail.com
3/19/2014 100www.datavail.com
3/19/2014 101www.datavail.com
3/19/2014 102www.datavail.com
Clone from base machine for 11g Single Instance
with ASM
Main highlights
• Setup ASM disks i...
3/19/2014 103www.datavail.com
3/19/2014 104www.datavail.com
3/19/2014 105www.datavail.com
3/19/2014 106www.datavail.com
3/19/2014 107www.datavail.com
3/19/2014 108www.datavail.com
3/19/2014 109www.datavail.com
3/19/2014 110www.datavail.com
3/19/2014 111www.datavail.com
3/19/2014 112www.datavail.com
3/19/2014 113www.datavail.com
3/19/2014 114www.datavail.com
3/19/2014 115www.datavail.com
3/19/2014 116www.datavail.com
3/19/2014 117www.datavail.com
3/19/2014 118www.datavail.com
3/19/2014 119www.datavail.com
3/19/2014 120www.datavail.com
3/19/2014 121www.datavail.com
3/19/2014 122www.datavail.com
3/19/2014 123www.datavail.com
3/19/2014 124www.datavail.com
3/19/2014 125www.datavail.com
3/19/2014 126www.datavail.com
3/19/2014 127www.datavail.com
3/19/2014 128www.datavail.com
3/19/2014 129www.datavail.com
3/19/2014 130www.datavail.com
Clone from base machine for the first node of a two
Node 11gR2 RAC install
Main highlights
•...
3/19/2014 131www.datavail.com
3/19/2014 132www.datavail.com
3/19/2014 133www.datavail.com
3/19/2014 134www.datavail.com
3/19/2014 135www.datavail.com
3/19/2014 136www.datavail.com
3/19/2014 137www.datavail.com
3/19/2014 138www.datavail.com
3/19/2014 139www.datavail.com
3/19/2014 140www.datavail.com
3/19/2014 141www.datavail.com
3/19/2014 142www.datavail.com
3/19/2014 143www.datavail.com
3/19/2014 144www.datavail.com
3/19/2014 145www.datavail.com
3/19/2014 146www.datavail.com
3/19/2014 147www.datavail.com
3/19/2014 148www.datavail.com
3/19/2014 149www.datavail.com
3/19/2014 150www.datavail.com
Clone from base machine for the second node of a
two Node 11gR2 RAC install
Main highlights
...
3/19/2014 151www.datavail.com
3/19/2014 152www.datavail.com
3/19/2014 153www.datavail.com
3/19/2014 154www.datavail.com
3/19/2014 155www.datavail.com
3/19/2014 156www.datavail.com
3/19/2014 157www.datavail.com
3/19/2014 158www.datavail.com
3/19/2014 159www.datavail.com
3/19/2014 160www.datavail.com
3/19/2014 161www.datavail.com
3/19/2014 162www.datavail.com
Installing Grid Infrastructure on DV-RAC1 and DV-
RAC2
Potential Gotchas
• Disable NTP
• Set...
3/19/2014 163www.datavail.com
3/19/2014 164www.datavail.com
3/19/2014 165www.datavail.com
3/19/2014 166www.datavail.com
3/19/2014 167www.datavail.com
vi /etc/udev/permissions.d/50-udev.permissions
add these lines to the file and save
oracleas...
3/19/2014 168www.datavail.com
if [ -e /usr/bin/nslookup.orig ]; then
echo "ERROR: /usr/bin/nslookup.orig already exist"
ex...
3/19/2014 169www.datavail.com
3/19/2014 170www.datavail.com
Clone from DV-130 (11g DB machine) to install 12c
OEM Cloud Control
Main highlights
• Cloned...
3/19/2014 171www.datavail.com
3/19/2014 172www.datavail.com
3/19/2014 173www.datavail.com
3/19/2014 174www.datavail.com
3/19/2014 175www.datavail.com
3/19/2014 176www.datavail.com
3/19/2014 177www.datavail.com
3/19/2014 178www.datavail.com
3/19/2014 179www.datavail.com
3/19/2014 180www.datavail.com
3/19/2014 181www.datavail.com
3/19/2014 182www.datavail.com
3/19/2014 183www.datavail.com
3/19/2014 184www.datavail.com
3/19/2014 185www.datavail.com
3/19/2014 186www.datavail.com
3/19/2014 187www.datavail.com
3/19/2014 188www.datavail.com
3/19/2014 189www.datavail.com
3/19/2014 190www.datavail.com
3/19/2014 191www.datavail.com
3/19/2014 192www.datavail.com
Questions?
Upcoming SlideShare
Loading in …5
×

Oracle Sandbox

1,589 views

Published on

Published in: Technology
  • Be the first to comment

Oracle Sandbox

  1. 1. Patrick Gates and Peter Schroeder How to Build a Personal DBA Lab to Keep Up with Ever Changing Demands
  2. 2. 3/19/2014 2www.datavail.com Our Virtual Box End State Configuration Oracle Base machine • DV_BASE_OEL_6.3 DNS server • DV_DNS 11g single instance with ASM • DV_130_11G 2 node RAC • DV_RAC1 • DV_RAC2 12c OEM Cloud Control • DV_EM12C
  3. 3. 3/19/2014 3www.datavail.com
  4. 4. 3/19/2014 4www.datavail.com What is Virtual Box? • VirtualBox is a general-purpose full virtualizer for x86 hardware, targeted at server, desktop and embedded use. • Presently, VirtualBox runs on Windows, Linux, Macintosh, and Solaris hosts and supports a large number of guest operating systems including but not limited to Windows (NT 4.0, 2000, XP, Server 2003, Vista, Windows 7), DOS/Windows 3.x, Linux (2.4 and 2.6), Solaris and OpenSolaris, OS/2, and OpenBSD. • Download Virtual Box at www.virtualbox.org
  5. 5. 3/19/2014 5www.datavail.com Why we used VirtualBox • It’s Free!! • Supports Multiple Operating Systems • Ability to setup host only networking • Testing becomes easier with features like immutable disks and snapshots • Can share disks between severs • i.e. RAC installations • Allows us to setup and perform functionality testing quickly • Standbys • Guaranteed Restore points • Basic RMAN recoveries • RMAN duplicates or clones • 10g to 11gR2 upgrades
  6. 6. 3/19/2014 6www.datavail.com How we did it We did not reinvent the wheel building this lab We used lots of good blogs and sites to build what is effective for us • Installing OEL • http://www.oracle-base.com/articles/linux/oracle-linux-6-installation.php • Setting up a DNS Server • http://www.oracle-base.com/articles/linux/dns-configuration-for-scan.php • Installing 11gR2 RAC • http://www.oracle-base.com/articles/11g/oracle-db-11gr2-rac-installation- on-oracle-linux-6-using-virtualbox.php • How to reconfigure oralce OHAS when the hostname is changed • http://www.dbaglobe.com/2011/05/summarize-of-how-to-change- hostname-in.html
  7. 7. 3/19/2014 7www.datavail.com How we did it - Continued • Enables you to get deeper understanding of how things actually work • Can try different configurations within the OS and database • OS • Install the OS, Change OS parameters, set kernel values • Create all the users and groups necessary for an Oracle install • Oracleasm configure • Oracleasm createdisk • Setup SCAN IP addresses • Database • Flashback Database • Fast Start Failover • Active Dataguard
  8. 8. 3/19/2014 8www.datavail.com Setup a BASE machine • Build everything into base machine so you don’t have redo it every time • Use NAT not bridged for networking • You can share you machine with a coworker with out having IP conflicts • Setup VirtualBox Host Only networking • What is in the BASE machine • OEL OS 6.3 • Root user • Oracle user • ASMLIB
  9. 9. 3/19/2014 9www.datavail.com
  10. 10. 3/19/2014 10www.datavail.com
  11. 11. 3/19/2014 11www.datavail.com
  12. 12. 3/19/2014 12www.datavail.com
  13. 13. 3/19/2014 13www.datavail.com
  14. 14. 3/19/2014 14www.datavail.com
  15. 15. 3/19/2014 15www.datavail.com
  16. 16. 3/19/2014 16www.datavail.com
  17. 17. 3/19/2014 17www.datavail.com
  18. 18. 3/19/2014 18www.datavail.com
  19. 19. 3/19/2014 19www.datavail.com
  20. 20. 3/19/2014 20www.datavail.com
  21. 21. 3/19/2014 21www.datavail.com
  22. 22. 3/19/2014 22www.datavail.com
  23. 23. 3/19/2014 23www.datavail.com
  24. 24. 3/19/2014 24www.datavail.com
  25. 25. 3/19/2014 25www.datavail.com
  26. 26. 3/19/2014 26www.datavail.com
  27. 27. 3/19/2014 27www.datavail.com
  28. 28. 3/19/2014 28www.datavail.com
  29. 29. 3/19/2014 29www.datavail.com
  30. 30. 3/19/2014 30www.datavail.com
  31. 31. 3/19/2014 31www.datavail.com
  32. 32. 3/19/2014 32www.datavail.com
  33. 33. 3/19/2014 33www.datavail.com
  34. 34. 3/19/2014 34www.datavail.com
  35. 35. 3/19/2014 35www.datavail.com
  36. 36. 3/19/2014 36www.datavail.com
  37. 37. 3/19/2014 37www.datavail.com
  38. 38. 3/19/2014 38www.datavail.com
  39. 39. 3/19/2014 39www.datavail.com
  40. 40. 3/19/2014 40www.datavail.com
  41. 41. 3/19/2014 41www.datavail.com
  42. 42. 3/19/2014 42www.datavail.com
  43. 43. 3/19/2014 43www.datavail.com
  44. 44. 3/19/2014 44www.datavail.com
  45. 45. 3/19/2014 45www.datavail.com
  46. 46. 3/19/2014 46www.datavail.com
  47. 47. 3/19/2014 47www.datavail.com
  48. 48. 3/19/2014 48www.datavail.com
  49. 49. 3/19/2014 49www.datavail.com
  50. 50. 3/19/2014 50www.datavail.com
  51. 51. 3/19/2014 51www.datavail.com
  52. 52. 3/19/2014 52www.datavail.com
  53. 53. 3/19/2014 53www.datavail.com
  54. 54. 3/19/2014 54www.datavail.com
  55. 55. 3/19/2014 55www.datavail.com
  56. 56. 3/19/2014 56www.datavail.com
  57. 57. 3/19/2014 57www.datavail.com
  58. 58. 3/19/2014 58www.datavail.com
  59. 59. 3/19/2014 59www.datavail.com
  60. 60. 3/19/2014 60www.datavail.com
  61. 61. 3/19/2014 61www.datavail.com
  62. 62. 3/19/2014 62www.datavail.com
  63. 63. 3/19/2014 63www.datavail.com
  64. 64. 3/19/2014 64www.datavail.com
  65. 65. 3/19/2014 65www.datavail.com
  66. 66. 3/19/2014 66www.datavail.com
  67. 67. 3/19/2014 67www.datavail.com Cloning the BASE machine Why clone the BASE machine? • OS is already loaded • Oracle OS user is created • You will not fail 11g prerequisite checks • Consistency • Takes about 2 to 3 minutes to clone Potential Gotchas • You need to make sure you always check “Reinitalize the MAC addresses for all network cards” on the screen when you are cloning the base machine.
  68. 68. 3/19/2014 68www.datavail.com Clone Base machine for a DNS server Why create a DNS Server? • Mimics a production environment • No need to update the /etc/hosts file on all virtual servers • You can setup a round robin SCAN IP for 11gR2 RAC installs
  69. 69. 3/19/2014 69www.datavail.com
  70. 70. 3/19/2014 70www.datavail.com
  71. 71. 3/19/2014 71www.datavail.com
  72. 72. 3/19/2014 72www.datavail.com
  73. 73. 3/19/2014 73www.datavail.com
  74. 74. 3/19/2014 74www.datavail.com
  75. 75. 3/19/2014 75www.datavail.com
  76. 76. 3/19/2014 76www.datavail.com
  77. 77. 3/19/2014 77www.datavail.com
  78. 78. 3/19/2014 78www.datavail.com
  79. 79. 3/19/2014 79www.datavail.com
  80. 80. 3/19/2014 80www.datavail.com
  81. 81. 3/19/2014 81www.datavail.com
  82. 82. 3/19/2014 82www.datavail.com
  83. 83. 3/19/2014 83www.datavail.com
  84. 84. 3/19/2014 84www.datavail.com
  85. 85. 3/19/2014 85www.datavail.com
  86. 86. 3/19/2014 86www.datavail.com
  87. 87. 3/19/2014 87www.datavail.com
  88. 88. 3/19/2014 88www.datavail.com
  89. 89. 3/19/2014 89www.datavail.com
  90. 90. 3/19/2014 90www.datavail.com
  91. 91. 3/19/2014 91www.datavail.com
  92. 92. 3/19/2014 92www.datavail.com
  93. 93. 3/19/2014 93www.datavail.com
  94. 94. 3/19/2014 94www.datavail.com
  95. 95. 3/19/2014 95www.datavail.com
  96. 96. 3/19/2014 96www.datavail.com
  97. 97. 3/19/2014 97www.datavail.com
  98. 98. 3/19/2014 98www.datavail.com
  99. 99. 3/19/2014 99www.datavail.com
  100. 100. 3/19/2014 100www.datavail.com
  101. 101. 3/19/2014 101www.datavail.com
  102. 102. 3/19/2014 102www.datavail.com Clone from base machine for 11g Single Instance with ASM Main highlights • Setup ASM disks in virtual box and server • Create in vbox • Fdisk steps • Oracleasm configure, createdisk • Changing network • Changing hostname • http://www.dbaglobe.com/2011/05/summarize-of- how-to-change-hostname-in.html
  103. 103. 3/19/2014 103www.datavail.com
  104. 104. 3/19/2014 104www.datavail.com
  105. 105. 3/19/2014 105www.datavail.com
  106. 106. 3/19/2014 106www.datavail.com
  107. 107. 3/19/2014 107www.datavail.com
  108. 108. 3/19/2014 108www.datavail.com
  109. 109. 3/19/2014 109www.datavail.com
  110. 110. 3/19/2014 110www.datavail.com
  111. 111. 3/19/2014 111www.datavail.com
  112. 112. 3/19/2014 112www.datavail.com
  113. 113. 3/19/2014 113www.datavail.com
  114. 114. 3/19/2014 114www.datavail.com
  115. 115. 3/19/2014 115www.datavail.com
  116. 116. 3/19/2014 116www.datavail.com
  117. 117. 3/19/2014 117www.datavail.com
  118. 118. 3/19/2014 118www.datavail.com
  119. 119. 3/19/2014 119www.datavail.com
  120. 120. 3/19/2014 120www.datavail.com
  121. 121. 3/19/2014 121www.datavail.com
  122. 122. 3/19/2014 122www.datavail.com
  123. 123. 3/19/2014 123www.datavail.com
  124. 124. 3/19/2014 124www.datavail.com
  125. 125. 3/19/2014 125www.datavail.com
  126. 126. 3/19/2014 126www.datavail.com
  127. 127. 3/19/2014 127www.datavail.com
  128. 128. 3/19/2014 128www.datavail.com
  129. 129. 3/19/2014 129www.datavail.com
  130. 130. 3/19/2014 130www.datavail.com Clone from base machine for the first node of a two Node 11gR2 RAC install Main highlights • Setup DNS entries for hostnames, SCANs, public IPs, private IPs and VIPs • Show how pinging the SCANs will round round through IPs from DNS server • Setup shared disks for RAC • Reconfirm how network is setup on VirtualBox Potential Gotchas • Disable NTP • Setup of network interface types from grid infrastructure install • install the cvuqdisk rpm from GI install media
  131. 131. 3/19/2014 131www.datavail.com
  132. 132. 3/19/2014 132www.datavail.com
  133. 133. 3/19/2014 133www.datavail.com
  134. 134. 3/19/2014 134www.datavail.com
  135. 135. 3/19/2014 135www.datavail.com
  136. 136. 3/19/2014 136www.datavail.com
  137. 137. 3/19/2014 137www.datavail.com
  138. 138. 3/19/2014 138www.datavail.com
  139. 139. 3/19/2014 139www.datavail.com
  140. 140. 3/19/2014 140www.datavail.com
  141. 141. 3/19/2014 141www.datavail.com
  142. 142. 3/19/2014 142www.datavail.com
  143. 143. 3/19/2014 143www.datavail.com
  144. 144. 3/19/2014 144www.datavail.com
  145. 145. 3/19/2014 145www.datavail.com
  146. 146. 3/19/2014 146www.datavail.com
  147. 147. 3/19/2014 147www.datavail.com
  148. 148. 3/19/2014 148www.datavail.com
  149. 149. 3/19/2014 149www.datavail.com
  150. 150. 3/19/2014 150www.datavail.com Clone from base machine for the second node of a two Node 11gR2 RAC install Main highlights • Finish configuration for shared disks with VirtualBox Potential Gotchas • Disable NTP • Setup of network interface types from grid infrastructure install • install the cvuqdisk rpm from GI install media
  151. 151. 3/19/2014 151www.datavail.com
  152. 152. 3/19/2014 152www.datavail.com
  153. 153. 3/19/2014 153www.datavail.com
  154. 154. 3/19/2014 154www.datavail.com
  155. 155. 3/19/2014 155www.datavail.com
  156. 156. 3/19/2014 156www.datavail.com
  157. 157. 3/19/2014 157www.datavail.com
  158. 158. 3/19/2014 158www.datavail.com
  159. 159. 3/19/2014 159www.datavail.com
  160. 160. 3/19/2014 160www.datavail.com
  161. 161. 3/19/2014 161www.datavail.com
  162. 162. 3/19/2014 162www.datavail.com Installing Grid Infrastructure on DV-RAC1 and DV- RAC2 Potential Gotchas • Disable NTP • Setup of network interface types from grid infrastructure install • resolv.conf bug, ignore check on install
  163. 163. 3/19/2014 163www.datavail.com
  164. 164. 3/19/2014 164www.datavail.com
  165. 165. 3/19/2014 165www.datavail.com
  166. 166. 3/19/2014 166www.datavail.com
  167. 167. 3/19/2014 167www.datavail.com vi /etc/udev/permissions.d/50-udev.permissions add these lines to the file and save oracleasm/disks/CRS01:0660 oracleasm/disks/DATA01:0660 oracleasm/disks/DATA02:0660 Fix for Device Checks for ASM bug
  168. 168. 3/19/2014 168www.datavail.com if [ -e /usr/bin/nslookup.orig ]; then echo "ERROR: /usr/bin/nslookup.orig already exist" exit 1 Fi mv /usr/bin/nslookup /usr/bin/nslookup.orig echo '#!/bin/bash /usr/bin/nslookup.orig $* exit 0' > /usr/bin/nslookup chmod a+x /usr/bin/nslookup Fix for resolv.conf bug
  169. 169. 3/19/2014 169www.datavail.com
  170. 170. 3/19/2014 170www.datavail.com Clone from DV-130 (11g DB machine) to install 12c OEM Cloud Control Main highlights • Cloned from an existing machine that had an 11g database because that is required for the OEM 12c install • How we used used Google as mail relay • Demonstrate 12c Grid monitoring targets Potential Gotchas • Hostname has to be changed • IP has to be changed • OHAS service has to be reconfigured • Specific database parameters required • Used this link as a guide for the steps above • http://www.dbaglobe.com/2011/05/summarize-of-how- to-change-hostname-in.html
  171. 171. 3/19/2014 171www.datavail.com
  172. 172. 3/19/2014 172www.datavail.com
  173. 173. 3/19/2014 173www.datavail.com
  174. 174. 3/19/2014 174www.datavail.com
  175. 175. 3/19/2014 175www.datavail.com
  176. 176. 3/19/2014 176www.datavail.com
  177. 177. 3/19/2014 177www.datavail.com
  178. 178. 3/19/2014 178www.datavail.com
  179. 179. 3/19/2014 179www.datavail.com
  180. 180. 3/19/2014 180www.datavail.com
  181. 181. 3/19/2014 181www.datavail.com
  182. 182. 3/19/2014 182www.datavail.com
  183. 183. 3/19/2014 183www.datavail.com
  184. 184. 3/19/2014 184www.datavail.com
  185. 185. 3/19/2014 185www.datavail.com
  186. 186. 3/19/2014 186www.datavail.com
  187. 187. 3/19/2014 187www.datavail.com
  188. 188. 3/19/2014 188www.datavail.com
  189. 189. 3/19/2014 189www.datavail.com
  190. 190. 3/19/2014 190www.datavail.com
  191. 191. 3/19/2014 191www.datavail.com
  192. 192. 3/19/2014 192www.datavail.com Questions?

×