1212312232

2,291 views

Published on

Published in: Lifestyle, Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,291
On SlideShare
0
From Embeds
0
Number of Embeds
1,383
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

1212312232

  1. 1. Building a RAC Test Environmenton VMWare For FreeDan Norrisdnorris@piocon.comd i @ ihttp://www.dannorris.com/Thanks to Tim Hall @ oracle-base.com
  2. 2. Agenda•Who is Dan?•VMWare Orientation•Importance of prerequisite checking•Configuring node #1•Cloning (VM) node #1•Installing Clusterware•Installing ASM and Database•Creating Database•Common Ch ll C Challenges and I d Issues 2
  3. 3. Who is Dan?• Virgo• Scuba Diver (PADI Advanced OW, Nitrox)• Over 21, under 35• Oracle DBA & UNIX Admin backgroundg• Certifiable: OCM, ACE Director, RHCE• Consultant, mostly fixing things that are broken• Active community participant: RAC SIG, SIG Council, DBA Track Manager, blogger, tweeter• ESA Practice Manager at Piocon Technologies 3
  4. 4. VMWare Orientation•Desktop Products –VMWare Workstation: “Full” product $ VMWare Full product, –VMWare Player: Free, no creation, just use•Server Products Server –VMWare Infrastructure (ESX Server): $$$, robust, clustering capable, OS replacement , g p , p –VMWare VirtualCenter: $$, management tool for whole environment, Windows –VMWare Server: free, limited, on top of OS 4
  5. 5. VMWare Server Hardware•Almost any server will do, mine is: –Dell SC440: 1 x dual-core 3.0 Ghz –2 x 500Gb internal SATA drives, using LVM –4 Gb RAM•I use Linux (OEL 5) as host OS•Let Linux use LVM (default), allows disk space to be pooled for 900+ Gb on /u01 900•VMWare Server 1.0.5•Two Virtual Networks: 1 bridged & 1 host- g only 5
  6. 6. RAC: System Architecture Node1 Cluster Cl t Node2 Interconnect rac1 rac2 Local Shared Local Storage Storage Storage 6
  7. 7. First: Build Node1 Node1 Node2 Local Shared Local Storage Storage Storage 7
  8. 8. Node1: Create New VM 8
  9. 9. Node1: Create New VM 9
  10. 10. Node1: Create New VM 10
  11. 11. Node1: Create New VM 11
  12. 12. Node1: Create New VM 12
  13. 13. Node1: Create New VM 13
  14. 14. Node1: Create New VM 14
  15. 15. Node1: Create New VM 15
  16. 16. Node1: Create New VM (Pretend it says RAC1) 16
  17. 17. Node1: Create New VM 17
  18. 18. Node1: Create New VM 18
  19. 19. Node1: Create New VM 19
  20. 20. Node1: Create New VM 20
  21. 21. Node1: Create New VM 21
  22. 22. Node1: Create New VM 22
  23. 23. Node1: Create New VM 23
  24. 24. Install Node1 OS•Plan the network addresses for public and private networks•Start the VM and the CD image should boot•Install the OS ensuring enough swap and OS, /tmp•Assign static addresses t each i t f A i t ti dd to h interface•See whitepaper for package and setup details d t il 24
  25. 25. Post-Install Steps on Node1•Modify /etc/hosts•Set kernel parameters Set•Set limits in /etc/security/limits.conf•Add groups (oinstall, dba), oracle user•Set up oracle user’s profile (shell limits, Set user s ORACLE_BASE)•Set up SSH user equivalency 25
  26. 26. Prerequisite Checking•Most issues arise from misconfiguration of OS, packages networks or storage OS packages, networks,•Tools exist to help check prerequisites –CVU CVU –RDA•Some basic networking knowledge helps•Recipes can help, be sure y p p, you understand why, not just follow steps 26
  27. 27. Prerequisite Checking - RDA•Metalink Note 314422.1 will orient you to the RDA and has links for download•Download the .zip file, unzip and run (as root or oracle): –./rda.sh –T hcve –provide the release you want to check –answer additional questions –obtain results in ./output/RDA* files 27
  28. 28. Prerequisite Checking - RDA 28
  29. 29. Prerequisite Checking - RDA 29
  30. 30. Prerequisite Checking - CVU•http://www.oracle.com/technology/produ cts/database/clustering/cvu/cvu_downloa cts/database/clustering/cvu/cvu downloa d_homepage.html•Download a .zip file unzip it run it: Download zip file, it, –cluvfy stage -list 30
  31. 31. Prerequisite Checking - CVU On AMD64 platforms, some packages will be listed twice, once for i386 and once for AMD64 31
  32. 32. On AMD64 platforms,some packages will belisted twice, once fori386 and once forAMD64 32
  33. 33. Done: Build Node1 Node1 Node2 Local Shared Local Storage Storage Storage 33
  34. 34. Next: Add Shared Storage Node1 Node2 Local Shared Local Storage Storage Storage 34
  35. 35. Node1: Add Shared Storage•Shut down guest OS in Node1•On the VMWare ser er host OS create On server OS, a new directory for shared disk files:mkdir –p /u01/vmware/vms/rac disks p /u01/vmware/vms/rac-disks•Back at the VMWare Server Console, add new disks to the Node1 (RAC1) VM dd di k t th N d 1 as shown in the following example. 35
  36. 36. Node1: Add Shared Storage 36
  37. 37. Node1: Add Shared Storage 37
  38. 38. Node1: Add Shared Storage 38
  39. 39. Node1: Add Shared Storage 39
  40. 40. Node1: Add Shared Storage 40
  41. 41. Node1: Add Shared Storage 41
  42. 42. Node1: Add Shared Storage•Repeat to create the following: Virtual D i Vi t l Device Path Size Node/u01/vmware/vms/rac-disks/ocr1.vmdk 10 Gb SCSI 1:0/u01/vmware/vms/rac-disks/voting1.vmdk 10 Gb SCSI 1:1/u01/vmware/vms/rac-disks/asm1.vmdk 10 Gb SCSI 1:2/u01/vmware/vms/rac-disks/asm2.vmdk 10 Gb SCSI 1:3/u01/vmware/vms/rac-disks/asm3.vmdk 10 Gb SCSI 1:4 42
  43. 43. Node1: Add Shared Storage•Now for the tricky part: hand-editing VMWare configuration filefile.•The changes we make enable the disks to be shared with other VMs VMs.•Add or update the following parameters in the “/u01/vmware/vms/RAC1/Red i th “/ / / / / Hat Enterprise Linux 4.vmx” file 43
  44. 44. Node1: Add Shared Storagedisk.locking = "FALSE"diskLib.dataCacheMaxSize = "0"diskLib.dataCacheMaxReadAheadSize = "0"diskLib.dataCacheMinReadAheadSize = "0"diskLib.dataCachePageSize = "4096"diskLib.maxUnsyncedWrites = "0"scsi1.sharedBus = "VIRTUAL" i1 h dB 44
  45. 45. Node1: Add Shared Storagescsi1:0.deviceType = "disk"scsi1:0.redo = ""scsi1:1.deviceType = "disk"scsi1:1.redo = ""scsi1:2.deviceTypescsi1:2 deviceType = "disk"scsi1:2.redo = ""scsi1:3.deviceType yp = "disk"scsi1:3.redo = ""scsi1:4.deviceType = "disk"scsi1:4.redo = "" 45
  46. 46. Node1: Add Shared Storage•Boot Node1 again and partition the disks to create partition 1 as the whole disk•“Quick and Dirty” setup: add this to /etc/rc.local: /etc/rc local:chown oracle:oinstall /dev/sd[b-f]1chmod 600 / /dev/sd[b-f]1 / 46
  47. 47. Done: Add Shared Storage Node1 Node2 Local Shared Local Storage Storage Storage 47
  48. 48. Next: Clone Node1 to Make Node2 Node1 Node2 Local Shared Local Storage Storage Storage 48
  49. 49. Create Node2: Clone Node1•Node2 is a VMWare clone of Node1•Shut do n Node1 nicel Sh t down nicely•On host OS, docp –Rp /u01/vmware/vms/RAC1 /u01/vmware/vms/RAC2mv /u01/vmware/vms/RAC2/RAC1.vmx /u01/vmware/vms/RAC2/RAC2.vmx /u01/vmware/vms/RAC2/RAC2 vmx 49
  50. 50. Create Node2: Clone Node1•Boot Node2 (leave Node1 down)•During power on choose Create when on, prompted about handling the identifer.•Modify –/etc/sysconfig/network –/etc/sysconfig/network-scripts/ifcfg-eth0 –/etc/sysconfig/network-scripts/ifcfg-eth1 / / fi / k i /if f h1•From the console, run “service network restart” 50
  51. 51. Create Node2: Clone Node1•Boot Node2 (leave Node1 down)•During power on choose Create when During on, prompted about handling the identifer.•Modify y – /etc/sysconfig/network – /etc/sysconfig/network-scripts/ifcfg-eth0 – /etc/sysconfig/network-scripts/ifcfg-eth1 / / y g/ p / g•From the virtual machine’s console, run “service network restart”•You can now power up Node1 51
  52. 52. Prerequisite Checking - CVU•Finally able to complete more prerequisite checking•As the oracle user, run CVU to confirm private networking and user networking, equivalence:./cluvfy stage –post h / l f t t hwos –n rac1,rac2 1 2 52
  53. 53. 53
  54. 54. Prerequisite Checking - CVU•Shared storage checks don’t work on VMWare since the disks do not support SMART (used by CVU to verify disk accessibility)•Next, check the pre-crsinst stage like this:./cluvfy –pre crsinst –n rac1,rac2 / l f i t 1 2 –r 11gR1 54
  55. 55. Prerequisite Checking - CVU 55
  56. 56. Prerequisite Checking - CVU 56
  57. 57. Prerequisite Checking - CVU•With all CVU checks complete and successful, successful we can move ahead to installing clusterware! 57
  58. 58. Done: Node2 Created Node1 Node2 Local Shared Local Storage Storage Storage 58
  59. 59. Next: Install Clusterware Node1 Cluster Cl t Node2 Interconnect Local Shared Local Storage Storage Storage 59
  60. 60. Installing Clusterware•Clusterware installation performed from one node (Node1) (Node1).•Ensure /etc/hosts is configured with host addresses, addresses VIP addresses and private addresses, addresses 60
  61. 61. Installing Clusterware 61
  62. 62. Installing Clusterware 62
  63. 63. Installing Clusterware 63
  64. 64. Installing Clusterware 64
  65. 65. Installing Clusterware 65
  66. 66. Installing Clusterware 66
  67. 67. Installing Clusterware 67
  68. 68. Installing Clusterware 68
  69. 69. Installing Clusterware 69
  70. 70. Installing Clusterware 70
  71. 71. Installing Clusterware 71
  72. 72. Installing Clusterware 72
  73. 73. Installing Clusterware Note: Do not run these scripts in parallel. They should be run sequentially. 73
  74. 74. Installing Clusterware 74
  75. 75. Installing Clusterware Now go back and click OK on the installer p prompt box. p 75
  76. 76. Installing Clusterware 76
  77. 77. Installing Clusterware 77
  78. 78. Installing Clusterware 78
  79. 79. Installing Clusterware•Once the installation is complete, you can check the status with crs stat -t crs_stat t 79
  80. 80. Finished: Install Clusterware Node1 Cluster Cl t Node2 Interconnect Local Shared Local Storage Storage Storage 80
  81. 81. Next: Install ASM and Database Node1 Cluster Cl t Node2 Interconnect +ASM1 +ASM2 Local Shared Local Storage Storage Storage 81
  82. 82. Installing ASM & Database 82
  83. 83. Installing ASM & Database 83
  84. 84. Installing ASM & Database 84
  85. 85. Installing ASM & Database 85
  86. 86. Installing ASM & Database 86
  87. 87. Installing ASM & Database 87
  88. 88. Installing ASM & Database 88
  89. 89. Installing ASM & Database 89
  90. 90. Installing ASM & Database 90
  91. 91. Installing ASM & Database 91
  92. 92. Installing ASM & Database 92
  93. 93. Installing ASM & Database 93
  94. 94. Installing ASM & Database 94
  95. 95. Installing ASM & Database 95
  96. 96. Installing ASM & Database 96
  97. 97. Installing ASM & Database 97
  98. 98. Installing ASM & Database 98
  99. 99. Finished: Install ASM and Database Node1 Cluster Cl t Node2 Interconnect +ASM1 +ASM2 Local Shared Local Storage Storage Storage 99
  100. 100. Next: Create Database rac1 Cluster Cl t rac2 Node1 Interconnect Node2 +ASM1 +ASM2 Local Shared Local Storage Storage Storage 100
  101. 101. Create Database 101
  102. 102. Create Database 102
  103. 103. Create Database 103
  104. 104. Create Database 104
  105. 105. Create Database 105
  106. 106. Create Database 106
  107. 107. Create Database 107
  108. 108. Create Database 108
  109. 109. Create Database 109
  110. 110. Create Database 110
  111. 111. Create Database 111
  112. 112. Create Database 112
  113. 113. Create Database 113
  114. 114. Create Database 114
  115. 115. Create Database 115
  116. 116. Create Database 116
  117. 117. Create Database 117
  118. 118. Create Database 118
  119. 119. Create Database 119
  120. 120. Create Database 120
  121. 121. Create Database 121
  122. 122. Create Database …after a rather long pause while instances are started…(drumroll, please) i t t t d (d ll l ) 122
  123. 123. Done: Create Database rac1 Cluster Cl t rac2 Node1 Interconnect Node2 +ASM1 +ASM2 Local Shared Local Storage Storage Storage 123
  124. 124. Common Issues #1: Firewalls•Firewall and SELinux should be disabled•Can be done d ring install with some Can during ith versions of Linux. If not, then…•Set SELINUX=disabled i S t in /etc/sysconfig/selinux•chkconfig iptables off•Reboot Reboot 124
  125. 125. Common Issues #2: Storage•Shared Storage misconfiguration – different names for disks on each node•For RHEL or OEL 5, use udev for privileges and device mapping•Note Linux v4 and v5 handle things differently•Ensure you use proper device in multipath configurations (not applicable in VMWare) 125
  126. 126. Common Issues #2: Storage, cont’d Note Title443996.1 How to map raw device on RHEL5 and OEL5 How to setup /etc/udev/rules.d for OCR and Voting for RAC on414897.1 SLES10/RH5/EL5371814.1 Using udev with Oracle Architecture (RAC & ASM) – Red Hat 4.0456239.1 Understanding device-mapper in Linux 2.6 kernel357472.1 Configuring device-mapper for CRS/ASM Configuring raw devices for Oracle 10gR2 (10.2.0) Clusterware on465001.1465001 1 RH/EL 5 126
  127. 127. Common Issues #3: Networking•Networking, especially private network, must be configured properly•NO CROSSOVER CABLES ALLOWED!•In VMWare ensure you use correct VMWare, interface for public, private networks•Use h t l network f private, bridged U host-only t k for i t b id d for public•Set up b di b f S t bonding before i t lli clusterware installing l t 127
  128. 128. Common Issues #4: Filesystems•Don’t “go nuts” with Linux filesystem layout•Avoid too many filesystems (especially on a test system)•Ensure you create sufficient swap space 128
  129. 129. Common Issues #5: Host equivalency•Don’t hate SSH, just set it up properly!ssh-keygen tssh keygen –t dsacat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys / /scp –rp ~/.ssh othernode:.ssh <remote_node> date•Metalink Note 300548 1 is helpful 300548.1 129
  130. 130. References•Much of this article originated from Tim Hall s Hall’s article at oracle-base com (with oracle-base.com permission--Thanks, Tim!): http://tinyurl.com/39ywx2 http://tinyurl com/39ywx2•Oracle 11g Documentation•Oracle Metalink, multiple notes and articles 130
  131. 131. 131
  132. 132. RAC SIG Events• See www.oracleracsig.org for details –Webcasts: Average 2x per month, live –Conference Events: C f • Customer Panel, Monday @ 10:30a • Experts Panel, Tuesday @ 12:15p • Bi d of a F th W d Birds f Feather, Wednesday @ 1 30 d 1:30p –Forums (via OTN): Lots of participation from RAC SIG as well as Oracle gurus• J i th RAC SIG at www.oracleracsig.org! Join the t l i ! 132
  133. 133. Building a RAC Test Environment onVMWare For FreeSession #341S iDan Norris, Twitter: dannorris NorrisPiocon booth #1619dnorris@piocon.comdnorris@piocon comhttp://www.dannorris.com/
  134. 134. LegalThe information contained herein should be deemed reliablebut not guaranteed. The author has made every attempt topprovide current and accurate information. If y have any you ycomments or suggestions, please contact the author at: dnorris@piocon.comYou may request redistribution permission fromdnorris@piocon.com. Copyright © 2008, Piocon Technologies 134

×