1. Building a RAC Test Environment
on VMWare For Free
Dan Norris
dnorris@piocon.com
d i @ i
http://www.dannorris.com/
Thanks to Tim Hall @ oracle-base.com
2. Agenda
â˘Who is Dan?
â˘VMWare Orientation
â˘Importance of prerequisite checking
â˘Configuring node #1
â˘Cloning (VM) node #1
â˘Installing Clusterware
â˘Installing ASM and Database
â˘Creating Database
â˘Common Ch ll
C Challenges and I
d Issues
2
3. Who is Dan?
⢠Virgo
⢠Scuba Diver (PADI Advanced OW, Nitrox)
⢠Over 21, under 35
⢠Oracle DBA & UNIX Admin backgroundg
⢠Certifiable: OCM, ACE Director, RHCE
⢠Consultant, mostly fixing things that are broken
⢠Active community participant: RAC SIG, SIG
Council, DBA Track Manager, blogger, tweeter
⢠ESA Practice Manager at Piocon Technologies
3
4. VMWare Orientation
â˘Desktop Products
âVMWare Workstation: âFullâ product $
VMWare Full product,
âVMWare Player: Free, no creation, just use
â˘Server Products
Server
âVMWare Infrastructure (ESX Server): $$$,
robust, clustering capable, OS replacement
, g p , p
âVMWare VirtualCenter: $$, management
tool for whole environment, Windows
âVMWare Server: free, limited, on top of OS
4
5. VMWare Server Hardware
â˘Almost any server will do, mine is:
âDell SC440: 1 x dual-core 3.0 Ghz
â2 x 500Gb internal SATA drives, using LVM
â4 Gb RAM
â˘I use Linux (OEL 5) as host OS
â˘Let Linux use LVM (default), allows disk
space to be pooled for 900+ Gb on /u01
900
â˘VMWare Server 1.0.5
â˘Two Virtual Networks: 1 bridged & 1 host-
g
only
5
6. RAC: System Architecture
Node1 Cluster
Cl t Node2
Interconnect
rac1 rac2
Local Shared Local
Storage Storage Storage
6
7. First: Build Node1
Node1 Node2
Local Shared Local
Storage Storage Storage
7
24. Install Node1 OS
â˘Plan the network addresses for public and
private networks
â˘Start the VM and the CD image should boot
â˘Install the OS ensuring enough swap and
OS,
/tmp
â˘Assign static addresses t each i t f
A i t ti dd to h interface
â˘See whitepaper for package and setup
details
d t il
24
25. Post-Install Steps on Node1
â˘Modify /etc/hosts
â˘Set kernel parameters
Set
â˘Set limits in /etc/security/limits.conf
â˘Add groups (oinstall, dba), oracle user
â˘Set up oracle userâs profile (shell limits,
Set user s
ORACLE_BASE)
â˘Set up SSH user equivalency
25
26. Prerequisite Checking
â˘Most issues arise from misconfiguration
of OS, packages networks or storage
OS packages, networks,
â˘Tools exist to help check prerequisites
âCVU
CVU
âRDA
â˘Some basic networking knowledge helps
â˘Recipes can help, be sure y
p p, you
understand why, not just follow steps
26
27. Prerequisite Checking - RDA
â˘Metalink Note 314422.1 will orient you to
the RDA and has links for download
â˘Download the .zip file, unzip and run (as
root or oracle):
â./rda.sh âT hcve
âprovide the release you want to check
âanswer additional questions
âobtain results in ./output/RDA* files
27
33. Done: Build Node1
Node1 Node2
Local Shared Local
Storage Storage Storage
33
34. Next: Add Shared Storage
Node1 Node2
Local Shared Local
Storage Storage Storage
34
35. Node1: Add Shared Storage
â˘Shut down guest OS in Node1
â˘On the VMWare ser er host OS create
On server OS,
a new directory for shared disk files:
mkdir âp /u01/vmware/vms/rac disks
p /u01/vmware/vms/rac-disks
â˘Back at the VMWare Server Console,
add new disks to the Node1 (RAC1) VM
dd di k t th N d 1
as shown in the following example.
35
42. Node1: Add Shared Storage
â˘Repeat to create the following:
Virtual D i
Vi t l Device
Path Size
Node
/u01/vmware/vms/rac-disks/ocr1.vmdk 10 Gb SCSI 1:0
/u01/vmware/vms/rac-disks/voting1.vmdk 10 Gb SCSI 1:1
/u01/vmware/vms/rac-disks/asm1.vmdk 10 Gb SCSI 1:2
/u01/vmware/vms/rac-disks/asm2.vmdk 10 Gb SCSI 1:3
/u01/vmware/vms/rac-disks/asm3.vmdk 10 Gb SCSI 1:4
42
43. Node1: Add Shared Storage
â˘Now for the tricky part: hand-editing
VMWare configuration filefile.
â˘The changes we make enable the disks
to be shared with other VMs
VMs.
â˘Add or update the following parameters
in the â/u01/vmware/vms/RAC1/Red
i th â/ / / / /
Hat Enterprise Linux 4.vmxâ file
43
46. Node1: Add Shared Storage
â˘Boot Node1 again and partition the disks
to create partition 1 as the whole disk
â˘âQuick and Dirtyâ setup: add this to
/etc/rc.local:
/etc/rc local:
chown oracle:oinstall /dev/sd[b-f]1
chmod 600 /
/dev/sd[b-f]1
/
46
47. Done: Add Shared Storage
Node1 Node2
Local Shared Local
Storage Storage Storage
47
48. Next: Clone Node1 to Make Node2
Node1 Node2
Local Shared Local
Storage Storage Storage
48
49. Create Node2: Clone Node1
â˘Node2 is a VMWare clone of Node1
â˘Shut do n Node1 nicel
Sh t down nicely
â˘On host OS, do
cp âRp /u01/vmware/vms/RAC1
/u01/vmware/vms/RAC2
mv /u01/vmware/vms/RAC2/RAC1.vmx
/u01/vmware/vms/RAC2/RAC2.vmx
/u01/vmware/vms/RAC2/RAC2 vmx
49
50. Create Node2: Clone Node1
â˘Boot Node2 (leave Node1 down)
â˘During power on choose Create when
on,
prompted about handling the identifer.
â˘Modify
â/etc/sysconfig/network
â/etc/sysconfig/network-scripts/ifcfg-eth0
â/etc/sysconfig/network-scripts/ifcfg-eth1
/ / fi / k i /if f h1
â˘From the console, run âservice network
restartâ
50
51. Create Node2: Clone Node1
â˘Boot Node2 (leave Node1 down)
â˘During power on choose Create when
During on,
prompted about handling the identifer.
â˘Modify
y
â /etc/sysconfig/network
â /etc/sysconfig/network-scripts/ifcfg-eth0
â /etc/sysconfig/network-scripts/ifcfg-eth1
/ / y g/ p / g
â˘From the virtual machineâs console, run
âservice network restartâ
â˘You can now power up Node1
51
52. Prerequisite Checking - CVU
â˘Finally able to complete more
prerequisite checking
â˘As the oracle user, run CVU to confirm
private networking and user
networking,
equivalence:
./cluvfy stage âpost h
/ l f t t hwos ân rac1,rac2
1 2
52
54. Prerequisite Checking - CVU
â˘Shared storage checks donât work on
VMWare since the disks do not support
SMART (used by CVU to verify disk
accessibility)
â˘Next, check the pre-crsinst stage like this:
./cluvfy âpre crsinst ân rac1,rac2
/ l f i t 1 2
âr 11gR1
54
57. Prerequisite Checking - CVU
â˘With all CVU checks complete and
successful,
successful we can move ahead to
installing clusterware!
57
58. Done: Node2 Created
Node1 Node2
Local Shared Local
Storage Storage Storage
58
59. Next: Install Clusterware
Node1 Cluster
Cl t Node2
Interconnect
Local Shared Local
Storage Storage Storage
59
60. Installing Clusterware
â˘Clusterware installation performed from
one node (Node1)
(Node1).
â˘Ensure /etc/hosts is configured with host
addresses,
addresses VIP addresses and private
addresses,
addresses
60
122. Create Database
âŚafter a rather long pause while
instances are startedâŚ(drumroll, please)
i t t t d (d ll l )
122
123. Done: Create Database
rac1 Cluster
Cl t
rac2
Node1 Interconnect Node2
+ASM1 +ASM2
Local Shared Local
Storage Storage Storage
123
124. Common Issues #1: Firewalls
â˘Firewall and SELinux should be disabled
â˘Can be done d ring install with some
Can during ith
versions of Linux. If not, thenâŚ
â˘Set SELINUX=disabled i
S t in
/etc/sysconfig/selinux
â˘chkconfig iptables off
â˘Reboot
Reboot
124
125. Common Issues #2: Storage
â˘Shared Storage misconfiguration â
different names for disks on each node
â˘For RHEL or OEL 5, use udev for
privileges and device mapping
â˘Note Linux v4 and v5 handle things
differently
â˘Ensure you use proper device in
multipath configurations (not applicable
in VMWare)
125
126. Common Issues #2: Storage, contâd
Note Title
443996.1 How to map raw device on RHEL5 and OEL5
How to setup /etc/udev/rules.d for OCR and Voting for RAC on
414897.1
SLES10/RH5/EL5
371814.1 Using udev with Oracle Architecture (RAC & ASM) â Red Hat 4.0
456239.1 Understanding device-mapper in Linux 2.6 kernel
357472.1 Configuring device-mapper for CRS/ASM
Configuring raw devices for Oracle 10gR2 (10.2.0) Clusterware on
465001.1
465001 1
RH/EL 5
126
127. Common Issues #3: Networking
â˘Networking, especially private network,
must be configured properly
â˘NO CROSSOVER CABLES ALLOWED!
â˘In VMWare ensure you use correct
VMWare,
interface for public, private networks
â˘Use h t l network f private, bridged
U host-only t k for i t b id d
for public
â˘Set up b di b f
S t bonding before i t lli clusterware
installing l t
127
128. Common Issues #4: Filesystems
â˘Donât âgo nutsâ with Linux filesystem
layout
â˘Avoid too many filesystems (especially
on a test system)
â˘Ensure you create sufficient swap space
128
129. Common Issues #5: Host equivalency
â˘Donât hate SSH, just set it up properly!
ssh-keygen t
ssh keygen ât dsa
cat ~/.ssh/id_dsa.pub >>
~/.ssh/authorized_keys
/ /
scp ârp ~/.ssh othernode:.
ssh <remote_node> date
â˘Metalink Note 300548 1 is helpful
300548.1
129
130. References
â˘Much of this article originated from Tim
Hall s
Hallâs article at oracle-base com (with
oracle-base.com
permission--Thanks, Tim!):
http://tinyurl.com/39ywx2
http://tinyurl com/39ywx2
â˘Oracle 11g Documentation
â˘Oracle Metalink, multiple notes and
articles
130
132. RAC SIG Events
⢠See www.oracleracsig.org for details
âWebcasts: Average 2x per month, live
âConference Events:
C f
⢠Customer Panel, Monday @ 10:30a
⢠Experts Panel, Tuesday @ 12:15p
⢠Bi d of a F th W d
Birds f Feather, Wednesday @ 1 30
d 1:30p
âForums (via OTN): Lots of participation from RAC
SIG as well as Oracle gurus
⢠J i th RAC SIG at www.oracleracsig.org!
Join the t l i !
132
133. Building a RAC Test Environment on
VMWare For Free
Session #341
S i
Dan Norris, Twitter: dannorris
Norris
Piocon booth #1619
dnorris@piocon.com
dnorris@piocon com
http://www.dannorris.com/
134. Legal
The information contained herein should be deemed reliable
but not guaranteed. The author has made every attempt to
p
provide current and accurate information. If y have any
you y
comments or suggestions, please contact the author at:
dnorris@piocon.com
You may request redistribution permission from
dnorris@piocon.com.
Copyright Š 2008, Piocon Technologies
134