Christo Kutrovsky presented on Pythian's experience with RAC and ASM configurations over 3 years of production use. Key lessons included: 1) A 2-node RAC requires a 3rd observer node to avoid split-brain issues during maintenance. 2) ASMLIB can cause issues if multipathing is present and devices are not configured properly. It is best to access disks directly without ASMLIB. 3) Restoring backups into the same ASM diskgroup as production can cause data corruption if the restore overwrites existing files. 4) Device naming conventions and spfile locations must be standardized across RAC nodes to avoid parameter conflicts. 5) Adding new

1. RAC+ASM 3 years in production Stories to sharePresented by: Christo Kutrovsky

2. Who Am I2Oracle ACE

3. 10 years in Oracle field

4. Joined Pythian 2003

5. Part of Pythian Consulting Team

6. Special projects

7. Performance tuning

8. Critical services“oracle pinup”

9. Pythian FactsFounded in 199790 employees120 customers worldwide20 customers more than $1 billion in revenue5 offices in 5 countries10 years profitable private company

10. What Pythian doesPythian provides database and application infrastructure services.

11. Agenda52 nodes RAC

12. ASMLIB with multipathing

13. Migrating to new servers with ASM

14. Thin provisioning

15. ASM + restores = danger

16. Device naming conventions

17. spfile location

18. JBOD configuration62 Node RAC for High Availability

19. 2 Node RACs for HA7Two node RAC nodes

20. 13 databases

21. Dev databases

22. Shutdown databases (and ASM) on node1

23. Perform maintenance

24. Unplug the interconnect cable

25. What happens?2 nodes RAC8VIPVIPNode 1Node 2SID_A1SID_A2SID_B1SID_B2InterconnectASMDGFibre ChannelOCR/V

26. 2 nodes RAC9VIPVIPNode 1Node 2SID_A1SID_A2SID_B1SID_B2InterconnectASMDGFibre ChannelOCR/V

27. 10I can’tSee Node 1I can’tSee Node 2VIPVIPNode 1Node 2SID_A1SID_A2SID_B1SID_B2InterconnectASMDGFibre ChannelOCR/V

28. One is not Quorum1150% chance your working node gets restarted

29. Depends on clusterware version

30. Who will shoot the other guy firstOne is not Quorum12Conclusion?

31. Turn off clusterware when you have only 2 nodes and performing maintenance

32. Upgrade to a more predicable clusterware

33. Lowest ‘leader’ always survives

34. Add a 3th tie-breaker node

35. doesn’t have to run a database, just clusterware (observer)One is not Quorum13Production cases, what happens ifAll Network dies on one node?

36. All disk dies on one node?14ASMLIBwith Multi Pathing

37. Building ASMLIB devices when multipathing is present15Devices used for creating asmlib

38. /dev/emcpowerc1

39. /dev/mapper/raid10_data_disk

40. Devices used to create asmdiskgroup

41. ASMLIB

42. The reboot changes everything

43. ASMLIB re-discovers the devices without multipath

44. Difficult to diagnoseVisual16/dev/mapper/data1/dev/mapper/data2/dev/sdb/dev/sdc/dev/sdd/dev/sdeHBA1HBA2LUN_1LUN_2

45. Building ASMLIB devices when multipathing is present17Do not use ASMLIB

46. If you have to (why?)

47. Must setup “ORACLEASM_SCANORDER”

48. asm_diskstring parameter

49. Permissions

50. Udev files

51. Boot/startup scriptRemoving ASMLIB18Why

52. Extra layer

53. Requires new driver for every new kernel

54. Can cause downtime if not careful

55. ASMLIB header is the same as ASM DISK header

56. Just has extra field for ASMLIB name

57. Disks can be accessed directly, without ASMLIB without having to drop/recreate themRemoving ASMLIB19Unmount all affected diskgroups

58. Change or set asm_diskstring

59. Remount diskgroups via new paths

60. Can be done in rolling fashion in RAC20SAN Migration

61. Migrating from EMC to 3PAR21New SAN

62. New concept

63. Thin provisioning

64. A big project

65. Or notAdd/drop/go home22No brainer

66. Thin provisioning rocks

67. SA adds disks

68. Add disk to diskgroup

69. Drop all old disks

70. Wait

71. Never be paged on space23Server Migration

72. Server migration24Current setup

73. 2 nodes RAC with ASM

74. New servers

75. Better, Faster, Stronger

76. Fastest (effort wise) way to migrate, with minimal downtime

77. Possible with zero downtimeServer migration options25Create standby on new server

78. Requires extra copy of data

79. Add the new nodes, drop existing ones

80. Possible clusterware issues

81. Move the LUNs

82. Easy

83. New servers testedLun Migration26Install clusterware and create RAC database with same name

84. Test hardware / wiring / configuration

85. Migrate

86. Stop production

87. Re-assigning LUNs

88. Start production27ASM Restore creates database black hole

89. ASM + Same host restore = DANGER28Production database

90. Diskgroup +PROD

91. Snapshot database

92. Diskgroup +SNAP

93. Rebuild monthly via duplicate database

94. Except this one time…The concept29“SNAP” backups not taken

95. If a given “SNAP” backup is to be restored, simply re-create the given “PROD” backup

96. Independent from ProductionRestore with ASM30Restore FRA files into separate directory

97. Startup SNAP instance

98. Catalog backup files

99. Restore into SNAP diskgroup

100. The missing piece?“restore” writes into original backup file location

101. Must use “set new name for datafile” in run blockRestore with ASM – the result31Unrecoverable corruption on production database

102. Lost about 3-4 hours of changes

103. If this was filesystem and not ASM, no corruption would have occuredCorruption – what happened32SGA5rows5rows2rowsBLK1 add Row 6BLK3 add Row 35rows5rows2rows2rowsDiskPartially overwritten datafile5rows5rows5 rows5 rowsREDODiskOriginal datafile

104. Corruption – what should’ve hap.33SGA5rows5rows5rowsBLK1 add Row 6BLK3 add Row 35rows5rows2rows2rowsBLK3 add Row 6DiskPartially overwritten datafile5rows5rows5 rows5 rowsREDODiskOriginal datafile

105. Corruption – what happened34

106. Corruption35Why this wouldn’t have happened with filesystem?

107. File names are just pointers to data stream

108. If a file is re-created, a new data streams is associated with it

109. Processes that have the file currently open still use the old data stream

110. This is why “undelete” is possible

111. My blog about undeleting filesCorruptionOpen “file 1”36Process 1File 1Stream X1

112. Corruption – recreate File 1Open “file 1”37Process 1File 1Stream X1Stream X2

113. 38Device names convention causes user error

114. Device naming conventions39Using /dev/mapper/<name>

115. Asm uses <name>p1 – first partition

116. Permissions set script uses: “*p1”

117. Then came /dev/mapper/backup1

118. First partition is: /dev/mapper/backup1p1Device naming conventions40V$ASM_DISKPATH HEADER_STATUS--------------------------- -------------/dev/mapper/backup1 CANDIDATE/dev/mapper/redop1 MEMBER/dev/mapper/backup1p1 MEMBER/dev/mapper/data2p1 MEMBER/dev/mapper/data1p1 MEMBER

119. Naming conventions41ADDEDDISKPartition 1IN USE

120. New convention42Now we use generic names, as we do re-assign disks

121. We also use prefix and suffix with a clear dilimiter/dev/mapper/asm-raid5-dev01-part1

122. 43spfile location in RAC

123. spfile location44Intended configuration

124. init.oraspfile=‘+ASM_DSKGRP/dbname.spfile’

125. no spfileChanging parameters in masses45create pfile=‘your_initials.ora’ from spfile;

126. edit

127. create spfile=‘+ASM_DSK/spfile’ from pfile=‘ck.ora’What not to do46create pfile from spfile

128. edit

129. create spfile from pfile;Result47One node uses local spfile

130. Other(s) uses global spfile

131. Parameter changes to “BAD” node are sent to other nodes

132. not persistent on GOOD nodes

133. persistent on BAD nodes

134. Paramer changes on GOOD nodes have reversed behaviour48Adding ASM disks crashes databases

135. Adding disks49Must be visible on all servers

136. Otherwise your diskgroup gets dismounted on the nodes that don’t see the disk

137. All databases using this diskgroup crashASM add disk process50Is the disk visible locally?Initialize disk header, add it to diskgroupNotify all nodes to rescan disks and add the new diskIf one or more nodes cannot see the disk, raise errorDismount diskgroup on all nodes not seeing the new disk

138. 51ASM with JBODwelcomes simplicity

139. JBOD Configuration52Linux Datawarehouse

140. 10 TB space

141. 28 disks of 430/285 GB

142. All redundancy/striping provided by ASMJBOD Configuration53Simplicity

143. No ASMLIB

144. Straight devices

145. Naming convention – use only 1 partition, and use partition 4

146. /dev/sd*4

147. is ASM partition

148. is permissions wildcard

149. is asm_diskstringTesting your speed54Verify read speed of each device

150. Verifies each device is performing as expected

151. Verify read speed from all devices

152. Verify your total bandwith

153. Verify read speed from all devices, towards the end of the device

154. Disk read speed is not linearRead Speed of a single disk55* Courtesy google image search

155. Testing your speed56One device at a timefor dsk in /dev/sd[c-q]; do echo $dsk; dd if=$dsk of=/dev/null iflag=direct bs=2M count=100; doneAll devices (total bandwith)for dsk in /dev/sd[c-q]; do echo $dsk; dd if=$dsk of=/dev/null iflag=direct bs=2M count=100 &; doneTest end speed

156. Add SKIP=xSample output57/dev/sdc100+0 records in100+0 records out209715200 bytes (210 MB) copied, 1.60325 seconds, 131 MB/s/dev/sdd100+0 records in100+0 records out209715200 bytes (210 MB) copied, 1.60188 seconds, 131 MB/s/dev/sde100+0 records in100+0 records out209715200 bytes (210 MB) copied, 1.60067 seconds, 131 MB/s/dev/sdf100+0 records in100+0 records out209715200 bytes (210 MB) copied, 1.59928 seconds, 131 MB/s/dev/sdg100+0 records in100+0 records out

157. JBOD configuration58Disk adding/removal is very easy

158. Add disks in bulk:alter diskgroup XXX add disk ‘/dev/sd[c-q]4’;

159. Performance rocks

160. controller speed

161. Diagnostic is easy

162. Iostat –x 5 /dev/sd*4

163. Manageability is easy

164. 1 diskgroup – no filenames, no mountpointsFinal Thoughts59RAC for HA requires 3 nodes

165. ASM

166. Keep it simple

167. Reduce layers

168. Runs fast

169. Still need to be carefull60The EndThank YouQuestions?I blog athttp://www.pythian.com/news/author/kutrovsky/

170. 61Transition/Section Marker