Christo Kutrovsky presented on Pythian's experience with RAC and ASM configurations over 3 years of production use. Key lessons included: 1) A 2-node RAC requires a 3rd observer node to avoid split-brain issues during maintenance. 2) ASMLIB can cause issues if multipathing is present and devices are not configured properly. It is best to access disks directly without ASMLIB. 3) Restoring backups into the same ASM diskgroup as production can cause data corruption if the restore overwrites existing files. 4) Device naming conventions and spfile locations must be standardized across RAC nodes to avoid parameter conflicts. 5) Adding new

1. RAC+ASM 3 years in production Stories to share Presented by: Christo Kutrovsky

3. 10 years in Oracle field

4. Joined Pythian 2003

5. Part of Pythian Consulting Team

6. Special projects

7. Performance tuning

8. Critical services“oracle pinup”

9. Pythian Facts Founded in 1997 90 employees 120 customers worldwide 20 customers more than $1 billion in revenue 5 offices in 5 countries 10 years profitable private company

10. What Pythian does Pythian provides database and application infrastructure services.

12. ASMLIB with multipathing

13. Migrating to new servers with ASM

14. Thin provisioning

15. ASM + restores = danger

16. Device naming conventions

17. spfile location

20. 13 databases

21. Dev databases

22. Shutdown databases (and ASM) on node1

23. Perform maintenance

24. Unplug the interconnect cable

26. 2 nodes RAC 9 VIP VIP Node 1 Node 2 SID_A1 SID_A2 SID_B1 SID_B2 Interconnect ASM DG Fibre Channel OCR/V

27. 10 I can’t See Node 1 I can’t See Node 2 VIP VIP Node 1 Node 2 SID_A1 SID_A2 SID_B1 SID_B2 Interconnect ASM DG Fibre Channel OCR/V

29. Depends on clusterware version

31. Turn off clusterware when you have only 2 nodes and performing maintenance

32. Upgrade to a more predicable clusterware

33. Lowest ‘leader’ always survives

34. Add a 3th tie-breaker node

38. /dev/emcpowerc1

39. /dev/mapper/raid10_data_disk

40. Devices used to create asmdiskgroup

41. ASMLIB

42. The reboot changes everything

43. ASMLIB re-discovers the devices without multipath

46. If you have to (why?)

47. Must setup “ORACLEASM_SCANORDER”

48. asm_diskstring parameter

49. Permissions

50. Udev files

52. Extra layer

53. Requires new driver for every new kernel

54. Can cause downtime if not careful

55. ASMLIB header is the same as ASM DISK header

56. Just has extra field for ASMLIB name

58. Change or set asm_diskstring

59. Remount diskgroups via new paths

62. New concept

63. Thin provisioning

64. A big project

66. Thin provisioning rocks

67. SA adds disks

68. Add disk to diskgroup

69. Drop all old disks

70. Wait

73. 2 nodes RAC with ASM

74. New servers

75. Better, Faster, Stronger

76. Fastest (effort wise) way to migrate, with minimal downtime

78. Requires extra copy of data

79. Add the new nodes, drop existing ones

80. Possible clusterware issues

81. Move the LUNs

82. Easy

84. Test hardware / wiring / configuration

85. Migrate

86. Stop production

87. Re-assigning LUNs

90. Diskgroup +PROD

91. Snapshot database

92. Diskgroup +SNAP

93. Rebuild monthly via duplicate database

95. If a given “SNAP” backup is to be restored, simply re-create the given “PROD” backup

97. Startup SNAP instance

98. Catalog backup files

99. Restore into SNAP diskgroup

100. The missing piece?“restore” writes into original backup file location

102. Lost about 3-4 hours of changes

104. Corruption – what should’ve hap. 33 SGA 5 rows 5 rows 5 rows BLK1 add Row 6 BLK3 add Row 3 5 rows 5 rows 2 rows 2 rows BLK3 add Row 6 Disk Partially overwritten datafile 5 rows 5 rows 5 rows 5 rows REDO Disk Original datafile

105. Corruption – what happened 34

107. File names are just pointers to data stream

108. If a file is re-created, a new data streams is associated with it

109. Processes that have the file currently open still use the old data stream

110. This is why “undelete” is possible

112. Corruption – recreate File 1 Open “file 1” 37 Process 1 File 1 Stream X1 Stream X2

113. 38 Device names convention causes user error

115. Asm uses <name>p1 – first partition

116. Permissions set script uses: “*p1”

117. Then came /dev/mapper/backup1

119. Naming conventions 41 ADDED DISK Partition 1 IN USE

121. We also use prefix and suffix with a clear dilimiter/dev/mapper/asm-raid5-dev01-part1

122. 43 spfile location in RAC

124. init.oraspfile=‘+ASM_DSKGRP/dbname.spfile’

126. edit

128. edit

130. Other(s) uses global spfile

131. Parameter changes to “BAD” node are sent to other nodes

132. not persistent on GOOD nodes

133. persistent on BAD nodes

136. Otherwise your diskgroup gets dismounted on the nodes that don’t see the disk

138. 51 ASM with JBODwelcomes simplicity

140. 10 TB space

141. 28 disks of 430/285 GB

143. No ASMLIB

144. Straight devices

145. Naming convention – use only 1 partition, and use partition 4

146. /dev/sd*4

147. is ASM partition

148. is permissions wildcard

150. Verifies each device is performing as expected

151. Verify read speed from all devices

152. Verify your total bandwith

153. Verify read speed from all devices, towards the end of the device

158. Add disks in bulk:alter diskgroup XXX add disk ‘/dev/sd[c-q]4’;

159. Performance rocks

160. controller speed

161. Diagnostic is easy

162. Iostat –x 5 /dev/sd*4

163. Manageability is easy

165. ASM

166. Keep it simple

167. Reduce layers

168. Runs fast

170. 61 Transition/Section Marker