Oracle ASM 11g - The Evolution

1. Oracle ASM 11g The Evolution Alex Gorbachev RMOUG Training Days 2010 Denver, 18-Feb-10

3. Why Companies Trust Pythian • Recognized Leader: • Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and SQL Server • Work with over 150 multinational companies such as Forbes.com, Fox Interactive media, and MDS Inc. to help manage their complex IT deployments • Expertise: • One of the world’s largest concentrations of dedicated, full-time DBA expertise. • Global Reach & Scalability: • 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response 3 © 2009/2010 Pythian

5. Pre-ASM - simple ﬁle system on JBOD • Resilience? • multiplexing • redo + controlfiles Server • hardware mirroring /u01 /u02 • Scaleability? • Manual datafile layout • separate indexes and tables? • Hardware striping • Manageability? Disk Disk • poor • resize? move? 5 © 2009/2010 Pythian

7. Pre-ASM - raw devices (cluster) Server 1 • One ﬁle => disk or partition • Why raw devices? /dev/sdb /dev/sdc • remove overhead of FS • Later CIO, direct IO, etc Disk Disk • Clusters - shared storage /dev/sdb /dev/sdc Server 2 7 © 2009/2010 Pythian

10. Pre-ASM - cluster VM and FS • Cluster Volume Manager (CVM) • Synchronizes volume metadata • Still lots of manageability overhead • 1 volume = one file • Cluster File System (CFS) • On top of CVM • Synchronize file metadata & concurrent IO • Network Attached Storage (NAS) • FS and VM on storage box • ZFS opens the whole new dimension in manageability 9 © 2009/2010 Pythian

13. Pre-ASM storage issues • Too complex • Too simple • 3rd party vendors • Doesn’t scale • Not reliable • Additional licensing cost • Not ﬂexible • Many groups involved • Storage engineers • SA’s • DBA’s • Loosing layout visibility 10 © 2009/2010 Pythian

14. ASM introduction • Simplify • Fewer vendors involved • Fewer layers • Move many operations under DBA umbrella • oh well, Storage Administrator role • Cost efficient • No need to license volume manager • Cheap storage boxes are good for ASM • Manageable 11 © 2009/2010 Pythian

15. ASM principles • Organizing Oracle files on physical disks • Mirroring • reliability & failure tolerance • Striping • performance & scaleability • Rebalancing • manageability 12 © 2009/2010 Pythian

26. ASM striping 10g • Allocation unit (AU) 1MB • Coarse striping - whole AU • Fine striping - 128K inside AU • One extent = 1 AU • Random striping -> equal distribution • No auto-magic re-layout based on performance ﬁle1 ﬁle2 13 © 2009/2010 Pythian

33. ASM rebalancing • Add / remove disks • Migrate to another storage • Extend / shrink • asm_power_limit - rebalancing power/overhead • No mirroring (external redundancy) • planned maintenance 14 © 2009/2010 Pythian

47. ASM instance • Special kind of Oracle instance • Clustered in RAC • ASM disks and diskgroups • ASM extent map • Rebalancing operations • Must be started before database instance 17 © 2009/2010 Pythian

48. ASM 10g issues • ASM upgrade requires full RAC outage • Fixed 1MB allocation unit • Overhead with multi-TB databases • Recovery from failure is inefficient • Extended clusters read IO is inefficient • Limited FS functionality and tools • No snapshot-like functionality 18 © 2009/2010 Pythian

49. ASM 11gR1 - diskgroup compatibility • Default SQL> select name, compatibility, database_compatibility from v$asm_diskgroup; NAME COMPATIBIL DATABASE_C ---- ---------- ---------- DG1 10.1.0.0.0 10.1.0.0.0 • New features SQL> ALTER DISKGROUP dg1 SET ATTRIBUTE 'compatible.asm'='11.1'; Diskgroup altered. SQL> ALTER DISKGROUP dg1 SET ATTRIBUTE 'compatible.rdbms'='11.1'; Diskgroup altered. SQL> select name, compatibility, database_compatibility from v$asm_diskgroup; NAME COMPATIBIL DATABASE_C ---- ---------- ---------- DG1 11.1.0.0.0 11.1.0.0.0 19 © 2009/2010 Pythian

50. ASM 11g - rolling updates for RAC ALTER SYSTEM START ROLLING MIGRATION TO 11.2.0.0.0; • Limited services from ASM • Normal database operation • No diskgroup configuration changes • Mount / unmount diskgroups ALTER SYSTEM STOP ROLLING MIGRATION; © 2009/2010 Pythian

51. ASM 11gR1 - variable AU size • Allocation Unit - 1/2/4/.../64 MB • 10g - 1MB (_asm_ausize) • Data sits closer together and can be read in bigger chunks • Can improve sequential IO • Reduces striping over SAN striped LUN’s • Configuring Oracle ASM hidden parameters for EVA8000, HP Knowledge Brief • Reduces SGA for metadata for large files • 11g - can be done per diskgroup © 2009/2010 Pythian

52. ASM 11gR1 - variable extent size • 10g • 1 extent = 1 AU • 1 TB = 1+ mil. extents • 11g • 1-20,000 extents • 1MB AU: 1 extent = 1 AU • 1 TB = 53,572 extents • 20,001 - 40,000 • 64MB AU: 1 extent = 8 AU’s • 1 TB = 16,384 extents • 40,001 - ... • 100 TB = 66,788 extents 1 extent = 64 AU’s 22 © 2009/2010 Pythian

53. ASM 11gR1 - recovery from failures • Fast Mirror Resync • OFFLINE disks • disk_repair_time attribute • ASM extent change tracking • Suitable for transient failures and maintenance • Fast Rebalancing in restricted mount • Rebalancing => many lock/unlock extent map • Restrict mode rebalancing => no locks • Restricted mounted DG => service outage © 2009/2010 Pythian

54. ASM 11gR1 - Preferred Mirror Read • For extended clusters • Storage mirrored across 2 or 3 datacenters • 10g • Read is first done on primary extent • 11g • Preferred read failure groups (to local disks) • asm_preferred_read_failure_groups in init.ora © 2009/2010 Pythian

55. ASM Dynamic Volume Manager (ADVM) • Clustervolume manager • Can place any filesystem on top of an ASM Volume • Oracle ACFS or third-party like ext3 • Unix/Linux & Windows • Dynamically resizable volumes • Leveraging all standard ASM features (like mirroring) • Extent size 64MB - allocation unit • Number of columns - 1 to 8 configurable • Striping across columns • Dirty Region Logging (DRL) 25 © 2009/2010 Pythian

56. ADVM on Linux Loadable kernel modules [root@epa01 init.d]# ./acfsload start -s [root@epa01 init.d]# lsmod | grep ora oracleacfs 787460 0 oracleadvm 177792 0 oracleoks 226656 2 oracleacfs,oracleadvm [root@epa01 init.d]# 26 © 2009/2010 Pythian

57. ADVM: Creating a volume ASMCMD & SQL Interfaces (+GUI Tools: ASMCA & OEM) ASMCMD> volcreate -G DG_A1 -s 50M DV_A2 Volume Name: DV_A2 Volume Device: /dev/asm/dv_a2-354 State: ENABLED Size (MB): 256 Resize Unit (MB): 256 Redundancy: UNPROT Stripe Columns: 4 Stripe Width (K): 128 Usage: Mountpath: SQL> ALTER DISKGROUP DG_A1 ADD VOLUME DV_A2 SIZE 50M; 27 © 2009/2010 Pythian

58. ADVM in the OS OS devices [oradb@epa01 ~]$ ls -l /dev/asm/ total 0 brwxrwx--- 1 root dba 252, 12289 Nov 16 02:15 dv_a1-24 brwxrwx--- 1 root dba 252, 111617 Nov 16 02:15 dv_main_db-218 [oradb@epa01 ~]$ Dynamically resizable ASMCMD> volresize -G DG_A1 -s 400M DV_A2 28 © 2009/2010 Pythian

59. ADVM: creating ﬁlesystems # /sbin/mkfs -t ext4 /dev/asm/dv_a3-461 # /sbin/mkfs -t acfs -b 4k /dev/asm/dv_a3-461 mkfs.acfs: version = 11.2.0.1.0.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/dv_a3-461 mkfs.acfs: volume size = 268435456 mkfs.acfs: Format complete. # mount -t acfs /dev/asm/dv_a3-461 /a03 # df –h /a03 Filesystem Size Used Avail Use% Mounted on /dev/asm/dv_a3-461 256M 37M 220M 15% /a03 29 © 2009/2010 Pythian

60. Oracle ASM Cluster Filesystem (ACFS) • General purpose cluster filesystem • Journaling, variable extent-based • Windows/Linux/Unix • Oracle homes, dumps, traces, export files and etc - *anything* • Native ASM files (i.e. no datafiles, Diagram from Oracle ASM 10gR2 Documentation controlfiles and etc) unsupported • Direct path IO or cached • ASM tools • Dynamic resize SQL/ASMCMD/ASMCA/OEM • Mount registry (like /etc/fstab) • Standard OS commands mkfs / mount / fsck / fuser / cp / chmod / chown / ... 30 © 2009/2010 Pythian

61. ACFS Snapshots • Filesystem technology • No snapshots for ASM diskgroups and ADVM volumes • Read-only • Space-efficient • Initially sparse with copy-on-write technology • Up to 63 snapshots per filesystem • Access via .ACFS/snaps hidden directory • CLI acfsutil snap and OEM support 31 © 2009/2010 Pythian

62. ASM 11gR2 - “small” enhancements • ASM supports placement of OCR and voting disks • Intelligent Data Placement (hot/cold -> outer/inner) • Disk Group rename • renamedg utility • ASMCMD enhancements • New ASMCA tool, OEM support • Access control for ASM files (owner/group/others) • 4K sector-size for diskgroups 32 © 2009/2010 Pythian

63. ASM Overhead? • ASM is not in the IO code path for Oracle Database • ASM only operates extent maps and disk headers • Database -> thousands IO’s per second vs ASM -> few IO’s per minute • Additional ASM instance - memory and processes but not much • Database instance keeps ASM extent maps in SGA • Mirroring generates additional write IO • ADVM - another layer • ACFS - yet another layer 33 © 2009/2010 Pythian

66. What’s still missing? • Read-write ACFS snapshots / clones • ASM diskgroup level snapshots for database files • ADVM snapshots (do we need them?) • ACFS or ADVM compression • Diskgroup mirroring conversion • Split mirror functionality • De-duplication capabilities • Still need multipathing • Proper throttling of rebalancing operations • Is ADVM and ACFS are production ready? 35 © 2009/2010 Pythian

67. What’s next? • Book draw • Leave me your cards or email+phone Q&A Email me - gorbachev@pythian.com Read my blog - http://www.pythian.com Follow me on Twitter - @AlexGorbachev Join Pythian fan club on Facebook & LinkedIn 36 © 2009/2010 Pythian