Oracle Flex ASM - What’s New and Best Practices by Jim Williams

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Oracle Flex Automatic Storage Management: What’s New and Best Practices
Jim Williams
ASM Product Manager
October 1, 2014

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
3

Program Agenda
4
Flex ASM
Best Practices - Avoiding 3:00 AM calls
Q & A
1
2
3

ASM History 101
•Provide an integrated volume manager and file system
•Stripe and mirror files across disks in a ASM Disk Group
•Automatic “Rebalance” after storage configuration changes
•Built on the Oracle instance architecture
•I/O operations DO NOT go through the ASM instance!
•Manage storage as a global cluster of shared Disk Groups
The Simple Idea
5

ASM History 101
The Simple Idea
6

ASM History 101
The Simple Idea
Disk1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7
Disk Group A Disk Group B
Database Database Database
File 1 File 2
File 3 File 4
ASM Cluster Pool of Storage
Oracle
RAC
Servers
ASM
Instance
ASM
Instance
ASM
Instance
ASM
Instance
ASM
Instance
Database
Instance
Database
Instance
Database
Instance
Database
Instance
Database
Instance
Shared Disk Groups
Wide File Striping
1-1 ASM to Server
7

Pre-12c ASM Architecture
ASM architecture utilized an ASM
instance on every server
– Database instances dependent on node-specific
ASM instances
– ASM overhead scaled with size of cluster
– Cluster reconfiguration events increased
with number of servers in a cluster
DB1 DB2 DB3
ASM
DB1 DB2 DB4
ASM
DB1 DB5 DB4
ASM
DB1 DB5 DB6
ASM
Server Server
Server Server
Cluster
8

Flex ASM Architecture
Eliminates requirement for an ASM instance on every cluster server
–Database instances connect to any ASM instance in the cluster
–Database instances survive loss of ASM instance
–Administrators specify the cardinality of ASM instances (default is 3)
–Clusterware ensures ASM cardinality is maintained
9

Flex ASM
•Increased maximum number of Disk Groups increased to 511
–Previous limit was 63
•Replicated physical metadata
–Improves reliability
–Virtual metadata has always been replicated with ASM mirroring
•Replace ASM Disk command
•Administrators can now specify a Failure Group repair time:
–Similar to existing disk repair time
–New disk group attribute - failgroup_repair_time
–Default setting is 24 hours
Other Flex Features
10

Flex ASM
•In previous versions, database instances used OS-authentication to connect to ASM
–This worked because ASM clients and servers were always on the same server
•With Oracle Database 12c, database instances and ASM servers can be on different servers
–Flex ASM uses password file authentication
–Password file is in an ASM Disk Group
–A default configuration is created when the ASM cluster is configured
•Databases can use a shared password file as well!
Remote Access
11

Program Agenda
12
Flex ASM
Avoiding 3:00 AM Calls
Q & A
1
2
3

Deploying Flex ASM
Do not accept the default – choose “Advanced Installation”
•Typical Installation
–Does not provide an option to use “Flex ASM”
•Advanced Option
–Recommended for all configurations
13

Deploying Flex ASM
“Advanced Installation” – Storage Options
•Four storage options are available:
1.Standard ASM
•Pre 12c ASM configuration mode
2.Oracle Flex ASM
•Recommended
3.ASM Client Cluster
•Ignore for now
4.Non-ASM managed storage
14

Green Field Deployment
Pre 12c
Flex ASM
Isolate Flex ASM deployment
from pre 12c databases
– Pre 12c databases require a local
ASM instance
– Dedicated Flex ASM cluster
DB DB DB
ASM
Server
DB DB DB
ASM
Server
DB DB DB
ASM
Server
DB DB DB
ASM
Server
DB DB DB
ASM
Server
DB DB DB
ASM
Server
DB DB DB
ASM
Server
DB DB DB
Server
DB DB DB
ASM
Server
DB DB DB
Server
ASM
Pre
12c
12c
15

Mixed-mode Deployment
Flex ASM cluster supporting pre-12c
databases
– Set cardinality to ALL
– ASM instance on every server
– Only 12c database instances can
reconnect to a surviving ASM instance
after a server failure
DB DB DB
ASM
Server
DB DB DB
Server
DB DB
ASM
Server
DB DB DB
Server
DB DB
ASM
Server
DB DB DB
Server
Pre
12c
12c
DB
DB
ASM
ASM
ASM
Mixed
16

Deploying Flex ASM
•Less than four nodes
–ASM instance on every node
•Four or more nodes
–Three ASM Instances or ALL if pre 12c databases
•SRVCTL command for:
•Checking ASM instance status
–srvctl status asm
•Setting cardinality
–srvctl modify asm –count ALL
•Starting, stopping, and relocating ASM instances
–srvctl start asm –n node-name
–srvctl stop asm –n node-name
How many ASM instances?
17

Deploying Flex ASM
•Use Automatic Memory Management (AMM) “MEMORY_TARGET”
–Defaults to 1 GB – adequate for most configurations
–Exadata uses a custom memory configuration
•Process Count “PROCESSES”
–For #_DBs < 10, PROCESSES = 50* #_DBs + 50
–For #_DBs >= 10, PROCESSES = 10* #_DBs + 450
–Oracle Exadata environment, the MAX(450 + 10* #_DBs, 1024)
•See ASM Administration Guide for details on setting
Memory and Process Count Considerations
18

ASM Diskgroup Configuration
•ASM_DISKSTRING: search path that ASM uses to discover Disk Groups
–ASM examines all the devices specified as possible candidates as an ASM Disk
•When the candidate list is excessively large, Disk Group discovery becomes unnecessarily long causing slow ASM response
•Default value often is sufficient “/dev/sd*”, but if there is a need to change it, don’t make it too inclusive e.g. “/dev/*”
Disk Group Discovery
19

•Best practice is DATA, FRA, GRID
–GRID DG for OCR, Voting File, SPFILE
–Use External Redundancy for most high-end storage arrays
•Separate Disk Groups for different storage tiers
–REDO Disk Group for flash storage
How many Disk Groups?
20

•How many Failure Groups
–Engineered systems automatically configure Failure Groups
–Choose by hardware boundary
–Use default of disk per Failure Group for small number of ASM Disks
–Failure Groups must be balanced – equal number of disks in each Failure Group
•Except Quorum Failure Groups need not be balanced
–Failure Groups for Extended Clusters need to be site-based and require Quorum Failure Group
How many Failure Groups?
21

•All disks must be same size for Normal and High Redundancy
–Similar performance characteristics
•Minimum disks: 4 times the number of paths for each Disk Group
–Normal Redundancy Disk Group with 2-way multipathing >= 8 disks
•Maximum: < 1000 disks in a Disk Group
–Long disk discovery times and frequent capacity additions with too many disks
How big and how many disks in a Diskgroup?
22

ASM Disk Groups with Mirroring
•With ASM Mirroring, the Partner Status Table (PST) is replicated
–3 copies for Normal Redundancy, and 5 copies for High Redundancy
–Each copy in separate Failure Groups, special case for <3 FG (Normal) and <5 FG (High)
•Two Failure Groups are problematic because quorum cannot be established!
–Create Quorum Failure Group(s) to satisfy PST quorum
–Quorum Failure Groups are exempt from the homogeneity requirements
–Quorum Failure need not be large – not used for data
Quorum Failure Groups
23

ASM Capacity Management
SELECT name, type, total_mb, free_mb, required_mirror_free_mb, usable_file_mb FROM V$ASM_DISKGROUP_STAT;
NAME TYPE TOTAL_MB FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
DATA NORMAL 51180 42204 10236 15984
24
Diskgroup Name
Redundancy
Total raw space in Diskgroup
Unused raw space in Diskgroup
Largest FG capacity
Logical space that can be allocated and still have ASM restore redundancy after Failure Group failure
•USABLE_FILE_MB = (FREE_MB - REQUIRED_MIRROR_FREE_MB) / 2 [normal redundancy]

•When creating a new Disk Group, 4 MB AUs provide minor benefit for data warehouse environments
–AU Size cannot be changed online and requires recreating the Disk Group
•Don’t invest effort in reconfiguring to accommodate 4 MB AU Disk Groups
–Variable extents introduced in 11.2 reduce benefit of larger AUs
–Storage array architecture affects I/O performance benefit of 4 MB AUs
AU Size
25

•Use External Redundancy if you have complete confidence in your storage
–A small percentage of customer use Normal Redundancy with high-end storage arrays
•Mirror data across storage arrays
•Need to provide Quorum Failure Group for Partner Status Table quorum
•Always use multi-pathing
–Provide ASM the MP O/S device path name
–Set MP timeout to less than clusterware heartbeat timeout (< 120 seconds)
–MOS note 294869.1
•Advance COMPATIBLE.ASM to 12.1
–Replicates physical metadata and makes External Redundancy Disk Groups more resilient to accidental corruption
Fault Tolerance
26

•Disk Groups are containers with internal “versioned” data structures
–Support backward compatibility
–Compatibility settings determines the availability of ASM features (see matrix in ASM admin guide)
–Disk Group compatible attribute can only be advanced
•BUT CANNOT BE REVERTED TO PREVIOUS VERSION
•The COMPATIBLE.ASM attribute must be >= value of other disk group compatibility attributes
–Advanced with SQL, ASMCA, ASMCMD
Compatibility Settings
27

•Two new features in 12.1 are support for LUNs larger than 2 TBs and 511 Disk Groups
–Large LUN support
•LUNs larger than 2TB require COMPATIBLE.ASM >= 12.1 and COMPATIBLE.RDBMS => 12.1
•ASM environments planning to use Disk Groups with large LUNs must only have post-12c databases
–511 Disk Group support: not controlled by any compatibility attribute
•Disk Group numbers are assigned at mount time (1..511) in the order they are discovered
•Pre 12c databases cannot access a Disk Group that is numbered greater than 63
•ASM environments planning to use more than 63 Disk Groups must only have post-12c databases
12.1 Feature Compatibility
28

ASM Diskgroup Reconfiguration
•Power determines the number of concurrent I/O operations
–Highly configuration dependent, but values greater than 32 often have declining benefit with respect to rebalance performance
–Can be dynamically changed to manage performance impact
•A “Power” setting can now be used for disk resync (disk online)
•Administrators can now replace a disk as a fast and efficient operation
–Disk Group reorganization is not required
–Replacement disk is populated with copies of ASM extents from online disks
ASM Power
29

Preventing Accidental Corruption
•The most common cause of corruption is accidental administrative action made to the wrong disk
–Over writing ASM Disk with a File System
–Assigning an ASM Disk to an LVM
•Employ operational procedures that established hard separation between ASM Disks and all others. A few ideas:
–Unique ASM Disk partitions e.g. /dev/sdu2
•First partition is small partition that aligns second partition to 1 MB boundary
•ASM is assigned second partition
•NEVER use second partitions elsewhere
•ASM Filter Driver – prevents over writing ASM Disks
30

Checking for Corruption
•Silent data corruption is a fact of life in today’s storage world
•The database checks for logical consistency when reading data
–If a logical corruption is detected then automatic recovery can be performed using the ASM mirror copies
–For seldom accessed data, over time all mirror copies of data could be corrupted
•With Oracle 12c ASM data can be proactively scrubbed:
–Scrubbing occurs automatically during rebalance operations
–Scrubbing of a Disk Groups, individual files, or individual disks
–ALTER DISKGROUP <NAME> SCRUB [POWER AUTO|LOW|HIGH|MAX];
Silent Data Corruption
31

Disk Failure
•Disk and Failure Group Repair Timers
–Disk Repair Time default value is 3.6 hours
–Failure Group Repair Time feature provides extra opportunity to avoid unnecessary rebalance – default value is 24 hours
–ALTER DISKGROUP <NAME> SET ATTRIBUTE 'DISK_REPAIR_TIME' = ‘12H';
–DROP AFTER clause of "OFFLINE DISK"|"OFFLINE DISKS IN FAILGROUP“ can be used to reset active timer
–Timer runs only while Disk Group mounted
–REPAIR_TIMER column in V$ASM_DISK reflects remaining time
–When timer expires and the disk is force dropped, you cannot use ONLINE DISK or REPLACE DISK
Repair Times when mirroring
32

Recovery After Disk Failure
33
•Normal Redundancy Disk Group with two Failure Groups.
•USABLE_FILE_MB is negative

34
•What happens when there is a disk failure?
•ASM takes Disk OFFLINE
•Timer counts for disk to be DROPPED from Disk Group

35
•After timer expires disk is dropped from Disk Group
•ASM begins rebalancing Disk Group

36
•Eventually, one disk will become full and allocations in Disk Group cannot be made
•ASM begins rebalancing Disk Group, but cannot continue because the Disk Group is full.

37
•Even adding a new disk cannot allow rebalance of Disk Group to continue because of partnering of individual extents.

38
•Possible solution:
–Drop the disk that is full (disk E)
–Then undrop disk after a brief period once some of the extents have been relocated and freed up space

Q & A

Oracle Flex ASM - What’s New and Best Practices by Jim Williams

Oracle Flex ASM - What’s New and Best Practices by Jim Williams

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (19)

Similar to Oracle Flex ASM - What’s New and Best Practices by Jim Williams

Similar to Oracle Flex ASM - What’s New and Best Practices by Jim Williams (20)

More from Markus Michalewicz

More from Markus Michalewicz (20)

Recently uploaded

Recently uploaded (20)

Oracle Flex ASM - What’s New and Best Practices by Jim Williams