Db2 for z/OS and FlashCopy - Practical use cases (June 2019 Edition)

Data & AI
Florence Dubois
IBM Db2 for z/OS Development
flodubois@uk.ibm.com
@floDB2z
Db2 for z/OS
and FlashCopy
Practical use cases
June 2019

Disclaimer/Trademarks
© Copyright IBM Corporation 2019. All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
THE INFORMATION CONTAINED IN THIS DOCUMENT HAS NOT BEEN SUBMITTED TO ANY FORMAL IBM TEST AND IS DISTRIBUTED AS
IS. THE USE OF THIS INFORMATION OR THE IMPLEMENTATION OF ANY OF THESE TECHNIQUES IS A CUSTOMER RESPONSIBILITY
AND DEPENDS ON THE CUSTOMER’S ABILITY TO EVALUATE AND INTEGRATE THEM INTO THE CUSTOMER’S OPERATIONAL
ENVIRONMENT. WHILE IBM MAY HAVE REVIEWED EACH ITEM FOR ACCURACY IN A SPECIFIC SITUATION, THERE IS NO GUARANTEE
THAT THE SAME OR SIMILAR RESULTS WILL BE OBTAINED ELSEWHERE. ANYONE ATTEMPTING TO ADAPT THESE TECHNIQUES TO
THEIR OWN ENVIRONMENTS DO SO AT THEIR OWN RISK.
ANY PERFORMANCE DATA CONTAINED IN THIS DOCUMENT WERE DETERMINED IN VARIOUS CONTROLLED LABORATORY
ENVIRONMENTS AND ARE FOR REFERENCE PURPOSES ONLY. CUSTOMERS SHOULD NOT ADAPT THESE PERFORMANCE NUMBERS TO
THEIR OWN ENVIRONMENTS AS SYSTEM PERFORMANCE STANDARDS. THE RESULTS THAT MAY BE OBTAINED IN OTHER OPERATING
ENVIRONMENTS MAY VARY SIGNIFICANTLY. USERS OF THIS DOCUMENT SHOULD VERIFY THE APPLICABLE DATA FOR THEIR
SPECIFIC ENVIRONMENT.
Trademarks
IBM, the IBM logo, ibm.com, Db2, FlashCopy, System z and z/OS are trademarks of International Business Machines Corp., registered in
many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml.
Data and AI on IBM Z / © 2019 IBM Corporation 2

• FlashCopy overview
• FlashCopy practical use cases
– Non-disruptive data consistency checks
– Optimisation of object-level recovery
– Consistent image copy backups
– Non-disruptive consistent UNLOAD with Db2 HPU for z/OS
– System-level backups for recovery or cloning
• Additional information
– Troubleshooting
– RACF FACILITY class profile
– LRSN-to-timestamp translation
– Options to check data consistency
Topics

FlashCopy Overview

• IBM FlashCopy (FC)
– Optional licensed feature available on IBM disk storage systems as well as non-IBM
• Creates an instant copy of a volume or a dataset at a specific point-in-time
– Initial logical FC phase (also called establish)
• After the logical FC, read and write activity are possible on both source and target devices
– Physical FC phase (also called background copy)
• Advantages
– Can copy huge volumes of data very quickly (seconds to a few minutes)
– Minimal disruption for the running applications
– Copy process is offloaded to the storage subsystem -> opportunity to lower host CPU
FlashCopy overview
Copy FC
NoCopy FC
Incremental FC Space-efficient FC
Consistency Group FC
Remote Pair FC
Fast Reverse Restore
Volume-level FC
Dataset-level FC

FlashCopy overview …
S
T
1
0
0
0
0
1
0
Establish phase – Logical FlashCopy
A FlashCopy relationship between source and target is established within
seconds
A bitmap tracks the relationship between source and target tracks
No writes are allowed to the volumes during this phase
S
T
1
0
0
0
0
1
0
Read and write activity are
possible on both the
source and target devices
Writes to the source will cause a ‘copy-on-write’ (a.k.a.
on-demand copy) if the track has not been copied to
the target – done asynchronously when the write is
destaged onto disk
Reads of tracks on the target that have not been
copied from the source will be redirected to the source
S
T
1
0
0
0
0
1
0
Background copy phase – Physical FlashCopy (optional)
Copy tracks from the source to the target
When all tracks have been copied to the target volume, the FC relationship
automatically ends unless the persistent FC option was specified

• Distributed interfaces
– Command-line interface (DS CLI)
– Web-based graphical user interface (DS Storage Manager GUI)
– Copy Services Manager (formerly known as IBM TotalStorage Productivity Center for Replication (TPC-R))
• Additional interfaces for z/OS users
– DFSMSdss
– TSO commands
– ICKDSF
– ANTRQST application
programming interface (API)
– IBM Copy Services Manager
z/OS FlashCopy Manager
Interface used by Db2 for z/OS utilities
Invoked either directly via the DFSMSdss COPY command
or via DFSMShsm Fast Replication commands (FRBACKUP/FRRECOV)
Whether dss COPY calls FC depends on the FASTREPLICATION keyword
PREFERRED (default) – use FC if possible, otherwise ‘traditional’ COPY
REQUIRED – if FC cannot be used, COPY fails
NONE – always use ‘traditional’ COPY

• Volume-level FC
– Source and target must be on the same physical disk subsystem (within any LSS)
– Require one target volume per source volume
– Target volume must be the same size or bigger than the source volume
– Target and source must have the same logical track format
– A source volume can have up to 12 target volumes (multiple relationship FC)
– At any point-in-time, a volume can be only a source or a target
– A volume can only be a target to one source at a time
• Dataset-level FC
– All types of z/OS data sets are supported (sequential, partitioned, VSAM data sets)
– No location restriction – can copy to same volume
– At any point-in-time, a data set can be only a source or a target
– Source and target data set must be fully contained on the same physical control unit
Restrictions lifted by
support for cascading
FlashCopy introduced
in DS8880 Release 8.3
NEW

• Full background COPY (default for ‘standard’ FC)
– All tracks from the source are copied to the target
– Option available for volume and dataset level FC
– 1:1 space requirements
– FC relationship ends as soon as all data is copied (unless it is defined as persistent)
– How long the actual physical copy takes depends on
• The amount of data being copied
• The number of background copy processes that are occurring
• The other activities on the ESS
– Background copy task runs at a lower priority than normal I/O, so as not to affect the normal application I/O
processing
FlashCopy overview … Volume-level FC
Dataset-level FC

• NOCOPY
– Suppresses the background copy
• Only copy source tracks before they are updated for the first time after FC (‘copy-on-write’)
– Option available for volume and dataset level FC
– 1:1 space requirements
• Still require one target volume per source volume for volume-level FC
– Relationship will last until it is explicitly withdrawn or until all data in the source volume has been modified
– Target volume becomes unusable if the relationship is withdrawn before all source tracks are copied
• Track data on the target device is unpredictable
– Typical usage: Transient copy e.g. backup to be dumped immediately to tape
FCNOCOPY option on the DFSMSdss COPY command
Volume-level FC
Dataset-level FC

• Incremental volume-level FC
– Option available only for volume-level FC
– ‘Refresh’ target volume
• Copy only source tracks that have changed since the last incremental FC was taken
– Advantages
• Minimize I/O impact of the background physical copy
• Considerably reduce elapsed time of the physical copy
– Incremental FC g Db2 Incremental IC
• Initial incremental FC invocation results in a full background copy
• Subsequent incremental FC result in ‘delta’ copy
• Resulting backup is always a ‘full’ backup
– 1:1 space requirements (one target volume per source volume)
– DS8870 Release 7.4 supports Multiple Incremental FC: up to 12 targets from a single source
(only one before)
– Relationship will last until it is explicitly withdrawn
FCINCREMENTAL on DFSMSdss COPY command
Volume-level FC

• IBM Remote Copy Services
– Metro Mirror – MM (a.k.a. PPRC)
– z/OS Global Mirror – zGM (a.k.a. XRC)
– Global Copy – GC (a.k.a. PPRC-XD)
– Global Mirror – GM
• Restrictions when combining copy services functions with FlashCopy
FlashCopy and IBM Remote Copy Services
Device
isMay
become
(1) If using Remote Pair FlashCopy
(2) Multiple incremental relationships
(3) If using Remote Pair FlashCopy for XRC (OA55068)
(4) Cascading FlashCopy
MM or GC
Primary
MM or GC
Secondary
zGM
Primary
zGM
Secondary
GM
Primary
GM
Secondary
FC
Source
FC
Target
Incrementa
l FC Source
Incrementa
l FC Target
FlashCopy
Source
Yes Yes Yes Yes Yes Yes Yes Yes (4) Yes (2) Yes (4)
FlashCopy
Target
Yes (1) No Yes (3) No No No Yes (4) No Yes (4) No
NEW

V8 • BACKUP SYSTEM
• RESTORE SYSTEM
• Dataset FC support for CHECK INDEX SHRLEVEL CHANGE
V9 • Incremental FC for BACKUP SYSTEM
• Dataset FC for RECOVER with system-level backup (SLB) as input
• Dataset FC for CHECK DATA SHRLEVEL CHANGE and CHECK LOB SHRLEVEL CHANGE
V10 • Dataset FC for COPY
• Dataset FC for inline copy in REORG TABLESPACE, REORG INDEX, REBUILD INDEX, LOAD
• FC image copies with consistency and no application outage (SHRLEVEL CHANGE)
• FCIC accepted as input to RECOVER, COPYTOCOPY, DSN1COPY, DSN1COMP, DSN1PRNT
FlashCopy and Db2 for z/OS utilities
• IBM Db2 for z/OS utilities make more and more use of FlashCopy

• Data consistency checking
– CHECK INDEX
• Consistency between table space and indexes
• Consistency between auxiliary table index and LOB table space
– CHECK DATA
• Violations of referential constraints
• Violations of table check constraints
• Consistency between a base table space and corresponding LOB/XML table spaces
– CHECK LOB
• Invalid LOB values or structural problems with the pages in a LOB table space
• CHECK utilities are critical diagnosis tools in the event of data corruption
– Identify objects that need repairing/recovering and assess the extent of the damage
CHECK utilities

• CHECK SHRLEVEL REFERENCE
– Utility executes directly on the data
– Allows concurrent read access to the data but no write access
• CHECK SHRLEVEL CHANGE
– Introduced in V8 for CHECK INDEX, V9 for CHECK DATA and CHECK LOB
– Utility executes on a shadow copy of the objects
– Allows concurrent read and write access to the data …
… AFTER creation of the shadow objects
CHECK utilities …
Without FlashCopy, the CHECK utilities are disruptive
even with SHRLEVEL CHANGE
Will you take an application outage,
or delay the discovery of the damage and the extent thereof?

• Requires only a small investment in temporary disk storage to create shadow objects
• Recommend ZPARM CHECK_FASTREPLICATION = REQUIRED (default since V10)
– Protection against misuse of CHECK SHRLEVEL CHANGE
– CHECK utility will fail if FlashCopy cannot be used
• Optional: ZPARM UTIL_TEMP_STORCLAS can be used to specify a storage class for the
shadow data sets (if blank, the shadow data sets are defined in the same storage class as the
production page set)
CHECK utilities …
COPY DATASET(FILTERDD(SYS00009)) DEBUG(FRMSG(DTL))-
REPLACEUNCONDITIONAL RENUNC( -
(*.*.*.*.I0001.*,*.*.*.*.J0001.*), -
(*.*.*.*.J0001.*,*.*.*.*.I0001.*)) -
STORCLAS(SCDB2FC ) BYPASSACS(**) -
FCNOCOPY FASTREPLICATION(REQUIRED ) -
FCTOPPRCPRIMARY(PRESMIRREQ ) -
SHARE CANCELERROR TOLERATE(ENQFAILURE)
Use case: Exploit data set FlashCopy to run the CHECK utilities non disruptively
COPY DATASET(FILTERDD(SYS00009)) DEBUG(FRMSG(DTL))-
REPLACEUNCONDITIONAL RENUNC( -
(*.*.*.*.I0001.*,*.*.*.*.J0001.*), -
(*.*.*.*.J0001.*,*.*.*.*.I0001.*)) -
FCNOCOPY FASTREPLICATION(REQUIRED ) -
FCTOPPRCPRIMARY(PRESMIRREQ ) -
SHARE CANCELERROR TOLERATE(ENQFAILURE)
ZPARM UTIL_TEMP_STORCLAS = SCDB2FCZPARM UTIL_TEMP_STORCLAS = blank

• Special considerations if you have DASD-based replication
– Asynchronous mirroring with Global Mirror
• Shadow data sets must be allocated on a pool of volumes OUTSIDE the mirror
– Use UTIL_TEMP_STORCLAS with a corresponding storage group
–Alternatively, use the DFSMSdss FlashCopy Batch Protection feature to force data sets to be allocated in a
specific storage group when FlashCopy is used
– Synchronous mirroring with Metro Mirror (PPRC)
• Option #1: Use a pool of volumes OUTSIDE of Metro Mirror (see above)
• Option #2: Set ZPARM FLASHCOPY_PPRC = REQUIRED (default V10)
– Protection against losing full duplex as a result of a FlashCopy operation
–FCTOPPRCPrimary(PresMirReq) will be added to the DFSMSdss COPY command
– If Remote Pair FlashCopy (aka preserve mirror) cannot be used, the utility will not use FlashCopy
→ if CHECK_FASTREPLICATION = REQUIRED, the CHECK utility will fail
CHECK utilities …
PROC &STORGRP
IF &ACSENVR2 = 'FLASHCPY' THEN SET &STORGRP = 'fcstrgrp';

• Special considerations if you have DASD-based replication …
– Asynchronous mirroring with z/OS Global Mirror (XRC)
• Option #1: Use a pool of volumes OUTSIDE of Metro Mirror (see previous slide)
• Option #2: Set new ZPARM FLASHCOPY_XRCP = YES
–z/OS APAR OA55068: New function REMOTE PAIR FLASHCOPY FOR XRC
» Requires DS8K Release 8.5 and higher
– FCTOXRCPrimary will be added to the DFSMSdss COPY command and will allow FlashCopy to XRC primary
volumes
CHECK utilities …
NEW

• Db2 restart is the key ingredient for disk-based DR solutions
– Normal Db2 warm restart
– Db2 restart times have a direct impact on the ability to meet the RTO
– In data sharing, GRECP/LPL is likely to be the largest contributor to Db2 restart time
• DR testing not always representative
– Executing a planned site swap is an important validation but it is not enough
– Need representative application workloads running concurrently to stress test Db2 group restart and
exercise a wide variety of conditions
• Validate procedures and maintain readiness
– Understand the system behaviour and prepare for potential residual clean up actions
» Data recovery/index rebuild might be required if GRECP/LPL conditions cannot be resolved
• Provide realistic service levels
• Flush out issues and get them fixed
Disaster Recovery (DR) testing

• Capture a snapshot copy of production (all volumes) at the DR site
– Vary the time of day when the copy is created (online/batch)
• Restart off that copy in an isolated environment using your DR restart procedures
• Develop procedures to proactively check for any sign of data inconsistency
– Broken pages, data vs. index mismatches, etc.
– A lot can be done without involving application teams (see Appendix)
• Capitalise on your investment
– Create a system-wide backup of last resort e.g. protection during resync of the mirror
– Create a ‘golden’ copy for forensic analysis immediately before any major restart/recovery action is
intended at the local site
Disaster Recovery (DR) testing …
P
T
SUse case: Exploit volume-level FlashCopy for realistic Db2 DR restart testing

• V10 introduced the option to create FlashCopy image copies (FCIC)
– FCIC created with COPY, REORG, LOAD, REBUILD INDEX, REORG INDEX
• VSAM data set(s) created via the DFSMSdss COPY function
– Separate VSAM data set for each partition or piece of the object that is being copied
• FCIC data sets are SMS-managed and always catalogued
–Allocation based on ACS routines
• FCIC data set names generated according to FCCOPYDDN template specification
– Storage Class, Data Class or Management Class are NOT honoured
– If no TEMPLATE is provided, name is generated using ZPARM FCCOPYDDN
» Default is HLQ.&DB..&SN..N&DSNUM..&UQ
FlashCopy image copies
TEMPLATE FCCOPY DSN DB2FCIC.D&DATE..T&HO.&MI..&DB..&SN..P&DSNUM.
COPY TABLESPACE DBNAME.TSNAME SHRLEVEL CHANGE
FLASHCOPY YES|CONSISTENT
FCCOPYDDN(FCCOPY)

• V10 introduced the option to create FlashCopy image copies (FCIC) …
– FCIC are always FULL image copies
• And you cannot create an incremental sequential image copies after an FCIC
– SYSCOPY records for FCIC
• ICTYPE=F
• ICBACKUP=FC (even if it was created with ‘slow’ copy instead of FlashCopy)
• STYPE=N (FCIC is not consistent) or T (FCIC is consistent)
– FCIC are accepted as input to RECOVER, COPYTOCOPY, DSN1COPY, DSN1PRNT
• Db2 UNLOAD utility cannot run directly from FCIC
–Use COPYTOCOPY to create a sequential copy that can then serve as input to UNLOAD
– Or use Db2 High Performance Unload for z/OS, which can unload directly from FCIC
FlashCopy image copies …

• Remote Pair FlashCopy (aka preserve mirror) → allows use of FlashCopy during COPY and RECOVER
• Set ZPARM FLASHCOPY_PPRC = REQUIRED (default V10) → FCTOPPRCP (PMR) will be added to the DFSMSdss
COPY command
– Protection against losing full duplexing as a result of a FlashCopy operation
• New Remote Pair FlashCopy for XRC (z/OS APAR OA55068 ) → allows use of FlashCopy during COPY and RECOVER
– Requires DS8K Release 8.5 and higher
• Set new ZPARM FLASHCOPY_XRCP = YES → FCTOXRCPrimary will be added to the DFSMSdss COPY command
– Asynchronous mirroring with Global Mirror
• FCIC must be allocated on a pool of volumes OUTSIDE the mirror
• Must find alternative solution to transfer object backups to DR site
• Cannot take advantage of FlashCopy during RECOVER
NEW

• Operational considerations
– All data sets have to be on SMS managed volumes
– Source and target data sets cannot span multiple physical DASD controller
– Db2 object and FCIC backup are allocated in same physical DASD controller
• To provide a recovery base for media failures, you can create one or more sequential image copies when you create
an FCIC
– The additional sequential image copies are created directly off the FCIC
TEMPLATE FCCOPY DSN DB2FCIC.D&DATE..T&HO.&MI..&DB..&SN..P&DSNUM.
TEMPLATE SYSCOPY DSN DB2IC&IC..D&DATE..T&HO.&MI..&DB..&SN..P&DSNUM.
COPY TABLESPACE DBNAME.TSNAME SHRLEVEL CHANGE
FLASHCOPY YES|CONSISTENT
FCCOPYDDN(FCCOPY)
COPYDDN(SYSCOPY)

• Operational considerations …
– If the FCIC data set is migrated to tape or VTS by DFSMShsm, RECOVER will not use it
• Error message DSNU1562I
• RECOVER will look for an alternate local copy first and then for an older image copy
• Work-around: Add a step to recall the FCIC data set (HRECALL) before the RECOVER executes
– Inline FCIC with REORG SHRLEVEL CHANGE
• Taking an inline FCIC can reduce the outage window for REORG
– Eliminates inline copy processing → no need to copy pages at each log iteration
• Uses option FASTREPLICATION(REQUIRED)
–If FC cannot be used:
» V11: REORG utility successfully completes but leaves the object in COPY-pending status
» V12: REORG utility fails and leaves the object in RW status

• FCIC have the potential to speed up object-level recovery
– If FlashCopy can be used during RECOVER, the LOGAPPLY phase can start as soon as the logical
FlashCopy completes → eliminates RESTORE time
• Large table spaces
– Take regular FCIC backups and keep at least one version on DASD at all times
– Backup frequency in line with update activity to avoid very long LOG APPLY phase
• Large NPIs
– Define/alter as COPY YES and include in FCIC cycle
– Prepare to run RECOVER INDEX in parallel with RECOVER TABLESPACE [PART]
• Instead of waiting for RECOVER TABLESPACE [PART] to complete for all parts and then running REBUILD INDEX
Use case: Selective use of FCIC for large objects to optimise recovery

• Applications sometimes require consistent image copy backups
• COPY SHRLEVEL REFERENCE is disruptive
– Allows concurrent read access to the data but no write access
• Option #1: Create consistent backups non-disruptively
• Option #2: Reduce application ‘outage’
Consistent image copies
LISTDEF APPL1SET INCLUDE TABLESPACE APPL1.*
TEMPLATE FCCOPY DSN DB2FCIC.D&DATE..T&HO.&MI..&DB..&SN..P&PA.
COPY LIST APPL1SET FLASHCOPY YES SHRLEVEL REFERENCE FCCOPYDDN(FCCOPY)
LISTDEF APPL1SET INCLUDE TABLESPACE APPL1.*
TEMPLATE FCCOPY DSN DB2FCIC.D&DATE..T&HO.&MI..&DB..&SN..P&PA.
COPY LIST APPL1SET FLASHCOPY CONSISTENT SHRLEVEL CHANGE FCCOPYDDN(FCCOPY)
Use case: Exploit data set-level FlashCopy to create consistent image copies

• Option #1: FLASHCOPY CONSISTENT SHRLEVEL CHANGE
– Consistency processing
• LOGAPPLY: reads the logs and applies all missing updates to the FCIC data sets → ensures that all activity is
reflected up to the point of consistency (PIT_RBA)
• LOGCSR: identifies uncommitted URs running at PIT_RBA
• LOGUNDO: backs out uncommitted URs from the FCIC data sets
Consistent image copies …
08:40:51
UR1 starts
08:49:27
COPY ends
08:41:39
END of logical FC
CONSISTENCY
PROCESSING
starts
08:41:47.55
CONSISTENCY
PROCESSING
ends
08:40:54
COPY
starts
08:41:47.88
SYSCOPY updated
UR1
UR2
08:41:00
UR2
starts
08:41:10
UR2
ends
START_RBA
Could be different
for each object
PIT_RBA
Same for
all objects
TIMESTAMP

• Option #1: FLASHCOPY CONSISTENT SHRLEVEL CHANGE …
– Automation considerations
• DSNU1540I: all objects have been successfully copied (logical completion)
• DSNU15401I: all FCIC data sets have been made consistent (end of LOGUNDO)
08:40:51
UR1 starts
08:49:27
COPY ends
08:41:39
END of logical FC
DSNU1540I
CONSISTENCY
PROCESSING
starts
08:41:47.55
DSNU15401I
CONSISTENCY
PROCESSING
ends
08:40:54
COPY
starts
08:41:47.88
SYSCOPY updated
UR1
UR2
08:41:00
UR2
starts
08:41:10
UR2
ends
START_RBA
Could be different
for each object
PIT_RBA
Same for
all objects
TIMESTAMP

• Option #2: FLASHCOPY YES SHRLEVEL REFERENCE
– COPY utility processing
• Writers are drained from the objects to be copied
• All changed pages in buffer pools and group buffer pools are externalised to DASD
• FlashCopy is invoked to copy the objects
• Optionally, additional sequential image copies can be created out of the FCIC
– Application ‘outage’ is minimised
• UTRO held only until all objects are successfully copied (logical completion)
• During the SEQCOPY phase, objects are in UTRW
– Point of consistency is established at a 'quiet' time (no inflight UR) and you have more control over it than
with FLASHCOPY CONSISTENT SHRLEVEL CHANGE

• IBM Db2 High Performance Unload for z/OS can operate directly from an FCIC
• Operational considerations for ‘basic’ Db2 UNLOAD utility
– Cannot do run the Db2 UNLOAD utility directly from an FCIC, only from a sequential image copy
– Sequential copies created out of an FCIC (as part of the COPY utility or with COPYTOCOPY) are always
created with SYSTEMPAGES NO
• Does not ensure that the dictionary and version system pages are copied at the beginning of the image copy data set
• Might cause the Db2 UNLOAD utility to fail
– Recommendations
• If your data is versioned (via immediate ALTERs), be sure to REORG your data before creating the FCIC
• If your data has been compressed by 'compression on insert', be sure to REORG your data before creating the FCIC
– REORG is required only once after the compression during insert occurred
Use case: Non-disruptive consistent UNLOAD with Db2 HPU for z/OS

• Db2 utility BACKUP SYSTEM
– System-level backup (SLB) using volume-level FlashCopy
– Fast and minimally-disruptive
• No need to suspend the logs or hold off write I/Os
– Backup and recovery are managed by Db2 and DFSMShsm
• SLBs can be on DASD and/or tapes
– SLB can be used for
• Recovery of the entire system to the time of the backup
• Recovery of the entire system to a PIT after backup (RESTORE SYSTEM utility)
• Recovery of individual Db2 table spaces and indexes (RECOVER utility)
• Quick cloning of an environment
– Test/QA, reporting system, forensic’ system to investigate data corruption
– Very simple from an operational perspective
– Requires careful planning to achieve (and maintain) Db2 data and logs separation
System-level backup

• Important planning considerations
– All volumes containing Db2 data sets must be SMS managed
• Including the Db2 catalog/directory, logs and BSDS, and all user data
– Db2 logs & BSDS must be separated from the rest of the Db2 data sets
• Own pool of volumes
• Own ICF user catalog
• Defined in a separate SMS storage group
– Special care should be taken to ensure that the ICF user catalogs stay synchronized with the data
• Must have separate ICF user catalogs for data and logs
–ICF user catalog(s) for data must reside in the data copy pool
– ICF user catalog for logs must reside in the log copy pool
• Do not share ICF user catalog with non-Db2 data
System-level backup …

• Sample SMS definitions
CopyPool
Name: DSN$P870$DB
Versions: 2
Storage Group Storage Group
Storage Group
Storage Group
Name: DSNCAT
CopyPool Backup Name: CPB1
Name: DSNDB
Type: Pool
Name: EX1
Type: Extend
Name: CPB1
CopyPool Backup Name: N/A
Type: CopyPool Backup
CopyPool
Name: DSN$P870$LG
Versions: 2
Storage Group
Name: DSNLOG
Type: PoolType: Pool
• Create new ICF user catalogs
• Create SMS copy pools
• Move/rename BSDSs
• Move/rename active log pairs
• Move/rename Db2 data sets
• Merge ICF user catalog aliases

• Db2 utility BACKUP SYSTEM …
– When BACKUP SYSTEM is issued, Db2
• Suspends 32K page writes if not using variable CI size (DSVCI=NO)
• Suspends data set creation, deletion, rename and extension operations
• Prevents data sets from pseudo-close
• Records the Recover Based Log Point (RBLP) in DBD01
– Starting point for RESTORE SYSTEM to apply log records
– Data sharing – Oldest system checkpoint across all members of group
– Recommend using time based system checkpointing to avoid trailing member
//BACKUP EXEC DSNUPROC,PARM='P870,BACKUP‘
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
BACKUP SYSTEM FULL | DATA ONLY

– During the backup …
• Invokes DFSMShsm to take FlashCopy of the database copy pool
– A FlashCopy relationship is established between source and target volumes
– The copy is considered successful when the relationship is established
• (Optional if BACKUP SYSTEM FULL) Invokes DFSMShsm to take FlashCopy of the log copy pool
• Updates BSDS with the system backup information
– In data sharing only the submitting member logs BSDS information
• Resumes the quiesced activities
• Ends the utility
– A physical copy continues in the background that will result in target volumes that look like the source volumes
when the relationship was established
• The FC relationship goes away when the background copy completes

– hsm VERSIONS parameter controls how many backup generations are kept on DASD
• If VERSIONS>0 → (default) |
• If VERSIONS=0 →  SLB on TAPE
– Having the backup on DASD is key to fast recovery
• For mass recovery following data corruption, the base for recovery may be 48-72h old
• Customers with very aggressive SLAs may need to keep more than one SLB on DASD
– Using incremental FC is key to limiting impact on the DASD subsystem
• Minimize I/O impact of the background physical copy
• Considerably reduce elapsed time of the physical copy
• DS8870 Release 7.4 supports Multiple Incremental FC: up to 12 targets from a single source
Copy FC
NoCopy FC
Incremental FC

– Support for backup and restore to/from tape
• Integrated tape management between Db2 and DFSMShsm
• Reduce disk space for maintaining multiple backups
• Retain backup for long term use
• Provide a means of recovery from media failure
• Can allow remote site recovery
– Keywords for BACKUP SYSTEM
• DUMP initiates dump processing to tape when the DASD backup is logically complete
• DUMPONLY dumps to tape an existing DASD backup
• FORCE indicates that the oldest copy can be overwritten even if dump to tape is still in progress
– Support for z/OS 2.2 Common Dump Queues (and Common Restore Queues)
• Improves overall throughput by distributing the activity across the sysplex instead of concentrating it on a single
LPAR

• Db2 utility RESTORE SYSTEM
– Recovery is for the entire system or data sharing group
– Very simple from an operational perspective
– Before running RESTORE SYSTEM:
• Establish the 'PITR' conditional restart record (DSNJU003)
– CRESTART CREATE SYSPITR=log-truncation-point (RBA/LRSN) or log-truncation-timestamp (GMT)
– You can specify a value of FFFFFFFFFFFF to recover to current
– For recovery to the time of the backup, use the DATA COMPLETE LRSN
• Purge the Db2 CF structures
• Start Db2 with a PITR CRCR
– Db2 system enters into System RECOVER-pending mode
» Implicitly apply DEFER ALL, FORWARD = NO (except for in-doubt URs) and Access(Maint)
–In that mode, all you can do is run the RESTORE SYSTEM utility
Volume-level FC

• Db2 utility RESTORE SYSTEM …
– Execute RESTORE SYSTEM utility
(1) Restore the database copy pool version that was taken by BACKUP SYSTEM prior to the specified PIT recovery point
(2) Db2 reads the DBD01 header page to retrieve the recovery base log point (RBLP) → starting point for the log apply
function
– Uses fast log apply (FLA) to recover objects in parallel
–Handles table spaces and index spaces CREATEs and DROPs
– For LOG NO events, associated object are marked RECP or RBDP
– Stop and start Db2 to remove it from ACCESS(MAINT) mode
– Recover all objects that are marked RECP, RBDP or in LPL
//RESTORE EXEC DSNUPROC,PARM='P870,RESTORE‘
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
RESTORE SYSTEM

• RECOVER using SLB as input
– Can recover a single object or list of objects using an SLB
• Set ZPARM SYSTEM_LEVEL_BACKUPS=YES to allow SLB to be considered for object recovery
• COPY YES indexes can be included in this recovery
• If FROMDUMP is specified, data sets are restored from tapes
– RECOVER to point in time or to current
– To allow recovery even when data sets have moved to a different set of volumes since the SLB:
• Use DFSMShsm option to capture ICF catalog information is turned ON (not the default)
‘CAPTURE CATALOG INFORMATION FOR DATA SET RECOVERY’ must be set to ‘R’ (REQUIRED) in the copy pool
definition
RECOVER hsm FRRECOV FASTREPLICATION(PREFERRED) dss COPY
Flash
Copy
Standard
I/O
Dataset-level FC

• Use Remote Pair FlashCopy (aka preserve mirror)
• FCTOPPRCPrimary is set at the Copy Pool level via the SMS definition panel
– FRBACKUP to PPRC Primary Vols allowed (NO|PN|PR)
–FRRECOV to PPRC Primary Vols allowed (NO|PN|PR)
– Default (NO): MM primary devices cannot be used as FlashCopy target volumes
• Use new Remote Pair FlashCopy for XRC (z/OS APAR OA55068 )
• FCTOXRCPrimary attribute is set at the Copy Pool level via the SMS definition panel
– FRBACKUP to XRC Primary Volumes allowed (NO|YES)
– FRRECOV to XRC Primary Volumes allowed (NO|YES)
– Default (NO): XRC primary devices cannot be used as FlashCopy target volumes
NEW

• Special considerations if you have DASD-based replication …
– Asynchronous mirroring with Global Mirror or 3-site Metro Global Mirror
• SLB and FCIC must be allocated on a pool of volumes OUTSIDE the Global Mirror
• Must find alternative solution to transfer object backups to DR site
• RECOVER using SLB or FCIC will always use ‘standard’ I/O
• RESTORE SYSTEM cannot be done unless Global Mirror is disabled
S1
S2
P1
P2
GM
T1
MM
Tape mirroring
MM

• If you are interested in system-level backup,
strongly recommend the following publication:
– Tightly Integrated Db2 Backup, Recovery and Cloning for SAP Environments
– Very detailed information and many examples
– Relevant to all types of environments, not just SAP…
• Also recommend tooling to support the solution
(e.g. IBM Db2 Recovery Expert for z/OS,
IBM Db2 Cloning Tool for z/OS)
– Initial setup
– Backup validation at every execution
– Creation of Db2 image copy backups from SLB
– Simplified cloning procedures
• Dataset renames, VTOC and VVDS updates, Db2 catalog updates, etc.

• FlashCopy opens up a whole new world of possibilities
– Increased availability for Db2 utilities
– Local or remote backups
– Enhanced testing capability
– Quick cloning of the environment
Summary
PREPARATION INVESTMENT
CHECK utilities
DR testing
FlashCopy image copies
Consistent image copies
Consistent HPU unload
System-level backup 45

Cascading FlashCopy
NEW

Before Cascading FlashCopy

Cascading FlashCopy
48
A B C
Forward Cascaded Relationship
Existing target of a relationship
becomes the source of a new
relationship
T0 T1
A B C
Backward Cascaded Relationship
Existing source of a relationship
becomes the target of a new
relationship
T1 T0
Data and AI on IBM Z / © 2019 IBM Corporation

Cascading FlashCopy ...
49
A B
Cyclic Cascaded Relationship
When a relationship is created from an
existing target to its existing source, without
using restore or Fast Reverse Restore, a new
relationship is created, resulting in a cyclic
cascading relationship
A B
Reversed Relationship
When using restore or Fast Reverse Restore,
the existing relationship is Reversed
Establish
Reverse
Establish
Establish
Data and AI on IBM Z / © 2019 IBM Corporation

• Hardware prerequisites
– DS8880 Release 8.3
– For minimum microcode levels and current maintenance required to exploit this new function, refer to the
Technical Information section of announcement letter ENUS117-061
• http://www-01.ibm.com/common/ssi
• Software prerequisites
– z/OS 2.1 and 2.2 enablement APAR OA52391
• OA52391 (AOM) OA52394 (SDM) PI78748 (ICKDSF) OA52393 (DSS) OA52395 (HSM)
– z/OS 2.3
• OA53543 (AOM) OA53536 (SDM) OA53577 (DSS) OA53652 (HSM)

• Design limitations
– Can only be a target to one source at a time
– Cannot create a forward cascade from an Incremental target volume until the background
copy has completed
– Cannot create a forward cascade if Incremental relationship was created using different
Incremental types (i.e. Type-1 and Type-2)
– Cyclic relations
•Cannot create Incremental cyclic relationships
•* You can however create full volume (non-incremental) and data set cyclic relations *
* NOTE: All Cyclic relationships were temporarily restricted by microcode DS8880 in Release 8.3 but was
lifted in Release 3.1 (made generally available in November 2017)

• 1a - BACKUP SYSTEM and RECOVER
A
B
Source
Cluster
SLB
BACKUP SYSTEM with Full
FC
RECOVER single object(s)
Old: - FlashCopy, if background copy is done
- Slow copy, if background copy is still running
New: FlashCopy
1
2

• 1b - BACKUP SYSTEM and RECOVER
A
B
BACKUP SYSTEM with
Incremental FC
Old: Slow copy
New: - FlashCopy, if background copy is done
- Slow copy, if background copy is still running
(supposed to be fast as only changed tracks are copied)
1
2
Source
Cluster
SLB

• 1c - BACKUP SYSTEM and RECOVER
A
B
BACKUP SYSTEM with NOCOPY
(Version 0)
Old/New: HSM does not support data set level recovery from disk
backup. Data sets need to be recovered from the dump version.
1
2
Source
Cluster
SLB

• 2a - BACKUP SYSTEM and RESTORE SYSTEM
A
B
BACKUP SYSTEM
RESTORE SYSTEM
Old: - FlashCopy, if background copy is done
- FlashCopy, if background copy is still running and Fast Reverse Restore is allowed → but then SLB is lost
- Failure, if background copy is still running and Fast Reverse Restore cannot be used
(e.g. Metro Mirror and Remote Pair FlashCopy not incompatible with Fast Reverse Restore)
New: - FlashCopy even if background copy is still running → if FRR is not used, SLB is not lost
1
2
Source
Cluster
SLB
RECOVER objects in RECP
Unchanged: Slow copy,
if background copy is
still running
3
C

• 2b - BACKUP SYSTEM and RESTORE SYSTEM
A
C
BACKUP SYSTEM
with Multiple Incremental FC
RESTORE SYSTEM
Old: A-C cannot be reversed until persistent incremental FC relationship A-B has been manually deleted
→ B will no longer be available for subsequent recovery. Dump copies will still be available if they were not
deleted.
New: A-C can be reversed without having to remove any existing FC relationship from the original source
2
3
Source
Cluster
SLB
B
SLB
1

• RESTORE SYSTEM and inline copies
A
B
REORG with FC inline copies
RESTORE SYSTEM
Old: Failure
New: FlashCopy
2
1
Source
Cluster
SLB
C
While the background process triggered by RESTORE SYSTEM is running, the day-to-
day business has already started, e.g. REORGs with FlashCopy inline copies

• 3a - BACKUP SYSTEM and cloning process
A
B
Cloning process
Old: Slow copy
New: FlashCopy
2
Source
Cluster
SLB
C
While physical copy for BACKUP SYSTEM is running in the background, the clusters can be copied to fill a new Db2
system or refresh an existing one (clone scenario).
Advantage: The SLB is available for recovery and cloning, and copy with FC can run during the cloning process.
BACKUP SYSTEM with Full FC
1
RECOVER single
object(s)
3
Old: Slow copy
New: FlashCopy
COPY single object(s)
FLASHCOPY YES
D
4
Old: Slow copy
New: FlashCopy
Target
Cluster

• 3b - BACKUP SYSTEM and cloning process
A
B
Cloning process
Old:
After background copy is complete, must
withdraw incremental copy relationship
New:
Cloning process can be done as soon as
background copy is complete, without having
to break the incremental relationship
2
Source
Cluster
SLB
C
BACKUP SYSTEM with
incremental FC
1
Target
Cluster

Appendix

• Common reasons why FlashCopy cannot be used
– Missing RACF authorisations (or equivalent)
– Source or target dataset not SMS managed
– Source or target data set allocated across multiple physical DASD controllers
– Target volume in a Metro Mirror (PPRC) relationship and preserve mirror is not used
– Target volume in a z/OS Global Mirror (XRC) relationship and preserve mirror for XRC is not used
– Target volume in an asynchronous mirroring relationship, other than XRC (e.g. Global Mirror)
– Target volume in a Metro Global Mirror relationship
– Target volume already in a FlashCopy relationship and Cascading FlashCopy support is not used
• RECOVER while physical background copy is still ongoing
• RECOVER when Incremental FlashCopy is used for System Level Backup
Troubleshooting

• Determining why FlashCopy cannot be used
– What can you do if you think that a Db2 utility using DFSMSdss should have used FlashCopy but did not
(no ADR806I message in the utility output)?
– Extract the DFSMSdss COPY command from the Db2 utility output
– Execute the same command with ADRDSSU
– Add the DEBUG keyword to collect diagnostic information
• DEBUG(FRMSG(MINIMAL|SUMMARIZED|DETAILED))
–DFSMSdss will write an informational message indicating why FlashCopy was not used
» The message level controls the type and amount of information DFSMSdss provides
• DEBUG(SMSMSG) – new with z/OS 2.1
– DFSMSdss will display ACS WRITE statements to the job output
» Can be useful if you suspect the problem lies in the ACS routines
Troubleshooting …
ADR806I (001)-T0MI (03), DATA SET DB2FLSH.DSNDBC.RCDB0001.RCTS0001.I0001.A001
COPIED USING A FAST REPLICATION FUNCTION

• RACF FACILITY class profiles
– When the RACF FACILITY class is active and one of the profiles listed below is defined, the user submitting
the Db2 utility must have READ access authority to use the indicated command or keyword
RACF authorisations
Keyword or function Profile name Used by…
FRBACKUP STGADMIN.ARC.FB.cpname BACKUP SYSTEM
FRRECOV STGADMIN.ARC.FR.cpname RESTORE SYSTEM, RECOVER from SLB
BYPASSACS with COPY STGADMIN.ADR.COPY.BYPASSACS CHECK if using UTIL_TEMP_STORCLAS
FCCGFREEZE with COPY STGADMIN.ADR.COPY.FCFREEZE Volume-level backup outside of Db2's control
FCFASTREVERSERESTORE with COPY STGADMIN.ADR.COPY.FCFRR Volume-level recover outside of Db2's control
FCTOPPRCPRIMARY with COPY STGADMIN.ADR.COPY.FCTOPPRCP CHECK, COPY(*), RECOVER with RPFC for PPRC
FCTOXRCPRIMARY with COPY STGADMIN.ADR.COPY.FCTOXRCP CHECK, COPY(*), RECOVER with RPFC for XRC
FlashCopy with COPY STGADMIN.ADR.COPY.FLASHCPY CHECK, COPY(*), RECOVER
ICH408I USER(JSMITH ) GROUP(DB2ADM ) NAME(SMITH, JOHN )
STGADMIN.ADR.COPY.FCTOPPRCP CL(FACILITY)
INSUFFICIENT ACCESS AUTHORITY
ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )
(*) Also applies to Db2 utilities that create inline FCIC

• RACF data set access authorisation levels
– When a Db2 utility calls the DFSMSdss COPY command, the COPY command is executed with the authority
of the utility batch address space → the user ID that invokes the Db2 utility must have at the minimum the
following authorities:
• READ access level on the source dataset
– COPY FLASHCOPY YES|CONSISTENT: Db2 VSAM tablespace/indexspace dataset(*)
– CHECK SHRLEVEL SHANGE: Db2 VSAM tablespace/indexspace dataset
– RECOVER from FCIC: FCIC dataset
• ALTER access level on the target dataset
– COPY FLASHCOPY YES|CONSISTENT : FCIC dataset (*)
– CHECK SHRLEVEL SHANGE: shadow dataset
– RECOVER from FCIC: Db2 VSAM tablespace/indexspace dataset
– For COPY FLASHCOPY CONSISTENT, the user ID running the DBM1 address space must have at least
UPDATE access level on the FCIC dataset (to allow changes during LOGAPPLY phase)
RACF authorisations …
(*) Also applies to Db2 utilities that create inline FCIC

• In V10 or V11 CM → 6-byte LRSN format in SYSCOPY
• In V11 NFM → 10-byte LRSN format in SYSCOPY
LRSN-to-timestamp translation
SELECT HEX(START_RBA) FROM SYSIBM.SYSCOPY WHERE ... ;
---------+---------+---------+---------+---------+------
00D04A01271BDB5A0800
SET CURRENT APPLICATION COMPATIBILITY = 'V11R1';
SELECT TIMESTAMP(BX'00D04A01271BDB5A0800'!!BX'000000000000')
FROM SYSIBM.SYSDUMMY1 ;
------+---------+---------+---------+---------+---------+---
2016-02-12-10.04.07.961013
SELECT HEX(START_RBA) FROM SYSIBM.SYSCOPY WHERE ... ;
---------+---------+---------+---------+---------+------
D04A01271BDB
SELECT TIMESTAMP(X'D04A01271BDB'!!X'0000')
FROM SYSIBM.SYSDUMMY1 ;
------+---------+---------+---------+---------+---------+---
2016-02-12-10.04.07.961008

• Invalid LOB values or structural problems with the pages in a LOB table space
– CHECK LOB TABLESPACE dbname.tsname for each LOB table space
• Consistency between table spaces and indexes
– CHECK INDEX (ALL) TABLESPACE dbname.tsname
• Db2 RI and table check constraint violations + consistency between base table and
corresponding LOB/XML table spaces
– CHECK DATA TABLESPACE dbname.tsname part SCOPE ALL
• Options to check each page of a table space for consistency
– DSN1COPY with CHECK option
– SELECT * from tbname
– Basic RUNSTATS to touch each row
Options to check data consistency

• SQL queries against Db2 catalog from migration job DSNTESQ
– Should always return zero rows
• REPAIR DBD TEST or DIAGNOSE
• For Db2 catalog/directory objects with LOB columns:
– CHECK LOB on the LOB table space
– CHECK INDEX on the AUX index
– CHECK DATA on the base table space SCOPE AUXONLY AUXERROR REPORT
Options to check data consistency …

Db2 for z/OS and FlashCopy - Practical use cases (June 2019 Edition)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Db2 for z/OS and FlashCopy - Practical use cases (June 2019 Edition)

Similar to Db2 for z/OS and FlashCopy - Practical use cases (June 2019 Edition) (19)

Recently uploaded

Recently uploaded (20)

Db2 for z/OS and FlashCopy - Practical use cases (June 2019 Edition)