A DBA’s guide to using TSA

A DBA’s guide to using TSA
Frederik Engelen
RealDolmen
Session Code: C03
16 November 2011 | Platform: LUW

Click to edit Master title style
Session Goal
• Quick overview of HADR functionality and advantages
• Description of Tivoli System Automation for Multiplatforms
(TSAMP) software and integration with DB2
• Operational tasks and lessons learned
2

Why HADR?
• Failover solution available from DB2 v8.2
• Design choices:
• Ultra-fast failover capability
• Negligible impact on performance
• Configurable degree of consistency
• Low administrative costs
• Avoid service interruption for fixpack installs and changes
requiring instance restart
• Transparent failover and failback for applications
• Spread over different geographic locations
• Built in clustering software in DB2 9.5
• Cheap
3

Basic Principle HADR
• Database scope
• Allows flexible configurations
• Primary database
• Processes transactions
• Send logs to standby
• Standby database
• Applies transactions recieved from primary
• Available for read-only workload (DB2 v9.7 FP1)
• In case of primary failure, standby database can take over
primary role
4
Primary Standby
Log shipping
Primary connection

HADR Synchronisation Modes
• Configurable degree of consistency
• SYNC, NEARSYNC, ASYNC, SUPERASYNC (NEW)
5
New logs
send() recieve()
Logs
ASYNC NEARSYNC
SYNC
TCP/IP
SUPER
ASYNC

Ready to implement?
• Network layout
• High speed network
• Private or public HADR communication?
• Reserve HADR communication port
• Use identical OS & database version and hardware family
• For automation with TSA, make sure DB2 > v9.5 FP5 or v9.7 FP1
• Use identical file system layout
• Keep clock synchronised
• Recommended:
• Same amount of memory & hardware
6

Implementing
• Configure primary database and backup
• Restore on standby server
• Adjust db cfg on standby database
• Start HADR
• Configure Automatic Client Reroute
7

Configure primary database
• Database configuration
db2 update db cfg for <DBNAME> using LOGARCHMETH1 DISK:/share/logarchs
db2 update db cfg for <DBNAME> using INDEXREC RESTART
db2 update db cfg for <DBNAME> using LOGINDEXBUILD ON
db2 update db cfg for <DBNAME> using HADR_TIMEOUT 30
db2 update db cfg for <DBNAME> using HADR_PEER_WINDOW 120
db2 update db cfg for <DBNAME> using HADR_LOCAL_HOST <PRIMARY_HOST>
db2 update db cfg for <DBNAME> using HADR_LOCAL_SVC <PRIMARY_HADR_PORT>
db2 update db cfg for <DBNAME> using HADR_REMOTE_HOST <STANDBY_HOST>
db2 update db cfg for <DBNAME> using HADR_REMOTE_SVC <STANDBY_HADR_PORT>
db2 update db cfg for <DBNAME> using HADR_REMOTE_INST <STANDBY_INST>
db2 update db cfg for <DBNAME> using HADR_SYNCMODE NEARSYNC
• Take a backup
8

Configure standby server
• Restore backup
• Keep name
• Database configuration
db2 update db cfg for <DBNAME> using HADR_LOCAL_HOST <STANDBY_HOST>
db2 update db cfg for <DBNAME> using HADR_LOCAL_SVC <STANDBY_HADR_PORT>
db2 update db cfg for <DBNAME> using HADR_REMOTE_HOST <PRIMARY_HOST>
db2 update db cfg for <DBNAME> using HADR_REMOTE_SVC <PRIMARY_HADR_PORT>
db2 update db cfg for <DBNAME> using HADR_REMOTE_INST <PRIMARY_INST>
9

Start HADR
• Standby
db2 update hadr on db <DBNAME> as standby
• Replay local log files
• Primary
db2 start hadr on db <DBNAME> as primary
• Replay primary archived logs
• Replay primary active logs
• Replay in-memory log buffer
-> Peer state
10
HADR start
Local Catchup
Remote Catch-
up Pending
Catch-up
Pending
Peer
Disconnected
Peer
Connectionlost
Connect
Disconnected
Reconnected
Peerwindowexpires

HADR result
• db2pd –hadr –db SAMPLE
Database Partition 0 -- Database SAMPLE -- Active -- Up 0 days 00:16:25 -- Date
08/21/2011 08:28:39
HADR Information:
Role State SyncMode HeartBeatsMissed LogGapRunAvg (bytes)
Primary Peer Nearsync 0 0
ConnectStatus ConnectTime Timeout
Connected Sun Aug 21 08:13:29 2011 (1313928809) 120
PeerWindowEnd PeerWindow
Sun Aug 21 08:30:39 2011 (1313929839) 120
LocalHost LocalService
batman 60000
RemoteHost RemoteService RemoteInstance
robin 60000 db2insts
PrimaryFile PrimaryPg PrimaryLSN
S0000005.LOG 0 0x00000000032C8010
StandByFile StandByPg StandByLSN
S0000005.LOG 0 0x00000000032C8010
11

Basic Operations
• Takeover
db2 takeover hadr on db <dbname> [by force [peer window
only]]
• Stop database
db2 deactivate db <dbname> && db2stop
 “stop hadr“ will turn db to standalone db
• Actually, that’s about it…
12

Automatic Client Reroute
• DB2 client reroutes connection to alternative server in
case of communication error
db2 update alternate server for <dbname> using hostname <host>
port <svcename>
• On connection failure:
SQL30108N A connection failed but has been reestablished
• HADR is not required
• JDBC URL
jdbc:db2://<dbhost>:<port>/<db>:clientRerouteAlternateServerNam
e=<althost>;clientRerouteAlternatePortNumber=<altport>;
• Configure on client and server
13

Things to keep in mind
• What gets replicated? Everything that’s logged.
• What not?
• DB CFG
• History file
• Not Logged Initially
• LOB columns that are not logged (>1G)
• Common operations
• LOAD: COPY YES
• REORG: consider reorg online or deactivate standby to avoid
blocking primary
• Tuning
• HADR simulator
14

Archival Logs
• Only primary server archives logs
• Store on shared file system
• Faster catch-up process
• Avoid manually copying logs when restore on standby
• Attention when using AUTO_DEL_REC_OBJ
15

Blocknonlogged
• Introduced in DB2 v9.5 FP4 / v9.7
• Blocks non-logged operations
• Not logged initially
• Not logged colums (BLOB and CLOB > 1G)
• Simple measure to protect from human error or an
unsuspecting application
16

Monitoring
• db2pd -hadr
• Database snapshot
• Health Monitor
• db2diag.log – check for messages from d2hadrp and
db2hadrs
db2insts@robin:~> db2diag -g "eduname:=db2hadr" -fmt "%eduname @msg
@changeevent“
db2hadrs Initiating HADR takeover request, forcing applications.
db2hadrs Info: Standby has initiated a takeover.
db2hadrs Info: Standby switching roles to primary.
db2hadrs HADR role set to Primary (was Standby)
db2hadrp HADR state set to P-Peer (was S-Peer)
db2hadrp Info: Standby has completed takeover (now primary).
db2hadrp Info: Standby Finished.
db2hadrp Info: Primary Started.
db2hadrp Info: HADR Socket send buffer size, SO_SNDBUF: 16384 bytes
db2hadrp Info: HADR Socket receive buffer size, SO_RCVBUF: 87380 bytes
17

Where does TSA fit in?
• TSA provides automatic monitoring and availability
management of “resources”
• TSA helps you when one of the cluster components fails
• But…TSA will also be there when you are
• Performing maintenance activities
• Troubleshooting
-> Basic knowledge of TSA and how it interacts with DB2 is
essential to succesfully perform your tasks as a DBA
18

Goal
• Components and terminology
• TSA domain configuration
• Operations
• Monitoring & troubleshooting
19

RSCT/TSA Terminology
• Cluster domain
• Node
• Resource
• Resource group
• Quorum
• Tiebreaker
• Equivalency
• Let’s see that in more detail…
20

Domain/Node
• Group of server nodes hosting resources configured for
high availability with RSCT
• Each node runs RSCT subsystems/daemons
• preprpnode, lsrpnode, addrpnode, rmrpnode,
stoprpnode, startrpnode
• mkrpdomain, lsrpdomain, rmrpdomain,
startrpdomain, stoprpdomain
21

Domain – RSCT components
# lssrc -a
Subsystem Group PID Status
IBM.ConfigRM rsct_rm 3610 active
ctcas rsct 3891 active
cthats cthats 5457 active
cthags cthags 5505 active
ctrmc rsct 5625 active
IBM.GblResRM rsct_rm 5792 active
IBM.RecoveryRM rsct_rm 5793 active
IBM.StorageRM rsct_rm 5794 active
IBM.TestRM rsct_rm 5795 active
22

Domain – RSCT components
• CTHATS: Topology Services
• Node connectivity (heartbeat)
• Messaging
• CTHAGS: Groups Services
• Synchronised configuration
• Messagging
• CTRMC: Resource Monitoring & Control
• Backbone of RSCT
• Monitoring & control infrastructure across cluster
• CTCAS: Cluster Authentication Services
• IBM.ConfigRM: Configuration Resource Manager
• Management application
• Create and administer a domain
23

Resource
• Any piece of hard- or software (configuration) that
provides services
• HADR role, DB2 instance, virtual IP, file system, tiebreaker,…
• Instance of Resource Class
• Most common IBM.Application (start/stop/monitor scriptsà)
• Provided by resource managers
• GlbResRM, TestRM, StorageRM, RecoveryRM
• Fixed/floating/concurrent
• Not directly controllable
• lsrsrc, rmrsrc, chrsrc, resetrsrc,
lsrsrcdef
24

IBM.Application class
Example of a DB2 HADR Resource
# lsrsrc –s “Name = ‘db2_db2instp_db2insts_SAMPLE-rs’ ” -Ab IBM.Application
resource 1:
Name = "db2_db2instp_db2insts_SAMPLE-rs"
ResourceType = 0 (Constituent)
AggregateResource = "0x2028 0xffff 0x5e5ca445 0xb9d31ef8 0x9239e701
0x23ba7500"
StartCommand = "/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh
db2instp db2insts SAMPLE"
StopCommand = "/usr/sbin/rsct/sapolicies/db2/hadrV97_stop.ksh
MonitorCommand = "/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh
MonitorCommandPeriod = 21
MonitorCommandTimeout = 29
StartCommandTimeout = 330
StopCommandTimeout = 140
ActivePeerDomain = "gotham"
NodeNameList = {"robin"}
OpState = 2 (Offline on this node)
25

resource 2:
0x23ba7500"
NodeNameList = {"batman"}
OpState = 1 (Online on this node)
26

resource 3:
ResourceType = 1 (Aggregate)
AggregateResource = "0x3fff 0xffff 0x00000000 0x00000000 0x00000000
0x00000000"
NodeNameList = {"batman","robin"}
OpState = 1 (Online globally)
27

Example of a DB2 Instance Resource
# lsrsrc –s “Name = ‘db2_db2instp_batman_0-rs’” –Ab IBM.Application
resource 1:
Name = "db2_db2instp_batman_0-rs"
AggregateResource = "0x2028 0xffff 0x5e5ca445 0xb9d31ef8 0x9239e6fa
0x97f134b0"
StartCommand = "/usr/sbin/rsct/sapolicies/db2/db2V97_start.ksh
db2instp 0"
StopCommand = "/usr/sbin/rsct/sapolicies/db2/db2V97_stop.ksh
db2instp 0"
MonitorCommand = "/usr/sbin/rsct/sapolicies/db2/db2V97_monitor.ksh
db2instp 0"
28

Example of a DB2 Instance Resource
# lsrsrc –s “Name = ‘db2_db2instp_batman_0-rs’” –Ab IBM.Application
resource 2:
Name = "db2_db2instp_batman_0-rs"
0x00000000"
StartCommand = "/usr/sbin/rsct/sapolicies/db2/db2V97_start.ksh
db2instp 0"
StopCommand = "/usr/sbin/rsct/sapolicies/db2/db2V97_stop.ksh
db2instp 0"
MonitorCommand = "/usr/sbin/rsct/sapolicies/db2/db2V97_monitor.ksh
db2instp 0"
29

IBM.ServiceIP class
Example of a HADR Virtual IP
# lsrsrc -Ab IBM.ServiceIP
resource 1:
Name = "db2ip_192_168_13_132-rs"
0x5d11b368"
IPAddress = "192.168.13.132"
NetMask = "255.255.255.0"
ProtectionMode = 1
NetPrefix = 0
NodeNameList = {"robin"}
OpState = 2 (Offline on this node)
ConfigChanged = 0
ChangedAttributes = {}
30

IBM.ServiceIP class
resource 2:
Name = "db2ip_192_168_13_132-rs"
0x5d11b368"
IPAddress = "192.168.13.132"
NetMask = "255.255.255.0"
ProtectionMode = 1
NetPrefix = 0
ConfigChanged = 0
31

IBM.ServiceIP class
resource 3:
Name = "db2ip_192_168_13_132-rs"
0x00000000"
IPAddress = "192.168.13.132"
NetMask = "255.255.255.0"
ProtectionMode = 1
NetPrefix = 0
NodeNameList = {"batman","robin"}
ConfigChanged = 0
32

Resource Managers
• IBM.GlbResRM
• Provides the IBM.Application and IBM.ServiceIP resource class
• IBM.RecoveryRM
• Brain of TSA, decision engine
• Gathers monitoring info and decides on action to take
• IBM.StorageRM
• Montoring and control of storage resources
• IBM.TestRM
• Provides IBM.Test resource class
• Simulates automation scenarios
• RM logfiles: /var/ct/${domain}/log/mc/IBM.${daemon}RM/
33

Resource Managers
# lssrc -ls IBM.RecoveryRM
Subsystem : IBM.RecoveryRM
Cluster Name : gotham
Node Number : 2
Daemon start time : 09/04/11 06:55:01
Daemon State:
My Node Name : robin
Master Node Name : batman (node number = 1)
Our IVN : 3.2.1.1
Our AVN : 3.2.1.1
Total Node Count : 2
Joined Member Count : 2
Config Quorum Count : 2
Startup Quorum Count : 1
Operational Quorum State: HAS_QUORUM
In Config Quorum : TRUE
In Config State : TRUE
In Jeopardy : FALSE
34

Equivalency
• Set of resources that provide the same functionality
• Main resource classes:
• IBM.PeerNode
• IBM.NetworkInterface
• lsequ, mkequ, rmequ, chequ
35

Equivalency example
HADR nodes
# lsequ -s "Name='db2_db2instp_db2insts_SAMPLE-rg_group-equ'"
Equivalency 1:
Name = db2_db2instp_db2insts_SAMPLE-rg_group-equ
MemberClass = IBM.PeerNode
Resource:Node[Membership] = {batman:batman,robin:robin}
SelectString = ""
SelectFromPolicy = ANY
MinimumNecessary = 1
Subscription = {}
Color = 0
ActivePeerDomain = gotham
ConfigValidity =
36

Equivalency example
Virtual IP
# lsequ -s "Name='db2_public_network_0'"
Equivalency 1:
Name = db2_public_network_0
MemberClass = IBM.NetworkInterface
Resource:Node[Membership] = {eth0:batman,eth0:robin}
SelectString = ""
SelectFromPolicy = ANY
MinimumNecessary = 1
Subscription = {}
Color = 0
ConfigValidity =
37

Resource Group
• Groups resources (surprise…)
• All included resources share the same Nominal State
• MemberLocation attribute controls placement on nodes
• Controllable operational state
• Resides on node/equivalency
38

Resource Group Operations
• Move(cancel)
• rgreq -o move(cancel) <resource_group_name>
• Change status
• chrg -o offline <resource_group_name>
• Un(lock)
• rgreq -o move(cancel) <resource_group_name>
• lsrg, mkrg, chrg, rmrg
• addrgmbr, chrgmbr, rmrgmbr, rgmbrreq
• rgreq, lsrgreq, …
39

Resource Group example
HADR DB
# lsrg -s "Name='db2_db2instp_db2insts_SAMPLE-rg'"
Resource Group 1:
Name = db2_db2instp_db2insts_SAMPLE-rg
MemberLocation = Collocated
Priority = 0
AllowedNode = db2_db2instp_db2insts_SAMPLE-rg_group-equ
NominalState = Online
OpState = Online
TopGroup = db2_db2instp_db2insts_SAMPLE-rg
TopGroupNominalState = Online
40

Resource Group example
HADR DB
# lsrg -m -g db2_db2instp_db2insts_SAMPLE-rg
Member Resource 1:
Class:Resource:Node[ManagedResource] =
IBM.Application:db2_db2instp_db2insts_SAMPLE-rs
Mandatory = True
MemberOf = db2_db2instp_db2insts_SAMPLE-rg
SelectFromPolicy = ORDERED
OpState = Online
Member Resource 2:
Class:Resource:Node[ManagedResource] = IBM.ServiceIP:db2ip_192_168_13_132-rs
Mandatory = True
MemberOf = db2_db2instp_db2insts_SAMPLE-rg
SelectFromPolicy = ORDERED
ConfigValidity =
OpState = Online
41

Relationship
• Configured between resources, resource groups and
equivalencies (1-N)
• Mainly DependsOn type
• StartAfter, ForcedDownBy, …
• lsrel, mkrel, chrel, rmrel
42

Relationship example
DB2 instance -> public network
# lsrel -s "Name = 'db2_db2instp_batman_0-rs_DependsOn_db2_public_network_0-
rel'"
Name = db2_db2instp_batman_0-
rs_DependsOn_db2_public_network_0-rel
Class:Resource:Node[Source] = IBM.Application:db2_db2instp_batman_0-rs
Class:Resource:Node[Target] = {IBM.Equivalency:db2_public_network_0}
Relationship = DependsOn
Conditional = NoCondition
ConfigValidity =
ResourceGroup[Source] = db2_db2instp_batman_0-rg
43

Quorum
• Quorum: the number of nodes in a cluster that are
required to control the resources (majority)
• Example log entry
ConfigRM[3610]: (Recorded using libct_ffdc.a cv 2):::Error ID: :::Reference
ID: :::Template ID: 0:::Details File: :::Location:
RSCT,PeerDomain.C,1.99.22.19,18004 :::CONFIGRM_PENDINGQUORUM_ER
The operational quorum state of the active peer domain has changed to
PENDING_QUORUM. This state usually indicates that exactly half of the nodes
that are defined in the peer domain are online. In this state cluster
resources cannot be recovered although none will be stopped explicitly.
• Majority with only 2 HADR nodes? Tiebreaker
44

Tiebreaker
• Determine quorum when half of the nodes survives
• Network Tiebreaker
• Only supported type out of the box
• A pingable IP address (default gateway is a good choice)
• Disk Tiebreaker
• Possible to add manually
• SCSI Reservation, faster
• More secure
• Without quorum (+ tiebreaker), no failover will occur !
45

Communication TSA/DB2
• DB2->TSA
• API implemented by cluster manager
• CLUSTER_MGR = TSA
• TSA->DB2
• HA Policy scripts in /usr/sbin/rsct/sapolicies
# ls /usr/sbin/rsct/sapolicies/db2/
db2V97_monitor|start|stop.ksh
hadrV97_monitor|start|stop.ksh
mountV97_monitor|start|stop.ksh
nfsserverctrl-server
forceAllApps
rovingV97_failover.ksh
46

Policy script example
batman:~ # head -n 17 /usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh
#!/bin/ksh -p
#-----------------------------------------------------------------------
# (C) COPYRIGHT International Business Machines Corp. 2001-2009
# All Rights Reserved
#
# US Government Users Restricted Rights - Use, duplication or
# disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
#
# VERSION: db2_v97fp4 s110330
# INPUT: hadrV97_monitor.ksh db2instp db2insts hadrdb [verbose]
#
# OUTPUT: 1 if online, 2 if offline, 0 if not known
#
# NOTES: Can only be used in the context of a TSA environment configured
# with the db2 HA Integrated Configuration Utility (db2haicu)
#
#-----------------------------------------------------------------------
47

Configuring for HA
• Install TSA + DB2 (lots of fixes -> latest fixpack)
• Who would’ve guessed…
• Configure HADR (Peer State)
• We already know how to do that
• Disable db2fm
• Prepare nodes
• #preprpnode batman robin
• Run DB2 High Availability Instance Configuration Utility
(db2haicu)
• Interactively
• XML
48

DB2HAICU XML File example
<DB2Cluster xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="db2ha.xsd" clusterManagerName="TSA"
version="1.0">
<ClusterDomain domainName=“gotham">
<Quorum quorumDeviceProtocol="network“ quorumDeviceName="192.168.13.1"/>
<PhysicalNetwork physicalNetworkName="db2_public_network_0"
physicalNetworkProtocol="ip">
<Interface interfaceName="eth0" clusterNodeName="batman">
<IPAddress baseAddress="192.169.13.130“
subnetMask="255.255.255.0"
networkName="db2_public_network_0"/>
</Interface>
<Interface interfaceName="eth0" clusterNodeName=“robin">
<IPAddress baseAddress="192.169.13.131"
subnetMask="255.255.255.0"
</Interface>
</PhysicalNetwork>
49

DB2HAICU XML File example (continued)
<PhysicalNetwork physicalNetworkName="db2_private_network_0“
physicalNetworkProtocol="ip">
<Interface interfaceName="eth1" clusterNodeName="batman">
subnetMask="255.255.255.0"
networkName="db2_private_network_0"/>
</Interface>
<Interface interfaceName="eth1" clusterNodeName="robin">
subnetMask="255.255.255.0”
networkName="db2_private_network_0"/>
</Interface>
</PhysicalNetwork>
<ClusterNode clusterNodeName="batman"/>
<ClusterNode clusterNodeName="robin“/>
</ClusterDomain>
50

DB2HAICU XML File example (continued)
<FailoverPolicy>
<HADRFailover></HADRFailover>
</FailoverPolicy>
<DB2PartitionSet>
<DB2Partition dbpartitionnum="0" instanceName="db2instp"/>
</DB2PartitionSet>
<HADRDBSet>
<HADRDB databaseName="SAMPLE" localInstance="db2instp"
remoteInstance="db2insts" localHost ="batman“ remoteHost="robin" />
<VirtualIPAddress baseAddress="192.168.13.132“
subnetMask="255.255.255.0"
</HADRDBSet>
</DB2Cluster>
51

DB2HAICU
• Start on standby instance
• Don’t mix primary/standby databases in the same instance
$db2haicu –f ha_cfg.xml
 Repeat for primary instance
 Done
52

DB2HAICU standby output
db2haicu determined the current DB2 database manager instance is db2insts. The cluster
configuration that follows will apply to this instance.
Creating domain gotham in the cluster ...
Creating domain gotham in the cluster was successful.
Configuring quorum device for domain gotham ...
Configuring quorum device for domain gotham was successful.
Adding network interface card eth0 on cluster node batman to the network db2_public_network_0 ...
Adding network interface card eth0 on cluster node batman to the network db2_public_network_0 was
successful.
Adding network interface card eth0 on cluster node robin to the network db2_public_network_0 ...
Adding network interface card eth0 on cluster node robin to the network db2_public_network_0 was
successful.
Adding network interface card eth1 on cluster node batman to the network db2_private_network_0 ...
Adding network interface card eth1 on cluster node batman to the network db2_private_network_0 was
successful.
Adding network interface card eth1 on cluster node robin to the network db2_private_network_0 ...
Adding network interface card eth1 on cluster node robin to the network db2_private_network_0 was
successful.
Adding DB2 database partition 0 to the cluster ...
Adding DB2 database partition 0 to the cluster was successful.
The HADR database SAMPLE has been determined to be valid for high availability. However, the
database cannot be added to the cluster from this node because db2haicu detected this node is the
standby for the HADR database SAMPLE. Run db2haicu on the primary for the HADR database SAMPLE to
configure the database for automated failover.
All cluster configurations have been completed successfully. db2haicu exiting ...
53

DB2HAICU primary output
db2haicu determined the current DB2 database manager instance is db2instp. The cluster
configuration that follows will apply to this instance.
Configuring quorum device for domain gotham ...
Configuring quorum device for domain gotham was successful.
The network adapter eth0 on node batman is already defined in network db2_public_network_0 and
cannot be added to another network until it is removed from its current network.
The network adapter eth0 on node robin is already defined in network db2_public_network_0 and
The network adapter eth1 on node batman is already defined in network db2_private_network_0 and
The network adapter eth1 on node robin is already defined in network db2_private_network_0 and
Adding DB2 database partition 0 to the cluster ...
Adding DB2 database partition 0 to the cluster was successful.
Adding HADR database SAMPLE to the domain ...
Adding HADR database SAMPLE to the domain was successful.
All cluster configurations have been completed successfully. db2haicu exiting ...
54

Cluster domain overview
55
PeerNode: BATMAN PeerNode: ROBIN
Equivalency: db2_db2instp_db2insts_SAMPLE-rg_group-equ
Resource Group: db2_db2instp_db2insts_SAMPLE-rg
Floating Resource:
db2_db2instp_db2insts_SAMPLE-rs
HADR Primary HADR Secondary
Floating Resource:
db2ip_192_168_13_132-rs
Virtual IP Virtual IP
db2_db2instp_batman_0-
rg_group-equ
db2_db2instp_
batman-rg
db2_db2insts_robin_0-
rg_group-equ
db2_db2insts_
robin-rg
Resource:
db2_db2instp_
batman-rs
Resource:
db2_db2insts_
robin-rs
Private Network
Equivalency: db2_public_network_0 eth0eth0
Equivalency: db2_private_network_0 eth1eth1
Public Network
Relation:
db2_db2instp_batman_0-rs_
DependsOn_
db2_public_network_0-rel
Relation:
db2_db2insts_robin_0-rs_
DependsOn_
db2_public_network_0-rel

DB2HAICU result
~# lssam
Online IBM.ResourceGroup:db2_db2instp_batman_0-rg Nominal=Online
'- Online IBM.Application:db2_db2instp_batman_0-rs
'- Online IBM.Application:db2_db2instp_batman_0-rs:batman
Online IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Nominal=Online
|- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs
|- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:batman
'- Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:robin
'- Online IBM.ServiceIP:db2ip_192_168_13_132-rs
|- Online IBM.ServiceIP:db2ip_192_168_13_132-rs:batman
'- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs:robin
Online IBM.ResourceGroup:db2_db2insts_robin_0-rg Nominal=Online
'- Online IBM.Application:db2_db2insts_robin_0-rs
'- Online IBM.Application:db2_db2insts_robin_0-rs:robin
Online IBM.Equivalency:db2_db2instp_batman_0-rg_group-equ
'- Online IBM.PeerNode:batman:batman
Online IBM.Equivalency:db2_db2instp_db2insts_SAMPLE-rg_group-equ
|- Online IBM.PeerNode:batman:batman
'- Online IBM.PeerNode:robin:robin
Online IBM.Equivalency:db2_db2insts_robin_0-rg_group-equ
'- Online IBM.PeerNode:robin:robin
…
56

DB2HAICU result (continued)
…
Online IBM.Equivalency:db2_private_network_0
|- Online IBM.NetworkInterface:eth1:batman
'- Online IBM.NetworkInterface:eth1:robin
Online IBM.Equivalency:db2_public_network_0
|- Online IBM.NetworkInterface:eth0:batman
'- Online IBM.NetworkInterface:eth0:robin
~> db2 get dbm cfg | grep CLUSTER_MGR
Cluster manager (CLUSTER_MGR) = TSA
57

Operations
• Use regular commands whenever possible:
• Takeover, db2stop, db2start,…
• Check the result on the cluster domain
• TSA commands still work
• rgreq –o move <HADR RG>
• chrg -o offline <DB2 Instance RG>
• Takeover example
• $ db2 takeover hadr on db <DBNAME>
58

Takeover example
~# lssam
…
…
59

Takeover example (continued)
~# lssam
…
Online IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Request=Lock Nominal=Online
|- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs Control=SuspendedPropagated
'- Online IBM.ServiceIP:db2ip_192_168_13_132-rs Control=SuspendedPropagated
…
60

~# lssam
…
Online IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Request=Move Nominal=Online
…
61

~# lssam
…
Pending Offline IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Request=Move
Nominal=Online
|- Pending Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs
|- Pending Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:batman
'- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs Control=StartInhibited
|- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs:batman
…
62

~# lssam
…
Pending Offline IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Request=Move
Binding=Sacrificial Nominal=Online
|- Pending Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs Binding=Sacrificial
|- Pending Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:batman
'- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs Binding=Sacrificial
…
63

~# lssam
…
Pending Online IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Request=Move
Nominal=Online
|- Pending Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs
|- Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:batman
'- Pending Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:robin
'- Online IBM.ServiceIP:db2ip_192_168_13_132-rs:robin
…
64

~# lssam
…
|- Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:batman
'- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:robin
'- Online IBM.ServiceIP:db2ip_192_168_13_132-rs:robin
…
65

Regain control
• Disable TSA
• $ db2haicu –disable
• Remove DB2 from TSA config
• $ db2haicu –delete
• Change TSAMP to manual/automatic mode
• # samctrl –m T/F
• Remove domain
• # rmrpdomain –f <domainname>
66

Monitoring & troubleshooting
• Main commands:
• db2pd –ha
• lssam
• samdiag
• HA Policy scripts (verbose)
• Syslog (/var/log/messages)
• Output from automation scripts
• Error messages from TSA
67

Monitoring & troubleshooting
• TSAMP Daemon logs
• /var/ct/${domain}/log/ctha[t|g]s
• /var/ct/${domain}/log/mc/IBM.${daemon}RM/
• Format using
• getsadata –traces
• rpttr
• Example: putting verbose start information in RecoveryRM trace:
chrsrc -s "Name = 'db2_db2instp_db2insts_SAMPLE-rs'"
IBM.Application
StartCommand="/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh
db2instp db2insts SAMPLE verbose"
68

Syslog example
/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh[9874]: Entering : db2instp db2insts SAMPLE
/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh[10427]: manual mode, or no state flags, use db2pd:
hdrRole:Primary, manual:-1, usepd:0, flagFile:0
/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh[10449]: Returning 1 : db2instp db2insts SAMPLE
/usr/sbin/rsct/sapolicies/db2/db2V97_monitor.ksh[10549]: Returning 2 (db2insts, 0)
db2V97_start.ksh[10554]: Entered /usr/sbin/rsct/sapolicies/db2/db2V97_start.ksh, db2insts, 0
db2V97_start.ksh[10582]: Able to cd to /home/db2insts/sqllib :
/usr/sbin/rsct/sapolicies/db2/db2V97_start.ksh, db2insts, 0
db2V97_start.ksh[10631]: 1 partitions total: /usr/sbin/rsct/sapolicies/db2/db2V97_start.ksh,
db2insts, 0
hadrV97_monitor.ksh[10727]: /usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh is offline, hadr
monitor must return offline
/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh[10766]: Entering : db2instp db2insts SAMPLE
/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh[10779]: PROGNAME: hadrV97_start.ksh,PROGPATH:
/usr/sbin/rsct/sapolicies/db2,DB2HADRINSTANCE1: db2instp, DB2HADRINSTANCE2: db2in
sts, dbname: SAMPLE, DB2HADRDBNAME: SAMPLE, VERBOSE: noverbose
/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh[10786]: candidate_P_instance: db2instp,
candidate_S_instance: db2insts, localShorthost: robin
69

Syslog example
/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh[10793]: db2insts,db2instp,,
/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh[10794]: candidate_P_instance: db2insts,
candidate_S_instance: db2instp, localShorthost: robin, nn1: , nn1s:
/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh[10796]: su - db2insts -c db2gcf -t 3600 -u -i
db2insts -i db2instp -h SAMPLE -L ...
db2V97_start.ksh[13107]: Returning 0 from /usr/sbin/rsct/sapolicies/db2/db2V97_start.ksh ( db2insts,
0)
/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh[13611]: manual mode, or no state flags, use db2pd:
hdrRole:Primary, manual:-1, usepd:0, flagFile:0
70

IBM.GblResRM trace_summary example
08:16:29 AM.757887 T(4124900208) _GBD Monitor detect OpState change for resource
Name=db2_db2instp_batman_0-rs OldOpState=1 NewOpState=2 Handle=0x6028 0xffff 0xb506ae54
0xb9d31ef8 0x12404f85 0x3ba
d7160
08:16:30 AM.181871 T(4151659376) _GBD Bringing application resource online:
Name=db2_db2instp_batman_0-rs Handle=0x6028 0xffff 0xb506ae54 0xb9d31ef8 0x12404f85
0x3bad7160
Name=db2_db2instp_db2insts_SAMPLE-rs OldOpState=1 NewOpState=2 Handle=0x6028 0xffff
0xb506ae54 0xb9d31ef8 0x12404f8
b 0x7e81efb0
08:16:37 AM.219586 T(4151659376) _GBD Bringing application resource online:
Name=db2_db2instp_db2insts_SAMPLE-rs Handle=0x6028 0xffff 0xb506ae54 0xb9d31ef8 0x12404f8b
0x7e81efb0
Name=db2_db2instp_batman_0-rs OldOpState=5 NewOpState=1 Handle=0x6028 0xffff 0xb506ae54
0xb9d31ef8 0x12404f85 0x3ba
d7160
08:17:11 AM.220830 T(4124703600) _GBD Start command for application resource
"db2_db2instp_batman_0-rs" (handle 0x6028 0xffff 0xb506ae54 0xb9d31ef8 0x12404f85 0x3bad7160)
succeeded with exit code
0
71

Samdiag output
DB2 instance
# samdiag -g db2_db2instp_batman_0-rg
Displaying information for the following:
Resource Group "db2_db2instp_batman_0-rg":
Diagnosis::Resource: db2_db2instp_batman_0-rg/ResGroup/IBM.ResourceGroup
type: CHARM Resource Group
Status -
Observed: Offline - SoftDown
Desired: Offline - Requested Offline
(Nominal: Offline - Nominal State: Offline)
Automation: Idle - CharmBase trigger linked
Startable: Yes - Resource is startable
Binding: Unbound - Unbound
Compound: Satisfactory - Satisfactory
Move: Not_Supported - Resource Move State is Not Supported
72

Samdiag output
DB2 instance (continued)
Resource Based Quorum: In Quorum - Resource has Quorum
Members and Memberships:
+---HasMember ---> db2_db2instp_batman_0-
rs/Float/IBM.Application
Group Constraint: Collocated
Binding Constraints:
Flags:
None
Orders:
Outstanding Order: None - Resource is Unavailable
Dependencies:
Start: Satisfied
+---InCluster ---> Cluster
Stop: Satisfied
Binding exceptions:
There are unbound members.
Static Relationships:
+---InCluster ---> Cluster
Dynamic Relationships:"
73

Tips &tricks
• Use TSA to improve availability of standalone databases
• Replace db2fm
• Use HADR failover without specifying HADR database
• Instance start-script will also activate databases
• Keep the # of databases / instance low
• Rebuild the domain quickly with sampolicy
74

Library & References
• DB2 Information Center
• TSAMP documentation
• http://publib.boulder.ibm.com/tividd/td/IBMTivoliSystemAutomationforMultiplatforms3.1.html
• Automated Cluster Controlled HADR Configuration Setup
using the IBM DB2 High Availability Instance Utility
• ftp://ftp.software.ibm.com/software/data/pubs/papers/HADR_db2haicu.pdf
• Automating DB2 HADR Failover on Linux using Tivoli System
Automation for Multiplatforms
• ftp://ftp.software.ibm.com/software/data/pubs/papers/hadr_tsa.pdf
• DB2 system topology and configuration for automated multi-site HA
and DR
• http://public.dhe.ibm.com/software/dw/data/dm-1002automateddb2hadr/Automated_long-
distance_HADR_configurations.pdf
75

Library & References
• Integrating Tivoli System Automation for Multiplatforms” (TSAMP)
with DB2 High Availability & Disaster Recovery (HADR) v9.5 (Garreth
Holl)
• http://www-01.ibm.com/support/docview.wss?uid=swg27016774&aid=1
• High Availability and Disaster Recovery Options for DB2 on Linux,
UNIX, and Windows
• www.redbooks.ibm.com/redbooks/pdfs/sg247363.pdf
• DB2 Best Practices – HADR
• http://www.ibm.com/developerworks/data/bestpractices/
76

Click to edit Master title style 77
Thank you for attending.
Any questions?

A DBA’s guide to using TSA

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (7)

Similar to A DBA’s guide to using TSA

Similar to A DBA’s guide to using TSA (20)

Recently uploaded

Recently uploaded (20)

A DBA’s guide to using TSA

Editor's Notes