TSA provides automatic monitoring and availability management of resources configured for high availability in a cluster domain. It monitors DB2 HADR resources and DB2 instance resources, and can start, stop, and fail over these resources between nodes when failures occur. The document provides examples of how DB2 HADR and instance resources are defined and monitored by TSA using the IBM.Application resource type.

1. A DBA’s guide to using TSA
Frederik Engelen
RealDolmen
Session Code: C03
16 November 2011 | Platform: LUW

2. Click to edit Master title style
Session Goal
• Quick overview of HADR functionality and advantages
• Description of Tivoli System Automation for Multiplatforms
(TSAMP) software and integration with DB2
• Operational tasks and lessons learned
2

3. Click to edit Master title style
Why HADR?
• Failover solution available from DB2 v8.2
• Design choices:
• Ultra-fast failover capability
• Negligible impact on performance
• Configurable degree of consistency
• Low administrative costs
• Avoid service interruption for fixpack installs and changes
requiring instance restart
• Transparent failover and failback for applications
• Spread over different geographic locations
• Built in clustering software in DB2 9.5
• Cheap
3

4. Click to edit Master title style
Basic Principle HADR
• Database scope
• Allows flexible configurations
• Primary database
• Processes transactions
• Send logs to standby
• Standby database
• Applies transactions recieved from primary
• Available for read-only workload (DB2 v9.7 FP1)
• In case of primary failure, standby database can take over
primary role
4
Primary Standby
Log shipping
Primary connection

5. Click to edit Master title style
HADR Synchronisation Modes
• Configurable degree of consistency
• SYNC, NEARSYNC, ASYNC, SUPERASYNC (NEW)
5
New logs
send() recieve()
Logs
ASYNC NEARSYNC
SYNC
TCP/IP
SUPER
ASYNC

6. Click to edit Master title style
Ready to implement?
• Network layout
• High speed network
• Private or public HADR communication?
• Reserve HADR communication port
• Use identical OS & database version and hardware family
• For automation with TSA, make sure DB2 > v9.5 FP5 or v9.7 FP1
• Use identical file system layout
• Keep clock synchronised
• Recommended:
• Same amount of memory & hardware
6

7. Click to edit Master title style
Implementing
• Configure primary database and backup
• Restore on standby server
• Adjust db cfg on standby database
• Start HADR
• Configure Automatic Client Reroute
7

8. Click to edit Master title style
Configure primary database
• Database configuration
db2 update db cfg for <DBNAME> using LOGARCHMETH1 DISK:/share/logarchs
db2 update db cfg for <DBNAME> using INDEXREC RESTART
db2 update db cfg for <DBNAME> using LOGINDEXBUILD ON
db2 update db cfg for <DBNAME> using HADR_TIMEOUT 30
db2 update db cfg for <DBNAME> using HADR_PEER_WINDOW 120
db2 update db cfg for <DBNAME> using HADR_LOCAL_HOST <PRIMARY_HOST>
db2 update db cfg for <DBNAME> using HADR_LOCAL_SVC <PRIMARY_HADR_PORT>
db2 update db cfg for <DBNAME> using HADR_REMOTE_HOST <STANDBY_HOST>
db2 update db cfg for <DBNAME> using HADR_REMOTE_SVC <STANDBY_HADR_PORT>
db2 update db cfg for <DBNAME> using HADR_REMOTE_INST <STANDBY_INST>
db2 update db cfg for <DBNAME> using HADR_SYNCMODE NEARSYNC
• Take a backup
8

9. Click to edit Master title style
Configure standby server
• Restore backup
• Keep name
• Database configuration
db2 update db cfg for <DBNAME> using HADR_LOCAL_HOST <STANDBY_HOST>
db2 update db cfg for <DBNAME> using HADR_LOCAL_SVC <STANDBY_HADR_PORT>
db2 update db cfg for <DBNAME> using HADR_REMOTE_HOST <PRIMARY_HOST>
db2 update db cfg for <DBNAME> using HADR_REMOTE_SVC <PRIMARY_HADR_PORT>
db2 update db cfg for <DBNAME> using HADR_REMOTE_INST <PRIMARY_INST>
9

10. Click to edit Master title style
Start HADR
• Standby
db2 update hadr on db <DBNAME> as standby
• Replay local log files
• Primary
db2 start hadr on db <DBNAME> as primary
• Replay primary archived logs
• Replay primary active logs
• Replay in-memory log buffer
-> Peer state
10
HADR start
Local Catchup
Remote Catch-
up Pending
Catch-up
Pending
Peer
Disconnected
Peer
Connectionlost
Connect
Disconnected
Reconnected
Peerwindowexpires

11. Click to edit Master title style
HADR result
• db2pd –hadr –db SAMPLE
Database Partition 0 -- Database SAMPLE -- Active -- Up 0 days 00:16:25 -- Date
08/21/2011 08:28:39
HADR Information:
Role State SyncMode HeartBeatsMissed LogGapRunAvg (bytes)
Primary Peer Nearsync 0 0
ConnectStatus ConnectTime Timeout
Connected Sun Aug 21 08:13:29 2011 (1313928809) 120
PeerWindowEnd PeerWindow
Sun Aug 21 08:30:39 2011 (1313929839) 120
LocalHost LocalService
batman 60000
RemoteHost RemoteService RemoteInstance
robin 60000 db2insts
PrimaryFile PrimaryPg PrimaryLSN
S0000005.LOG 0 0x00000000032C8010
StandByFile StandByPg StandByLSN
S0000005.LOG 0 0x00000000032C8010
11

12. Click to edit Master title style
Basic Operations
• Takeover
db2 takeover hadr on db <dbname> [by force [peer window
only]]
• Stop database
db2 deactivate db <dbname> && db2stop
 “stop hadr“ will turn db to standalone db
• Actually, that’s about it…
12

13. Click to edit Master title style
Automatic Client Reroute
• DB2 client reroutes connection to alternative server in
case of communication error
db2 update alternate server for <dbname> using hostname <host>
port <svcename>
• On connection failure:
SQL30108N A connection failed but has been reestablished
• HADR is not required
• JDBC URL
jdbc:db2://<dbhost>:<port>/<db>:clientRerouteAlternateServerNam
e=<althost>;clientRerouteAlternatePortNumber=<altport>;
• Configure on client and server
13

14. Click to edit Master title style
Things to keep in mind
• What gets replicated? Everything that’s logged.
• What not?
• DB CFG
• History file
• Not Logged Initially
• LOB columns that are not logged (>1G)
• Common operations
• LOAD: COPY YES
• REORG: consider reorg online or deactivate standby to avoid
blocking primary
• Tuning
• HADR simulator
14

15. Click to edit Master title style
Archival Logs
• Only primary server archives logs
• Store on shared file system
• Faster catch-up process
• Avoid manually copying logs when restore on standby
• Attention when using AUTO_DEL_REC_OBJ
15

16. Click to edit Master title style
Blocknonlogged
• Introduced in DB2 v9.5 FP4 / v9.7
• Blocks non-logged operations
• Not logged initially
• Not logged colums (BLOB and CLOB > 1G)
• Simple measure to protect from human error or an
unsuspecting application
16

17. Click to edit Master title style
Monitoring
• db2pd -hadr
• Database snapshot
• Health Monitor
• db2diag.log – check for messages from d2hadrp and
db2hadrs
db2insts@robin:~> db2diag -g "eduname:=db2hadr" -fmt "%eduname @msg
@changeevent“
db2hadrs Initiating HADR takeover request, forcing applications.
db2hadrs Info: Standby has initiated a takeover.
db2hadrs Info: Standby switching roles to primary.
db2hadrs HADR role set to Primary (was Standby)
db2hadrp HADR state set to P-Peer (was S-Peer)
db2hadrp Info: Standby has completed takeover (now primary).
db2hadrp Info: Standby Finished.
db2hadrp Info: Primary Started.
db2hadrp Info: HADR Socket send buffer size, SO_SNDBUF: 16384 bytes
db2hadrp Info: HADR Socket receive buffer size, SO_RCVBUF: 87380 bytes
17

18. Click to edit Master title style
Where does TSA fit in?
• TSA provides automatic monitoring and availability
management of “resources”
• TSA helps you when one of the cluster components fails
• But…TSA will also be there when you are
• Performing maintenance activities
• Troubleshooting
-> Basic knowledge of TSA and how it interacts with DB2 is
essential to succesfully perform your tasks as a DBA
18

19. Click to edit Master title style
Goal
• Components and terminology
• TSA domain configuration
• Operations
• Monitoring & troubleshooting
19

20. Click to edit Master title style
RSCT/TSA Terminology
• Cluster domain
• Node
• Resource
• Resource group
• Quorum
• Tiebreaker
• Equivalency
• Let’s see that in more detail…
20

21. Click to edit Master title style
Domain/Node
• Group of server nodes hosting resources configured for
high availability with RSCT
• Each node runs RSCT subsystems/daemons
• preprpnode, lsrpnode, addrpnode, rmrpnode,
stoprpnode, startrpnode
• mkrpdomain, lsrpdomain, rmrpdomain,
startrpdomain, stoprpdomain
21

22. Click to edit Master title style
Domain – RSCT components
# lssrc -a
Subsystem Group PID Status
IBM.ConfigRM rsct_rm 3610 active
ctcas rsct 3891 active
cthats cthats 5457 active
cthags cthags 5505 active
ctrmc rsct 5625 active
IBM.GblResRM rsct_rm 5792 active
IBM.RecoveryRM rsct_rm 5793 active
IBM.StorageRM rsct_rm 5794 active
IBM.TestRM rsct_rm 5795 active
22

23. Click to edit Master title style
Domain – RSCT components
• CTHATS: Topology Services
• Node connectivity (heartbeat)
• Messaging
• CTHAGS: Groups Services
• Synchronised configuration
• Messagging
• CTRMC: Resource Monitoring & Control
• Backbone of RSCT
• Monitoring & control infrastructure across cluster
• CTCAS: Cluster Authentication Services
• IBM.ConfigRM: Configuration Resource Manager
• Management application
• Create and administer a domain
23

24. Click to edit Master title style
Resource
• Any piece of hard- or software (configuration) that
provides services
• HADR role, DB2 instance, virtual IP, file system, tiebreaker,…
• Instance of Resource Class
• Most common IBM.Application (start/stop/monitor scriptsà)
• Provided by resource managers
• GlbResRM, TestRM, StorageRM, RecoveryRM
• Fixed/floating/concurrent
• Not directly controllable
• lsrsrc, rmrsrc, chrsrc, resetrsrc,
lsrsrcdef
24

25. Click to edit Master title style
IBM.Application class
Example of a DB2 HADR Resource
# lsrsrc –s “Name = ‘db2_db2instp_db2insts_SAMPLE-rs’ ” -Ab IBM.Application
resource 1:
Name = "db2_db2instp_db2insts_SAMPLE-rs"
ResourceType = 0 (Constituent)
AggregateResource = "0x2028 0xffff 0x5e5ca445 0xb9d31ef8 0x9239e701
0x23ba7500"
StartCommand = "/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh
db2instp db2insts SAMPLE"
StopCommand = "/usr/sbin/rsct/sapolicies/db2/hadrV97_stop.ksh
db2instp db2insts SAMPLE"
MonitorCommand = "/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh
db2instp db2insts SAMPLE"
MonitorCommandPeriod = 21
MonitorCommandTimeout = 29
StartCommandTimeout = 330
StopCommandTimeout = 140
ActivePeerDomain = "gotham"
NodeNameList = {"robin"}
OpState = 2 (Offline on this node)
25

26. Click to edit Master title style
IBM.Application class
Example of a DB2 HADR Resource
# lsrsrc –s “Name = ‘db2_db2instp_db2insts_SAMPLE-rs’ ” -Ab IBM.Application
resource 2:
Name = "db2_db2instp_db2insts_SAMPLE-rs"
ResourceType = 0 (Constituent)
AggregateResource = "0x2028 0xffff 0x5e5ca445 0xb9d31ef8 0x9239e701
0x23ba7500"
StartCommand = "/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh
db2instp db2insts SAMPLE"
StopCommand = "/usr/sbin/rsct/sapolicies/db2/hadrV97_stop.ksh
db2instp db2insts SAMPLE"
MonitorCommand = "/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh
db2instp db2insts SAMPLE"
MonitorCommandPeriod = 21
MonitorCommandTimeout = 29
StartCommandTimeout = 330
StopCommandTimeout = 140
ActivePeerDomain = "gotham"
NodeNameList = {"batman"}
OpState = 1 (Online on this node)
26

27. Click to edit Master title style
IBM.Application class
Example of a DB2 HADR Resource
# lsrsrc –s “Name = ‘db2_db2instp_db2insts_SAMPLE-rs’ ” -Ab IBM.Application
resource 3:
Name = "db2_db2instp_db2insts_SAMPLE-rs"
ResourceType = 1 (Aggregate)
AggregateResource = "0x3fff 0xffff 0x00000000 0x00000000 0x00000000
0x00000000"
StartCommand = "/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh
db2instp db2insts SAMPLE"
StopCommand = "/usr/sbin/rsct/sapolicies/db2/hadrV97_stop.ksh
db2instp db2insts SAMPLE"
MonitorCommand = "/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh
db2instp db2insts SAMPLE"
MonitorCommandPeriod = 21
MonitorCommandTimeout = 29
StartCommandTimeout = 330
StopCommandTimeout = 140
ActivePeerDomain = "gotham"
NodeNameList = {"batman","robin"}
OpState = 1 (Online globally)
27

28. Click to edit Master title style
IBM.Application class
Example of a DB2 Instance Resource
# lsrsrc –s “Name = ‘db2_db2instp_batman_0-rs’” –Ab IBM.Application
resource 1:
Name = "db2_db2instp_batman_0-rs"
ResourceType = 0 (Constituent)
AggregateResource = "0x2028 0xffff 0x5e5ca445 0xb9d31ef8 0x9239e6fa
0x97f134b0"
StartCommand = "/usr/sbin/rsct/sapolicies/db2/db2V97_start.ksh
db2instp 0"
StopCommand = "/usr/sbin/rsct/sapolicies/db2/db2V97_stop.ksh
db2instp 0"
MonitorCommand = "/usr/sbin/rsct/sapolicies/db2/db2V97_monitor.ksh
db2instp 0"
MonitorCommandPeriod = 10
MonitorCommandTimeout = 120
StartCommandTimeout = 330
StopCommandTimeout = 140
ActivePeerDomain = "gotham"
NodeNameList = {"batman"}
OpState = 1 (Online on this node)
28

29. Click to edit Master title style
IBM.Application class
Example of a DB2 Instance Resource
# lsrsrc –s “Name = ‘db2_db2instp_batman_0-rs’” –Ab IBM.Application
resource 2:
Name = "db2_db2instp_batman_0-rs"
ResourceType = 1 (Aggregate)
AggregateResource = "0x3fff 0xffff 0x00000000 0x00000000 0x00000000
0x00000000"
StartCommand = "/usr/sbin/rsct/sapolicies/db2/db2V97_start.ksh
db2instp 0"
StopCommand = "/usr/sbin/rsct/sapolicies/db2/db2V97_stop.ksh
db2instp 0"
MonitorCommand = "/usr/sbin/rsct/sapolicies/db2/db2V97_monitor.ksh
db2instp 0"
MonitorCommandPeriod = 10
MonitorCommandTimeout = 120
StartCommandTimeout = 330
StopCommandTimeout = 140
ActivePeerDomain = "gotham"
NodeNameList = {"batman"}
OpState = 1 (Online globally)
29

30. Click to edit Master title style
IBM.ServiceIP class
Example of a HADR Virtual IP
# lsrsrc -Ab IBM.ServiceIP
resource 1:
Name = "db2ip_192_168_13_132-rs"
ResourceType = 0 (Constituent)
AggregateResource = "0x2029 0xffff 0x5e5ca445 0xb9d31ef8 0x9239e864
0x5d11b368"
IPAddress = "192.168.13.132"
NetMask = "255.255.255.0"
ProtectionMode = 1
NetPrefix = 0
ActivePeerDomain = "gotham"
NodeNameList = {"robin"}
OpState = 2 (Offline on this node)
ConfigChanged = 0
ChangedAttributes = {}
30

31. Click to edit Master title style
IBM.ServiceIP class
Example of a HADR Virtual IP
# lsrsrc -Ab IBM.ServiceIP
resource 2:
Name = "db2ip_192_168_13_132-rs"
ResourceType = 0 (Constituent)
AggregateResource = "0x2029 0xffff 0x5e5ca445 0xb9d31ef8 0x9239e864
0x5d11b368"
IPAddress = "192.168.13.132"
NetMask = "255.255.255.0"
ProtectionMode = 1
NetPrefix = 0
ActivePeerDomain = "gotham"
NodeNameList = {"batman"}
OpState = 1 (Online on this node)
ConfigChanged = 0
ChangedAttributes = {}
31

32. Click to edit Master title style
IBM.ServiceIP class
Example of a HADR Virtual IP
# lsrsrc -Ab IBM.ServiceIP
resource 3:
Name = "db2ip_192_168_13_132-rs"
ResourceType = 1 (Aggregate)
AggregateResource = "0x3fff 0xffff 0x00000000 0x00000000 0x00000000
0x00000000"
IPAddress = "192.168.13.132"
NetMask = "255.255.255.0"
ProtectionMode = 1
NetPrefix = 0
ActivePeerDomain = "gotham"
NodeNameList = {"batman","robin"}
OpState = 1 (Online globally)
ConfigChanged = 0
ChangedAttributes = {}
32

33. Click to edit Master title style
Resource Managers
• IBM.GlbResRM
• Provides the IBM.Application and IBM.ServiceIP resource class
• IBM.RecoveryRM
• Brain of TSA, decision engine
• Gathers monitoring info and decides on action to take
• IBM.StorageRM
• Montoring and control of storage resources
• IBM.TestRM
• Provides IBM.Test resource class
• Simulates automation scenarios
• RM logfiles: /var/ct/${domain}/log/mc/IBM.${daemon}RM/
33

34. Click to edit Master title style
Resource Managers
# lssrc -ls IBM.RecoveryRM
Subsystem : IBM.RecoveryRM
Cluster Name : gotham
Node Number : 2
Daemon start time : 09/04/11 06:55:01
Daemon State:
My Node Name : robin
Master Node Name : batman (node number = 1)
Our IVN : 3.2.1.1
Our AVN : 3.2.1.1
Total Node Count : 2
Joined Member Count : 2
Config Quorum Count : 2
Startup Quorum Count : 1
Operational Quorum State: HAS_QUORUM
In Config Quorum : TRUE
In Config State : TRUE
In Jeopardy : FALSE
34

35. Click to edit Master title style
Equivalency
• Set of resources that provide the same functionality
• Main resource classes:
• IBM.PeerNode
• IBM.NetworkInterface
• lsequ, mkequ, rmequ, chequ
35

36. Click to edit Master title style
Equivalency example
HADR nodes
# lsequ -s "Name='db2_db2instp_db2insts_SAMPLE-rg_group-equ'"
Equivalency 1:
Name = db2_db2instp_db2insts_SAMPLE-rg_group-equ
MemberClass = IBM.PeerNode
Resource:Node[Membership] = {batman:batman,robin:robin}
SelectString = ""
SelectFromPolicy = ANY
MinimumNecessary = 1
Subscription = {}
Color = 0
ActivePeerDomain = gotham
ConfigValidity =
36

37. Click to edit Master title style
Equivalency example
Virtual IP
# lsequ -s "Name='db2_public_network_0'"
Equivalency 1:
Name = db2_public_network_0
MemberClass = IBM.NetworkInterface
Resource:Node[Membership] = {eth0:batman,eth0:robin}
SelectString = ""
SelectFromPolicy = ANY
MinimumNecessary = 1
Subscription = {}
Color = 0
ActivePeerDomain = gotham
ConfigValidity =
37

38. Click to edit Master title style
Resource Group
• Groups resources (surprise…)
• All included resources share the same Nominal State
• MemberLocation attribute controls placement on nodes
• Controllable operational state
• Resides on node/equivalency
38

39. Click to edit Master title style
Resource Group Operations
• Move(cancel)
• rgreq -o move(cancel) <resource_group_name>
• Change status
• chrg -o offline <resource_group_name>
• Un(lock)
• rgreq -o move(cancel) <resource_group_name>
• lsrg, mkrg, chrg, rmrg
• addrgmbr, chrgmbr, rmrgmbr, rgmbrreq
• rgreq, lsrgreq, …
39

40. Click to edit Master title style
Resource Group example
HADR DB
# lsrg -s "Name='db2_db2instp_db2insts_SAMPLE-rg'"
Resource Group 1:
Name = db2_db2instp_db2insts_SAMPLE-rg
MemberLocation = Collocated
Priority = 0
AllowedNode = db2_db2instp_db2insts_SAMPLE-rg_group-equ
NominalState = Online
ActivePeerDomain = gotham
OpState = Online
TopGroup = db2_db2instp_db2insts_SAMPLE-rg
TopGroupNominalState = Online
40

41. Click to edit Master title style
Resource Group example
HADR DB
# lsrg -m -g db2_db2instp_db2insts_SAMPLE-rg
Member Resource 1:
Class:Resource:Node[ManagedResource] =
IBM.Application:db2_db2instp_db2insts_SAMPLE-rs
Mandatory = True
MemberOf = db2_db2instp_db2insts_SAMPLE-rg
SelectFromPolicy = ORDERED
ActivePeerDomain = gotham
OpState = Online
Member Resource 2:
Class:Resource:Node[ManagedResource] = IBM.ServiceIP:db2ip_192_168_13_132-rs
Mandatory = True
MemberOf = db2_db2instp_db2insts_SAMPLE-rg
SelectFromPolicy = ORDERED
ActivePeerDomain = gotham
ConfigValidity =
OpState = Online
41

42. Click to edit Master title style
Relationship
• Configured between resources, resource groups and
equivalencies (1-N)
• Mainly DependsOn type
• StartAfter, ForcedDownBy, …
• lsrel, mkrel, chrel, rmrel
42

43. Click to edit Master title style
Relationship example
DB2 instance -> public network
# lsrel -s "Name = 'db2_db2instp_batman_0-rs_DependsOn_db2_public_network_0-
rel'"
Name = db2_db2instp_batman_0-
rs_DependsOn_db2_public_network_0-rel
Class:Resource:Node[Source] = IBM.Application:db2_db2instp_batman_0-rs
Class:Resource:Node[Target] = {IBM.Equivalency:db2_public_network_0}
Relationship = DependsOn
Conditional = NoCondition
ConfigValidity =
ResourceGroup[Source] = db2_db2instp_batman_0-rg
43

44. Click to edit Master title style
Quorum
• Quorum: the number of nodes in a cluster that are
required to control the resources (majority)
• Example log entry
ConfigRM[3610]: (Recorded using libct_ffdc.a cv 2):::Error ID: :::Reference
ID: :::Template ID: 0:::Details File: :::Location:
RSCT,PeerDomain.C,1.99.22.19,18004 :::CONFIGRM_PENDINGQUORUM_ER
The operational quorum state of the active peer domain has changed to
PENDING_QUORUM. This state usually indicates that exactly half of the nodes
that are defined in the peer domain are online. In this state cluster
resources cannot be recovered although none will be stopped explicitly.
• Majority with only 2 HADR nodes? Tiebreaker
44

45. Click to edit Master title style
Tiebreaker
• Determine quorum when half of the nodes survives
• Network Tiebreaker
• Only supported type out of the box
• A pingable IP address (default gateway is a good choice)
• Disk Tiebreaker
• Possible to add manually
• SCSI Reservation, faster
• More secure
• Without quorum (+ tiebreaker), no failover will occur !
45

46. Click to edit Master title style
Communication TSA/DB2
• DB2->TSA
• API implemented by cluster manager
• CLUSTER_MGR = TSA
• TSA->DB2
• HA Policy scripts in /usr/sbin/rsct/sapolicies
# ls /usr/sbin/rsct/sapolicies/db2/
db2V97_monitor|start|stop.ksh
hadrV97_monitor|start|stop.ksh
mountV97_monitor|start|stop.ksh
nfsserverctrl-server
forceAllApps
rovingV97_failover.ksh
46

47. Click to edit Master title style
Policy script example
batman:~ # head -n 17 /usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh
#!/bin/ksh -p
#-----------------------------------------------------------------------
# (C) COPYRIGHT International Business Machines Corp. 2001-2009
# All Rights Reserved
#
# US Government Users Restricted Rights - Use, duplication or
# disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
#
# VERSION: db2_v97fp4 s110330
# INPUT: hadrV97_monitor.ksh db2instp db2insts hadrdb [verbose]
#
# OUTPUT: 1 if online, 2 if offline, 0 if not known
#
# NOTES: Can only be used in the context of a TSA environment configured
# with the db2 HA Integrated Configuration Utility (db2haicu)
#
#-----------------------------------------------------------------------
47

48. Click to edit Master title style
Configuring for HA
• Install TSA + DB2 (lots of fixes -> latest fixpack)
• Who would’ve guessed…
• Configure HADR (Peer State)
• We already know how to do that
• Disable db2fm
• Prepare nodes
• #preprpnode batman robin
• Run DB2 High Availability Instance Configuration Utility
(db2haicu)
• Interactively
• XML
48

49. Click to edit Master title style
DB2HAICU XML File example
<DB2Cluster xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="db2ha.xsd" clusterManagerName="TSA"
version="1.0">
<ClusterDomain domainName=“gotham">
<Quorum quorumDeviceProtocol="network“ quorumDeviceName="192.168.13.1"/>
<PhysicalNetwork physicalNetworkName="db2_public_network_0"
physicalNetworkProtocol="ip">
<Interface interfaceName="eth0" clusterNodeName="batman">
<IPAddress baseAddress="192.169.13.130“
subnetMask="255.255.255.0"
networkName="db2_public_network_0"/>
</Interface>
<Interface interfaceName="eth0" clusterNodeName=“robin">
<IPAddress baseAddress="192.169.13.131"
subnetMask="255.255.255.0"
networkName="db2_public_network_0"/>
</Interface>
</PhysicalNetwork>
49

50. Click to edit Master title style
DB2HAICU XML File example (continued)
<PhysicalNetwork physicalNetworkName="db2_private_network_0“
physicalNetworkProtocol="ip">
<Interface interfaceName="eth1" clusterNodeName="batman">
<IPAddress baseAddress="192.168.102.2"
subnetMask="255.255.255.0"
networkName="db2_private_network_0"/>
</Interface>
<Interface interfaceName="eth1" clusterNodeName="robin">
<IPAddress baseAddress="192.168.102.3"
subnetMask="255.255.255.0”
networkName="db2_private_network_0"/>
</Interface>
</PhysicalNetwork>
<ClusterNode clusterNodeName="batman"/>
<ClusterNode clusterNodeName="robin“/>
</ClusterDomain>
50

51. Click to edit Master title style
DB2HAICU XML File example (continued)
<FailoverPolicy>
<HADRFailover></HADRFailover>
</FailoverPolicy>
<DB2PartitionSet>
<DB2Partition dbpartitionnum="0" instanceName="db2instp"/>
</DB2PartitionSet>
<HADRDBSet>
<HADRDB databaseName="SAMPLE" localInstance="db2instp"
remoteInstance="db2insts" localHost ="batman“ remoteHost="robin" />
<VirtualIPAddress baseAddress="192.168.13.132“
subnetMask="255.255.255.0"
networkName="db2_public_network_0"/>
</HADRDBSet>
</DB2Cluster>
51

52. Click to edit Master title style
DB2HAICU
• Start on standby instance
• Don’t mix primary/standby databases in the same instance
$db2haicu –f ha_cfg.xml
 Repeat for primary instance
 Done
52

53. Click to edit Master title style
DB2HAICU standby output
db2haicu determined the current DB2 database manager instance is db2insts. The cluster
configuration that follows will apply to this instance.
Creating domain gotham in the cluster ...
Creating domain gotham in the cluster was successful.
Configuring quorum device for domain gotham ...
Configuring quorum device for domain gotham was successful.
Adding network interface card eth0 on cluster node batman to the network db2_public_network_0 ...
Adding network interface card eth0 on cluster node batman to the network db2_public_network_0 was
successful.
Adding network interface card eth0 on cluster node robin to the network db2_public_network_0 ...
Adding network interface card eth0 on cluster node robin to the network db2_public_network_0 was
successful.
Adding network interface card eth1 on cluster node batman to the network db2_private_network_0 ...
Adding network interface card eth1 on cluster node batman to the network db2_private_network_0 was
successful.
Adding network interface card eth1 on cluster node robin to the network db2_private_network_0 ...
Adding network interface card eth1 on cluster node robin to the network db2_private_network_0 was
successful.
Adding DB2 database partition 0 to the cluster ...
Adding DB2 database partition 0 to the cluster was successful.
The HADR database SAMPLE has been determined to be valid for high availability. However, the
database cannot be added to the cluster from this node because db2haicu detected this node is the
standby for the HADR database SAMPLE. Run db2haicu on the primary for the HADR database SAMPLE to
configure the database for automated failover.
All cluster configurations have been completed successfully. db2haicu exiting ...
53

54. Click to edit Master title style
DB2HAICU primary output
db2haicu determined the current DB2 database manager instance is db2instp. The cluster
configuration that follows will apply to this instance.
Configuring quorum device for domain gotham ...
Configuring quorum device for domain gotham was successful.
The network adapter eth0 on node batman is already defined in network db2_public_network_0 and
cannot be added to another network until it is removed from its current network.
The network adapter eth0 on node robin is already defined in network db2_public_network_0 and
cannot be added to another network until it is removed from its current network.
The network adapter eth1 on node batman is already defined in network db2_private_network_0 and
cannot be added to another network until it is removed from its current network.
The network adapter eth1 on node robin is already defined in network db2_private_network_0 and
cannot be added to another network until it is removed from its current network.
Adding DB2 database partition 0 to the cluster ...
Adding DB2 database partition 0 to the cluster was successful.
Adding HADR database SAMPLE to the domain ...
Adding HADR database SAMPLE to the domain was successful.
All cluster configurations have been completed successfully. db2haicu exiting ...
54

55. Click to edit Master title style
Cluster domain overview
55
PeerNode: BATMAN PeerNode: ROBIN
Equivalency: db2_db2instp_db2insts_SAMPLE-rg_group-equ
Resource Group: db2_db2instp_db2insts_SAMPLE-rg
Floating Resource:
db2_db2instp_db2insts_SAMPLE-rs
HADR Primary HADR Secondary
Floating Resource:
db2ip_192_168_13_132-rs
Virtual IP Virtual IP
db2_db2instp_batman_0-
rg_group-equ
db2_db2instp_
batman-rg
db2_db2insts_robin_0-
rg_group-equ
db2_db2insts_
robin-rg
Resource:
db2_db2instp_
batman-rs
Resource:
db2_db2insts_
robin-rs
Private Network
Equivalency: db2_public_network_0 eth0eth0
Equivalency: db2_private_network_0 eth1eth1
Public Network
Relation:
db2_db2instp_batman_0-rs_
DependsOn_
db2_public_network_0-rel
Relation:
db2_db2insts_robin_0-rs_
DependsOn_
db2_public_network_0-rel

56. Click to edit Master title style
DB2HAICU result
~# lssam
Online IBM.ResourceGroup:db2_db2instp_batman_0-rg Nominal=Online
'- Online IBM.Application:db2_db2instp_batman_0-rs
'- Online IBM.Application:db2_db2instp_batman_0-rs:batman
Online IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Nominal=Online
|- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs
|- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:batman
'- Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:robin
'- Online IBM.ServiceIP:db2ip_192_168_13_132-rs
|- Online IBM.ServiceIP:db2ip_192_168_13_132-rs:batman
'- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs:robin
Online IBM.ResourceGroup:db2_db2insts_robin_0-rg Nominal=Online
'- Online IBM.Application:db2_db2insts_robin_0-rs
'- Online IBM.Application:db2_db2insts_robin_0-rs:robin
Online IBM.Equivalency:db2_db2instp_batman_0-rg_group-equ
'- Online IBM.PeerNode:batman:batman
Online IBM.Equivalency:db2_db2instp_db2insts_SAMPLE-rg_group-equ
|- Online IBM.PeerNode:batman:batman
'- Online IBM.PeerNode:robin:robin
Online IBM.Equivalency:db2_db2insts_robin_0-rg_group-equ
'- Online IBM.PeerNode:robin:robin
…
56

57. Click to edit Master title style
DB2HAICU result (continued)
…
Online IBM.Equivalency:db2_private_network_0
|- Online IBM.NetworkInterface:eth1:batman
'- Online IBM.NetworkInterface:eth1:robin
Online IBM.Equivalency:db2_public_network_0
|- Online IBM.NetworkInterface:eth0:batman
'- Online IBM.NetworkInterface:eth0:robin
~> db2 get dbm cfg | grep CLUSTER_MGR
Cluster manager (CLUSTER_MGR) = TSA
57

58. Click to edit Master title style
Operations
• Use regular commands whenever possible:
• Takeover, db2stop, db2start,…
• Check the result on the cluster domain
• TSA commands still work
• rgreq –o move <HADR RG>
• chrg -o offline <DB2 Instance RG>
• Takeover example
• $ db2 takeover hadr on db <DBNAME>
58

59. Click to edit Master title style
Takeover example
~# lssam
…
Online IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Nominal=Online
|- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs
|- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:batman
'- Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:robin
'- Online IBM.ServiceIP:db2ip_192_168_13_132-rs
|- Online IBM.ServiceIP:db2ip_192_168_13_132-rs:batman
'- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs:robin
…
59

60. Click to edit Master title style
Takeover example (continued)
~# lssam
…
Online IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Request=Lock Nominal=Online
|- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs Control=SuspendedPropagated
|- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:batman
'- Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:robin
'- Online IBM.ServiceIP:db2ip_192_168_13_132-rs Control=SuspendedPropagated
|- Online IBM.ServiceIP:db2ip_192_168_13_132-rs:batman
'- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs:robin
…
60

61. Click to edit Master title style
Takeover example (continued)
~# lssam
…
Online IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Request=Move Nominal=Online
|- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs
|- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:batman
'- Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:robin
'- Online IBM.ServiceIP:db2ip_192_168_13_132-rs
|- Online IBM.ServiceIP:db2ip_192_168_13_132-rs:batman
'- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs:robin
…
61

62. Click to edit Master title style
Takeover example (continued)
~# lssam
…
Pending Offline IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Request=Move
Nominal=Online
|- Pending Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs
|- Pending Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:batman
'- Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:robin
'- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs Control=StartInhibited
|- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs:batman
'- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs:robin
…
62

63. Click to edit Master title style
Takeover example (continued)
~# lssam
…
Pending Offline IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Request=Move
Binding=Sacrificial Nominal=Online
|- Pending Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs Binding=Sacrificial
|- Pending Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:batman
'- Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:robin
'- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs Binding=Sacrificial
|- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs:batman
'- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs:robin
…
63

64. Click to edit Master title style
Takeover example (continued)
~# lssam
…
Pending Online IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Request=Move
Nominal=Online
|- Pending Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs
|- Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:batman
'- Pending Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:robin
'- Online IBM.ServiceIP:db2ip_192_168_13_132-rs
|- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs:batman
'- Online IBM.ServiceIP:db2ip_192_168_13_132-rs:robin
…
64

65. Click to edit Master title style
Takeover example (continued)
~# lssam
…
Online IBM.ResourceGroup:db2_db2instp_db2insts_SAMPLE-rg Nominal=Online
|- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs
|- Offline IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:batman
'- Online IBM.Application:db2_db2instp_db2insts_SAMPLE-rs:robin
'- Online IBM.ServiceIP:db2ip_192_168_13_132-rs
|- Offline IBM.ServiceIP:db2ip_192_168_13_132-rs:batman
'- Online IBM.ServiceIP:db2ip_192_168_13_132-rs:robin
…
65

66. Click to edit Master title style
Regain control
• Disable TSA
• $ db2haicu –disable
• Remove DB2 from TSA config
• $ db2haicu –delete
• Change TSAMP to manual/automatic mode
• # samctrl –m T/F
• Remove domain
• # rmrpdomain –f <domainname>
66

67. Click to edit Master title style
Monitoring & troubleshooting
• Main commands:
• db2pd –ha
• lssam
• samdiag
• HA Policy scripts (verbose)
• Syslog (/var/log/messages)
• Output from automation scripts
• Error messages from TSA
67

68. Click to edit Master title style
Monitoring & troubleshooting
• TSAMP Daemon logs
• /var/ct/${domain}/log/ctha[t|g]s
• /var/ct/${domain}/log/mc/IBM.${daemon}RM/
• Format using
• getsadata –traces
• rpttr
• Example: putting verbose start information in RecoveryRM trace:
chrsrc -s "Name = 'db2_db2instp_db2insts_SAMPLE-rs'"
IBM.Application
StartCommand="/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh
db2instp db2insts SAMPLE verbose"
68

69. Click to edit Master title style
Syslog example
/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh[9874]: Entering : db2instp db2insts SAMPLE
/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh[10427]: manual mode, or no state flags, use db2pd:
hdrRole:Primary, manual:-1, usepd:0, flagFile:0
/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh[10449]: Returning 1 : db2instp db2insts SAMPLE
/usr/sbin/rsct/sapolicies/db2/db2V97_monitor.ksh[10549]: Returning 2 (db2insts, 0)
db2V97_start.ksh[10554]: Entered /usr/sbin/rsct/sapolicies/db2/db2V97_start.ksh, db2insts, 0
/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh[10572]: Entering : db2instp db2insts SAMPLE
db2V97_start.ksh[10582]: Able to cd to /home/db2insts/sqllib :
/usr/sbin/rsct/sapolicies/db2/db2V97_start.ksh, db2insts, 0
/usr/sbin/rsct/sapolicies/db2/db2V97_monitor.ksh[10626]: Returning 2 (db2insts, 0)
/usr/sbin/rsct/sapolicies/db2/db2V97_monitor.ksh[10630]: Returning 2 (db2insts, 0)
db2V97_start.ksh[10631]: 1 partitions total: /usr/sbin/rsct/sapolicies/db2/db2V97_start.ksh,
db2insts, 0
hadrV97_monitor.ksh[10727]: /usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh is offline, hadr
monitor must return offline
/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh[10730]: Returning 2 : db2instp db2insts SAMPLE
/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh[10766]: Entering : db2instp db2insts SAMPLE
/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh[10779]: PROGNAME: hadrV97_start.ksh,PROGPATH:
/usr/sbin/rsct/sapolicies/db2,DB2HADRINSTANCE1: db2instp, DB2HADRINSTANCE2: db2in
sts, dbname: SAMPLE, DB2HADRDBNAME: SAMPLE, VERBOSE: noverbose
/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh[10786]: candidate_P_instance: db2instp,
candidate_S_instance: db2insts, localShorthost: robin
69

70. Click to edit Master title style
Syslog example
/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh[10793]: db2insts,db2instp,,
/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh[10794]: candidate_P_instance: db2insts,
candidate_S_instance: db2instp, localShorthost: robin, nn1: , nn1s:
/usr/sbin/rsct/sapolicies/db2/hadrV97_start.ksh[10796]: su - db2insts -c db2gcf -t 3600 -u -i
db2insts -i db2instp -h SAMPLE -L ...
/usr/sbin/rsct/sapolicies/db2/db2V97_monitor.ksh[12055]: Returning 2 (db2insts, 0)
/usr/sbin/rsct/sapolicies/db2/db2V97_monitor.ksh[12224]: Returning 1 (db2insts, 0)
/usr/sbin/rsct/sapolicies/db2/db2V97_monitor.ksh[13074]: Returning 1 (db2insts, 0)
/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh[13078]: Entering : db2instp db2insts SAMPLE
db2V97_start.ksh[13107]: Returning 0 from /usr/sbin/rsct/sapolicies/db2/db2V97_start.ksh ( db2insts,
0)
/usr/sbin/rsct/sapolicies/db2/db2V97_monitor.ksh[13122]: Returning 1 (db2insts, 0)
/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh[13611]: manual mode, or no state flags, use db2pd:
hdrRole:Primary, manual:-1, usepd:0, flagFile:0
/usr/sbin/rsct/sapolicies/db2/hadrV97_monitor.ksh[13633]: Returning 1 : db2instp db2insts SAMPLE
70

71. Click to edit Master title style
IBM.GblResRM trace_summary example
08:16:29 AM.757887 T(4124900208) _GBD Monitor detect OpState change for resource
Name=db2_db2instp_batman_0-rs OldOpState=1 NewOpState=2 Handle=0x6028 0xffff 0xb506ae54
0xb9d31ef8 0x12404f85 0x3ba
d7160
08:16:30 AM.181871 T(4151659376) _GBD Bringing application resource online:
Name=db2_db2instp_batman_0-rs Handle=0x6028 0xffff 0xb506ae54 0xb9d31ef8 0x12404f85
0x3bad7160
08:16:37 AM.200645 T(4124801904) _GBD Monitor detect OpState change for resource
Name=db2_db2instp_db2insts_SAMPLE-rs OldOpState=1 NewOpState=2 Handle=0x6028 0xffff
0xb506ae54 0xb9d31ef8 0x12404f8
b 0x7e81efb0
08:16:37 AM.219586 T(4151659376) _GBD Bringing application resource online:
Name=db2_db2instp_db2insts_SAMPLE-rs Handle=0x6028 0xffff 0xb506ae54 0xb9d31ef8 0x12404f8b
0x7e81efb0
08:16:57 AM.088710 T(4124900208) _GBD Monitor detect OpState change for resource
Name=db2_db2instp_batman_0-rs OldOpState=5 NewOpState=1 Handle=0x6028 0xffff 0xb506ae54
0xb9d31ef8 0x12404f85 0x3ba
d7160
08:17:11 AM.220830 T(4124703600) _GBD Start command for application resource
"db2_db2instp_batman_0-rs" (handle 0x6028 0xffff 0xb506ae54 0xb9d31ef8 0x12404f85 0x3bad7160)
succeeded with exit code
0
71

72. Click to edit Master title style
Samdiag output
DB2 instance
# samdiag -g db2_db2instp_batman_0-rg
Displaying information for the following:
Resource Group "db2_db2instp_batman_0-rg":
Diagnosis::Resource: db2_db2instp_batman_0-rg/ResGroup/IBM.ResourceGroup
type: CHARM Resource Group
Status -
Observed: Offline - SoftDown
Desired: Offline - Requested Offline
(Nominal: Offline - Nominal State: Offline)
Automation: Idle - CharmBase trigger linked
Startable: Yes - Resource is startable
Binding: Unbound - Unbound
Compound: Satisfactory - Satisfactory
Move: Not_Supported - Resource Move State is Not Supported
72

73. Click to edit Master title style
Samdiag output
DB2 instance (continued)
Resource Based Quorum: In Quorum - Resource has Quorum
Members and Memberships:
+---HasMember ---> db2_db2instp_batman_0-
rs/Float/IBM.Application
Group Constraint: Collocated
Binding Constraints:
Flags:
None
Orders:
Outstanding Order: None - Resource is Unavailable
Dependencies:
Start: Satisfied
+---InCluster ---> Cluster
Stop: Satisfied
Binding exceptions:
There are unbound members.
Static Relationships:
+---InCluster ---> Cluster
Dynamic Relationships:"
73

74. Click to edit Master title style
Tips &tricks
• Use TSA to improve availability of standalone databases
• Replace db2fm
• Use HADR failover without specifying HADR database
• Instance start-script will also activate databases
• Keep the # of databases / instance low
• Rebuild the domain quickly with sampolicy
74

75. Click to edit Master title style
Library & References
• DB2 Information Center
• TSAMP documentation
• http://publib.boulder.ibm.com/tividd/td/IBMTivoliSystemAutomationforMultiplatforms3.1.html
• Automated Cluster Controlled HADR Configuration Setup
using the IBM DB2 High Availability Instance Utility
• ftp://ftp.software.ibm.com/software/data/pubs/papers/HADR_db2haicu.pdf
• Automating DB2 HADR Failover on Linux using Tivoli System
Automation for Multiplatforms
• ftp://ftp.software.ibm.com/software/data/pubs/papers/hadr_tsa.pdf
• DB2 system topology and configuration for automated multi-site HA
and DR
• http://public.dhe.ibm.com/software/dw/data/dm-1002automateddb2hadr/Automated_long-
distance_HADR_configurations.pdf
75

76. Click to edit Master title style
Library & References
• Integrating Tivoli System Automation for Multiplatforms” (TSAMP)
with DB2 High Availability & Disaster Recovery (HADR) v9.5 (Garreth
Holl)
• http://www-01.ibm.com/support/docview.wss?uid=swg27016774&aid=1
• High Availability and Disaster Recovery Options for DB2 on Linux,
UNIX, and Windows
• www.redbooks.ibm.com/redbooks/pdfs/sg247363.pdf
• DB2 Best Practices – HADR
• http://www.ibm.com/developerworks/data/bestpractices/
76

77. Click to edit Master title style 77
Thank you for attending.
Any questions?