Session Type : Breakout Session Date/Time : Thu, 26-Feb, 10:30 AM-11:30 AM Venue : Mandalay Bay Room : Surf Ballroom E Descriptions: Active-Active is the target model of modern data center, its successfully adoption includes not only the mainframe, but also the heterogeneous and periphery distributed platforms which makes it much complex to implement. Data synchronization is the heart in the various technologies of active-active, which messaging technology been chose in its implementation. This session gives an overview of active-active technologies on both z and distributed platforms; highlight how does the Active-Active gives the benefits of both high availability and workload balancing, we also discuss China customer cases to implement messaging based active-active.

1. Enable Active-Active Enterprise Messaging Technology
to extend workload balancing and high availability
Session AME-1934
© 2015 IBM Corporation
Wang Bo - IBM CDL
wangbowb@cn.ibm.com

2. Agenda
• Concepts of Business Continuity
• Business Continuity
• High Availability
• Continuous Serviceability
• Continuous Availability Cross Sites
• Messaging Technologies for Business Continuity
• Cases Sharing
1

3. What does business continuity mean to you?
• Why we need to have a business continuity plan (BCP)?
• Don’t panic in the event of disaster crisis
• What we need to consider when preparing a BCP?
• "backups" and their locations
• a central command center, which we call it as "Crisis
Management Team (CMT)" in IBMManagement Team (CMT)" in IBM
• maintain a "contact list“
• think about all possible "scenarios" and their corresponding
action plans
• consider "critical" information or applications first
2

4. Different levels of business continuity
• Enterprise Business Requires Business Continuity
Standby Active-Active
0. Disaster 1. High-Availability 2. Continuous 3. Continuous
Recovery
• Restore the
business after
a disaster
• Meet Service
Availability objectives
e.g., 99.9% availability
or no more than 8
hours of down-time a
year for maintenance
and failures
serviceability
• No downtime
within one data
center (planned
or not)
3
Availability cross
sites
• No downtime ever
(planned or not)

5. BC Level 1 – High Availability
• HA at different levels （AIX example)
• Apps follow HA principles
• Middleware HA technologies
– Clustering, DB2 pureScale, MQ multi-ins
• OS HA technologies
– PowerHA (HACMP)
• Hardware HA technologies
– Disk redundancy (RAID, SDD, etc)
–– FlashCopy, Metro/Global mirror
– Server redundancy (CPU, power, etc)
– Network redundancy
• Key point is eliminating SPOF
• Redundancy
• RPO = 0!
4

6. BC Level 2 – Continuous Serviceability
• Usually based on workload take over
• Automatically take over
• A challenge for application affinity and sequence
• Decoupling of components – easier maintenance
• Old data may be lost – could combine with HA
• Maintenance• Maintenance
• Planed and unplanned downtime
• Rolling updates
• Coexistence
• Short RTO !
5

7. BC Level 3 – Continuous Availability Cross Sites
• Two or more sites, separated by unlimited distances, running
the same applications and having the same data to provide
cross-site workload balancing and Continuous Availability /
Disaster Recovery
• Customer data at geographically dispersed sites kept in sync
via synchronization
GDPS/PPRC GDPS/XRC or GDPS/GM Active/ActiveGDPS/PPRC GDPS/XRC or GDPS/GM Active/Active
Failover model Failover model Near CA model
Recovery time = 2 minutes Recovery time < 1 hour Recovery time < 1 minute
Distance < 20 KM Unlimited distance Unlimited distance
CD1SOURCE
CD1TABLE
CD1SOURCE
CD1TABLE
CD1SOURCE
CD1TABLE
CD1SOURCE
CD1TABLE
CD1SOURCECD1SOURCE
CD1TABLECD1CD1TABLE
CD1SOURCE
CD1TABLE
CD1SOURCE
CD1TABLE
CD1SOURCECD1SOURCE
CD1TABLECD1CD1TABLE
6
• Care about both
RPO & RTO!

8. Workload Balancing Through Data Replication
• Both sides run workload simultaneously, may with same or
different volumes. But both have the full picture of data!
• Replicate data from one platform to another
• Both sides may work equally, or have different focus, like below:
• Main server still do the existing critical work.
• Meanwhile, the offloaded server can run data analysis, query data, etc.
• New business requirements, but don’t want to touch the existing server!• New business requirements, but don’t want to touch the existing server!
• When purchase a new organization, may involve a different database
on a different platform. How to centralize the data?
Site A Site B
Synchronization
OLTP QLTP
Powerful
Critical production work
(DB updates/inserts)
Strict maintenance
process
Cautions: Nobody
wants it down
Less powerful
Less critical work
(DB queries)
Work can be delayed,
but may cost high CPU
(Data analysis, credit
card anti-fraud, etc)
New workloads
7

9. Agenda
• Concepts of Business Continuity
• Messaging Technologies for Business Continuity
• HA Technologies
• Continuous Serviceability Technologies
• Continuous Availability Cross Sites
• Cases Sharing• Cases Sharing
8

10. MQ Technologies
• HA Technologies
• QSG for MQ on zOS
• Failover Technologies
• Application HA
• Continuous Serviceability Technologies
• MQ Clustering
• Rolling Upgrade• Rolling Upgrade
• Continuous Availability Cross Sites
• Data Synchronization
• Synchronization Application Design
• How To Replicate Data
• Performance Consideration
9

11. HA - QSG for MQ on z/OS
Queue
manager
Private
queues
Queue
manager
Private
queues
Coupling facility failure
Queue
manager
Private
queues
Queue
manager
Private
queues
Nonpersistent
messages on
private queues
OK (kept)
Queue manager failure
10
Queue
manager
Private
queues
Shared
queues
Messages on
shared queues
OK (kept)
Nonpersistent
messages on
shared queues
lost (deleted)
Queue
manager
Private
queues
Shared
queues
Messages on
shared queues
OK (kept)
Nonpersistent
messages on
private queues
lost (deleted)
Persistent
messages on
shared queues
restored from log

12. HA - Failover Technologies
• Failover
• The automatic switching of availability of a service
• Data accessible on all servers
• Multi-instance queue manager
• Integrated into the WebSphere MQ product
• Faster failover than HA cluster
• Runtime performance of networked storage• Runtime performance of networked storage
• More susceptible to MQ and OS defects
• HA cluster
• Capable of handling a wider range of failures
• Failover historically slower, but some HA clusters are improving
• Some customers frustrated by unnecessary failovers
• Extra product purchase and skills required

13. HA - Application Availability
• Application environment
• Dependencies like a specific DB, broker, WAS?
• machine-specific or server-specific?
• Start/stop operations – sequence?
• Message loss
• Really need every message delivered?
• Application affinities• Application affinities
• MQ connectivity
12
QM1
MQ Client
Application
QM3
QM2

14. App 1App 1Client 1
Gateway
QMgr
QMgr
Site 1
Continuous Serviceability – MQ Cluster
• Workload Balancing
• Service Availability
• Location Transparency (of a kind)
Service 1
Client 1
Service 1
QMgr
Site 2
13

15. QMgr QMgr
Service Service
QMgr
QMgr
App 1App 1Client
New York
but separated by an ocean and 3500 miles
Global applications
Multi - Data Center using MQ Cluster
QMgr QMgr
Service Service
QMgr
QMgr
App 1App 1Client
London
• Prefer traffic to stay geographically local
• Except when you have to look further afield
• How do you do this with clusters that span geographies?…
14

16. QMgr
Service
QMgr
App 1App 1Client
New York
DEF QALIAS(AppQ)
TARGET(NYQ)
DEF QALIAS(NYQ)
TARGET(ReqQ)
CLUSTER(Global)
CLWLPRTY(9)
AppQ NYQ
ReqQ
A A
LonQ
A
DEF QALIAS(LonQ)
TARGET(ReqQ)
CLUSTER(Global)
CLWLPRTY(4)
Set this up – The one cluster solution
London
• Clients always open AppQ
• Local alias determines the preferred region
• Cluster workload priority is used to target geographically local cluster aliases
• Use of CLWLPRTY enables automatic failover
•CLWLRANK can be used for manual failover
Service
App 1App 1Client
QMgr
AppQ
A
QMgr
NYQ
ReqQ
A
LonQ
A
DEF QALIAS(AppQ)
TARGET(LonQ)
DEF QALIAS(LonQ)
TARGET(ReqQ)
CLUSTER(Global)
CLWLPRTY(9)
DEF QALIAS(NYQ)
TARGET(ReqQ)
CLUSTER(Global)
CLWLPRTY(4)
15

17. QMgr QMgr
Service Service
QMgr
QMgr
App 1App 1Client
New York
USA
QMgr
QMgr
Set this up - The two cluster solution
QMgr QMgr
Service Service
QMgr
QMgr
App 1App 1Client
London
EUROPE
QMgr
QMgr
• The service queue managers join both geographical clusters
•Each with separate cluster receivers for each cluster, at different cluster priorities. Queues are clustered in both clusters.
• The client queue managers are in their local cluster only.
16

18. Continuous Availability Cross Sites
• Data Synchronization is the key component in Active-Active
• Capture transaction change in real-time
• Publish the change in high performance with low latency
• Messaging based implementation is proven to be the simplest
way among kinds of methods of data transmission
• A high performance, reliable messaging product is needed for
the following requirements:
• Simplifies application development
• Ease of use
• Assured message delivery
• High Performance and Scalability
• Easy of Management
17

19. Active-Active Common Model based on Messaging
Workload Distributor
•Cross Site Workload Distribution
•Data synchronization
•Reply on high performance, reliable messaging transmission
•Flexible application design
•Automation & Management
Business
App
Business
Data
Sync
App
Messaging
Sync
App
Messaging
Business
App
Business
Data
Sites at a distance
18

20. How to replicate data?
• Capture transaction activities through DB2 logs – an independent tool
Log-based
Capture
WebSphere MQ
Source Target
Highly parallel
Apply
Q
Capture
Q
Apply
• Modify the existing applications – Send out transactional data with MQ
API
• At the end of existing logic, add MQPUT call to send the data. Program an
apply application at the target end.
• Flexible, can cross different platforms, even different database products. But
need a robust application.
• Option to choose within or without syncpoint. – Will the existing transaction
fail(roll back) if the send fails?
WebSphere MQ
Q-Replication
19

21. Performance Tuning Considerations
• Synchronize only the changed data, thus reduce the data
volume
• Introduce more parallelism
• Multiple synchronization channels for different type of workload
• More threads in sync application for parallel processing
• Multiple MQ channels to leverage single channel busy problem• Multiple MQ channels to leverage single channel busy problem
• Invest to use MQ new feature
• Bigger buffer pools above the bar
• Sequential pre-fetch
• Page set read/write performance enhancement
• Channel performance improvement
20

22. MQ Buffer pools read ahead enhancement
• Symptom : When the number of messages overruns the buffer
pool allocated for the queue, messages are spilled to disk and
must then be retrieved from disk.
• The read ahead enhancement enables message pre-fetch from
disk storage and improves MQGET performance.
• Available in PM PM63802/UK79853 in 2012 and PM81785/
UK91439 in 2013.
• Internal testing shows ~50% improvement with read ahead
enabled (msglen=6KB).
• Enable this feature if MQ buffer pool may overrun.
21

23. Agenda
• Concepts of Business Continuity
• Messaging Technologies for Business Continuity
• Cases sharing
• Case 1 (Active/Active with QREP tool )
• Case 2 (Active/Active with application)
• Case 3 (Workload offload )• Case 3 (Workload offload )
• Case 4 (Workload offload to multiple systems)
22

24. Beijing data center:
For disaster recovery
Requirements of a bank – Active/Active
• A commercial bank - data centers in Shanghai and Beijing
• Beijing: One existing data center for disaster recovery
• Shanghai: One existing data center for production, and one new data center for Active-
Active. 70 km between two data centers
• This bank plans to achieve Active-Active between two data centers in Shanghai for core banking
business.
rows/s MB/s
OLTP 45K-50K 45
Batch 140K 50
Month-End Batch 130K 70-80
1200 km
70 km
For disaster recovery
Shanghai data center 1
Production center
Shanghai data center 2
23
Month-End Batch 130K 70-80
Interest Accrual Batch 440K 172.5

25. MQ in Q Replication
• Part of the InfoSphere Data Replication product
• A software-based asynchronous replication solution
• For Relational Databases
• Changes are captured from the database recovery log; transmitted as
(compact) binary data; and then applied to the remote database(s) using SQL
statements.
• Leverages WebSphere MQ for Staging/Transport
• Each captured database transactions published in an MQ message (messages
sent at each commit interval)sent at each commit interval)
• Staging makes it possible to achieve continuous operation even when the target
database is down for some time or the network encounter some problem.
24
DB2
Control Tables
Site A
DB2
Control tables
Q Capture
Q Apply
agent
agent
agentUser
tables
database
recovery
log
User
tables
Unlimited
Distance
Site B
Configuration &
Monitoring
logrdr publish
Data CenterWebSphere MQ
DB2 Transaction
Parallel Replay
Asynchronous
LOG change
data capture
Active DB2Active DB2 Persistent
Staging
Capability
SQL
statements

26. MQ v8.0 features for Q Rep scenarios
• Sequential pre-fetch on z/OS
• The TUNE READAHEAD(ON), TUNE RAHGET(ON) delivered to
the bank as PTF in V71 and still applicable to V8
• Pageset read/write performance enhancements for QREP on z/OS
• Changes to the queue manager deferred write processor. Now it’s
the default behavious in the V8
• 64-bit enablement of buffer pools on z/OS
• More real storages can be used as buffers
• SMF Enhancements on z/OS
• Chinit SMF helps on tuning channel performance
• 64-bit log RBA
• We probably want QREP users to get to this
• Other improvement
• z/OS miscellaneous improvements (performance and serviceability)
• Channel performance on z/OS
25

27. Case 2(Active/Active with application)
• Active-Active Adaptability in Small/Medium-sized Banks
• China banks have setup storage based DR solution, but the
business recovery time is too long
• Sysplex solution is expensive, and input-output ratio is not high.
The distance is also limited.
• Need to consider application based solution, and mix with the
storage based solutionstorage based solution
• Active-Active is the target model of modern data center
• Not only for mainframe, but heterogeneous and
periphery distributed platform also need to be active-active
26

28. Business Requirement of Active-Active
• Credit card system on mainframe is based on the VisionPlus
(V+) solution by First Data.
• Improve the capacity and availability of the whole credit card
system.
• More comprehensive and more efficient services by payment
systems of the banks.systems of the banks.
• More flexibility accesses, more comprehensive functions of
liquidity risk management, extension of the scope of system
monitoring
• Refinement of backup infrastructure
27

29. The target Active-Active System Structure
• Both the main system and the secondary system are active
• Real data synchronization for OLTP transactions
• The main system and the secondary system backup each other
• Workload can be taken over in case of planned or unplanned failure
File TransferOLTP Batch Terminal Anti-fraud Reporting Debt-collection
2. File Transfer
(Secondary) V+ Mainframe
Batch Processing
(Main) V+ Mainframe
Batch Processing
DRNET
Headquarter
Gateway
Finance
Processing in BJ
Finance
Processing in SH
OLTP Processing
OLTP Processing
VISA/MC/JCB .
Non-Finance
Processing
3. Global Mirror
files
Workload Split by
Card BIN, and send to
BJ and SH
1.OLTP Transaction (MQ)
28

30. Active-Active Deployment Model
Continuous Availability
– Active-Active
Encryption
Core
Data
Beijing Business Continuous Availability
Achieve Business Continuous AvailabilityAchieve Business Continuous Availability
by front end and mainframe activeby front end and mainframe active--activeactive
Reliable Services
Synchronize application data based onSynchronize application data based on
Headquarter Gateway
(Route by BIN)
Encryption
Front-end
App System
Core
Data
Sync
Sync
Shanghai
Front-end App
System(Main)
Synchronize application data based onSynchronize application data based on
MQ reliable messaging, keep dataMQ reliable messaging, keep data
consistency in real timeconsistency in real time
Data Backup
Backup key business data through MQBackup key business data through MQ
seriesseries
Data interchange in real time
The data centers could be located in longThe data centers could be located in long
distancedistance
29

31. Active-Active Logical model for OLTP
• Self implemented replication service based on WebSphere MQ
for z/OS
Beijing Site Shanghai Site
Credit Card System
Credit Card System
Workload
Distributor
MQ queue manager 1
send
VSAM
AOR
Transaction
Publisher
VSAM
Transaction
Replay
retrieve
MQ queue manager 2
AOR
Transaction
Publisher
Transaction
Replay
retrieve
send
Credit Card System
30

32. Planned Site Switch Over Procedure
• Stop workload routing to BJ site
• Waiting for SH site duplex as BJ site data
• Workload re-rout to SH site
• Reverse GM from site B to site A
31

33. Unplanned Site Switch Over Procedure
• Stop workload routing to BJ site
• Workload re-rout to SH site
• Reverse GM from site B to site A
32

34. Characteristics of this case
• For business which has less complex master data with less
dependent database tables. For example, Credit Card business.
• The synchronization applications need to be developed
according to your business and technical requirements, rather
than an out-of-box product.than an out-of-box product.
33

35. Case 3 (Workload offload )
• Purpose
• A new business – SELECT frequently.
• Existing DB2 on zOS, but wants to buy an existing solution on Linux.
• So this is an active-active data replication within the same data center,
cross platform.
• Implementation
• Modify the existing core banking applications + Send with MQ logic at
the end.
• On the distributed side, develop another application for DB
updates/inserts.
• Minimize the impact on the existing applications - out of syncpoint.
34

36. Workload offload
• Easier and Faster expand the business
• The existing business is slight touched (nearly untouched).
• Flexible, no dependencies on the type of target database.
Workload
Distributor
35
Core banking system(zOS)
zOS System
Core
Workload
Standby
Linux System
Query
Workload
Active
QUERY system(Linux)
App Logic:
• Existing logic
• MQPUT (data to update in DB)
• EXEC CICS SYNCPOINT
Apply
Application
App Logic:
• According to the data received,
update target with SQL statement
or SP
MQ Channel
zOS MQ Linux MQ

37. Case 4 (Workload offload to multiple systems)
• Purpose
• Replicate zOS database of core credit card system to a Linux
database in a near real time window. There are multiple consumers on
different Linux boxes want the same data.
• Implementation
• zOS MQ dose a normal put(same as the data replication discussed in
previous pages), only one copy of data is transferred to Linux MQ.
Then this MQ dose the 1-n publication with the MQ pub/sub engine.Then this MQ dose the 1-n publication with the MQ pub/sub engine.
36
MQPUT(/credit/deposit/)
CICS/Batch
QM on zOS （（（（QM1）））） QM distributed （（（（QM2））））
SUB2.Q
APP1.SUB
SUB1.Q
MQGET or
Remote QMGR
APP2.SUB
Cluster XMITQ
Or XMITQ(hierarchy)
……
10 Subs in total
MQGET or
Remote QMGR

38. Detailed implementation on pub/sub + HA
PublisherPublisherPublisher
MQ cluster
QM0A QM0B
QMGW01 QMGW03 QMGW02
QM0A/QM0B:
DEFINE TOPIC(MYTOPIC) TOPICSTR('/Price/Bread')
DEFINE QALIAS(MYTARGET) TARGET(MYTOPIC)
TARGTYPE(TOPIC) CLUSTER(CL0)
Duplicated Apps(On gateway QMGRs):
Just put messages to queue 'MYTARGET', the cluster
will use work-load balancing logic to route them to
either QM0A or QM0B.
MQPUT
CLWLPRTY = 7CLWLPRTY = 5
37
Hierarchy
App
5 Subscription
App
5 Subscription
App
5 Subscription
App
5 Subscription
QM01 QM02 QM03 QM04
TARGTYPE(TOPIC) CLUSTER(CL0)
QM01/QM02/QM03/QM04:
ALTER QMGR PARENT(QM0A)
/* For QM03/QM04, the parent is QM0B */
DEFINE QL(MYTARGETQ1)
DEFINE QL(MYTARGETQ2)
DEFINE QL(MYTARGETQ3)
DEFINE QL(MYTARGETQ4)
DEFINE QL(MYTARGETQ5)
DEFINE SUB(SUB01) TOPICSTR('/Price/Bread')
DEST(MYTARGETQ1)
DEFINE SUB(SUB02) TOPICSTR('/Price/Bread')
DEST(MYTARGETQ2)
DEFINE SUB(SUB03) TOPICSTR('/Price/Bread')
DEST(MYTARGETQ3)
DEFINE SUB(SUB04) TOPICSTR('/Price/Bread')
DEST(MYTARGETQ4)
DEFINE SUB(SUB05) TOPICSTR('/Price/Bread')
DEST(MYTARGETQ5)

39. 38

40. Notices and Disclaimers
Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or
transmitted in any form without written permission from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been
reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM
shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY,
EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF
THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT
OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the
agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without
notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are
presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual
performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products,
programs or services available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not
necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither
intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal
counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s
business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or
represent or warrant that its services or products will ensure that the customer is in compliance with any law.

41. Notices and Disclaimers (con’t)
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products in connection with this
publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to
interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED,
INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any
IBM patents, copyrights, trademarks or other intellectual property right.
• IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document
Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand,Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand,
ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™,
PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®,
pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®,
urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of
International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and
service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on
the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

42. Thank You
Your Feedback is
Important!
Access the InterConnect 2015Access the InterConnect 2015
Conference CONNECT Attendee
Portal to complete your session
surveys from your smartphone,
laptop or conference kiosk.