Downloaded 472 times
Uploaded byDavid Ware
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ Clusters
The document discusses managing workloads, scaling, and service availability using IBM MQ clusters. It emphasizes the importance of cluster design for high availability, addresses routing around failures, and elaborates on location dependency and the need to avoid interference in message distribution. Key points include balancing workloads, the significance of monitoring service failures, and strategies for disaster recovery.
In this document
Powered by AI
Managing workloads, scaling, and availability with IBM MQ clusters by David Ware.
Notice regarding IBM's statements about product plans, directions, and potential future products.
Overview of topics: cluster reasoning, service availability, location dependency, interference, and disaster recovery.
Exploring the rationale behind implementing clusters within the IBM MQ environment.
Illustrates how clients and services grow over time in a clustered environment.
Emphasizing workload balancing, service availability, and potential bottlenecks in message queues.
Management of message reallocation and ensuring service application availability during failures.
Monitoring tools to track service failures and ensure messages are redirected effectively.
Handling scenarios where client connections fail and implications for message replies.
Mechanism of name resolution for messages and how reply management is handled.
Addressing geographical distribution of services and how to maintain optimal traffic flow.
Structuring service queue managers for effective communication across geographical clusters.
Managing message traffic types within a cluster to prevent interference and ensure efficiency.
Introduce concepts of separate transmission queues and management strategies for message flow.
Discussing requirements for quick and efficient disaster recovery in MQ clusters.
Explaining various replication approaches for maintaining data integrity in disaster recovery.
Outline the totality of recovery needs including resources, application availability, and costs.
Summary of cluster management, service availability, geo-dependency, interference, and recovery strategies.
Time allocated for audience questions and answers following the presentation.
Detailed schedule of upcoming IBM MQ sessions including topics and speakers.
Links and contacts for further training and information on IBM MQ.
Legal disclaimers regarding the presentation content and IBM's responsibilities.
More Related Content
![[오픈소스컨설팅]클라우드기반U2L마이그레이션 전략 및 고려사항](https://cdn.slidesharecdn.com/ss_thumbnails/u2l-171114134117-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ Clusters
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ Clusters
- 1. A17: Managing workloads,scaling and availability with IBM MQ clusters David Ware, IBM © 2014 IBM Corporation
- 2. © 2014 IBMCorporation 2 Please Note IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
- 3. © 2014 IBMCorporation 3 Agenda Why a cluster? Starting to scale Service availability Routing around failures Location dependency Active active scenarios with ties to home Avoiding interference What are we sharing? When things go wrong Levels of DR and clustering implications
- 4. © 2014 IBMCorporation 6 Why a Cluster? 6
- 5. © 2014 IBMCorporation 7 Where it all begins… Client 1 QMgr Service 1
- 6. QQMMggrr QQMMggrr Service2 Service 1 © 2014 IBM Corporation 9 Over time… Client 1 Service 1 Client 2 Client 2 Client 3 Service 3 CClileienntt 1 1
- 7. © 2014 IBMCorporation 10 Over time… App 1 Service 1 CAlpiepn 1t 1 Client 2 Client 2 Client 3 Service 2 Service 1 QMgr QMgr QMgr QMgr Client 3 Client 1 Client 4 ACAplpipep n4 4t 4 Service 4 Service 3 SSeerrvvicicee 1 1 Service 3
- 8. QMgr QMgr ©2014 IBM Corporation 11 Until... App 1 Service 1 CAlpiepn 1t 1 Client 2 Client 2 Client 3 Service 2 Service 1 QMgr QMgr QMgr QMgr Client 3 Client 1 Client 4 ACAplpipep n4 4t 4 Service 4 Service 3 SSeerrvvicicee 1 1 Service 3
- 9. Zooming in onan application… QMgr QMgr © 2014 IBM Corporation 13 Service 1 ACAplpipep n1 1t 1 Client 2 Client 2 Client 3 Service 2 Service 1 QMgr QMgr QMgr QMgr Client 3 Client 1 Client 4 ACAplpipep n4 4t 4 Service 4 Service 3 SSeerrvvicicee 1 1 Service 3
- 10. Starting to scalehorizontally… QUEUE 1 © 2014 IBM Corporation 14 • Workload Balancing • Service Availability Service 1 ACAplpipep n1 1t 1 Service 1 QMgr QMgr QMgr QUEUE 1
- 11. © 2014 IBMCorporation 16 Availability
- 12. © 2014 IBMCorporation 17 • Target queues • Transmission queues Service 1 ACAplpipep n1 1t 1 Service 1 QMgr QMgr QMgr Where can the messages get stuck? Target queue Transmission queue
- 13. Service 1 Theservice queue manager/host fails © 2014 IBM Corporation 18 ACAplpipep n1 1t 1 Service 1 QMgr QMgr QMgr Message reallocation Unbound messages on the transmission queue can be diverted QMgr Locked messages Messages on the failed queue manager are locked until it is restarted Restart the queue manager Use multi-instance queue managers or HA clusters to automatically restart a queue manager Reconnect the service Make sure the service is restarted/reconnects to the restarted queue manager When a queue manager fails: • Ensure messages are not bound to it • Restart it to release queued messages
- 14. Service application availability © 2014 IBM Corporation 20 20
- 15. Unserviced messages Halfthe messages will quickly start to build up on the service queue QMgr QMgr Service 1 Service 1 The service application fails © 2014 IBM Corporation 21 QMgr Blissful ignorance This queue manager is unaware of the failure to one of the service instances ACAplpipep n1 1t 1 • Cluster workload balancing does not take into account the availability of receiving applications. • Or a build up of messages.
- 16. Service 1 ©2014 IBM Corporation 22 ACAplpipep n1 1t 1 Service 1 QMgr QMgr QMgr Monitoring for service failures
- 17. Service 1 •WebSphere MQ provides a sample monitoring service • Regularly checks for attached consuming applications • Updates the cluster workload balancing configuration to reflect this • So, what happens?... • Generally suited to steady state service applications © 2014 IBM Corporation 23 ACAplpipep n1 1t 1 Service 1 QMgr QMgr QMgr Monitoring for service failures QMgr QMgr Moving messages Any messages that slipped through will be transferred to an active instance of the queue Detecting a change When a change to the open handles is detected the cluster workload balancing state is modified Sending queue managers Newly sent messages will be sent to active instances of the queue
- 18. © 2014 IBMCorporation 26 Client failures
- 19. • Multiple locationsfor a client to connect to Service 1 •Allows new requests when one queue manager is unavailable. • What happens to replies after a failure? © 2014 IBM Corporation 27 ACAplpipep n1 1t 1 Service 1 Client availability QMgr QMgr QQMMggrr
- 20. Service 1 ©2014 IBM Corporation 28 ACAplpipep n1 1t 1 Service 1 Client host failure with an in flight request/response QMgr QMgr QMgr QMgr • Reply messages are bound to the originating queue manager, with no ability to redirect. • This may or may not be what’s required… Reply message bound The reply message will be locked to that outbound queue manager Request message Typically a request message will fill in the reply ReplyToQmgr based on the outbound queue manager
- 21. Name resolution Outgoingrequest resolves the ReplyToQ to be ‘REPLYQ’ and ReplyToQMgr to be ‘DUMMY’ Replying application Application replies to ‘REPLYQ’ on queue manager ‘DUMMY’ Service 1 DEF QREMOTE(DUMMY) RNAME(‘ ’) RQMNAME(‘ ’) Name resolution Target queue manager ‘DUMMY’ is resolved to ‘ ’, allowing cluster resolution to occur © 2014 IBM Corporation 29 DEF QLOCAL(REPLYQ) CLUSTER(CLUSTER1) DEF QREMOTE(REPLYQALIAS) RNAME(REPLYQ) RQMNAME(DUMMY) Requesting application Request message sets ReplyToQ to be ‘REPLYQALIAS’ and ReplyToQMgr to ‘ ’ ACAplpipep n1 1t 1 Service 1 Unlocking a response message QMgr QMgr QMgr QMgr DEF QLOCAL(REPLYQ) CLUSTER(CLUSTER1) DEF QREMOTE(REPLYQALIAS) RNAME(REPLYQ) RQMNAME(DUMMY) • Reply-to queue aliases and reply-to queue manager aliases can be used to blank out the outbound resolution of the ReplyToQMgr field. • But watch out, replies will now go to queue managers where the requestor may not be connected – this only suits some scenarios
- 22. Location Dependency ©2014 IBM Corporation 31 31
- 23. Service Service ©2014 IBM Corporation 32 Global applications QMgr QMgr QMgr QMgr Service Service QMgr AAppCplipe n 1t1 QMgr QMgr AAppCplipe n 1t1 QMgr New York London but separated by an ocean and 3500 miles • Prefer traffic to stay geographically local • Except when you have to look further afield • How do you do this with clusters that span geographies?…
- 24. © 2014 IBMCorporation 34 Setting this up
- 25. Service A A A © 2014 IBM Corporation 35 One cluster QMgr QMgr AAppCplipe n 1t1 New York London DEF QALIAS(AppQ) A LonQ • Clients always open AppQ • Local alias determines the preferred region • Cluster workload priority is used to target geographically local cluster aliases • Use of CLWLPRTY enables automatic failover •CLWLRANK can be used for manual failover Service AAppCplipe n 1t1 TARGET(NYQ) DEF QALIAS(NYQ) TARGET(ReqQ) CLUSTER(Global) CLWLPRTY(9) AppQ NYQ ReqQ QMgr AppQ A LonQ QMgr NYQ ReqQ A DEF QALIAS(AppQ) TARGET(LonQ) DEF QALIAS(LonQ) TARGET(ReqQ) CLUSTER(Global) CLWLPRTY(4) DEF QALIAS(LonQ) TARGET(ReqQ) CLUSTER(Global) CLWLPRTY(9) DEF QALIAS(NYQ) TARGET(ReqQ) CLUSTER(Global) CLWLPRTY(4)
- 26. Service Service QMgrQMgr QMgr QMgr Service Service The two cluster alternative © 2014 IBM Corporation 36 QMgr AAppCplipe n 1t1 QMgr QMgr AAppCplipe n 1t1 QMgr New York London USA QMgr QMgr QMgr QMgr EUROPE • The service queue managers join both geographical clusters •Each with separate cluster receivers for each cluster, at different cluster priorities. Queues are clustered in both clusters. • The client queue managers are in their local cluster only.
- 27. Avoiding interference ©2014 IBM Corporation 38 38
- 28. © 2014 IBMCorporation 39 AAppCplipe n 1t1 SSeerrvvicicee AAppCplipe n 1t1 SSeerrvvicicee AAppCplipe n 1t1 SSeerrvvicicee AAppCplipe n 1t1 AAppCplipe n 1t1 SSeerrvvicicee Real time queries Big data transfer Audit events The cluster as a pipe • Often a WebSphere MQ backbone will be used for multiple types of traffic
- 29. © 2014 IBMCorporation 40 The cluster as a pipe AAppCplipe n 1t1 SSeerrvvicicee AAppCplipe n 1t1 SSeerrvvicicee AAppCplipe n 1t1 SSeerrvvicicee AAppCplipe n 1t1 AAppCplipe n 1t1 SSeerrvvicicee QMgr QMgr QMgr QMgr QMgr QMgr Channels • Often a WebSphere MQ backbone will be used for multiple types of traffic • When using a single cluster and the same queue managers, messages all share the same channels • Even multiple cluster receiver channels in the same cluster will not separate out the different traffic types
- 30. Cluster Cluster ©2014 IBM Corporation 42 The cluster as a pipe AAppCplipe n 1t1 SSeerrvvicicee AAppCplipe n 1t1 SSeerrvvicicee AAppCplipe n 1t1 SSeerrvvicicee AAppCplipe n 1t1 AAppCplipe n 1t1 SSeerrvvicicee QMgr QMgr QMgr QMgr QMgr QMgr Cluster Channels Channels Channels • Often a WebSphere MQ backbone will be used for multiple types of traffic • When using a single cluster and the same queue managers, messages all share the same channels • Even multiple cluster receiver channels in the same cluster will not separate out the different traffic types • Multiple overlaid clusters with different channels enable separation
- 31. QQMMggrr QMgr Workloadbalancing level interference Service 1 © 2014 IBM Corporation 44 Client 1 Service 1 QMgr Service 2 Client 2 Cluster workload balancing is at the channel level. Messages sharing the same channels, but to different target queues will be counted together. The two channels here have an even 50/50 split of messages… …but the two instances of Service 1 do not! Split Service 1 and Service 2 queues out into separate clusters, queue managers or customise workload balancing logic. x75 x100 x50 x75 x25 x50 Multiple applications sharing the same queue managers and the same cluster channels. x75 x150
- 32. © 2014 IBMCorporation 45 QMgr Cluster transmit queue Separation of Message Traffic With a single transmission queue there is potential for pending messages for cluster ChannelA to interfere with messages pending for cluster ChannelB Management of messages Use of queue concepts such as MAXDEPTH not useful when using a single transmission queue for more than one channel. Monitoring Tracking the number of messages processed by a cluster channel currently difficult/impossible using queues. Performance? In reality a shared transmission queue is not always the bottleneck, often other solutions to improving channel throughput (e.g. multiple cluster receiver channels) are really what’s needed, but sometimes it will help. A much requested feature… Multiple cluster transmission queues QMgr QMgr QMgr QMgr V7.5 V8 Distributed z/OS & IBM i
- 33. Multiple cluster transmitqueues: Automatic Configured on the sending queue manager, not the owners of the cluster receiver SYSTEM.CLUSTER.TRANSMIT.ChlB SYSTEM.CLUSTER.TRANSMIT.ChlC ChlB © 2014 IBM Corporation 46 channel definitions. Queue Manager switch to automatically create a dynamic transmission queue per cluster sender channel. ALTER QMGR DEFCLXQ( SCTQ | CHANNEL ) Dynamic queues based upon model queue. SYSTEM.CLUSTER.TRANSMIT.MODEL Well known queue names. SYSTEM.CLUSTER.TRANSMIT.<CHANNEL-NAME> QMgr ChlB QMgr QMgr ChlA SYSTEM.CLUSTER.TRANSMIT.ChlA ChlA ChlC ChlC
- 34. Still configuredon the sending queue manager, not the owners of the cluster receiver Pink.B QMgr QMgr Pink.B Pink.A PINK.XMITQ © 2014 IBM Corporation 47 QMgr channel definitions. Administratively define a transmission queue and configure which cluster sender channels will use this transmission queue. DEFINE QLOCAL(GREEN.XMITQ) CLCHNAME(GREEN.*) USAGE(XMITQ) Set a channel name pattern in CLCHNAME Single/multiple channels (wildcard) E.g. all channels for a specific cluster (assuming a suitable channel naming convention!) Any cluster sender channel not covered by a manual transmission queue defaults to the DEFCLXQ behaviour Pink.A Green.A GREEN.XMITQ Green.A Multiple cluster transmit queues: Manual
- 35. When things gowrong Disaster Recovery © 2014 IBM Corporation 48
- 36. Everything? Applications mustbe able to connect and target the exact same MQ resources. Every existing, in-flight, message must be processed whilst in DR mode. New messages must be processed whilst in DR mode. © 2014 IBM Corporation 49 What do we need to recover? The MQ resources? Applications must be able to connect and target the exact same MQ resources. New messages must be processed whilst in DR mode. Application availability? Applications must be able to connect and target equivalent MQ resources. New messages must be processed whilst in DR mode.
- 37. Datacenter 1 ©2014 IBM Corporation 51 Datacenter Replication ‘in step’ Synchronous Replication 1 QMgr 2 QMgr DB DB Datacenter 2 QMgr QMgr QMgr DB DB QMgr QMgr QMgr QMgr Outside the datacenter, DR queue managers must be locatable in either site IP switching layer or comma separated channel names QMgr 1 2
- 38. Datacenter 1 1 QMgr 2 QMgr 1 2 REFRESH REFRESH © 2014 IBM Corporation 53 Datacenter Replication ‘asynchronous’ Asynchronous Replication/one-time copy Datacenter 2 QMgr QMgr QMgr QMgr QMgr QMgr Backup queue managers contain historic messages (if any). Cluster state will also be historic. Refreshing cluster state on failover and failback is essential. – That’s why we moved the full repository QMgr QMgr
- 39. Datacenter 1 QMgr 1 QMgr 3 © 2014 IBM Corporation 55 No Replication ‘warm standby’ QMgr 2 Datacenter 2 QMgr QMgr QMgr QMgr QMgr Backup queue managers are always running. Cluster workload balancing used to direct traffic to live queue managers (CLWLPRTY/RANK) Messages will be trapped on failed queue managers in the event of a failure. QMgr 4 Applications and system must be designed to accommodate this configuration (as we’ve discussed)
- 40. Live disk replication Restore nearly live backup Restore from stale backup Duplicated queue manager configuration How much do we need to recover? © 2014 IBM Corporation 57 Everything Live, cross center, WebSphere MQ clustering synchronous replication Asynchronous replication active/active COST MQ resources Application availability REFRESH
- 41. © 2014 IBMCorporation 58 Summary My first cluster Service availability Location dependency Avoiding interference When things go wrong
- 42. © 2014 IBMCorporation 59 Questions & Answers
- 43. IBM MQ Sessionsthis week 10:30 - 12:00 13:15 - 14:15 14:30 - 15:30 16:00 - 17:00 17:15 - 18:15 © 2014 IBM Corporation 60 Tuesday Opening General Session- IBM Digital Experience and WebSphere Technical University Session A31: IBM MQ CHLAUTH rules – with MQ V8 updates Speaker: Morag Hughson Room 02 Session A4: WebSphere MQ for z/OS: Performance and Accounting Speaker: Alexander Ross Room 8 Session I26: DataPower- MQ Connectivity Deep Dive (Theory) Speaker: Robin Wiley Room 27 Session Z1: WebSphere MQ for z/OS V8: Latest Features Deep Dive Speaker: Damon Cross Room 6 9:00 - 10:00 10:30 - 11:30 11:45 - 12:45 14:00 - 15:00 15:15 - 16:15 16:45 - 17:45 Wednesday Session Z5: WebSphere MQ for z/OS: Security Speaker: Damon Cross Room 02 Session A21: What's New in IBM Messaging Speaker: Morag Hughson Room 8 Session C7: Messaging in the Cloud with IBM MQ Light and IBM Bluemix Speaker: Rob Nicholson Room 27 Session A17: Managing work-loads, scaling and availability with IBM MQ clusters Speaker: David Ware Room 6 Lab IL5: DataPower-MQ Connectivity Deep Dive (Hands-On) Speaker: Robin Wiley Room 7b Session A9: WebSphere MQ for z/OS: The Inside Story Speaker: Damon Cross Room 6 Thursday Session A35: How to Develop Responsive Applications with IBM MQ Light Speaker: Rob Nicholson Room 27 Session A22: New IBM MQ V8 Security Features Speaker: Morag Hughson Room 01 Session A3: WebSphere MQ for z/OS: Shared Queues Speaker: Alex Ross Room 6 Session A18: Using Publish /Subscribe with IBM MQ Speaker: David Ware Room 27 Friday Lab AL6: Developing a First Application with IBM WebSphere MQ Light Speakers: Robert Nicholson, Alex Ross Room 7b Session A16: Using IBM MQ Pub/Sub in an MQ network Speaker: David Ware Room 6
- 44. For Additional Information © 2014 IBM Corporation 61 IBM Training http://www.ibm.com/training IBM WebSphere http://www.ibm.com/software/websphere/ http://www.ibm.com/software/products/ibm-mq IBM developerWorks http://www.ibm.com/developerworks/websphere https://www.ibm.com/developerworks/community/blogs/messaging WebSphere forums and community http://www.ibm.com/developerworks/websphere/community/
- 45. Legal Disclaimer •© IBM Corporation 2014. All Rights Reserved. • The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. • References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. • All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. © 2014 IBM Corporation 62