SlideShare a Scribd company logo
1 of 42
Download to read offline
HHM-2833: Where is My Message?: Using
IBM MQ Tools to Work Out What Applications
Have Done
Matt Leming – lemingma@uk.ibm.com
IBM Hursley - UK
Symptoms include:
• Timeouts for request/response messaging
• Inconsistent data between front-end and back-end systems
• Customer complaints about ‘missing’ transactions
1
Agenda – the MQ Toolbox
• Who’s connected?
• Are messages flowing?
• Where are messages going?
• What are the apps doing?
• How can I look back in time? Presentation contains MQSC
example commands and output.
Your own admin tools will
vary by task, situation and
preference. Tasks can be
performed in PowerShell, MQ
Explorer etc. as desired.
2
Who’s connected?
DISPLAY CONN
DISPLAY CONN(*) TYPE(HANDLE) ALL DIS CONN(*) ALL
AMQ8276: Display Connection details.
CONN(577C425321295301)
EXTCONN(414D5143474154455741593120202020)
TYPE(HANDLE)
OBJNAME(WLMMDB.REQUEST) OBJTYPE(QUEUE)
ASTATE(NONE) HSTATE(INACTIVE)
OPENOPTS(MQOO_OUTPUT,MQOO_FAIL_IF_QUIESCING)
READA(NO)
OBJNAME(SENDINGAPP.REPLY) OBJTYPE(QUEUE)
ASTATE(ACTIVE) HSTATE(ACTIVE)
OPENOPTS(MQOO_INPUT_SHARED,MQOO_INQUIRE,MQOO_SAVE_ALL_CONTEXT,MQOO_FAIL_IF_QUIESCING)
READA(NO)
AMQ8276: Display Connection details.
CONN(577C425321295301)
EXTCONN(414D5143474154455741593120202020)
TYPE(CONN)
PID(9740) TID(185)
APPLDESC(WebSphere MQ Channel) APPLTAG(jms/GATEWAY1_CF)
APPLTYPE(SYSTEM) ASTATE(NONE)
CHANNEL(WAS.CLIENTS) CONNAME(127.0.0.1)
CONNOPTS(MQCNO_SHARED_BINDING) USERID(pbroad)
UOWLOG( ) UOWSTDA(2014-04-08)
UOWSTTI(13.24.00) UOWLOGDA( )
UOWLOGTI( ) URTYPE(XA)
EXTURID(XA_FORMATID[DSAW]
XA_GTRID[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809]
XA_BQUAL[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809000000010000000000000000
000000000001])
QMURID(0.7940075) UOWSTATE(ACTIVE)
Use CONN to match TYPE(CONN) and
TYPE(HANDLE) records
TYPE(HANDLE) records let you find
applications by the objects they access.
See all open handles for an app in one place,
unlike DIS QSTATUS records
Long running UOW information.
XID can be tied up with app server txn timeout
Channel name + IP help identify client apps.
MQ V7.5 and later JMS clients can supply
an application name in the CF
4
DISPLAY CHSTATUS
DISPLAY CHSTATUS(*) ALL
AMQ8417: Display Channel Status details.
CHANNEL(WAS.CLIENTS) CHLTYPE(SVRCONN)
BUFSRCVD(17) BUFSSENT(13)
BYTSRCVD(2296) BYTSSENT(2456)
CHSTADA(2014-04-08) CHSTATI(15.26.59)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(127.0.0.1) CURRENT
EXITTIME(0,0) HBINT(5)
JOBNAME(0000260C000000B9) LOCLADDR( )
LSTMSGDA(2014-04-08) LSTMSGTI(15.26.59)
MCASTAT(RUNNING) MCAUSER(pbroad)
MONCHL(OFF) MSGS(6)
RAPPLTAG(jar) SSLCERTI(CN=ExampleCA,O=Example)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER(SERIALNUMBER=53:43:FD:D6,CN=ExampleApp1,O=Example)
SSLRKEYS(0) STATUS(RUNNING)
STOPREQ(NO) SUBSTATE(RECEIVE)
CURSHCNV(1) MAXSHCNV(1)
RVERSION(00000000) RPRODUCT(MQJM)
See SSLPEER information not in DIS CONN
JOBNAME contains PID (except z/OS): 0x260C = PID(9740)
On Linux/UNIX (not Win) TID matches CONN: 0xB9 = TID(185)
Note that multiple CONN might share one SVRCONN channel instance
Check suitable heartbeats are negotiated
5
Automatic detection if no-one is
connected
AMQSCLM
1) Normal operation
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
Cluster
WLM
Round-robin message distribution
App
server 1
App
server 2
App
server N
. . .
IPPROCS(>=1) IPPROCS(>=1) IPPROCS(>=1)
AMQSCLM AMQSCLM AMQSCLM
7
AMQSCLM
2) Application failure
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
Cluster
WLM
App
server 1
App
server 2
. . .
IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1)
Messages
start
building
up
AMQSCLM AMQSCLM AMQSCLM
App
server N
Round-robin message distribution
8
AMQSCLM
3) Failure Detection
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
Cluster
WLM
App
server 1
App
server 2
. . .
IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1)
AMQSCLM
regularly
checks the
queue and
detects
IPPROCS(0)
AMQSCLM AMQSCLM AMQSCLM
App
server N
Round-robin message distribution
9
AMQSCLM
4) Redirect Messages
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
Cluster
WLM
App
server 1
App
server 2
. . .
IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1)
AMQSCLM
reduces
queue priority
in cluster, to
stop new
messages
arriving
AMQSCLM AMQSCLM AMQSCLM
App
server N
Round-robin message distribution
10
AMQSCLM
5) Un-maroon Msgs (optional)
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
Cluster
WLM
App
server 1
App
server 2
. . .
IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1)
AMQSCLM
tells qmgr to
redistribute
‘stuck’
messages via
the cluster
AMQSCLM AMQSCLM AMQSCLM
App
server N
Round-robin message distribution
11
AMQSCLM summary
• The Cluster Queue Monitoring sample program (AMQSCLM)
• Shipped with the product as a sample
– Precompiled
– Source code
– Platforms except on z/OS
• More information here:
http://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.
mq.dev.doc/q024620_.htm
12
Are messages flowing?
Real-time/online Monitoring – Queues
• Set detail level for QMGR. Override for individual Queues
• Gives live view of application responsiveness
ALTER QMGR MONQ(MEDIUM)
ALTER QLOCAL(QUEUE1) MONQ(HIGH)
ALTER QLOCAL(QUEUE2) MONQ(OFF)
AMQ8450: Display queue status details.
QUEUE(QUEUE1) TYPE(QUEUE)
CURDEPTH(16) IPPROCS(3)
LGETDATE(2014-04-08) LGETTIME(17.05.59)
LPUTDATE(2014-04-08) LPUTTIME(17.12.16)
MEDIALOG( ) MONQ(HIGH)
MSGAGE(112) OPPROCS(5)
QTIME(10101414, 10101414) UNCOM(NO)
DIS QSTATUS(QUEUE1) ALL
Age in seconds of the oldest
message on the queue
Estimations of the time in microseconds that messages are waiting on the queue for
processing.
First value: Calculated from recent activity
Second value: Calculated from longer term activity
Timestamps of last PUT/GET
to check for recent activity
Without MONQ you only get the depth and
how many handles are open
14
Real-time/online Monitoring – Channels
• Gives live view of channel throughput
ALTER QMGR MONCHL(MEDIUM)
ALTER CHANNEL(CLUSTER1.QM1) CHLTYPE(CLUSRCVR) MONCHL(HIGH)
AMQ8417: Display Channel Status details.
CHANNEL(MQHUB.GATEWAY2) CHLTYPE(CLUSSDR)
BATCHES(52) BATCHSZ(50)
BUFSRCVD(55) BUFSSENT(1616)
BYTSRCVD(1748) BYTSSENT(1192330)
CHSTADA(2014-04-08) CHSTATI(17.49.03)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(127.0.0.1(1422)) CURLUWID(0107445310001B34)
CURMSGS(50) CURRENT
CURSEQNO(11823) EXITTIME(0,0)
HBINT(5) INDOUBT(YES)
JOBNAME(00002DA4000047D0) LOCLADDR(127.0.0.1(53557))
LONGRTS(999999999) LSTLUWID(0107445310001B33)
LSTMSGDA(2014-04-08) LSTMSGTI(17.49.51)
LSTSEQNO(11773) MCASTAT(RUNNING)
MONCHL(MEDIUM) MSGS(1580)
NETTIME(137538,29555) NPMSPEED(FAST)
RQMNAME(GATEWAY2) SHORTRTS(180)
SSLCERTI( ) SSLKEYDA( )
SSLKEYTI( ) SSLPEER( )
SSLRKEYS(0) STATUS(RUNNING)
STOPREQ(NO) SUBSTATE(RECEIVE)
XBATCHSZ(20,17) XMITQ(SYSTEM.CLUSTER.TRANSMIT.QUEUE)
XQMSGSA(112) XQTIME(545784,3929968)
RVERSION(07050002) RPRODUCT(MQMM)
DIS CHSTATUS(MQHUB.GATEWAY2) ALL
Short/long term calculations of how
full your batches are getting, to help
you tune BATCHSZ/BATCHINT
Last time a message was sent over
the channel
Depth of messages on XMITQ
for this channel (capped at 999)
Short/long term calculations of how
long messages are waiting on the
XMITQ for transmission
15
Accounting and Statistics Overview
• Monitoring data sent as a PCF message at a configured interval
– Statistics – scoped to a Queue / Channel / QMGR
– Accounting – scoped to an individual CONN and Queue / QMGR
• Applications must consume the messages
– Enterprise monitoring – Tivoli ITCAM / Tivoli OMEGAMON XE for Messaging
– Sample applications and SupportPacs
– Custom applications – com.ibm.mq.headers Java package
PCF accounting
collection app
Queues Channels
CONNs
PCF statistics
collection app
Statistics
Messages
Accounting
Messages
QMGR
SYSTEM.ADMIN.STATISTICS.QUEUE SYSTEM.ADMIN.ACCOUNTING.QUEUE
16
Taking a look with SupportPac MS0P
• Time period
• MQ Statistics at QMGR level
• Detailed queue statistics
17
Simple practical example using amqsmon
• Overall QMGR busyness
• Simple data format
– Multiple values are
[Persistent, NonPersistent]
• One message every X seconds
– Use amqsmon directly (perl/cron)
• Low/high water marks for
subscriptions
– Grouped by subscription type
ALTER QMGR STATMQI(ON)
Wait a bit
RESET QMGR TYPE(STATISTICS)
amqsmon -m GATEWAY1 -t statistics -a -w 0
MonitoringType: MQIStatistics
QueueManager: 'GATEWAY1'
IntervalStartDate: '2014-04-09'
IntervalStartTime: '00.00.35'
IntervalEndDate: '2014-04-09'
IntervalEndTime: '00.01.13'
CommandLevel: 700
ConnCount: 35
PutCount: [271, 0]
PutFailCount: 0
Put1Count: [2, 0]
Put1FailCount: 0
PutBytes: [273976, 0]
GetCount: [270, 0]
GetBytes: [269468, 0]
GetFailCount: 19
DurableSubscriptionHighWater: [0, 0, 0, 0]
DurableSubscriptionLowWater: [0, 0, 0, 0]
NonDurableSubscriptionHighWater: [0, 0, 0, 0]
NonDurableSubscriptionLowWater: [0, 0, 0, 0]
PutTopicCount: [0, 0]
PutTopicFailCount: 0
Put1TopicCount: [0, 0]
Put1TopicFailCount: 0
PutTopicBytes: [0, 0]
PublishMsgCount: [0, 0]
PublishMsgBytes: [0, 0]
18
Where are messages going?
Cluster
MQ networks can be complex
• At each of the dots stuck / mis-sent messages are possible
– MQOPENs of the wrong Queue / Queue Manager by apps
– Full queues
– Stopped channels
– Stopped apps
– Incorrectly configured QREMOTE/QALIAS routing objects
– Cluster membership problems
• The standard problem diagnosis approach
– Methodically checking channels/queues/DLQs at each point
• Is there anything to speed up this process?
App1
App2A
App2B
Requests Replies
Replies
SDR/RCVR
SDR/RCVR
20
Cluster
Trace Route
• MQ has the ability to inject tracer messages
– (Can be) hidden from applications
– Generate activity reports as they pass through
• Tools are available to trace routes using these reports
– dspmqrte – command line tool supplied with the product
– MS0P – Cat 2 SupportPac extension to MQ Explorer
– Report messages need to get back to your ReplyQ of your tool
• Lets you see the path messages could have taken
– Test connectivity through the MQ network
– Test cluster workload balancing
• Can quickly jump you close to the problem
– The point your trace message veers off in the wrong direction
– The point the trail goes cold
dspmqrte SDR/RCVR
SDR/RCVR
dspmqrte
dspmqrte
21
What are the apps doing?
Application Activity Tracing
• Similar infrastructure to Accounting & Statistics
– PCF messages on SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE
– Configurable via mqat.ini
• Can be changed without QMGR restart
• Configurable detail level can include partial/full message payload
• Frequency options for tuning
– Can be enabled on a per-application basis
• Via MQCONNX flags
• Via application name
– In MQ V7.5 and later JMS clients can supply an application
name in the connection factory
Health warning: Performance impact
http://ow.ly/vA8wB
23
Application Activity Tracing
• Enables scenarios such as
– Application audit trail
– Message duplication
– Resource usage
• Which queues or topics are actually being used
– Problem Determination
• Which Queue / Queue Manager is the application
actually opening
– Application Coding Standards
• Does everyone use the MQI in the recommended way
– And more …
Health warning: Performance impact
http://ow.ly/vA8wB
24
Looking at the data with the amqsact sample
ALTER QMGR ACTVTRC(ON) <- should be tuned via mqat.ini
amqsact -m GATEWAY1 -v
MQI Operation: 6
Operation Id: MQXF_PUT
ApplicationTid: 12451
OperationDate: '2014-04-09'
OperationTime: '01:39:48'
High Res Time: 1397003988665548
Completion Code: MQCC_OK
Reason Code: 0
Hobj: 18225032
Put Options: 139330
Msg length: 460
Recs_present: 0
Known_dest_count: 1
Unknown_dest_count: 0
Invalid_dest_count: 0
Object_type: MQOT_Q
Object_name: 'SENDINGAPP.REPLY'
Object_Q_mgr_name: 'GATEWAY1'
Resolved_Q_Name: 'SENDINGAPP.REPLY'
Resolved_Q_mgr: 'GATEWAY1'
Resolved_local_Q_name: 'SENDINGAPP.REPLY'
Resolved_local_Q_mgr: 'GATEWAY1'
Resolved_type: MQOT_Q
Report Options: 0
Msg_type: MQMT_DATAGRAM
Expiry: -1
Format_name: 'MQHRF2'
Priority: 4
Persistence: 0
Msg_id:
00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 '
00000010: 0207 4453 2007 2603 '..DS .&. '
Correl_id:
00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 '
00000010: 0207 4453 2007 2203 '..DS .". '
Reply_to_Q : ' ^D'
Reply_to_Q_Mgr: ' ^C'
Coded_char_set_id: 1208
Encoding: 273
Put_date: '20140409'
Put_time: '00394866'
Check the options used for coding standards
Check queue name resolution, to find out
why messages are going to the wrong place
Track individual messages and request/reply
scenarios with Msg_id and Correl_id
25
Extract from Report
MonitoringType: MQI Activity Trace
QueueManager: 'V71'
Host Name: 'rockall.hursley.ibm.com'
CommandLevel: 710
ApplicationName: 'WebSphere MQ Client for Java'
ApplicationPid: 18612354
UserId: 'mquser'
ConnName: '9.20.95.106'
Channel Type: MQCHT_SVRCONN
Platform: MQPL_UNIX
=======================================================================
=
Time Operation CompCode MQRC HObj (ObjName)
10:04:09 MQXF_INQ MQCC_OK 0000 2
10:04:09 MQXF_CLOSE MQCC_OK 0000 2
10:04:09 MQXF_OPEN MQCC_OK 0000 4 ()
10:04:09 MQXF_INQ MQCC_OK 0000 4
10:04:09 MQXF_CLOSE MQCC_OK 0000 4
10:04:09 MQXF_OPEN MQCC_OK 0000 4 (SYSTEM.DEFAULT.LOCAL.QUEUE)
10:04:09 MQXF_INQ MQCC_OK 0000 4
26
How can I look back in time?
What happened to my messages at 2am this
morning?
• Enterprise monitoring solution
– DLQ alerts, queue depth alerts, channel status alerts
– Unresolved running units of work
– Historical MQ monitoring, accounting and stats data
• App logs from the time of the problem
– Exceptions, MQ error codes, timeouts
• MQ error logs for all qmgrs that could have been involved
– Channel errors
– Authentication issues
• ??? – what else is there
29
What about the MQ Recovery Log?
• For persistent messages inside transactions
– MQ logs each operation performed
• Why can't we use this to
– Look back in time to 2am and see what happened?
– Recovery the original payload if the app lost the message?
– See what happened inside long-running units of work?
– Provide a list of operations within the failed business transaction?
• MQ documents how you can… if
– You use the text formatting tool provided with MQ (dmpmqlog)
– The logging is linear so the historical data is available in the tool
– You follow the right steps to extract data from running qmgrs
– You do the work to follow through the logs
30
Dmpmqlog output is readable, but analysis is tedious
• Wouldn't it be easier to let
the computer do the
tedious bit?
LOG RECORD - LSN <0:0:954:44817>
**********
HLG Header: lrecsize 873, version 1, rmid 0, eyecatcher HLRH
LogRecdType . . : AQM Put Message (257)
Eyecatcher . . : ALRH Version . . . . : 1
LogRecdLen . . : 853 LogRecdOwnr . . : 256 (AQM)
XTranid . . . . : TranType: XA
XID: formatID 1463898948, gtrid_length 36, bqual_length 54
gtrid [000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F2]
bqual
[000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F20000000100000000000
00000000000000001]
QueueName . . . : Not known
Qid . . . . . . : {Hash 2147211283, Counter: 5}
ThisLSN . . . . : <0:0:0:0>
PrevLSN . . . . : <0:0:954:43944>
Version . . . . : 4
MapIndex . . . : 199
PrevLink.Locn . : 102408 PrevLink.Length : 8
PrevDataLink . : {High 0, Low 103424}
Data.Locn . . . : 103424 Data.Length . . : 613
Data . . . . . :
00000: 41 51 52 48 04 00 00 00 FF FF FF FF FF FF FF FF AQRH....ÿÿÿÿÿÿÿÿ
00016: 00 00 00 00 00 00 00 00 C7 00 00 00 02 00 C0 01 ........Ç.......
00032: 00 00 00 00 04 00 01 00 A5 00 00 00 00 00 00 00 ........¥.......
00048: 63 00 00 00 41 4D 51 20 49 49 42 30 31 5F 51 4D c...AMQ IIB01_QM
00064: 20 20 20 20 D9 26 B0 52 20 08 53 F5 30 30 30 30 .&.R .Sõ0000
00080: 30 30 39 39 00 00 00 00 00 00 00 00 00 00 00 00 0099............
00096: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00112: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ................
00128: 00 00 00 00 FF FF FF FF 00 00 00 00 04 00 00 09 ....ÿÿÿÿ........
00144: 00 00 00 00 E3 ED 04 80 10 FD 67 E4 FF FF FF FF ....ãí.€.ýgäÿÿÿÿ
00160: 4D 44 20 20 01 00 00 00 00 00 00 00 08 00 00 00 MD ............
00176: 00 00 00 00 11 01 00 00 B8 04 00 00 4D 51 48 52 ........¸...MQHR
00192: 46 32 20 20 04 00 00 00 01 00 00 00 20 20 20 20 F2 ........
00208: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
00224: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
00240: 20 20 20 20 20 20 20 20 20 20 20 20 49 49 42 30 IIB0
00256: 31 5F 51 4D 20 20 20 20 20 20 20 20 20 20 20 20 1_QM
00272: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
00288: 20 20 20 20 20 20 20 20 20 20 20 20 4D 55 53 52 MUSR
00304: 5F 4D 51 41 44 4D 49 4E 16 01 05 15 00 00 00 64 _MQADMIN.......d
00320: 20 3E AC 57 48 B3 09 B8 71 B0 4C F2 03 00 00 00 >¬WH..¸q.Lò....
00336: 00 00 00 00 00 00 00 0B 20 20 20 20 20 20 20 20 ........
00352: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
00368: 20 20 20 20 20 20 20 20 1C 00 00 00 57 65 62 53 ....WebS
00384: 70 68 65 72 65 20 4D 51 20 43 6C 69 65 6E 74 20 phere MQ Client
00400: 66 6F 72 20 4A 61 76 61 32 30 31 33 31 32 31 39 for Java20131219
00416: 31 34 32 32 33 32 32 32 20 20 20 20 00 00 00 00 14223222 ....
00432: A4 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ¤...............
00448: 52 46 48 20 00 00 00 02 00 00 00 A4 00 00 01 11 RFH .......¤....
00464: 00 00 04 B8 4D 51 53 54 52 20 20 20 00 00 00 00 ...¸MQSTR ....
00480: 00 00 04 B8 00 00 00 20 3C 6D 63 64 3E 3C 4D 73 ...¸... <mcd><Ms
00496: 64 3E 6A 6D 73 5F 74 65 78 74 3C 2F 4D 73 64 3E d>jms_text</Msd>
00512: 3C 2F 6D 63 64 3E 20 20 00 00 00 58 3C 6A 6D 73 </mcd> ...X<jms
00528: 3E 3C 44 73 74 3E 71 75 65 75 65 3A 2F 2F 2F 51 ><Dst>queue:///Q
00544: 31 3C 2F 44 73 74 3E 3C 54 6D 73 3E 31 33 38 37 1</Dst><Tms>1387
00560: 34 36 32 39 35 32 32 32 36 3C 2F 54 6D 73 3E 3C 462952226</Tms><
00576: 43 69 64 3E 30 30 30 30 30 30 39 39 3C 2F 43 69 Cid>00000099</Ci
00592: 64 3E 3C 44 6C 76 3E 32 3C 2F 44 6C 76 3E 3C 2F d><Dlv>2</Dlv></
Ordered unique IDs for each record (LSN)
A set of documented record types
Transaction information with XIDs, or re-used
MQ transaction IDs
MQMD header data at discoverable offsets
in the hex of a message Put
The message payload itself
31
Check out dmpmqlog scraper tool
• Takes the tedium out of analysing the output from dmpmqlog
• Created by Peter Broadhurst
• Download from
http://www.ibm.com/support/docview.wss?uid=swg21660642
• Generates file per message PUT in the supplied data.
• Summary file
java -jar dmpmqlog.scraper-20151201.jar -b little-endian -i dmpmqlog.txt -o .
32
dmpmqlog scraper tool output
• Generates file per message PUT in the supplied data.
• Summary file
33
Summary
• Lots of tools in your MQ toolbox!
• On-line Status Commands
– DISPLAY CONN
– DISPLAY QSTATUS
– DISPLAY CHSTATUS
• Cluster monitoring – AMQSCLM
• Off-line Statistics and Accounting
– amqsmon and MS0P to view
• Tracking
– Trace-route
– Application Activity Trace
• MQ Recovery Logs
– dmpmqlog scraper
34
Where can I get more information?
IBM Messaging developerWorks
developer.ibm.com/messaging
IBM Messaging Youtube
https://www.youtube.com/IBMmessagingMedia
LinkedIn
Ibm.biz/ibmmessaging
Twitter
@IBMMessaging
IBM MQ Facebook
Facebook.com/IBM-MQ-8304628654/
35
Monday
10:30-11:30 3592 New MQ features
3452 Managing applications
12:00-13:00 2835 MQ on z/OS and Distributed
15:00-16:00 3470 Latest MQ z/OS features
2833 Where is my message?
3544 MQ Light in an MQ infrastructure
16:30-17:30 3573 Hybrid cloud messaging
2941 MQ Advanced
Tuesday
08:30-09:30 3540 The MQ Light API
12:00-13:00 3456 The IBM MQ Appliance
13:15-14:15 3499 Introducing Message Hub
3458 MQ Appliance administration
14:30-15:30 6432 MQ updates and futures (InnerCircle)
2849 Messaging feedback roundtable
16:00-17:00 3544 MQ Light in an MQ infrastructure
3513 MQ hands on lab
Wednesday
08:30-09:30 3602 Managing your MQ environment
12:00-13:00 3613 Designing MQ self service
6408 Hybrid messaging roadmap (InnerCircle)
13:15-14:00 3416 HA and DR with MQ
3433 Why secure your messaging?
15:45-16:30 3429 Securing MQ
2847 Meet the messaging experts
16:00-17:00 3508 MQ Light hands on lab
16:45-17:30 2275 Migrating to the IBM MQ Appliance
Thursday
08:30-09:15 3420 MQ Clustering
2931 Business agility with self service MQ
09:30-10:15 3479 MQ z/OS clusters and shared queue
3450 Optimising MQ applications
2849 Messaging feedback roundtable
10:30-11:15 3465 MQ Appliance high availability
3481 MQ z/OS messaging connectivity
11:30-12:15 3474 Active-active messaging
3537 Monitoring and managing MQ
3425 MQ publish/subscribe
Find us at the EXPO:
Hybrid Integration peds 65-68
Check out the Hybrid Messaging sub topic under the
Hybrid Integration topic for further customer and business
partner sessions
Hybrid Messaging from the IBM experts at InterConnect 2016
Sunday
14:30-15:30 6408 Hybrid messaging roadmap (InnerCircle)
MQ as a service Redpaper now available
• IBM MQ as a Service: A Practical Approach
• Provides information about how to build, deploy, and use IBM MQ as a
service. The information in this paper includes the key factors that must
be considered while planning the use of IBM MQ as a service
• http://ibm.biz/mqaas_red
Questions?
Notices and Disclaimers
40
Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission
from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of
initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS
DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE
USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY.
IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers
have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in
which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and
discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their
specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and
interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such
laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law
Notices and Disclaimers Con’t.
41
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not
tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the
ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,
FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,
Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,
PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,
StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business
Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
Thank You
Your Feedback is Important!
Access the InterConnect 2016 Conference Attendee
Portal to complete your session surveys from your
smartphone,
laptop or conference kiosk.

More Related Content

What's hot

Redis and its Scaling and Obersvability
Redis and its Scaling and ObersvabilityRedis and its Scaling and Obersvability
Redis and its Scaling and ObersvabilityAbhishekDubey902839
 
MariaDB ColumnStore
MariaDB ColumnStoreMariaDB ColumnStore
MariaDB ColumnStoreMariaDB plc
 
Reading the LISTCAT entries for VSAM
Reading the LISTCAT entries for VSAMReading the LISTCAT entries for VSAM
Reading the LISTCAT entries for VSAMDan O'Dea
 
RocksDB detail
RocksDB detailRocksDB detail
RocksDB detailMIJIN AN
 
Elastic APM: Amping up your logs and metrics for the full picture
Elastic APM: Amping up your logs and metrics for the full pictureElastic APM: Amping up your logs and metrics for the full picture
Elastic APM: Amping up your logs and metrics for the full pictureElasticsearch
 
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&TCloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&TShapeBlue
 
Best Practices for Oracle Exadata and the Oracle Optimizer
Best Practices for Oracle Exadata and the Oracle OptimizerBest Practices for Oracle Exadata and the Oracle Optimizer
Best Practices for Oracle Exadata and the Oracle OptimizerEdgar Alejandro Villegas
 
IBM MQ - better application performance
IBM MQ - better application performanceIBM MQ - better application performance
IBM MQ - better application performanceMarkTaylorIBM
 
Exadata troubleshooting
Exadata troubleshootingExadata troubleshooting
Exadata troubleshootingpat2001
 
MariaDB High Availability
MariaDB High AvailabilityMariaDB High Availability
MariaDB High AvailabilityMariaDB plc
 
Designing and building a micro-services architecture. Stairway to heaven or a...
Designing and building a micro-services architecture. Stairway to heaven or a...Designing and building a micro-services architecture. Stairway to heaven or a...
Designing and building a micro-services architecture. Stairway to heaven or a...Sander Hoogendoorn
 
Presentation oracle on power power advantages and license optimization
Presentation   oracle on power power advantages and license optimizationPresentation   oracle on power power advantages and license optimization
Presentation oracle on power power advantages and license optimizationsolarisyougood
 
Migrating Monitoring to Observability – How to Transform DevOps from being Re...
Migrating Monitoring to Observability – How to Transform DevOps from being Re...Migrating Monitoring to Observability – How to Transform DevOps from being Re...
Migrating Monitoring to Observability – How to Transform DevOps from being Re...Liz Masters Lovelace
 
Writing and testing high frequency trading engines in java
Writing and testing high frequency trading engines in javaWriting and testing high frequency trading engines in java
Writing and testing high frequency trading engines in javaPeter Lawrey
 
MySQL Enterprise Edition
MySQL Enterprise EditionMySQL Enterprise Edition
MySQL Enterprise EditionMySQL Brasil
 
An Introduction to Redis for Developers.pdf
An Introduction to Redis for Developers.pdfAn Introduction to Redis for Developers.pdf
An Introduction to Redis for Developers.pdfStephen Lorello
 

What's hot (20)

Redis and its Scaling and Obersvability
Redis and its Scaling and ObersvabilityRedis and its Scaling and Obersvability
Redis and its Scaling and Obersvability
 
MariaDB ColumnStore
MariaDB ColumnStoreMariaDB ColumnStore
MariaDB ColumnStore
 
Reading the LISTCAT entries for VSAM
Reading the LISTCAT entries for VSAMReading the LISTCAT entries for VSAM
Reading the LISTCAT entries for VSAM
 
RocksDB detail
RocksDB detailRocksDB detail
RocksDB detail
 
Elastic APM: Amping up your logs and metrics for the full picture
Elastic APM: Amping up your logs and metrics for the full pictureElastic APM: Amping up your logs and metrics for the full picture
Elastic APM: Amping up your logs and metrics for the full picture
 
Monitoring with Prometheus
Monitoring with PrometheusMonitoring with Prometheus
Monitoring with Prometheus
 
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&TCloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
 
Best Practices for Oracle Exadata and the Oracle Optimizer
Best Practices for Oracle Exadata and the Oracle OptimizerBest Practices for Oracle Exadata and the Oracle Optimizer
Best Practices for Oracle Exadata and the Oracle Optimizer
 
IBM MQ - better application performance
IBM MQ - better application performanceIBM MQ - better application performance
IBM MQ - better application performance
 
Exadata troubleshooting
Exadata troubleshootingExadata troubleshooting
Exadata troubleshooting
 
Disruptor
DisruptorDisruptor
Disruptor
 
MariaDB High Availability
MariaDB High AvailabilityMariaDB High Availability
MariaDB High Availability
 
AWR Sample Report
AWR Sample ReportAWR Sample Report
AWR Sample Report
 
Designing and building a micro-services architecture. Stairway to heaven or a...
Designing and building a micro-services architecture. Stairway to heaven or a...Designing and building a micro-services architecture. Stairway to heaven or a...
Designing and building a micro-services architecture. Stairway to heaven or a...
 
Mutiny + quarkus
Mutiny + quarkusMutiny + quarkus
Mutiny + quarkus
 
Presentation oracle on power power advantages and license optimization
Presentation   oracle on power power advantages and license optimizationPresentation   oracle on power power advantages and license optimization
Presentation oracle on power power advantages and license optimization
 
Migrating Monitoring to Observability – How to Transform DevOps from being Re...
Migrating Monitoring to Observability – How to Transform DevOps from being Re...Migrating Monitoring to Observability – How to Transform DevOps from being Re...
Migrating Monitoring to Observability – How to Transform DevOps from being Re...
 
Writing and testing high frequency trading engines in java
Writing and testing high frequency trading engines in javaWriting and testing high frequency trading engines in java
Writing and testing high frequency trading engines in java
 
MySQL Enterprise Edition
MySQL Enterprise EditionMySQL Enterprise Edition
MySQL Enterprise Edition
 
An Introduction to Redis for Developers.pdf
An Introduction to Redis for Developers.pdfAn Introduction to Redis for Developers.pdf
An Introduction to Redis for Developers.pdf
 

Similar to HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applications Have Done

Where is My Message
Where is My MessageWhere is My Message
Where is My MessageMatt Leming
 
Where is my MQ message on z/OS?
Where is my MQ message on z/OS?Where is my MQ message on z/OS?
Where is my MQ message on z/OS?Matt Leming
 
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...Peter Broadhurst
 
Monitoring and problem determination of your mq distributed systems
Monitoring and problem determination of your mq distributed systemsMonitoring and problem determination of your mq distributed systems
Monitoring and problem determination of your mq distributed systemsmatthew1001
 
Where is My Message?: Use MQ Tools to Work Out What Applications Have Done
Where is My Message?: Use MQ Tools to Work Out What Applications Have DoneWhere is My Message?: Use MQ Tools to Work Out What Applications Have Done
Where is My Message?: Use MQ Tools to Work Out What Applications Have DoneMorag Hughson
 
Queue Manager both as sender and Receiver.docx
Queue Manager both as sender and Receiver.docxQueue Manager both as sender and Receiver.docx
Queue Manager both as sender and Receiver.docxsarvank2
 
Websphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentalsWebsphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentalsBiju Nair
 
Using Transaction Tracing to Determine Issues with Remote MQ Transactions
Using Transaction Tracing to Determine Issues with Remote MQ TransactionsUsing Transaction Tracing to Determine Issues with Remote MQ Transactions
Using Transaction Tracing to Determine Issues with Remote MQ TransactionsjKool
 
MSMQ - Microsoft Message Queueing
MSMQ - Microsoft Message QueueingMSMQ - Microsoft Message Queueing
MSMQ - Microsoft Message QueueingPeter R. Egli
 
5th KuVS Meeting
5th KuVS Meeting5th KuVS Meeting
5th KuVS Meetingsteccami
 
Network-Connected Development with ZeroMQ
Network-Connected Development with ZeroMQNetwork-Connected Development with ZeroMQ
Network-Connected Development with ZeroMQICS
 
Functional and non-functional testing with IoT-Testware
Functional and non-functional testing with IoT-TestwareFunctional and non-functional testing with IoT-Testware
Functional and non-functional testing with IoT-TestwareAxel Rennoch
 
Overview of ZeroMQ
Overview of ZeroMQOverview of ZeroMQ
Overview of ZeroMQpieterh
 
Timing attacks against web applications: Are they still practical?
Timing attacks against web applications: Are they still practical?Timing attacks against web applications: Are they still practical?
Timing attacks against web applications: Are they still practical?DefCamp
 
RTOS implementation
RTOS implementationRTOS implementation
RTOS implementationRajan Kumar
 
3450 - Writing and optimising applications for performance in a hybrid messag...
3450 - Writing and optimising applications for performance in a hybrid messag...3450 - Writing and optimising applications for performance in a hybrid messag...
3450 - Writing and optimising applications for performance in a hybrid messag...Timothy McCormick
 
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)Jakub Botwicz
 
Webinar Monitoring in era of cloud computing
Webinar Monitoring in era of cloud computingWebinar Monitoring in era of cloud computing
Webinar Monitoring in era of cloud computingCREATE-NET
 

Similar to HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applications Have Done (20)

Where is My Message
Where is My MessageWhere is My Message
Where is My Message
 
Where is my MQ message on z/OS?
Where is my MQ message on z/OS?Where is my MQ message on z/OS?
Where is my MQ message on z/OS?
 
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
 
Monitoring and problem determination of your mq distributed systems
Monitoring and problem determination of your mq distributed systemsMonitoring and problem determination of your mq distributed systems
Monitoring and problem determination of your mq distributed systems
 
Where is My Message?: Use MQ Tools to Work Out What Applications Have Done
Where is My Message?: Use MQ Tools to Work Out What Applications Have DoneWhere is My Message?: Use MQ Tools to Work Out What Applications Have Done
Where is My Message?: Use MQ Tools to Work Out What Applications Have Done
 
Queue Manager both as sender and Receiver.docx
Queue Manager both as sender and Receiver.docxQueue Manager both as sender and Receiver.docx
Queue Manager both as sender and Receiver.docx
 
Websphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentalsWebsphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentals
 
Using Transaction Tracing to Determine Issues with Remote MQ Transactions
Using Transaction Tracing to Determine Issues with Remote MQ TransactionsUsing Transaction Tracing to Determine Issues with Remote MQ Transactions
Using Transaction Tracing to Determine Issues with Remote MQ Transactions
 
MSMQ - Microsoft Message Queueing
MSMQ - Microsoft Message QueueingMSMQ - Microsoft Message Queueing
MSMQ - Microsoft Message Queueing
 
5th KuVS Meeting
5th KuVS Meeting5th KuVS Meeting
5th KuVS Meeting
 
Network-Connected Development with ZeroMQ
Network-Connected Development with ZeroMQNetwork-Connected Development with ZeroMQ
Network-Connected Development with ZeroMQ
 
Functional and non-functional testing with IoT-Testware
Functional and non-functional testing with IoT-TestwareFunctional and non-functional testing with IoT-Testware
Functional and non-functional testing with IoT-Testware
 
Overview of ZeroMQ
Overview of ZeroMQOverview of ZeroMQ
Overview of ZeroMQ
 
Timing attacks against web applications: Are they still practical?
Timing attacks against web applications: Are they still practical?Timing attacks against web applications: Are they still practical?
Timing attacks against web applications: Are they still practical?
 
RTOS implementation
RTOS implementationRTOS implementation
RTOS implementation
 
3450 - Writing and optimising applications for performance in a hybrid messag...
3450 - Writing and optimising applications for performance in a hybrid messag...3450 - Writing and optimising applications for performance in a hybrid messag...
3450 - Writing and optimising applications for performance in a hybrid messag...
 
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)
 
dnp3 Protocol Master Client simulator user manual
dnp3 Protocol Master Client simulator user manualdnp3 Protocol Master Client simulator user manual
dnp3 Protocol Master Client simulator user manual
 
WebSphere MQ introduction
WebSphere MQ introductionWebSphere MQ introduction
WebSphere MQ introduction
 
Webinar Monitoring in era of cloud computing
Webinar Monitoring in era of cloud computingWebinar Monitoring in era of cloud computing
Webinar Monitoring in era of cloud computing
 

More from Matt Leming

533-MigratingYourMQIApplicationsToJMS.pdf
533-MigratingYourMQIApplicationsToJMS.pdf533-MigratingYourMQIApplicationsToJMS.pdf
533-MigratingYourMQIApplicationsToJMS.pdfMatt Leming
 
IBM MQ Whats new - up to 9.3.4.pptx
IBM MQ Whats new - up to 9.3.4.pptxIBM MQ Whats new - up to 9.3.4.pptx
IBM MQ Whats new - up to 9.3.4.pptxMatt Leming
 
Going Deep with MQ
Going Deep with MQGoing Deep with MQ
Going Deep with MQMatt Leming
 
What's new with MQ on z/OS 9.3 and 9.3.1
What's new with MQ on z/OS 9.3 and 9.3.1What's new with MQ on z/OS 9.3 and 9.3.1
What's new with MQ on z/OS 9.3 and 9.3.1Matt Leming
 
Connecting mq&amp;kafka
Connecting mq&amp;kafkaConnecting mq&amp;kafka
Connecting mq&amp;kafkaMatt Leming
 
What's New In MQ 9.2 on z/OS
What's New In MQ 9.2 on z/OSWhat's New In MQ 9.2 on z/OS
What's New In MQ 9.2 on z/OSMatt Leming
 
Building a resilient and scalable solution with IBM MQ on z/OS
Building a resilient and scalable solution with IBM MQ on z/OSBuilding a resilient and scalable solution with IBM MQ on z/OS
Building a resilient and scalable solution with IBM MQ on z/OSMatt Leming
 
What's new in MQ 9.1.* on z/OS
What's new in MQ 9.1.* on z/OSWhat's new in MQ 9.1.* on z/OS
What's new in MQ 9.1.* on z/OSMatt Leming
 
REST APIs and MQ
REST APIs and MQREST APIs and MQ
REST APIs and MQMatt Leming
 
What's new in MQ 9.1 on z/OS
What's new in MQ 9.1 on z/OSWhat's new in MQ 9.1 on z/OS
What's new in MQ 9.1 on z/OSMatt Leming
 
The enterprise differentiator of mq on zos
The enterprise differentiator of mq on zosThe enterprise differentiator of mq on zos
The enterprise differentiator of mq on zosMatt Leming
 
New Tools and Interfaces for Managing IBM MQ
New Tools and Interfaces for Managing IBM MQNew Tools and Interfaces for Managing IBM MQ
New Tools and Interfaces for Managing IBM MQMatt Leming
 
MQ Support for z/OS Connect
MQ Support for z/OS ConnectMQ Support for z/OS Connect
MQ Support for z/OS ConnectMatt Leming
 
HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...
 HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen... HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...
HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...Matt Leming
 
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
 HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ... HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...Matt Leming
 

More from Matt Leming (15)

533-MigratingYourMQIApplicationsToJMS.pdf
533-MigratingYourMQIApplicationsToJMS.pdf533-MigratingYourMQIApplicationsToJMS.pdf
533-MigratingYourMQIApplicationsToJMS.pdf
 
IBM MQ Whats new - up to 9.3.4.pptx
IBM MQ Whats new - up to 9.3.4.pptxIBM MQ Whats new - up to 9.3.4.pptx
IBM MQ Whats new - up to 9.3.4.pptx
 
Going Deep with MQ
Going Deep with MQGoing Deep with MQ
Going Deep with MQ
 
What's new with MQ on z/OS 9.3 and 9.3.1
What's new with MQ on z/OS 9.3 and 9.3.1What's new with MQ on z/OS 9.3 and 9.3.1
What's new with MQ on z/OS 9.3 and 9.3.1
 
Connecting mq&amp;kafka
Connecting mq&amp;kafkaConnecting mq&amp;kafka
Connecting mq&amp;kafka
 
What's New In MQ 9.2 on z/OS
What's New In MQ 9.2 on z/OSWhat's New In MQ 9.2 on z/OS
What's New In MQ 9.2 on z/OS
 
Building a resilient and scalable solution with IBM MQ on z/OS
Building a resilient and scalable solution with IBM MQ on z/OSBuilding a resilient and scalable solution with IBM MQ on z/OS
Building a resilient and scalable solution with IBM MQ on z/OS
 
What's new in MQ 9.1.* on z/OS
What's new in MQ 9.1.* on z/OSWhat's new in MQ 9.1.* on z/OS
What's new in MQ 9.1.* on z/OS
 
REST APIs and MQ
REST APIs and MQREST APIs and MQ
REST APIs and MQ
 
What's new in MQ 9.1 on z/OS
What's new in MQ 9.1 on z/OSWhat's new in MQ 9.1 on z/OS
What's new in MQ 9.1 on z/OS
 
The enterprise differentiator of mq on zos
The enterprise differentiator of mq on zosThe enterprise differentiator of mq on zos
The enterprise differentiator of mq on zos
 
New Tools and Interfaces for Managing IBM MQ
New Tools and Interfaces for Managing IBM MQNew Tools and Interfaces for Managing IBM MQ
New Tools and Interfaces for Managing IBM MQ
 
MQ Support for z/OS Connect
MQ Support for z/OS ConnectMQ Support for z/OS Connect
MQ Support for z/OS Connect
 
HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...
 HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen... HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...
HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...
 
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
 HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ... HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
 

Recently uploaded

ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...
ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...
ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...Christina Lin
 
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...soniya singh
 
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASEBATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASEOrtus Solutions, Corp
 
MYjobs Presentation Django-based project
MYjobs Presentation Django-based projectMYjobs Presentation Django-based project
MYjobs Presentation Django-based projectAnoyGreter
 
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...stazi3110
 
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样umasea
 
What is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need ItWhat is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need ItWave PLM
 
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceCRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceBrainSell Technologies
 
Building Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
Building Real-Time Data Pipelines: Stream & Batch Processing workshop SlideBuilding Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
Building Real-Time Data Pipelines: Stream & Batch Processing workshop SlideChristina Lin
 
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptxKnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptxTier1 app
 
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...OnePlan Solutions
 
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024StefanoLambiase
 
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEECloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEEVICTOR MAESTRE RAMIREZ
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtimeandrehoraa
 
Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)OPEN KNOWLEDGE GmbH
 
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdfXen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdfStefano Stabellini
 
英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作qr0udbr0
 
Introduction Computer Science - Software Design.pdf
Introduction Computer Science - Software Design.pdfIntroduction Computer Science - Software Design.pdf
Introduction Computer Science - Software Design.pdfFerryKemperman
 

Recently uploaded (20)

ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...
ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...
ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...
 
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
 
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASEBATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
 
MYjobs Presentation Django-based project
MYjobs Presentation Django-based projectMYjobs Presentation Django-based project
MYjobs Presentation Django-based project
 
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
 
2.pdf Ejercicios de programación competitiva
2.pdf Ejercicios de programación competitiva2.pdf Ejercicios de programación competitiva
2.pdf Ejercicios de programación competitiva
 
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
 
What is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need ItWhat is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need It
 
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceCRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. Salesforce
 
Building Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
Building Real-Time Data Pipelines: Stream & Batch Processing workshop SlideBuilding Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
Building Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
 
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptxKnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
 
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
 
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
 
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEECloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEE
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtime
 
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort ServiceHot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
 
Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)
 
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdfXen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdf
 
英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作
 
Introduction Computer Science - Software Design.pdf
Introduction Computer Science - Software Design.pdfIntroduction Computer Science - Software Design.pdf
Introduction Computer Science - Software Design.pdf
 

HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applications Have Done

  • 1. HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applications Have Done Matt Leming – lemingma@uk.ibm.com IBM Hursley - UK
  • 2. Symptoms include: • Timeouts for request/response messaging • Inconsistent data between front-end and back-end systems • Customer complaints about ‘missing’ transactions 1
  • 3. Agenda – the MQ Toolbox • Who’s connected? • Are messages flowing? • Where are messages going? • What are the apps doing? • How can I look back in time? Presentation contains MQSC example commands and output. Your own admin tools will vary by task, situation and preference. Tasks can be performed in PowerShell, MQ Explorer etc. as desired. 2
  • 5. DISPLAY CONN DISPLAY CONN(*) TYPE(HANDLE) ALL DIS CONN(*) ALL AMQ8276: Display Connection details. CONN(577C425321295301) EXTCONN(414D5143474154455741593120202020) TYPE(HANDLE) OBJNAME(WLMMDB.REQUEST) OBJTYPE(QUEUE) ASTATE(NONE) HSTATE(INACTIVE) OPENOPTS(MQOO_OUTPUT,MQOO_FAIL_IF_QUIESCING) READA(NO) OBJNAME(SENDINGAPP.REPLY) OBJTYPE(QUEUE) ASTATE(ACTIVE) HSTATE(ACTIVE) OPENOPTS(MQOO_INPUT_SHARED,MQOO_INQUIRE,MQOO_SAVE_ALL_CONTEXT,MQOO_FAIL_IF_QUIESCING) READA(NO) AMQ8276: Display Connection details. CONN(577C425321295301) EXTCONN(414D5143474154455741593120202020) TYPE(CONN) PID(9740) TID(185) APPLDESC(WebSphere MQ Channel) APPLTAG(jms/GATEWAY1_CF) APPLTYPE(SYSTEM) ASTATE(NONE) CHANNEL(WAS.CLIENTS) CONNAME(127.0.0.1) CONNOPTS(MQCNO_SHARED_BINDING) USERID(pbroad) UOWLOG( ) UOWSTDA(2014-04-08) UOWSTTI(13.24.00) UOWLOGDA( ) UOWLOGTI( ) URTYPE(XA) EXTURID(XA_FORMATID[DSAW] XA_GTRID[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809] XA_BQUAL[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809000000010000000000000000 000000000001]) QMURID(0.7940075) UOWSTATE(ACTIVE) Use CONN to match TYPE(CONN) and TYPE(HANDLE) records TYPE(HANDLE) records let you find applications by the objects they access. See all open handles for an app in one place, unlike DIS QSTATUS records Long running UOW information. XID can be tied up with app server txn timeout Channel name + IP help identify client apps. MQ V7.5 and later JMS clients can supply an application name in the CF 4
  • 6. DISPLAY CHSTATUS DISPLAY CHSTATUS(*) ALL AMQ8417: Display Channel Status details. CHANNEL(WAS.CLIENTS) CHLTYPE(SVRCONN) BUFSRCVD(17) BUFSSENT(13) BYTSRCVD(2296) BYTSSENT(2456) CHSTADA(2014-04-08) CHSTATI(15.26.59) COMPHDR(NONE,NONE) COMPMSG(NONE,NONE) COMPRATE(0,0) COMPTIME(0,0) CONNAME(127.0.0.1) CURRENT EXITTIME(0,0) HBINT(5) JOBNAME(0000260C000000B9) LOCLADDR( ) LSTMSGDA(2014-04-08) LSTMSGTI(15.26.59) MCASTAT(RUNNING) MCAUSER(pbroad) MONCHL(OFF) MSGS(6) RAPPLTAG(jar) SSLCERTI(CN=ExampleCA,O=Example) SSLKEYDA( ) SSLKEYTI( ) SSLPEER(SERIALNUMBER=53:43:FD:D6,CN=ExampleApp1,O=Example) SSLRKEYS(0) STATUS(RUNNING) STOPREQ(NO) SUBSTATE(RECEIVE) CURSHCNV(1) MAXSHCNV(1) RVERSION(00000000) RPRODUCT(MQJM) See SSLPEER information not in DIS CONN JOBNAME contains PID (except z/OS): 0x260C = PID(9740) On Linux/UNIX (not Win) TID matches CONN: 0xB9 = TID(185) Note that multiple CONN might share one SVRCONN channel instance Check suitable heartbeats are negotiated 5
  • 7. Automatic detection if no-one is connected
  • 8. AMQSCLM 1) Normal operation Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps Cluster WLM Round-robin message distribution App server 1 App server 2 App server N . . . IPPROCS(>=1) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM AMQSCLM AMQSCLM 7
  • 9. AMQSCLM 2) Application failure Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) Messages start building up AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution 8
  • 10. AMQSCLM 3) Failure Detection Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM regularly checks the queue and detects IPPROCS(0) AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution 9
  • 11. AMQSCLM 4) Redirect Messages Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM reduces queue priority in cluster, to stop new messages arriving AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution 10
  • 12. AMQSCLM 5) Un-maroon Msgs (optional) Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM tells qmgr to redistribute ‘stuck’ messages via the cluster AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution 11
  • 13. AMQSCLM summary • The Cluster Queue Monitoring sample program (AMQSCLM) • Shipped with the product as a sample – Precompiled – Source code – Platforms except on z/OS • More information here: http://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm. mq.dev.doc/q024620_.htm 12
  • 15. Real-time/online Monitoring – Queues • Set detail level for QMGR. Override for individual Queues • Gives live view of application responsiveness ALTER QMGR MONQ(MEDIUM) ALTER QLOCAL(QUEUE1) MONQ(HIGH) ALTER QLOCAL(QUEUE2) MONQ(OFF) AMQ8450: Display queue status details. QUEUE(QUEUE1) TYPE(QUEUE) CURDEPTH(16) IPPROCS(3) LGETDATE(2014-04-08) LGETTIME(17.05.59) LPUTDATE(2014-04-08) LPUTTIME(17.12.16) MEDIALOG( ) MONQ(HIGH) MSGAGE(112) OPPROCS(5) QTIME(10101414, 10101414) UNCOM(NO) DIS QSTATUS(QUEUE1) ALL Age in seconds of the oldest message on the queue Estimations of the time in microseconds that messages are waiting on the queue for processing. First value: Calculated from recent activity Second value: Calculated from longer term activity Timestamps of last PUT/GET to check for recent activity Without MONQ you only get the depth and how many handles are open 14
  • 16. Real-time/online Monitoring – Channels • Gives live view of channel throughput ALTER QMGR MONCHL(MEDIUM) ALTER CHANNEL(CLUSTER1.QM1) CHLTYPE(CLUSRCVR) MONCHL(HIGH) AMQ8417: Display Channel Status details. CHANNEL(MQHUB.GATEWAY2) CHLTYPE(CLUSSDR) BATCHES(52) BATCHSZ(50) BUFSRCVD(55) BUFSSENT(1616) BYTSRCVD(1748) BYTSSENT(1192330) CHSTADA(2014-04-08) CHSTATI(17.49.03) COMPHDR(NONE,NONE) COMPMSG(NONE,NONE) COMPRATE(0,0) COMPTIME(0,0) CONNAME(127.0.0.1(1422)) CURLUWID(0107445310001B34) CURMSGS(50) CURRENT CURSEQNO(11823) EXITTIME(0,0) HBINT(5) INDOUBT(YES) JOBNAME(00002DA4000047D0) LOCLADDR(127.0.0.1(53557)) LONGRTS(999999999) LSTLUWID(0107445310001B33) LSTMSGDA(2014-04-08) LSTMSGTI(17.49.51) LSTSEQNO(11773) MCASTAT(RUNNING) MONCHL(MEDIUM) MSGS(1580) NETTIME(137538,29555) NPMSPEED(FAST) RQMNAME(GATEWAY2) SHORTRTS(180) SSLCERTI( ) SSLKEYDA( ) SSLKEYTI( ) SSLPEER( ) SSLRKEYS(0) STATUS(RUNNING) STOPREQ(NO) SUBSTATE(RECEIVE) XBATCHSZ(20,17) XMITQ(SYSTEM.CLUSTER.TRANSMIT.QUEUE) XQMSGSA(112) XQTIME(545784,3929968) RVERSION(07050002) RPRODUCT(MQMM) DIS CHSTATUS(MQHUB.GATEWAY2) ALL Short/long term calculations of how full your batches are getting, to help you tune BATCHSZ/BATCHINT Last time a message was sent over the channel Depth of messages on XMITQ for this channel (capped at 999) Short/long term calculations of how long messages are waiting on the XMITQ for transmission 15
  • 17. Accounting and Statistics Overview • Monitoring data sent as a PCF message at a configured interval – Statistics – scoped to a Queue / Channel / QMGR – Accounting – scoped to an individual CONN and Queue / QMGR • Applications must consume the messages – Enterprise monitoring – Tivoli ITCAM / Tivoli OMEGAMON XE for Messaging – Sample applications and SupportPacs – Custom applications – com.ibm.mq.headers Java package PCF accounting collection app Queues Channels CONNs PCF statistics collection app Statistics Messages Accounting Messages QMGR SYSTEM.ADMIN.STATISTICS.QUEUE SYSTEM.ADMIN.ACCOUNTING.QUEUE 16
  • 18. Taking a look with SupportPac MS0P • Time period • MQ Statistics at QMGR level • Detailed queue statistics 17
  • 19. Simple practical example using amqsmon • Overall QMGR busyness • Simple data format – Multiple values are [Persistent, NonPersistent] • One message every X seconds – Use amqsmon directly (perl/cron) • Low/high water marks for subscriptions – Grouped by subscription type ALTER QMGR STATMQI(ON) Wait a bit RESET QMGR TYPE(STATISTICS) amqsmon -m GATEWAY1 -t statistics -a -w 0 MonitoringType: MQIStatistics QueueManager: 'GATEWAY1' IntervalStartDate: '2014-04-09' IntervalStartTime: '00.00.35' IntervalEndDate: '2014-04-09' IntervalEndTime: '00.01.13' CommandLevel: 700 ConnCount: 35 PutCount: [271, 0] PutFailCount: 0 Put1Count: [2, 0] Put1FailCount: 0 PutBytes: [273976, 0] GetCount: [270, 0] GetBytes: [269468, 0] GetFailCount: 19 DurableSubscriptionHighWater: [0, 0, 0, 0] DurableSubscriptionLowWater: [0, 0, 0, 0] NonDurableSubscriptionHighWater: [0, 0, 0, 0] NonDurableSubscriptionLowWater: [0, 0, 0, 0] PutTopicCount: [0, 0] PutTopicFailCount: 0 Put1TopicCount: [0, 0] Put1TopicFailCount: 0 PutTopicBytes: [0, 0] PublishMsgCount: [0, 0] PublishMsgBytes: [0, 0] 18
  • 21. Cluster MQ networks can be complex • At each of the dots stuck / mis-sent messages are possible – MQOPENs of the wrong Queue / Queue Manager by apps – Full queues – Stopped channels – Stopped apps – Incorrectly configured QREMOTE/QALIAS routing objects – Cluster membership problems • The standard problem diagnosis approach – Methodically checking channels/queues/DLQs at each point • Is there anything to speed up this process? App1 App2A App2B Requests Replies Replies SDR/RCVR SDR/RCVR 20
  • 22. Cluster Trace Route • MQ has the ability to inject tracer messages – (Can be) hidden from applications – Generate activity reports as they pass through • Tools are available to trace routes using these reports – dspmqrte – command line tool supplied with the product – MS0P – Cat 2 SupportPac extension to MQ Explorer – Report messages need to get back to your ReplyQ of your tool • Lets you see the path messages could have taken – Test connectivity through the MQ network – Test cluster workload balancing • Can quickly jump you close to the problem – The point your trace message veers off in the wrong direction – The point the trail goes cold dspmqrte SDR/RCVR SDR/RCVR dspmqrte dspmqrte 21
  • 23. What are the apps doing?
  • 24. Application Activity Tracing • Similar infrastructure to Accounting & Statistics – PCF messages on SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE – Configurable via mqat.ini • Can be changed without QMGR restart • Configurable detail level can include partial/full message payload • Frequency options for tuning – Can be enabled on a per-application basis • Via MQCONNX flags • Via application name – In MQ V7.5 and later JMS clients can supply an application name in the connection factory Health warning: Performance impact http://ow.ly/vA8wB 23
  • 25. Application Activity Tracing • Enables scenarios such as – Application audit trail – Message duplication – Resource usage • Which queues or topics are actually being used – Problem Determination • Which Queue / Queue Manager is the application actually opening – Application Coding Standards • Does everyone use the MQI in the recommended way – And more … Health warning: Performance impact http://ow.ly/vA8wB 24
  • 26. Looking at the data with the amqsact sample ALTER QMGR ACTVTRC(ON) <- should be tuned via mqat.ini amqsact -m GATEWAY1 -v MQI Operation: 6 Operation Id: MQXF_PUT ApplicationTid: 12451 OperationDate: '2014-04-09' OperationTime: '01:39:48' High Res Time: 1397003988665548 Completion Code: MQCC_OK Reason Code: 0 Hobj: 18225032 Put Options: 139330 Msg length: 460 Recs_present: 0 Known_dest_count: 1 Unknown_dest_count: 0 Invalid_dest_count: 0 Object_type: MQOT_Q Object_name: 'SENDINGAPP.REPLY' Object_Q_mgr_name: 'GATEWAY1' Resolved_Q_Name: 'SENDINGAPP.REPLY' Resolved_Q_mgr: 'GATEWAY1' Resolved_local_Q_name: 'SENDINGAPP.REPLY' Resolved_local_Q_mgr: 'GATEWAY1' Resolved_type: MQOT_Q Report Options: 0 Msg_type: MQMT_DATAGRAM Expiry: -1 Format_name: 'MQHRF2' Priority: 4 Persistence: 0 Msg_id: 00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 ' 00000010: 0207 4453 2007 2603 '..DS .&. ' Correl_id: 00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 ' 00000010: 0207 4453 2007 2203 '..DS .". ' Reply_to_Q : ' ^D' Reply_to_Q_Mgr: ' ^C' Coded_char_set_id: 1208 Encoding: 273 Put_date: '20140409' Put_time: '00394866' Check the options used for coding standards Check queue name resolution, to find out why messages are going to the wrong place Track individual messages and request/reply scenarios with Msg_id and Correl_id 25
  • 27. Extract from Report MonitoringType: MQI Activity Trace QueueManager: 'V71' Host Name: 'rockall.hursley.ibm.com' CommandLevel: 710 ApplicationName: 'WebSphere MQ Client for Java' ApplicationPid: 18612354 UserId: 'mquser' ConnName: '9.20.95.106' Channel Type: MQCHT_SVRCONN Platform: MQPL_UNIX ======================================================================= = Time Operation CompCode MQRC HObj (ObjName) 10:04:09 MQXF_INQ MQCC_OK 0000 2 10:04:09 MQXF_CLOSE MQCC_OK 0000 2 10:04:09 MQXF_OPEN MQCC_OK 0000 4 () 10:04:09 MQXF_INQ MQCC_OK 0000 4 10:04:09 MQXF_CLOSE MQCC_OK 0000 4 10:04:09 MQXF_OPEN MQCC_OK 0000 4 (SYSTEM.DEFAULT.LOCAL.QUEUE) 10:04:09 MQXF_INQ MQCC_OK 0000 4 26
  • 28. How can I look back in time?
  • 29. What happened to my messages at 2am this morning? • Enterprise monitoring solution – DLQ alerts, queue depth alerts, channel status alerts – Unresolved running units of work – Historical MQ monitoring, accounting and stats data • App logs from the time of the problem – Exceptions, MQ error codes, timeouts • MQ error logs for all qmgrs that could have been involved – Channel errors – Authentication issues • ??? – what else is there 29
  • 30. What about the MQ Recovery Log? • For persistent messages inside transactions – MQ logs each operation performed • Why can't we use this to – Look back in time to 2am and see what happened? – Recovery the original payload if the app lost the message? – See what happened inside long-running units of work? – Provide a list of operations within the failed business transaction? • MQ documents how you can… if – You use the text formatting tool provided with MQ (dmpmqlog) – The logging is linear so the historical data is available in the tool – You follow the right steps to extract data from running qmgrs – You do the work to follow through the logs 30
  • 31. Dmpmqlog output is readable, but analysis is tedious • Wouldn't it be easier to let the computer do the tedious bit? LOG RECORD - LSN <0:0:954:44817> ********** HLG Header: lrecsize 873, version 1, rmid 0, eyecatcher HLRH LogRecdType . . : AQM Put Message (257) Eyecatcher . . : ALRH Version . . . . : 1 LogRecdLen . . : 853 LogRecdOwnr . . : 256 (AQM) XTranid . . . . : TranType: XA XID: formatID 1463898948, gtrid_length 36, bqual_length 54 gtrid [000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F2] bqual [000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F20000000100000000000 00000000000000001] QueueName . . . : Not known Qid . . . . . . : {Hash 2147211283, Counter: 5} ThisLSN . . . . : <0:0:0:0> PrevLSN . . . . : <0:0:954:43944> Version . . . . : 4 MapIndex . . . : 199 PrevLink.Locn . : 102408 PrevLink.Length : 8 PrevDataLink . : {High 0, Low 103424} Data.Locn . . . : 103424 Data.Length . . : 613 Data . . . . . : 00000: 41 51 52 48 04 00 00 00 FF FF FF FF FF FF FF FF AQRH....ÿÿÿÿÿÿÿÿ 00016: 00 00 00 00 00 00 00 00 C7 00 00 00 02 00 C0 01 ........Ç....... 00032: 00 00 00 00 04 00 01 00 A5 00 00 00 00 00 00 00 ........¥....... 00048: 63 00 00 00 41 4D 51 20 49 49 42 30 31 5F 51 4D c...AMQ IIB01_QM 00064: 20 20 20 20 D9 26 B0 52 20 08 53 F5 30 30 30 30 .&.R .Sõ0000 00080: 30 30 39 39 00 00 00 00 00 00 00 00 00 00 00 00 0099............ 00096: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00112: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ................ 00128: 00 00 00 00 FF FF FF FF 00 00 00 00 04 00 00 09 ....ÿÿÿÿ........ 00144: 00 00 00 00 E3 ED 04 80 10 FD 67 E4 FF FF FF FF ....ãí.€.ýgäÿÿÿÿ 00160: 4D 44 20 20 01 00 00 00 00 00 00 00 08 00 00 00 MD ............ 00176: 00 00 00 00 11 01 00 00 B8 04 00 00 4D 51 48 52 ........¸...MQHR 00192: 46 32 20 20 04 00 00 00 01 00 00 00 20 20 20 20 F2 ........ 00208: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00224: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00240: 20 20 20 20 20 20 20 20 20 20 20 20 49 49 42 30 IIB0 00256: 31 5F 51 4D 20 20 20 20 20 20 20 20 20 20 20 20 1_QM 00272: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00288: 20 20 20 20 20 20 20 20 20 20 20 20 4D 55 53 52 MUSR 00304: 5F 4D 51 41 44 4D 49 4E 16 01 05 15 00 00 00 64 _MQADMIN.......d 00320: 20 3E AC 57 48 B3 09 B8 71 B0 4C F2 03 00 00 00 >¬WH..¸q.Lò.... 00336: 00 00 00 00 00 00 00 0B 20 20 20 20 20 20 20 20 ........ 00352: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00368: 20 20 20 20 20 20 20 20 1C 00 00 00 57 65 62 53 ....WebS 00384: 70 68 65 72 65 20 4D 51 20 43 6C 69 65 6E 74 20 phere MQ Client 00400: 66 6F 72 20 4A 61 76 61 32 30 31 33 31 32 31 39 for Java20131219 00416: 31 34 32 32 33 32 32 32 20 20 20 20 00 00 00 00 14223222 .... 00432: A4 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ¤............... 00448: 52 46 48 20 00 00 00 02 00 00 00 A4 00 00 01 11 RFH .......¤.... 00464: 00 00 04 B8 4D 51 53 54 52 20 20 20 00 00 00 00 ...¸MQSTR .... 00480: 00 00 04 B8 00 00 00 20 3C 6D 63 64 3E 3C 4D 73 ...¸... <mcd><Ms 00496: 64 3E 6A 6D 73 5F 74 65 78 74 3C 2F 4D 73 64 3E d>jms_text</Msd> 00512: 3C 2F 6D 63 64 3E 20 20 00 00 00 58 3C 6A 6D 73 </mcd> ...X<jms 00528: 3E 3C 44 73 74 3E 71 75 65 75 65 3A 2F 2F 2F 51 ><Dst>queue:///Q 00544: 31 3C 2F 44 73 74 3E 3C 54 6D 73 3E 31 33 38 37 1</Dst><Tms>1387 00560: 34 36 32 39 35 32 32 32 36 3C 2F 54 6D 73 3E 3C 462952226</Tms>< 00576: 43 69 64 3E 30 30 30 30 30 30 39 39 3C 2F 43 69 Cid>00000099</Ci 00592: 64 3E 3C 44 6C 76 3E 32 3C 2F 44 6C 76 3E 3C 2F d><Dlv>2</Dlv></ Ordered unique IDs for each record (LSN) A set of documented record types Transaction information with XIDs, or re-used MQ transaction IDs MQMD header data at discoverable offsets in the hex of a message Put The message payload itself 31
  • 32. Check out dmpmqlog scraper tool • Takes the tedium out of analysing the output from dmpmqlog • Created by Peter Broadhurst • Download from http://www.ibm.com/support/docview.wss?uid=swg21660642 • Generates file per message PUT in the supplied data. • Summary file java -jar dmpmqlog.scraper-20151201.jar -b little-endian -i dmpmqlog.txt -o . 32
  • 33. dmpmqlog scraper tool output • Generates file per message PUT in the supplied data. • Summary file 33
  • 34. Summary • Lots of tools in your MQ toolbox! • On-line Status Commands – DISPLAY CONN – DISPLAY QSTATUS – DISPLAY CHSTATUS • Cluster monitoring – AMQSCLM • Off-line Statistics and Accounting – amqsmon and MS0P to view • Tracking – Trace-route – Application Activity Trace • MQ Recovery Logs – dmpmqlog scraper 34
  • 35. Where can I get more information? IBM Messaging developerWorks developer.ibm.com/messaging IBM Messaging Youtube https://www.youtube.com/IBMmessagingMedia LinkedIn Ibm.biz/ibmmessaging Twitter @IBMMessaging IBM MQ Facebook Facebook.com/IBM-MQ-8304628654/ 35
  • 36. Monday 10:30-11:30 3592 New MQ features 3452 Managing applications 12:00-13:00 2835 MQ on z/OS and Distributed 15:00-16:00 3470 Latest MQ z/OS features 2833 Where is my message? 3544 MQ Light in an MQ infrastructure 16:30-17:30 3573 Hybrid cloud messaging 2941 MQ Advanced Tuesday 08:30-09:30 3540 The MQ Light API 12:00-13:00 3456 The IBM MQ Appliance 13:15-14:15 3499 Introducing Message Hub 3458 MQ Appliance administration 14:30-15:30 6432 MQ updates and futures (InnerCircle) 2849 Messaging feedback roundtable 16:00-17:00 3544 MQ Light in an MQ infrastructure 3513 MQ hands on lab Wednesday 08:30-09:30 3602 Managing your MQ environment 12:00-13:00 3613 Designing MQ self service 6408 Hybrid messaging roadmap (InnerCircle) 13:15-14:00 3416 HA and DR with MQ 3433 Why secure your messaging? 15:45-16:30 3429 Securing MQ 2847 Meet the messaging experts 16:00-17:00 3508 MQ Light hands on lab 16:45-17:30 2275 Migrating to the IBM MQ Appliance Thursday 08:30-09:15 3420 MQ Clustering 2931 Business agility with self service MQ 09:30-10:15 3479 MQ z/OS clusters and shared queue 3450 Optimising MQ applications 2849 Messaging feedback roundtable 10:30-11:15 3465 MQ Appliance high availability 3481 MQ z/OS messaging connectivity 11:30-12:15 3474 Active-active messaging 3537 Monitoring and managing MQ 3425 MQ publish/subscribe Find us at the EXPO: Hybrid Integration peds 65-68 Check out the Hybrid Messaging sub topic under the Hybrid Integration topic for further customer and business partner sessions Hybrid Messaging from the IBM experts at InterConnect 2016 Sunday 14:30-15:30 6408 Hybrid messaging roadmap (InnerCircle)
  • 37. MQ as a service Redpaper now available • IBM MQ as a Service: A Practical Approach • Provides information about how to build, deploy, and use IBM MQ as a service. The information in this paper includes the key factors that must be considered while planning the use of IBM MQ as a service • http://ibm.biz/mqaas_red
  • 38.
  • 40. Notices and Disclaimers 40 Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law
  • 41. Notices and Disclaimers Con’t. 41 Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
  • 42. Thank You Your Feedback is Important! Access the InterConnect 2016 Conference Attendee Portal to complete your session surveys from your smartphone, laptop or conference kiosk.