SlideShare a Scribd company logo
1 of 37
Download to read offline
© 2015 IBM Corporation@MoragHughson
Session AME-2289
Where’s my message?
Morag Hughson - hughson@uk.ibm.com
IBM Hursley - UK
@MoragHughson
Symptoms include:
Timeouts for request/response messaging
Inconsistent data between front-end and back-end systems
Customer complaints about ‘missing’ transactions
Developer frustration when testing
an app / MQ environment
@MoragHughson
Agenda – the MQ Toolbox
Who’s connected?
Are messages flowing?
Where are messages going?
What are the apps doing?
How can I look back in time?
Presentation contains MQSC
example commands and output.
Your own admin tools will
vary by task, situation and
preference. Tasks can be
performed in PowerShell, MQ
Explorer etc. as desired.
Who’s connected?
@MoragHughson
DISPLAY CONN
DISPLAY CONN(*) TYPE(HANDLE) ALL DIS CONN(*) ALL
AMQ8276: Display Connection details.
CONN(577C425321295301)
EXTCONN(414D5143474154455741593120202020)
TYPE(HANDLE)
OBJNAME(WLMMDB.REQUEST) OBJTYPE(QUEUE)
ASTATE(NONE) HSTATE(INACTIVE)
OPENOPTS(MQOO_OUTPUT,MQOO_FAIL_IF_QUIESCING)
READA(NO)
OBJNAME(SENDINGAPP.REPLY) OBJTYPE(QUEUE)
ASTATE(ACTIVE) HSTATE(ACTIVE)
OPENOPTS(MQOO_INPUT_SHARED,MQOO_INQUIRE,MQOO_SAVE_ALL_CONTEXT,MQOO_FAIL_IF_QUIESCING)
READA(NO)
AMQ8276: Display Connection details.
CONN(577C425321295301)
EXTCONN(414D5143474154455741593120202020)
TYPE(CONN)
PID(9740) TID(185)
APPLDESC(WebSphere MQ Channel) APPLTAG(jms/GATEWAY1_CF)
APPLTYPE(SYSTEM) ASTATE(NONE)
CHANNEL(WAS.CLIENTS) CONNAME(127.0.0.1)
CONNOPTS(MQCNO_SHARED_BINDING) USERID(pbroad)
UOWLOG( ) UOWSTDA(2014-04-08)
UOWSTTI(13.24.00) UOWLOGDA( )
UOWLOGTI( ) URTYPE(XA)
EXTURID(XA_FORMATID[DSAW]
XA_GTRID[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809]
XA_BQUAL[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809000000010000000000000000
000000000001])
QMURID(0.7940075) UOWSTATE(ACTIVE)
Use CONN to match TYPE(CONN) and
TYPE(HANDLE) records
Use CONN to match TYPE(CONN) and
TYPE(HANDLE) records
TYPE(HANDLE) records let you find
applications by the objects they access.
See all open handles for an app in one place,
unlike DIS QSTATUS records
TYPE(HANDLE) records let you find
applications by the objects they access.
See all open handles for an app in one place,
unlike DIS QSTATUS records
Long running UOW information.
XID can be tied up with app server txn timeout
Long running UOW information.
XID can be tied up with app server txn timeout
Channel name + IP help identify client apps.Channel name + IP help identify client apps.
MQ V7.5 and later JMS clients can supply
an application name in the CF
MQ V7.5 and later JMS clients can supply
an application name in the CF
@MoragHughson
DISPLAY CHSTATUS
DISPLAY CHSTATUS(*)
AMQ8417: Display Channel Status details.
CHANNEL(WAS.CLIENTS) CHLTYPE(SVRCONN)
BUFSRCVD(17) BUFSSENT(13)
BYTSRCVD(2296) BYTSSENT(2456)
CHSTADA(2014-04-08) CHSTATI(15.26.59)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(127.0.0.1) CURRENT
EXITTIME(0,0) HBINT(5)
JOBNAME(0000260C000000B9) LOCLADDR( )
LSTMSGDA(2014-04-08) LSTMSGTI(15.26.59)
MCASTAT(RUNNING) MCAUSER(pbroad)
MONCHL(OFF) MSGS(6)
RAPPLTAG(jar) SSLCERTI(CN=ExampleCA,O=Example)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER(SERIALNUMBER=53:43:FD:D6,CN=ExampleApp1,O=Example)
SSLRKEYS(0) STATUS(RUNNING)
STOPREQ(NO) SUBSTATE(RECEIVE)
CURSHCNV(1) MAXSHCNV(1)
RVERSION(00000000) RPRODUCT(MQJM)
See SSLPEER information not in DIS CONNSee SSLPEER information not in DIS CONN
JOBNAME contains PID (except z/OS): 0x260C = PID(9740)
On Linux/UNIX (not Win) TID matches CONN: 0xB9 = TID(185)
JOBNAME contains PID (except z/OS): 0x260C = PID(9740)
On Linux/UNIX (not Win) TID matches CONN: 0xB9 = TID(185)
Note that multiple CONN might share one SVRCONN channel instanceNote that multiple CONN might share one SVRCONN channel instance
Check suitable heartbeats are negotiatedCheck suitable heartbeats are negotiated
Automatic detection if
no-one is connected
@MoragHughson
AMQSCLM
1) Normal operation
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
WLM
Cluster
WLM
Round-robin message distribution
App
server 1
App
server 2
App
server N
. . .
IPPROCS(>=1) IPPROCS(>=1) IPPROCS(>=1)
AMQSCLM AMQSCLM AMQSCLM
@MoragHughson
AMQSCLM
2) Application failure
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
WLM
Cluster
WLM
App
server 1
App
server 2
. . .
IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1)
Messages
start
building
up
AMQSCLM AMQSCLM AMQSCLM
App
server N
Round-robin message distribution
@MoragHughson
AMQSCLM
3) Failure Detection
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
WLM
Cluster
WLM
App
server 1
App
server 2
. . .
IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1)
AMQSCLM
regularly
checks the
queue and
detects
IPPROCS(0)
AMQSCLM AMQSCLM AMQSCLM
App
server N
Round-robin message distribution
@MoragHughson
AMQSCLM
4) Redirect Messages
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
WLM
Cluster
WLM
App
server 1
App
server 2
. . .
IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1)
AMQSCLM
reduces
queue priority
in cluster, to
stop new
messages
arriving
AMQSCLM AMQSCLM AMQSCLM
App
server N
Round-robin message distribution
@MoragHughson
AMQSCLM
5) Un-maroon Msgs
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
WLM
Cluster
WLM
App
server 1
App
server 2
. . .
IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1)
AMQSCLM
tells qmgr to
redistribute
‘stuck’
messages via
the cluster
AMQSCLM AMQSCLM AMQSCLM
App
server N
Round-robin message distribution
@MoragHughson
AMQSCLM summary
The Cluster Queue Monitoring sample program (AMQSCLM)
Shipped with the product as a sample
Precompiled
Source code
Platforms except on z/OS
More information here:
http://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.
mq.dev.doc/q024620_.htm
Are messages
flowing?
@MoragHughson
Real-time/online Monitoring – Queues
Set detail level for QMGR. Override for individual Queues
Gives live view of application responsiveness
ALTER QMGR MONQ(MEDIUM)
ALTER QLOCAL(QUEUE1) MONQ(HIGH)
ALTER QLOCAL(QUEUE2) MONQ(OFF)
AMQ8450: Display queue status details.
QUEUE(QUEUE1) TYPE(QUEUE)
CURDEPTH(16) IPPROCS(3)
LGETDATE(2014-04-08) LGETTIME(17.05.59)
LPUTDATE(2014-04-08) LPUTTIME(17.12.16)
MEDIALOG( ) MONQ(HIGH)
MSGAGE(112) OPPROCS(5)
QTIME(10101414, 10101414) UNCOM(NO)
DIS QSTATUS(QUEUE1) ALL
Age in Seconds of the oldest
message on the queue
Age in Seconds of the oldest
message on the queue
Estimations of the time in Microseconds that messages are waiting on the queue for processing.
First value: Calculated from recent activity
Second value: Calculated from longer term activity
Estimations of the time in Microseconds that messages are waiting on the queue for processing.
First value: Calculated from recent activity
Second value: Calculated from longer term activity
Timestamps of last PUT/GET
to check for recent activity
Timestamps of last PUT/GET
to check for recent activity
Without MONQ you only get the depth and
how many handles are open
Without MONQ you only get the depth and
how many handles are open
@MoragHughson
Real-time/online Monitoring – Channels
Gives live view of channel throughput
ALTER QMGR MONCHL(MEDIUM)
ALTER CHANNEL(CLUSTER1.QM1) CHLTYPE(CLUSRCVR) MONCHL(HIGH)
AMQ8417: Display Channel Status details.
CHANNEL(MQHUB.GATEWAY2) CHLTYPE(CLUSSDR)
BATCHES(52) BATCHSZ(50)
BUFSRCVD(55) BUFSSENT(1616)
BYTSRCVD(1748) BYTSSENT(1192330)
CHSTADA(2014-04-08) CHSTATI(17.49.03)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(127.0.0.1(1422)) CURLUWID(0107445310001B34)
CURMSGS(50) CURRENT
CURSEQNO(11823) EXITTIME(0,0)
HBINT(5) INDOUBT(YES)
JOBNAME(00002DA4000047D0) LOCLADDR(127.0.0.1(53557))
LONGRTS(999999999) LSTLUWID(0107445310001B33)
LSTMSGDA(2014-04-08) LSTMSGTI(17.49.51)
LSTSEQNO(11773) MCASTAT(RUNNING)
MONCHL(MEDIUM) MSGS(1580)
NETTIME(137538,29555) NPMSPEED(FAST)
RQMNAME(GATEWAY2) SHORTRTS(180)
SSLCERTI( ) SSLKEYDA( )
SSLKEYTI( ) SSLPEER( )
SSLRKEYS(0) STATUS(RUNNING)
STOPREQ(NO) SUBSTATE(RECEIVE)
XBATCHSZ(20,17) XMITQ(SYSTEM.CLUSTER.TRANSMIT.QUEUE)
XQMSGSA(112) XQTIME(545784,3929968)
RVERSION(07050002) RPRODUCT(MQMM)
DIS CHSTATUS(MQHUB.GATEWAY2) ALL
Short/long term calculations of how
full your batches are getting, to help
you tune BATCHSZ/BATCHINT
Short/long term calculations of how
full your batches are getting, to help
you tune BATCHSZ/BATCHINT
Last time a message was sent over
the channel
Last time a message was sent over
the channel
Depth of messages on XMITQ
for this channel (capped at 999)
Depth of messages on XMITQ
for this channel (capped at 999)
Short/long term calculations of how
long messages are waiting on the
XMITQ for transmission
Short/long term calculations of how
long messages are waiting on the
XMITQ for transmission
@MoragHughson
Accounting and Statistics Overview
Monitoring data sent as a PCF message at a configured interval
Statistics – scoped to a Queue / Channel / QMGR
Accounting – scoped to an individual CONN and Queue / QMGR
Applications must consume the messages
Enterprise monitoring – Tivoli ITCAM / Tivoli OMEGAMON XE for Messaging
Sample applications and SupportPacs
Custom applications – com.ibm.mq.headers Java package
PCF accounting
collection app
PCF accounting
collection app
QueuesQueues ChannelsChannels
CONNsCONNs
PCF statistics
collection app
PCF statistics
collection app
Statistics
Messages
Accounting
Messages
QMGRQMGR
SYSTEM.ADMIN.STATISTICS.QUEUE SYSTEM.ADMIN.ACCOUNTING.QUEUE
@MoragHughson
Taking a look with SupportPac MS0P
Time period
MQ Statistics at QMGR level
Detailed queue statistics
@MoragHughson
Simple practical example using amqsmon
Overall QMGR busyness
Simple data format
Multiple values are
[Persistent, NonPersistent]
One message every X seconds
Use amqsmon directly (perl/cron)
Low/high water marks for
Connections
Subscriptions
ALTER QMGR STATMQI(ON)
Wait a bit
RESET QMGR TYPE(STATISTICS)
amqsmon -m GATEWAY1 -t statistics -a -w 0
MonitoringType: MQIStatistics
QueueManager: 'GATEWAY1'
IntervalStartDate: '2014-04-09'
IntervalStartTime: '00.00.35'
IntervalEndDate: '2014-04-09'
IntervalEndTime: '00.01.13'
CommandLevel: 700
ConnCount: 35
PutCount: [271, 0]
PutFailCount: 0
Put1Count: [2, 0]
Put1FailCount: 0
PutBytes: [273976, 0]
GetCount: [270, 0]
GetBytes: [269468, 0]
GetFailCount: 19
DurableSubscriptionHighWater: [0, 0, 0, 0]
DurableSubscriptionLowWater: [0, 0, 0, 0]
NonDurableSubscriptionHighWater: [0, 0, 0, 0]
NonDurableSubscriptionLowWater: [0, 0, 0, 0]
PutTopicCount: [0, 0]
PutTopicFailCount: 0
Put1TopicCount: [0, 0]
Put1TopicFailCount: 0
PutTopicBytes: [0, 0]
PublishMsgCount: [0, 0]
PublishMsgBytes: [0, 0]
Where are messages
going?
@MoragHughson
ClusterCluster
MQ networks can be complex
At each of the dots stuck / mis-sent messages are possible
MQOPENs of the wrong Queue / Queue Manager by apps
Full queues
Stopped channels
Stopped apps
Incorrectly configured QREMOTE/QALIAS routing objects
Cluster membership problems
The standard problem diagnosis approach
Methodically checking channels/queues/DLQs at each point
Is there anything to speed up this process?
App1App1
App2AApp2A
App2BApp2B
Requests Replies
Replies
SDR/RCVRSDR/RCVR
SDR/RCVRSDR/RCVR
@MoragHughson
ClusterCluster
Trace Route
MQ has the ability to inject tracer messages
Hidden from applications
Generate activity reports as they pass through
Tools are available to trace routes using these reports
dspmqrte – command line tool supplied with the product
MS0P – Cat 2 SupportPac extension to MQ Explorer
Report messages need to get back to your ReplyQ of your tool
Lets you see the path messages could have taken
Test connectivity through the MQ network
Test cluster workload balancing
Can quickly jump you close to the problem
The point your trace message veers off in the wrong direction
The point the trail goes cold
dspmqrtedspmqrte SDR/RCVRSDR/RCVR
SDR/RCVRSDR/RCVR
dspmqrtedspmqrte
dspmqrtedspmqrte
What are the apps
doing?
@MoragHughson
Application Activity Tracing
Similar infrastructure to Accounting & Statistics
PCF messages on SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE
Configurable via ini file
Can be changed without QMGR restart
Configurable detail level can include partial/full message payload
Frequency options for tuning
Can be enabled on a per-application basis
Via MQCONNX flags
Via application name
– MQ V7.5 JMS clients can supply an application name in the CF
Enables scenarios such as
Application audit trail
Message duplication
Resource usage
Which queues or topics are actually being used
Problem Determination
Which Queue / Queue Manager is the application
actually opening
Application Coding Standards
Does everyone use the MQI in the recommended way
And more …
Health warning: Performance impact
http://ow.ly/vA8wB
Health warning: Performance impact
http://ow.ly/vA8wB
@MoragHughson
Extract from Report
MonitoringType: MQI Activity Trace
QueueManager: 'V71'
Host Name: 'rockall.hursley.ibm.com'
CommandLevel: 710
ApplicationName: 'WebSphere MQ Client for Java'
ApplicationPid: 18612354
UserId: 'mquser'
ConnName: '9.20.95.106'
Channel Type: MQCHT_SVRCONN
Platform: MQPL_UNIX
=======================================================================
=
Time Operation CompCode MQRC HObj (ObjName)
10:04:09 MQXF_INQ MQCC_OK 0000 2
10:04:09 MQXF_CLOSE MQCC_OK 0000 2
10:04:09 MQXF_OPEN MQCC_OK 0000 4 ()
10:04:09 MQXF_INQ MQCC_OK 0000 4
10:04:09 MQXF_CLOSE MQCC_OK 0000 4
10:04:09 MQXF_OPEN MQCC_OK 0000 4 (SYSTEM.DEFAULT.LOCAL.QUEUE)
10:04:09 MQXF_INQ MQCC_OK 0000 4
@MoragHughson
Looking at the data with the amqsact sample
ALTER QMGR ACTVTRC(ON) <- should be tuned via mqat.ini
amqsact -m GATEWAY1 -v
MQI Operation: 6
Operation Id: MQXF_PUT
ApplicationTid: 12451
OperationDate: '2014-04-09'
OperationTime: '01:39:48'
High Res Time: 1397003988665548
Completion Code: MQCC_OK
Reason Code: 0
Hobj: 18225032
Put Options: 139330
Msg length: 460
Recs_present: 0
Known_dest_count: 1
Unknown_dest_count: 0
Invalid_dest_count: 0
Object_type: MQOT_Q
Object_name: 'SENDINGAPP.REPLY'
Object_Q_mgr_name: 'GATEWAY1'
Resolved_Q_Name: 'SENDINGAPP.REPLY'
Resolved_Q_mgr: 'GATEWAY1'
Resolved_local_Q_name: 'SENDINGAPP.REPLY'
Resolved_local_Q_mgr: 'GATEWAY1'
Resolved_type: MQOT_Q
Report Options: 0
Msg_type: MQMT_DATAGRAM
Expiry: -1
Format_name: 'MQHRF2'
Priority: 4
Persistence: 0
Msg_id:
00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 '
00000010: 0207 4453 2007 2603 '..DS .&. '
Correl_id:
00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 '
00000010: 0207 4453 2007 2203 '..DS .". '
Reply_to_Q : ' ^D'
Reply_to_Q_Mgr: ' ^C'
Coded_char_set_id: 1208
Encoding: 273
Put_date: '20140409'
Put_time: '00394866'
Check the options used for coding standardsCheck the options used for coding standards
Check queue name resolution, to find out
why messages are going to the wrong place
Check queue name resolution, to find out
why messages are going to the wrong place
Track individual messages and request/reply
scenarios with Msg_id and Correl_id
Track individual messages and request/reply
scenarios with Msg_id and Correl_id
@MoragHughson
Taking a look with SupportPac MS0P
How can I look back in
time?
@MoragHughson
What happened to my messages at 2am this morning?
Enterprise monitoring solution
DLQ alerts, queue depth alerts, channel status alerts
Unresolved running units of work
Historical MQ monitoring, accounting and stats data
App logs from the time of the problem
Exceptions, MQ error codes, timeouts
MQ error logs for all qmgrs that could have been involved
Channel errors
Authentication issues
??? – what else is there
@MoragHughson
What about the MQ Recovery Log?
For persistent messages inside transactions
MQ logs each operation performed
Why can't we use this to
Look back in time to 2am and see what happened
Recovery the original payload if the app lost the message
See what happened inside long-running units of work
Provide a list of operations within the failed business transaction
MQ documents how you can… if
You use the text formatting tool provided with MQ (dmpmqlog)
The logging is linear so the historical data is available in the tool
You follow the right steps to extract data from running qmgrs
You do the work to follow through the logs
@MoragHughson
Dmpmqlog output is readable, but analysis is tedious
Wouldn't it be easier to let
the computer do the
tedious bit?
LOG RECORD - LSN <0:0:954:44817>
**********
HLG Header: lrecsize 873, version 1, rmid 0, eyecatcher HLRH
LogRecdType . . : AQM Put Message (257)
Eyecatcher . . : ALRH Version . . . . : 1
LogRecdLen . . : 853 LogRecdOwnr . . : 256 (AQM)
XTranid . . . . : TranType: XA
XID: formatID 1463898948, gtrid_length 36, bqual_length 54
gtrid [000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F2]
bqual
[000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F20000000100000000000
00000000000000001]
QueueName . . . : Not known
Qid . . . . . . : {Hash 2147211283, Counter: 5}
ThisLSN . . . . : <0:0:0:0>
PrevLSN . . . . : <0:0:954:43944>
Version . . . . : 4
MapIndex . . . : 199
PrevLink.Locn . : 102408 PrevLink.Length : 8
PrevDataLink . : {High 0, Low 103424}
Data.Locn . . . : 103424 Data.Length . . : 613
Data . . . . . :
00000: 41 51 52 48 04 00 00 00 FF FF FF FF FF FF FF FF AQRH....ÿÿÿÿÿÿÿÿ
00016: 00 00 00 00 00 00 00 00 C7 00 00 00 02 00 C0 01 ........Ç.......
00032: 00 00 00 00 04 00 01 00 A5 00 00 00 00 00 00 00 ........¥.......
00048: 63 00 00 00 41 4D 51 20 49 49 42 30 31 5F 51 4D c...AMQ IIB01_QM
00064: 20 20 20 20 D9 26 B0 52 20 08 53 F5 30 30 30 30 .&.R .Sõ0000
00080: 30 30 39 39 00 00 00 00 00 00 00 00 00 00 00 00 0099............
00096: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00112: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ................
00128: 00 00 00 00 FF FF FF FF 00 00 00 00 04 00 00 09 ....ÿÿÿÿ........
00144: 00 00 00 00 E3 ED 04 80 10 FD 67 E4 FF FF FF FF ....ãí.€.ýgäÿÿÿÿ
00160: 4D 44 20 20 01 00 00 00 00 00 00 00 08 00 00 00 MD ............
00176: 00 00 00 00 11 01 00 00 B8 04 00 00 4D 51 48 52 ........¸...MQHR
00192: 46 32 20 20 04 00 00 00 01 00 00 00 20 20 20 20 F2 ........
00208: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
00224: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
00240: 20 20 20 20 20 20 20 20 20 20 20 20 49 49 42 30 IIB0
00256: 31 5F 51 4D 20 20 20 20 20 20 20 20 20 20 20 20 1_QM
00272: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
00288: 20 20 20 20 20 20 20 20 20 20 20 20 4D 55 53 52 MUSR
00304: 5F 4D 51 41 44 4D 49 4E 16 01 05 15 00 00 00 64 _MQADMIN.......d
00320: 20 3E AC 57 48 B3 09 B8 71 B0 4C F2 03 00 00 00 >¬WH..¸q.Lò....
00336: 00 00 00 00 00 00 00 0B 20 20 20 20 20 20 20 20 ........
00352: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
00368: 20 20 20 20 20 20 20 20 1C 00 00 00 57 65 62 53 ....WebS
00384: 70 68 65 72 65 20 4D 51 20 43 6C 69 65 6E 74 20 phere MQ Client
00400: 66 6F 72 20 4A 61 76 61 32 30 31 33 31 32 31 39 for Java20131219
00416: 31 34 32 32 33 32 32 32 20 20 20 20 00 00 00 00 14223222 ....
00432: A4 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ¤...............
00448: 52 46 48 20 00 00 00 02 00 00 00 A4 00 00 01 11 RFH .......¤....
00464: 00 00 04 B8 4D 51 53 54 52 20 20 20 00 00 00 00 ...¸MQSTR ....
00480: 00 00 04 B8 00 00 00 20 3C 6D 63 64 3E 3C 4D 73 ...¸... <mcd><Ms
00496: 64 3E 6A 6D 73 5F 74 65 78 74 3C 2F 4D 73 64 3E d>jms_text</Msd>
00512: 3C 2F 6D 63 64 3E 20 20 00 00 00 58 3C 6A 6D 73 </mcd> ...X<jms
00528: 3E 3C 44 73 74 3E 71 75 65 75 65 3A 2F 2F 2F 51 ><Dst>queue:///Q
00544: 31 3C 2F 44 73 74 3E 3C 54 6D 73 3E 31 33 38 37 1</Dst><Tms>1387
00560: 34 36 32 39 35 32 32 32 36 3C 2F 54 6D 73 3E 3C 462952226</Tms><
00576: 43 69 64 3E 30 30 30 30 30 30 39 39 3C 2F 43 69 Cid>00000099</Ci
00592: 64 3E 3C 44 6C 76 3E 32 3C 2F 44 6C 76 3E 3C 2F d><Dlv>2</Dlv></
00608: 6A 6D 73 3E 61 jms>a
Ordered unique IDs for each record (LSN)Ordered unique IDs for each record (LSN)
A set of documented record typesA set of documented record types
Transaction information with XIDs, or re-used
MQ transaction IDs
Transaction information with XIDs, or re-used
MQ transaction IDs
MQMD header data at discoverable offsets
in the hex of a message Put
MQMD header data at discoverable offsets
in the hex of a message Put
The message payload itselfThe message payload itself
@MoragHughson
Check out dmpmqlog scraper tool
Takes the tedium out of analysing the output from dmpmqlog
Created by Peter Broadhurst
Download from
http://www.ibm.com/support/docview.wss?uid=swg21660642
Generates file per message PUT in the supplied data.
Summary file
java -jar dmpmqlog.scraper-20151201.jar -b little-endian -i dmpmqlog.txt -o .
@MoragHughson
dmpmqlog scraper tool output
Generates file per message PUT in the supplied data.
Summary file
@MoragHughson
Summary
Lots of tools in your MQ toolbox!
On-line Status Commands
DISPLAY CONN
DISPLAY QSTATUS
DISPLAY CHSTATUS
Cluster monitoring – AMQSCLM
Off-line Statistics and Accounting
amqsmon and MS0P to view
Tracking
Trace-route
Application Activity Trace
MQ Recovery Logs
dmpmqlog scraper
@MoragHughson
Notices and Disclaimers
Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or
transmitted in any form without written permission from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been
reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM
shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY,
EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF
THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT
OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements
under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without
notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are
presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual
performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products,
programs or services available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily
reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to,
nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal
counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s
business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or
represent or warrant that its services or products will ensure that the customer is in compliance with any law.
@MoragHughson
Notices and Disclaimers (con’t)
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products in connection with this
publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to
interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED,
INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any
IBM patents, copyrights, trademarks or other intellectual property right.
• IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document
Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand,
ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™,
PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®,
pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®,
urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International
Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names
might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at
"Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
Thank You
Your Feedback is
Important!
Session AME-2289
Access the InterConnect 2015
Conference CONNECT Attendee
Portal to complete your session
surveys from your smartphone,
laptop or conference kiosk.

More Related Content

What's hot

MQ Security Overview
MQ Security OverviewMQ Security Overview
MQ Security OverviewMarkTaylorIBM
 
IBM MQ - better application performance
IBM MQ - better application performanceIBM MQ - better application performance
IBM MQ - better application performanceMarkTaylorIBM
 
HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applicati...
HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applicati...HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applicati...
HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applicati...Matt Leming
 
What's new in IBM MQ Messaging
What's new in IBM MQ MessagingWhat's new in IBM MQ Messaging
What's new in IBM MQ MessagingMarkTaylorIBM
 
IBM MQ - Comparing Distributed and z/OS platforms
IBM MQ - Comparing Distributed and z/OS platformsIBM MQ - Comparing Distributed and z/OS platforms
IBM MQ - Comparing Distributed and z/OS platformsMarkTaylorIBM
 
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ Clusters
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ ClustersIBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ Clusters
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ ClustersDavid Ware
 
IBM MQ v8 and JMS 2.0
IBM MQ v8 and JMS 2.0IBM MQ v8 and JMS 2.0
IBM MQ v8 and JMS 2.0Matthew White
 
MQ What's New Beyond V8 - V8003 level
MQ What's New Beyond V8 - V8003 levelMQ What's New Beyond V8 - V8003 level
MQ What's New Beyond V8 - V8003 levelMarkTaylorIBM
 
IBM MQ: Managing Workloads, Scaling and Availability with MQ Clusters
IBM MQ: Managing Workloads, Scaling and Availability with MQ ClustersIBM MQ: Managing Workloads, Scaling and Availability with MQ Clusters
IBM MQ: Managing Workloads, Scaling and Availability with MQ ClustersDavid Ware
 
IBM MQ - Monitoring and Managing Hybrid Messaging Environments
IBM MQ - Monitoring and Managing Hybrid Messaging EnvironmentsIBM MQ - Monitoring and Managing Hybrid Messaging Environments
IBM MQ - Monitoring and Managing Hybrid Messaging EnvironmentsMarkTaylorIBM
 
Where is my MQ message on z/OS?
Where is my MQ message on z/OS?Where is my MQ message on z/OS?
Where is my MQ message on z/OS?Matt Leming
 
IBM WebSphere MQ: Using Publish/Subscribe in an MQ Network
IBM WebSphere MQ: Using Publish/Subscribe in an MQ NetworkIBM WebSphere MQ: Using Publish/Subscribe in an MQ Network
IBM WebSphere MQ: Using Publish/Subscribe in an MQ NetworkDavid Ware
 
Where is My Message
Where is My MessageWhere is My Message
Where is My MessageMatt Leming
 
IBM MQ: Using Publish/Subscribe in an MQ Network
IBM MQ: Using Publish/Subscribe in an MQ NetworkIBM MQ: Using Publish/Subscribe in an MQ Network
IBM MQ: Using Publish/Subscribe in an MQ NetworkDavid Ware
 
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
 HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ... HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...Matt Leming
 
IBM MQ for z/OS The Latest and Greatest Enhancements
IBM MQ for z/OS The Latest and Greatest EnhancementsIBM MQ for z/OS The Latest and Greatest Enhancements
IBM MQ for z/OS The Latest and Greatest EnhancementsPete Siddall
 
IBM WebSphere MQ for z/OS - Security
IBM WebSphere MQ for z/OS - SecurityIBM WebSphere MQ for z/OS - Security
IBM WebSphere MQ for z/OS - SecurityDamon Cross
 
Hhm 3479 mq clustering and shared queues for high availability
Hhm 3479 mq clustering and shared queues for high availabilityHhm 3479 mq clustering and shared queues for high availability
Hhm 3479 mq clustering and shared queues for high availabilityPete Siddall
 

What's hot (20)

MQ Security Overview
MQ Security OverviewMQ Security Overview
MQ Security Overview
 
IBM MQ - better application performance
IBM MQ - better application performanceIBM MQ - better application performance
IBM MQ - better application performance
 
HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applicati...
HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applicati...HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applicati...
HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applicati...
 
What's new in IBM MQ Messaging
What's new in IBM MQ MessagingWhat's new in IBM MQ Messaging
What's new in IBM MQ Messaging
 
IBM MQ - Comparing Distributed and z/OS platforms
IBM MQ - Comparing Distributed and z/OS platformsIBM MQ - Comparing Distributed and z/OS platforms
IBM MQ - Comparing Distributed and z/OS platforms
 
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ Clusters
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ ClustersIBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ Clusters
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ Clusters
 
IBM MQ v8 and JMS 2.0
IBM MQ v8 and JMS 2.0IBM MQ v8 and JMS 2.0
IBM MQ v8 and JMS 2.0
 
WebSphere MQ tutorial
WebSphere MQ tutorialWebSphere MQ tutorial
WebSphere MQ tutorial
 
MQ What's New Beyond V8 - V8003 level
MQ What's New Beyond V8 - V8003 levelMQ What's New Beyond V8 - V8003 level
MQ What's New Beyond V8 - V8003 level
 
IBM MQ: Managing Workloads, Scaling and Availability with MQ Clusters
IBM MQ: Managing Workloads, Scaling and Availability with MQ ClustersIBM MQ: Managing Workloads, Scaling and Availability with MQ Clusters
IBM MQ: Managing Workloads, Scaling and Availability with MQ Clusters
 
IBM What's New in MQ V8
IBM What's New in MQ V8IBM What's New in MQ V8
IBM What's New in MQ V8
 
IBM MQ - Monitoring and Managing Hybrid Messaging Environments
IBM MQ - Monitoring and Managing Hybrid Messaging EnvironmentsIBM MQ - Monitoring and Managing Hybrid Messaging Environments
IBM MQ - Monitoring and Managing Hybrid Messaging Environments
 
Where is my MQ message on z/OS?
Where is my MQ message on z/OS?Where is my MQ message on z/OS?
Where is my MQ message on z/OS?
 
IBM WebSphere MQ: Using Publish/Subscribe in an MQ Network
IBM WebSphere MQ: Using Publish/Subscribe in an MQ NetworkIBM WebSphere MQ: Using Publish/Subscribe in an MQ Network
IBM WebSphere MQ: Using Publish/Subscribe in an MQ Network
 
Where is My Message
Where is My MessageWhere is My Message
Where is My Message
 
IBM MQ: Using Publish/Subscribe in an MQ Network
IBM MQ: Using Publish/Subscribe in an MQ NetworkIBM MQ: Using Publish/Subscribe in an MQ Network
IBM MQ: Using Publish/Subscribe in an MQ Network
 
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
 HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ... HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
 
IBM MQ for z/OS The Latest and Greatest Enhancements
IBM MQ for z/OS The Latest and Greatest EnhancementsIBM MQ for z/OS The Latest and Greatest Enhancements
IBM MQ for z/OS The Latest and Greatest Enhancements
 
IBM WebSphere MQ for z/OS - Security
IBM WebSphere MQ for z/OS - SecurityIBM WebSphere MQ for z/OS - Security
IBM WebSphere MQ for z/OS - Security
 
Hhm 3479 mq clustering and shared queues for high availability
Hhm 3479 mq clustering and shared queues for high availabilityHhm 3479 mq clustering and shared queues for high availability
Hhm 3479 mq clustering and shared queues for high availability
 

Viewers also liked

IBM WebSphere MQ V8 Security Features: Deep Dive
IBM WebSphere MQ V8 Security Features: Deep DiveIBM WebSphere MQ V8 Security Features: Deep Dive
IBM WebSphere MQ V8 Security Features: Deep DiveMorag Hughson
 
Secure Messages with IBM WebSphere MQ Advanced Message Security
Secure Messages with IBM WebSphere MQ Advanced Message SecuritySecure Messages with IBM WebSphere MQ Advanced Message Security
Secure Messages with IBM WebSphere MQ Advanced Message SecurityMorag Hughson
 
Secure Your Messages with IBM MQ Advanced Message Security
Secure Your Messages with IBM MQ Advanced Message SecuritySecure Your Messages with IBM MQ Advanced Message Security
Secure Your Messages with IBM MQ Advanced Message SecurityMorag Hughson
 
DevOps & Continuous Test for IIB and IBM MQ
DevOps & Continuous Test for IIB and IBM MQDevOps & Continuous Test for IIB and IBM MQ
DevOps & Continuous Test for IIB and IBM MQStuart Feasey
 
IBM MQ V8 Security: Latest Features Deep-Dive
IBM MQ V8 Security: Latest Features Deep-DiveIBM MQ V8 Security: Latest Features Deep-Dive
IBM MQ V8 Security: Latest Features Deep-DiveMorag Hughson
 
Websphere MQ admin guide
Websphere MQ admin guideWebsphere MQ admin guide
Websphere MQ admin guideRam Babu
 
J abrahams biologic design uk
J abrahams biologic design ukJ abrahams biologic design uk
J abrahams biologic design ukJoy Elliott
 
Hoe ontwerp je een goede koffie training? Omstandigheden, structuur en vaardi...
Hoe ontwerp je een goede koffie training? Omstandigheden, structuur en vaardi...Hoe ontwerp je een goede koffie training? Omstandigheden, structuur en vaardi...
Hoe ontwerp je een goede koffie training? Omstandigheden, structuur en vaardi...Cerianne Bury
 
Ac fbrewingessentials cbury506060
Ac fbrewingessentials cbury506060Ac fbrewingessentials cbury506060
Ac fbrewingessentials cbury506060Cerianne Bury
 
Ready For Liftoff: Instructional Shifts in Social Studies
Ready For Liftoff: Instructional Shifts in Social StudiesReady For Liftoff: Instructional Shifts in Social Studies
Ready For Liftoff: Instructional Shifts in Social Studiesammurtlow
 
L ramirez agrosilvopastoril systems mexico
L ramirez agrosilvopastoril systems mexicoL ramirez agrosilvopastoril systems mexico
L ramirez agrosilvopastoril systems mexicoJoy Elliott
 
How to Create a Stand Out Resume
How to Create a Stand Out ResumeHow to Create a Stand Out Resume
How to Create a Stand Out ResumeNicole Steen
 

Viewers also liked (20)

IBM WebSphere MQ V8 Security Features: Deep Dive
IBM WebSphere MQ V8 Security Features: Deep DiveIBM WebSphere MQ V8 Security Features: Deep Dive
IBM WebSphere MQ V8 Security Features: Deep Dive
 
Secure Messages with IBM WebSphere MQ Advanced Message Security
Secure Messages with IBM WebSphere MQ Advanced Message SecuritySecure Messages with IBM WebSphere MQ Advanced Message Security
Secure Messages with IBM WebSphere MQ Advanced Message Security
 
Secure Your Messages with IBM MQ Advanced Message Security
Secure Your Messages with IBM MQ Advanced Message SecuritySecure Your Messages with IBM MQ Advanced Message Security
Secure Your Messages with IBM MQ Advanced Message Security
 
DevOps & Continuous Test for IIB and IBM MQ
DevOps & Continuous Test for IIB and IBM MQDevOps & Continuous Test for IIB and IBM MQ
DevOps & Continuous Test for IIB and IBM MQ
 
IBM MQ V8 Security
IBM MQ V8 SecurityIBM MQ V8 Security
IBM MQ V8 Security
 
IBM MQ V8 Security: Latest Features Deep-Dive
IBM MQ V8 Security: Latest Features Deep-DiveIBM MQ V8 Security: Latest Features Deep-Dive
IBM MQ V8 Security: Latest Features Deep-Dive
 
Websphere MQ admin guide
Websphere MQ admin guideWebsphere MQ admin guide
Websphere MQ admin guide
 
Sunu
SunuSunu
Sunu
 
J abrahams biologic design uk
J abrahams biologic design ukJ abrahams biologic design uk
J abrahams biologic design uk
 
Cöv
CövCöv
Cöv
 
Evaluation 2
Evaluation 2Evaluation 2
Evaluation 2
 
Addeo smp4
Addeo   smp4Addeo   smp4
Addeo smp4
 
Evaluation 2
Evaluation 2Evaluation 2
Evaluation 2
 
Evaluation q6
Evaluation q6Evaluation q6
Evaluation q6
 
Hoe ontwerp je een goede koffie training? Omstandigheden, structuur en vaardi...
Hoe ontwerp je een goede koffie training? Omstandigheden, structuur en vaardi...Hoe ontwerp je een goede koffie training? Omstandigheden, structuur en vaardi...
Hoe ontwerp je een goede koffie training? Omstandigheden, structuur en vaardi...
 
Evaluation 2
Evaluation 2Evaluation 2
Evaluation 2
 
Ac fbrewingessentials cbury506060
Ac fbrewingessentials cbury506060Ac fbrewingessentials cbury506060
Ac fbrewingessentials cbury506060
 
Ready For Liftoff: Instructional Shifts in Social Studies
Ready For Liftoff: Instructional Shifts in Social StudiesReady For Liftoff: Instructional Shifts in Social Studies
Ready For Liftoff: Instructional Shifts in Social Studies
 
L ramirez agrosilvopastoril systems mexico
L ramirez agrosilvopastoril systems mexicoL ramirez agrosilvopastoril systems mexico
L ramirez agrosilvopastoril systems mexico
 
How to Create a Stand Out Resume
How to Create a Stand Out ResumeHow to Create a Stand Out Resume
How to Create a Stand Out Resume
 

Similar to Where is My Message?: Use MQ Tools to Work Out What Applications Have Done

Monitoring and problem determination of your mq distributed systems
Monitoring and problem determination of your mq distributed systemsMonitoring and problem determination of your mq distributed systems
Monitoring and problem determination of your mq distributed systemsmatthew1001
 
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...Peter Broadhurst
 
Queue Manager both as sender and Receiver.docx
Queue Manager both as sender and Receiver.docxQueue Manager both as sender and Receiver.docx
Queue Manager both as sender and Receiver.docxsarvank2
 
Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...
Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...
Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...abdenour boussioud
 
Performance Metrics Overview
Performance Metrics OverviewPerformance Metrics Overview
Performance Metrics OverviewRudyansyah -
 
Cisco data center support
Cisco data center supportCisco data center support
Cisco data center supportKrunal Shah
 
IBM MQ Whats new - up to 9.3.4.pptx
IBM MQ Whats new - up to 9.3.4.pptxIBM MQ Whats new - up to 9.3.4.pptx
IBM MQ Whats new - up to 9.3.4.pptxMatt Leming
 
GSM TRAINING ENGINEER
GSM TRAINING ENGINEERGSM TRAINING ENGINEER
GSM TRAINING ENGINEERrizfadil98
 
XL Messaging Manager P2P - TOK
XL Messaging Manager P2P - TOKXL Messaging Manager P2P - TOK
XL Messaging Manager P2P - TOKpusatbelajar
 
Cassandra 2.1 boot camp, Overview
Cassandra 2.1 boot camp, OverviewCassandra 2.1 boot camp, Overview
Cassandra 2.1 boot camp, OverviewJoshua McKenzie
 
Deep review of LMS process
Deep review of LMS processDeep review of LMS process
Deep review of LMS processRiyaj Shamsudeen
 
Technical Overview of QUIC
Technical  Overview of QUICTechnical  Overview of QUIC
Technical Overview of QUICshigeki_ohtsu
 
eRAN8.1 Radio & Performance-Interference Handling-CAMC feature introduction.ppt
eRAN8.1 Radio & Performance-Interference Handling-CAMC feature introduction.ppteRAN8.1 Radio & Performance-Interference Handling-CAMC feature introduction.ppt
eRAN8.1 Radio & Performance-Interference Handling-CAMC feature introduction.pptAhmed963381
 
transport layer protocols
transport layer protocolstransport layer protocols
transport layer protocolsBE Smârt
 

Similar to Where is My Message?: Use MQ Tools to Work Out What Applications Have Done (20)

Monitoring and problem determination of your mq distributed systems
Monitoring and problem determination of your mq distributed systemsMonitoring and problem determination of your mq distributed systems
Monitoring and problem determination of your mq distributed systems
 
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
 
Queue Manager both as sender and Receiver.docx
Queue Manager both as sender and Receiver.docxQueue Manager both as sender and Receiver.docx
Queue Manager both as sender and Receiver.docx
 
Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...
Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...
Best practices for catalyst 4500 4000, 5500-5000, and 6500-6000 series switch...
 
Performance Metrics Overview
Performance Metrics OverviewPerformance Metrics Overview
Performance Metrics Overview
 
CDMA BSC 6600
CDMA BSC 6600CDMA BSC 6600
CDMA BSC 6600
 
Ccnpswitch
CcnpswitchCcnpswitch
Ccnpswitch
 
Cisco data center support
Cisco data center supportCisco data center support
Cisco data center support
 
IBM MQ Whats new - up to 9.3.4.pptx
IBM MQ Whats new - up to 9.3.4.pptxIBM MQ Whats new - up to 9.3.4.pptx
IBM MQ Whats new - up to 9.3.4.pptx
 
GSM TRAINING ENGINEER
GSM TRAINING ENGINEERGSM TRAINING ENGINEER
GSM TRAINING ENGINEER
 
XL Messaging Manager P2P - TOK
XL Messaging Manager P2P - TOKXL Messaging Manager P2P - TOK
XL Messaging Manager P2P - TOK
 
Cassandra 2.1 boot camp, Overview
Cassandra 2.1 boot camp, OverviewCassandra 2.1 boot camp, Overview
Cassandra 2.1 boot camp, Overview
 
Deep review of LMS process
Deep review of LMS processDeep review of LMS process
Deep review of LMS process
 
Lte protocols
Lte protocolsLte protocols
Lte protocols
 
PhD Slides
PhD SlidesPhD Slides
PhD Slides
 
Technical Overview of QUIC
Technical  Overview of QUICTechnical  Overview of QUIC
Technical Overview of QUIC
 
eRAN8.1 Radio & Performance-Interference Handling-CAMC feature introduction.ppt
eRAN8.1 Radio & Performance-Interference Handling-CAMC feature introduction.ppteRAN8.1 Radio & Performance-Interference Handling-CAMC feature introduction.ppt
eRAN8.1 Radio & Performance-Interference Handling-CAMC feature introduction.ppt
 
transport layer protocols
transport layer protocolstransport layer protocols
transport layer protocols
 
Cics faqs
Cics faqsCics faqs
Cics faqs
 
WebSphere MQ introduction
WebSphere MQ introductionWebSphere MQ introduction
WebSphere MQ introduction
 

Recently uploaded

Software Project Health Check: Best Practices and Techniques for Your Product...
Software Project Health Check: Best Practices and Techniques for Your Product...Software Project Health Check: Best Practices and Techniques for Your Product...
Software Project Health Check: Best Practices and Techniques for Your Product...Velvetech LLC
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...Technogeeks
 
Folding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesFolding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesPhilip Schwarz
 
What is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need ItWhat is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need ItWave PLM
 
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024StefanoLambiase
 
Automate your Kamailio Test Calls - Kamailio World 2024
Automate your Kamailio Test Calls - Kamailio World 2024Automate your Kamailio Test Calls - Kamailio World 2024
Automate your Kamailio Test Calls - Kamailio World 2024Andreas Granig
 
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML DiagramsUnveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML DiagramsAhmed Mohamed
 
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmIntelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmSujith Sukumaran
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringHironori Washizaki
 
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaReact Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaHanief Utama
 
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...Natan Silnitsky
 
PREDICTING RIVER WATER QUALITY ppt presentation
PREDICTING  RIVER  WATER QUALITY  ppt presentationPREDICTING  RIVER  WATER QUALITY  ppt presentation
PREDICTING RIVER WATER QUALITY ppt presentationvaddepallysandeep122
 
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company OdishaBalasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odishasmiwainfosol
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationBradBedford3
 
Ahmed Motair CV April 2024 (Senior SW Developer)
Ahmed Motair CV April 2024 (Senior SW Developer)Ahmed Motair CV April 2024 (Senior SW Developer)
Ahmed Motair CV April 2024 (Senior SW Developer)Ahmed Mater
 
MYjobs Presentation Django-based project
MYjobs Presentation Django-based projectMYjobs Presentation Django-based project
MYjobs Presentation Django-based projectAnoyGreter
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanyChristoph Pohl
 
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEECloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEEVICTOR MAESTRE RAMIREZ
 

Recently uploaded (20)

Software Project Health Check: Best Practices and Techniques for Your Product...
Software Project Health Check: Best Practices and Techniques for Your Product...Software Project Health Check: Best Practices and Techniques for Your Product...
Software Project Health Check: Best Practices and Techniques for Your Product...
 
Advantages of Odoo ERP 17 for Your Business
Advantages of Odoo ERP 17 for Your BusinessAdvantages of Odoo ERP 17 for Your Business
Advantages of Odoo ERP 17 for Your Business
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...
 
Folding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesFolding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a series
 
What is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need ItWhat is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need It
 
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
 
Automate your Kamailio Test Calls - Kamailio World 2024
Automate your Kamailio Test Calls - Kamailio World 2024Automate your Kamailio Test Calls - Kamailio World 2024
Automate your Kamailio Test Calls - Kamailio World 2024
 
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML DiagramsUnveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML Diagrams
 
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmIntelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalm
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their Engineering
 
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaReact Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief Utama
 
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort ServiceHot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
 
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
 
PREDICTING RIVER WATER QUALITY ppt presentation
PREDICTING  RIVER  WATER QUALITY  ppt presentationPREDICTING  RIVER  WATER QUALITY  ppt presentation
PREDICTING RIVER WATER QUALITY ppt presentation
 
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company OdishaBalasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion Application
 
Ahmed Motair CV April 2024 (Senior SW Developer)
Ahmed Motair CV April 2024 (Senior SW Developer)Ahmed Motair CV April 2024 (Senior SW Developer)
Ahmed Motair CV April 2024 (Senior SW Developer)
 
MYjobs Presentation Django-based project
MYjobs Presentation Django-based projectMYjobs Presentation Django-based project
MYjobs Presentation Django-based project
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
 
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEECloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEE
 

Where is My Message?: Use MQ Tools to Work Out What Applications Have Done

  • 1. © 2015 IBM Corporation@MoragHughson Session AME-2289 Where’s my message? Morag Hughson - hughson@uk.ibm.com IBM Hursley - UK
  • 2. @MoragHughson Symptoms include: Timeouts for request/response messaging Inconsistent data between front-end and back-end systems Customer complaints about ‘missing’ transactions Developer frustration when testing an app / MQ environment
  • 3. @MoragHughson Agenda – the MQ Toolbox Who’s connected? Are messages flowing? Where are messages going? What are the apps doing? How can I look back in time? Presentation contains MQSC example commands and output. Your own admin tools will vary by task, situation and preference. Tasks can be performed in PowerShell, MQ Explorer etc. as desired.
  • 5. @MoragHughson DISPLAY CONN DISPLAY CONN(*) TYPE(HANDLE) ALL DIS CONN(*) ALL AMQ8276: Display Connection details. CONN(577C425321295301) EXTCONN(414D5143474154455741593120202020) TYPE(HANDLE) OBJNAME(WLMMDB.REQUEST) OBJTYPE(QUEUE) ASTATE(NONE) HSTATE(INACTIVE) OPENOPTS(MQOO_OUTPUT,MQOO_FAIL_IF_QUIESCING) READA(NO) OBJNAME(SENDINGAPP.REPLY) OBJTYPE(QUEUE) ASTATE(ACTIVE) HSTATE(ACTIVE) OPENOPTS(MQOO_INPUT_SHARED,MQOO_INQUIRE,MQOO_SAVE_ALL_CONTEXT,MQOO_FAIL_IF_QUIESCING) READA(NO) AMQ8276: Display Connection details. CONN(577C425321295301) EXTCONN(414D5143474154455741593120202020) TYPE(CONN) PID(9740) TID(185) APPLDESC(WebSphere MQ Channel) APPLTAG(jms/GATEWAY1_CF) APPLTYPE(SYSTEM) ASTATE(NONE) CHANNEL(WAS.CLIENTS) CONNAME(127.0.0.1) CONNOPTS(MQCNO_SHARED_BINDING) USERID(pbroad) UOWLOG( ) UOWSTDA(2014-04-08) UOWSTTI(13.24.00) UOWLOGDA( ) UOWLOGTI( ) URTYPE(XA) EXTURID(XA_FORMATID[DSAW] XA_GTRID[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809] XA_BQUAL[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809000000010000000000000000 000000000001]) QMURID(0.7940075) UOWSTATE(ACTIVE) Use CONN to match TYPE(CONN) and TYPE(HANDLE) records Use CONN to match TYPE(CONN) and TYPE(HANDLE) records TYPE(HANDLE) records let you find applications by the objects they access. See all open handles for an app in one place, unlike DIS QSTATUS records TYPE(HANDLE) records let you find applications by the objects they access. See all open handles for an app in one place, unlike DIS QSTATUS records Long running UOW information. XID can be tied up with app server txn timeout Long running UOW information. XID can be tied up with app server txn timeout Channel name + IP help identify client apps.Channel name + IP help identify client apps. MQ V7.5 and later JMS clients can supply an application name in the CF MQ V7.5 and later JMS clients can supply an application name in the CF
  • 6. @MoragHughson DISPLAY CHSTATUS DISPLAY CHSTATUS(*) AMQ8417: Display Channel Status details. CHANNEL(WAS.CLIENTS) CHLTYPE(SVRCONN) BUFSRCVD(17) BUFSSENT(13) BYTSRCVD(2296) BYTSSENT(2456) CHSTADA(2014-04-08) CHSTATI(15.26.59) COMPHDR(NONE,NONE) COMPMSG(NONE,NONE) COMPRATE(0,0) COMPTIME(0,0) CONNAME(127.0.0.1) CURRENT EXITTIME(0,0) HBINT(5) JOBNAME(0000260C000000B9) LOCLADDR( ) LSTMSGDA(2014-04-08) LSTMSGTI(15.26.59) MCASTAT(RUNNING) MCAUSER(pbroad) MONCHL(OFF) MSGS(6) RAPPLTAG(jar) SSLCERTI(CN=ExampleCA,O=Example) SSLKEYDA( ) SSLKEYTI( ) SSLPEER(SERIALNUMBER=53:43:FD:D6,CN=ExampleApp1,O=Example) SSLRKEYS(0) STATUS(RUNNING) STOPREQ(NO) SUBSTATE(RECEIVE) CURSHCNV(1) MAXSHCNV(1) RVERSION(00000000) RPRODUCT(MQJM) See SSLPEER information not in DIS CONNSee SSLPEER information not in DIS CONN JOBNAME contains PID (except z/OS): 0x260C = PID(9740) On Linux/UNIX (not Win) TID matches CONN: 0xB9 = TID(185) JOBNAME contains PID (except z/OS): 0x260C = PID(9740) On Linux/UNIX (not Win) TID matches CONN: 0xB9 = TID(185) Note that multiple CONN might share one SVRCONN channel instanceNote that multiple CONN might share one SVRCONN channel instance Check suitable heartbeats are negotiatedCheck suitable heartbeats are negotiated
  • 8. @MoragHughson AMQSCLM 1) Normal operation Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps WLM Cluster WLM Round-robin message distribution App server 1 App server 2 App server N . . . IPPROCS(>=1) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM AMQSCLM AMQSCLM
  • 9. @MoragHughson AMQSCLM 2) Application failure Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps WLM Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) Messages start building up AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution
  • 10. @MoragHughson AMQSCLM 3) Failure Detection Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps WLM Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM regularly checks the queue and detects IPPROCS(0) AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution
  • 11. @MoragHughson AMQSCLM 4) Redirect Messages Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps WLM Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM reduces queue priority in cluster, to stop new messages arriving AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution
  • 12. @MoragHughson AMQSCLM 5) Un-maroon Msgs Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps WLM Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM tells qmgr to redistribute ‘stuck’ messages via the cluster AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution
  • 13. @MoragHughson AMQSCLM summary The Cluster Queue Monitoring sample program (AMQSCLM) Shipped with the product as a sample Precompiled Source code Platforms except on z/OS More information here: http://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm. mq.dev.doc/q024620_.htm
  • 15. @MoragHughson Real-time/online Monitoring – Queues Set detail level for QMGR. Override for individual Queues Gives live view of application responsiveness ALTER QMGR MONQ(MEDIUM) ALTER QLOCAL(QUEUE1) MONQ(HIGH) ALTER QLOCAL(QUEUE2) MONQ(OFF) AMQ8450: Display queue status details. QUEUE(QUEUE1) TYPE(QUEUE) CURDEPTH(16) IPPROCS(3) LGETDATE(2014-04-08) LGETTIME(17.05.59) LPUTDATE(2014-04-08) LPUTTIME(17.12.16) MEDIALOG( ) MONQ(HIGH) MSGAGE(112) OPPROCS(5) QTIME(10101414, 10101414) UNCOM(NO) DIS QSTATUS(QUEUE1) ALL Age in Seconds of the oldest message on the queue Age in Seconds of the oldest message on the queue Estimations of the time in Microseconds that messages are waiting on the queue for processing. First value: Calculated from recent activity Second value: Calculated from longer term activity Estimations of the time in Microseconds that messages are waiting on the queue for processing. First value: Calculated from recent activity Second value: Calculated from longer term activity Timestamps of last PUT/GET to check for recent activity Timestamps of last PUT/GET to check for recent activity Without MONQ you only get the depth and how many handles are open Without MONQ you only get the depth and how many handles are open
  • 16. @MoragHughson Real-time/online Monitoring – Channels Gives live view of channel throughput ALTER QMGR MONCHL(MEDIUM) ALTER CHANNEL(CLUSTER1.QM1) CHLTYPE(CLUSRCVR) MONCHL(HIGH) AMQ8417: Display Channel Status details. CHANNEL(MQHUB.GATEWAY2) CHLTYPE(CLUSSDR) BATCHES(52) BATCHSZ(50) BUFSRCVD(55) BUFSSENT(1616) BYTSRCVD(1748) BYTSSENT(1192330) CHSTADA(2014-04-08) CHSTATI(17.49.03) COMPHDR(NONE,NONE) COMPMSG(NONE,NONE) COMPRATE(0,0) COMPTIME(0,0) CONNAME(127.0.0.1(1422)) CURLUWID(0107445310001B34) CURMSGS(50) CURRENT CURSEQNO(11823) EXITTIME(0,0) HBINT(5) INDOUBT(YES) JOBNAME(00002DA4000047D0) LOCLADDR(127.0.0.1(53557)) LONGRTS(999999999) LSTLUWID(0107445310001B33) LSTMSGDA(2014-04-08) LSTMSGTI(17.49.51) LSTSEQNO(11773) MCASTAT(RUNNING) MONCHL(MEDIUM) MSGS(1580) NETTIME(137538,29555) NPMSPEED(FAST) RQMNAME(GATEWAY2) SHORTRTS(180) SSLCERTI( ) SSLKEYDA( ) SSLKEYTI( ) SSLPEER( ) SSLRKEYS(0) STATUS(RUNNING) STOPREQ(NO) SUBSTATE(RECEIVE) XBATCHSZ(20,17) XMITQ(SYSTEM.CLUSTER.TRANSMIT.QUEUE) XQMSGSA(112) XQTIME(545784,3929968) RVERSION(07050002) RPRODUCT(MQMM) DIS CHSTATUS(MQHUB.GATEWAY2) ALL Short/long term calculations of how full your batches are getting, to help you tune BATCHSZ/BATCHINT Short/long term calculations of how full your batches are getting, to help you tune BATCHSZ/BATCHINT Last time a message was sent over the channel Last time a message was sent over the channel Depth of messages on XMITQ for this channel (capped at 999) Depth of messages on XMITQ for this channel (capped at 999) Short/long term calculations of how long messages are waiting on the XMITQ for transmission Short/long term calculations of how long messages are waiting on the XMITQ for transmission
  • 17. @MoragHughson Accounting and Statistics Overview Monitoring data sent as a PCF message at a configured interval Statistics – scoped to a Queue / Channel / QMGR Accounting – scoped to an individual CONN and Queue / QMGR Applications must consume the messages Enterprise monitoring – Tivoli ITCAM / Tivoli OMEGAMON XE for Messaging Sample applications and SupportPacs Custom applications – com.ibm.mq.headers Java package PCF accounting collection app PCF accounting collection app QueuesQueues ChannelsChannels CONNsCONNs PCF statistics collection app PCF statistics collection app Statistics Messages Accounting Messages QMGRQMGR SYSTEM.ADMIN.STATISTICS.QUEUE SYSTEM.ADMIN.ACCOUNTING.QUEUE
  • 18. @MoragHughson Taking a look with SupportPac MS0P Time period MQ Statistics at QMGR level Detailed queue statistics
  • 19. @MoragHughson Simple practical example using amqsmon Overall QMGR busyness Simple data format Multiple values are [Persistent, NonPersistent] One message every X seconds Use amqsmon directly (perl/cron) Low/high water marks for Connections Subscriptions ALTER QMGR STATMQI(ON) Wait a bit RESET QMGR TYPE(STATISTICS) amqsmon -m GATEWAY1 -t statistics -a -w 0 MonitoringType: MQIStatistics QueueManager: 'GATEWAY1' IntervalStartDate: '2014-04-09' IntervalStartTime: '00.00.35' IntervalEndDate: '2014-04-09' IntervalEndTime: '00.01.13' CommandLevel: 700 ConnCount: 35 PutCount: [271, 0] PutFailCount: 0 Put1Count: [2, 0] Put1FailCount: 0 PutBytes: [273976, 0] GetCount: [270, 0] GetBytes: [269468, 0] GetFailCount: 19 DurableSubscriptionHighWater: [0, 0, 0, 0] DurableSubscriptionLowWater: [0, 0, 0, 0] NonDurableSubscriptionHighWater: [0, 0, 0, 0] NonDurableSubscriptionLowWater: [0, 0, 0, 0] PutTopicCount: [0, 0] PutTopicFailCount: 0 Put1TopicCount: [0, 0] Put1TopicFailCount: 0 PutTopicBytes: [0, 0] PublishMsgCount: [0, 0] PublishMsgBytes: [0, 0]
  • 21. @MoragHughson ClusterCluster MQ networks can be complex At each of the dots stuck / mis-sent messages are possible MQOPENs of the wrong Queue / Queue Manager by apps Full queues Stopped channels Stopped apps Incorrectly configured QREMOTE/QALIAS routing objects Cluster membership problems The standard problem diagnosis approach Methodically checking channels/queues/DLQs at each point Is there anything to speed up this process? App1App1 App2AApp2A App2BApp2B Requests Replies Replies SDR/RCVRSDR/RCVR SDR/RCVRSDR/RCVR
  • 22. @MoragHughson ClusterCluster Trace Route MQ has the ability to inject tracer messages Hidden from applications Generate activity reports as they pass through Tools are available to trace routes using these reports dspmqrte – command line tool supplied with the product MS0P – Cat 2 SupportPac extension to MQ Explorer Report messages need to get back to your ReplyQ of your tool Lets you see the path messages could have taken Test connectivity through the MQ network Test cluster workload balancing Can quickly jump you close to the problem The point your trace message veers off in the wrong direction The point the trail goes cold dspmqrtedspmqrte SDR/RCVRSDR/RCVR SDR/RCVRSDR/RCVR dspmqrtedspmqrte dspmqrtedspmqrte
  • 23. What are the apps doing?
  • 24. @MoragHughson Application Activity Tracing Similar infrastructure to Accounting & Statistics PCF messages on SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE Configurable via ini file Can be changed without QMGR restart Configurable detail level can include partial/full message payload Frequency options for tuning Can be enabled on a per-application basis Via MQCONNX flags Via application name – MQ V7.5 JMS clients can supply an application name in the CF Enables scenarios such as Application audit trail Message duplication Resource usage Which queues or topics are actually being used Problem Determination Which Queue / Queue Manager is the application actually opening Application Coding Standards Does everyone use the MQI in the recommended way And more … Health warning: Performance impact http://ow.ly/vA8wB Health warning: Performance impact http://ow.ly/vA8wB
  • 25. @MoragHughson Extract from Report MonitoringType: MQI Activity Trace QueueManager: 'V71' Host Name: 'rockall.hursley.ibm.com' CommandLevel: 710 ApplicationName: 'WebSphere MQ Client for Java' ApplicationPid: 18612354 UserId: 'mquser' ConnName: '9.20.95.106' Channel Type: MQCHT_SVRCONN Platform: MQPL_UNIX ======================================================================= = Time Operation CompCode MQRC HObj (ObjName) 10:04:09 MQXF_INQ MQCC_OK 0000 2 10:04:09 MQXF_CLOSE MQCC_OK 0000 2 10:04:09 MQXF_OPEN MQCC_OK 0000 4 () 10:04:09 MQXF_INQ MQCC_OK 0000 4 10:04:09 MQXF_CLOSE MQCC_OK 0000 4 10:04:09 MQXF_OPEN MQCC_OK 0000 4 (SYSTEM.DEFAULT.LOCAL.QUEUE) 10:04:09 MQXF_INQ MQCC_OK 0000 4
  • 26. @MoragHughson Looking at the data with the amqsact sample ALTER QMGR ACTVTRC(ON) <- should be tuned via mqat.ini amqsact -m GATEWAY1 -v MQI Operation: 6 Operation Id: MQXF_PUT ApplicationTid: 12451 OperationDate: '2014-04-09' OperationTime: '01:39:48' High Res Time: 1397003988665548 Completion Code: MQCC_OK Reason Code: 0 Hobj: 18225032 Put Options: 139330 Msg length: 460 Recs_present: 0 Known_dest_count: 1 Unknown_dest_count: 0 Invalid_dest_count: 0 Object_type: MQOT_Q Object_name: 'SENDINGAPP.REPLY' Object_Q_mgr_name: 'GATEWAY1' Resolved_Q_Name: 'SENDINGAPP.REPLY' Resolved_Q_mgr: 'GATEWAY1' Resolved_local_Q_name: 'SENDINGAPP.REPLY' Resolved_local_Q_mgr: 'GATEWAY1' Resolved_type: MQOT_Q Report Options: 0 Msg_type: MQMT_DATAGRAM Expiry: -1 Format_name: 'MQHRF2' Priority: 4 Persistence: 0 Msg_id: 00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 ' 00000010: 0207 4453 2007 2603 '..DS .&. ' Correl_id: 00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 ' 00000010: 0207 4453 2007 2203 '..DS .". ' Reply_to_Q : ' ^D' Reply_to_Q_Mgr: ' ^C' Coded_char_set_id: 1208 Encoding: 273 Put_date: '20140409' Put_time: '00394866' Check the options used for coding standardsCheck the options used for coding standards Check queue name resolution, to find out why messages are going to the wrong place Check queue name resolution, to find out why messages are going to the wrong place Track individual messages and request/reply scenarios with Msg_id and Correl_id Track individual messages and request/reply scenarios with Msg_id and Correl_id
  • 27. @MoragHughson Taking a look with SupportPac MS0P
  • 28. How can I look back in time?
  • 29. @MoragHughson What happened to my messages at 2am this morning? Enterprise monitoring solution DLQ alerts, queue depth alerts, channel status alerts Unresolved running units of work Historical MQ monitoring, accounting and stats data App logs from the time of the problem Exceptions, MQ error codes, timeouts MQ error logs for all qmgrs that could have been involved Channel errors Authentication issues ??? – what else is there
  • 30. @MoragHughson What about the MQ Recovery Log? For persistent messages inside transactions MQ logs each operation performed Why can't we use this to Look back in time to 2am and see what happened Recovery the original payload if the app lost the message See what happened inside long-running units of work Provide a list of operations within the failed business transaction MQ documents how you can… if You use the text formatting tool provided with MQ (dmpmqlog) The logging is linear so the historical data is available in the tool You follow the right steps to extract data from running qmgrs You do the work to follow through the logs
  • 31. @MoragHughson Dmpmqlog output is readable, but analysis is tedious Wouldn't it be easier to let the computer do the tedious bit? LOG RECORD - LSN <0:0:954:44817> ********** HLG Header: lrecsize 873, version 1, rmid 0, eyecatcher HLRH LogRecdType . . : AQM Put Message (257) Eyecatcher . . : ALRH Version . . . . : 1 LogRecdLen . . : 853 LogRecdOwnr . . : 256 (AQM) XTranid . . . . : TranType: XA XID: formatID 1463898948, gtrid_length 36, bqual_length 54 gtrid [000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F2] bqual [000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F20000000100000000000 00000000000000001] QueueName . . . : Not known Qid . . . . . . : {Hash 2147211283, Counter: 5} ThisLSN . . . . : <0:0:0:0> PrevLSN . . . . : <0:0:954:43944> Version . . . . : 4 MapIndex . . . : 199 PrevLink.Locn . : 102408 PrevLink.Length : 8 PrevDataLink . : {High 0, Low 103424} Data.Locn . . . : 103424 Data.Length . . : 613 Data . . . . . : 00000: 41 51 52 48 04 00 00 00 FF FF FF FF FF FF FF FF AQRH....ÿÿÿÿÿÿÿÿ 00016: 00 00 00 00 00 00 00 00 C7 00 00 00 02 00 C0 01 ........Ç....... 00032: 00 00 00 00 04 00 01 00 A5 00 00 00 00 00 00 00 ........¥....... 00048: 63 00 00 00 41 4D 51 20 49 49 42 30 31 5F 51 4D c...AMQ IIB01_QM 00064: 20 20 20 20 D9 26 B0 52 20 08 53 F5 30 30 30 30 .&.R .Sõ0000 00080: 30 30 39 39 00 00 00 00 00 00 00 00 00 00 00 00 0099............ 00096: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00112: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ................ 00128: 00 00 00 00 FF FF FF FF 00 00 00 00 04 00 00 09 ....ÿÿÿÿ........ 00144: 00 00 00 00 E3 ED 04 80 10 FD 67 E4 FF FF FF FF ....ãí.€.ýgäÿÿÿÿ 00160: 4D 44 20 20 01 00 00 00 00 00 00 00 08 00 00 00 MD ............ 00176: 00 00 00 00 11 01 00 00 B8 04 00 00 4D 51 48 52 ........¸...MQHR 00192: 46 32 20 20 04 00 00 00 01 00 00 00 20 20 20 20 F2 ........ 00208: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00224: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00240: 20 20 20 20 20 20 20 20 20 20 20 20 49 49 42 30 IIB0 00256: 31 5F 51 4D 20 20 20 20 20 20 20 20 20 20 20 20 1_QM 00272: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00288: 20 20 20 20 20 20 20 20 20 20 20 20 4D 55 53 52 MUSR 00304: 5F 4D 51 41 44 4D 49 4E 16 01 05 15 00 00 00 64 _MQADMIN.......d 00320: 20 3E AC 57 48 B3 09 B8 71 B0 4C F2 03 00 00 00 >¬WH..¸q.Lò.... 00336: 00 00 00 00 00 00 00 0B 20 20 20 20 20 20 20 20 ........ 00352: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00368: 20 20 20 20 20 20 20 20 1C 00 00 00 57 65 62 53 ....WebS 00384: 70 68 65 72 65 20 4D 51 20 43 6C 69 65 6E 74 20 phere MQ Client 00400: 66 6F 72 20 4A 61 76 61 32 30 31 33 31 32 31 39 for Java20131219 00416: 31 34 32 32 33 32 32 32 20 20 20 20 00 00 00 00 14223222 .... 00432: A4 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ¤............... 00448: 52 46 48 20 00 00 00 02 00 00 00 A4 00 00 01 11 RFH .......¤.... 00464: 00 00 04 B8 4D 51 53 54 52 20 20 20 00 00 00 00 ...¸MQSTR .... 00480: 00 00 04 B8 00 00 00 20 3C 6D 63 64 3E 3C 4D 73 ...¸... <mcd><Ms 00496: 64 3E 6A 6D 73 5F 74 65 78 74 3C 2F 4D 73 64 3E d>jms_text</Msd> 00512: 3C 2F 6D 63 64 3E 20 20 00 00 00 58 3C 6A 6D 73 </mcd> ...X<jms 00528: 3E 3C 44 73 74 3E 71 75 65 75 65 3A 2F 2F 2F 51 ><Dst>queue:///Q 00544: 31 3C 2F 44 73 74 3E 3C 54 6D 73 3E 31 33 38 37 1</Dst><Tms>1387 00560: 34 36 32 39 35 32 32 32 36 3C 2F 54 6D 73 3E 3C 462952226</Tms>< 00576: 43 69 64 3E 30 30 30 30 30 30 39 39 3C 2F 43 69 Cid>00000099</Ci 00592: 64 3E 3C 44 6C 76 3E 32 3C 2F 44 6C 76 3E 3C 2F d><Dlv>2</Dlv></ 00608: 6A 6D 73 3E 61 jms>a Ordered unique IDs for each record (LSN)Ordered unique IDs for each record (LSN) A set of documented record typesA set of documented record types Transaction information with XIDs, or re-used MQ transaction IDs Transaction information with XIDs, or re-used MQ transaction IDs MQMD header data at discoverable offsets in the hex of a message Put MQMD header data at discoverable offsets in the hex of a message Put The message payload itselfThe message payload itself
  • 32. @MoragHughson Check out dmpmqlog scraper tool Takes the tedium out of analysing the output from dmpmqlog Created by Peter Broadhurst Download from http://www.ibm.com/support/docview.wss?uid=swg21660642 Generates file per message PUT in the supplied data. Summary file java -jar dmpmqlog.scraper-20151201.jar -b little-endian -i dmpmqlog.txt -o .
  • 33. @MoragHughson dmpmqlog scraper tool output Generates file per message PUT in the supplied data. Summary file
  • 34. @MoragHughson Summary Lots of tools in your MQ toolbox! On-line Status Commands DISPLAY CONN DISPLAY QSTATUS DISPLAY CHSTATUS Cluster monitoring – AMQSCLM Off-line Statistics and Accounting amqsmon and MS0P to view Tracking Trace-route Application Activity Trace MQ Recovery Logs dmpmqlog scraper
  • 35. @MoragHughson Notices and Disclaimers Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.
  • 36. @MoragHughson Notices and Disclaimers (con’t) Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. • IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
  • 37. Thank You Your Feedback is Important! Session AME-2289 Access the InterConnect 2015 Conference CONNECT Attendee Portal to complete your session surveys from your smartphone, laptop or conference kiosk.