NZS-4409 - Enterprise Java Monitoring on zOS Discover, Alert, Optimize As presented at IBM InterConnect 2016

1. Enterprise Java Monitoring on z/OS:
Discover, Alert, Optimize
Chris Walker – IBM

2. Please Note:
1
• IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole
discretion.
• Information regarding potential future products is intended to outline our general product direction and it should not be relied on in
making a purchasing decision.
• The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any
material, code or functionality. Information about potential future products may not be incorporated into any contract.
• The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.
• Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual
throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the
amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

3. Abstract
2
With the number of Java deployments increasing on z/OS, it is even more important today to understand
across your z environment which JVMs you currently have executing and which ones are in need of
attention, and understand whether there are any issues within the context of your z/OS sub-systems.
In this session you will learn how IBM's z/OS Monitoring solutions can give you greater insight into your
JVM deployment, and hear how customers are leveraging the latest technology to get better resilience
across their enterprises.

4. Background: Java on z/OS
3
z/OS is rapidly becoming
one of the optimal places to
run Java workloads
Virtually all major z/OS subsystems now
have some Java component
IBM Java 8 implementation working
with z13 hardware provides significant
performance throughput improvements
Vast majority of the
world’s corporate
data in systems of
record reside or
originated on
mainframe systems
Ability to optimize performance and
manage subsystem performance is key
to z/OS success…Java is part of this

5. Growth Factors
4
Skills
Application Development on z/OS is growing.
New generation of developers comfortable
with Java as primary language
Creation of new workloads (for example,
Liberty in CICS)
Integration
Integration of new Java applications with
existing applications are a cost-effective
means of modernizing.
z/OS Connect providing new access to
systems of record from mobile workloads
Performance
Access from Java applications to Systems of
Record can be optimized. Control flow can
be managed better. Reduction in network
traffic and effects.
Cost
Java workloads eligible for specialty
processor offloading. Deployment of Java
based servers on z/OS maximize this benefit.

6. Challenges to monitor Java on z/OS
5
The JVM is a black-box
“Do we know why performance is an issue?”
If there is a problem, do you just recycle and
start again?
Java is Everywhere
“Are we clear on what is running?”
Can you be certain you are making efficient
use of resources?
Complexity of Existing Solutions
Lots of point products (monitor a single JVM)
Designed for distributed platforms – not z/OS aware?
May rely on JMX and TCP/IP connections – do they scale?
Could use Byte Code Injection to track data – what is the cost?

7. Development Hills
6
“As an SME responsible for the performance and availability of JVMs on z/OS I expect to be able to view all
active JVMs within an LPAR, whether instrumented or not, such that I am aware of any active JVM within
1 minute of it being started so that further investigation can take place.”
“As an SME responsible for the performance and availability of JVMs on z/OS I need to be able to see
garbage collection related statistics for any active JVM in my system which will allow me to identify issues
where garbage collection is performing sub-optimally in less than 5 seconds.”
“As an SME responsible for the performance and availability of JVMs on z/OS I need to be able to see critical
statistics related to thread locks for any JVM in my system which will allow me to detect issues that are
related to thread locks in less than 5 seconds, leading to faster problem resolution of thread lock related
issues.”

8. Introducing OMEGAMON Monitoring for JVM
7
• Brand new OMEGAMON monitoring agent focused on assisting z/OS system administrators,
operators and SMEs identify problems, resolve quicker and optimize performance
• Lightweight overhead compared to other offerings.
– 90% of data collected is through Health Center API
• Ability to view all JVMs side-by-side. No disconnect when switching between JVMs
• Collects data on any online JVM on z/OS
– Subsystems: CICS, IMS DB2, WAS, z/OS Connect, ODM
– Standalone Batch USS Java applications
– Can identify and distinguish Liberty JVM servers
• Data presented on both OMEGAMON enhanced 3270 UI and Tivoli Enterprise Portal
• Reports on Garbage Collection, Active Threads, Lock Utilization, JVM Environment, CPU Utilization
• Provides the standard OMEGAMON features:
– Look back in time with historical data collection
– Be alerted to abnormal conditions through defined event generation (Situations)
– Easy to configure and deploy using PARMGEN

9. Clear Summary of JVM Data – e3270UI
8
Each column represents the highest/worst current status for a given attribute (GC
Rate, Thread Blocked etc.). Each value is ‘zoomable’, that is, click or hit enter on
the cell value that interests or concerns you and you will be automatically
navigated to detailed information for the JVM that produced that value.
This is the initial workspace for JVM monitoring. Each row represents data collected
by a JVM monitoring agent. Typically, we would expect a user to deploy one agent
per LPAR and instrument all JVMs to report data back to that agent.

10. Clear Summary of JVM Data – TEP
9
Equivalent workspaces are developed for the TEP
interface. The JVM Monitor appears in the physical
navigator tree with an entry for each agent.
Multiple JVMs can send data to each agent.

11. Drilling down to important information
10
Highest JVM Statistics
JVM Health Summary CPU Statistics
Garbage Collection
Statistics
Thread DetailsLock Details
JVM Environment
JVM Command Line
System Variables
Env Variables
JVM Parameters
Classpath
Boot Classpath
GET Count
Average Hold Time
Slow Gets
Recursive Acquires
Lock Utilization %
Thread State
Contending Object
Stack Trace
Nursery GC Details
Global GC Details
% Time Paused
Heap Allocation
General CPU
IFA CPU
IFA Work on
General CPU

12. Scenario: Visibility of all JVMs
11
• “We need to be able to see all JVMs that are currently online”
• As noted, JVMs can be found all over the environment. Can you be clear what is online, are there
JVMs online that are unplanned?
• Starting the JVM Monitor will seek out and find all JVMs on an LPAR regardless of subsystem type
whether they have been configured for full monitoring or not.
• The agent will capture the jobname, ASID, subsystem type and basic details of the JVM.
CICS TS
JVM
WAS
JVM
DB2
JVM
IMS
JVM
OMEGAMON
JVM
Agent

13. Scenario: Visibility of all JVMs
12
For a JVM to be fully monitored, it must be
instrumented to allow OMEGAMON to collect
data. If not, we can still determine online JVMs
and their subsystem type. These are reported
on the second subpanel here. A user can then
determine if they want to instrument that JVM
for full monitoring.

14. Scenario: Visibility of all JVMs
13
Equivalent Tivoli Enterprise Portal screen
showing JVMs currently being fully
monitored and those detected as being
online but not monitored by JVM agent

15. Scenario: Visibility of all JVMs
14
• To enable full monitoring of a JVM it must be instrumented to allow the OMEGAMON agent to interact
with the JVM and issue requests via the Health Center API.
• Typical configuration is a minor change to the JVM startup parameters:
-Xhealthcenter:level=inprocess
-javaagent:/omegamon/uss/install/dir/kan/bin/IBM/kjj.jar
• OMEGAMON code will collect JVM environment information, capture JVM events (for example GCs)
and push the details to the OMEGAMON JVM agent.
CICS TS
JVM
WAS
JVM
DB2
JVM
IMS
JVM
OMEGAMON
JVM
Agent

16. Scenario: Optimizing Garbage Collection
15
• “Performance of JVM is poor. Could Garbage Collection be a cause?”
• Performance of the Garbage Collector has improved significantly in recent releases of Java however
poor performance can still occur due to:
• Insufficient heap allocation
• Poorly written applications
• The symptoms of such problems might be:
• Excessive number GC events occurring within a given period of time
• High heap occupancy even after a GC
• Long pause times when GC event is occurring
• System GC events occurring
• The Garbage Collection Details workspaces provide insight into the performance of the JVM GC
allowing the operator to confirm (or dismiss) the JVM as a bottleneck in the performance throughput.

17. Scenario: Optimizing Garbage Collection
16
The Highest JVM Statistics subpanel shows the
poorest performing statistics in key GC metrics
If a threshold is exceeded (example GC Rate per
Minute), zoom into the JVM that potentially has
an issue.

18. Scenario: Optimizing Garbage Collection
17
GC Details can point out key values that may
indicate a problem. A rolling 5 minute interval is
used to scale values.
Does the Occupancy look OK? Average Heap
size fine?

19. Historical Data Collection
18
A snapshot of data taken a regular intervals
to allow viewing of system status a specified
point in the past

20. Scenario: Identifying Locks and Thread Blocks
19
• “Our applications are performing poorly. Can we see what might be causing this?”
• If not GC issues, perhaps threads are being blocked for an excessive period of time or locks within the
JVM are being held for long periods causing application to wait for the monitor to yield.
• If high values found here, the application owner (if applicable) can be alerted or adjustments to the
JVM environment could be made.

21. Scenario: Identifying Locks and Thread Blocks
20
Thread Statistics drills-down to all active threads
making BLOCKED threads easy to spot

22. Scenario: Identifying Locks and Thread Blocks
21
The Lock Statistics shows which monitor objects
were used as lock most often an how long they
were held for.

23. Scenario: Identify Environment Issues
22
• “We need to ensure that the Java levels being used are up to date”
• We are able to deep-dive into JVM environment details to view information like the classpath, system
properties and the version of Java being used.
• We can also define a situation to check setting and alert us to a problem. In this case, if a ‘bad’ Java
version is being used

24. Scenario: Identify Environment Issues
23
In the TEP Situation Editor we create a new
Situation to check against the JVMs
Version attribute. If this condition is ever
met, a Warning alert will be raised.

25. Scenario: Identify Environment Issues
24
Once the situation is tripped, you can
analyze the current conditions, identify
the offending job and take appropriate
action

26. Scenario: Identify Environment Issues
25
The Situation Status Tree in enhanced
3270 UI will if there is a JVM online with the
offending Java level. A user could then take
appropriate action

27. Product Packaging
26
 OMEGAMON Monitoring Feature for JVM is provided as a priced feature of OMEGAMON XE
on z/OS
 Willbe available also as part of all suite offerings:
 Service Management Suite
 OMEGAMON Performance Management Suite
 OMEGAMON z/OS Management Suite
 There is no technical dependency between OM on z/OS and OMEGAMON Monitoring Feature
for JVM
 Can deploy JVM agent without deployment of OM for z/OS agent, remote TEMS etc
 Agent cuts across all subsystems and applicable to users of Java regardless of whether they have the subsystem

28. Suite Offerings
27
Service Management Suite on z/OS
OMEGAMON Performance Management Suite on z/OS
OMEGAMON z/OS Management Suite
OMEGAMON on
z/OS
Monitoring
Feature for JVM
OMEGAMON
Mainframe Networks
OMEGAMON for
Storage
OMEGAMON Dashboard
Edition
OMEGAMON for
CICS
OMEGAMON for
DB2 PE
OMEGAMON for
IMS
OMEGAMON for
Messaging
ITCAM for Web
Resources
Service Management
Unite
NetView for z/OS
System Automation for
z/OS
Tivoli Asset
Discovery

29. Suite Offerings
28
Enable resource
monitoring of JVMs on
z/OS with the new
Tivoli OMEGAMON XE
on z/OS Monitoring
Feature for JVM.
It‘s a new priced
feature.
The new release
provides new function
to enable resource
monitoring of JVMs on
z/OS with the new
Monitoring Feature for
JVM as an integral
part of the suite.
The new release
provides new function
to enable resource
monitoring of JVMs on
z/OS with the new
Monitoring Feature for
JVM as an integral
part of the suite.
The new release
delivers valuable
Service Management
Unite dashboard
enhancements and
JVM Monitoring
Tivoli OMEGAMON XE on
z/OS, V5.3.0 (Monitoring
Feature for JVM)
Tivoli OMEGAMON for
z/OS Management Suite
V5.4.0
Tivoli OMEGAMON
Performance Management
Suite for z/OS V5.4.0
Service Management Suite
for z/OS V1.4.0
Helps manage
performance and
availability of IBM z/OS
operating system,
middleware, mainframe
networks, and z/OS
storage.
Provides detailed
monitoring and problem
management for IBM z
Systems.
Provides real-time and
historical performance,
and availability
management capabilities
for z/OS operating
systems, networks, and
storage.
Automation, network
mgmt., high availability,
asset discovery for critical
z business workloads.
Integration of systems
management and
analytics.

30. More Information/References
29
 OMEGAMON Product Home
 Overview and product information for all OMEGAMON products
 http://www.ibm.com/OMEGAMON
 Service Management Connect
 Blogs, forums, articles, best practices videos for IBM z Systems monitoring
 http://www.ibm.com/developerworks/servicemanagement/z
 IBM Servers and Systems Software RFE Community
 Process to raise requests for enhancements to OMEGAMON products
 https://www.ibm.com/developerworks/rfe/?BRAND_ID=352

31. OMEGAMON at InterConnect 2016
30
 NZS-3272 : OMEGAMON Family Update: See What’s New in Monitoring
 NZS-3645 : Increase Your Business Agility with the Latest Release of OMEGAMON
 NZS-4409 : Enterprise Java Monitoring on z/OS: Discover, Alert and Optimize
 NZS-1531 : Panel Discussion in IBM Service Management on z Systems
 NZS-6326 : z Software Monitoring/Automation Futures
 NZS-2879 : z/OS Monitoring and Automation to Enable Predictive Analysis for Critical
Applications at BPM
 NZS-6763 : The Orange Code of OMEGAMON at ING
 NZS-5044 : OMEGAMON on z/VM and Linux at Bank of Brazil

32. Notices and Disclaimers
31
Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission
from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of
initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS
DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE
USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY.
IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers
have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in
which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials
and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or
their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and
interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such
laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law

33. Notices and Disclaimers Con’t.
32
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not
tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the
ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,
FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,
Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,
PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,
StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business
Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

34. Thank You
Your Feedback is Important!
Access the InterConnect 2016 Conference Attendee
Portal to complete your session surveys from your
smartphone,
laptop or conference kiosk.