Best Practices for Troubleshooting Four Real-world Java Performance Issues

© eG Innovations, Inc. | www.eginnovations.com | Confidential @perfclarity
Agenda
Introduction
Digital Experience - Why?
3rd party vendor problems
Code-level infinite loop
Hands-on Tech Demo Scenarios
Table-level locks
Browser slowness
eG Full-Stack APM Features
Q&A
Description of the real-world microservices application

Arun Aravamudhan
Product Head - APM & Web products
 Lead APM product development @eGInnovations
 6+ years in eG. Areas of focus: Java, Cloud, Containers.
 20+ years experience in distributed systems (e-commerce,
banking, telco) across Dev, Architecture and Ops.
Live in Chennai, India.
@perfclarity
Introduction

Agenda
Introduction
Digital Experience - Quick introduction
3rd party vendor problems
Code-level infinite loop
Hands-on Tech Demo Scenarios
Table-level locks
Browser slowness
eG Full-Stack APM Features
Q&A
Description of the real-world microservices application

What?
Why?
How?
Who?
Observes performance from the point of view of the end user
Top-down and outside-in approach
Measure business impact of performance
Web apps are important. Customer experience (CX) is a key differentiator
Revenue loss Reputation damage
• Synthetic transaction monitoring (Proactive scripted jobs)
• Network path monitoring
• Install endpoint agents on client machines
• Real user monitoring (APM centric, easy to inject, wide applicability,
reactive)
5 personas in particular: App Owner / LOB, IT Ops, AppDev, Helpdesk/ Support, DevOps
Digital Experience

Application
Owner or LOB
Personas – Digital User Journey
What’s the End User Experience?
I need to know if performance is
impacting business.
IT Operations
Even as a generalist, I need early-
warning notification of problems
quickly before customers are
impacted. I need to prioritize high
user impact incidents first.
Developers
I need method-level code visibility and
specific steps in the user journey that
experience higher number of errors
Support &
Helpdesk
I need to triage this
ticket quickly. Need
cross-application tracing to
hand it off it to the right
team.
DevOps
How can I deploy
faster? Are code /
config changes
affecting real users?
References: Various articles from Gartner, Forrester, EMA, 451
research

IT Operations leaders should:
Improve their ability to show the business impact of APM by
ensuring that the APM suite integrates performance statistics
with their impact on business metrics and processes, such as the
order process.
Critical Capabilities for Application Performance
Monitoring
Why is the User Journey capability important?

http://bit.ly/egdemo
“Webstore” (props for
creativity) is your
company’s E-Commerce
application that sells books
and mobile phones.
Customers are complaining
that they are not able to
place orders.
Hands-on Demo Application
Please access the above link on your mobile phone or
desktop

User Browser
AWS Elastic Kubernetes Service (EKS)
Docker
AWS
Technology Stack
Webstore PODs
Payments PODs
Order PODs
Customer PODs
Product PODs
Quote PODs
3rd party
payment
gateway
Docker
Containers
Containers
Containers
Containers
Containers
Containers
MySQL
(also running
on Docker)
The Microservices application - Architecture
Tomcat

The microservices app is running on AWS EKS – Elastic Kubernetes Service (EU West region)
The microservices app has 8 PODs (including the database service)
The Microservices application on Kubernetes (AWS)

Live Demo - http://bit.ly/egdemo

USER INTERFACE
Network looks good
JVM CPU looks good
No DB locks
OS is looking ok.
APPLICATION
Angular JS code is fine. I just tested on my
laptop and the UI loads fine. Smooth as silk!
Our tools are showing green!
INFRA
• I entered my credit card 3 times and it keeps
erroring out
• The page keeps spinning!
My application logs look good. I tested
everything on staging before checking in.
Customers are happy, right?

Click #1
Problem: Payment submission is very slow or generating errors
Scenario 1: Payment Gateway – 3rd party dependency
RUM AppFlow
Topology
(Micro) Service
Topology

(Micro) Service
Topology
Business Transaction Monitoring (BTM) - Detailed Diagnosis
Click #2

Detailed Diagnosis
Business Transaction Monitoring (BTM) Snapshot
Root-cause:
Mastercard
payment
gateway
malfunction
Click #3
From Problem to
Root-cause
in 3 Clicks
 Affected user’s name:
Peter Payments.

You run a tight ship in your data center. But your 3rd
party vendor’s perf problems can cause revenue loss

Scenario 2: Infinite loop in code causing High CPU
Problem: A code-level bug is causing high CPU and thereby causing web pages to hang.
boolean keepLooping = true;
long cpuDuration = Long.parseLong(request.getParameter("duration"));
long timeStart = System.currentTimeMillis();
while(keepLooping){
long timeNow = System.currentTimeMillis();
if((timeNow - timeStart) >= (cpuDuration * 1000) ||
(timeNow - timeStart) >= (300 * 1000)){
keepLooping = false;
}
}
Actual code causing CPU
CustomActionController.java

Click #1
(Micro) Service
Topology
Thread Diagnosis
From Problem to Root-cause
in 1 Click
Root-cause:
Code-level bug

Click #1
See the affected user’s name
Root-cause:
Code-level bug
 Affected user’s name:
Chris CPU.

Table Level Locks
Online Transactions
Batch Job (that
updates product and
inventory
information)
Time
Start Trans
Start Trans
Add items
to cart
Update price
information

Click #1
Table lock is clearly shown
Affected SQL (update cart)
Problem: Another process that creates a table lock prevents items from being added or removed from the
Shopping Cart
Root-cause:
Table lock
Scenario 3: Table level Locks impact AddToCart

Limitation of APM tracing tools (BTM is blind to DB infra
issues such as table locks)
 Affected
user’s name:
Larry Locker. 49 seconds!
Standalone APM can only show
you that the SQL was slow.
But why slow? Due to table level
locks –
that’s Converged APM =
RUM + APM + Infra.

These images (or any static
resources) are blocking/
waiting to get fetched since
they are all served by the
same domain
Best practice : Shard the domain from which
static resources are served.
Content1.webstore.com
Content2.webstore.com etc
Scenario 4: Frontend Browser Slowness

# Scenario Impacted User
(example)
Remediation
1 3rd party payment
gateway malfunction
Peter Payments • Disable Mastercard button
• Notify vendor
2 Code-level infinite loop Chris CPU • Notify dev team to fix bug
3 Table-level locks Larry Locker • Notify DBA to kill external
process / batch job and
reschedule to nightly
4 Browser slowness Bob Browser • Notify Web Admin to
create sharded content
domains for static assets
Summary of scenarios and impacted users

• When the end user requests a page from your browser application, the eG BTM
agent sends additional HTTP headers identifying the business transaction as well as
a correlation GUID (Globally unique identifier).
• The eG RUM JavaScript agent sends the correlation GUID to the eG Manager
• The eG agent aggregates backend metrics and sends the metrics and business
transaction identifiers to the eG Manager with the unique BTM GUID .
• The eG RUM JavaScript Agent sends the browser metrics and business transaction
identifiers (from the HTTP header) to the RUM Collector. This is the browser detailed
diagnosis snapshot.
• The dashboards can join the same RUM and BTM snapshot using the common GUID.
How does RUM <-> BTM correlation work?

This is a common GUID for both RUM
and BTM and thus be joined as a single
record.

How are users identified in eG Enterprise?
eG Enterprise v7 can uniquely identify individual users across different
browsers, devices, and user sessions.
There are 3 options available to tag Users:
Based on page
metadata such as a
DOM element.
Use CSS Selector,
XPath or DOM
identifier
User tagging based on
request attributes.
Based on a data
collector
Method invocation
data collectors capture
code data such as
method arguments,
variables, and return
values
1 2 3
/AddToCart?userid=123 Public String getUser()

INFRA
APPLICATION
USER INTERFACE
What should we focus on?
Browser? Server? Network?
If it is the server, which tier is the
root cause ?
Code? SQL/ DB? 3rd Party?
Is the JVM and App Server optimal?
Wider Infra issue that affects ALL
transactions?
Docker? Kubernetes? Operating system?
Key Checklist Questions

eG Enterprise – the APM journey
Rich heritage and depth in
Infrastructure and Unified
Monitoring
Prior
Years 2016 2018 2020
Introduced Real User
Monitoring (RUM) and
Business Transaction
Monitoring (BTM) with v6
Added more depth in BTM
with new pointcuts,
frameworks with v6.3.
Converged APM.
New User Journey visualization
tools. RUM-BTM correlation,
Kubernetes, Docker, Microservices
and much more (scalability, point
cuts, optimizations, more
frameworks) with v7
The product is “well-baked” in high scale transactional environments such as Core Banking
(case studies available)

• eG Enterprise is licensed by server
• There can be Multiple JVMs installed on a server of any Host RAM
• Other APMs license by every JVM monitored or by host units
• Spend your valuable time solving business & IT problems, not worrying
about host unit overages or quotas
JVM 1
JVM 2
JVM 3
JVM 1
JVM 2
JVM 3
Server Server
5 Licenses Needed per server in Other APMs Only 1 License Needed per Server in eG Enterprise
eG Enterprise: Cost-efficient Licensing
JVM 4
JVM 5
JVM 4
JVM 5

Key Takeaways
Challenge Opportunity What IT Ops Needs
Full stack APMeG Enterprise v7
Expectation on IT Ops is high to
ensure top UX
IT Ops leaders can impact
business
Digital service center - UX
Use reactive war rooms less
often
• Better prioritize
performance incidents
based on users impacted.
• Triage and diagnose in
the context of customer
experience.
• Remediate and validate
bottlenecks in the
customer experience.
Web apps and UX is important
Single converged solution (not
siloed tools)
Eliminate finger pointing
Connect all dependencies
with UX context
Single-pane-of-glass visibility
Automatic correlation &
diagnosis
Actionable insights
KPIs, analytics, reports and
machine learning

Why eG Enterprise 7.1 for Full-Stack APM
• Extremely cost-efficient: License by OS,
not JVMs/CLRs
• The only truly converged application and
IT infrastructure monitoring solution
• Synthetic and real user experience
monitoring for Java, .NET, PHP, Node.js,
and COTS applications from one console

Q&A

We plan to conduct more Hands-On Tech Training with deep
dives into:
• Single Page Applications (such as Angular)
• Microservices, Kubernetes & Docker, AWS
• Multi-threading performance problems
• Memory performance problems
Upcoming Webinars in APM
Please let us know your feedback / comments and requests for any other hot performance topics!
Email: jecilia.chong@eginnovations.com

THANK YOU

Best Practices for Troubleshooting Four Real-world Java Performance Issues

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Best Practices for Troubleshooting Four Real-world Java Performance Issues

Similar to Best Practices for Troubleshooting Four Real-world Java Performance Issues (20)

More from eG Innovations

More from eG Innovations (9)

Recently uploaded

Recently uploaded (20)

Best Practices for Troubleshooting Four Real-world Java Performance Issues