Alfresco DevCon 2019 Performance Tools of the Trade

Performance Tools of the Trade
Luis Colorado, Zia Consulting

Learn. Connect. Collaborate.
Happy New
Alfresco User

But time
passes…
• Slow searches
• Slow pages
• Slow everything!
• Crashes!
What
to do?

Caveat
Emptor
• Not covering:
– Performance techniques
– Architecture
– Development techniques
– (Okay, okay… we will cover a bit of it)
• Focus on Alfresco on-premise
– But it also applies Cloud deployments

Slowness:
level 1
• Things are slower than demo
• We have only loaded like a million documents!
• Users are complaining that Alfresco is very
slow

Slowness:
level 2
• Sometimes pages or searches take more than
30 seconds to load
• Searches are terrible slow
• Users start to click two or three times on links
• Some pages will just not load

Slowness:
level 3
• Alfresco Servers starts to crash once on a
while
• But they start crashing more and more often

Finding the
Bottleneck
• But it feels like finding a needle in a haystack
or ….

Step Zero
Follow or Set …
… before Troubleshooting Performance.

Step 0.1

Step 0.2
In Virtual Environments (e.g., VMWare)
Make sure that
Memory and
CPU are
Pinned/Allocated to Alfresco

Possible
Performance
Bottlenecks
Alfresco
Memory
Database
Repository
CPU
Virtualization
Storage

Cutting the
Performance
Gordian Knot

Slicing the
Performance
Problem
Low
CPU
CPU
Usage
High
CPU
I/O,
Database,
Repo,
Other Service
Alfresco
Other
Process

First Half of the Gordian Knot:
What is Killing the CPU?

Who is killing
the CPU?
• Java (Alfresco)?
• LibreOffice?
• Other Process?

CPU
Utilization
Tools
(System)
• Linux
– htop
– top
– sar (part of sysstat)
• Task Manager (Windows)

High Java
CPU usage
Find that hot thread!
• Support Tools
• Hot Threads
• jVisualvm or jConsole
• Application Performance Monitor (APM)
Gotcha Note!
– Docker/Kubernetes may not report CPU usage
correctly

High Java
CPU usage
• Support Tools for the Admin Console
– Enterprise:
• 4.2 – 5.1: Support Tools AMP
(https://github.com/Alfresco/alfresco-
support-tools)
• 5.2+: Integrated
– Community:
• Order of the Bee
(https://community.alfresco.com/docs/DOC-
7941-order-of-the-bee-support-tools)
• Pros: integrated, easy to use, non-intrusive
• Cons: may require 5.2+ or AMP installation,
not persistent

High Java
CPU usage
• Hot Threads
– Pros: some tools provide it, very light,
straightforward analysis, free, command line
(https://web.archive.org/web/20111225025057/h
ttp://weblogs.java.net/blog/brucechapman/archiv
e/2008/03/hot_threads.html), minimal
installation
– Cons: no UI, only instant snapshots

High Java
CPU usage
• jVisualVM, included with JDK
– Pros: low/medium overhead, persistency for
hours or days, thread analysis, free
– Cons: connecting not always straightforward or
even possible, needs to set java options in
Alfresco start script

High Java
CPU usage
• YourKit, jvisualvm on steroids, “Back to the
Future”
– Pros: database analysis, monitoring and
profiling, reviewing previous, best for random
crashes
– Cons: not free, requires installing agent, may be
heavy on CPU, complex configuration

High Java
CPU usage
• Application Performance Monitor (APM)
– Pros: some Open Source, constant connection,
ideal for constant monitoring
– Cons: some commercial (not free), shop
carefully (capabilities vary widely)

“The Gordian Knot: Part II”
Low CPU Usage

Low CPU
usage, now
what?
Alfresco
Memory
Database
Repository
CPU
Virtualization
Storage

Low Java
CPU usage
• Low CPU usage (but Alfresco is slow!)
• The problem is related to I/O issues or other
systems:
– Database issues
– Storage issues (too slow, too full)
– Problematic repository (too many children in a
folder, lots of sites, lots of users or permissions)
– Remove services (e.g., Solr) are slow

Database
(latency,
maintenance)
• ping is your friend (but not always)
• If your DBA is not your friend, it should be!
• Long-running queries
• Refreshing statistics

Memory
Problems
• Java
– Small heap size
– Garbage Collection (too much time, improper
algorithm)
– Small New/Eden space
• Operating System
– 35% to 25% should available

Tools to
Diagnose
Memory
Problems
(Java)
• jstat – JDK utility
• Use -verbose:gc command line option
– Requires visualgc
• jconsole
• jvisualvm
• Support Tools

Memory
Problems:
Pro Tips
• Java Heap Size
– < 75% OS Memory (Linux)
– < 60% Os Memory (Windows)
• Understand “free” OS memory in Linux
• Java:
– (Actually) Used <= Committed (allocated)
• Watch the Java Large Garbage Collections
• Try to use G1C for Java:
-XX:+UseG1GC
• G1C uses more Memory

Low Java
CPU usage:
Diagnostic
Tools
• If DB, storage and repo are okay, let’s look for
stuck/slow threads
– Obtaining Thread dumps:
• jstack
• jconsole
• Support Tools
– Spotify (https://spotify.github.io/threaddump-
analyzer/) thread analyzer: most useful
• Pros: needs only one thread dump
• Cons: will not find stuck
– Thread Dump Analyzers: no silver bullet
• Multiple thread dumps needed at even
intervals

Storage
• Content Store doesn’t have to be super fast
• Solr and tomcat/temp should use the fastest
storage available

Monitoring is
critical (if your
environment is
critical)

Application
Performance
Management
(APM)
• Commercial or Open Source?
• Commercial
– AppDynamics
– Wily
– Dynatrace
– New Relic
– More…
• Open Source
– Pinpoint
– MoSKito
– Glowroot

Conclusions
• Most of the times the tools to troubleshoot
performance problems are free, but
commercial tools may be necessary.
• Be ready to collect information in a crisis.
• It’s a good idea to use APM tools to monitor
24x7 to prevent crashes and performance
degradation.
• Keep your mind open.

Alfresco DevCon 2019 Performance Tools of the Trade

More Related Content

What's hot

Similar to Alfresco DevCon 2019 Performance Tools of the Trade

Recently uploaded

Alfresco DevCon 2019 Performance Tools of the Trade

Editor's Notes