Zing Vision
Answering your toughest production
Java performance questions
Outline
• What is Zing Vision?
• Where does Zing Vision fit in your Java environment?
• Key features

• How it works
• Using ZVRobot
• Q&A

©2014 Azul Systems Inc.

2
What is Zing Vision?
• Zing Vision is a browser-based, visual window into the Zing VM
– Hyperlinked display provides drill-down to root cause
– Both JVM internal and Java program information
– No additional performance overhead
– Nothing special to install or configure
– Easy to get started as a user

– ZVRobot collects and stores monitoring data
• Collected data is same as ZVision!

©2014 Azul Systems Inc.

3
Example Zing Vision deployment

HTTP

Linux host
Zing Vision
(Web browser)

zvision server
HTTP

Zing VM

©2014 Azul Systems Inc.

4
Fitting in: Production focus
• Most tools suitable for developers, but too ‘heavy’ for production
•
•
•
•

Use JVMTI (JVM Tools Interface) and BCI (Byte Code Instrumentation)
High overhead, so must be used carefully in production
May require configuration to lower the overhead
Not practical for Operations teams

• Zing Vision is designed for safe use on production systems
• Ideal diagnostic tool to use when you know you have a problem in
production (“a flashlight in a dark place”)
• Integrated with the JVM, so it won’t add overhead or perturb the running
Java program
• It’s meant to be used - click on items in the web UI to explore!

©2014 Azul Systems Inc.

5
Key features
• Tick profiler
• Thread-level views
• Lock contention view

• Garbage collection views
• Java heap object views
– Live objects
– Object types that increase in number as application runs

©2014 Azul Systems Inc.

6
Tick Profiler: Find “hot” code
• Goal: Determine what code is hot (consuming most CPU time)
• Approach used by other profiling tools:
– Method tracing
– Byte code instrumentation – heavy weight
– Solution: Profile only a user-selected portion of the application
– Drawback: You need to know the location in the code where the
performance bottleneck occurs to select the area of the code to profile!

©2014 Azul Systems Inc.

7
Tick Profiler: Find “hot” code
Zing Vision uses runtime thread sampling:
• Fast snapshots of the code running in executing Java threads
• Identifies where the work is done in the application
• Lightweight, low overhead
– Always on
– Even when you’re not looking at the generated metrics

• Instruction-level granularity
– JVM internal threads, too!
• Detailed information about the entire process
– Interpretation sometimes requires understanding of the JVM runtime

©2014 Azul Systems Inc.

8
Tick Profiler: How it works
1. Thread registers SIG61 signal handler with kernel at
intervals 1 ms (1000 times per second, configurable)
2. Thread runs
3. Kernel delivers SIG61
4. Thread is interrupted on its normal stack
5. Signal handler runs, creates a new (youngest) profile

Youngest profile added

6. Zing Vision aggregates all of
the profiles on the belt in the
Timer Tick Profile view

Oldest profile drops off… (into the trash)

Tick profile conveyor belt has a fixed number of positions for profiles.
Note that the conveyor belt only moves when a new profile is added!
This means Zing Vision always shows the most recent data.
©2014 Azul Systems Inc.

9
Tick Profiler: Where is work done?

©2014 Azul Systems Inc.

10
Tick Profiler: Where is work done?

Collection controllers
Filters

Select link to see details

©2014 Azul Systems Inc.

11
Tick Profiler: Work at instruction level

Callee | Caller
Ticks for each instruction

©2014 Azul Systems Inc.

C2 compiled JDK and
application code

12
Tick Profiler: How did I get here?

©2014 Azul Systems Inc.

13
Tick Profiler: How did I get here?
Sorted based on highest CPU consumers
0 com.sun.tools.javac.util.Name.fromUtf

Top of stack

Top of stack

©2014 Azul Systems Inc.

14
Thread-level questions
• What threads are executing my application?
• What are the housekeeping threads for the JVM?
• Can I examine a thread object?

• What are the threads doing?
– Profiling information
– Stack traces

• Where is my application stalled?

©2014 Azul Systems Inc.

15
Thread-level views: Standard method
Thread number: 1 2 3

4 5

Request thread stacks (kill -3)
Some threads blocked – NO work done
Thread 3 continues to run!

Time

Last thread stopped
O(numThreads) delay  application visible work stoppage
Resume threads

©2014 Azul Systems Inc.

16
Thread-level views: Zing Vision method
Thread number: 1 2 3

4 5

Request thread stacks (ZVision)
Each thread:
1. Stopped individually
2. Delay order is O(1)

Time

No thread waits for any other thread to stop!
The application never perceives a work
stoppage because N-1 threads continue to
run at all times
Resume each thread seamlessly

Pro: Safe to use in production (guilt-free clicking)
Con: Not 100% consistent (especially for lock data)
©2014 Azul Systems Inc.

17
Thread-level view: Application threads

Filters and control
Select link to
see stack
Individual
thread stack
trace and
tick profile
access
©2014 Azul Systems Inc.

18
Thread-level view: Thread stack trace

Thread CPU time consumed
Stack frames
Select link to see details

©2014 Azul Systems Inc.

19
Thread-level view: Thread object details

Values of the object’s fields

©2014 Azul Systems Inc.

20
Thread-level view: Application threads

Filters and control
Select link to
see profile
Individual
thread stack
trace and
tick profile
access
©2014 Azul Systems Inc.

21
Thread-level: Where is the work done?

Thread 8120

©2014 Azul Systems Inc.

22
Lock contention: Where is my app stalled?

Lock acquisition
Total and Max times Blocking count
Select link to
see details

©2014 Azul Systems Inc.

23
Lock contention: Where is my app stalled?

How did I get to the contended lock in my code?

Summary metrics

©2014 Azul Systems Inc.

24
Memory: Answering your questions
• What’s the collector doing?
• How much memory is the application using?
• What type of objects are in the heap?
– What’s keeping those objects live?

• What object types are increasing in number?

©2014 Azul Systems Inc.

25
Memory: Resource use summary

Java heap overview

Process Zing memory use

Linux memory
©2014 Azul Systems Inc.

26
Memory: Collection details

Collection details
©2014 Azul Systems Inc.

27
Memory: Collection details summary

Detailed
summary
calculating
metric
averages

©2014 Azul Systems Inc.

28
Memory: What objects are in the heap?

Objects in Old Generation

Default sorting:
Sum of size of each
object of that type

Expand to see
types with
references to
objects of that type
©2014 Azul Systems Inc.

29
Memory: Which object type is growing?

Object types with increasing memory
consumption in the Old Generation
Selectable time interval

Growth
rate

©2014 Azul Systems Inc.

30
Example ZVRobot deployment

Linux host

Linux host
Write a file for each Zing Vision web page at
each sample period (for example, every 1
minute) into a directory

ZVRobot

HTTP

Zing VM

Use a web browser to look at the saved snapshot files anytime after
program has run or even during collection
©2014 Azul Systems Inc.

31
How to Try Zing Vision - Free
• Request a trial copy of Zing http://www.azulsystems.com/trial
• Download and run Azul Inspector to check your system
• Download and install Zing
– Zing Vision and ZVRobot are included

• Run your application
• Observe your application and the JVM’s activity using Zing Vision

©2014 Azul Systems Inc.

32
Questions?
Zing Vision – providing answers to your Java performance questions
• Tick profiler

• Thread-level views
• Lock contention view
• Garbage collection views
• Java heap object views
• Live objects
• Object types that increase in number as application runs

©2014 Azul Systems Inc.

33

Webinar: Zing Vision: Answering your toughest production Java performance questions

  • 1.
    Zing Vision Answering yourtoughest production Java performance questions
  • 2.
    Outline • What isZing Vision? • Where does Zing Vision fit in your Java environment? • Key features • How it works • Using ZVRobot • Q&A ©2014 Azul Systems Inc. 2
  • 3.
    What is ZingVision? • Zing Vision is a browser-based, visual window into the Zing VM – Hyperlinked display provides drill-down to root cause – Both JVM internal and Java program information – No additional performance overhead – Nothing special to install or configure – Easy to get started as a user – ZVRobot collects and stores monitoring data • Collected data is same as ZVision! ©2014 Azul Systems Inc. 3
  • 4.
    Example Zing Visiondeployment HTTP Linux host Zing Vision (Web browser) zvision server HTTP Zing VM ©2014 Azul Systems Inc. 4
  • 5.
    Fitting in: Productionfocus • Most tools suitable for developers, but too ‘heavy’ for production • • • • Use JVMTI (JVM Tools Interface) and BCI (Byte Code Instrumentation) High overhead, so must be used carefully in production May require configuration to lower the overhead Not practical for Operations teams • Zing Vision is designed for safe use on production systems • Ideal diagnostic tool to use when you know you have a problem in production (“a flashlight in a dark place”) • Integrated with the JVM, so it won’t add overhead or perturb the running Java program • It’s meant to be used - click on items in the web UI to explore! ©2014 Azul Systems Inc. 5
  • 6.
    Key features • Tickprofiler • Thread-level views • Lock contention view • Garbage collection views • Java heap object views – Live objects – Object types that increase in number as application runs ©2014 Azul Systems Inc. 6
  • 7.
    Tick Profiler: Find“hot” code • Goal: Determine what code is hot (consuming most CPU time) • Approach used by other profiling tools: – Method tracing – Byte code instrumentation – heavy weight – Solution: Profile only a user-selected portion of the application – Drawback: You need to know the location in the code where the performance bottleneck occurs to select the area of the code to profile! ©2014 Azul Systems Inc. 7
  • 8.
    Tick Profiler: Find“hot” code Zing Vision uses runtime thread sampling: • Fast snapshots of the code running in executing Java threads • Identifies where the work is done in the application • Lightweight, low overhead – Always on – Even when you’re not looking at the generated metrics • Instruction-level granularity – JVM internal threads, too! • Detailed information about the entire process – Interpretation sometimes requires understanding of the JVM runtime ©2014 Azul Systems Inc. 8
  • 9.
    Tick Profiler: Howit works 1. Thread registers SIG61 signal handler with kernel at intervals 1 ms (1000 times per second, configurable) 2. Thread runs 3. Kernel delivers SIG61 4. Thread is interrupted on its normal stack 5. Signal handler runs, creates a new (youngest) profile Youngest profile added 6. Zing Vision aggregates all of the profiles on the belt in the Timer Tick Profile view Oldest profile drops off… (into the trash) Tick profile conveyor belt has a fixed number of positions for profiles. Note that the conveyor belt only moves when a new profile is added! This means Zing Vision always shows the most recent data. ©2014 Azul Systems Inc. 9
  • 10.
    Tick Profiler: Whereis work done? ©2014 Azul Systems Inc. 10
  • 11.
    Tick Profiler: Whereis work done? Collection controllers Filters Select link to see details ©2014 Azul Systems Inc. 11
  • 12.
    Tick Profiler: Workat instruction level Callee | Caller Ticks for each instruction ©2014 Azul Systems Inc. C2 compiled JDK and application code 12
  • 13.
    Tick Profiler: Howdid I get here? ©2014 Azul Systems Inc. 13
  • 14.
    Tick Profiler: Howdid I get here? Sorted based on highest CPU consumers 0 com.sun.tools.javac.util.Name.fromUtf Top of stack Top of stack ©2014 Azul Systems Inc. 14
  • 15.
    Thread-level questions • Whatthreads are executing my application? • What are the housekeeping threads for the JVM? • Can I examine a thread object? • What are the threads doing? – Profiling information – Stack traces • Where is my application stalled? ©2014 Azul Systems Inc. 15
  • 16.
    Thread-level views: Standardmethod Thread number: 1 2 3 4 5 Request thread stacks (kill -3) Some threads blocked – NO work done Thread 3 continues to run! Time Last thread stopped O(numThreads) delay  application visible work stoppage Resume threads ©2014 Azul Systems Inc. 16
  • 17.
    Thread-level views: ZingVision method Thread number: 1 2 3 4 5 Request thread stacks (ZVision) Each thread: 1. Stopped individually 2. Delay order is O(1) Time No thread waits for any other thread to stop! The application never perceives a work stoppage because N-1 threads continue to run at all times Resume each thread seamlessly Pro: Safe to use in production (guilt-free clicking) Con: Not 100% consistent (especially for lock data) ©2014 Azul Systems Inc. 17
  • 18.
    Thread-level view: Applicationthreads Filters and control Select link to see stack Individual thread stack trace and tick profile access ©2014 Azul Systems Inc. 18
  • 19.
    Thread-level view: Threadstack trace Thread CPU time consumed Stack frames Select link to see details ©2014 Azul Systems Inc. 19
  • 20.
    Thread-level view: Threadobject details Values of the object’s fields ©2014 Azul Systems Inc. 20
  • 21.
    Thread-level view: Applicationthreads Filters and control Select link to see profile Individual thread stack trace and tick profile access ©2014 Azul Systems Inc. 21
  • 22.
    Thread-level: Where isthe work done? Thread 8120 ©2014 Azul Systems Inc. 22
  • 23.
    Lock contention: Whereis my app stalled? Lock acquisition Total and Max times Blocking count Select link to see details ©2014 Azul Systems Inc. 23
  • 24.
    Lock contention: Whereis my app stalled? How did I get to the contended lock in my code? Summary metrics ©2014 Azul Systems Inc. 24
  • 25.
    Memory: Answering yourquestions • What’s the collector doing? • How much memory is the application using? • What type of objects are in the heap? – What’s keeping those objects live? • What object types are increasing in number? ©2014 Azul Systems Inc. 25
  • 26.
    Memory: Resource usesummary Java heap overview Process Zing memory use Linux memory ©2014 Azul Systems Inc. 26
  • 27.
    Memory: Collection details Collectiondetails ©2014 Azul Systems Inc. 27
  • 28.
    Memory: Collection detailssummary Detailed summary calculating metric averages ©2014 Azul Systems Inc. 28
  • 29.
    Memory: What objectsare in the heap? Objects in Old Generation Default sorting: Sum of size of each object of that type Expand to see types with references to objects of that type ©2014 Azul Systems Inc. 29
  • 30.
    Memory: Which objecttype is growing? Object types with increasing memory consumption in the Old Generation Selectable time interval Growth rate ©2014 Azul Systems Inc. 30
  • 31.
    Example ZVRobot deployment Linuxhost Linux host Write a file for each Zing Vision web page at each sample period (for example, every 1 minute) into a directory ZVRobot HTTP Zing VM Use a web browser to look at the saved snapshot files anytime after program has run or even during collection ©2014 Azul Systems Inc. 31
  • 32.
    How to TryZing Vision - Free • Request a trial copy of Zing http://www.azulsystems.com/trial • Download and run Azul Inspector to check your system • Download and install Zing – Zing Vision and ZVRobot are included • Run your application • Observe your application and the JVM’s activity using Zing Vision ©2014 Azul Systems Inc. 32
  • 33.
    Questions? Zing Vision –providing answers to your Java performance questions • Tick profiler • Thread-level views • Lock contention view • Garbage collection views • Java heap object views • Live objects • Object types that increase in number as application runs ©2014 Azul Systems Inc. 33