Diagnose Your Microservices

Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
Diagnose Your Microservices
Open Tracing / Oracle Application Performance Monitoring Cloud
Marcus Hirt / Consulting Member of Technical Staff
John Sullivan / Senior Director of Product Management
OMC
October, 2018

Agenda
1. Microservices
2. OpenTracing
3. JDK Flight Recorder
4. Oracle Management Cloud (APM)
2

Microservices
Confidential – Oracle
3

Monolith vs Microserves
4
Illustrations from https://martinfowler.com/articles/microservices.html

Microservices
• Decentralized
• Independent
• Black box
• Polyglot
• Do one thing, and do it well
• You build it, you run it
5
Characteristics

Conway’s Law
6

The Robotshop
7
Customers
OrdersLoad Generator
Factory
Add/Remove
Verify
Build/Pickup
Submit/Pickup
Check available robot types and colors
https://github.com/thegreystone/problematic-microservices

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 8
Microservices in the Cloud
LoadBalancer
Incoming
Request

OpenTracing
9

Distributed Tracing
• Dapper - Google’s paper on distributed tracing (2010)
• Inspired other distributed tracer implementations
– Zipkin
– Jaeger
– AppDash
– …
10
Dynatrace (2005)
AppDynamics (2008)
New Relic (2008)
DataDog (2010)
…

Java Tracing
• All Tracers/APMs pretty much doing the same thing
– BCI for getting the data (value add - knowing what data to get)
– Presenting findings to the user (value add - knowing what to do with the data)
• Want to add additional contextual information?
– Work with vendor specific proprietary API
• Framework/library want to add more detailed trace information?
– Work with vendor specific proprietary API
11

Open Tracing
• Vendor neutral distributed tracing API
– Library builders can interact with one API to support multiple vendors
– Customers can add contextual information without worrying about vendor lock-in
• Open Tracing is…
– Application level API for producing tracing and trace related information
– API for context propagation
– Mainly contributed by LightStep, Jaeger, Skywalking and Datadog
• Spec and language specific APIs on github:
https://github.com/opentracing
12

OpenTracing
• Trace
– A distributed operation, potentially spanning multiple processes
– Implicitly defined by the individual Spans in the trace (more soon)
– Can be thought of as a directed acyclic graph (DAG) of Spans
– The span in the root of the DAG is called the root Span
– The edges between the Spans are called References
13
Core API Concepts

OpenTracing
• Span
– Has an operation name
– Has a start timestamp
– Has a finish timestamp
– Has a SpanContext
• Has Baggage Items (key/value pairs which cross process boundaries)
• Implementation specific state (used to identify the span across process boundaries)
– Zero or more key/value Span Tags
– Zero or more Span Logs (key/value + timestamp)
14
Core API Concepts, cont’d

OpenTracing
• Reference
– Define a direct casual relationship between two spans
– ChildOf
• Parent depends on the child in some way
• Note that it is legal for a finish timestamp of a child to be after that of any parent
– FollowFrom
• Parent does not depend on the result of the child in any way
• Note that is legal for a FollowsFrom child to be started after the end of any ancestor
– ChildOf is by far the most common reference type
15
Core API Concepts, cont’d

createRobot
FactoryService
createRobot
FactoryService
Example Trace DAG
16
Robotshop
createCustomer
CustomerService
fullSystemTest
LoadGenerator
postOrder
OrderService
validateCustomer
CustomerService
createRobot
FactoryService
pickUpOrder
OrderService
pickUpRobot
FactoryService
deleteCustomer
CustomerService
pickUpRobot
FactoryService
pickUpRobot
FactoryService

Jaeger + Robotshop Demo

JDK Flight Recorder
And JDK Mission Control
Confidential – Oracle
18

JDK Flight Recorder (JFR)
• JVM technology for production time profiling and diagnostics
• Analogous to a Flight Data Recorder in a commercial aircraft
• Open sourced with JDK 11
• Features added by the Hotspot runtime team continuously
• Used at Oracle for years for sorting out problems in production systems
19
"In Fusion we create hundreds of thousands of Flight
Recordings, and we can figure out 95% of the issues
using nothing but the recordings."
- Joe Albowicz, Oracle (Fusion Application Development)

Simple JFR Event Example
20
For more examples, see the JFR part of: https://github.com/thegreystone/java-svc
public class HelloJfr {
@Label("Hello World!")
static class HelloWorldEvent extends Event {
@Label("Message")
String message;
}
public static void main(String ... args) throws IOException {
HelloWorldEvent event = new HelloWorldEvent();
event.message = "Hello World!";
event.commit();
}
}

JDK Mission Control
• An application for, among other things, analysing flight recordings
• Parser and analysis libraries for JDK Flight Recordings
– Statistical operations
– Compact code
– Reports
• Open sourced
• Early access builds of the application available (http://jdk.java.net/jmc/)
• Snapshot builds of the libraries available at Sonatype (org.openjdk.jmc.*)
21

Example: Generating an Analysis Report (HTML)
22
public static void main(String[] args) throws Exception {
String report = JfrHtmlRulesReport.createReport(
JfrLoaderToolkit.loadEvents(new File(args[0])));
System.out.println(report);
}
For more examples, see the JMC part of: https://github.com/thegreystone/java-svc
- or just run the main method of JfrHtmlRulesReport

Example: Calculating Statistics for Monitor Enter Events
23
For more examples, see the JMC part of: https://github.com/thegreystone/java-svc
public static void main(String[] args) throws IOException, CouldNotLoadRecordingException {
IItemCollection events = JfrLoaderToolkit.loadEvents(new File(args[0]));
IItemCollection monitorEnterEvents = events.apply(JdkFilters.MONITOR_ENTER);
IQuantity eventCount = monitorEnterEvents.getAggregate(Aggregators.count());
IQuantity avg = monitorEnterEvents.getAggregate(Aggregators.avg((JfrAttributes.DURATION));
IQuantity stddev = monitorEnterEvents.getAggregate(Aggregators.stddev(
JfrAttributes.DURATION));
System.out.println(String.format("# of events: %d, avg: %s, stdddev: %sn",
eventCount.longValue(),
avg.displayUsing(IDisplayable.AUTO),
stddev.displayUsing(IDisplayable.AUTO)));
}

Idea: Marrying OpenTracing and JDK Flight Recorder
• Combine OpenTracing and Flight Recorder
– Distributed tracing with deep diagnostics capabilities
• Create a JFR OpenTracing Tracer
– Emit flight recording events for Spans and Scopes (thread local)
– Delegate to whichever Tracer is someone’s favourite tracer
– Since context is implementation specific, provide plug-in mechanism for custom
context extractors
– Have it support Oracle JDK 7+ and OpenJDK 11+ by building it as an mrjar
• Open sourced prototype (links towards the end)
24

jfr-tracer + Robotshop Demo

But…
• What if you have thousands of these Java processes?
• How do you trigger the recording dumps?
• Where do you store the recording data?
• What if you also want the logs, and the means to process the logs?
• What if you want some metrics already at the distributed tracing level?
26

Oracle Management Cloud
Application Performance Management
27

APM Diagnostics Use Case Examples
28
Performance issues in the past
• Use APM to monitor and analyze server requests
• Use APM with deep dive thread samples to locate the slow code
• Very recent past - Dump JFR and analyze in APM and in JMC
The problem is happening right NOW!!
• Dump JFR
• Run the profiler
Problem is about to happen (i.e. memory leak)
• Take class histograms and compare/trend later

End-2-End Transaction Tracing, Code-level Visibility
• Measure all ingress/egress points
• Measure preformance across
service and tiers
• User defined custom tracing points
(OpenTracing, POJO, etc.) when
OOB are not sufficent
• Leverage diagnostics on
monitoring data anywhere/anytime
29

Custom Calls Added OpenTrace in APM
30

Continuous
Integration
Java Flight
Recorder
Build Better Applications with Oracle
Code
Repository
Telemetry
from test
Telemetry
from pre-
prod/prod
AVAILABILITY
APM
ALERTSSYS
PAGING
VIZ.FRAMEWORK
DBMONITORING
CUSTOM
MONITORING
LOGGING
BIGDATA
MONITORING
DISTRIBUTED
TRACING
CLOUD
MONITORING
Management At Cloud Scale
Monitoring Large-Scale Distributed Applications in
Oracle Cloud - Thursday, Oct 25, 10:00 a.m. - 10:45 a.m.
| Moscone West - Room 2009 - DEV6151
Diagnose Your Microservices: OpenTracing/Oracle
Application Performance Monitoring Cloud -
Wednesday, Oct 24, 4:00 p.m. - 4:45 p.m. | Moscone
West - Room 2011 - DEV5435
Microservices: Ten Steps to Build, Manage, and Monitor
Your Microservices - Wednesday, Oct 24, 9:00 a.m. -
9:45 a.m. | Moscone West - Room 2008 - DEV5564
Build
Diagnose
Run
38

Summary
• OpenTracing is a vendor neutral initiative for a standard API for supporting
distributed tracing
• JDK Flight Recorder is a framework for production time profiling and
distnostics built into the JVM
• JDK Flight Recorder is open sourced, and included in OpenJDK, since JDK 11
• Oracle has a powerful APM in Oracle Management Cloud
– With support for OpenTracing as a Zipkin agent communication endpoint, and…
– …as a custom OpenTracing Tracer (metrics++ from the OMC APM agent)
– With JFR support
39

Other JMC-/JFR-related Presentations
Day Time Title Location
Monday 10:30 Contributing to the Mission Control OpenJDK Project MW 2004
Tuesday 11:30 Three Productive Ways to Use Open Source Java Flight Recorder and
Java Mission Control
MW 2024
Tuesday 14:30 Fast and Furious: Java Flight Recorder and Flame Graphs MW 2020
Wednesday 10:30 Production-Time Profiling and Diagnostics on the JVM MW 2004
Wednesday 12:30 OpenJDK Mission Control: The Hands-on-Lab MW 2001A
Wednesday 16:30 Diagnose Your Microservices: OpenTracing/Oracle APM Cloud MW 2011
Thursday 11:00 Getting Started with the (Open Source) JDK Mission Control MW 2014
Thursday 12:30 Flight Recorder in OpenJDK MW 2018
40

Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

Resources
• OMC: http://cloud.oracle.com/OMC
• Marcus’ blog: http://hirt.se/blog
• Marcus’ GitHub: https://github.com/thegreystone
https://github.com/thegreystone/jfr-tracer
(Delegating JFR tracer for OpenTracing)
https://github.com/thegreystone/problematic-microservices
(Robotshop, use the jfr-tracer branch)
42

Diagnose Your Microservices

More Related Content

What's hot

Similar to Diagnose Your Microservices

Recently uploaded

Diagnose Your Microservices