This document discusses different ways to instrument software applications for monitoring purposes. It begins with definitions of instrumentation and telemetry. It then discusses the different types of data that can be collected through instrumentation like metrics and events. It describes how to instrument various parts of applications including the frontend like browsers and mobile apps as well as the backend for languages like Java, .NET, Node.js and PHP. It provides best practices for instrumentation and logging. Finally, it discusses challenges with correlating data across distributed systems and some open source options and solutions for transaction correlation.

1. The differing ways to monitor and
instrument
Jonah Kowall
VP Market Development and Insights
Twitter : @jkowall

2. APPDYNAMICS CONFIDENTIAL AND PROPRIETARY 2
Notice
The information and materials included in this presentation (collectively, the
“Materials”) are the proprietary information of AppDynamics, Inc. (“AppDynamics” or
the “Company”). No part of the Materials may be reproduced, distributed,
communicated or displayed in any form or by any means, or used to make any
derivative work, without prior written permission from AppDynamics.
The Materials may contain product roadmap information of AppDynamics.
AppDynamics reserves the right to change any product roadmap information at any
time, for any reason and without notice. This information is intended to outline
AppDynamics' general product direction, it is not a guarantee of future product
features, and it should not be relied on in making a purchasing decision. The
development, release, and timing of any features or functionality described for
AppDynamics' products remains at AppDynamics' sole discretion. AppDynamics
reserves the right to change any planned features at any time before making them
generally available as well as never making them generally available.
All third-party trademarks, including names, logos and brands, referenced by
AppDynamics in this presentation are property of their respective owners. All
references to third-party trademarks are for identification purposes only and shall be
considered nominative fair use under trademark law. © 2016 AppDynamics, Inc. All
rights reserved.

3. Jonah Kowall’s background
• 23 years in IT
• Infrastructure and Operations enterprises and startups
(17 yrs)
– Security - CISSP, CISA, PCI
– Started one of the first content filtering companies
• Head of global monitoring at Thomson Reuters
• Head of IT Operations at MFG.com – Bezos Expeditions
• Gartner Research VP 3.5 years
• Strategy AppDynamics 1.5 years

4. Agenda
• Introduction to instrumentation
• Instrumentation defined
• Priorities in instrumentation
• Technical instrumentation of frontends (Browser and Mobile)
• Technical instrumentation of backend (Java, .NET, Python, Node.js,
PHP)
• Logging best practices
• How transaction correlation options (AppDynamics and Open
Source)

5. Definitions
Instrumentation
“The design, construction, and provision of instruments
for measurement, control, etc; the state of being
equipped with or controlled by such instruments
collectively.”
https://en.wikipedia.org/wiki/Telemetry
Telemetry
“Automated communications process by which
measurements are made and other data collected
at remote or inaccessible points and transmitted to
receiving equipment for monitoring.”
https://en.wikipedia.org/wiki/Instrumentation

6. Software instrumentation data types
• Metrics
– Key value pairs
– Numeric values
– Time series
• Events
– Informational
– Errors
– Critical events

7. Use cases for data types
• Metrics
– Average, peak
– Percentage
– Correlation to metrics and events
• Events
– Search
– Parse
– Correlate to metrics

8. Software instrumentation
• Logging
– Supplied by vendor
– Created by developers
– Not easily controlled
• Push Collection
– Attach and extract
– Software agent or network tap
• Pull Collection
– Polling APIs - HTTP, SNMP, WMI

9. Instrumentation
misalignment

10. Priorities for instrumentation
• Business
• Application
• Services
• Infrastructure
Events and metrics

11. Uplevel the conversation
• Understand the customer
– Internally and externally
• Requirements should be gathered across business and IT teams
• Responsibility for definition of monitoring should be shared

12. Business metrics and KPIs
• Customer metrics
– Conversion between
products
– Loyalty and retention
(churn)
– Usage metrics (feature
and product)
• Sales / marketing metrics
– Revenue
– Cost of customer
acquisition
– User flows through
applications

13. Technical metrics and KPIs
• End to end performance
– User through transaction hops
– Error isolation
• End user experience
– Client side errors
– Latency per element (page or app) + 3rd
party
– Client side DNS
• Application component performance
– Metrics from app server
– Metrics from code
– Queries
– Errors
• Intra application component performance

14. Use cases for business and technical data
• Usage
• Problem identification - MTTI
• Problem resolution - MTTR
• User satisfaction
• Usability
• Performance
• Change analysis
– A/B testing
– data center moves
– technology changes

15. Frontend instrumentation

16. Instrumenting browsers

17. Instrumenting mobile

18. Simulating users
• Synthetic transactions are good for
• SLAs
• Availability
• Baseline performance
• DNS
• SSL
• But if you try to use it as a
barometer you will fail

19. Backend instrumentation
of Java and .NET

20. Java instrumentation
JSR-163 (JavaTM Platform
Profiling Architecture)
added in Java 1.5
Overloads the default
behavior of Java to allow
hooks into code for many
use cases
Since JDK 1.6, for the Oracle
HotSpot JVM, a javaagent
may be dynamically attached
to a running JVM by
specifying the process-id
(pid).

21. .NET Instrumentation
● Profiling API loaded into the same process as the application process that is being profiled.
● Callback interface (ICorProfilerCallback in the .NET Framework version 1.0 and 1.1,
ICorProfilerCallback2 in version 2.0 and later)
● CLR calls the methods in that interface to notify the .NET agent of events in the profiled process
● Profiler can also call into the CLR by using the methods in the ICorProfilerInfo and ICorProfilerInfo2
interfaces to obtain information about the state of the profiled application
● Callbacks are used to inject MSIL (Microsoft Intermediate Language) bytecode into existing application
code for instrumentation.

22. Backend instrumentation
of interpreted languages

23. Monkey patching
Wikipedia relevant definition:
In Ruby,[3] Python,[4] and many other dynamic
programming languages... dynamic modifications of a
class or module at runtime, motivated by the intent to
patch existing third-party code as a workaround to a
bug or feature which does not act as desired
Disclaimer : Can be very dangerous, hard to maintain
• Replace methods / attributes / functions at runtime
• Apply a patch at runtime to the objects in memory,
instead of the source code on disk;

24. PHP instrumentation
Zend callback methods zend_execute(…), zend_execute_internal(…)
and zend_compile_file(…) so that it can wrap the original
implementations with instrumentation code.
Handles state changes
and new web server
initialization (which are
PHP instances)

25. Node.js instrumentation
● Wrap methods using before, after and around aspect
interceptors.
● Callback along with after, before and around aspect interceptor.
● Notifications when asynchronous calls are complete.

26. Python instrumentation

27. Transaction correlation

28. Correlation in end to end APM

29. Correlation in asynchronous calls (headache)

30. Correlation in Open Source
• OpenZipkin is the most advanced (backed by Pivotal)
– New instrumentations (Java, Scala, JavaScript, Python, Ruby, C#, Go)
– Requires code changes
– No async support, no overhead controls (dependant on how
implemented in code)

31. Future of correlation in Open Source
• OpenTracing an open standard API for instrumentation
– Vendors must implement specific code per instrumentation library which
doesn’t scale
– Must add code manually
– No overhead controls
• PivotTracing runs distributed traces on demand backed by Brown
University
– Currently has been written into forks of
• Apache Hadoop (YARN, MapReduce, HDFS)
• HBase
• ZooKeeper

32. Open Source is starting to merge (Zipkin + Spigo)
Spigo created by Adrian Cockroft for Simulation and Visualization of a
complex App Topology
Can now feed Zipkin traces into Spigo and
visualize response times for a trace in a
topology (still manual using 3erd party
tool, and it’s missing any kind of real UI)

33. Logging and log
correlation

34. Logging best practices
• Logs are not transaction records
• Log errors and exceptions
• Easily parsed (JSON)
• Time (long), Source
• Write your own identifiers for each statement logged (or instrument
and inject)
• Think about security implications (plain text, on disk, syslog are all
insecure)
• Keep log statements small (thanks Java, .NET…)
• Every log statement introduces overhead, so don’t overdo it
• Exception logs create even more overhead, so fix them
• Do not try to use log tools as metric stores

35. Correlation in logs
• Log every transaction segment
• Persist a GUID or transaction ID
• This is very difficult in large teams
• Inefficient to analyze and pull metrics from logs
• Doesn’t work unless you own the code
[code]
PERF,2013-04-03 11:29:52.640,external,0x123456,NA,service1,MyAPP,jimmy,NA,336,NA,NA
INFO,2013-04-03 11:29:53.189,internal,789012,0x123456,service2,TheirApp,jimmy,NA,174,NA,NA
INFO,2013-04-03 11:29:52.892,internal,345678,789012,service3,TheirApp,jimmy,NA,163,NA,NA
[/code]

36. Transaction correlation and logs!
• Many integrations across APM
and Log vendors
• Can add correlation in code
and use any log tool
– ex: [%X{AD.requestGUID}]
• We auto inject and correlate
(one platform)

37. Please give us your feedback—Session T8455
• Complete the online survey you'll receive via
email later today or via text at:
Text this number: 878787
Text this word: APPSPHERE
• Every time you submit a session survey, your
name will be entered in a random drawing.
We're giving away Amazon Echos
to 5 lucky winners!
• Thank you for your input
APPDYNAMICS CONFIDENTIAL AND PROPRIETARY 37
Win!

38. Thank you