SlideShare a Scribd company logo
1 of 51
Download to read offline
§ Michael Dawson/Michael Tunnicliffe
§ September 2015
A Taste of Monitoring and Post Mortem
Debugging with Node
A Workshop
2
To talk about: Application reliability and performance
●
Common problem scenarios
●
Monitoring
●
Problem diagnosis
●
Tailoring tooling to your application
Try it for yourself with prepared hands-on activities
To share and learn from each others experiences
Why are we here ?
3
• Meet and Greet
• Runtime Monitoring
• Concepts and Tools
• Hands on - Monitor an App
• GC Visualisation
• Concepts and Tools
• Hands on – visualize gc activity
• Post Mortem Debugging
• Concepts and Tools
• Hands on with a heap dump
• Hands on with a core dump
• Experience/Gaps Brainstorm
• Monitoring – Instrument a Module
Heap usage/Object layout
Initial Problem Determination
Dig into GC behaviour (if appropriate)
Execution State/Native Memory
Retrospective
Improve your tools
Modelled on typical workflow
Agenda
4
About Michael Dawson
Loves the web and building software
Senior Software Developer @ IBM
22 years development experience
10 years in runtime development
Contact me:
Michael_dawson@ca.ibm.com
https://www.linkedin.com/pub/michael-dawson/2/128/605
Twitter: @mhdawson1
5
About Mike Tunnicliffe
Bug detective and low level systems programmer
Software Developer @ IBM
15 years development experience
12 years in runtime development
Contact me:
mike.tunnicliffe@uk.ibm.com
@fierydrake
6
• Interest in Monitoring/Post mortem debugging ?
• What you hope to get out of the workshop today ?
About You (workshop participants)
Used Node Before ?
Role (Dev/QA/Ops) ?
Apps in Production ?
Production Monitoring ?
Background
Memory Usage ?
GC activity
Throughput
Reponse Time
Something else ?
Metrics – what do you track
High Memory
Memory Exhaustion
Memory Leaks
Footprint
Performance
Crash
Unhandled Exceptions
Problems – what have you seen
7
• USB drive provided for easy installation
• IBM is not a distributor
• While freely available, licences apply to some components
●
Licences are included on USB drive and in install packages
●
By copying from USB drive you are accepting licence terms in USB drive
as licensee
• Printed copy also available
• Alternatively follow full download/install instructions.
• Install Instructions
●
Pre-Install/Instructions.txt
●
Self-Install/Instructions.txt
Workshop materials
8
• Runtime Monitoring
• GC Visualization
• Post Mortem Debugging
• Experience/Gaps Brainstorm
• Monitoring – Instrument a Module
Heap usage/Object layout
Initial Problem Determination
Dig into GC behaviour
Execution State/Native Memory
Retrospective
Improve your tools
Modelled on typical workflow
Runtime Monitoring
9
To understand the “real” characteristics of the application
• In production – health check under real load
• During development – test under artificial load
Why do runtime monitoring?
10
Data about the application and the environment on which it
runs, typically:
• Machine and operating system
• Node.js and dependencies (eg V8)
• Application and dependencies (eg http, mongodb)
What will it give you?
11
Typically production monitoring consists of:
• Data sources
• Data store aggregator
• Web-based visualizer
How it works
Browser
Monitoring
Data
HTT
P LoadBalancer
12
Production monitoring solutions
13
Setup
• Run startelk
• Run startapps
• Open Kibana in a browser
http://localhost:5601/
• Open the dashboard
Try to identify any potential problems with the monitored
applications
Monitoring – Hands on – 20 minutes
14
Data Sources (open source)
• ElasticSearch / Kibana (ELK)
●
bunyan-child, hapi-good-logstash, winston-kibana, log4js-elasticsearch,
black-pearl, appmetrics-elk, etc
• StatsD / Graphite
●
Express-metrics, connect-statsd, mongodb-datadog-statsd, node-statsd,
statsd-profiler, node-statsd-instrument, appmetrics-statsd, etc.
• Munin
●
nodejs-munin, munin-plugin, etc
• Custom:
●
monitor, appmetrics, etc
15
Node Application Metrics ('appmetrics')
• https://www.npmjs.com/package/appmetrics
• https://github.com/RuntimeTools/appmetrics
• Open Source under Apache 2.0 License*
• Provides:
●
producer and consumer APIs
●
monkey-patching framework and “dropins”
●
collection of connectors to existing monitoring stacks
*has a (currently) proprietary dependency
16
appmetrics Data Collection
Data Type Event Name Data Provided
Application Data
HTTP request/response 'http' Time, Method, URL, Duration
MySQL database queries 'mysql' Time, SQL, Duration
MongoDB database queries 'mongo' Time, Query, Duration
Request tracking 'request' Time, Event Tree, Duration
JavaScript Runtime
V8 Garbage Collection 'gc' Time, Type, Used, Size, Duration
V8 Profiler 'profiling' Time, Function Array, Samples
System Resources
CPU Usage 'cpu' Time, Process, System
Memory Usage 'memory' Time, Process (Physical, Reserved, Virtual),
System (Physical, Total)
17
Production Monitoring with 'appmetrics'
18
Development Monitoring with 'appmetrics'
Health Center
for Eclipse
appmetrics-dash
19
Appmetrics and ElasticSearch/Kibana (ELK)
var appmetrics = require('appmetrics-elk').monitor();
var http = require('http');
var server = http.createServer(function handler(req, res) {
…
}).listen(process.env.PORT || 3000);
console.log('App listening on port 3000');
});
ES ES ES ES
client.bulk()
visualize
20
'appmetrics' and Health Center for Eclipse
• Free from Eclipse Marketplace
• http://marketplace.eclipse.org/content/ibm-monitoring-and-diagnostic-tools-health-
• Supports Node and Java
• Uses MQTT for the connection
Pub/Sub Broker
Topic namespace:
/ibm/healthcenter
/<namespace>/<agent_id>/<datatype>
Node.js
Application
Liberty
Java
Application
MQTT
MQTT
MQTT
MQTT
MQTT Broker, eg. mosca
(npm install -g mosca)
21
Health Center
Environment
Hardware and Operating System Configuration
Process environment and configuration
Highlights incorrect or non-standard configurations
Memory Utilization
Detect native memory leaks in application
Determine if external forces are using more memory
View components using the most native memory
CPU Utilization
Visualizes process CPU usage over time
Visualizes system CPU usage over time
22
Health Center
Garbage Collection
Visualizes heap usage and GC pause times
Identifies memory leaks
Suggests command-line and tuning parameters
Function Profiling
Function profiling shows application activity
Identifies the hottest methods in an application
Full call stacks to identify where functions are being
called from and what functions they call
23
Monitoring – Hands on – 20 minutes
• Run sample app (gcload.bat/gcload.sh) and connect with Health Center
• Look at environment information
• Look through other available panels
• Look at gc activity so see periodic allocation load
• Run sample app with heavy cpu usage (cpuload.bat/cpuload.sh)
• Enable profiling
• Look at profile, map back to application source
• Run custom program that consumes data.
• Edit gcload.js to un-comment line that create monitor instance
• Re-run and see output on console.
24
• Runtime Monitoring
• GC Visualisation
• Post Mortem Debugging
• Experience/Gaps Brainstorm
• Monitoring – Instrument a Module
Heap usage/Object layout
Initial Problem Determination
Dig into GC behaviour
Execution State/Native Memory
Retrospective
Improve your tools
Modelled on typical workflow
Runtime Monitoring
25
GC Visualization
• Visualisation of gc data long known to be important
• There are many existing gc visualizers (mostly Java)
•
•
•
•
•
• Javascript
●
https://github.com/adobe-research/GCview
●
http://marketplace.eclipse.org/content/ibm-monitoring-and-diagnostic-tools-garbage
We’ll use GCMV since we are most familiar with it
26
GCMV
• Free from Eclipse Marketplace
• http://marketplace.eclipse.org/content/ibm-monitoring-and-diagnostic-tools-
garbage-collection-and-memory-visualizer-gcmv
• Supports both Node and Java
27
GC Visualization – Hands on (15mins)
• Run sample app with memory leak with gc options enabled (gcloadWithVerbose)
• Observe there is a lot of information
• Run again sending to file
• Open up resulting file in GCMV
28
• Runtime Monitoring
• GC Visualisation
• Post Mortem Debugging
• Experience/Gaps Brainstorm
• Monitoring – Instrument a Module
Heap usage/Object layout
Initial Problem Determination
Dig into GC behaviour
Execution State/Native Memory
Retrospective
Improve your tools
Post Mortem Debugging
29
• 2 Common artifacts
• Heap Dumps
• Core Dumps
• No standards yet but workgroup:
• https://github.com/nodejs/post-mortem
• Heap dumps provide insight into Heap memory usage
• Core Dumps can provide insight into:
• Heap memory usage
• Runtime State
• What was running at the time of a crash – native and JS stacks
• Values in native memory
• Native memory usage
Post Mortem Debugging
30
• Run Sample OOM program
• Generate Heapdump on OOM
• Load OOM into Chrome developer tools
• Show how to find what is keeps memory alive
• Participants explore
Chrome Developer Tools – Hands on – 10 minutes
31
• Free from Eclipse Marketplace
• http://marketplace.eclipse.org/content/ibm-monitoring-and-diagnostic-tools-interac
• Multi-runtime → Supports Node and Java
• Currently requires dumps from IBM SDK for Node.js
• http://www.ibm.com/developerworks/web/nodesdk/
Interactive Diagnostic Data Explorer (IDDE)
32
• Open pre-provided core file
• Use IDDE commands to show where crash occurred
• Help to list all commands
• Explore
IDDE – Hands On – 10 minutes
33
• Runtime Monitoring
• GC Visualisation
• Post Mortem Debugging
• Experience/Gaps Brainstorm
• Monitoring – Instrument a Module
Heap usage/Object layout
Initial Problem Determination
Dig into GC behaviour
Execution State/Native Memory
Retrospective
Improve your tools
Retrospective
34
• Experience from Participants (Open discussion)
– Other tools
– Key issues
– Workflow
• Gaps Brainstorm
– Key gaps
– Post mortem debugging – useful commands ?
Other Experiences/Gaps Brainstorm (10-15 mins)
35
• Runtime Monitoring
• GC Visualisation
• Post Mortem Debugging
• Experience/Gaps Brainstorm
• Monitoring – Instrument a Module
Heap usage/Object layout
Initial Problem Determination
Dig into GC behaviour
Execution State/Native Memory
Retrospective
Improve your tools
Improve
36
You may collect timing and event data from your application,
or modules that you use.
Wouldn't it be great if you could capture and view that in the
same way as the other monitoring data?
Custom monitoring
37
The appmetrics module provides:
• A producer API for emitting custom data
• A consumer API for receiving all data
Using appmetrics APIs
38
Using the emit() and Event APIs
var appmetrics = require('appmetrics');
var data = {
time: Date.now(),
mydata: 7
};
appmetrics.emit('myevent', data);
var appmetrics = require('appmetrics');
var monitor = appmetrics.monitor();
monitor.on('myevent', function(data) {
console.log('[myevent] ' + data.mydata);
});
• emit() API is on the appmetrics object
• Convention to have a “time” field
– Allows it to be used in ElasticSearch
• Events are on the monitored instance object
– Allows for future remote monitoring
39
Instrument a simple piece of test code:
• Emit a custom 'iteration' event with the following data:
count
time taken to generate the array
time taken to serialize
• Add a handler that logs the data
Improve – Instrument an application (10 mins)
// Crunch some numbers
setInterval(function() {
var object = [];
var count = Math.random()*10000000;
for (var i=0; i<count; i++) {
object.push(i)
}
JSON.stringify(object);
}, 1000);
40
You may not be able to modify the code you want to
instrument
• For example: a third-party or core module
• You could monkey patch it
Instrumenting modules
41
A simple monkey patching example
var appmetrics = require('appmetrics');
var fs = require('fs');
var originalReadFileSync = fs.readFileSync;
// Instrument readFileSync
fs.readFileSync = function() {
var startTime = Date.now();
originalReadFileSync.apply(fs, arguments);
var endTime = Date.now();
appmetrics.emit('loadfile', { 'duration': endTime - startTime });
};
// Set up monitoring listeners
appmetrics.monitor().on('loadfile', function(e) {
console.log('File load duration=' + e['duration'] + 'ms');
});
// Application
for (var i=0; i<100; i++) {
fs.readFileSync('bigfile.dat');
}
Store function to
patch
Patch function
by replacing
Custom
“visualizer”
42
Asynchronous monkey patching example
var appmetrics = require('appmetrics');
var fs = require('fs');
var originalReadFile = fs.readFile;
// Instrument readFileSync
fs.readFile = function() {
var startTime = Date.now();
var callbackIndex = (typeof(arguments[1]) === 'function') ? 1 :
(typeof(arguments[2]) === 'function') ? 2 : -1;
if (callbackIndex == -1) return;
var originalCallback = arguments[callbackIndex];
var wrap = function() {
appmetrics.emit('loadfile', { 'duration': Date.now() - startTime });
originalCallback.call(arguments);
};
arguments[callbackIndex] = wrap;
originalReadFile.apply(fs, arguments);
};
// Set up monitoring listeners
appmetrics.monitor().on('loadfile', console.log);
// Application
for (var i=0; i<100; i++) {
fs.readFile('bigfile.dat', function(err, data) {
console.log('readFile completed');
});
}
Store function to
patch
Patch function
by replacing
Custom
“visualizer”
43
Deferred monkey patching example
var module = require('module');
var moduleLoad = module._load;
module._load = function(mod, parent, isMain) {
var loadedModule = moduleLoad.apply(this, arguments);
if (mod === 'http') {
var addListener = loadedModule.Server.prototype.addListener;
loadedModule.Server.prototype.addListener = function(type, listener) {
if (type === 'request' && typeof listener === 'function'){
var oldListener = listener;
listener = function(req, res){
if (typeof res.startTime === 'undefined') {
res.startTime = new Date().getTime();
res.once('finish', function() {
res.endTime = new Date().getTime();
var dur = res.endTime - res.startTime;
console.log('['+ res.startTime + '] ' + req.method + ' : ' + req.url + ' ('+dur+')');
});
};
oldListener.call(this, req, res);
};
return addListener.call(this, type, listener);
} else {
return addListener.apply(this, arguments);
}
};
};
return loadedModule;
};
44
The appmetrics module provides:
• Drop-in “probes”
Self-contained facility to deploy patches in appmetrics
• An aspect framework for use by probes
Simplify complex method replacement scenarios
Making patching easier
45
Using the Probes framework
Probe scripts are automatically loaded from:
node_modules/appmetrics/probes
Template probe:
var Probe = require('../lib/probe.js');
var aspect = require('../lib/aspect.js');
var util = require('util');
function MyExampleProbe() {
Probe.call(this, 'name_of_module_to_patch');
}
util.inherits(MyExampleProbe, Probe);
MyExampleProbe.prototype.attach = function(name, target, am) {
if (target.__probeAttached__) return target;
target.__probeAttached__ = true;
// Code to patch and emit event (can use aspect)
return target;
};
module.exports = MyExampleProbe;
46
Aspect API
aspect.before(obj, methods, func);
aspect.after(obj, methods, func);
aspect.around(obj, methods, func1, func2);
aspect.aroundCallback(obj, methods, func1, func2);
• Provides APIs to locate and patch functions
– before, after and around supported
HTTPProbe.prototype.attach = function(name, target, am) {
if (target.__probeAttached__) return target;
target.__probeAttached__ = true;
aspect.before(target.Server.prototype, ['on', 'addListener'], function(obj, args) {
if (args[0] !== 'request') return;
aspect.aroundCallback(args, function(obj, args) {
var request = args[0], response = args[1]; // args for callback are [request, response]
response.startTime = Date.now();
aspect.after(response, 'finish', function(obj, args, ret) {
var dur = Date.now() - response.startTime;
console.log('['+ response.startTime + '] ' + request.method + ' : ' + request.url + ' ('+dur+')');
});
});
});
return target;
};
• Reduces the complex monkey patch example to...
47
Using the Aspects framework
Aspects framework makes it easy to patch specific functions in the module
Simple use:
The “around” function is basically “before” and “after” combined
MyExampleProbe.prototype.attach = function(name, target, am) {
if (target.__probeAttached__) return target;
target.__probeAttached__ = true;
var start;
aspect.around(target, 'readFileSync',
function(obj, args) {
start = Date.now();
},
function(obj, args, returnObject) {
am.emit('file', {time: start, method: 'readFileSync', file: args[0], duration: Date.now() - start});
return returnObject;
}
);
return target;
};
48
Using the Aspects framework
This, more complex, scenario can happen when you want to patch async functions
that accept callback arguments
The “aroundCallback” function will search a given list of objects and patch the
first function it finds (good for patching async handlers)
MyExampleProbe.prototype.attach = function(name, target, am) {
if (target.__probeAttached__) return target;
target.__probeAttached__ = true;
// 1. Run some code after moduleToPatch.functionToPatch() to...
// 2. - run some code before a function on its return value (returnObject.functionOnReturnObjectToPatch()) to...
// 3. - search that function's arguments for a callback and run some code before that callback
aspect.after(target, 'functionToPatch', function(target, args, returnObject) {
aspect.before(returnObject, 'functionOnReturnObjectToPatch',
function(target, methodArgs) {
aspect.aroundCallback(methodArgs, function(target, args){
// code
});
}
);
return returnObject;
});
return target;
};
1
2
3
49
Instrument fs.readFile() by creating a probe
Start with the template code →
●
Set the name to 'fs'
●
Replace the comment with
a call into the aspect framework
to patch readFile() and associated
callback
●
Drop into the probes directory
inside appmetrics
Use this simple script to test →
Improve – Instrument a module (10 mins)
var fs = require('fs');
setInterval(function() {
fs.readFile('bigfile.dat', function(err, data) {
console.log(data.length);
});
}, 1000);
var Probe = require('../lib/probe.js');
var aspect = require('../lib/aspect.js');
var util = require('util');
function MyExampleProbe() {
Probe.call(this, 'name_of_module_to_patch');
}
util.inherits(MyExampleProbe, Probe);
MyExampleProbe.prototype.attach = function(name, target, am) {
if (target.__probeAttached__) return target;
target.__probeAttached__ = true;
// Code to patch and emit event (can use aspect)
return target;
};
module.exports = MyExampleProbe;
50
• Runtime Monitoring
• GC Visualisation
• Post Mortem Debugging
• Experience/Gaps Brainstorm
• Monitoring – Instrument a Module
Heap usage/Object layout
Initial Problem Determination
Dig into GC behaviour
Execution State/Native Memory
Retrospective
Improve your tools
Summary and Wrap up
51
Copyrights and Trademarks
© IBM Corporation 2015. All Rights Reserved
IBM, the IBM logo, ibm.com are trademarks or registered
trademarks of International Business Machines Corp.,
registered in many jurisdictions worldwide. Other product and
service names might be trademarks of IBM or other companies.
A current list of IBM trademarks is available on the Web at
“Copyright and trademark information” at
www.ibm.com/legal/copytrade.shtml
Node.js is an official trademark of Joyent. IBM SDK for Node.js is not formally
related to or endorsed by the official Joyent Node.js open source or
commercial project.
Java, JavaScript and all Java-based trademarks and logos are trademarks or
registered trademarks of Oracle and/or its affiliates.
"All other servicemarks and trademarks are the property of their respective
owner."

More Related Content

What's hot

Csw2016 d antoine_automatic_exploitgeneration
Csw2016 d antoine_automatic_exploitgenerationCsw2016 d antoine_automatic_exploitgeneration
Csw2016 d antoine_automatic_exploitgeneration
CanSecWest
 
Eclipse DemoCamp Bucharest 2014 - Continuous Integration Jenkins/Hudson
Eclipse DemoCamp Bucharest 2014 - Continuous Integration Jenkins/HudsonEclipse DemoCamp Bucharest 2014 - Continuous Integration Jenkins/Hudson
Eclipse DemoCamp Bucharest 2014 - Continuous Integration Jenkins/Hudson
VladLica
 
Java Development EcoSystem
Java Development EcoSystemJava Development EcoSystem
Java Development EcoSystem
Alex Tumanoff
 
Intro to XPages for Administrators (DanNotes, November 28, 2012)
Intro to XPages for Administrators (DanNotes, November 28, 2012)Intro to XPages for Administrators (DanNotes, November 28, 2012)
Intro to XPages for Administrators (DanNotes, November 28, 2012)
Per Henrik Lausten
 

What's hot (20)

Csw2016 d antoine_automatic_exploitgeneration
Csw2016 d antoine_automatic_exploitgenerationCsw2016 d antoine_automatic_exploitgeneration
Csw2016 d antoine_automatic_exploitgeneration
 
Eclipse DemoCamp Bucharest 2014 - Continuous Integration Jenkins/Hudson
Eclipse DemoCamp Bucharest 2014 - Continuous Integration Jenkins/HudsonEclipse DemoCamp Bucharest 2014 - Continuous Integration Jenkins/Hudson
Eclipse DemoCamp Bucharest 2014 - Continuous Integration Jenkins/Hudson
 
The Ultimate Android Security Checklist (Mdevcon 2014)
The Ultimate Android Security Checklist (Mdevcon 2014)The Ultimate Android Security Checklist (Mdevcon 2014)
The Ultimate Android Security Checklist (Mdevcon 2014)
 
Using NuGet the way you should - TechDays NL 2014
Using NuGet the way you should - TechDays NL 2014Using NuGet the way you should - TechDays NL 2014
Using NuGet the way you should - TechDays NL 2014
 
Finding and exploiting novel flaws in Java software (SyScan 2015)
Finding and exploiting novel flaws in Java software (SyScan 2015)Finding and exploiting novel flaws in Java software (SyScan 2015)
Finding and exploiting novel flaws in Java software (SyScan 2015)
 
EuroBSDCon 2013 - Mitigating DDoS Attacks at Layer 7
EuroBSDCon 2013 - Mitigating DDoS Attacks at Layer 7EuroBSDCon 2013 - Mitigating DDoS Attacks at Layer 7
EuroBSDCon 2013 - Mitigating DDoS Attacks at Layer 7
 
Java Development EcoSystem
Java Development EcoSystemJava Development EcoSystem
Java Development EcoSystem
 
The Ultimate Android Security Checklist (AnDevCon Boston 2014)
The Ultimate Android Security Checklist (AnDevCon Boston 2014)The Ultimate Android Security Checklist (AnDevCon Boston 2014)
The Ultimate Android Security Checklist (AnDevCon Boston 2014)
 
The Seven Habits of Highly Effective Puppet Users - PuppetConf 2014
The Seven Habits of Highly Effective Puppet Users - PuppetConf 2014The Seven Habits of Highly Effective Puppet Users - PuppetConf 2014
The Seven Habits of Highly Effective Puppet Users - PuppetConf 2014
 
Android ndk: Entering the native world
Android ndk: Entering the native worldAndroid ndk: Entering the native world
Android ndk: Entering the native world
 
Android ndk
Android ndkAndroid ndk
Android ndk
 
Webinar: Code Faster on Kubernetes
Webinar: Code Faster on KubernetesWebinar: Code Faster on Kubernetes
Webinar: Code Faster on Kubernetes
 
Testing fácil con Docker: Gestiona dependencias y unifica entornos
Testing fácil con Docker: Gestiona dependencias y unifica entornosTesting fácil con Docker: Gestiona dependencias y unifica entornos
Testing fácil con Docker: Gestiona dependencias y unifica entornos
 
Jenkins Shared Libraries Workshop
Jenkins Shared Libraries WorkshopJenkins Shared Libraries Workshop
Jenkins Shared Libraries Workshop
 
Intro to XPages for Administrators (DanNotes, November 28, 2012)
Intro to XPages for Administrators (DanNotes, November 28, 2012)Intro to XPages for Administrators (DanNotes, November 28, 2012)
Intro to XPages for Administrators (DanNotes, November 28, 2012)
 
Puppetize PDX 2019 - Automated Patching with Bolt by Nick Maludy
Puppetize PDX 2019 - Automated Patching with Bolt by Nick MaludyPuppetize PDX 2019 - Automated Patching with Bolt by Nick Maludy
Puppetize PDX 2019 - Automated Patching with Bolt by Nick Maludy
 
Java Deserialization Vulnerabilities - The Forgotten Bug Class (DeepSec Edition)
Java Deserialization Vulnerabilities - The Forgotten Bug Class (DeepSec Edition)Java Deserialization Vulnerabilities - The Forgotten Bug Class (DeepSec Edition)
Java Deserialization Vulnerabilities - The Forgotten Bug Class (DeepSec Edition)
 
2017 DevSecCon ZAP Scripting Workshop
2017 DevSecCon ZAP Scripting Workshop2017 DevSecCon ZAP Scripting Workshop
2017 DevSecCon ZAP Scripting Workshop
 
Source Control with Domino Designer 8.5.3 and Git (DanNotes, November 28, 2012)
Source Control with Domino Designer 8.5.3 and Git (DanNotes, November 28, 2012)Source Control with Domino Designer 8.5.3 and Git (DanNotes, November 28, 2012)
Source Control with Domino Designer 8.5.3 and Git (DanNotes, November 28, 2012)
 
Mastering IntelliTrace in Development and Production
Mastering IntelliTrace in Development and ProductionMastering IntelliTrace in Development and Production
Mastering IntelliTrace in Development and Production
 

Viewers also liked

Architecture for monitoring applications in Cloud
Architecture for monitoring applications in CloudArchitecture for monitoring applications in Cloud
Architecture for monitoring applications in Cloud
Onkar Kadam
 
ACM Applicative System Methodology 2016
ACM Applicative System Methodology 2016ACM Applicative System Methodology 2016
ACM Applicative System Methodology 2016
Brendan Gregg
 

Viewers also liked (20)

Automating AWS with Ansible
Automating AWS with AnsibleAutomating AWS with Ansible
Automating AWS with Ansible
 
Architecture for monitoring applications in Cloud
Architecture for monitoring applications in CloudArchitecture for monitoring applications in Cloud
Architecture for monitoring applications in Cloud
 
Server Monitoring 101
Server Monitoring 101Server Monitoring 101
Server Monitoring 101
 
Server Monitoring from the Cloud
Server Monitoring from the CloudServer Monitoring from the Cloud
Server Monitoring from the Cloud
 
Server Monitoring (Scaling while bootstrapped)
Server Monitoring  (Scaling while bootstrapped)Server Monitoring  (Scaling while bootstrapped)
Server Monitoring (Scaling while bootstrapped)
 
Backend server monitoring and alarm system (collectd, graphite, grafana, zabb...
Backend server monitoring and alarm system (collectd, graphite, grafana, zabb...Backend server monitoring and alarm system (collectd, graphite, grafana, zabb...
Backend server monitoring and alarm system (collectd, graphite, grafana, zabb...
 
Micro services infrastructure with AWS and Ansible
Micro services infrastructure with AWS and AnsibleMicro services infrastructure with AWS and Ansible
Micro services infrastructure with AWS and Ansible
 
InterConnect2016 Monitoring Nodejs
InterConnect2016 Monitoring NodejsInterConnect2016 Monitoring Nodejs
InterConnect2016 Monitoring Nodejs
 
Stop the Guessing: Performance Methodologies for Production Systems
Stop the Guessing: Performance Methodologies for Production SystemsStop the Guessing: Performance Methodologies for Production Systems
Stop the Guessing: Performance Methodologies for Production Systems
 
ACM Applicative System Methodology 2016
ACM Applicative System Methodology 2016ACM Applicative System Methodology 2016
ACM Applicative System Methodology 2016
 
Nodejs intro
Nodejs introNodejs intro
Nodejs intro
 
Monitoring the #DevOps way
Monitoring the #DevOps wayMonitoring the #DevOps way
Monitoring the #DevOps way
 
Automation with Ansible and Containers
Automation with Ansible and ContainersAutomation with Ansible and Containers
Automation with Ansible and Containers
 
SREcon 2016 Performance Checklists for SREs
SREcon 2016 Performance Checklists for SREsSREcon 2016 Performance Checklists for SREs
SREcon 2016 Performance Checklists for SREs
 
NGINX High-performance Caching
NGINX High-performance CachingNGINX High-performance Caching
NGINX High-performance Caching
 
Andrew Nelson - Zabbix and SNMP on Linux
Andrew Nelson - Zabbix and SNMP on LinuxAndrew Nelson - Zabbix and SNMP on Linux
Andrew Nelson - Zabbix and SNMP on Linux
 
Netflix: From Clouds to Roots
Netflix: From Clouds to RootsNetflix: From Clouds to Roots
Netflix: From Clouds to Roots
 
Blazing Performance with Flame Graphs
Blazing Performance with Flame GraphsBlazing Performance with Flame Graphs
Blazing Performance with Flame Graphs
 
Performance Tuning EC2 Instances
Performance Tuning EC2 InstancesPerformance Tuning EC2 Instances
Performance Tuning EC2 Instances
 
Linux BPF Superpowers
Linux BPF SuperpowersLinux BPF Superpowers
Linux BPF Superpowers
 

Similar to A Taste of Monitoring and Post Mortem Debugging with Node

DevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and ProjectsDevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and Projects
Fedir RYKHTIK
 
Threat Modeling the CI/CD Pipeline to Improve Software Supply Chain Security ...
Threat Modeling the CI/CD Pipeline to Improve Software Supply Chain Security ...Threat Modeling the CI/CD Pipeline to Improve Software Supply Chain Security ...
Threat Modeling the CI/CD Pipeline to Improve Software Supply Chain Security ...
Denim Group
 

Similar to A Taste of Monitoring and Post Mortem Debugging with Node (20)

TechGIG_Memory leaks in_java_webnair_26th_july_2012
TechGIG_Memory leaks in_java_webnair_26th_july_2012TechGIG_Memory leaks in_java_webnair_26th_july_2012
TechGIG_Memory leaks in_java_webnair_26th_july_2012
 
JDK Tools For Performance Diagnostics
JDK Tools For Performance DiagnosticsJDK Tools For Performance Diagnostics
JDK Tools For Performance Diagnostics
 
Reproducibility in artificial intelligence
Reproducibility in artificial intelligenceReproducibility in artificial intelligence
Reproducibility in artificial intelligence
 
DockerCon Europe 2018 Monitoring & Logging Workshop
DockerCon Europe 2018 Monitoring & Logging WorkshopDockerCon Europe 2018 Monitoring & Logging Workshop
DockerCon Europe 2018 Monitoring & Logging Workshop
 
Impact2014: Introduction to the IBM Java Tools
Impact2014: Introduction to the IBM Java ToolsImpact2014: Introduction to the IBM Java Tools
Impact2014: Introduction to the IBM Java Tools
 
Testing - How Vital and How Easy to use
Testing - How Vital and How Easy to useTesting - How Vital and How Easy to use
Testing - How Vital and How Easy to use
 
WebSphere Technical University: Introduction to the Java Diagnostic Tools
WebSphere Technical University: Introduction to the Java Diagnostic ToolsWebSphere Technical University: Introduction to the Java Diagnostic Tools
WebSphere Technical University: Introduction to the Java Diagnostic Tools
 
PyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web ApplicationsPyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web Applications
 
Visual Studio Profiler
Visual Studio ProfilerVisual Studio Profiler
Visual Studio Profiler
 
Guider: An Integrated Runtime Performance Analyzer on AGL
Guider: An Integrated Runtime Performance Analyzer on AGLGuider: An Integrated Runtime Performance Analyzer on AGL
Guider: An Integrated Runtime Performance Analyzer on AGL
 
Nginx conference 2015
Nginx conference 2015Nginx conference 2015
Nginx conference 2015
 
DevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and ProjectsDevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and Projects
 
Threat Modeling the CI/CD Pipeline to Improve Software Supply Chain Security ...
Threat Modeling the CI/CD Pipeline to Improve Software Supply Chain Security ...Threat Modeling the CI/CD Pipeline to Improve Software Supply Chain Security ...
Threat Modeling the CI/CD Pipeline to Improve Software Supply Chain Security ...
 
POD-Diagnosis: Error Detection and Diagnosis of Sporadic Operations on Cloud ...
POD-Diagnosis: Error Detection and Diagnosis of Sporadic Operations on Cloud ...POD-Diagnosis: Error Detection and Diagnosis of Sporadic Operations on Cloud ...
POD-Diagnosis: Error Detection and Diagnosis of Sporadic Operations on Cloud ...
 
DockerCon EU 2015: Monitoring Docker
DockerCon EU 2015: Monitoring DockerDockerCon EU 2015: Monitoring Docker
DockerCon EU 2015: Monitoring Docker
 
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit EuropeAutomation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
 
DevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
DevOpsGuys - DevOps Automation - The Good, The Bad and The UglyDevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
DevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
 
MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)
 
Synapseindia android middleware
Synapseindia android middlewareSynapseindia android middleware
Synapseindia android middleware
 
Tool up your lamp stack
Tool up your lamp stackTool up your lamp stack
Tool up your lamp stack
 

More from ibmwebspheresoftware

Don't Miss a Thing at IBM InterConnect 2015
Don't Miss a Thing at IBM InterConnect 2015Don't Miss a Thing at IBM InterConnect 2015
Don't Miss a Thing at IBM InterConnect 2015
ibmwebspheresoftware
 

More from ibmwebspheresoftware (6)

JavaOne 2015 Keynote Presentation
JavaOne 2015 Keynote PresentationJavaOne 2015 Keynote Presentation
JavaOne 2015 Keynote Presentation
 
NodeConf EU 2015 Keynote
NodeConf EU 2015 Keynote NodeConf EU 2015 Keynote
NodeConf EU 2015 Keynote
 
Node on Guard!
Node on Guard! Node on Guard!
Node on Guard!
 
A Node.js Developer's Guide to Bluemix
A Node.js Developer's Guide to BluemixA Node.js Developer's Guide to Bluemix
A Node.js Developer's Guide to Bluemix
 
Don't Miss a Thing at IBM InterConnect 2015
Don't Miss a Thing at IBM InterConnect 2015Don't Miss a Thing at IBM InterConnect 2015
Don't Miss a Thing at IBM InterConnect 2015
 
IBM WebSphere Application Foundation Sessions at IBM InterConnect 2015
IBM WebSphere Application Foundation Sessions at IBM InterConnect 2015IBM WebSphere Application Foundation Sessions at IBM InterConnect 2015
IBM WebSphere Application Foundation Sessions at IBM InterConnect 2015
 

Recently uploaded

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Victor Rentea
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Victor Rentea
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
FIDO Alliance
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 

Recently uploaded (20)

Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering Developers
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps Productivity
 
Simplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptxSimplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptx
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 

A Taste of Monitoring and Post Mortem Debugging with Node

  • 1. § Michael Dawson/Michael Tunnicliffe § September 2015 A Taste of Monitoring and Post Mortem Debugging with Node A Workshop
  • 2. 2 To talk about: Application reliability and performance ● Common problem scenarios ● Monitoring ● Problem diagnosis ● Tailoring tooling to your application Try it for yourself with prepared hands-on activities To share and learn from each others experiences Why are we here ?
  • 3. 3 • Meet and Greet • Runtime Monitoring • Concepts and Tools • Hands on - Monitor an App • GC Visualisation • Concepts and Tools • Hands on – visualize gc activity • Post Mortem Debugging • Concepts and Tools • Hands on with a heap dump • Hands on with a core dump • Experience/Gaps Brainstorm • Monitoring – Instrument a Module Heap usage/Object layout Initial Problem Determination Dig into GC behaviour (if appropriate) Execution State/Native Memory Retrospective Improve your tools Modelled on typical workflow Agenda
  • 4. 4 About Michael Dawson Loves the web and building software Senior Software Developer @ IBM 22 years development experience 10 years in runtime development Contact me: Michael_dawson@ca.ibm.com https://www.linkedin.com/pub/michael-dawson/2/128/605 Twitter: @mhdawson1
  • 5. 5 About Mike Tunnicliffe Bug detective and low level systems programmer Software Developer @ IBM 15 years development experience 12 years in runtime development Contact me: mike.tunnicliffe@uk.ibm.com @fierydrake
  • 6. 6 • Interest in Monitoring/Post mortem debugging ? • What you hope to get out of the workshop today ? About You (workshop participants) Used Node Before ? Role (Dev/QA/Ops) ? Apps in Production ? Production Monitoring ? Background Memory Usage ? GC activity Throughput Reponse Time Something else ? Metrics – what do you track High Memory Memory Exhaustion Memory Leaks Footprint Performance Crash Unhandled Exceptions Problems – what have you seen
  • 7. 7 • USB drive provided for easy installation • IBM is not a distributor • While freely available, licences apply to some components ● Licences are included on USB drive and in install packages ● By copying from USB drive you are accepting licence terms in USB drive as licensee • Printed copy also available • Alternatively follow full download/install instructions. • Install Instructions ● Pre-Install/Instructions.txt ● Self-Install/Instructions.txt Workshop materials
  • 8. 8 • Runtime Monitoring • GC Visualization • Post Mortem Debugging • Experience/Gaps Brainstorm • Monitoring – Instrument a Module Heap usage/Object layout Initial Problem Determination Dig into GC behaviour Execution State/Native Memory Retrospective Improve your tools Modelled on typical workflow Runtime Monitoring
  • 9. 9 To understand the “real” characteristics of the application • In production – health check under real load • During development – test under artificial load Why do runtime monitoring?
  • 10. 10 Data about the application and the environment on which it runs, typically: • Machine and operating system • Node.js and dependencies (eg V8) • Application and dependencies (eg http, mongodb) What will it give you?
  • 11. 11 Typically production monitoring consists of: • Data sources • Data store aggregator • Web-based visualizer How it works Browser Monitoring Data HTT P LoadBalancer
  • 13. 13 Setup • Run startelk • Run startapps • Open Kibana in a browser http://localhost:5601/ • Open the dashboard Try to identify any potential problems with the monitored applications Monitoring – Hands on – 20 minutes
  • 14. 14 Data Sources (open source) • ElasticSearch / Kibana (ELK) ● bunyan-child, hapi-good-logstash, winston-kibana, log4js-elasticsearch, black-pearl, appmetrics-elk, etc • StatsD / Graphite ● Express-metrics, connect-statsd, mongodb-datadog-statsd, node-statsd, statsd-profiler, node-statsd-instrument, appmetrics-statsd, etc. • Munin ● nodejs-munin, munin-plugin, etc • Custom: ● monitor, appmetrics, etc
  • 15. 15 Node Application Metrics ('appmetrics') • https://www.npmjs.com/package/appmetrics • https://github.com/RuntimeTools/appmetrics • Open Source under Apache 2.0 License* • Provides: ● producer and consumer APIs ● monkey-patching framework and “dropins” ● collection of connectors to existing monitoring stacks *has a (currently) proprietary dependency
  • 16. 16 appmetrics Data Collection Data Type Event Name Data Provided Application Data HTTP request/response 'http' Time, Method, URL, Duration MySQL database queries 'mysql' Time, SQL, Duration MongoDB database queries 'mongo' Time, Query, Duration Request tracking 'request' Time, Event Tree, Duration JavaScript Runtime V8 Garbage Collection 'gc' Time, Type, Used, Size, Duration V8 Profiler 'profiling' Time, Function Array, Samples System Resources CPU Usage 'cpu' Time, Process, System Memory Usage 'memory' Time, Process (Physical, Reserved, Virtual), System (Physical, Total)
  • 18. 18 Development Monitoring with 'appmetrics' Health Center for Eclipse appmetrics-dash
  • 19. 19 Appmetrics and ElasticSearch/Kibana (ELK) var appmetrics = require('appmetrics-elk').monitor(); var http = require('http'); var server = http.createServer(function handler(req, res) { … }).listen(process.env.PORT || 3000); console.log('App listening on port 3000'); }); ES ES ES ES client.bulk() visualize
  • 20. 20 'appmetrics' and Health Center for Eclipse • Free from Eclipse Marketplace • http://marketplace.eclipse.org/content/ibm-monitoring-and-diagnostic-tools-health- • Supports Node and Java • Uses MQTT for the connection Pub/Sub Broker Topic namespace: /ibm/healthcenter /<namespace>/<agent_id>/<datatype> Node.js Application Liberty Java Application MQTT MQTT MQTT MQTT MQTT Broker, eg. mosca (npm install -g mosca)
  • 21. 21 Health Center Environment Hardware and Operating System Configuration Process environment and configuration Highlights incorrect or non-standard configurations Memory Utilization Detect native memory leaks in application Determine if external forces are using more memory View components using the most native memory CPU Utilization Visualizes process CPU usage over time Visualizes system CPU usage over time
  • 22. 22 Health Center Garbage Collection Visualizes heap usage and GC pause times Identifies memory leaks Suggests command-line and tuning parameters Function Profiling Function profiling shows application activity Identifies the hottest methods in an application Full call stacks to identify where functions are being called from and what functions they call
  • 23. 23 Monitoring – Hands on – 20 minutes • Run sample app (gcload.bat/gcload.sh) and connect with Health Center • Look at environment information • Look through other available panels • Look at gc activity so see periodic allocation load • Run sample app with heavy cpu usage (cpuload.bat/cpuload.sh) • Enable profiling • Look at profile, map back to application source • Run custom program that consumes data. • Edit gcload.js to un-comment line that create monitor instance • Re-run and see output on console.
  • 24. 24 • Runtime Monitoring • GC Visualisation • Post Mortem Debugging • Experience/Gaps Brainstorm • Monitoring – Instrument a Module Heap usage/Object layout Initial Problem Determination Dig into GC behaviour Execution State/Native Memory Retrospective Improve your tools Modelled on typical workflow Runtime Monitoring
  • 25. 25 GC Visualization • Visualisation of gc data long known to be important • There are many existing gc visualizers (mostly Java) • • • • • • Javascript ● https://github.com/adobe-research/GCview ● http://marketplace.eclipse.org/content/ibm-monitoring-and-diagnostic-tools-garbage We’ll use GCMV since we are most familiar with it
  • 26. 26 GCMV • Free from Eclipse Marketplace • http://marketplace.eclipse.org/content/ibm-monitoring-and-diagnostic-tools- garbage-collection-and-memory-visualizer-gcmv • Supports both Node and Java
  • 27. 27 GC Visualization – Hands on (15mins) • Run sample app with memory leak with gc options enabled (gcloadWithVerbose) • Observe there is a lot of information • Run again sending to file • Open up resulting file in GCMV
  • 28. 28 • Runtime Monitoring • GC Visualisation • Post Mortem Debugging • Experience/Gaps Brainstorm • Monitoring – Instrument a Module Heap usage/Object layout Initial Problem Determination Dig into GC behaviour Execution State/Native Memory Retrospective Improve your tools Post Mortem Debugging
  • 29. 29 • 2 Common artifacts • Heap Dumps • Core Dumps • No standards yet but workgroup: • https://github.com/nodejs/post-mortem • Heap dumps provide insight into Heap memory usage • Core Dumps can provide insight into: • Heap memory usage • Runtime State • What was running at the time of a crash – native and JS stacks • Values in native memory • Native memory usage Post Mortem Debugging
  • 30. 30 • Run Sample OOM program • Generate Heapdump on OOM • Load OOM into Chrome developer tools • Show how to find what is keeps memory alive • Participants explore Chrome Developer Tools – Hands on – 10 minutes
  • 31. 31 • Free from Eclipse Marketplace • http://marketplace.eclipse.org/content/ibm-monitoring-and-diagnostic-tools-interac • Multi-runtime → Supports Node and Java • Currently requires dumps from IBM SDK for Node.js • http://www.ibm.com/developerworks/web/nodesdk/ Interactive Diagnostic Data Explorer (IDDE)
  • 32. 32 • Open pre-provided core file • Use IDDE commands to show where crash occurred • Help to list all commands • Explore IDDE – Hands On – 10 minutes
  • 33. 33 • Runtime Monitoring • GC Visualisation • Post Mortem Debugging • Experience/Gaps Brainstorm • Monitoring – Instrument a Module Heap usage/Object layout Initial Problem Determination Dig into GC behaviour Execution State/Native Memory Retrospective Improve your tools Retrospective
  • 34. 34 • Experience from Participants (Open discussion) – Other tools – Key issues – Workflow • Gaps Brainstorm – Key gaps – Post mortem debugging – useful commands ? Other Experiences/Gaps Brainstorm (10-15 mins)
  • 35. 35 • Runtime Monitoring • GC Visualisation • Post Mortem Debugging • Experience/Gaps Brainstorm • Monitoring – Instrument a Module Heap usage/Object layout Initial Problem Determination Dig into GC behaviour Execution State/Native Memory Retrospective Improve your tools Improve
  • 36. 36 You may collect timing and event data from your application, or modules that you use. Wouldn't it be great if you could capture and view that in the same way as the other monitoring data? Custom monitoring
  • 37. 37 The appmetrics module provides: • A producer API for emitting custom data • A consumer API for receiving all data Using appmetrics APIs
  • 38. 38 Using the emit() and Event APIs var appmetrics = require('appmetrics'); var data = { time: Date.now(), mydata: 7 }; appmetrics.emit('myevent', data); var appmetrics = require('appmetrics'); var monitor = appmetrics.monitor(); monitor.on('myevent', function(data) { console.log('[myevent] ' + data.mydata); }); • emit() API is on the appmetrics object • Convention to have a “time” field – Allows it to be used in ElasticSearch • Events are on the monitored instance object – Allows for future remote monitoring
  • 39. 39 Instrument a simple piece of test code: • Emit a custom 'iteration' event with the following data: count time taken to generate the array time taken to serialize • Add a handler that logs the data Improve – Instrument an application (10 mins) // Crunch some numbers setInterval(function() { var object = []; var count = Math.random()*10000000; for (var i=0; i<count; i++) { object.push(i) } JSON.stringify(object); }, 1000);
  • 40. 40 You may not be able to modify the code you want to instrument • For example: a third-party or core module • You could monkey patch it Instrumenting modules
  • 41. 41 A simple monkey patching example var appmetrics = require('appmetrics'); var fs = require('fs'); var originalReadFileSync = fs.readFileSync; // Instrument readFileSync fs.readFileSync = function() { var startTime = Date.now(); originalReadFileSync.apply(fs, arguments); var endTime = Date.now(); appmetrics.emit('loadfile', { 'duration': endTime - startTime }); }; // Set up monitoring listeners appmetrics.monitor().on('loadfile', function(e) { console.log('File load duration=' + e['duration'] + 'ms'); }); // Application for (var i=0; i<100; i++) { fs.readFileSync('bigfile.dat'); } Store function to patch Patch function by replacing Custom “visualizer”
  • 42. 42 Asynchronous monkey patching example var appmetrics = require('appmetrics'); var fs = require('fs'); var originalReadFile = fs.readFile; // Instrument readFileSync fs.readFile = function() { var startTime = Date.now(); var callbackIndex = (typeof(arguments[1]) === 'function') ? 1 : (typeof(arguments[2]) === 'function') ? 2 : -1; if (callbackIndex == -1) return; var originalCallback = arguments[callbackIndex]; var wrap = function() { appmetrics.emit('loadfile', { 'duration': Date.now() - startTime }); originalCallback.call(arguments); }; arguments[callbackIndex] = wrap; originalReadFile.apply(fs, arguments); }; // Set up monitoring listeners appmetrics.monitor().on('loadfile', console.log); // Application for (var i=0; i<100; i++) { fs.readFile('bigfile.dat', function(err, data) { console.log('readFile completed'); }); } Store function to patch Patch function by replacing Custom “visualizer”
  • 43. 43 Deferred monkey patching example var module = require('module'); var moduleLoad = module._load; module._load = function(mod, parent, isMain) { var loadedModule = moduleLoad.apply(this, arguments); if (mod === 'http') { var addListener = loadedModule.Server.prototype.addListener; loadedModule.Server.prototype.addListener = function(type, listener) { if (type === 'request' && typeof listener === 'function'){ var oldListener = listener; listener = function(req, res){ if (typeof res.startTime === 'undefined') { res.startTime = new Date().getTime(); res.once('finish', function() { res.endTime = new Date().getTime(); var dur = res.endTime - res.startTime; console.log('['+ res.startTime + '] ' + req.method + ' : ' + req.url + ' ('+dur+')'); }); }; oldListener.call(this, req, res); }; return addListener.call(this, type, listener); } else { return addListener.apply(this, arguments); } }; }; return loadedModule; };
  • 44. 44 The appmetrics module provides: • Drop-in “probes” Self-contained facility to deploy patches in appmetrics • An aspect framework for use by probes Simplify complex method replacement scenarios Making patching easier
  • 45. 45 Using the Probes framework Probe scripts are automatically loaded from: node_modules/appmetrics/probes Template probe: var Probe = require('../lib/probe.js'); var aspect = require('../lib/aspect.js'); var util = require('util'); function MyExampleProbe() { Probe.call(this, 'name_of_module_to_patch'); } util.inherits(MyExampleProbe, Probe); MyExampleProbe.prototype.attach = function(name, target, am) { if (target.__probeAttached__) return target; target.__probeAttached__ = true; // Code to patch and emit event (can use aspect) return target; }; module.exports = MyExampleProbe;
  • 46. 46 Aspect API aspect.before(obj, methods, func); aspect.after(obj, methods, func); aspect.around(obj, methods, func1, func2); aspect.aroundCallback(obj, methods, func1, func2); • Provides APIs to locate and patch functions – before, after and around supported HTTPProbe.prototype.attach = function(name, target, am) { if (target.__probeAttached__) return target; target.__probeAttached__ = true; aspect.before(target.Server.prototype, ['on', 'addListener'], function(obj, args) { if (args[0] !== 'request') return; aspect.aroundCallback(args, function(obj, args) { var request = args[0], response = args[1]; // args for callback are [request, response] response.startTime = Date.now(); aspect.after(response, 'finish', function(obj, args, ret) { var dur = Date.now() - response.startTime; console.log('['+ response.startTime + '] ' + request.method + ' : ' + request.url + ' ('+dur+')'); }); }); }); return target; }; • Reduces the complex monkey patch example to...
  • 47. 47 Using the Aspects framework Aspects framework makes it easy to patch specific functions in the module Simple use: The “around” function is basically “before” and “after” combined MyExampleProbe.prototype.attach = function(name, target, am) { if (target.__probeAttached__) return target; target.__probeAttached__ = true; var start; aspect.around(target, 'readFileSync', function(obj, args) { start = Date.now(); }, function(obj, args, returnObject) { am.emit('file', {time: start, method: 'readFileSync', file: args[0], duration: Date.now() - start}); return returnObject; } ); return target; };
  • 48. 48 Using the Aspects framework This, more complex, scenario can happen when you want to patch async functions that accept callback arguments The “aroundCallback” function will search a given list of objects and patch the first function it finds (good for patching async handlers) MyExampleProbe.prototype.attach = function(name, target, am) { if (target.__probeAttached__) return target; target.__probeAttached__ = true; // 1. Run some code after moduleToPatch.functionToPatch() to... // 2. - run some code before a function on its return value (returnObject.functionOnReturnObjectToPatch()) to... // 3. - search that function's arguments for a callback and run some code before that callback aspect.after(target, 'functionToPatch', function(target, args, returnObject) { aspect.before(returnObject, 'functionOnReturnObjectToPatch', function(target, methodArgs) { aspect.aroundCallback(methodArgs, function(target, args){ // code }); } ); return returnObject; }); return target; }; 1 2 3
  • 49. 49 Instrument fs.readFile() by creating a probe Start with the template code → ● Set the name to 'fs' ● Replace the comment with a call into the aspect framework to patch readFile() and associated callback ● Drop into the probes directory inside appmetrics Use this simple script to test → Improve – Instrument a module (10 mins) var fs = require('fs'); setInterval(function() { fs.readFile('bigfile.dat', function(err, data) { console.log(data.length); }); }, 1000); var Probe = require('../lib/probe.js'); var aspect = require('../lib/aspect.js'); var util = require('util'); function MyExampleProbe() { Probe.call(this, 'name_of_module_to_patch'); } util.inherits(MyExampleProbe, Probe); MyExampleProbe.prototype.attach = function(name, target, am) { if (target.__probeAttached__) return target; target.__probeAttached__ = true; // Code to patch and emit event (can use aspect) return target; }; module.exports = MyExampleProbe;
  • 50. 50 • Runtime Monitoring • GC Visualisation • Post Mortem Debugging • Experience/Gaps Brainstorm • Monitoring – Instrument a Module Heap usage/Object layout Initial Problem Determination Dig into GC behaviour Execution State/Native Memory Retrospective Improve your tools Summary and Wrap up
  • 51. 51 Copyrights and Trademarks © IBM Corporation 2015. All Rights Reserved IBM, the IBM logo, ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml Node.js is an official trademark of Joyent. IBM SDK for Node.js is not formally related to or endorsed by the official Joyent Node.js open source or commercial project. Java, JavaScript and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. "All other servicemarks and trademarks are the property of their respective owner."