Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Real-Life Node.js
Troubleshooting
Damian Schenkelman, Auth0
4 years
Usual suspects
Memory Leaks
–Sebastian Peyrott (a co-worker)
“The main cause for leaks in garbage collected
languages are unwanted references”
Memory Model
https://developers.google.com/web/tools/chrome-
devtools/memory-problems/memory-101
A
B
C
F
E
D
GGC
H I
b.d	=	null
A
B
C
F
E
D
GGC
H I
D not referenced
A
B
C
F
E
D
GGC
H I
D is GCed
A
B
C
F
E GGC
H I
Sawtooth
Memory Leak
A
B
C
F
E
D
G
Things we don’t need and are
taking up a LOT of space
GC
Memory Leak
A
B
C
F
E
D
G
Things we don’t need and are
taking up a LOT of space
GC
More things we don’t need
Crashes
Memory leak busting
Take Control
--max-old-space-size
is your friend
Drain connections
Restarts
Heap Snapshot
v8-profiler
https://github.com/node-inspector/v8-profiler
var	profiler	=	require('v8-profiler');	
var	snapshot	=	profiler.ta...
Demo
Mental Picture
{		
		'kinesis.us-west-1.amazonaws.co:443'	:						
}
{		
		sockets	:		
}
[		
		…,	…,	…,	…,	…,	…,	…,	…,…,	TL...
Not Easy
https://github.com/auth0/kinesis-writable/pull/6/files
forever-agent
https://github.com/request/forever-agent/blob/
ece900a6e8dfac734186db3080c00875c0300450/index.js#L70
if	(thi...
Fixed
CPU bottlenecks
Auth Service
DB
Scenario
Client
Expected response times
Actual response times
Flame graphs
a()
b()
c()
d()
h()
i()
e() f()
g()
Flame graphs
Demo
The problem
Scale
Faster Hash
Caching
Scale up
Multiple Auth Services
LB
Auth Service
Auth Service
Auth Service
Auth Service
BaaS
https://github.com/auth0/node-baas
//	compare	bcrypt	for	req.password	to	db	hash	
baas.compare(req.body.password,	use...
Auto Scaling
Auth Service LB
BaaS
BaaS
BaaS
BaaS
Cost comparison
Price / (1M req)
#req per sec /
vCPU
t2-micro $0.36 10.00
t2-medium $0.76 9.50
c4-large $1.53 10.00
c3-8xl...
Fail gracefully
Cat picture
Memory Leaks
• Memory terminology: https://developers.google.com/web/tools/
chrome-devtools/memory-problems/memory-101
• M...
Memory Leaks (2)
• Connection draining: https://cloud.google.com/
compute/docs/load-balancing/enabling-connection-
drainin...
Memory Leaks (3)
• AWS SDK issues: https://github.com/aws/aws-sdk-js/issues/329,
https://github.com/aws/aws-sdk-js/issues/...
Performance
• http://security.stackexchange.com/a/83382
• http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
• htt...
Thanks
https://github.com/dschenkelman/node-troubleshooting
@dschenkelman
npm	i	dschenkelman
Upcoming SlideShare
Loading in …5
×

Real-Life Node.js Troubleshooting - Damian Schenkelman, Auth0

524 views

Published on

When building a large enough set of services using node.js, there will be a point when you find that your application is suffering from performance or memory issues. When this happens, you have to roll up your sleeves, get your tools and start digging. This talk explains how you can use tools such as ab, flame graphs, heap snapshots and Chrome's memory inspector to find the cause of these. We will go over two real life issues, a CPU bottleneck and a memory leak, we found while building our services at Auth0, and also explain how we fixed them.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Real-Life Node.js Troubleshooting - Damian Schenkelman, Auth0

  1. 1. Real-Life Node.js Troubleshooting Damian Schenkelman, Auth0
  2. 2. 4 years
  3. 3. Usual suspects
  4. 4. Memory Leaks
  5. 5. –Sebastian Peyrott (a co-worker) “The main cause for leaks in garbage collected languages are unwanted references”
  6. 6. Memory Model https://developers.google.com/web/tools/chrome- devtools/memory-problems/memory-101 A B C F E D GGC H I
  7. 7. b.d = null A B C F E D GGC H I
  8. 8. D not referenced A B C F E D GGC H I
  9. 9. D is GCed A B C F E GGC H I
  10. 10. Sawtooth
  11. 11. Memory Leak A B C F E D G Things we don’t need and are taking up a LOT of space GC
  12. 12. Memory Leak A B C F E D G Things we don’t need and are taking up a LOT of space GC More things we don’t need
  13. 13. Crashes
  14. 14. Memory leak busting
  15. 15. Take Control
  16. 16. --max-old-space-size is your friend
  17. 17. Drain connections
  18. 18. Restarts
  19. 19. Heap Snapshot
  20. 20. v8-profiler https://github.com/node-inspector/v8-profiler var profiler = require('v8-profiler'); var snapshot = profiler.takeSnapshot(); snapshot.export(function(error, result) { fs.writeFileSync('snapshot.json', result); snapshot.delete(); });
  21. 21. Demo
  22. 22. Mental Picture { 'kinesis.us-west-1.amazonaws.co:443' : } { sockets : } [ …, …, …, …, …, …, …, …,…, TLSSocket ] Something related to logging…
  23. 23. Not Easy https://github.com/auth0/kinesis-writable/pull/6/files
  24. 24. forever-agent https://github.com/request/forever-agent/blob/ ece900a6e8dfac734186db3080c00875c0300450/index.js#L70 if (this.freeSockets[name] && this.freeSockets[name].length > 0 && !req.useChunkedEncodingByDefault) { var idleSocket = this.freeSockets[name].pop() idleSocket.removeListener('error', idleSocket._onIdleError) delete idleSocket._onIdleError req._reusedSocket = true req.onSocket(idleSocket) } else { this.addRequestNoreuse(req, host, port) }
  25. 25. Fixed
  26. 26. CPU bottlenecks
  27. 27. Auth Service DB Scenario Client
  28. 28. Expected response times
  29. 29. Actual response times
  30. 30. Flame graphs a() b() c() d() h() i() e() f() g()
  31. 31. Flame graphs
  32. 32. Demo
  33. 33. The problem
  34. 34. Scale
  35. 35. Faster Hash
  36. 36. Caching
  37. 37. Scale up
  38. 38. Multiple Auth Services LB Auth Service Auth Service Auth Service Auth Service
  39. 39. BaaS https://github.com/auth0/node-baas // compare bcrypt for req.password to db hash baas.compare(req.body.password, user.passwordHash, (err, success) => { res.send(err || !success ? 401 : 200); });
  40. 40. Auto Scaling Auth Service LB BaaS BaaS BaaS BaaS
  41. 41. Cost comparison Price / (1M req) #req per sec / vCPU t2-micro $0.36 10.00 t2-medium $0.76 9.50 c4-large $1.53 10.00 c3-8xlarge $1.64 8.88
  42. 42. Fail gracefully
  43. 43. Cat picture
  44. 44. Memory Leaks • Memory terminology: https://developers.google.com/web/tools/ chrome-devtools/memory-problems/memory-101 • Memory leaks in JS: https://auth0.com/blog/four-types-of-leaks- in-your-javascript-code-and-how-to-get-rid-of-them/ • Roots: https://stackoverflow.com/questions/9748358/when- does-the-js-engine-create-a-garbage-collection-root • Increase heap size: https://twitter.com/tjholowaychuk/status/ 480753206301966336 • v8 GC logs speaking to you: https://github.com/joyeecheung/ v8-gc-talk
  45. 45. Memory Leaks (2) • Connection draining: https://cloud.google.com/ compute/docs/load-balancing/enabling-connection- draining • master-process: https://github.com/jfromaniello/ master-process • Express connection draining: https://github.com/ expressjs/express/issues/1366 • v8-profiler: https://github.com/node-inspector/v8- profiler
  46. 46. Memory Leaks (3) • AWS SDK issues: https://github.com/aws/aws-sdk-js/issues/329, https://github.com/aws/aws-sdk-js/issues/855 • Kinesis event fix: https://github.com/auth0/kinesis-writable/pull/6/files • Forever agent: https://github.com/request/forever-agent/blob/ ece900a6e8dfac734186db3080c00875c0300450/index.js#L70 • Memory profiling for mere mortals: http://thlorenz.com/talks/memory- profiling/book/memory_profiling_for_mere_mortals_/ memory_profiling_for_mere_mortals__0.html • @thlorenz notes on mem profiling: https://github.com/thlorenz/v8- perf/blob/master/memory-profiling.md
  47. 47. Performance • http://security.stackexchange.com/a/83382 • http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html • https://github.com/davidmarkclements/0x • https://github.com/thlorenz/v8-perf/issues/4 • https://github.com/auth0/node-baas • http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ US_SetUpASLBApp.html • https://gist.github.com/trevnorris/9616784 • https://github.com/dschenkelman/bcrypt-sample
  48. 48. Thanks https://github.com/dschenkelman/node-troubleshooting @dschenkelman npm i dschenkelman

×