When Node.js Goes Wrong: Debugging Node in Production
The event-oriented approach underlying Node.js enables significant concurrency using a deceptively simple programming model, which has been an important factor in Node's growing popularity for building large scale web services. But what happens when these programs go sideways? Even in the best cases, when such issues are fatal, developers have historically been left with just a stack trace. Subtler issues, including latency spikes (which are just as bad as correctness bugs in the real-time domain where Node is especially popular) and other buggy behavior often leave even fewer clues to aid understanding. In this talk, we will discuss the issues we encountered in debugging Node.js in production, focusing upon the seemingly intractable challenge of extracting runtime state from the black hole that is a modern JIT'd VM.
We will describe the tools we've developed for examining this state, which operate on running programs (via DTrace), as well as VM core dumps (via a postmortem debugger). Finally, we will describe several nasty bugs we encountered in our own production environment: we were unable to understand these using existing tools, but we successfully root-caused them using these new found abilities to introspect the JavaScript VM.
The .NET Garbage Collector (GC) is really cool. It helps providing our applications with virtually unlimited memory, so we can focus on writing code instead of manually freeing up memory. But how does .NET manage that memory? What are hidden allocations? Are strings evil? It still matters to understand when and where memory is allocated. In this talk, we’ll go over the base concepts of .NET memory management and explore how .NET helps us and how we can help .NET – making our apps better. Expect profiling, Intermediate Language (IL), ClrMD and more!
Performance has always been a major concern in software development and should not be taken lightly even when commodity computers have multicore CPUs and a few gigabytes of RAM. One of the most handy, simple tools for performance testing are microbenchmarks. Unfortunately, developing correct Java microbenchmarks is a complex task with many pitfalls on the way. This presentation is about the Do's and Don'ts of Java microbenchmarking and about what tools are out there to help with this tricky task.
Rainbow Over the Windows: More Colors Than You Could ExpectPeter Hlavaty
As time goes on operating systems keep evolving, like Microsoft Windows do, it ships new designs, features and codes from time to time. However sometimes it also ships more than bit of codes for complex subsystems residing in its kernel ... and at some future point it starts implementing new designs to prevent unnecessary access to it. However is it safe enough?
As we can see from security bulletins, win32k subsystem attracts lots of attention. It looks that with efforts of many security researchers who has dug into this area, finding bugs here shall becomes pretty tough and almost fruitless. But unfortunately this is not true, as win32k is backed up by very complex logic and large amount of code by nature..
We will present our point of view to Windows graphic subsystem, as well as schema of our fuzzing strategies. We will introduce some unusual areas of win32k, its extensions and how it can breaks even locked environments.
Part of our talk will be dedicated to CVE-2016-0176, the bug we used for this year's Pwn2Own Edge sandbox bypass, from its discovery to its exploitation techniques, which could serves as an example for universal DirectX escape which is independent of graphics vendors.
Security research over Windows #defcon chinaPeter Hlavaty
Past several years Microsoft Windows undergo lot of fundamental security changes. Where one can argue still imperfect and bound to tons of legacy issues, on the other hand those changes made important shifts in attacker perspective. From tightened sandboxing, restricting attack surface, introducing mitigations, applying virtualization up to stronger focus even on win32k. In our talk we will go trough those changes, how it affects us and how we tackle them from choosing targets, finding bugs up to exploitation primitives we are using. While also empathize that windows research is not only about sandbox, and there are many more interesting target to look for.
In the past few years, the bar for exploitation was raised highly, and in the current state of software security it is harder and harder to make successful exploitation on newest operating systems.
But as some systems continue to evolve and introduce new mitigations, the others just freeze a few years behind. In our talk we will focus on rooting Android by two racing conditions vulnerabilities. We will show the differences between level of exploitation needed, and how some mobile vendors are killing offered security features.
You didnt see it’s coming? "Dawn of hardened Windows Kernel" Peter Hlavaty
Past few years our team was focusing on different operating systems including Microsoft windows kernel. Honestly our first pwn at Windows kernel was not that challenging. Number of available targets with friendly environment for straightforward pwn, from user up to reliable kernel code execution.
However, step by step, security policies continue to evolve, and it becomes more troublesome to choose ideal attack surface from various sandboxes. In addition, what steps to follow for digging security holes is highly dependent upon the chosen target. In general, a few common strategies are available for researchers to choose: e.g choose “unknown” one which hasn’t been researched before; Select well fuzzed or well audited one, or research on kernel module internals to find “hidden” attack surfaces which are not explicitly interconnected. In the first part of the talk we introduce our methodology of selecting, alongside with cost of tricks around to choose seemingly banned targets, illustrated by notable examples.
After getting hands on potential bug available from targeted sandbox, it is time for Microsoft windows taking hardening efforts to put attacker into corner. Strong mitigations are being introduced more frequently than ever, with promising direction which cuts lots of attack surface off, and a several exploitation techniques being killed. We will show difficulties of developing universal exploitation techniques, and demonstrate needed technical level depending on code quality of target. We will examine how different it becomes with era of Redstone and following versions even with those techniques and good vulnerability in hand. How it changed attacker landscape and how it will (and will not) kill those techniques and applications. However will it really change the game or not?
The .NET Garbage Collector (GC) is really cool. It helps providing our applications with virtually unlimited memory, so we can focus on writing code instead of manually freeing up memory. But how does .NET manage that memory? What are hidden allocations? Are strings evil? It still matters to understand when and where memory is allocated. In this talk, we’ll go over the base concepts of .NET memory management and explore how .NET helps us and how we can help .NET – making our apps better. Expect profiling, Intermediate Language (IL), ClrMD and more!
Performance has always been a major concern in software development and should not be taken lightly even when commodity computers have multicore CPUs and a few gigabytes of RAM. One of the most handy, simple tools for performance testing are microbenchmarks. Unfortunately, developing correct Java microbenchmarks is a complex task with many pitfalls on the way. This presentation is about the Do's and Don'ts of Java microbenchmarking and about what tools are out there to help with this tricky task.
Rainbow Over the Windows: More Colors Than You Could ExpectPeter Hlavaty
As time goes on operating systems keep evolving, like Microsoft Windows do, it ships new designs, features and codes from time to time. However sometimes it also ships more than bit of codes for complex subsystems residing in its kernel ... and at some future point it starts implementing new designs to prevent unnecessary access to it. However is it safe enough?
As we can see from security bulletins, win32k subsystem attracts lots of attention. It looks that with efforts of many security researchers who has dug into this area, finding bugs here shall becomes pretty tough and almost fruitless. But unfortunately this is not true, as win32k is backed up by very complex logic and large amount of code by nature..
We will present our point of view to Windows graphic subsystem, as well as schema of our fuzzing strategies. We will introduce some unusual areas of win32k, its extensions and how it can breaks even locked environments.
Part of our talk will be dedicated to CVE-2016-0176, the bug we used for this year's Pwn2Own Edge sandbox bypass, from its discovery to its exploitation techniques, which could serves as an example for universal DirectX escape which is independent of graphics vendors.
Security research over Windows #defcon chinaPeter Hlavaty
Past several years Microsoft Windows undergo lot of fundamental security changes. Where one can argue still imperfect and bound to tons of legacy issues, on the other hand those changes made important shifts in attacker perspective. From tightened sandboxing, restricting attack surface, introducing mitigations, applying virtualization up to stronger focus even on win32k. In our talk we will go trough those changes, how it affects us and how we tackle them from choosing targets, finding bugs up to exploitation primitives we are using. While also empathize that windows research is not only about sandbox, and there are many more interesting target to look for.
In the past few years, the bar for exploitation was raised highly, and in the current state of software security it is harder and harder to make successful exploitation on newest operating systems.
But as some systems continue to evolve and introduce new mitigations, the others just freeze a few years behind. In our talk we will focus on rooting Android by two racing conditions vulnerabilities. We will show the differences between level of exploitation needed, and how some mobile vendors are killing offered security features.
You didnt see it’s coming? "Dawn of hardened Windows Kernel" Peter Hlavaty
Past few years our team was focusing on different operating systems including Microsoft windows kernel. Honestly our first pwn at Windows kernel was not that challenging. Number of available targets with friendly environment for straightforward pwn, from user up to reliable kernel code execution.
However, step by step, security policies continue to evolve, and it becomes more troublesome to choose ideal attack surface from various sandboxes. In addition, what steps to follow for digging security holes is highly dependent upon the chosen target. In general, a few common strategies are available for researchers to choose: e.g choose “unknown” one which hasn’t been researched before; Select well fuzzed or well audited one, or research on kernel module internals to find “hidden” attack surfaces which are not explicitly interconnected. In the first part of the talk we introduce our methodology of selecting, alongside with cost of tricks around to choose seemingly banned targets, illustrated by notable examples.
After getting hands on potential bug available from targeted sandbox, it is time for Microsoft windows taking hardening efforts to put attacker into corner. Strong mitigations are being introduced more frequently than ever, with promising direction which cuts lots of attack surface off, and a several exploitation techniques being killed. We will show difficulties of developing universal exploitation techniques, and demonstrate needed technical level depending on code quality of target. We will examine how different it becomes with era of Redstone and following versions even with those techniques and good vulnerability in hand. How it changed attacker landscape and how it will (and will not) kill those techniques and applications. However will it really change the game or not?
Java Colombo Meetup on 22nd March 2018
Speaker: Isuru Perera, Technical Lead at WSO2
Flame graphs are a visualization of profiled software and it was developed by Brendan Gregg, an industry expert in computing performance and cloud computing. Finding out why CPUs are busy is an important task when troubleshooting performance issues and we often use a sampling profiler to see which code-paths are hot. However, a profiler will dump a lot of data with thousands of lines and it is not easy to go through all data. With Flame Graphs, we can identify the most frequent code-paths quickly and accurately. Basically, a Flame Graph can simply visualize the stack traces output of a sampling profiler.
There are many ways to profile Java applications and Java Flight Recorder (JFR) is a really good tool to profile a Java application with a very low overhead. I will show how we can generate a Flame Graph from a Java Flight Recording using the JFR Flame Graph tool (https://github.com/chrishantha/jfr-flame-graph) I developed.
Since Flame Graphs can visualize any stack profiles, we can also use a Linux system profiler (perf) and create a Java Mixed-Mode Flame Graph, which will show how much CPU time is spent in Java methods, system libraries and the kernel. We can troubleshoot performance issues related to high CPU usage easily with a flame graph showing profile information from both system code paths and Java code paths. I will discuss how we can use the -XX:+PreserveFramePointer option in JDK and the perf system profiler to generate a Java Mixed-mode flame graph.
OWASP SD: Deserialize My Shorts: Or How I Learned To Start Worrying and Hate ...Christopher Frohoff
Object deserialization is an established but poorly understood attack vector in applications that is disturbingly prevalent across many languages, platforms, formats, and libraries.
In January 2015 at AppSec California, Chris Frohoff and Gabe Lawrence gave a talk on this topic, covering deserialization vulnerabilities across platforms, the many forms they take, and places they can be found. It covered, among other things, somewhat novel techniques using classes in commonly used libraries for attacking Java serialization that were subsequently released in the form of the ysoserial tool. Few people noticed until late 2015, when other researchers used these techniques/tools to exploit well known products such as Bamboo, WebLogic, WebSphere, ApacheMQ, and Jenkins, and then services such as PayPal. Since then, the topic has gotten some long-overdue attention and great work is being done by many to improve our understanding and developer awareness on the subject.
This talk will review the details of Java deserialization exploit techniques and mitigations, as well as report on some of the recent (and future) activity in this area.
http://www.meetup.com/Open-Web-Application-Security-Project-San-Diego-OWASP-SD/events/226242635/
Recently our team researched various ntos subsystem attack vectors, and one of the outputs we will present in our talk. DeathNote as our internal code name to this component, which resides in Microsoft Windows kernel, hiding behind different interfaces and exposed to user differently.
What can goes bad with it?
Basically two kinds of problems, one is syscall handling via direct user interaction. We will describe how to obtain basic understanding of what's going on, how it interacts with other components and what is its purpose. With those knowledge we will dig deeper how to make more complex fuzzing logic to cause enough chaos that will end up in unexpected behaviors in Windows kernel, and demonstrate some of them.
And as for second, as it hints from title, this module does bit of data parsing, so we will dive deep into internals, pointing out some available materials, and move on to reverse engineered structures and internal mechanism. We will show how some tricks can outcome with various results, and how structured approach can expose more problems than is expected.
Talk given by Pierre Ernst, Product Security Lead at Salesforce, at Hack Fest 2016 on November 2016
Pierre Ernst has 20 years of professional experience in building and breaking applications. His current focus is helping organisations improve their security posture by playing both offense and defense. In his spare time, he still enjoys finding high-value vulnerabilities and tries to make open source components more secure using his weapon of choice: code review. His favorite research topics include: weaponizing XML External Entity (XXE) attacks and XPath injections, finding novel ways of triggering hash table collisions and exploiting all sorts of deserialization technologies.
Fixing the Java Serialization mess
Deserializing untrusted input with Java has been known to be a risky proposition for at least 10 years. More recently, several vulnerabilities exploiting this flaw have been published. These deserialization vulnerabilities can be divided into 2 groups: endpoints allowing deserialization of arbitrary classes known to the application, or serialization “gadgets” allowing to weaponize malicious input for these endpoints. When it comes to fixing this class of vulnerabilities, it is hard to reach a consensus: some library maintainers consider that there is no point fixing the “gadgets” and that all application should simply stop accepting serialized input. Easier said than done…
While the root cause of the issue lies with a lenient Java API (not allowing to specify which class is to be deserialized), we need an immediate fix. This is why Pierre Ernst came up with the seminal “Look-ahead Java deserialization” concept in 2013.
During this talk, the current look-ahead implementation will be bypassed with a live demo, and a more robust mitigation will be presented.
Software Profiling: Understanding Java Performance and how to profile in JavaIsuru Perera
Guest lecture at University of Colombo School of Computing on 27th May 2017
Covers following topics:
Software Profiling
Measuring Performance
Java Garbage Collection
Sampling vs Instrumentation
Java Profilers. Java Flight Recorder
Java Just-in-Time (JIT) compilation
Flame Graphs
Linux Profiling
Inside the JVM - Follow the white rabbit! / Breizh JUGSylvain Wallez
Presentation given at the Rennes (FR) Java User Group in Feb 2019.
How do we go from your Java code to the CPU assembly that actually runs it? Using high level constructs has made us forget what happens behind the scenes, which is however key to write efficient code.
Starting from a few lines of Java, we explore the different layers that constribute to running your code: JRE, byte code, structure of the OpenJDK virtual machine, HotSpot, intrinsic methds, benchmarking.
An introductory presentation to these low-level concerns, based on the practical use case of optimizing 6 lines of code, so that hopefully you to want to explore further!
syzbot and the tale of million kernel bugsDmitry Vyukov
The root cause of most software exploits is bugs. Hardening, mitigations and containers are important, but they can't protect a system with thousands of bugs. In this presentation, Dmitry Vyukov will review the current [sad] situation with Linux kernel bugs and security implications based on their experience testing kernel for the past 3 years; overview a set of bug finding tools they are developing (syzbot, syzkaller, KASAN, KMSAN, KTSAN); and discuss problems and areas that require community help to improve the situation.
JetBrains Day Seoul - Exploring .NET’s memory management – a trip down memory...Maarten Balliauw
The .NET Garbage Collector (GC) is really cool. It helps providing our applications with virtually unlimited memory, so we can focus on writing code instead of manually freeing up memory. But how does .NET manage that memory? What are hidden allocations? Are strings evil? It still matters to understand when and where memory is allocated. In this talk, we’ll go over the base concepts of .NET memory management and explore how .NET helps us and how we can help .NET – making our apps better. Expect profiling, Intermediate Language (IL), ClrMD and more!
The .NET Garbage Collector (GC) is really cool. It helps providing our applications with virtually unlimited memory, so we can focus on writing code instead of manually freeing up memory. But how does .NET manage that memory? What are hidden allocations? Are strings evil? It still matters to understand when and where memory is allocated. In this talk, we’ll go over the base concepts of .NET memory management and explore how .NET helps us and how we can help .NET – making our apps better. Expect profiling, Intermediate Language (IL), ClrMD and more!
It is easy to monitor the performance of JVM if one knows how GC and Threads work in JVM. This presentation throws light on Collector types, HotSpot Collection Algorithms, Thread Monitoring, Method Profiling and Heap Profiling
XCon 2014 => http://xcon.xfocus.org/
In the past was quite common to exploit heap / pool manager vulnerabilities attacking its internal linked structures. However current memory management improve a lot and at current date it is quite ineffective to attack heap in this way. But still those techniques come into hand when we start to looking at linked structures widespread throughout kernel that are unfortunately not hardened enough.
In this presentation we will examine power of these vulnerabilities by famous example “CVE – 2013 - 3660”. Showing bypass on ‘lazy’ assertions of _LIST_ENTRY, present exploitation after party and teleport to kernel.
Java Colombo Meetup on 22nd March 2018
Speaker: Isuru Perera, Technical Lead at WSO2
Flame graphs are a visualization of profiled software and it was developed by Brendan Gregg, an industry expert in computing performance and cloud computing. Finding out why CPUs are busy is an important task when troubleshooting performance issues and we often use a sampling profiler to see which code-paths are hot. However, a profiler will dump a lot of data with thousands of lines and it is not easy to go through all data. With Flame Graphs, we can identify the most frequent code-paths quickly and accurately. Basically, a Flame Graph can simply visualize the stack traces output of a sampling profiler.
There are many ways to profile Java applications and Java Flight Recorder (JFR) is a really good tool to profile a Java application with a very low overhead. I will show how we can generate a Flame Graph from a Java Flight Recording using the JFR Flame Graph tool (https://github.com/chrishantha/jfr-flame-graph) I developed.
Since Flame Graphs can visualize any stack profiles, we can also use a Linux system profiler (perf) and create a Java Mixed-Mode Flame Graph, which will show how much CPU time is spent in Java methods, system libraries and the kernel. We can troubleshoot performance issues related to high CPU usage easily with a flame graph showing profile information from both system code paths and Java code paths. I will discuss how we can use the -XX:+PreserveFramePointer option in JDK and the perf system profiler to generate a Java Mixed-mode flame graph.
OWASP SD: Deserialize My Shorts: Or How I Learned To Start Worrying and Hate ...Christopher Frohoff
Object deserialization is an established but poorly understood attack vector in applications that is disturbingly prevalent across many languages, platforms, formats, and libraries.
In January 2015 at AppSec California, Chris Frohoff and Gabe Lawrence gave a talk on this topic, covering deserialization vulnerabilities across platforms, the many forms they take, and places they can be found. It covered, among other things, somewhat novel techniques using classes in commonly used libraries for attacking Java serialization that were subsequently released in the form of the ysoserial tool. Few people noticed until late 2015, when other researchers used these techniques/tools to exploit well known products such as Bamboo, WebLogic, WebSphere, ApacheMQ, and Jenkins, and then services such as PayPal. Since then, the topic has gotten some long-overdue attention and great work is being done by many to improve our understanding and developer awareness on the subject.
This talk will review the details of Java deserialization exploit techniques and mitigations, as well as report on some of the recent (and future) activity in this area.
http://www.meetup.com/Open-Web-Application-Security-Project-San-Diego-OWASP-SD/events/226242635/
Recently our team researched various ntos subsystem attack vectors, and one of the outputs we will present in our talk. DeathNote as our internal code name to this component, which resides in Microsoft Windows kernel, hiding behind different interfaces and exposed to user differently.
What can goes bad with it?
Basically two kinds of problems, one is syscall handling via direct user interaction. We will describe how to obtain basic understanding of what's going on, how it interacts with other components and what is its purpose. With those knowledge we will dig deeper how to make more complex fuzzing logic to cause enough chaos that will end up in unexpected behaviors in Windows kernel, and demonstrate some of them.
And as for second, as it hints from title, this module does bit of data parsing, so we will dive deep into internals, pointing out some available materials, and move on to reverse engineered structures and internal mechanism. We will show how some tricks can outcome with various results, and how structured approach can expose more problems than is expected.
Talk given by Pierre Ernst, Product Security Lead at Salesforce, at Hack Fest 2016 on November 2016
Pierre Ernst has 20 years of professional experience in building and breaking applications. His current focus is helping organisations improve their security posture by playing both offense and defense. In his spare time, he still enjoys finding high-value vulnerabilities and tries to make open source components more secure using his weapon of choice: code review. His favorite research topics include: weaponizing XML External Entity (XXE) attacks and XPath injections, finding novel ways of triggering hash table collisions and exploiting all sorts of deserialization technologies.
Fixing the Java Serialization mess
Deserializing untrusted input with Java has been known to be a risky proposition for at least 10 years. More recently, several vulnerabilities exploiting this flaw have been published. These deserialization vulnerabilities can be divided into 2 groups: endpoints allowing deserialization of arbitrary classes known to the application, or serialization “gadgets” allowing to weaponize malicious input for these endpoints. When it comes to fixing this class of vulnerabilities, it is hard to reach a consensus: some library maintainers consider that there is no point fixing the “gadgets” and that all application should simply stop accepting serialized input. Easier said than done…
While the root cause of the issue lies with a lenient Java API (not allowing to specify which class is to be deserialized), we need an immediate fix. This is why Pierre Ernst came up with the seminal “Look-ahead Java deserialization” concept in 2013.
During this talk, the current look-ahead implementation will be bypassed with a live demo, and a more robust mitigation will be presented.
Software Profiling: Understanding Java Performance and how to profile in JavaIsuru Perera
Guest lecture at University of Colombo School of Computing on 27th May 2017
Covers following topics:
Software Profiling
Measuring Performance
Java Garbage Collection
Sampling vs Instrumentation
Java Profilers. Java Flight Recorder
Java Just-in-Time (JIT) compilation
Flame Graphs
Linux Profiling
Inside the JVM - Follow the white rabbit! / Breizh JUGSylvain Wallez
Presentation given at the Rennes (FR) Java User Group in Feb 2019.
How do we go from your Java code to the CPU assembly that actually runs it? Using high level constructs has made us forget what happens behind the scenes, which is however key to write efficient code.
Starting from a few lines of Java, we explore the different layers that constribute to running your code: JRE, byte code, structure of the OpenJDK virtual machine, HotSpot, intrinsic methds, benchmarking.
An introductory presentation to these low-level concerns, based on the practical use case of optimizing 6 lines of code, so that hopefully you to want to explore further!
syzbot and the tale of million kernel bugsDmitry Vyukov
The root cause of most software exploits is bugs. Hardening, mitigations and containers are important, but they can't protect a system with thousands of bugs. In this presentation, Dmitry Vyukov will review the current [sad] situation with Linux kernel bugs and security implications based on their experience testing kernel for the past 3 years; overview a set of bug finding tools they are developing (syzbot, syzkaller, KASAN, KMSAN, KTSAN); and discuss problems and areas that require community help to improve the situation.
JetBrains Day Seoul - Exploring .NET’s memory management – a trip down memory...Maarten Balliauw
The .NET Garbage Collector (GC) is really cool. It helps providing our applications with virtually unlimited memory, so we can focus on writing code instead of manually freeing up memory. But how does .NET manage that memory? What are hidden allocations? Are strings evil? It still matters to understand when and where memory is allocated. In this talk, we’ll go over the base concepts of .NET memory management and explore how .NET helps us and how we can help .NET – making our apps better. Expect profiling, Intermediate Language (IL), ClrMD and more!
The .NET Garbage Collector (GC) is really cool. It helps providing our applications with virtually unlimited memory, so we can focus on writing code instead of manually freeing up memory. But how does .NET manage that memory? What are hidden allocations? Are strings evil? It still matters to understand when and where memory is allocated. In this talk, we’ll go over the base concepts of .NET memory management and explore how .NET helps us and how we can help .NET – making our apps better. Expect profiling, Intermediate Language (IL), ClrMD and more!
It is easy to monitor the performance of JVM if one knows how GC and Threads work in JVM. This presentation throws light on Collector types, HotSpot Collection Algorithms, Thread Monitoring, Method Profiling and Heap Profiling
XCon 2014 => http://xcon.xfocus.org/
In the past was quite common to exploit heap / pool manager vulnerabilities attacking its internal linked structures. However current memory management improve a lot and at current date it is quite ineffective to attack heap in this way. But still those techniques come into hand when we start to looking at linked structures widespread throughout kernel that are unfortunately not hardened enough.
In this presentation we will examine power of these vulnerabilities by famous example “CVE – 2013 - 3660”. Showing bypass on ‘lazy’ assertions of _LIST_ENTRY, present exploitation after party and teleport to kernel.
Have you ever cried yourself to sleep unable to find the cause of a horrendous bug in your node app? Cry no more, your tears will be reshaped into blinding swords as we explore uncharted territories laced with mystical creatures.
JavaScript debugging is an often avoided topic due to the uncertainty of how best to accomplish it and the lack of powerful introspective tools. This talk will explore new territory and showcase tools that help you debug complex and difficult issues in your node or frontend app. Libraries and tools such as node-inspector, Time-Travel Debugging, Visual Studio Code, vorlon.js, Vantage and memory leak catchers will be used to slay dragons.
No more shall you fear building complex apps with JavaScript!
Delivered as plenary at USENIX LISA 2013. video here: https://www.youtube.com/watch?v=nZfNehCzGdw and https://www.usenix.org/conference/lisa13/technical-sessions/plenary/gregg . "How did we ever analyze performance before Flame Graphs?" This new visualization invented by Brendan can help you quickly understand application and kernel performance, especially CPU usage, where stacks (call graphs) can be sampled and then visualized as an interactive flame graph. Flame Graphs are now used for a growing variety of targets: for applications and kernels on Linux, SmartOS, Mac OS X, and Windows; for languages including C, C++, node.js, ruby, and Lua; and in WebKit Web Inspector. This talk will explain them and provide use cases and new visualizations for other event types, including I/O, memory usage, and latency.
Documentation: https://izumi.7mind.io/latest/release/doc/distage/
Github: https://github.com/pshirshov/izumi-r2
Pavel Shirshov - DIStage: purely functional programming without sacrificing modularity with modern dependency injection for Scala
- Modularity and its importance
- DI-like mechanisms and their issues in Scala
- Why people think that "DI doesn't compose with functional programming", and why that's not true
- Designing a staged DI, for wiring at runtime, at compile-time, or mixed
- Staging programs for reliability, power and performance
- The pains of supporting rich Scala types (incl. How to emulate kind-polymorphism in Scala 2)
- Garbage collection in DI for better tests and deployments
Pavel's bio: Language-agnostic software engineer, coding for 18 years,
10 years of hands-on commercial engineering experience.
Led a cluster orchestration team at Yandex, "the Russian Google"; implemented an internal orchestration solution, "ISS" (Scala/Java/C++), managing 50K+ physical hosts across 6 datacenters.
Today, Pavel owns Irish R&D company Septimal Mind.
[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...Ambassador Labs
One of the challenges facing Telepresence is growing the contributor community. It’s a complex application that requires a good understanding of OS networking, VPNs, Kubernetes, and everything in between. We’ll kick off this meeting with a general architectural overview of Telepresence. We’ll talk about how we’ve managed the project to date, and our investments to make it easier. We want to then turn it over for an interactive discussion with participants to see what we can do to make it easier to contribute and grow the Telepresence community.
Java in 2019 was predicted to be business as usual by many. We have seen new Java releases coming out as planned, AdoptOpenJDK became the main trustful source of binaries and Oracle fighting for the trademark again by preventing the use of javax as namespace. Everything looks like it would be a silent year for Java. But one thing seems obvious. Java's popularity is not gaining any more traction. New language features keep it up to date but people are getting more selective when it comes to implementation choices. Especially in the age of containers and cloud infrastructures. This talk walks you through the why and how of Java in containers. We will talk about image size and development and deployment processes.
OCCIware Project at EclipseCon France 2016, by Marc Dutoo, Open WideOCCIware
Hear hear dev & ops alike - ever got bitten by the fragmentation of the Cloud space at deployment time, By AWS vs Azure, Open Shift vs Heroku ? in a word, ever dreamt of configuring at once your Cloud application along with both its VMs and database ? Well, the extensible Open Cloud Computing Interface (OCCI) REST API (see http://occi-wg.org/) allows just that, by addressing the whole XaaS spectrum.
And now, OCCI is getting powerboosted by Eclipse Modeling and formal foundations. Enter Cloud Designer and other outputs of the OCCIware project (See http://www.occiware.org) : multiple visual representations, one per Cloud layer and technology. XaaS Cloud extension model validation, documentation & ops scripting generation. Simulation, decision-making comparison. Connectors that bring those models to life by getting their status from common Cloud services. Runtime middleware, deployed, monitored, adminstrated. And tackling the very interesting challenge of modeling a meta API in EMF's metamodel, while staying true to EMF, Eclipse tools and the OCCI standard.
Featuring Eclipse Sirius, Acceleo generators, EMF at runtime. Coming soon to a new Eclipse Foundation project near you, if so you'd like.
This talk includes a demonstration of the Docker connector and of how to use Cloud Designer to configure a simple Cloud application's deployment on the Roboconf PaaS system and OpenStack infrastructure.
Production debugging is hard, and it’s getting harder. With architectures becoming more distributed and code more asynchronous and reactive, pinpointing and resolving errors that happen in production is no child’s game. This session covers some essential tools and more advanced techniques Scala developers can use to debug live applications and resolve errors quickly. It explores crucial techniques for distributed debugging - and some of the pitfalls that make resolution much harder, and can lead to downtime. The talk also touches on some little-known JVM tools and capabilities that give you super-deep visibility at high scale without making you restart it or attach debuggers.
Experiences from debugging ZFS in production in Illumos and Linux from Delphix. Introduction of the SDB debugger and how it can be used to debug ZFS on Linux.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Surge2012
1. When Node.js goes wrong:
Debugging Node in production
Surge 2012
David Pacheco (@dapsays)
Joyent
2. The Rise of Node.js
• We see Node.js as the confluence of three ideas:
• JavaScript’s friendliness and rich support for asynchrony (i.e., closures)
• High-performance JavaScript VMs (e.g., V8)
• Time-tested system abstractions (i.e. Unix, in the form of streams)
• Event-oriented model delivers consistent performance in the
presence of long latency events (i.e. no artificial latency bubbles)
• Node.js is displacing C for a lot of highly reliable, high performance
core infrastructure software (at Joyent alone: DNS, DHCP,
SNMP, LDAP, key value stores, public-facing web services, ...).
• This has been great for rapid development, but historically has
come with a cost in debuggability.
• Debugging is decidedly not a new problem...
2
3. The Genesis of debugging
“As soon as we started programming,
we found to our surprise that it wasn't
as easy to get programs right as we
had thought. Debugging had to be
discovered. I can remember the exact
instant when I realized that a large
part of my life from then on was going
to be spent in finding mistakes in my
own programs.”
—Sir Maurice Wilkes, 1913 - 2010
3
4. Debugging Node: a run-away service
• February, 2011: Joyent is preparing to launch no.de, a Node PaaS.
• During testing, Cloud Analytics becomes intermittently
unresponsive.
• Quickly traced the problem to a rogue data aggregator using 100%
of 1 CPU core and not responding over HTTP or AMQP.
• How do you debug this?
4
8. Give up?
• Could add more logging for next time, but we don’t know how to
reproduce it. (Plus, what would we log?)
• ... but it’s still exhibiting these symptoms! Can’t we figure out why?!
Text
Text
8
9. The more general problem
• The software: a moderately complex concurrent service
(that is, where concurrent requests can affect one another).
• The deployment: in production, 10s to 1000s of instances.
• The problem: ~once/day, one of the instances crashes, leaving
behind a stacktrace where an assertion was blown. (Or worse: one
of the instances simply misbehaves, leaving nothing to help figure
out what’s going wrong.)
• How do you debug this?
9
10. A first approach
• Add instrumentation (console.log) and redeploy.
• How easy is it to deploy a new version?
• How risky is it to deploy a new version? What’s the impact?
What if it’s a very common code path that you need to instrument?
• What if this isn’t your code, but a customer’s, whose deployment
you cannot control?
• Are you sure you’ll only need to do this once?
(You lose credibility with each lap with ops and customers.)
• If you’re lucky or if the problem is relatively simple, this can work
okay.
10
11. “The postmortem technique”
“Experience with the EDSAC has
shown that although a high proportion
of mistakes can be removed by
preliminary checking, there frequently
remain mistakes which could only
have been detected in the early
stages by prolonged and laborious
study. Some attention, therefore, has
been given to the problem of dealing
with mistakes after the programme
has been tried and found to fail.”
—Stanley Gill, 1926 - 1975
“The diagnosis of mistakes in programmes on the EDSAC”, 1951
11
12. A better approach
• For C programs, we have rich tools for postmortem analysis of a
system based on a snapshot of its state.
• This technique is so old, the term for this state snapshot dates
from the dawn of computing: it’s a core dump.
• Once a core dump has been generated, either automatically after
a crash or on-demand using gcore(1), the program can be
immediately restarted to restore service quickly so that
engineers can debug the problem asynchronously.
• Using the debugger on the core dump, you can inspect all internal
program state: global variables, threads, and objects.
• Can also use the same tools with a live process.
• Can’t we do this with Node.js?
12
13. Debugging dynamic environments
• Historically, native postmortem tools have been unable to
meaningfully observe dynamic environments like Node.
• Such tools would need to translate native abstractions from the
dump (symbols, functions, structs) into their higher-level
counterparts in the dynamic environment (variables, Functions,
Objects).
• Some abstractions don’t even exist explicitly in the language itself.
(e.g., JavaScript’s event queue)
• Node is not alone! The state of the art is no better in Python, Ruby,
or PHP, and not nearly solved for Java or Erlang either.
13
14. Aside: MDB
• illumos-based systems like SmartOS and OmniOS have MDB, the
modular debugger built specifically for postmortem analysis
• MDB was originally built for postmortem analysis of the operating
system kernel and later extended to applications
• Plug-ins (“dmods”) can easily build on one another to deliver
powerful postmortem analysis tools, e.g.:
• ::stacks coalesces threads based on stack trace, with optional filtering
by module, caller, etc
• ::findleaks performs postmortem garbage collection on a core dump to find
memory leaks in native code
• Could we build a dmod for Node?
14
15. mdb_v8: postmortem debugging for Node
• With some excruciating pain and some ugly layer violations, we
were able to build mdb_v8
• With ::jsstack, prints call stacks, including native C++ and
JavaScript functions and arguments.
• With ::jsprint, given a pointer, prints out as a C++ object and
its JavaScript counterpart.
• With ::v8function, given a JSFunction pointer, show the assembly
for that function.
• With ::findjsobjects, scans the heap to identify how many
instances of each object type exist (incredible visibility into memory
usage).
• Demo
15
16. Remember that run-away Node program?
• In February, 2011, we had essentially no way to see what this
program was doing.
• We saved a core dump in case we might one day have a way to
read it. We also added instrumentation in case we saw it again.
(We expected to see it again very soon after going to production.)
• We didn’t see it again until October, while the mdb_v8 work was
underway. So we applied what we had to the original core file...
16
17. And the winner is:
> ::jsstack
8046a9c <anonymous> (as exports.bucketize) at lib/heatmap.js position 7838
8046af8 caAggrValueHeatmapImage at lib/ca/ca-agg.js position 48960
...
> 8046a9c::jsframe -v
8046a9c <anonymous> (as exports.bucketize)
func: fc435fcd
file: lib/heatmap.js
posn: position 7838
arg1: fc070719 (JSObject)
arg2: fc070709 (JSArray)
> fc070719::jsprint
{
base: 1320886447,
height: 281,
Invalid input resulted in infinite loop in JavaScript
width: 624, Time to root cause: 10 minutes
max: 11538462,
min: 11538462,
...
}
17
18. mdb_v8: How the sausage is made
• V8 (libv8.a) includes a small amount (a few KB) of metadata that
describes the heap’s classes, type information, and class layouts.
(Small enough to include in production builds.)
• mdb_v8 knows how to identify stack frames, iterate function
arguments, iterate object properties, and walk basic V8 structures
(arrays, functions, strings).
• mdb_v8 uses the debug metadata encoded in the binary to avoid
hardcoding the way heap structures are laid out in memory. (Still
has intimate knowledge of things like property iteration.)
18
19. What did you say was in this sausage?
• Goal: debugger module shouldn’t hardcode structure offsets and
other constants, but rather rely on metadata included in the “node”
binary.
• Generally speaking, these offsets are computed at compile-time
and used in inline functions defined by macros. So they get
compiled out and are not available at runtime.
• The build process was modified to:
• Generate a new C++ source file with references to the constants that we need,
using extern “C” constants that our debugger module can look for and read.
• Build this with the rest of libv8_base.a.
• Result: this “debug metadata” is embedded in $PREFIX/bin/node,
and the debugger can read it directly from the core file.
• (Should) generally work for 32-bit/64-bit, different architectures,
and no matter how complex the expressions for the constants are.
19
20. Problems with this approach
• We strongly believe in the general approach of having the
debugger grok program state from a snapshot, because it’s
comprehensive and has zero runtime overhead, meaning it
works in production. (This is a constraint.)
• With the current implementation, the debugger module is built and
delivered separately from the VM, which means that changes in
the VM can (and do) break the debugger module.
• Additionally, each debugger feature requires reverse engineering
and reimplementing some piece of the VM.
• Ideally, the VM would embed programmatic logic for decoding the
in-memory state (e.g., iterating objects, iterating object properties,
walking the stack, and so on) -- without relying on the VM itself to
be running.
20
21. Debugging live programs
• Postmortem tools can be applied to live processes, and
core files can be generated for running processes.
• Examining processes and core dumps is useful for many kinds of
failure, but sometimes you want to trace runtime activity.
21
22. DTrace
• Provides comprehensive tracing of kernel and application-level
events in real-time (from “thread on-CPU” to “Node GC done”)
• Scales arbitrarily with the number of traced events.
(first class in situ data aggregation)
• Suitable for production systems because it’s safe, has minimal
overhead (usually no disabled probe effect), and can be enabled/
disabled dynamically (no application restart required).
• Open-sourced in 2005. Available on illumos-derived systems like
SmartOS and OmniOS, Solaris-derived systems, BSD, and
MacOS (Linux ports in progress).
22
23. DTrace in dynamic environments
• DTrace instruments the system holistically, which is to say, from
the kernel, which poses a challenge for interpreted environments
• User-level statically defined tracing (USDT) providers describe
semantically relevant points of instrumentation
• Some interpreted environments (e.g., Ruby, Python, PHP, Erlang)
have added USDT providers that instrument the interpreter itself
• This approach is very fine-grained (e.g., every function call) and
doesn’t work in JIT’d environments
• We decided to take a different tack for Node.js...
23
24. DTrace for Node.js
• Given the nature of the paths that we wanted to instrument, we
introduced a function into JavaScript that Node can call to get into
USDT-instrumented C++
• Introduces disabled probe effect: calling from JavaScript into C++
costs even when probes are not enabled
• Use USDT is-enabled probes to minimize disabled probe effect
once in C++
• If (and only if) the probe is enabled, prepare a structure for the
kernel that allows for translation into a structure that is familiar to
node programmers
24
26. DTrace probes in Node.js modules
• Our technique is adequate for DTrace probes in the Node.js core,
but it’s very cumbersome for pure Node.js modules
• Fortunately, Chris Andrews has generalized this technique in his
node-dtrace-provider module:
https://github.com/chrisa/node-dtrace-provider
• This module allows one to declare and fire one’s own probes
entirely in JavaScript
• Used extensively by Joyent’s Mark Cavage in his node-restify and
ldap.js modules, especially to allow for measurement of latency
26
27. DTrace stack traces
• ustack(): DTrace looks at (%ebp, %eip) and follows frame
pointers to the top of the stack (standard approach).
Asynchronously, looks for symbols in the process’s address space
to map instruction offsets to function names:
0x80ed9ab becomes malloc+0x16
• Great for C, C++. Doesn’t work for JIT’d environments.
• Functions are compiled at runtime => they have no corresponding symbols
=> the VM must be called upon at runtime to map frames to function names
• Garbage collection => functions themselves move around at arbitrary points
=> mapping of frames to function names must be done “synchronously”
• jstack(): Like ustack(), but invokes VM-specific ustack helper,
expressed in D and attached to the VM binary, to resolve names.
27
28. DTrace ustack helpers
• For JIT’d code, DTrace supports ustack helper mechanism, by
which the VM itself includes logic to translate from
(frame pointer, instruction pointer) -> human-readable function name
• When jstack() action is processed in probe context (in the kernel),
DTrace invokes the helper to translate frames:
Before After
0xfe772a8c toJSON at native date.js position 39314
0xfe84d962 BasicJSONSerialize at native json.js position 8444
0xfea6b6ed BasicSerializeObject at native json.js position 7622
0xfe84db11 BasicJSONSerialize at native json.js position 8444
0xfeaba5ee stringify at native json.js position 10128
28
29. V8 ustack helper
• The ustack helper has to do much of the same work that mdb_v8
does to identify stack frames and pick apart heap objects.
• The same debug metadata that’s used for mdb_v8 is used for the
helper, but unlike mdb_v8, the helper is embedded directly into the
node binary (good!).
• The implementation is written in D, and subject to all the same
constraints as other DTrace scripts (and then some): no functions,
no iteration, no if/else.
• Particularly nasty pieces include expanding ConsStrings and
binary searching to compute line numbers.
• The helper only depends on V8, not Node.js. (With MacOS support
for ustack helpers from profile probes, we could use the same
helper to profile webapps running under Chrome!)
29
30. Profiling Node with DTrace
• “profile” provider: probe that fires N times per second per CPU
• ustack()/jstack() actions: collect user-level stacktrace when
a probe fires.
• Low-overhead runtime profiling (via stack sampling) that can be
turned on and off without restarting your program.
• Demo.
30
32. More real-world examples
• The infinite loop problem we saw earlier was debugged with
mdb_v8, and could have also been debugged with DTrace.
• @izs used mdb_v8‘s heap scanning to zero in on a memory leak
in Node 0.7 that was seriously impacting several users, including
Voxer.
• @mranney (Voxer) has used Node profiling + flame graphs to
identify several performance issues (unoptimized OpenSSL
implementation, poor memory allocation behavior).
• Debugging RangeError (stack overflow, with no stack trace).
32
33. Final thoughts
• Node is a great for rapidly building complex or distributed system
software. But in order to achieve the reliability we expect from such
systems, we must be able to understand both fatal and non-
fatal failure in production from the first occurrence.
• One year ago: we had no way to solve the “infinite loop” problem
without adding more logging and hoping to see it again.
• Now we have tools to inspect both running and crashed Node
programs (mdb_v8 and the DTrace ustack helper), and we’ve used
them to debug problems in minutes that we either couldn’t solve at
all before or which took days or weeks to solve.
• But the postmortem tools are still primitive (like a flashlight in a
dark room). Need better support from the VM.
33
34. • Thanks:
• @mraleph for help with V8 and landing patches
• @izs and the Node core team for help integrating DTrace and MDB support
• @brendangregg for flame graphs
• @chrisandrews for node-dtrace-provider
• @mcavage for putting it to such great use in node-restify and ldap.js
• @mranney and Voxer for pushing Node hard, running into lots of issues, and
helping us refine the tools to debug them. (God bless the early adopters!)
• For more info:
• http://dtrace.org/blogs/dap/2012/04/25/profiling-node-js/
• http://dtrace.org/blogs/dap/2012/01/13/playing-with-nodev8-postmortem-debugging/
• https://github.com/joyent/illumos-joyent/blob/master/usr/src/cmd/mdb/common/modules/v8/mdb_v8.c
• https://github.com/joyent/node/blob/master/src/v8ustack.d
34