fd0 is the listening socket descriptor
The other descriptors are initialized to -1 to indicate they are unused.
Max is the highest descriptor number currently in use, which is fd0 (the listening socket).
rset is the read file descriptor set for select. It has a 1 bit for fd0 to indicate the listening socket should be checked for connections. All other bits are 0.
Client[] array tracks the client socket descriptors and is initialized to -1.
So initially it is just waiting for a connection on the listening socket fd0.
Formbook is a malware that steals passwords and harvests credentials by injecting code into targeted applications like web browsers, mail clients, and IM apps. It uses various anti-analysis techniques like manually mapping ntdll.dll and checking for debuggers. Formbook employs process hollowing to inject its code into explorer.exe and other processes, then sets up inline userland hooks to intercept function calls and harvest passwords.
Possibility of arbitrary code execution by Step-Oriented Programmingkozossakai
Step-Oriented Programming (SOP) allows executing arbitrary code on embedded systems by repeating step execution and changing the program counter value. A debugger communicates with a target system's stub using the Remote Serial Protocol to read/write memory and registers, enabling full control via simple commands if the connection is compromised. SOP constructs code by combining pieces of existing machine code and executes it without needing to directly inject new code. Therefore attacks are possible even if execution from data areas is prevented. The presentation will demonstrate this attack principle and results from actual experimentation.
The Art of Exploiting Unconventional Use-after-free Bugs in Android Kernel by...CODE BLUE
In this talk, we are going to disclose two unconventional Use-after-free kernel bugs on Android we found last year, and introduce the new techniques we used to make these exploits 100% reliable.
The first bug is CVE-2017-0403, which we used to gain root privilege on almost all devices shipped with 3.10 or earlier Linux kernel last year. So far more than 14 million users have successfully rooted their smartphones with this exploit. With this vulnerability, an attacker only can overwrite the freed object at a fixed offset with a pointer to object itself. How to achieve kernel code execution with this bug can be very challenging.To solve the problem, we propose a new method which is using iovec to re-fill the freed object and compromising the pipe subsystem in kernel.In this way we can covert this unusual memory corruption to arbitrary kernel memory overwriting.
The second bug is CVE-2016-6787. The bug is an UAF due to race condition, may corrupt a critical kernel structure, and lead to the kernel crash when scheduler switched context back to attacker's process. So we'll introduce a way to freeze the attacker's process soon after UAF happened ,stop kernel from crashing, and make the exploit reliable.
In summary, this presentation gives out the new techniques of exploiting use-after-free bugs we just found in Android kernel. The ideas of exploitation are fresh, detail of bugs is also never discussed before.
No instrumentation Golang Logging with eBPF (GoSF talk 11/11/20)Pixie Labs
The document discusses using eBPF for instrumentation and logging in Golang applications without source code modifications. It provides an example of using eBPF to log function arguments by attaching a BPF program to the computeE function via uprobes. This allows viewing function parameters in production without recompiling or using a debugger. eBPF provides low overhead dynamic tracing of all application code compared to other options like debuggers or static tracing tools.
Speculative Execution of Parallel Programs with Precise Exception Semantics ...Akihiro Hayashi
Akihiro Hayashi, Max Grossman, Jisheng Zhao, Jun Shirako, Vivek Sarkar. The 26th International Workshop on Languages and Compilers for Parallel Computing (LCPC2013), September 25-27, 2013 Qualcomm Research Silicon Valley, Santa Clara, CA (co-located with CnC-2013).
Troubleshooting Linux Kernel Modules And Device DriversSatpal Parmar
The document discusses various techniques for debugging Linux kernel modules and device drivers, including:
1) Using printk statements to output debug messages from kernel space.
2) Watching system calls with strace to debug interactions between user and kernel space.
3) Adding /proc file system entries and write functions to dynamically modify driver values at runtime.
4) Enabling source-level debugging with tools like kgdb to debug at the level of C source code.
fd0 is the listening socket descriptor
The other descriptors are initialized to -1 to indicate they are unused.
Max is the highest descriptor number currently in use, which is fd0 (the listening socket).
rset is the read file descriptor set for select. It has a 1 bit for fd0 to indicate the listening socket should be checked for connections. All other bits are 0.
Client[] array tracks the client socket descriptors and is initialized to -1.
So initially it is just waiting for a connection on the listening socket fd0.
Formbook is a malware that steals passwords and harvests credentials by injecting code into targeted applications like web browsers, mail clients, and IM apps. It uses various anti-analysis techniques like manually mapping ntdll.dll and checking for debuggers. Formbook employs process hollowing to inject its code into explorer.exe and other processes, then sets up inline userland hooks to intercept function calls and harvest passwords.
Possibility of arbitrary code execution by Step-Oriented Programmingkozossakai
Step-Oriented Programming (SOP) allows executing arbitrary code on embedded systems by repeating step execution and changing the program counter value. A debugger communicates with a target system's stub using the Remote Serial Protocol to read/write memory and registers, enabling full control via simple commands if the connection is compromised. SOP constructs code by combining pieces of existing machine code and executes it without needing to directly inject new code. Therefore attacks are possible even if execution from data areas is prevented. The presentation will demonstrate this attack principle and results from actual experimentation.
The Art of Exploiting Unconventional Use-after-free Bugs in Android Kernel by...CODE BLUE
In this talk, we are going to disclose two unconventional Use-after-free kernel bugs on Android we found last year, and introduce the new techniques we used to make these exploits 100% reliable.
The first bug is CVE-2017-0403, which we used to gain root privilege on almost all devices shipped with 3.10 or earlier Linux kernel last year. So far more than 14 million users have successfully rooted their smartphones with this exploit. With this vulnerability, an attacker only can overwrite the freed object at a fixed offset with a pointer to object itself. How to achieve kernel code execution with this bug can be very challenging.To solve the problem, we propose a new method which is using iovec to re-fill the freed object and compromising the pipe subsystem in kernel.In this way we can covert this unusual memory corruption to arbitrary kernel memory overwriting.
The second bug is CVE-2016-6787. The bug is an UAF due to race condition, may corrupt a critical kernel structure, and lead to the kernel crash when scheduler switched context back to attacker's process. So we'll introduce a way to freeze the attacker's process soon after UAF happened ,stop kernel from crashing, and make the exploit reliable.
In summary, this presentation gives out the new techniques of exploiting use-after-free bugs we just found in Android kernel. The ideas of exploitation are fresh, detail of bugs is also never discussed before.
No instrumentation Golang Logging with eBPF (GoSF talk 11/11/20)Pixie Labs
The document discusses using eBPF for instrumentation and logging in Golang applications without source code modifications. It provides an example of using eBPF to log function arguments by attaching a BPF program to the computeE function via uprobes. This allows viewing function parameters in production without recompiling or using a debugger. eBPF provides low overhead dynamic tracing of all application code compared to other options like debuggers or static tracing tools.
Speculative Execution of Parallel Programs with Precise Exception Semantics ...Akihiro Hayashi
Akihiro Hayashi, Max Grossman, Jisheng Zhao, Jun Shirako, Vivek Sarkar. The 26th International Workshop on Languages and Compilers for Parallel Computing (LCPC2013), September 25-27, 2013 Qualcomm Research Silicon Valley, Santa Clara, CA (co-located with CnC-2013).
Troubleshooting Linux Kernel Modules And Device DriversSatpal Parmar
The document discusses various techniques for debugging Linux kernel modules and device drivers, including:
1) Using printk statements to output debug messages from kernel space.
2) Watching system calls with strace to debug interactions between user and kernel space.
3) Adding /proc file system entries and write functions to dynamically modify driver values at runtime.
4) Enabling source-level debugging with tools like kgdb to debug at the level of C source code.
Implementation of the ZigBee ZCL Reporting Configuration FeaturesSimen Li
This slide presents the implementation of the ZigBee ZCL Reporting Configuration features by taking TI Z-Stack Home 1.2.1 SampleLight project as an example.
WinDbg is a low-level debugger for Windows that provides features like usermode debugging, kernel debugging, post-mortem debugging, and support for debugging extensions. It can be used to debug crashes, analyze memory leaks, find deadlocks, and investigate other issues when the higher-level Visual Studio debugger is not sufficient. The document provides examples of using WinDbg commands and extensions like SOS to debug memory leaks, analyze crashes based on offset or dump files, and investigate .NET deadlocks.
Return oriented programming (ROP) allows an attacker to bypass address space layout randomization (ASLR) and data execution prevention (DEP). It works by identifying small "gadgets" in a program's code that end with a return instruction. These gadgets can be stitched together to perform operations or redirect execution flow. First, gadgets are found in the program using tools like ROPeMe or objdump. Useful gadgets include those that load registers from memory or call functions indirectly. The gadgets can then be chained to build ROP payloads that copy shellcode into memory and pivot the stack to execute it.
Take a Jailbreak -Stunning Guards for iOS Jailbreak- by Kaoru OtsukaCODE BLUE
This document discusses methods for jailbreaking iOS devices by exploiting vulnerabilities to escalate privileges and disable security mitigations. It summarizes 3 key steps:
1) Using CVE-2016-7637 and CVE-2016-7661 to perform privilege escalation to root level access via a man-in-the-middle attack on the com.apple.iohideventsystem port.
2) Using CVE-2016-7644, a use-after-free bug, to gain the kernel task port, allowing control of the kernel.
3) Overwriting ipc_port data structures to fake the kernel task port and obtain full kernel write privileges, bypassing security mechanisms like AMCC
For the Greater Good: Leveraging VMware's RPC Interface for fun and profit by...CODE BLUE
Virtual machines play a crucial role in modern computing. They often are used to isolate multiple customers with instances on the same physical server. Virtual machines are also used by researchers and security practitioners to isolate potentially harmful code for analysis and review. The assumption being made is that by running in a virtual machine, the potentially harmful code cannot execute anywhere else. However, this is not foolproof, as a vulnerability in the virtual machine hypervisor can give access to the entire system. While this was once thought of as just hypothetical, two separate demonstrations at Pwn2Own 2017 proved this exact scenario.
This talk details the host-to-guest communications within VMware. Additionally, the presentation covers the functionalities of the RPC interface. In this section of the presentation, we discuss the techniques that can be used to record or sniff the RPC requests sent from the Guest OS to the Host OS automatically. We also demonstrate how to write tools to query the RPC Interface in C++ and Python for fuzzing purposes.
Finally, we demonstrate how to exploit Use-After-Free vulnerabilities in VMware by walking through a patched vulnerability.
This document discusses software vulnerabilities and exploits. It begins with an introduction to remote code execution and provides legal aspects of data interception for Germany and Switzerland. It then discusses the goals of preventing teaching how to write malware or hack systems illegally. The document covers basics of function calls, stack smashing vulnerabilities, and how to redirect program flow to injected shellcode through buffer overflows. It provides examples of finding vulnerabilities using debuggers and generating exploits in Python to execute shellcode by overwriting return addresses. Methods discussed include placing shellcode in registers or on the stack and dealing with gaps between the stack pointer and shellcode.
You're Off the Hook: Blinding Security SoftwareCylance
User-mode hooking is dead. It’s also considered harmful due to interference with OS-level exploit mitigations like Control Flow Guard (CFG). At BlackHat US 2016, the “Captain Hook” talk revealed there were multiple serious security issues in AV hooking — we will put the final nail in the coffin by showing how trivial it is to bypass user-mode hooks. We will demonstrate a universal user-mode unhooking approach that can be included in any binary to blind security software from monitoring code execution and perform heuristic analysis. The tool and source code will be released on GitHub after the talk.
Alex Matrosov | Principal Research Scientist
Jeff Tang | Senior Security Researcher
Debugging linux kernel tools and techniquesSatpal Parmar
This document discusses tools and techniques for debugging the Linux kernel, including debuggers like gdb, built-in debugging facilities, system logs, and crash dump analysis tools like LKCD. It outlines common issues like kernel crashes and hangs, and provides an example of analyzing an "oops" crash dump to identify the failing line of code through tools like ksymoops. It also covers generating a full system memory dump using LKCD for thorough crash investigation.
This document provides an overview of fuzzing techniques and the Sulley fuzzing framework. It begins with definitions of fuzzing and different fuzzing techniques like static testing, randomized fuzzing, and mutation-based fuzzing. The rest of the document demonstrates how to setup and use the Sulley framework to fuzz protocols like HTTP and file formats. It includes explanations of the Sulley API and how to generate test cases, monitor for crashes, and analyze results. Examples are provided of fuzzing HTTP servers and file formats.
Course lecture - An introduction to the Return Oriented ProgrammingJonathan Salwan
This document provides an introduction and overview of Return Oriented Programming (ROP). It discusses classical stack overflow attacks and the mitigations put in place like Address Space Layout Randomization (ASLR) and No eXecute (NX) bit. It then introduces ROP as a technique to bypass these mitigations by chaining small snippets of existing code, called gadgets, to perform malicious tasks without injecting code. Several tools for finding gadgets are presented, and an example exploitation of the CVE-2011-1938 vulnerability is discussed to demonstrate ROP in practice. The document concludes with discussing mitigations against ROP and some related research topics.
Androsia: A step ahead in securing in-memory Android application data by Sami...CODE BLUE
This document describes Androsia, a tool that analyzes Android applications to determine the last usage points of StringBuilder objects that may contain sensitive data. It uses the Soot framework to convert Dalvik bytecode to Jimple code and perform an interprocedural data flow analysis. The analysis computes summaries for each method specifying the last usage points of local variables and static fields. These summaries are then used to determine the last usage points of StringBuilder objects at the whole program level to ensure sensitive data is cleared.
CppUnit is a unit testing framework for C++ that was originally ported from JUnit. The document discusses using CppUnit framework classes like TestCase, TestRunner, and TestFixture for unit testing. It also covers integrating CppUnit into the build process and using helper macros to minimize coding errors when creating tests.
The document discusses various types of shellcodes used in reverse engineering, including local shellcodes, remote shellcodes like reverse shellcodes and bindshell shellcodes, and other less common types. It provides details on the different techniques shellcodes use, such as staged shellcodes that download additional code in stages. The document also introduces the msfpayload command used to generate shellcodes in Metasploit and provides examples of configuring options and outputting shellcode in various formats.
This document discusses building a virtual platform for the OpenRISC architecture using SystemC and transaction-level modeling. It covers setting up the toolchain, writing test programs, and simulating the platform using event-driven or cycle-accurate simulation with Icarus Verilog or the Vorpsoc simulator. The virtual platform allows fast development and debugging of OpenRISC code without requiring physical hardware.
[ZigBee 嵌入式系統] ZigBee 應用實作 - 使用 TI Z-Stack FirmwareSimen Li
The document outlines an application called BasicApp that is used to understand the basic operations of the OSAL (Operating System Abstraction Layer) embedded in the ZigBee firmware. It describes initializing tasks using OSAL, processing events, and handling key presses to control an LED using the HAL (Hardware Abstraction Layer) APIs. The code files for the BasicApp include OSAL_BasicApp.c for task initialization, BasicApp.h for constants and function declarations, and BasicApp.c which implements the task event processing and key handling functions.
Implementation of the ZigBee ZCL Reporting Configuration FeaturesSimen Li
This slide presents the implementation of the ZigBee ZCL Reporting Configuration features by taking TI Z-Stack Home 1.2.1 SampleLight project as an example.
WinDbg is a low-level debugger for Windows that provides features like usermode debugging, kernel debugging, post-mortem debugging, and support for debugging extensions. It can be used to debug crashes, analyze memory leaks, find deadlocks, and investigate other issues when the higher-level Visual Studio debugger is not sufficient. The document provides examples of using WinDbg commands and extensions like SOS to debug memory leaks, analyze crashes based on offset or dump files, and investigate .NET deadlocks.
Return oriented programming (ROP) allows an attacker to bypass address space layout randomization (ASLR) and data execution prevention (DEP). It works by identifying small "gadgets" in a program's code that end with a return instruction. These gadgets can be stitched together to perform operations or redirect execution flow. First, gadgets are found in the program using tools like ROPeMe or objdump. Useful gadgets include those that load registers from memory or call functions indirectly. The gadgets can then be chained to build ROP payloads that copy shellcode into memory and pivot the stack to execute it.
Take a Jailbreak -Stunning Guards for iOS Jailbreak- by Kaoru OtsukaCODE BLUE
This document discusses methods for jailbreaking iOS devices by exploiting vulnerabilities to escalate privileges and disable security mitigations. It summarizes 3 key steps:
1) Using CVE-2016-7637 and CVE-2016-7661 to perform privilege escalation to root level access via a man-in-the-middle attack on the com.apple.iohideventsystem port.
2) Using CVE-2016-7644, a use-after-free bug, to gain the kernel task port, allowing control of the kernel.
3) Overwriting ipc_port data structures to fake the kernel task port and obtain full kernel write privileges, bypassing security mechanisms like AMCC
For the Greater Good: Leveraging VMware's RPC Interface for fun and profit by...CODE BLUE
Virtual machines play a crucial role in modern computing. They often are used to isolate multiple customers with instances on the same physical server. Virtual machines are also used by researchers and security practitioners to isolate potentially harmful code for analysis and review. The assumption being made is that by running in a virtual machine, the potentially harmful code cannot execute anywhere else. However, this is not foolproof, as a vulnerability in the virtual machine hypervisor can give access to the entire system. While this was once thought of as just hypothetical, two separate demonstrations at Pwn2Own 2017 proved this exact scenario.
This talk details the host-to-guest communications within VMware. Additionally, the presentation covers the functionalities of the RPC interface. In this section of the presentation, we discuss the techniques that can be used to record or sniff the RPC requests sent from the Guest OS to the Host OS automatically. We also demonstrate how to write tools to query the RPC Interface in C++ and Python for fuzzing purposes.
Finally, we demonstrate how to exploit Use-After-Free vulnerabilities in VMware by walking through a patched vulnerability.
This document discusses software vulnerabilities and exploits. It begins with an introduction to remote code execution and provides legal aspects of data interception for Germany and Switzerland. It then discusses the goals of preventing teaching how to write malware or hack systems illegally. The document covers basics of function calls, stack smashing vulnerabilities, and how to redirect program flow to injected shellcode through buffer overflows. It provides examples of finding vulnerabilities using debuggers and generating exploits in Python to execute shellcode by overwriting return addresses. Methods discussed include placing shellcode in registers or on the stack and dealing with gaps between the stack pointer and shellcode.
You're Off the Hook: Blinding Security SoftwareCylance
User-mode hooking is dead. It’s also considered harmful due to interference with OS-level exploit mitigations like Control Flow Guard (CFG). At BlackHat US 2016, the “Captain Hook” talk revealed there were multiple serious security issues in AV hooking — we will put the final nail in the coffin by showing how trivial it is to bypass user-mode hooks. We will demonstrate a universal user-mode unhooking approach that can be included in any binary to blind security software from monitoring code execution and perform heuristic analysis. The tool and source code will be released on GitHub after the talk.
Alex Matrosov | Principal Research Scientist
Jeff Tang | Senior Security Researcher
Debugging linux kernel tools and techniquesSatpal Parmar
This document discusses tools and techniques for debugging the Linux kernel, including debuggers like gdb, built-in debugging facilities, system logs, and crash dump analysis tools like LKCD. It outlines common issues like kernel crashes and hangs, and provides an example of analyzing an "oops" crash dump to identify the failing line of code through tools like ksymoops. It also covers generating a full system memory dump using LKCD for thorough crash investigation.
This document provides an overview of fuzzing techniques and the Sulley fuzzing framework. It begins with definitions of fuzzing and different fuzzing techniques like static testing, randomized fuzzing, and mutation-based fuzzing. The rest of the document demonstrates how to setup and use the Sulley framework to fuzz protocols like HTTP and file formats. It includes explanations of the Sulley API and how to generate test cases, monitor for crashes, and analyze results. Examples are provided of fuzzing HTTP servers and file formats.
Course lecture - An introduction to the Return Oriented ProgrammingJonathan Salwan
This document provides an introduction and overview of Return Oriented Programming (ROP). It discusses classical stack overflow attacks and the mitigations put in place like Address Space Layout Randomization (ASLR) and No eXecute (NX) bit. It then introduces ROP as a technique to bypass these mitigations by chaining small snippets of existing code, called gadgets, to perform malicious tasks without injecting code. Several tools for finding gadgets are presented, and an example exploitation of the CVE-2011-1938 vulnerability is discussed to demonstrate ROP in practice. The document concludes with discussing mitigations against ROP and some related research topics.
Androsia: A step ahead in securing in-memory Android application data by Sami...CODE BLUE
This document describes Androsia, a tool that analyzes Android applications to determine the last usage points of StringBuilder objects that may contain sensitive data. It uses the Soot framework to convert Dalvik bytecode to Jimple code and perform an interprocedural data flow analysis. The analysis computes summaries for each method specifying the last usage points of local variables and static fields. These summaries are then used to determine the last usage points of StringBuilder objects at the whole program level to ensure sensitive data is cleared.
CppUnit is a unit testing framework for C++ that was originally ported from JUnit. The document discusses using CppUnit framework classes like TestCase, TestRunner, and TestFixture for unit testing. It also covers integrating CppUnit into the build process and using helper macros to minimize coding errors when creating tests.
The document discusses various types of shellcodes used in reverse engineering, including local shellcodes, remote shellcodes like reverse shellcodes and bindshell shellcodes, and other less common types. It provides details on the different techniques shellcodes use, such as staged shellcodes that download additional code in stages. The document also introduces the msfpayload command used to generate shellcodes in Metasploit and provides examples of configuring options and outputting shellcode in various formats.
This document discusses building a virtual platform for the OpenRISC architecture using SystemC and transaction-level modeling. It covers setting up the toolchain, writing test programs, and simulating the platform using event-driven or cycle-accurate simulation with Icarus Verilog or the Vorpsoc simulator. The virtual platform allows fast development and debugging of OpenRISC code without requiring physical hardware.
[ZigBee 嵌入式系統] ZigBee 應用實作 - 使用 TI Z-Stack FirmwareSimen Li
The document outlines an application called BasicApp that is used to understand the basic operations of the OSAL (Operating System Abstraction Layer) embedded in the ZigBee firmware. It describes initializing tasks using OSAL, processing events, and handling key presses to control an LED using the HAL (Hardware Abstraction Layer) APIs. The code files for the BasicApp include OSAL_BasicApp.c for task initialization, BasicApp.h for constants and function declarations, and BasicApp.c which implements the task event processing and key handling functions.
Breaking Parser Logic: Take Your Path Normalization Off and Pop 0days Out!Priyanka Aash
"We propose a new exploit technique that brings a whole-new attack surface to defeat path normalization, which is complicated in implementation due to many implicit properties and edge cases. This complication, being under-estimated or ignored by developers for a long time, has made our proposed attack vector possible, lethal, and general. Therefore, many 0days have been discovered via this approach in popular web frameworks written in trending programming languages, including Python, Ruby, Java, and JavaScript.
Being a very fundamental problem that exists in path normalization logic, sophisticated web frameworks can also suffer. For example, we've found various 0days on Java Spring Framework, Ruby on Rails, Next.js, and Python aiohttp, just to name a few. This general technique can also adapt to multi-layered web architecture, such as using Nginx or Apache as a proxy for Tomcat. In that case, reverse proxy protections can be bypassed. To make things worse, we're able to chain path normalization bugs to bypass authentication and achieve RCE in real world Bug Bounty Programs. Several scenarios will be demonstrated to illustrate how path normalization can be exploited to achieve sensitive information disclosure, SMB-Relay and RCE.
Understanding the basics of this technique, the audience won't be surprised to know that more than 10 vulnerabilities have been found in sophisticated frameworks and multi-layered web architectures aforementioned via this technique."
This slide deck is used as an introduction to the internals of Hadoop MapReduce, as part of the Distributed Systems and Cloud Computing course I hold at Eurecom.
Course website:
http://michiard.github.io/DISC-CLOUD-COURSE/
Sources available here:
https://github.com/michiard/DISC-CLOUD-COURSE
5/13/13 presentation to Austin DevOps Meetup Group, describing our system for deploying 15 websites and supporting services in multiple languages to bare redhat 6 VMs. All system-wide software is installed using RPMs, and all application software is installed using GIT or Tarball.
- Legacy Perl code is code that uses outdated practices, has no tests or documentation, and is difficult to understand and modify. It often results from organic growth over many years and developers.
- Unit testing legacy code provides safety during refactoring, speeds up development by replacing debugging, and creates regression tests. However, the code's dependencies make it difficult to isolate and test.
- Techniques like dependency injection, sprouting, monkey patching, and temporary object reblessing can help break dependencies and make legacy code more testable. Instrumentation with profilers also aids understanding the code.
This document contains biographical information about Boulos Dib, an independent consultant specializing in software development. It provides details about Dib's early experience with personal computers and programming languages. It also lists upcoming presentations by Dib on LightSwitch and Silverlight at the NYC Code Camp in October 2011. The document concludes with an overview of PowerShell scripting.
My talk in Bessemer VP R&D / CTO yearly event (Jan 2020).
The presentation discusses major concept in resilience testing and MyHeritage's path to Chaos Engineering.
FIWARE Wednesday Webinars - How to Debug IoT AgentsFIWARE
How to Debug IoT Agents Webinar - 17th April 2019
Corresponding webinar recording: https://youtu.be/FRqJsywi9e8
Chapter: IoT Agents
Difficulty: 3
Audience: Any Technical
Presenter: Jason Fox (Senior Technical Evangelist, FIWARE Foundation)
How to debug IoT Agents - investigating what goes wrong and how to fix it.
LibOS as a regression test framework for Linux networking #netdev1.1Hajime Tazaki
This document describes using the LibOS framework to build a regression testing system for Linux networking code. LibOS allows running the Linux network stack in a library, enabling deterministic network simulation. Tests can configure virtual networks and run network applications and utilities to identify bugs in networking code by detecting changes in behavior across kernel versions. Example tests check encapsulation protocols like IP-in-IP and detect past kernel bugs. Results are recorded in JUnit format for integration with continuous integration systems.
The document discusses various .NET debugging techniques, including:
1. The importance of debugging and the tools available for debugging .NET applications, such as Visual Studio and Debugging Tools for Windows.
2. Basic debugging tasks like setting breakpoints, stepping through code, and examining variables and call stacks.
3. Advanced techniques like debugging managed code, threads, and memory issues.
Debugging is an important part of the software development process that helps developers write more reliable code. There are several tools available for debugging .NET applications, including Visual Studio and Debugging Tools for Windows. Some basic debugging tasks involve setting breakpoints, stepping through code, examining variables and call stacks, and understanding memory usage and threads. Postmortem debugging techniques like dump file analysis can help debug problems that occur in production environments where live debugging is not possible.
Bug bites Elephant? Test-driven Quality Assurance in Big Data Application Dev...inovex GmbH
Around the currently available large piles of Big Data, there's happening quite a mixed gathering: Business Engineers define which insightswould be precious, Analysts build models, Hadoop programmers tame the flood of data, and Operations people setup machines and networks. It's exactly the interplay of all participants which is central to project success. This setup together with the distributed nature of processing poses new challenges to well-established models of assuring software artifact quality: How can non-programmers define acceptance criteria? How can functionalities be tested which depend on cluster execution, orchestration of, e.g., different hadoop jobs without delaying the development process? Which data selection is suited best for simulating the live environment? How can intermediate results in arbitrary serialization formats be inspected?
In this talk, experiences and best practices from approaching these problems in a large-scale log data analysis project will be presented. At 1&1, our team develops hadoop applications which process roughly 1 billion log events (~1 TB) per day. We will give an overview of the hard- and software setup of our quality assurance environment, which includes FitNesse as a wiki-style acceptance testing framework.Starting from a comparison with existing test frameworks like MRUnit, we will explain how we automate the parameterized deployment of our applications, choose test data sampling strategies, perform workflow management and orchestration of jobs / applications, and use Pig for inspection of intermediate results and definition of final acceptance criteria. Our conclusion is that test-driven development in the field of Big Data requires adaption of existing paradigms, but is crucial for maintaining high quality standards for the resulting applications.
The lecture by Norman Feske for Summer Systems School'12.
Genode Compositions
SSS'12 - Education event, organized by ksys labs[1] in 2012, for students interested in system software development and information security.
Genode[2] - The Genode operating-system framework provides a uniform API for applications on top of 8 existing microkernels/hypervisors: Linux, L4ka::Pistachio, L4/Fiasco, OKL4, NOVA, Fiasco.OC, Codezero, and a custom kernel for the MicroBlaze architecture.
1. http://ksyslabs.org/
2. http://genode.org
This document provides an overview of new features in Apache Lenya 1.4, including modularization, use of UUIDs for internal links, changes to the repository API, configurable meta data, publication templating, and the introduction of a usecase framework. Key points include improved separation of concerns through modularization, more efficient linking using UUIDs instead of URLs, a more queryable metadata model, inheritance and overriding of publication resources using templating, and a standardized way to declare interactive use cases in Lenya.
Java 9 is expected to include several new features and changes, including:
- New collection factory methods like Set.of() and Map.of() that provide immutable collections.
- Enhancements to the Stream API such as takeWhile() and dropWhile().
- Syntax changes like allowing effectively final variables in try-with-resources and @SafeVarargs for private methods.
- The addition of JShell to provide a Java REPL.
- Garbage First (G1) garbage collector becoming the default collector.
- Various performance and logging improvements.
Stopping the Rot - Putting Legacy C++ Under TestSeb Rose
The document discusses introducing unit testing to legacy C++ code. It covers choosing a testing framework, writing initial tests and mocks, and various refactoring techniques like wrapping dependencies, extracting components, and adding non-intrusive C seams to facilitate testing. The goal is to incrementally make the code more testable while maintaining functionality through practices like test-driven development.
Real World Lessons on the Pain Points of Node.js ApplicationsBen Hall
The document discusses several pain points experienced with Node.js applications and solutions for resolving them. It covers creating a strong foundation by upgrading to Node.js v5, locking down NPM dependencies, handling errors properly with try/catch blocks and promises, deploying applications using Docker for scaling, addressing security issues, and using tools like debug and profilers to improve performance.
The document discusses dependency injection (DI) and inversion of control (IoC). It defines DI as a design principle where object creation and injection of dependencies is handled by external code rather than the objects themselves. IoC is described as a style where framework code calls application code rather than vice versa. The document outlines several DI patterns like constructor injection and property injection. It also discusses common DI anti-patterns and provides examples of using DI containers to manage object graphs and dependencies.
Similar to Hadoop: Code Injection, Distributed Fault Injection (20)
The document discusses using Cloudera DataFlow to address challenges with collecting, processing, and analyzing log data across many systems and devices. It provides an example use case of logging modernization to reduce costs and enable security solutions by filtering noise from logs. The presentation shows how DataFlow can extract relevant events from large volumes of raw log data and normalize the data to make security threats and anomalies easier to detect across many machines.
Cloudera Data Impact Awards 2021 - Finalists Cloudera, Inc.
The document outlines the 2021 finalists for the annual Data Impact Awards program, which recognizes organizations using Cloudera's platform and the impactful applications they have developed. It provides details on the challenges, solutions, and outcomes for each finalist project in the categories of Data Lifecycle Connection, Cloud Innovation, Data for Enterprise AI, Security & Governance Leadership, Industry Transformation, People First, and Data for Good. There are multiple finalists highlighted in each category demonstrating innovative uses of data and analytics.
2020 Cloudera Data Impact Awards FinalistsCloudera, Inc.
Cloudera is proud to present the 2020 Data Impact Awards Finalists. This annual program recognizes organizations running the Cloudera platform for the applications they've built and the impact their data projects have on their organizations, their industries, and the world. Nominations were evaluated by a panel of independent thought-leaders and expert industry analysts, who then selected the finalists and winners. Winners exemplify the most-cutting edge data projects and represent innovation and leadership in their respective industries.
The document outlines the agenda for Cloudera's Enterprise Data Cloud event in Vienna. It includes welcome remarks, keynotes on Cloudera's vision and customer success stories. There will be presentations on the new Cloudera Data Platform and customer case studies, followed by closing remarks. The schedule includes sessions on Cloudera's approach to data warehousing, machine learning, streaming and multi-cloud capabilities.
Machine Learning with Limited Labeled Data 4/3/19Cloudera, Inc.
Cloudera Fast Forward Labs’ latest research report and prototype explore learning with limited labeled data. This capability relaxes the stringent labeled data requirement in supervised machine learning and opens up new product possibilities. It is industry invariant, addresses the labeling pain point and enables applications to be built faster and more efficiently.
Data Driven With the Cloudera Modern Data Warehouse 3.19.19Cloudera, Inc.
In this session, we will cover how to move beyond structured, curated reports based on known questions on known data, to an ad-hoc exploration of all data to optimize business processes and into the unknown questions on unknown data, where machine learning and statistically motivated predictive analytics are shaping business strategy.
Introducing Cloudera DataFlow (CDF) 2.13.19Cloudera, Inc.
Watch this webinar to understand how Hortonworks DataFlow (HDF) has evolved into the new Cloudera DataFlow (CDF). Learn about key capabilities that CDF delivers such as -
-Powerful data ingestion powered by Apache NiFi
-Edge data collection by Apache MiNiFi
-IoT-scale streaming data processing with Apache Kafka
-Enterprise services to offer unified security and governance from edge-to-enterprise
Introducing Cloudera Data Science Workbench for HDP 2.12.19Cloudera, Inc.
Cloudera’s Data Science Workbench (CDSW) is available for Hortonworks Data Platform (HDP) clusters for secure, collaborative data science at scale. During this webinar, we provide an introductory tour of CDSW and a demonstration of a machine learning workflow using CDSW on HDP.
Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19Cloudera, Inc.
Join Cloudera as we outline how we use Cloudera technology to strengthen sales engagement, minimize marketing waste, and empower line of business leaders to drive successful outcomes.
Leveraging the cloud for analytics and machine learning 1.29.19Cloudera, Inc.
Learn how organizations are deriving unique customer insights, improving product and services efficiency, and reducing business risk with a modern big data architecture powered by Cloudera on Azure. In this webinar, you see how fast and easy it is to deploy a modern data management platform—in your cloud, on your terms.
Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19Cloudera, Inc.
Join us to learn about the challenges of legacy data warehousing, the goals of modern data warehousing, and the design patterns and frameworks that help to accelerate modernization efforts.
Leveraging the Cloud for Big Data Analytics 12.11.18Cloudera, Inc.
Learn how organizations are deriving unique customer insights, improving product and services efficiency, and reducing business risk with a modern big data architecture powered by Cloudera on AWS. In this webinar, you see how fast and easy it is to deploy a modern data management platform—in your cloud, on your terms.
Explore new trends and use cases in data warehousing including exploration and discovery, self-service ad-hoc analysis, predictive analytics and more ways to get deeper business insight. Modern Data Warehousing Fundamentals will show how to modernize your data warehouse architecture and infrastructure for benefits to both traditional analytics practitioners and data scientists and engineers.
Explore new trends and use cases in data warehousing including exploration and discovery, self-service ad-hoc analysis, predictive analytics and more ways to get deeper business insight. Modern Data Warehousing Fundamentals will show how to modernize your data warehouse architecture and infrastructure for benefits to both traditional analytics practitioners and data scientists and engineers.
The document discusses the benefits and trends of modernizing a data warehouse. It outlines how a modern data warehouse can provide deeper business insights at extreme speed and scale while controlling resources and costs. Examples are provided of companies that have improved fraud detection, customer retention, and machine performance by implementing a modern data warehouse that can handle large volumes and varieties of data from many sources.
Extending Cloudera SDX beyond the PlatformCloudera, Inc.
Cloudera SDX is by no means no restricted to just the platform; it extends well beyond. In this webinar, we show you how Bardess Group’s Zero2Hero solution leverages the shared data experience to coordinate Cloudera, Trifacta, and Qlik to deliver complete customer insight.
Federated Learning: ML with Privacy on the Edge 11.15.18Cloudera, Inc.
Join Cloudera Fast Forward Labs Research Engineer, Mike Lee Williams, to hear about their latest research report and prototype on Federated Learning. Learn more about what it is, when it’s applicable, how it works, and the current landscape of tools and libraries.
Analyst Webinar: Doing a 180 on Customer 360Cloudera, Inc.
451 Research Analyst Sheryl Kingstone, and Cloudera’s Steve Totman recently discussed how a growing number of organizations are replacing legacy Customer 360 systems with Customer Insights Platforms.
Build a modern platform for anti-money laundering 9.19.18Cloudera, Inc.
In this webinar, you will learn how Cloudera and BAH riskCanvas can help you build a modern AML platform that reduces false positive rates, investigation costs, technology sprawl, and regulatory risk.
Introducing the data science sandbox as a service 8.30.18Cloudera, Inc.
How can companies integrate data science into their businesses more effectively? Watch this recorded webinar and demonstration to hear more about operationalizing data science with Cloudera Data Science Workbench on Cazena’s fully-managed cloud platform.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Speck&Tech
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceIndexBug
Imagine a world where machines not only perform tasks but also learn, adapt, and make decisions. This is the promise of Artificial Intelligence (AI), a technology that's not just enhancing our lives but revolutionizing entire industries.
Project Management Semester Long Project - Acuityjpupo2018
Acuity is an innovative learning app designed to transform the way you engage with knowledge. Powered by AI technology, Acuity takes complex topics and distills them into concise, interactive summaries that are easy to read & understand. Whether you're exploring the depths of quantum mechanics or seeking insight into historical events, Acuity provides the key information you need without the burden of lengthy texts.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providersakankshawande
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
Building Production Ready Search Pipelines with Spark and MilvusZilliz
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Driving Business Innovation: Latest Generative AI Advancements & Success StorySafe Software
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
2. Few assumptions
The following work has been done with use of AOP injection technology called
AspectJ
Similar results could be achieved with
direct code implementation
MOP (monkey-patching)
Direct byte-code manipulations
Offered approaches aren't limited by the scope of Hadoop platform ;)
The scope of the talk isn't about AspectJ nor AOP/MOP technology
2
4. What for?
Some APIs as extremely useful as dangerous if made public
stop/blacklist a node or daemon
change a node configuration
certain functionality is experimental and needn't to be in production
a component's source code is unavailable
a build's re-spin isn't practical
many changes of the same nature need to be applied
your application doesn't have enough bugs yet
4
5. Use cases
producing a build for developer's testing
simulate faults and test error recovery before deployment
to sneak-in to the production something your boss don't
need to know
5
6. Injecting away
pointcut execGetBlockFile() :
// the following will inject faults inside of the method in question
execution (* FSDataset.getBlockFile(..)) && !within(FSDatasetAspects +);
// This aspect specifies the logic of our fault point.
// In this case it simply throws DiskErrorException before invoking
// the method, specified by callGetBlockFile() pointcut
before() throws DiskErrorException : execGetBlockFile() {
if (ProbabilityModel.injectCriteria(FSDataset.class.getSimpleName())) {
LOG.info("Before the injection point");
Thread.dumpStack();
throw new DiskErrorException("FI: injected fault point at "
+ thisJoinPoint.getStaticPart().getSourceLocation());
}
}
6
7. Injecting away (intercept & mock)
pointcut callCreateUri() : call (URI FileDataServlet.createUri(
String, HdfsFileStatus, UserGroupInformation, ClientProtocol,
HttpServletRequest, String));
/** Replace host name with "localhost" for unit test environment. */
URI around () throws URISyntaxException : callCreateUri() {
final URI original = proceed();
LOG.info("FI: original uri = " + original);
final URI replaced = new URI(original.getScheme(),
original.getUserInfo(),
"localhost", original.getPort(), original.getPath(),
original.getQuery(),
original.getFragment()) ;
LOG.info("FI: replaced uri = " + replaced);
return replaced;
}
7
9. Why Fault Injection
Hadoop deals with many kinds of faults
Block corruption
Failures of disk, Datanode, Namenode, Clients, Jobtracker, Tasktrackers and Tasks
Varying rates of bandwidth and latency
These are hard to test
Unit tests mostly deal with specific single faults or patterns
Faults do not occur frequently and hard to reproduce
Need to inject fault in the real system (as opposed to a simulated system)
More info
http://wiki.apache.org/hadoop/HowToUseInjectionFramework
9
10. Usage models
An actor configures a Hadoop cluster and “dials-in” a desired faults then runs a
set of applications on the cluster.
Test the behavior of particular feature under faults
Test time and consistency of recovery at high rate of faults
Observe loss of data under certain pattern and frequency of faults
Observe performance/utilization
Note: can inject faults in the real system's (as opposed to a simulated system)
running jobs
An actor write/reuse a unit/function test using the fault inject framework to
introduce faults during the test
Recovery procedures testing (!)
10
11. Fault examples (Hdfs)
Link/communication failure and communication corruption
Namenode to Datanode communication
Client to Datanode communications
Client to Namenode communications
Namenode related failures
General slow downs
Edit logs slow downs
NFS-mounted volume is slow or not responding
Datanode related failures
Hardware corruption and data failures
Storage latencies and bandwidth anomalies
11
12. Fault examples (Mapreduce)
Task tracker
Lost task trackers
Tasks
Timeouts
Slow downs
Shuffle failures
Sort/merge failures
Local storage issues
JobTracker failures
Link communication failures and corruptions
12
19. Goals
Write cluster-based tests using Java object model
Automate many types of tests on real clusters:
Functional
System
Load
Recovery
More information
http://wiki.apache.org/hadoop/HowToUseSystemTestFramework
19
20. Main Features
Remote daemon Observability and Controllability APIs
Enables large cluster-based tests written in Java using JUnit
(TestNG) framework
Herriot is comprised of a library of utility APIs, and code
injections into Hadoop binaries
Assumes a deployed and instrumented cluster
Production build contains NO Herriot instrumentation
Supports fault injection
20
21. Major design considerations
Common
RPC-based utilities to control remote daemons
Daemons belong to different roles
Remote process management from Java: start/stop, change/push
configuration, etc.
HDFS and MR specific APIs on top of Common
21
22. Common Features
Get a daemon (a remote Hadoop process) current configuration
Get a daemon process info: thread#, heap, environment…
Ping a daemon (make sure it’s up and ready)
Get/list FileStatus of a path from a remote daemon
Deamon Log Inspection: Grep remote logs, count exceptions…
Cluster setup/tear down; restart
Change a daemon(s) configuration, push new configs…
22
23. Deployment Diagram
DN/TT host VM
Test host VM
JT host VM
NN host VM
Test Injected
Herriot
library Injected
code
Herriot
code
injections
23