Want to make rsyslog talk to some not-yet-supported output? With v8 it's extremly simple.This presentation contains everything you need to know about writing great plugins in ANY language (like perl or pyton).
The document compares the performance of NFS, GFS2, and OCFS2 filesystems on a high-performance computing cluster with nodes split across two datacenters. Generic load testing showed that NFS performance declined significantly with more than 6 nodes, while GFS2 maintained higher throughput. Further testing of GFS2 and OCFS2 using workload simulations modeling researcher usage found that OCFS2 outperformed GFS2 on small file operations and maintained high performance across nodes, making it the best choice for the shared filesystem needs of the project.
Video: https://www.youtube.com/watch?v=FJW8nGV4jxY and https://www.youtube.com/watch?v=zrr2nUln9Kk . Tutorial slides for O'Reilly Velocity SC 2015, by Brendan Gregg.
There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This tutorial explains methodologies for using these tools, and provides a tour of four tool types: observability, benchmarking, tuning, and static tuning. Many tools will be discussed, including top, iostat, tcpdump, sar, perf_events, ftrace, SystemTap, sysdig, and others, as well observability frameworks in the Linux kernel: PMCs, tracepoints, kprobes, and uprobes.
This tutorial is updated and extended on an earlier talk that summarizes the Linux performance tool landscape. The value of this tutorial is not just learning that these tools exist and what they do, but hearing when and how they are used by a performance engineer to solve real world problems — important context that is typically not included in the standard documentation.
Learn about structured logging with rsyslog and how it can be used to do actual format conversions. Include config samples for Linux and Windows log sources.
1. Graylog is log management software that can be used to collect and analyze logs from OpenStack. It provides centralized log management.
2. With hundreds of log files generating tens of gigabytes of logs daily across OpenStack controllers and compute nodes, effective log management is needed. Graylog can collect logs from all OpenStack services and components.
3. Using Graylog for OpenStack involves 3 simple steps - collecting logs from OpenStack services, analyzing the logs in Graylog, and using log information to understand and resolve issues. This allows incidents to be responded to quickly by finding the root cause in the logs.
This document summarizes an overview of symmetric encryption algorithms and modes, including one-time pads, stream ciphers, block ciphers like AES, and encryption modes like ECB, CBC, CTR, and XTS. It discusses the properties, advantages, and disadvantages of each. It also covers topics like AES-NI instructions, padding oracle attacks, parallelization of encryption/decryption, and attacks like plaintext recovery from ciphertext manipulation.
The document compares the performance of NFS, GFS2, and OCFS2 filesystems on a high-performance computing cluster with nodes split across two datacenters. Generic load testing showed that NFS performance declined significantly with more than 6 nodes, while GFS2 maintained higher throughput. Further testing of GFS2 and OCFS2 using workload simulations modeling researcher usage found that OCFS2 outperformed GFS2 on small file operations and maintained high performance across nodes, making it the best choice for the shared filesystem needs of the project.
Video: https://www.youtube.com/watch?v=FJW8nGV4jxY and https://www.youtube.com/watch?v=zrr2nUln9Kk . Tutorial slides for O'Reilly Velocity SC 2015, by Brendan Gregg.
There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This tutorial explains methodologies for using these tools, and provides a tour of four tool types: observability, benchmarking, tuning, and static tuning. Many tools will be discussed, including top, iostat, tcpdump, sar, perf_events, ftrace, SystemTap, sysdig, and others, as well observability frameworks in the Linux kernel: PMCs, tracepoints, kprobes, and uprobes.
This tutorial is updated and extended on an earlier talk that summarizes the Linux performance tool landscape. The value of this tutorial is not just learning that these tools exist and what they do, but hearing when and how they are used by a performance engineer to solve real world problems — important context that is typically not included in the standard documentation.
Learn about structured logging with rsyslog and how it can be used to do actual format conversions. Include config samples for Linux and Windows log sources.
1. Graylog is log management software that can be used to collect and analyze logs from OpenStack. It provides centralized log management.
2. With hundreds of log files generating tens of gigabytes of logs daily across OpenStack controllers and compute nodes, effective log management is needed. Graylog can collect logs from all OpenStack services and components.
3. Using Graylog for OpenStack involves 3 simple steps - collecting logs from OpenStack services, analyzing the logs in Graylog, and using log information to understand and resolve issues. This allows incidents to be responded to quickly by finding the root cause in the logs.
This document summarizes an overview of symmetric encryption algorithms and modes, including one-time pads, stream ciphers, block ciphers like AES, and encryption modes like ECB, CBC, CTR, and XTS. It discusses the properties, advantages, and disadvantages of each. It also covers topics like AES-NI instructions, padding oracle attacks, parallelization of encryption/decryption, and attacks like plaintext recovery from ciphertext manipulation.
Ansible Night in Tokyo 2018.04 でのLT発表資料。
https://ansible-users.connpass.com/event/84907
Ansibleから日本語Widows環境を操作する際に、CP932でのコマンド出力が壊れてしまう件とそのワークアラウンドについてです。
This document provides an overview of five steps to improve PostgreSQL performance: 1) hardware optimization, 2) operating system and filesystem tuning, 3) configuration of postgresql.conf parameters, 4) application design considerations, and 5) query tuning. The document discusses various techniques for each step such as selecting appropriate hardware components, spreading database files across multiple disks or arrays, adjusting memory and disk configuration parameters, designing schemas and queries efficiently, and leveraging caching strategies.
Live Memory Forensics on Android devicesNikos Gkogkos
This presentation deals with some RAM forensics on the Android OS using the LiME tool for getting a RAM dump and the Volatility framework for the analysis part!
Nginx is a web server and proxy server that is modular, allowing users to specify which modules they want. It has a main configuration file located at /etc/nginx/nginx.conf that includes other configuration files. Nginx uses server blocks and location directives to map URI requests to resources. It can serve static content from a specified root directory or act as a proxy server by forwarding requests to another server. Rewrite rules using the return or rewrite directives allow changing URLs in client requests to redirect users.
Extreme Linux Performance Monitoring and TuningMilind Koyande
This document provides an introduction to monitoring Linux system performance. It discusses determining the type of application running and establishing a baseline of typical system usage. Key CPU concepts are then outlined such as hardware interrupts, soft interrupts, real-time threads and kernel/user threads. Context switches between threads and the thread scheduling queue are also introduced. The goal is to understand typical system behavior and identify any bottlenecks.
Native Android Userspace part of the Embedded Android Workshop at Linaro Conn...Opersys inc.
This document provides an overview of the native Android user-space environment, including:
1) The filesystem layout and key directories like /system, /data, and /vendor.
2) How the build system determines where files are placed.
3) The adb debugging tool and its capabilities.
4) Common command line tools and properties.
5) The init process and ueventd daemon.
6) Libraries like Bionic and integration options for legacy systems.
This talk introduces and discusses a novel, mostly unpublished technique to successfully attack websites that are applied with state-of-the-art XSS protection. This attack labeled Mutation-XSS (mXSS) is capable of bypassing high-end filter systems by utilizing the browser and its unknown capabilities - every single f***** one of them. We analyzed the type and number of high-profile websites and applications that are affected by this kind of attack. Several live demos during the presentation will share these impressions and help understanding, what mXSS is, why mXSS is possible and why it is of importance for defenders as well as professional attackers to understand and examine mXSS even further. The talk wraps up several years of research on this field, shows the abhorrent findings, discusses the consequences and delivers a step-by-step guide on how to protect against this kind of mayhem - with a strong focus on feasibility and scalability.
Windows 10 Nt Heap Exploitation (English version)Angel Boy
The document discusses the Windows memory allocator and heap exploitation. It describes the core components and data structures of the NT heap, including the _HEAP structure, _HEAP_ENTRY chunks, BlocksIndex structure, and FreeLists. It also explains the differences between the backend and frontend allocators as well as how chunks of different sizes are managed.
ATF(ARM Trusted Firmware)は、ARMv8では重要なソフトウェア。
全体を利用するのではなく、その一部を利用可能。
この資料では、BL31(EL3 Runtime Firmware)を単体で使う場合、どうすればいいのかを、Xilinx社のZynq UltraScale+ MPSoCを例に説明しています。
ATF (ARM Trusted Firmware) is an important software in ARMv8.
Instead of using the whole, part of it is available.
This document explains how to do when using BL31 (EL3 Runtime Firmware) alone, for example, with Xilinx's Zynq UltraScale + MPSoC.
This document provides an overview of Postgresql, including its history, capabilities, advantages over other databases, best practices, and references for further learning. Postgresql is an open source relational database management system that has been in development for over 30 years. It offers rich SQL support, high performance, ACID transactions, and extensive extensibility through features like JSON, XML, and programming languages.
This document provides an overview of troubleshooting streaming replication in PostgreSQL. It begins with introductions to write-ahead logging and replication internals. Common troubleshooting tools are then described, including built-in views and functions as well as third-party tools. Finally, specific troubleshooting cases are discussed such as replication lag, WAL bloat, recovery conflicts, and high CPU recovery usage. Throughout, examples are provided of how to detect and diagnose issues using the various tools.
RSYSLOG v8 improvements and how to write plugins in any language.Rainer Gerhards
RSYSLOG is a next generation log processing tool. In the frist part, we will explain the new RSYSLOG v8 engine, its motivation and its benefits. Learn, for example, why writing to Elasticsearch is much faster with the new engine. We will describe the tuning parameters vital for making best use of the new features.
In the second part we will explain how to write RSYSLOG plugins in any language. Traditionally, writing rsyslog plugins has been considered quite hard, with at least C knowledge necessary. In v8, we have introduced new interfaces which make it possible to write plugins in any language - be it Python, Perl or Java. Even bash will do. In essence, this is a great tool for any admin to add special needs with just a bit of scripting. We will proivde concrete instructions on how to write a plugin, point to read-to-copy samples and tell how to integrate this into rsyslog.
NOTE: This is my LinuxTag Berlin 2014 talk.
This is the talk I have given on Fedora Developer's Conference 2014 in Brno. It provides insight into the security features we added to rsyslog v7, integration into systemd journal, enhancements of the v8 engine and a glimpse at how to write rsyslog plugins in languages other than C.
Ansible Night in Tokyo 2018.04 でのLT発表資料。
https://ansible-users.connpass.com/event/84907
Ansibleから日本語Widows環境を操作する際に、CP932でのコマンド出力が壊れてしまう件とそのワークアラウンドについてです。
This document provides an overview of five steps to improve PostgreSQL performance: 1) hardware optimization, 2) operating system and filesystem tuning, 3) configuration of postgresql.conf parameters, 4) application design considerations, and 5) query tuning. The document discusses various techniques for each step such as selecting appropriate hardware components, spreading database files across multiple disks or arrays, adjusting memory and disk configuration parameters, designing schemas and queries efficiently, and leveraging caching strategies.
Live Memory Forensics on Android devicesNikos Gkogkos
This presentation deals with some RAM forensics on the Android OS using the LiME tool for getting a RAM dump and the Volatility framework for the analysis part!
Nginx is a web server and proxy server that is modular, allowing users to specify which modules they want. It has a main configuration file located at /etc/nginx/nginx.conf that includes other configuration files. Nginx uses server blocks and location directives to map URI requests to resources. It can serve static content from a specified root directory or act as a proxy server by forwarding requests to another server. Rewrite rules using the return or rewrite directives allow changing URLs in client requests to redirect users.
Extreme Linux Performance Monitoring and TuningMilind Koyande
This document provides an introduction to monitoring Linux system performance. It discusses determining the type of application running and establishing a baseline of typical system usage. Key CPU concepts are then outlined such as hardware interrupts, soft interrupts, real-time threads and kernel/user threads. Context switches between threads and the thread scheduling queue are also introduced. The goal is to understand typical system behavior and identify any bottlenecks.
Native Android Userspace part of the Embedded Android Workshop at Linaro Conn...Opersys inc.
This document provides an overview of the native Android user-space environment, including:
1) The filesystem layout and key directories like /system, /data, and /vendor.
2) How the build system determines where files are placed.
3) The adb debugging tool and its capabilities.
4) Common command line tools and properties.
5) The init process and ueventd daemon.
6) Libraries like Bionic and integration options for legacy systems.
This talk introduces and discusses a novel, mostly unpublished technique to successfully attack websites that are applied with state-of-the-art XSS protection. This attack labeled Mutation-XSS (mXSS) is capable of bypassing high-end filter systems by utilizing the browser and its unknown capabilities - every single f***** one of them. We analyzed the type and number of high-profile websites and applications that are affected by this kind of attack. Several live demos during the presentation will share these impressions and help understanding, what mXSS is, why mXSS is possible and why it is of importance for defenders as well as professional attackers to understand and examine mXSS even further. The talk wraps up several years of research on this field, shows the abhorrent findings, discusses the consequences and delivers a step-by-step guide on how to protect against this kind of mayhem - with a strong focus on feasibility and scalability.
Windows 10 Nt Heap Exploitation (English version)Angel Boy
The document discusses the Windows memory allocator and heap exploitation. It describes the core components and data structures of the NT heap, including the _HEAP structure, _HEAP_ENTRY chunks, BlocksIndex structure, and FreeLists. It also explains the differences between the backend and frontend allocators as well as how chunks of different sizes are managed.
ATF(ARM Trusted Firmware)は、ARMv8では重要なソフトウェア。
全体を利用するのではなく、その一部を利用可能。
この資料では、BL31(EL3 Runtime Firmware)を単体で使う場合、どうすればいいのかを、Xilinx社のZynq UltraScale+ MPSoCを例に説明しています。
ATF (ARM Trusted Firmware) is an important software in ARMv8.
Instead of using the whole, part of it is available.
This document explains how to do when using BL31 (EL3 Runtime Firmware) alone, for example, with Xilinx's Zynq UltraScale + MPSoC.
This document provides an overview of Postgresql, including its history, capabilities, advantages over other databases, best practices, and references for further learning. Postgresql is an open source relational database management system that has been in development for over 30 years. It offers rich SQL support, high performance, ACID transactions, and extensive extensibility through features like JSON, XML, and programming languages.
This document provides an overview of troubleshooting streaming replication in PostgreSQL. It begins with introductions to write-ahead logging and replication internals. Common troubleshooting tools are then described, including built-in views and functions as well as third-party tools. Finally, specific troubleshooting cases are discussed such as replication lag, WAL bloat, recovery conflicts, and high CPU recovery usage. Throughout, examples are provided of how to detect and diagnose issues using the various tools.
RSYSLOG v8 improvements and how to write plugins in any language.Rainer Gerhards
RSYSLOG is a next generation log processing tool. In the frist part, we will explain the new RSYSLOG v8 engine, its motivation and its benefits. Learn, for example, why writing to Elasticsearch is much faster with the new engine. We will describe the tuning parameters vital for making best use of the new features.
In the second part we will explain how to write RSYSLOG plugins in any language. Traditionally, writing rsyslog plugins has been considered quite hard, with at least C knowledge necessary. In v8, we have introduced new interfaces which make it possible to write plugins in any language - be it Python, Perl or Java. Even bash will do. In essence, this is a great tool for any admin to add special needs with just a bit of scripting. We will proivde concrete instructions on how to write a plugin, point to read-to-copy samples and tell how to integrate this into rsyslog.
NOTE: This is my LinuxTag Berlin 2014 talk.
This is the talk I have given on Fedora Developer's Conference 2014 in Brno. It provides insight into the security features we added to rsyslog v7, integration into systemd journal, enhancements of the v8 engine and a glimpse at how to write rsyslog plugins in languages other than C.
Slides for GUUG FFG2018 talk on rsyslog and containers. Describes the initial steps the rsyslog project took towards containers, uses cases seen by the team, problems we have seen and use of docker inside rsyslog's CI.
This document provides an introduction to Node.js, a framework for building scalable server-side applications with asynchronous JavaScript. It discusses what Node.js is, how it uses non-blocking I/O and events to avoid wasting CPU cycles, and how external Node modules help create a full JavaScript stack. Examples are given of using Node modules like Express for building RESTful APIs and Socket.IO for implementing real-time features like chat. Best practices, limitations, debugging techniques and references are also covered.
From Ansible's website: "Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs."
This introduction is based on ansible official docs, capturing most important information to make it easy to understand Ansible main concepts.
Docker networking uses Linux bridges which only connect containers on the same host. Containers cannot communicate across hosts. NAT and iptables are used to expose container ports to external networks. Docker's process model runs containers as child processes of the Docker daemon, so if the daemon fails all containers will fail. Alternative container runtimes like Rocket have a more robust process model.
Docker and Containers are proven solutions, but are they ready to replace your current deployment? And more importantly, are you aware of the changes you'll have to make to accommodate them? Are there any risks involved? This talk will answer these questions and talk about how to plan, automate, build, deploy, and orchestrate the whole process.
Spenser Reinhardt's presentation on Detecting Security Breaches With Docker, Honeypots, & Nagios.
The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference
The document discusses new features in version 0.9.4 of the DivConq file transfer software, including file tasks that can be triggered by uploads, scheduling, or file system events. It introduces dcScript, the scripting language that allows users to string together various file operations and tasks. Key points include that dcScript scripts can run asynchronously, optimize file operations through in-memory streaming rather than disk reads/writes, and offer features to simplify complex multi-step file tasks. The document provides examples of using dcScript to encrypt, compress, split and transfer files with just a few lines of code.
Deep Postgres Extensions in Rust | PGCon 2019 | Jeff DavisCitus Data
Postgres relies heavily on an extension ecosystem, but that is almost 100% dependent on C; which cuts out developers, libraries, and ideas from the world of Postgres. postgres-extension.rs changes that by supporting development of extensions in Rust. Rust is a memory-safe language that integrates nicely in any environment, has powerful libraries, a vibrant ecosystem, and a prolific developer community.
Rust is a unique language because it supports high-level features but all the magic happens at compile-time, and the resulting code is not dependent on an intrusive or bulky runtime. That makes it ideal for integrating with postgres, which has a lot of its own runtime, like memory contexts and signal handlers. postgres-extension.rs offers this integration, allowing the development of extensions in rust, even if deeply-integrated into the postgres internals, and helping handle tricky issues like error handling. This is done through a collection of Rust function declarations, macros, and utility functions that allow rust code to call into postgres, and safely handle resulting errors.
Logstash is a tool for managing logs that allows for input, filter, and output plugins to collect, parse, and deliver logs and log data. It works by treating logs as events that are passed through the input, filter, and output phases, with popular plugins including file, redis, grok, elasticsearch and more. The document also provides guidance on using Logstash in a clustered configuration with an agent and server model to optimize log collection, processing, and storage.
This document discusses optimizing a development process by migrating from a VirtualBox setup to using Warden, an open source Docker development environment. It outlines issues with the old VirtualBox setup such as slow performance and high resource usage. Requirements for a new setup include fast environments, security, ability to run multiple projects, and compatibility across operating systems. Several solutions were analyzed and Warden was selected for its features like Traefik for routing, Portainer for visibility, and easy project migration. The document demonstrates how to install Warden and migrate a local Magento 2 project, including commands for common development tasks.
Save 10% off ANY FITC event with discount code 'slideshare' - http://www.fitc.ca
OVERVIEW
Node.js is a runtime environment and library for running JavaScript applications outside the browser. Node.js is mostly used to run real-time server applications and shines through its performance using non-blocking I/O and asynchronous events. This talk will introduce you to Node.js by showcasing the environment and its two most popular libraries: express and socket.io.
TARGET AUDIENCE
Beginner web developers
ASSUMED AUDIENCE KNOWLEDGE
Working knowledge of JavaScript and HTML5.
OBJECTIVE
Learn how to build a chat engine using Node.js and WebSockets.
FIVE THINGS AUDIENCE MEMBERS WILL LEARN
Node.js environment and basics
Node Package Manager overview
Web Framework, express, basics
WebSockets and Socket.io basics
Building a chat engine using Node.js
This document summarizes Fluentd v1.0 and provides details about its new features and release plan. It notes that Fluentd v1.0 will provide stable APIs and compatibility with previous versions while improving plugin APIs, adding Windows and multicore support, and increasing event time resolution to nanoseconds. The release is planned for Q3 2017 to allow feedback on v0.14 before finalizing v1.0 features.
Devoops: DoJ Annual Cybersecurity Training Symposium Edition 2015Chris Gates
This document summarizes a talk given on DevOps infrastructure security. It discusses how various DevOps tools like GitHub, Jenkins, AWS config files, Chef, and in-memory databases like Redis and Memcache can expose sensitive information if not properly secured. Specific issues covered include exposed Git repositories, weak default credentials, plaintext storage of secrets, and lack of authentication. The document provides recommendations on securing these tools such as enabling authentication, upgrading versions, and segmenting tools from public access.
Gianluca Varisco - DevOoops (Increase awareness around DevOps infra security)Codemotion
DevOps is increasingly blending the work of both application and network security professionals. In a quest to move faster, organisations can end up creating security vulnerabilities using the tools and products meant to protect them. What happens when these tools are used insecurely or - even worse - they are just insecure? Technologies discussed will encompass AWS, Puppet, Hudson/Jenkins, Vagrant, Docker and much, much more. Everything from common misconfigurations to remote code execution.
O365Con19 - Things I've Learned While Building a Product on SharePoint Modern...NCCOMMS
This document provides an overview of lessons learned from building products on SharePoint Modern. It discusses SharePoint Framework architecture, localization strategies, versioning approaches, challenges of supporting older browsers and environments, and the importance of testing. Key recommendations include using semantic versioning, sharing common code via libraries, automating releases through CI/CD, and testing across different environments.
Sicherheit im Internet - Wie kann man sich schützen?Rainer Gerhards
Einfache Methoden mit denen sich auch "Normalos" im Internet schützen können. Leicht erklärt. Mit Detailinfos unter https://www.rainer-gerhards.de/sicherheit-im-internet/
The rsyslog project has adopted a new versioning and release cycle scheme to provide features to users more quickly while maintaining stability. The major changes are:
1) Stable releases will now occur every 6 weeks rather than distinguishing between stable and development releases.
2) The minor version number will increment with each new release rather than distinguishing between odd and even numbers.
3) Development versions are now identified by their git commit hash rather than a version number.
This allows new features to reach users more rapidly without compromising stability through more frequent testing.
Which enhancements does the rsyslog v8 engine contain? This short presentation focuses on those changes in the output engine that greatly helps to improve performance. We describe what exactly was changed, how this looks in real code and what developers need to know before upgrading a plugin to v8.
Wetterbeobachtung - Ein Vortrag für die GrundschuleRainer Gerhards
Der Vortrag erklärt in einfachen Worten und anhand von Beispielbildern die Grundlagen der Wetterbeobachtung und eignet sich für den Sachkunde-Unterricht.
Do rsyslog and the journal cooperate? If so, how? This is the presentation from the LinuxTag 2013 conference. It details the rsyslog team's current position on the journal, how it affected rsyslog, what is being done for integration and some notes about how to configure rsyslog to do things that the journal announcement claimed to be impossible.
Do rsyslog and the journal cooperate? If so, how? This is the paper from the LinuxTag 2013 conference. It details the rsyslog team's current position on the journal, how it affected rsyslog, what is being done for integration and some notes about how to configure rsyslog to do things that the journal announcemnt claimed to be impossible.
CEE Log Integrity and the "Counterpane Paper"Rainer Gerhards
What are the problems in signing log data while it traverses the network? This paper was originaly written to support CEE discussion on log integrity but also clearly describes both the problem and a partial, but practical solution to it. While it uses some CEE terms, it should be easy to follow without CEE knowledge.
The paper was written in December 2010 and uploaded in May 2013 to make it easier to obtain it.
This presentation covers the state of the syslog protocol and its standardization as of 2005. It was created for and held at Linuxtag in Germany (and as such is in German).
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...DanBrown980551
This LF Energy webinar took place June 20, 2024. It featured:
-Alex Thornton, LF Energy
-Hallie Cramer, Google
-Daniel Roesler, UtilityAPI
-Henry Richardson, WattTime
In response to the urgency and scale required to effectively address climate change, open source solutions offer significant potential for driving innovation and progress. Currently, there is a growing demand for standardization and interoperability in energy data and modeling. Open source standards and specifications within the energy sector can also alleviate challenges associated with data fragmentation, transparency, and accessibility. At the same time, it is crucial to consider privacy and security concerns throughout the development of open source platforms.
This webinar will delve into the motivations behind establishing LF Energy’s Carbon Data Specification Consortium. It will provide an overview of the draft specifications and the ongoing progress made by the respective working groups.
Three primary specifications will be discussed:
-Discovery and client registration, emphasizing transparent processes and secure and private access
-Customer data, centering around customer tariffs, bills, energy usage, and full consumption disclosure
-Power systems data, focusing on grid data, inclusive of transmission and distribution networks, generation, intergrid power flows, and market settlement data
Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots - like ChatGPT - can converse on any topic, task-oriented chatbots - the focus of this paper - are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots.
To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.
"Scaling RAG Applications to serve millions of users", Kevin GoedeckeFwdays
How we managed to grow and scale a RAG application from zero to thousands of users in 7 months. Lessons from technical challenges around managing high load for LLMs, RAGs and Vector databases.
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving
What began over 115 years ago as a supplier of precision gauges to the automotive industry has evolved into being an industry leader in the manufacture of product branding, automotive cockpit trim and decorative appliance trim. Value-added services include in-house Design, Engineering, Program Management, Test Lab and Tool Shops.
QA or the Highway - Component Testing: Bridging the gap between frontend appl...zjhamm304
These are the slides for the presentation, "Component Testing: Bridging the gap between frontend applications" that was presented at QA or the Highway 2024 in Columbus, OH by Zachary Hamm.
ScyllaDB is making a major architecture shift. We’re moving from vNode replication to tablets – fragments of tables that are distributed independently, enabling dynamic data distribution and extreme elasticity. In this keynote, ScyllaDB co-founder and CTO Avi Kivity explains the reason for this shift, provides a look at the implementation and roadmap, and shares how this shift benefits ScyllaDB users.
This talk will cover ScyllaDB Architecture from the cluster-level view and zoom in on data distribution and internal node architecture. In the process, we will learn the secret sauce used to get ScyllaDB's high availability and superior performance. We will also touch on the upcoming changes to ScyllaDB architecture, moving to strongly consistent metadata and tablets.
Getting the Most Out of ScyllaDB Monitoring: ShareChat's TipsScyllaDB
ScyllaDB monitoring provides a lot of useful information. But sometimes it’s not easy to find the root of the problem if something is wrong or even estimate the remaining capacity by the load on the cluster. This talk shares our team's practical tips on: 1) How to find the root of the problem by metrics if ScyllaDB is slow 2) How to interpret the load and plan capacity for the future 3) Compaction strategies and how to choose the right one 4) Important metrics which aren’t available in the default monitoring setup.
"Choosing proper type of scaling", Olena SyrotaFwdays
Imagine an IoT processing system that is already quite mature and production-ready and for which client coverage is growing and scaling and performance aspects are life and death questions. The system has Redis, MongoDB, and stream processing based on ksqldb. In this talk, firstly, we will analyze scaling approaches and then select the proper ones for our system.
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...GlobalLogic Ukraine
Під час доповіді відповімо на питання, навіщо потрібно підвищувати продуктивність аплікації і які є найефективніші способи для цього. А також поговоримо про те, що таке кеш, які його види бувають та, основне — як знайти performance bottleneck?
Відео та деталі заходу: https://bit.ly/45tILxj
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...Jason Yip
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...AlexanderRichford
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation Functions to Prevent Interaction with Malicious QR Codes.
Aim of the Study: The goal of this research was to develop a robust hybrid approach for identifying malicious and insecure URLs derived from QR codes, ensuring safe interactions.
This is achieved through:
Machine Learning Model: Predicts the likelihood of a URL being malicious.
Security Validation Functions: Ensures the derived URL has a valid certificate and proper URL format.
This innovative blend of technology aims to enhance cybersecurity measures and protect users from potential threats hidden within QR codes 🖥 🔒
This study was my first introduction to using ML which has shown me the immense potential of ML in creating more secure digital environments!
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor IvaniukFwdays
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
2. Output plugins in rsyslog
• Output plugins permit to connect to a variety of log
destinations
• Pre-v8, plugins needed to be internal modules
▫ Loaded into rsyslog process space
▫ Must be written in C (or link to it)
▫ Deeply tied into rsyslog infrastructure
• with v8, we officially support external plugins
▫ Written in any language (Python, Perl, Java, ...)
▫ Have their own process
▫ Relatively de-coupled
Rainer Gerhards, http://blog.gerhards.net
4. External Interface Design Goals
• Keep it stupid simple
▫ Must support almost any language
▫ Dumb easy to use even for novice programmer
▫ Do not require explicit threading
• Speed is NOT the most important goal
▫ Don't make it unnecessarily slow
▫ Many real-world log destinations are slow in any case
(like when you connect via http...)
▫ Focus on “enable to build solution”
▫ If necessary, conversion to internal module can be
done later
Rainer Gerhards, http://blog.gerhards.net
5. Interface Details: communication
• use pipes
• stdin
▫ one message per line
▫ format can be customized via rsyslog templates
▫ multi-line messags via JSON
• stdout/stderr
▫ Must NOT be written in initial version
▫ Will later convey back state information via plain text
(e.g. “ERR”, “ERRMSG:xxx”, ...)
• JSON as input format is recommended
Rainer Gerhards, http://blog.gerhards.net
6. Interface Details: Threading
• Do NOT care about threading
• Write app according to single-thread paradigm
• rsyslog will spawn multiple instances of your plugin
if there is need to do so
▫
▫
▫
▫
Happens based on config in busy cases
Works well in most cases (e.g. http connects)
Can be disabled if necessary
If your program can run in multiple ter-minal
sessions concurrently, it can also be run as
multiple rsyslog action instances.
Rainer Gerhards, http://blog.gerhards.net
7. Startup & Termination
•
•
•
•
•
•
rsyslog will startup the plugin automatically
Plugin needs to read stdin until EOF
Do NOT terminate before EOF is reached
On EOF, cleanup and terminate
If the plugin dies, rsyslog restarts a new instance
Some signals (like sigint) are blocked and should
remain so
Rainer Gerhards, http://blog.gerhards.net
8. Skeletons
• The rsyslog project provides sample plugin
skeletons
• Available in ./plugins/external/skeletons
• These contain
▫ the necessary plumbing
▫ often a kind of abstraction layer to make writing
plugins even easier
▫ often performance-enhancement features
• Can simply be copied to create your own plugins,
don't care about the (minimal) plumbing!
Rainer Gerhards, http://blog.gerhards.net
9. Example: Python Skeleton
• Handles all interface plumbing
• Uses “eventHandlers” to call the acutal app coding
• Is used as a basis, for example, for the Solr output
plugin
• Available at
https://github.com/rsyslog/rsyslog/blob/master/plugin
Rainer Gerhards, http://blog.gerhards.net
10. onInit() Handler
• Called when the plugin is initially loaded
• Ready outbound connections, files, etc...
def onInit():
""" Do everything that is needed to initialize processing
(e.g. open files, create handles, connect to systems...)
"""
global outfile
outfile = open("/tmp/logfile", "w")
Rainer Gerhards, http://blog.gerhards.net
11. onReceive()
• Called when new messages arrive
• Receives one or more message via Python list object
--> e.g. place them into one transaction, HTTP
request, ...
def onReceive(msgs):
"""This is the entry point where actual work needs to be done. It receives
a list with all messages pulled from rsyslog. The list is of variable
length, but contains all messages that are currently available. It is
suggest NOT to use any further buffering, as we do not know when the
next message will arrive. It may be in a nanosecond from now, but it
may also be in three hours...
"""
global outfile
for msg in msgs:
outfile.write(msg)
Rainer Gerhards, http://blog.gerhards.net
12. onExit()
• Called immediately before the plugin terminates
• Gurantees that no more messages arrive
• Used to cleanup connections, write some final
records, ...
def onExit():
""" Do everything that is needed to finish processing (e.g.
close files, handles, disconnect from systems...). This is
being called immediately before exiting.
"""
global outfile
outfile.close()
Rainer Gerhards, http://blog.gerhards.net
13. Call to Action
• If you need to send logs to a destination that is not
yet supported, you can quickly write an external
plugin – in any language you know!
• Writing rsyslog plugins is easy
▫ If there is already a skeleton for your language, copy it
and add your app-specific code
▫ If not ... no problem, the interface is dumb easy
If you can write a script that reads stdin and does
something useful with it, you can also write a
rsyslog plugin!
Rainer Gerhards, http://blog.gerhards.net