At LinkedIn we run lots of Java services on Linux boxes. Java and Linux are a perfect pair. Except when they're not; then there's fireworks. This talk describes 5 situations we encountered where Java interacted with normal Linux behavior to create stunningly sub-optimal application behavior like minutes-long GC pauses. We'll deep dive to show What Java Got Wrong, why Linux behaves the way it does, and how the two can conspire to ruin your day. Finally we'll examine actual code samples showing how we fixed or hid the problems.
NovaProva, a new generation unit test framework for C programsGreg Banks
I wrote NovaProva because I was sick of writing unit tests using the venerable but clunky CUnit library. CUnit looked easy when I started writing unit tests but I soon discovered it's limitations and ended up screaming in frustration. Meanwhile over 18 months the Cyrus IMAP server project has gone from 0 unit tests to 461, of which 277 are written in C with CUnit. So that's a big itch!
Slides used for an internal training. This explains how to generate Flame Graphs using Java Flight Recorder dumps. There is also an example to use Linux "perf_events" to generate a Java Mixed-Mode Flame Graph.
NovaProva, a new generation unit test framework for C programsGreg Banks
I wrote NovaProva because I was sick of writing unit tests using the venerable but clunky CUnit library. CUnit looked easy when I started writing unit tests but I soon discovered it's limitations and ended up screaming in frustration. Meanwhile over 18 months the Cyrus IMAP server project has gone from 0 unit tests to 461, of which 277 are written in C with CUnit. So that's a big itch!
Slides used for an internal training. This explains how to generate Flame Graphs using Java Flight Recorder dumps. There is also an example to use Linux "perf_events" to generate a Java Mixed-Mode Flame Graph.
Lecture 7: Introduction to Quantum Chemical Simulation graduate course taught at MIT in Fall 2014 by Heather Kulik. This course covers: wavefunction theory, density functional theory, force fields and molecular dynamics and sampling.
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)Valeriy Kravchuk
eBPF on Linux 4.9+ is probably the best way to study performance problems. Basic usage of ftrace interface, bcc tools and bpftrace, as well as main bpftrace features and commands are presented. Several typical use cases
(including adding dynamic probes to MariaDB servers, working with stack traces and creating Flame Graphs) are discussed.
This talk is all about the Berkeley Packet Filters (BPF) and their uses in Linux.
Agenda:
* What is a BPF and why do we need it?
* Writing custom BPFs
* Notes on BPF implementation in the kernel
* Usage examples: SOCKET_FILTER & seccomp
Speaker:
Kfir Gollan, senior embedded software developer, Linux kernel hacker and software team leader.
Golang Performance : microbenchmarks, profilers, and a war storyAerospike
Slides for Brian Bulkowski's talk about Golang performance:
microbenchmarks, profilers, and a war story about optimizing the Aerospike Database Go client.
http://www.meetup.com/Go-lang-Developers-NYC/events/216650022/
Kernel Recipes 2019 - CVEs are dead, long live the CVE!Anne Nicolas
For the Linux kernel, CVEs do not work at all given the rate of fixes being applied and rapidly backported and pushed to users through a huge variety of different ways. The average “request to fix” date for Linux CVEs is -100 days, showing that either no one cares about CVEs for Linux, or engineers are using them to game their internal release processes, or no one happens to notice when the kernel developers resolve an issue, or all of the above. This talk will go into the problems with CVEs when it comes to a fast moving project like Linux, and show the decentralized solution that we have been using for the past 14 years instead. All other open source projects are encouraged to use these same methods to help resolve the problems that CVEs have.
Software Profiling: Java Performance, Profiling and FlamegraphsIsuru Perera
Guest lecture at University of Colombo School of Computing on 30th May 2018
Covers following topics:
Software Profiling
Measuring Performance
Java Garbage Collection
Sampling vs Instrumentation
Java Profilers. Java Flight Recorder
Java Just-in-Time (JIT) compilation
Flame Graphs
Linux Profiling
Linux kernel tracing superpowers in the cloudAndrea Righi
The Linux 4.x series introduced a new powerful engine of programmable tracing (BPF) that allows to actually look inside the kernel at runtime. This talk will show you how to exploit this engine in order to debug problems or identify performance bottlenecks in a complex environment like a cloud. This talk will cover the latest Linux superpowers that allow to see what is happening “under the hood” of the Linux kernel at runtime. I will explain how to exploit these “superpowers” to measure and trace complex events at runtime in a cloud environment. For example, we will see how we can measure latency distribution of filesystem I/O, details of storage device operations, like individual block I/O request timeouts, or TCP buffer allocations, investigating stack traces of certain events, identify memory leaks, performance bottlenecks and a whole lot more.
Tracing MariaDB server with bpftrace - MariaDB Server Fest 2021Valeriy Kravchuk
Bpftrace is a relatively new eBPF-based open source tracer for modern Linux versions (kernels 5.x.y) that is useful for analyzing production performance problems and troubleshooting software. Basic usage of the tool, as well as bpftrace one liners and advanced scripts useful for MariaDB DBAs are presented. Problems of MariaDB Server dynamic tracing with bpftrace and some possible solutions and alternative tracing tools are discussed.
Kernel Recipes 2019 - Formal modeling made easyAnne Nicolas
Modeling parts of Linux has become a recurring topic. For instance, the memory model, the model for PREEMPT_RT synchronization, and so on. But the term "formal model" causes panic for most of the developers. Mainly because of the complex notations and reasoning that involves formal languages. It seems to be a very theoretical thing, far from our day-by-day reality.
Believe me. Modeling can be more practical than you might guess!
This talk will discuss the challenges and benefits of modeling, based on the experience of developing the PREEMPT_RT model. It will present a methodology for modeling the Linux behavior as Finite-State Machines (automata), using terms that are very known by kernel developers: tracing events! With the particular focus on how to use models for the formal verification of Linux kernel, at runtime, with low overhead, and in many cases, without even modifying Linux kernel!
Daniel Bristot de Oliveira
All of Your Network Monitoring is (probably) Wrongice799
Monitorama 2016 talk about network monitoring covering topics like network device drivers, ethtool, and some interesting bugs/features.
For more information about monitoring and tuning the entire Linux network stack, see: blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/
Kernel Recipes 2016 - Kernel documentation: what we have and where it’s goingAnne Nicolas
The Linux kernel features an extensive array of, to put it kindly, somewhat disorganized documentation. A significant effort is underway to make things better, though. This talk will review the state of kernel documentation, cover the changes that are being made (including the adoption of a new system for formatted documentation), and discuss how interested developers can help.
Jonathan Corbet, LWN.net
Lecture 7: Introduction to Quantum Chemical Simulation graduate course taught at MIT in Fall 2014 by Heather Kulik. This course covers: wavefunction theory, density functional theory, force fields and molecular dynamics and sampling.
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)Valeriy Kravchuk
eBPF on Linux 4.9+ is probably the best way to study performance problems. Basic usage of ftrace interface, bcc tools and bpftrace, as well as main bpftrace features and commands are presented. Several typical use cases
(including adding dynamic probes to MariaDB servers, working with stack traces and creating Flame Graphs) are discussed.
This talk is all about the Berkeley Packet Filters (BPF) and their uses in Linux.
Agenda:
* What is a BPF and why do we need it?
* Writing custom BPFs
* Notes on BPF implementation in the kernel
* Usage examples: SOCKET_FILTER & seccomp
Speaker:
Kfir Gollan, senior embedded software developer, Linux kernel hacker and software team leader.
Golang Performance : microbenchmarks, profilers, and a war storyAerospike
Slides for Brian Bulkowski's talk about Golang performance:
microbenchmarks, profilers, and a war story about optimizing the Aerospike Database Go client.
http://www.meetup.com/Go-lang-Developers-NYC/events/216650022/
Kernel Recipes 2019 - CVEs are dead, long live the CVE!Anne Nicolas
For the Linux kernel, CVEs do not work at all given the rate of fixes being applied and rapidly backported and pushed to users through a huge variety of different ways. The average “request to fix” date for Linux CVEs is -100 days, showing that either no one cares about CVEs for Linux, or engineers are using them to game their internal release processes, or no one happens to notice when the kernel developers resolve an issue, or all of the above. This talk will go into the problems with CVEs when it comes to a fast moving project like Linux, and show the decentralized solution that we have been using for the past 14 years instead. All other open source projects are encouraged to use these same methods to help resolve the problems that CVEs have.
Software Profiling: Java Performance, Profiling and FlamegraphsIsuru Perera
Guest lecture at University of Colombo School of Computing on 30th May 2018
Covers following topics:
Software Profiling
Measuring Performance
Java Garbage Collection
Sampling vs Instrumentation
Java Profilers. Java Flight Recorder
Java Just-in-Time (JIT) compilation
Flame Graphs
Linux Profiling
Linux kernel tracing superpowers in the cloudAndrea Righi
The Linux 4.x series introduced a new powerful engine of programmable tracing (BPF) that allows to actually look inside the kernel at runtime. This talk will show you how to exploit this engine in order to debug problems or identify performance bottlenecks in a complex environment like a cloud. This talk will cover the latest Linux superpowers that allow to see what is happening “under the hood” of the Linux kernel at runtime. I will explain how to exploit these “superpowers” to measure and trace complex events at runtime in a cloud environment. For example, we will see how we can measure latency distribution of filesystem I/O, details of storage device operations, like individual block I/O request timeouts, or TCP buffer allocations, investigating stack traces of certain events, identify memory leaks, performance bottlenecks and a whole lot more.
Tracing MariaDB server with bpftrace - MariaDB Server Fest 2021Valeriy Kravchuk
Bpftrace is a relatively new eBPF-based open source tracer for modern Linux versions (kernels 5.x.y) that is useful for analyzing production performance problems and troubleshooting software. Basic usage of the tool, as well as bpftrace one liners and advanced scripts useful for MariaDB DBAs are presented. Problems of MariaDB Server dynamic tracing with bpftrace and some possible solutions and alternative tracing tools are discussed.
Kernel Recipes 2019 - Formal modeling made easyAnne Nicolas
Modeling parts of Linux has become a recurring topic. For instance, the memory model, the model for PREEMPT_RT synchronization, and so on. But the term "formal model" causes panic for most of the developers. Mainly because of the complex notations and reasoning that involves formal languages. It seems to be a very theoretical thing, far from our day-by-day reality.
Believe me. Modeling can be more practical than you might guess!
This talk will discuss the challenges and benefits of modeling, based on the experience of developing the PREEMPT_RT model. It will present a methodology for modeling the Linux behavior as Finite-State Machines (automata), using terms that are very known by kernel developers: tracing events! With the particular focus on how to use models for the formal verification of Linux kernel, at runtime, with low overhead, and in many cases, without even modifying Linux kernel!
Daniel Bristot de Oliveira
All of Your Network Monitoring is (probably) Wrongice799
Monitorama 2016 talk about network monitoring covering topics like network device drivers, ethtool, and some interesting bugs/features.
For more information about monitoring and tuning the entire Linux network stack, see: blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/
Kernel Recipes 2016 - Kernel documentation: what we have and where it’s goingAnne Nicolas
The Linux kernel features an extensive array of, to put it kindly, somewhat disorganized documentation. A significant effort is underway to make things better, though. This talk will review the state of kernel documentation, cover the changes that are being made (including the adoption of a new system for formatted documentation), and discuss how interested developers can help.
Jonathan Corbet, LWN.net
Introduction to Java programming - Java tutorial for beginners to teach Java ...Duckademy IT courses
Check out the course: https://www.duckademy.com/course/java-programming Learn Java from scratch in an easy and entertaining way. The Easy-to-follow Java programming course on Duckademy was made for beginners. In this course we start at the very basics (from zero) and go through lots of interesting exercises and analogies that will take you to an advanced level. By the end of the course you will have all the knowledge needed to move on and specialize in Java. The course is easy to follow and things are well explained. Furthermore, to make your learning easier and more enjoyable throughout the course we will develop a nostalgic, text-based fantasy game.
The course is recommended to anyone who wants to learn Java. Ideal for those who are new to programming, but it can be useful and enjoyable for people who want to switch to Java from a different programming language.
By the end of the course you will be able to build simple, but fully functional programs. You will also gain all the knowledge needed to specialize in Java and become a well-payed Java expert later on.
Check out the course: https://www.duckademy.com/course/java-programming
Down-to-Earth Microservices with Java EEReza Rahman
Microservices have become the new kid of the buzzword block in our ever colorful industry. In this session we will explore what microservices really mean within the relatively well established context of distributed computing/SOA, when they make sense and how to develop them using the lightweight, simple, productive Java EE programming model.
We'll explore microservices using a simple but representative example using Java EE. You'll see how the Java EE programming model and APIs like JAX-RS, WebSocket, JSON-P, JSON-B, Bean Validation, CDI, JPA, EJB 3, JMS 2 and JTA aligns with the concept of microservices.
It may or may not surprise you to learn in the end that you already know more about microservices than you realize and that it is an architectural style that does not really require you to learn an entirely new tool set beyond the ones you already have. You might even see that Java EE is a particularly powerful and elegant tool set for developing microservices.
Summary of the lessons we learned with Docker (Dockerfile, storage, distributed networking) during the first iteration of the AdamCloud project (Fall 2014).
The AdamCloud project (part I) was presented here:
http://www.slideshare.net/davidonlaptop/bdm29-adamcloud-planification
Functional Programming with Streams in node.jsAdam Crabtree
My talk at HTML5DevConf && FluentConf covering how I combined some basic functional programming techniques with node.js streams to abstract a streaming VFS zipping layer that can also (theoretically / eventually) run in the browser.
A talk given at Docker London on Wednesday, July 20th, 2016. This talk is a fast-paced overview of the potential threats faced when containerizing applications, married to a quick run-through of the "security toolbox" available in the Docker engine via Linux kernel capabilities and features enabled by OCI's libcontainer/runc and Docker.
A video recording of this talk is available here: https://skillsmatter.com/skillscasts/8551-container-security
How Secure Is Your Container? ContainerCon Berlin 2016Phil Estes
A conference talk at ContainerCon Europe in Berlin, Germany, given on October 5th, 2016. This is a slightly modified version of my talk first used at Docker London in July 2016.
Android is NOT just 'Java on Linux'.
Android uses Linux kernel. But only kernel. I show you how different Android is from normal Linux systems.
Visit this page.
http://kobablog.wordpress.com/2011/05/22/android-is-not-just-java-on-linux/
Kernel Recipes 2016 - Landlock LSM: Unprivileged sandboxingAnne Nicolas
Linux has multiple access-control features, which help to contain the damage from a malicious process. However, it is difficult and complex, especially for unprivileged users, to create a sandboxed application because of the currently administrator-oriented security.
seccomp-bpf was a big step forward in empowering any user with the ability to filter syscalls and therefore limit access to some resources. Nevertheless, it lacks the ability to create a full standalone sandbox (e.g. restrict access to a set of files), unlike Seatbelt/XNU Sandbox or OpenBSD Pledge.
In this talk, we present Landlock, a new Linux Security Module for unprivileged users. This brings some interesting challenges, from architecture design to userland API definition.
Mickaël Salaün
This presentation will provide the information about the Linux Root File systems and its hierarchy. So any technocrate who is willing to gain info about root files of Linux can easily understand . preffered for Embedded system design Students who are pursuing diploma courses in various CDAC centers.
Writing Character driver (loadable module) in linuxRajKumar Rampelli
It covers the step by step approach on how to write a simple loadable character device driver in linux. What are Device files in linux detail. How user application interact with character driver using a device file.
Docker is an open-source project to easily create lightweight, portable, self-sufficient containers from any application. The same container that a developer builds and tests on a laptop can run at scale, in production, on VMs, bare metal, OpenStack clusters, public clouds and more.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
2. Greg Banks
Staff Site Reliablity Engineer
LinkedIn
www.linkedin.com/in/gregnbanks
Java hates Linux.
Deal with it.
3. 3
This is a story about Java and Linux
Java and Linux are a perfectly
matched pair
Except when they’re not
Then it’s fireworks
@misslexirose
4. 4
Java: a portable application platform
supposedly
Alternatively: Java is written for
an imaginary OS
Which is sorta like the set-top
box Java was designed for
Then shoe-horned into Linux,
Solaris & Windows with hidden
portability layers
Kevin Fream adroit.blog
6. 6
Garbage collection vs virtual memory
Since 3BSD (1979) every Unix-
like system’s virtual memory
subsystem has been designed
around locality of reference
www.mckusick.com
7. 7
Locality of reference
A process’ memory space is
managed in fixed size pages
A program uses a working set of
pages frequently
Other pages are hardly ever
used. They can be swapped out
to disk, saving previous RAM
Brian0918 wikipedia.org CC BY-SA 3.0
8. 8
Java behaves just like that
…until GC happens
Every page in the Java heap - a
multi-GB chunk of virtually
contiguous address space - is
read and written as fast as
possible.
While the application is stopped.
This has to be milliseconds fast
Javaworld.com
9. 9
Daytrip to Failtown
If enough of the Java heap is
swapped out, GC can take
hundreds of seconds
Your service’s clients time out
Healthchecks fail
Latency spikes propagate
Your SLAs are shot
10. 10
Real World Example
gc.log
2014-10-15T18:42:44.931+0000: 4651814.348: [GC 4651814.546: [ParNew
...
: 1152488K->274696K(1572864K), 106.7471350 secs] 15021460K-
>14143829K(32944128K), 106.9300350 secs] [Times: user=53.97 sys=518.37,
real=107.11 secs]
Total time for which application threads were stopped: 107.5402200 seconds
11. 11
Why? Swap is SLOW
“If you’re swapping out, you’ve
already lost the battle”
Swapping in happens one page
at a time
Swap has none of the tricks used
to make filesystems fast
(readahead, contiguous extents)
Nationalgeographic.com
12. 12
Deal with it
The easy way
Disable swap entirely
Remove /etc/fstab entries
Check with swapon –s
Affects all processes
OS-wide configuration change
We couldn’t do this for $reasons
13. 13
Deal with it
The clever way
sysctl –w
vm.swappiness=0
Well-known kernel tunable to
limit swapping
Doesn’t eliminate swapping
Behavior depends on kernel
version
It just didn’t work
14. 14
Deal with it
The horrible/cunning way
Lock the Java heap in RAM
A native system call using
com.sun.jna
mlockall(MCL_CURRENT)
Call early in process lifetime
No more swapping
15. 15
Resource Limits
RLIMIT_MEMLOCK limits how
much memory a process can lock
Kernel default is 64K, needs raising
echo
"app - memlock unlimited"
>>/etc/security/limits.conf
You could also give the process the
CAP_IPC_LOCK capability
Stephen Codrington CC BY 2.5
16. 16
Checking it worked
For a process with ~32G heap (-Xmx32684m)
egrep '^Vm(Lck|RSS|Size|Swap)' /proc/$pid/status
VmSize: 48767632 kB
VmLck: 39652024 kB needs to be > heap size
VmRSS: 35127672 kB
VmSwap: 26444 kB needs to be small
The whole heap is locked and resident
Some non-heap allocations came along for the ride
Others are eligible for swapping
18. 18
Java Hides the OS
So you can write portable code
Also … so you can only write
portable code
But sometime you need access
to the underlying OS objects
Coachingbysubjectexperts.com
19. 19
File Descriptors
File descriptors are the small
integers used to name open files
and sockets to Unix system calls.
int read(int fd, void*buf, int len)
Java hides these from you
because they're different on
Windows.
Stationeryinfo.com
20. 20
The Goal
Gateway to the Rabbit Hole
Network server performance
analysis
Need the length of a socket's in-
kernel input queue
Not a complicated example, but
outside Java's API fence
Jenningswire.com
21. 21
Doing it in C
FILE *stream = …
int length;
int r = ioctl(fileno(stream), FIONREAD, &length);
/* error handling */
22. 22
Doing it in Java
…because the build system cannot cope with mixed C & Java
First we have to convince Java to let us call ioctl()
import com.sun.jna.Native;
private static native int ioctl(int fd, int cmd, IntByReference valuep) throws
LastErrorException;
Native.register(…)
private static final int FIONREAD = 0x541B;
23. 23
Doing it in Java
part 2
Now we just need the Java
equivalent of C's fileno()
Given an InputStream object
return the Unix file descriptor
So simple…so impossible
24. 24
FileDescriptor is Wrapped Tight
Class FileInputStream has
public FileDescriptor getFD();
And FileDescriptor has
private int fd;
But NO WAY TO GET IT
http://www.iconsdb.com/soylent-red-icons/private-4-icon.html
25. 25
How to Unwrap a FileDescriptor ?
Well known trick using Reflection
…performance concerns
Tricks using Unsafe…are unsafe
sun.misc.SharedSecrets
This is the 2nd of the big ol' rugs
under which Sun swept all the
dust bunnies
literateforlife.org
29. 29
The Problem
We take database backup
snapshots and copy them to a
directory on an NFS filer.
Directory has 1000s of files
A Java process lists this directory
and reports file names and sizes
SLOOOOWLY
Offtackleempire.com
30. 30
How NFSv3 Works
when we run "ls"
A process on the NFS client
wants to read the contents of a
directory
getdents() system call
NFS client sends READDIR rpc
to the server, caches result
Gxdmtl.com
31. 31
The READDIR call
greatly simplified
Maps to getdents()
Arguments
nfs_fh3, count
Results
list of {
fileid3 fileid; // inode #
filename3 name; // string
}
Theodysseyonline.com
32. 32
Message Flow for "ls /d"
Process NFS Client NFS Server
getdents
local system calls network RPCs
READDIR
f1, f2, f3, …
33. 33
The Trouble With READDIR
The "ls -l" problem
Only returns names, not file
attributes or a file handle
If the process does a stat() or
open() system call on the files,
the NFS client needs to do a
LOOKUP rpc, maybe GETATTR
Those RPCs are one per file
Reference.com
34. 34
Message Flow for "ls -l /d"
Process NFS Client NFS Serverlocal system calls network RPCs
getdents
READDIR
f1, f2, …
stat(/d/f1)
LOOKUP f1
fh, attrs
stat(/d/f2)
LOOKUP f2
fh, attrs
…
For N files, N+1 RPCs
serialized on a
single thread
35. 35
The READDIRPLUS call
READDIR but returns file
handles and attributes for each
file.
Frontloads the information
needed for the process to stat()
or open() a file.
Clipartkid.com
36. 36
Call Sequence for "ls -l /d"
with READDIRPLUS
Process NFS Client NFS Serverlocal system calls network RPCs
getdents
READDIR
f1, f2, …
stat(/d/f1)
LOOKUP f1
fh, attrs
stat(/d/f2)
LOOKUP f2
fh, attrs
…
For N files, N+1 RPCs
serialized on a
single thread
37. 37
So we just use READDIRPLUS right?
If only it were that easy
The NFS protocol puts an upper
limit on the encoded size of the
results
A really big directory, needs more
READDIRPLUS than READDIR
Tradeoff: READDIR is faster with
large directories if you don’t want
to stat() the files.
Iconfinder.com
38. 38
Heuristics To The Rescue
On the first getdents() send a
READDIRPLUS
If the process open()s or stat()s
set a flag
On subsequent getdents() if the
flag is set, send READDIRPLUS
else READDIR
111emergency.co.uk
39. "ls" = processes which do
getdents READDIRPLUS
getdents READDIR
getdents READDIR
39
These Patterns Are Optimal
"ls –l" = processes which do
getdents READDIRPLUS
stat, stat, … (cached)
getdents READDIRPLUS
stat, stat, … (cached)
getdents READDIRPLUS
stat, stat, … (cached)
40. 40
"ls -l /d" in Java
This is the solution you will find on StackOverflow
File dir = File("/d");
File[] list =
dir.listFiles();
for (File f: list) {
print(f.getName(),
f.length());
}
Steven James Keathley
41. 41
java.io.File
Wraps a string filename
File.length() stat()
File.listFiles() reads the whole
directory using N x getdents(),
returns File[]
The API forces this behavor
Stationeryinfo.com
42. C processes do
getdents READDIRPLUS
stat, stat, … (cached)
getdents READDIRPLUS
stat, stat, … (cached)
getdents READDIRPLUS
stat, stat, … (cached)
42
"ls -l": C vs Java
on large directories
Java processes do
getdents READDIRPLUS
getdents READDIR
getdents READDIR
stat, stat, … (cached)
stat LOOKUP
stat LOOKUP
…
100s x slower
43. 43
Deal With It: Rewrite Using java.nio.files
new in Java 8
Path dir = Paths.get("/d");
DirectoryStream<Path> stream =
Files.newDirectoryStream(dir);
for (Path p: stream) {
File = p.toFile();
print(f.getName(), f.length());
}
The iterator object delays the getdents()until they're needed
46. 46
Java GC is Important
GC is a limiting factor on the
availability of Java services
In production, it's important to keep
GC behaving well
"You can't manage what you can't
measure"
GC logging in production.
HearMeSayThis.org
47. 47
GC Logging Options We Use
-Xloggc:$filename
-XX:+PrintGC
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintTenuringDistribution
More data is better, right
Hank Rabe hankstruckpictures.com
48. 48
What gets logged?
Major GC event
2017-03-12T23:19:01.480+0000: 3382210.059: Total time for which application threads were stopped: 0.0004373 seconds
2017-03-12T23:22:01.186+0000: 3382389.765: Application time: 179.7055362 seconds
2017-03-12T23:22:01.186+0000: 3382389.766: [GC (Allocation Failure) 3382389.766: [ParNew
Desired survivor size 134217728 bytes, new threshold 15 (max 15)
- age 1: 998952 bytes, 998952 total
- age 2: 12312 bytes, 1011264 total
- age 3: 10672 bytes, 1021936 total
- age 4: 10480 bytes, 1032416 total
- age 5: 753312 bytes, 1785728 total
- age 6: 11208 bytes, 1796936 total
- age 7: 149688 bytes, 1946624 total
- age 8: 9904 bytes, 1956528 total
- age 9: 11000 bytes, 1967528 total
- age 10: 10136 bytes, 1977664 total
- age 11: 10184 bytes, 1987848 total
- age 12: 10304 bytes, 1998152 total
- age 13: 92360 bytes, 2090512 total
- age 14: 176648 bytes, 2267160 total
- age 15: 70328 bytes, 2337488 total
: 539503K->10753K(786432K), 0.0073635 secs] 2119127K->1590653K(3932160K), 0.0075104 secs] [Times: user=0.09 sys=0.00, real=0.01 secs]
49. 49
Byte & Line Rate of Logging
Major GC up to 1200 B
Minor GC ~200 B
Light Load Heavy Load
Bytes/day 3.4 MB/d 33.0 MB/d
Lines/day 39 KL/d 543 KL/d
Visitmelbourne.com
50. 50
The Problem
The GC log is written from JDK's C
code in Stop-The-World
No userspace buffering. Every line
is a write() and an opportunity to
block in the kernel
If the root disk is heavily loaded, the
long tail latency can be 1-5 sec
Client timeouts typically 1-5 sec
Eureferendum.com
51. 51
Possible Solutions
Disable GC logging. Flying blind.
Log4j 2.x async logging. GC iog is
written from C code
Log to a named FIFO. If the reader
process dies the app is blocked.
En.oxforddictionaries.com
52. 52
Deal With It: Log To FUSE
Log to a file mounted on a FUSE
fileystem.
The filesystem's daemon accepts
writes and queues them in
userspace, writes to disk. Provides
asynchrony.
If the daemon dies, app write()s fail
immediately with ENOTCONN, app
continues.
54. 54
Java Hostname Lookup
Java makes this easy
String host = "www.example.com";
int port = 80;
Socket sock = Socket();
Sock.connect(host, port);
Kabl00ey @ wikipedia.org CC BY-SA 3.0
55. 55
Peeling The Onion
The easy APIs are layers of
wrappers around
class InetAddress {
public static InetAddress[]
getAllByName(String host) …
By default calls libc's
getaddrinfo()
Which calls Linux's NSCD
High Mowing Seed Company
56. 56
Linux NSCD
Name Service Cache Daemon
Standard (in the glibc repo)
Runs locally on every box
Configurable. Supports multiple
name service providers, like a
DNS client.
Understands & obeys TTL
Theodysseyonline.com
57. 57
Java's In-Process Cache
InetAddress conveniently caches
the results in RAM
For 30 sec. Ignores the TTL
returned from DNS
Java could have gotten this right
int __gethostbyname3_r(const char *name, …,
int32_t *ttlp, …);
58. 58
Why Does Java Do This?
Hostname Resolver Latencies
Java cache 53 ± 2 µs
NSCD 72 ± 3 ms
DNS server 192 ± 8 ms
Measured in production using a
specially written Java program.
Norfolkwildlifetrust.co.uk
59. 59
The Problem: Long TTLs
Most stable A/AAAA records are
configured with a TTL of 1h or 1d
Talking to NSCD every 30 sec is
wasteful
Yathin S Krishnappa CC BY-SA 3.0
60. 60
Worse Problem: Short TTLs
One way of achieving load
balancing is with Round-Robin
DNS
Some implementations rely on
an ultra short TTL << 30sec to
achieve failover
spirit-animals.com
61. 61
Deal With It: Disable Java's Cache
easy but impactful
Add to
$JAVA_HOME/lib/security/java.securit
y
networkaddress.cache.ttl=0
or (older)
sun.net.inetaddr.ttl=0
This is global to the host
62. 62
Deal With It: Bypass Cache using Reflection
Reflection is almost never the answer
static InetAddress[] getAllByNameUncached(String hostname) {
Field field = InetAddress.class.getDeclaredField("impl");
field.setAccessible(true);
Object impl = field.get(null);
Method method = null;
for (Method m : impl.getClass().getDeclaredMethods()) {
if (m.getName().equals("lookupAllHostAddr")) {
method = m;
break;
}
}
method.setAccessible(true);
return (InetAddress[]) method.invoke(impl, hostname);
}
63. 63
Deal With It: DNS With JNDI
least worst option
Use com.sun.jndi.dns to make
your own DNS requests for
A/AAAA records
Quick, easy
Does not affect other lookups in
the same process
Bypasses nscd entirely; all
lookups talk to DNS server.
64. 64
Deal With It: DNS SRV Records with JNDI
very useful and very hard
Use com.sun.jndi.dns to make
your own DNS requests for SRV
records
Need to handle stale entries
Have to parse out SRV response
Does not affect other lookups in
the same process
Bypasses nscd entirely; all
lookups talk to DNS server.
66. 66
The Two Hardest Things in CS
1. Naming
2. Cache Invalidation
3. Off By One Errors
67. 67
The Three Hardest Things in CS
1. Naming
2. Cache Invalidation
3. Off By One Errors
4. Making Java Behave Rationally
68. 68
Java Is Not Magic
Understand that Java doesn’t
always do the right thing with
your OS.
Anne Zweiner
69. 69
Horrible Things Live In The Corners
Modern software is complicated
and has corner cases. Bad bad
horrible things live there.
Thekidsshouldseethis.com
70. 70
Portable code is nice
Working code is better
Do what is needful
Do not be afraid to subvert
Java’s portability fascism
Know your OS
rickele @ flickr.com
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/sun/misc/SharedSecrets.java#l126
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/sun/misc/JavaIOFileDescriptorAccess.java#l35
Two implementations of the interface, one exists on Unix platforms and throws an exception for getHandle(), one vice-versa
https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#newDirectoryStream-java.nio.file.Path-
http://stackoverflow.com/questions/33668859/files-newdirectorystream-vs-files-list is very very wrong, its not a matter of style
The comments in the JDK documentation and source cpde about security appear to be entirely bullcrap
LinkedIn also has a very clever internal product that exposes applied cluster topology information with heartbeat via DNS. This service uses a 3sec TTL.
Object impl = null;
private static Method method = null;
static InetAddress[] getAllByNameUncached(String hostname) ... {
if (impl == null) {
Field field = InetAddress.class.getDeclaredField("impl");
field.setAccessible(true);
impl = field.get(null);
for (Method m : impl.getClass().getDeclaredMethods()) {
if (m.getName().equals("lookupAllHostAddr")) {
method = m;
break;
}
}
method.setAccessible(true);
}
return (InetAddress[]) method.invoke(impl, hostname);
}