Adding support for you new shiny board in Xen on ARM is a simple task once you get a kernel running on bare metal.
This session will cover the different steps to port Xen on ARM from the firmware to the shell prompt in DOM0.
We will give you tips on the common pitfalls when you have your hypervisor, or your DOM0 kernel crashing. We will also provide suggestion on how to debug when the console is not working.
CRUSH is the powerful, highly configurable algorithm Red Hat Ceph Storage uses to determine how data is stored across the many servers in a cluster. A healthy Red Hat Ceph Storage deployment depends on a properly configured CRUSH map. In this session, we will review the Red Hat Ceph Storage architecture and explain the purpose of CRUSH. Using example CRUSH maps, we will show you what works and what does not, and explain why.
Presented at Red Hat Summit 2016-06-29.
OSNoise Tracer: Who Is Stealing My CPU Time?ScyllaDB
In the context of high-performance computing (HPC), the Operating System Noise (osnoise) refers to the interference experienced by an application due to activities inside the operating system. In the context of Linux, NMIs, IRQs, softirqs, and any other system thread can cause noise to the application. Moreover, hardware-related jobs can also cause noise, for example, via SMIs.
HPC users and developers that care about every microsecond stolen by the OS need not only a precise way to measure the osnoise but mainly to figure out who is stealing cpu time so that they can pursue the perfect tune of the system. These users and developers are the inspiration of Linux's osnoise tracer.
The osnoise tracer runs an in-kernel loop measuring how much time is available. It does it with preemption, softirq and IRQs enabled, thus allowing all the sources of osnoise during its execution. The osnoise tracer takes note of the entry and exit point of any source of interferences. When the noise happens without any interference from the operating system level, the tracer can safely point to a hardware-related noise. In this way, osnoise can account for any source of interference. The osnoise tracer also adds new kernel tracepoints that auxiliaries the user to point to the culprits of the noise in a precise and intuitive way.
At the end of a period, the osnoise tracer prints the sum of all noise, the max single noise, the percentage of CPU available for the thread, and the counters for the noise sources, serving as a benchmark tool.
In this talk we will discuss how to build and run containers without root privileges. As part of the discussion, we will introduce new programs like fuse-overlayfs and slirp4netns and explain how it is possible to do this using user namespaces. fuse-overlayfs allows to use the same storage model as "root" containers and use layered images. slirp4netns emulates a TCP/IP stack in userland and allows to use a network namespace from a container and let it access the outside world (with some limitations).
We will also introduce Usernetes, and how to run Kubernetes in an unprivileged user namespace
https://sched.co/Jcgg
Adding support for you new shiny board in Xen on ARM is a simple task once you get a kernel running on bare metal.
This session will cover the different steps to port Xen on ARM from the firmware to the shell prompt in DOM0.
We will give you tips on the common pitfalls when you have your hypervisor, or your DOM0 kernel crashing. We will also provide suggestion on how to debug when the console is not working.
CRUSH is the powerful, highly configurable algorithm Red Hat Ceph Storage uses to determine how data is stored across the many servers in a cluster. A healthy Red Hat Ceph Storage deployment depends on a properly configured CRUSH map. In this session, we will review the Red Hat Ceph Storage architecture and explain the purpose of CRUSH. Using example CRUSH maps, we will show you what works and what does not, and explain why.
Presented at Red Hat Summit 2016-06-29.
OSNoise Tracer: Who Is Stealing My CPU Time?ScyllaDB
In the context of high-performance computing (HPC), the Operating System Noise (osnoise) refers to the interference experienced by an application due to activities inside the operating system. In the context of Linux, NMIs, IRQs, softirqs, and any other system thread can cause noise to the application. Moreover, hardware-related jobs can also cause noise, for example, via SMIs.
HPC users and developers that care about every microsecond stolen by the OS need not only a precise way to measure the osnoise but mainly to figure out who is stealing cpu time so that they can pursue the perfect tune of the system. These users and developers are the inspiration of Linux's osnoise tracer.
The osnoise tracer runs an in-kernel loop measuring how much time is available. It does it with preemption, softirq and IRQs enabled, thus allowing all the sources of osnoise during its execution. The osnoise tracer takes note of the entry and exit point of any source of interferences. When the noise happens without any interference from the operating system level, the tracer can safely point to a hardware-related noise. In this way, osnoise can account for any source of interference. The osnoise tracer also adds new kernel tracepoints that auxiliaries the user to point to the culprits of the noise in a precise and intuitive way.
At the end of a period, the osnoise tracer prints the sum of all noise, the max single noise, the percentage of CPU available for the thread, and the counters for the noise sources, serving as a benchmark tool.
In this talk we will discuss how to build and run containers without root privileges. As part of the discussion, we will introduce new programs like fuse-overlayfs and slirp4netns and explain how it is possible to do this using user namespaces. fuse-overlayfs allows to use the same storage model as "root" containers and use layered images. slirp4netns emulates a TCP/IP stack in userland and allows to use a network namespace from a container and let it access the outside world (with some limitations).
We will also introduce Usernetes, and how to run Kubernetes in an unprivileged user namespace
https://sched.co/Jcgg
An introduction to Linux Container, Namespace & Cgroup.
Virtual Machine, Linux operating principles. Application constraint execution environment. Isolate application working environment.
Xen Project is a static partitioning hypervisor for embedded deployments (industrial, medical, etc.) Xen enforces strong isolation between domains so that one cannot affect the execution of another. Features such as cache coloring reduce interference and improve interrupt latency and determinism. A real-time workload can run alongside a more complex guest. But can it be used in safety-critical environments? The Xen hypervisor has a microkernel design: services and tools are non-essential and run in unprivileged VMs, while the core is less than 50K LOC. This architecture lends itself well to safety-critical applications as only the core is critical and needs to go through the certification process. This presentation will describe the activities of the Xen FuSa SIG (Special Interest Group) to make Xen easier to safety-certify. It will go through the aspects of Xen that pertain safety and it will explain how to set up a mixed-criticality system with Xen. The talk will discuss the challenges of making an Open Source project safety-certifiable and the progress that the Xen community made so far in the areas of documentation and requirements, MISRA-C code compliance, and interference reduction.
How To Monetise & Bill CloudStack - A Practical Open ApproachShapeBlue
This talk is for ISPs looking to bill CloudStack resources, and for software developers looking to build a billing solution around CloudStack. This talk looks at covering multiple business and technical use cases (for example: plans, catalogues, flexible billing, tiered offerings, account management, etc.) for running a public cloud and how the same can be achieved using CloudStack. It does not delve into any specific billing system but instead focuses on an open approach to how ACS features can be leveraged to implement billing and monetise CloudStack.
Shiv is the Co-Founder and CTO of IndiQus Technologies Pvt. Ltd. and a CloudStack user turned evangelist since 2013. He loves tinkering on CloudStack and the possibilities it offers. He has deployed multiple public and private clouds running CloudStack in the South Asian region and has also integrated legacy systems with CloudStack. He would love to share his experiences with like-minded professionals.
-----------------------------------------
CloudStack Collaboration Conference 2022 took place on 14th-16th November in Sofia, Bulgaria and virtually. The day saw a hybrid get-together of the global CloudStack community hosting 370 attendees. The event hosted 43 sessions from leading CloudStack experts, users and skilful engineers from the open-source world, which included: technical talks, user stories, new features and integrations presentations and more.
Introduces Ansible as DevOps favorite choice for Configuration Management and Server Provisioning. Enables audience to get started with using Ansible. Developed in Python which only needs YAML syntax knowledge to automate using this tool.
In the Cloud Native community, eBPF is gaining popularity, which can often be the best solution for solving different challenges with deep observability of system. Currently, eBPF is being embraced by major players.
Mydbops co-Founder, Kabilesh P.R (MySQL and Mongo Consultant) illustrates on debugging linux issues with eBPF. A brief about BPF & eBPF, BPF internals and the tools in actions for faster resolution.
Note: also see https://www.slideshare.net/xen_com_mgr/ossna18-xen-beginners-training-exercise-script
The Xen Project supports some of the biggest clouds in production today and is moving into new industries, like security and automotive. Usually, you will use Xen indirectly as part of a commercial product, a distro, a hosting or cloud service and only indirectly use Xen. By following this session you will learn how Xen and virtualization work under the hood exploring high-level topics like architecture concepts related to virtualization to more technical attributes of the hypervisor like memory management (ballooning), virtual CPUs, scheduling, pinning, saving/restoring and migrating VMs.
In this PowerPoint, learn how a security policy can be your first line of defense. Servers running AIX and other operating systems are frequent targets of cyberattacks, according to the Data Breach Investigations Report. From DoS attacks to malware, attackers have a variety of strategies at their disposal. Having a security policy in place makes it easier to ensure you have appropriate controls in place to protect mission-critical data.
An introduction to Linux Container, Namespace & Cgroup.
Virtual Machine, Linux operating principles. Application constraint execution environment. Isolate application working environment.
Xen Project is a static partitioning hypervisor for embedded deployments (industrial, medical, etc.) Xen enforces strong isolation between domains so that one cannot affect the execution of another. Features such as cache coloring reduce interference and improve interrupt latency and determinism. A real-time workload can run alongside a more complex guest. But can it be used in safety-critical environments? The Xen hypervisor has a microkernel design: services and tools are non-essential and run in unprivileged VMs, while the core is less than 50K LOC. This architecture lends itself well to safety-critical applications as only the core is critical and needs to go through the certification process. This presentation will describe the activities of the Xen FuSa SIG (Special Interest Group) to make Xen easier to safety-certify. It will go through the aspects of Xen that pertain safety and it will explain how to set up a mixed-criticality system with Xen. The talk will discuss the challenges of making an Open Source project safety-certifiable and the progress that the Xen community made so far in the areas of documentation and requirements, MISRA-C code compliance, and interference reduction.
How To Monetise & Bill CloudStack - A Practical Open ApproachShapeBlue
This talk is for ISPs looking to bill CloudStack resources, and for software developers looking to build a billing solution around CloudStack. This talk looks at covering multiple business and technical use cases (for example: plans, catalogues, flexible billing, tiered offerings, account management, etc.) for running a public cloud and how the same can be achieved using CloudStack. It does not delve into any specific billing system but instead focuses on an open approach to how ACS features can be leveraged to implement billing and monetise CloudStack.
Shiv is the Co-Founder and CTO of IndiQus Technologies Pvt. Ltd. and a CloudStack user turned evangelist since 2013. He loves tinkering on CloudStack and the possibilities it offers. He has deployed multiple public and private clouds running CloudStack in the South Asian region and has also integrated legacy systems with CloudStack. He would love to share his experiences with like-minded professionals.
-----------------------------------------
CloudStack Collaboration Conference 2022 took place on 14th-16th November in Sofia, Bulgaria and virtually. The day saw a hybrid get-together of the global CloudStack community hosting 370 attendees. The event hosted 43 sessions from leading CloudStack experts, users and skilful engineers from the open-source world, which included: technical talks, user stories, new features and integrations presentations and more.
Introduces Ansible as DevOps favorite choice for Configuration Management and Server Provisioning. Enables audience to get started with using Ansible. Developed in Python which only needs YAML syntax knowledge to automate using this tool.
In the Cloud Native community, eBPF is gaining popularity, which can often be the best solution for solving different challenges with deep observability of system. Currently, eBPF is being embraced by major players.
Mydbops co-Founder, Kabilesh P.R (MySQL and Mongo Consultant) illustrates on debugging linux issues with eBPF. A brief about BPF & eBPF, BPF internals and the tools in actions for faster resolution.
Note: also see https://www.slideshare.net/xen_com_mgr/ossna18-xen-beginners-training-exercise-script
The Xen Project supports some of the biggest clouds in production today and is moving into new industries, like security and automotive. Usually, you will use Xen indirectly as part of a commercial product, a distro, a hosting or cloud service and only indirectly use Xen. By following this session you will learn how Xen and virtualization work under the hood exploring high-level topics like architecture concepts related to virtualization to more technical attributes of the hypervisor like memory management (ballooning), virtual CPUs, scheduling, pinning, saving/restoring and migrating VMs.
In this PowerPoint, learn how a security policy can be your first line of defense. Servers running AIX and other operating systems are frequent targets of cyberattacks, according to the Data Breach Investigations Report. From DoS attacks to malware, attackers have a variety of strategies at their disposal. Having a security policy in place makes it easier to ensure you have appropriate controls in place to protect mission-critical data.
Systemd: the modern Linux init system you will learn to loveAlison Chaiken
The talk combines a design overview of systemd with some tutorial incofrmation about how to configure it. Systemd's features and pitfalls are illustrated by short demos and real-life examples. Files used in the demos are listed under "Presentations" at http://she-devel.com/
Video of the live presentation will appear here:
http://www.meetup.com/Silicon-Valley-Linux-Technology/events/208133972/
State of Containers and the Convergence of HPC and BigDatainside-BigData.com
In this deck from 2018 Swiss HPC Conference, Christian Kniep from Docker Inc. presents: State of Containers and the Convergence of HPC and BigData.
"This talk will recap the history of and what constitutes Linux Containers, before laying out how the technology is employed by various engines and what problems these engines have to solve. Afterward Christian will elaborate on why the advent of standards for images and runtimes moved the discussion from building and distributing containers to orchestrating containerized applications at scale. In conclusion attendees will get an update on how containers foster the convergence of Big Data and HPC workloads and the state of native HPC containers."
Learn more: http://docker.com
and
http://www.hpcadvisorycouncil.com/events/2018/swiss-workshop/agenda.php
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Advanced cgroups and namespaces
This talk picks up where we left off in the previous cgroups and namespaces talk and dive in even deeper!
Agenda:
* cgroups v2 design (cgroup v2 was started to be merged in the current kernel, 4.4)
* cgroups v2 examples (migrating tasks, enabling and disabling controllers, and more).
* comparison between cgroup v2 unified hierarchy and cgroup v1 legacy hierarchy.
* PIDs namespaces (from kernel 4.3)
* cgroup namespaces (not merged yet)
Hands on Virtualization with Ganeti (part 1) - LinuxCon 2012Lance Albertson
Ganeti is a robust cluster virtualization management software tool. It’s built on top of existing virtualization technologies such as Xen and KVM and other Open Source software. Its integration with various technologies such as DRBD and LVM results in a cheaper High Availability infrastructure and linear scaling.
This hands-on tutorial will cover a basic overview of Ganeti, the step-by-step install & setup of a single-node and multi-node Ganeti cluster, operating the cluster, and some best practices of Ganeti.
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...Yandex
Lightweight virtualization", also called "OS-level virtualization", is not new. On Linux it evolved from VServer to OpenVZ, and, more recently, to Linux Containers (LXC). It is not Linux-specific; on FreeBSD it's called "Jails", while on Solaris it’s "Zones". Some of those have been available for a decade and are widely used to provide VPS (Virtual Private Servers), cheaper alternatives to virtual machines or physical servers. But containers have other purposes and are increasingly popular as the core components of public and private Platform-as-a-Service (PAAS), among others.
Just like a virtual machine, a Linux Container can run (almost) anywhere. But containers have many advantages over VMs: they are lightweight and easier to manage. After operating a large-scale PAAS for a few years, dotCloud realized that with those advantages, containers could become the perfect format for software delivery, since that is how dotCloud delivers from their build system to their hosts. To make it happen everywhere, dotCloud open-sourced Docker, the next generation of the containers engine powering its PAAS. Docker has been extremely successful so far, being adopted by many projects in various fields: PAAS, of course, but also continuous integration, testing, and more.
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...Adrian Huang
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Current experience shows that a lot of developers working on Xen/Linux kernel use mainly only small set of debugging tools. Often they are sufficient for generic work. However, when unusual problem arises which could not be easily debugged using known tools sometimes they are trying to reinvent the wheel. Goal of this session is to present wide range of debugging tools starting from simplest one to most feature reach solutions in context of Xen/Linux kernel debugging. It will describe pros and cons of printk (serial, debug console, etc.), gdb, gdbsx, kgdb, QEMU, kdump and others. Additionally, there will be some information about possible new solutions and current kexec/kdump developments for Xen.
[KubeCon NA 2020] containerd: Rootless Containers 2020Akihiro Suda
Rootless Containers means running the container runtimes (e.g. runc, containerd, and kubelet) as well as the containers without the host root privileges. The most significant advantage of Rootless Containers is that it can mitigate potential container-breakout vulnerability of the runtimes, but it is also useful for isolating multi-user environments on HPC hosts. This talk will contain the introduction to rootless containers and deep-dive topics about the recent updates such as Seccomp User Notification. The main focus will be on containerd (CNCF Graduated Project) and its consumer projects including Kubernetes and Docker/Moby, but topics about other runtimes will be discussed as well.
https://sched.co/fGWc
This is to introduce the related components in SUSE Linux Enterprise High Availability Extension product to build High Available Storage (ha-lvm/drbd/iscsi/nfs, clvm, ocfs2, cluster-raid1).
Talk from Embedded Linux Conference, http://elcabs2015.sched.org/event/551ba3cdefe2d37c478810ef47d4ca4c?iframe=no&w=i:0;&sidebar=yes&bg=no#.VRUCknSQQQs
Running Applications on the NetBSD Rump Kernel by Justin Cormack eurobsdcon
Abstract
The NetBSD rump kernel has been developed for some years now, allowing NetBSD kernel drivers to be used unmodified in many environments, for example as userspace code. However it is only since last year that it has become possible to easily run unmodified applications on the rump kernel, initially with the rump kernel on Xen port, and then with the rumprun tools to run them in userspace on Linux, FreeBSD and NetBSD. This talk will look at how this is achieved, and look at use cases, including kernel driver development, and lightweight process virtualization.
Speaker bio
Justin Cormack has been a Unix user, developer and sysadmin since the early 1990s. He is based in London and works on open source cloud applications, Lua, and the NetBSD rump kernel project. He has been a NetBSD developer since early 2014.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
How world-class product teams are winning in the AI era by CEO and Founder, P...
Red Hat Global File System (GFS)
1. Red Hat Storage Cluster
(GFS, CLVM, GNBD)
GFS allows multiple nodes to share
storage at a block level as if the
storage were connected locally to each
cluster node.
Schubert Zhang, Guangxian Liao
Jul, 2008
1
3. Architecture
• Pay attention to the relationship of cluster software
components.
– Cluster Infrastructure (Common): CMAN, DLM, CCS, Fencing
– GNBD: Server and Client
– CLVM
– GFS
3
4. Cluster Infrastructure
• We only concern cluster type for storage.
• Necessary infrastructure components:
– Cluster management (CMAN)
• It’s a distributed cluster manager and runs in each cluster node.
• Keeps track of cluster quorum, avoids "split-brain". (Like chubby’s Paxos Protocol?)
• Keeps track of membership.
• libcman.so library, cman_tool
• dlm_controld: started by cman init script to manage dlm in kernel
• gfs_controld : started by cman init script to manage gfs in kernel
• groupd: started by cman init script to interface between openais/cman and
dlm_controld/gfs_controld/fenced , group_tool
– Lock management (DLM)
• To synchronize access to shared resources (shared storage, etc.).
• Runs in each cluster node.
• libdlm.so library.
• GFS and CLVM use locks from DLM.
• GFS uses locks from the lock manager to synchronize access to file system metadata
(on shared storage).
• CLVM uses locks from the lock manager to synchronize updates to LVM volumes and
volume groups (also on shared storage).
• Like chubby’s lock function?
4
5. Cluster Infrastructure (cont.)
– Cluster configuration management (CCS)
• Runs in each cluster node. (ccsd)
• Synchronization/up-to-date of cluster configuration file. Propagates
modification in a cluster.
• Other components (eg. CMAN) access configuration info via CCS.
• /etc/cluster/cluster.conf
(Cluster Name, Cluster nodes, Fence, Resources, etc.)
• ccs_tool: to make online updates of CCS configuration files
• ccs_test: to retrieve information from configuration files through ccsd.
– Fencing
• Fencing is the disconnection of a node from the cluster's shared storage.
Fencing cuts off I/O from shared storage, thus ensuring data integrity. The
cluster infrastructure performs fencing through the fence daemon, fenced.
• GNBD fencing: fence_gnbd?
• fence_tool
• cluster configuration file: fencing-method, fencing agent, fencing device for
each node in the cluster.
5
6. GNBD
• An ancillary component of GFS that exports
block-level storage to Ethernet.
• Global Network Block Device
– GNBD provides block-device access for Red Hat GFS
over TCP/IP. GNBD is similar in concept to NBD;
– GNBD is GFS-specific and tuned solely for use with
GFS.
• Two major components
– GNBD Server
– GNBD Client
6
7. GNBD (server)
• Exports block-level storage from its local
storage.
• gnbd_serv process
– GNBD server need not join the cluster
manager.
• gnbd_export
– Export block devices.
7
8. GNBD (client)
• A GNBD client runs in a node with GFS.
• Imports block device exported by GNBD server.
• Multiple GNBD clients can access a device
exported by a GNBD server, thus making a
GNBD suitable for use by a group of nodes
running GFS.
• gnbd.ko
– A kernel module
• gnbd_import
– Import remote block devices form GNBD server.
8
9. CLVM
• Provides volume management of cluster storage.
• A cluster-wide version of LVM2
• CLVM provides the same capabilities as LVM2 on a single node, but
makes the logical volumes created with CLVM available to all nodes
in a cluster.
• CLVM uses the lock-management service provided by the cluster
infrastructure.
• Using CLVM requires minor changes to /etc/lvm/lvm.conf for cluster-
wide locking.
• clvmd:
– A daemon that provides clustering extensions to the standard LVM2 tool
set and allows LVM2 commands to manage shared storage.
– Runs on each cluster node.
– Distributes LVM metadata updates in a cluster, thereby presenting each
cluster node with the same view of the logical volumes
9
10. CLVM (cont.)
• lvm: LVM2 command line tools.
• /etc/lvm/lvm.conf
• pvcreate
– block devices/partitions -> PV
• vgcreate
– PV(s) -> VG
• lvcreate
– VG -> LV(s)
• Ready for make file system on LV.
10
11. GFS
• To simultaneously access a block device that is shared among the
nodes.
• Single, consistent view of the FS name space across GFS nodes in
a cluster.
• Native FS under VFS, POSIX interface to applications.
• Distributed metadata and multiple journals.
• Uses lock manager to coordinate I/O.
• When one node changes data on a GFS file system, that change is
immediately visible to the other cluster nodes using that file system.
• Scale the cluster seamlessly by adding servers or storage on the fly.
• We use a “Economy and Performance” deployment.
11
12. GFS (cont.)
• gfs.ko: kernel module, loaded on each GFS
cluster node.
• gfs_mkfs: create a GFS on a storage device.
• gfs_tool: configures or tunes a GFS.
• gfs_grow: grows a mounted GFS.
• gfs_jadd: adds journals to a mounted GFS.
• gfs_quota: manages quotas on a mounted GFS.
• gfs_fsck: repairs an unmounted GFS.
• mount.gfs: mount helper called by mount.
12
13. Fencing
• We must configure each GFS node in the
cluster for at least one form of fencing.
13
14. Setup a Cluster
(prepare)
• Software Installation
– Install default packages of “Clustering” and “Storage Clustering” in each node.
– The rpms in cdrom: /ClusterStorage and /Cluster
– cman’s rpm in cdrom: /Server
– Major RPM (if need dependence, install depended rpm)
• cman-2.0.60-1.el5.i386.rpm
• modcluster-0.8-27.el5.i386.rpm
• gnbd-1.1.5-1.el5.i386.rpm
• kmod-gnbd-0.1.3-4.2.6.18_8.el5.i686.rpm
• kmod-gfs-0.1.16-5.2.6.18_8.el5.i686.rpm
• lvm2-cluster-2.02.16-3.el5.i386.rpm
• Global_File_System-en-US-5.0.0-4.noarch.rpm
• gfs-utils-0.1.11-1.el5.i386.rpm
• gfs2-utils-0.1.25-1.el5.i386.rpm
• etc.
• Network
– Disable firewall and SELinux
– Enable multicast and IGMP.
– Configure /etc/hosts or DNS for hostname (import!)
• Machine hostnames
– 192.168.1.251 test1 (gnbd server)
– 192.168.1.252 test2 (gfs node)
– 192.168.1.253 test3 (gfs node)
– 192.168.1.254 test4 (gnbd server) 14
15. Setup a Cluster
(GNBD server)
• In GNBD server node, need not start cman, i.e., the GNBD server is not a member of
the cluster.
(1) Start GNBD server process (man gnbd_serv)
# gnbd_serv -n
(2) Export block device (man gnbd_export)
# gnbd_export -v -d /dev/sda3 -e gnbdnode1 –c
Note1: must enable cache since no cman.
Note2: the block device should be a disk partition (how about LV? The document
says LV is not supported).
(3) Check the export
# gnbd_export -l
(4) Add to /etc/rc.local
gnbd_serv –n
gnbd_export -v /dev/sda3 -e gnbdnode1 -c
15
16. Setup a Cluster
(cluster infrastructure)
• Initially configure a cluster
– /etc/cluster/cluster.conf: generated by “system-config-cluster” or manually. (only config clustername and
node members)
– In one GFS node, use “system-config-cluster” to create a new cluster(name:cluster1) and add a node
(name: test2)
– The cluster.conf
<?xml version="1.0" ?>
<cluster alias="cluster1" config_version="5" name="cluster1">
<fence_daemon post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="test2" nodeid="1" votes="1">
<fence/>
</clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1"/>
<fencedevices/>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>
• Start infrastructure components
– service cman start (refer to /etc/init.d/cman)
• Load kernel modules (configfs, dlm, lock_dlm)
• Mount configfs (I think it like chubby’s files ystem space)
• Start ccsd daemon
• Start cman (no daemon, use cman_tool join to join this node to cluster)
• Start daemons (start groupd, fenced, dlm_controld, gfs_controld)
• Start fencing (start fenced daemon, and use fence_tool to join this node to fence domain)
16
17. Setup a Cluster
(GNBD client)
• Load kernel module gnbd.ko
# echo “modprobe gnbd” >/etc/sysconfig/modules/gnbd.modules
# chmod 755 /etc/sysconfig/modules/gnbd.modules
# modprobe gnbd
Then, the gnbd.ko will be loaded when the node boot-up.
• Import GNBD
# gnbd_import -i test1
Then, we can find a block device /dev/gnbd/gnbdnode1, and it is same as /dev/gnbd0.
create a /etc/init.d/gnbd-client as a service script
# chmod 755 /etc/init.d/gnbd-client
# chkconfig --add gnbd-client
Since gnbd_import -i should be done early then clvmd and gnbd_import –R should be done
later then clvmd, we assign special start-number (23<24) and stop-number(77>76) in script
/etc/init.d/gnbd-client.
Thus, the gnbd will be imported automatically when node boot-up.
17
18. Setup a Cluster
(CLVM)
• Start clvmd
# service clvmd start
# chkconfig --level 35 clvmd
• pvcreate
# pvcreate /dev/gnbd0, or
# pvcreate /dev/gnbd/gnbdnode1
Use lvmdiskscan or pvdisplay or pvscan to display status.
• vgcteare
# vgcreate vg1 /dev/gnbd0
Use vgdisplay or vgscan to display status.
• lvcreate
# lvcreate -l 100%FREE -n lv1 vg1
Use lvdisplay and lvscan to display status.
• Restart clvmd
# service clvmd rerestart
clvmd is responsible for sync the lvm configuration among the cluster nodes.
Now, we can find a new logical volume block device @ /dev/vg1/lv1, and we can make file
system on it.
18
19. Setup a Cluster
(GFS)
• Make sure that the clocks on the GFS nodes are synchronized. (Use
NTP)
• Make GFS file system
# gfs_mkfs -p lock_dlm -t cluster1:testgfs -j 4 /dev/vg1/lv1
Note: -j number same node number. One journal is required for each node
that mounts a GFS file system. Make sure to account for additional journals
needed for future expansion.
• Mount GFS
# mkdir /mnt/gfs
(1) # mount -t gfs -o acl /dev/vg1/lv1 /mnt/gfs
(2) add to /etc/fstab
/dev/vg1/lv1 /mnt/gfs gfs defaults,acl 0 0
# mount -a -t gfs
# chkconfig --level 35 gfs
(refer to /etc/init.d/gfs)
Now, the GFS file system is accessible @ /mnt/gfs/
19
20. Setup a Cluster
(add a new GFS node)
• Install packages in the new node
• Add new node information in cluster.conf in a existing node.
– Use ccs_tool addnode, or
– Use system-config-cluster
• Copy (scp) the /etc/cluster.conf to new node.
• Import GNBD on the new node
• Stop components on all running node (if there are more than 2 node existing, need
not do this step)
# service gfs stop
# service clvmd stop
# service cman stop
• Then start components on all running node and the new added node
# service cman start
# service clvmd start
# service gfs start
The clvmd will sync the metadata of logical volumes to the new added node, so, when clvmd
is started, the /dev/vg1/lv1 will be visible on the now node.
20
21. Setup a Cluster
(add a new GNBD node)
• Setup a new GNBD server (machine test4) # gnbd_import -l
and export a new block device. Device name : gnbdnode1
# gnbd_serv -n ----------------------
# gnbd_export -v -d /dev/sda3 -e
gnbdnode2 –c Minor # : 0
# gnbd_export -l sysfs name : /block/gnbd0
• Import the new GNBD (on all of the cluster Server : test1
nodes) Port : 14567
# gnbd_import -i test4 State : Open Connected Clear
# gnbd_import –l
Readonly : No
• Make a new PV (on one of the cluster node) Sectors : 3984120
# pvcreate –v /dev/gnbd1
Then, we can find the new PV on all nodes in the
cluster, by pvdisplay or lvmdiskscan or pvscan Device name : gnbdnode2
or pvs. ----------------------
• Extend the VG (on one of the cluster node) Minor # : 1
# vgextend -v vg1 /dev/gnbd1
sysfs name : /block/gnbd1
Then, we can find the extended VG and changed PV
on all nodes in the cluster, by vgdisplay or Server : test4
vgfscan or vgs. Port : 14567
• Extend the LV (on one of the cluster node) State : Close Connected
# lvextend -v -l +100%FREE Clear
/dev/vg1/lv1
Then, we can find the extended LV on all nodes in
Readonly : No
the cluster, by lvdisplay or lvfscan or lvs. Sectors : 4273290
• Grow the GFS (on one of the cluster node)
(next page) 21
22. Setup a Cluster
(grow the GFS)
• The gfs_grow command must be run on a
mounted file system. Only needs to be run
on one node in a cluster.
• Grow
# gfs_grow /mnt/gfs
• Sometimes after gfs_grow, df and
lvdisplay are hung. Need reboot the
system.
22
23. Evaluation
• Size
– GFS is based on a 64-bit architecture, which can theoretically
accommodate an 8 EB file system. However, the current supported
maximum size of a GFS file system is 25 TB. (But there is a note in
Red Hat document: “If your system requires GFS file systems larger
than 25 TB, contact your Red Hat service representative.”)
• Essential benchmarks
– Refer to GlusterFS’s benchmarks:
http://www.gluster.org/docs/index.php/GlusterFS#GlusterFS_Benchmar
ks
• Iozone benchmarks
– Refer to http://www.iozone.org
• Lock test
– Always blocks the process when it set a new lock which conflict with
other process.
23
24. Conclusions
• The cluster infrastructure is too complex and weak, sometimes fail (such as
cman, fencing, etc).
• The GNDB is simple and robust, but lacks flexibility.
• The CLVM is ok, but too complex to use.
• The GFS is very weak, sometime fail. (mount is ok, but umount often fail)
• The “two levels” (GNBDStorage+GFSCluster) deployment is not meet the
“cloud” goal.
• Not easy to add a new GFS cluster node or a GNBD storage node.
• No data replicas for safety.
• Risks when a GNBD node fails. When one GNBD node fails, the data GFS
is not accessible.
The Red Hat Cluster solution is not based on the Assumptions like
GoogleFS and GlusterFS, (i.e. “The system is built from many computer
that often fail.”), it is not easy and not safe to use in a moderate scale
cluster. So, I think the Red Hat Storage Cluster is not a good designed
solution, and has no good future.
24