ZFS is a modern file system that provides data integrity through checksums and a copy-on-write design. It simplifies storage administration with a pooled model where space can be shared across file systems. ZFS supports features like snapshots, clones, compression, and deduplication. It can leverage solid state drives for caching and improve performance.
In computing, ZFS is a combined file system and logical volume manager designed by Sun Microsystems, a subsidiary of Oracle Corporation. The features of ZFS include support for high storage capacities, integration of the concepts of file system and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z and native NFSv4 ACLs. Unlike traditional file systems, which reside on single devices and thus require a volume manager to use more than one device, ZFS file systems are built on top of virtual storage pools called zpools. A zpool is constructed of virtual devices (vdevs), which are themselves constructed of block devices: files, hard drive partitions, or entire drives, with the last being the recommended usage.[7] Thus, a vdev can be viewed as a group of hard drives. This means a zpool consists of one or more groups of drives.
In addition, pools can have hot spares to compensate for failing disks. In addition, ZFS supports both read and write caching, for which special devices can be used. Solid State Devices can be used for the L2ARC, or Level 2 ARC, speeding up read operations, while NVRAM buffered SLC memory can be boosted with supercapacitors to implement a fast, non-volatile write cache, improving synchronous writes. Finally, when mirroring, block devices can be grouped according to physical chassis, so that the filesystem can continue in the face of the failure of an entire chassis. Storage pool composition is not limited to similar devices but can consist of ad-hoc, heterogeneous collections of devices, which ZFS seamlessly pools together, subsequently doling out space to diverse file systems as needed. Arbitrary storage device types can be added to existing pools to expand their size at any time. The storage capacity of all vdevs is available to all of the file system instances in the zpool. A quota can be set to limit the amount of space a file system instance can occupy, and a reservation can be set to guarantee that space will be available to a file system instance.
Intel - optimizing ceph performance by leveraging intel® optane™ and 3 d nand...inwin stack
Kenny Chang (張任伯) (Storage Solution Architect, Intel)
With the trend that Solid State Drive (SSD) becomes more affordable, more and more cloud providers are trying to provide high performance, highly reliable storage for their customers with SSDs. Ceph is becoming one of most open source scale-out storage solutions in worldwide market. More and more customers have strong demands that using SSD in Ceph to build high performance storage solutions for their Openstack clouds.
The disrupted Intel® Optane SSDs based on 3D Xpoint technology fills the performance gap between DRAM and NAND based SSD while the Intel® 3D NAND TLC is reducing cost gap between SSD and traditional spindle hard drive and makes it possible for all flash storage. In this session, we will
1) Discuss OpenStack storage Ceph reference design on the first Intel Optane (3D Xpoint) and P4500 TLC NAND based all-flash Ceph cluster, it delivers multi-million IOPS with extremely low latency as well as increase storage density with competitive dollar-per-gigabyte costs
2) Share Ceph bluestore tunings and optimizations, latency analysis, TCO model, IOPS/TB, IOPS/$ based on the reference architecture to demonstrate this high performance, cost effective solution.
In computing, ZFS is a combined file system and logical volume manager designed by Sun Microsystems, a subsidiary of Oracle Corporation. The features of ZFS include support for high storage capacities, integration of the concepts of file system and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z and native NFSv4 ACLs. Unlike traditional file systems, which reside on single devices and thus require a volume manager to use more than one device, ZFS file systems are built on top of virtual storage pools called zpools. A zpool is constructed of virtual devices (vdevs), which are themselves constructed of block devices: files, hard drive partitions, or entire drives, with the last being the recommended usage.[7] Thus, a vdev can be viewed as a group of hard drives. This means a zpool consists of one or more groups of drives.
In addition, pools can have hot spares to compensate for failing disks. In addition, ZFS supports both read and write caching, for which special devices can be used. Solid State Devices can be used for the L2ARC, or Level 2 ARC, speeding up read operations, while NVRAM buffered SLC memory can be boosted with supercapacitors to implement a fast, non-volatile write cache, improving synchronous writes. Finally, when mirroring, block devices can be grouped according to physical chassis, so that the filesystem can continue in the face of the failure of an entire chassis. Storage pool composition is not limited to similar devices but can consist of ad-hoc, heterogeneous collections of devices, which ZFS seamlessly pools together, subsequently doling out space to diverse file systems as needed. Arbitrary storage device types can be added to existing pools to expand their size at any time. The storage capacity of all vdevs is available to all of the file system instances in the zpool. A quota can be set to limit the amount of space a file system instance can occupy, and a reservation can be set to guarantee that space will be available to a file system instance.
Intel - optimizing ceph performance by leveraging intel® optane™ and 3 d nand...inwin stack
Kenny Chang (張任伯) (Storage Solution Architect, Intel)
With the trend that Solid State Drive (SSD) becomes more affordable, more and more cloud providers are trying to provide high performance, highly reliable storage for their customers with SSDs. Ceph is becoming one of most open source scale-out storage solutions in worldwide market. More and more customers have strong demands that using SSD in Ceph to build high performance storage solutions for their Openstack clouds.
The disrupted Intel® Optane SSDs based on 3D Xpoint technology fills the performance gap between DRAM and NAND based SSD while the Intel® 3D NAND TLC is reducing cost gap between SSD and traditional spindle hard drive and makes it possible for all flash storage. In this session, we will
1) Discuss OpenStack storage Ceph reference design on the first Intel Optane (3D Xpoint) and P4500 TLC NAND based all-flash Ceph cluster, it delivers multi-million IOPS with extremely low latency as well as increase storage density with competitive dollar-per-gigabyte costs
2) Share Ceph bluestore tunings and optimizations, latency analysis, TCO model, IOPS/TB, IOPS/$ based on the reference architecture to demonstrate this high performance, cost effective solution.
Krzysztof Ksiazek - Severalnines AB
So, you are a developer or sysadmin and showed some abilities in dealing with databases issues. And now, you have been elected to the role of DBA. And as you start managing the databases, you wonder…
* How do I tune them to make best use of the hardware?
* How do I optimize the Operating System?
* How do I best configure MySQL or MariaDB for a specific database workload?
If you're asking yourself the following questions when it comes to optimally running your MySQL or MariaDB databases, then this talk is for you!
We will discuss some of the settings that are most often tweaked and which can bring you significant improvement in the performance of your MySQL or MariaDB database. We will also cover some of the variables which are frequently modified even though they should not.
Performance tuning is not easy, especially if you're not an experienced DBA, but you can go a surprisingly long way with a few basic guidelines.
Starting with 12c Release 1, Oracle introduced a completely new architecture concept for its database - the Container Database.
With this new architecture, new challenges came up but with the same breath a wide branch of new opportunities.
The presentation will address the capabilities to create fast and easy new (test) databases or clones for a running production database. Five different ways will be discussed.
- Using Local and Remote Cloning
- Using an Unplugged PDB (predefined master)
- Using Refreshable PDBs as a master for new (test) databases
- Snapshot Carousel
Another point of the agenda is the usage of the Snapshot features of ACFS and Direct NFS to speed up the creation process.
Everything you've learned about wait events in the single instance Oracle database also applies to clustered Oracle RAC databases. However, the special use of a global buffer cache in Oracle RAC makes it critical to monitor inter-instance communication via the cluster-specific wait events gc cr request and gc buffer busy.
OpenZFS novel algorithms: snapshots, space allocation, RAID-Z - Matt AhrensMatthew Ahrens
Guest lecture at Brown University's Computer Science Operating Systems class, CS167, by Matt Ahrens, co-creator of ZFS. Introduction by professor Tom Doeppner. Recording, March 2017: https://youtu.be/uJGkyMxdNFE
Topics:
- Data structures and algorithms used by ZFS snapshots
- Overview of ZFS on-disk structure
- Data structures used for ZFS space allocation
- RAID-Z compared with traditional RAID-4/5/6
Class website: http://cs.brown.edu/courses/cs167/
Talk by Brendan Gregg for USENIX LISA 2019: Linux Systems Performance. Abstract: "
Systems performance is an effective discipline for performance analysis and tuning, and can help you find performance wins for your applications and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes the topic for everyone, touring six important areas of Linux systems performance: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events) and tracing (Ftrace, bcc/BPF, and bpftrace/BPF), and much advice about what is and isn't important to learn. This talk is aimed at everyone: developers, operations, sysadmins, etc, and in any environment running Linux, bare metal or the cloud."
Computing Performance: On the Horizon (2021)Brendan Gregg
Talk by Brendan Gregg for USENIX LISA 2021. https://www.youtube.com/watch?v=5nN1wjA_S30 . "The future of computer performance involves clouds with hardware hypervisors and custom processors, servers running a new type of BPF software to allow high-speed applications and kernel customizations, observability of everything in production, new Linux kernel technologies, and more. This talk covers interesting developments in systems and computing performance, their challenges, and where things are headed."
Introduction to Ceph, an open-source, massively scalable distributed file system.
This document explains the architecture of Ceph and integration with OpenStack.
Learn how to setup Samba and NFS in ubuntu server-ubuntu client and ubuntu server-windows client. Also, comparsion of NAS vs SAN, NAS vs DAS, why we are using NAS, its comonents and challanges with actual real world scenario that what if we use NAS and what if we not use NAS.
Oracle Real Application Clusters 19c- Best Practices and Internals- EMEA Tour...Sandesh Rao
In this session, I will cover under-the-hood features that power Oracle Real Application Clusters (Oracle RAC) 19c specifically around Cache Fusion and Service management. Improvements in Oracle RAC helps in integration with features such as Multitenant and Data Guard. In fact, these features benefit immensely when used with Oracle RAC. Finally we will talk about changes to the broader Oracle RAC Family of Products stack and the algorithmic changes that helps quickly detect sick/dead nodes/instances and the reconfiguration improvements to ensure that the Oracle RAC Databases continue to function without any disruption
This presentation is from the ZFS Tutorial presented at the USENIX LISA09 Conference at Baltimore, Maryland in November 2009.
Later versions are available on slideshare.net, too.
Krzysztof Ksiazek - Severalnines AB
So, you are a developer or sysadmin and showed some abilities in dealing with databases issues. And now, you have been elected to the role of DBA. And as you start managing the databases, you wonder…
* How do I tune them to make best use of the hardware?
* How do I optimize the Operating System?
* How do I best configure MySQL or MariaDB for a specific database workload?
If you're asking yourself the following questions when it comes to optimally running your MySQL or MariaDB databases, then this talk is for you!
We will discuss some of the settings that are most often tweaked and which can bring you significant improvement in the performance of your MySQL or MariaDB database. We will also cover some of the variables which are frequently modified even though they should not.
Performance tuning is not easy, especially if you're not an experienced DBA, but you can go a surprisingly long way with a few basic guidelines.
Starting with 12c Release 1, Oracle introduced a completely new architecture concept for its database - the Container Database.
With this new architecture, new challenges came up but with the same breath a wide branch of new opportunities.
The presentation will address the capabilities to create fast and easy new (test) databases or clones for a running production database. Five different ways will be discussed.
- Using Local and Remote Cloning
- Using an Unplugged PDB (predefined master)
- Using Refreshable PDBs as a master for new (test) databases
- Snapshot Carousel
Another point of the agenda is the usage of the Snapshot features of ACFS and Direct NFS to speed up the creation process.
Everything you've learned about wait events in the single instance Oracle database also applies to clustered Oracle RAC databases. However, the special use of a global buffer cache in Oracle RAC makes it critical to monitor inter-instance communication via the cluster-specific wait events gc cr request and gc buffer busy.
OpenZFS novel algorithms: snapshots, space allocation, RAID-Z - Matt AhrensMatthew Ahrens
Guest lecture at Brown University's Computer Science Operating Systems class, CS167, by Matt Ahrens, co-creator of ZFS. Introduction by professor Tom Doeppner. Recording, March 2017: https://youtu.be/uJGkyMxdNFE
Topics:
- Data structures and algorithms used by ZFS snapshots
- Overview of ZFS on-disk structure
- Data structures used for ZFS space allocation
- RAID-Z compared with traditional RAID-4/5/6
Class website: http://cs.brown.edu/courses/cs167/
Talk by Brendan Gregg for USENIX LISA 2019: Linux Systems Performance. Abstract: "
Systems performance is an effective discipline for performance analysis and tuning, and can help you find performance wins for your applications and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes the topic for everyone, touring six important areas of Linux systems performance: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events) and tracing (Ftrace, bcc/BPF, and bpftrace/BPF), and much advice about what is and isn't important to learn. This talk is aimed at everyone: developers, operations, sysadmins, etc, and in any environment running Linux, bare metal or the cloud."
Computing Performance: On the Horizon (2021)Brendan Gregg
Talk by Brendan Gregg for USENIX LISA 2021. https://www.youtube.com/watch?v=5nN1wjA_S30 . "The future of computer performance involves clouds with hardware hypervisors and custom processors, servers running a new type of BPF software to allow high-speed applications and kernel customizations, observability of everything in production, new Linux kernel technologies, and more. This talk covers interesting developments in systems and computing performance, their challenges, and where things are headed."
Introduction to Ceph, an open-source, massively scalable distributed file system.
This document explains the architecture of Ceph and integration with OpenStack.
Learn how to setup Samba and NFS in ubuntu server-ubuntu client and ubuntu server-windows client. Also, comparsion of NAS vs SAN, NAS vs DAS, why we are using NAS, its comonents and challanges with actual real world scenario that what if we use NAS and what if we not use NAS.
Oracle Real Application Clusters 19c- Best Practices and Internals- EMEA Tour...Sandesh Rao
In this session, I will cover under-the-hood features that power Oracle Real Application Clusters (Oracle RAC) 19c specifically around Cache Fusion and Service management. Improvements in Oracle RAC helps in integration with features such as Multitenant and Data Guard. In fact, these features benefit immensely when used with Oracle RAC. Finally we will talk about changes to the broader Oracle RAC Family of Products stack and the algorithmic changes that helps quickly detect sick/dead nodes/instances and the reconfiguration improvements to ensure that the Oracle RAC Databases continue to function without any disruption
This presentation is from the ZFS Tutorial presented at the USENIX LISA09 Conference at Baltimore, Maryland in November 2009.
Later versions are available on slideshare.net, too.
Introduction to Flocker which is a lightweight volume and container manager.
Meetup details of my presentation:
http://www.meetup.com/Docker-Bangalore/events/222476025/
Apple eMac Computer manual for this Apple system directly has accessible eMac specification. It includes Apple configuration, performance, design manual.
This slide was presented at Mydbops Database Meetup 4 by Bajranj ( Zenefits ). ZFS as a filesystem has good features that can enhance MySQL by compression, Quick Snapshots and others.
OSDC 2016 - Interesting things you can do with ZFS by Allan Jude&Benedict Reu...NETWAYS
ZFS is the next generation filesystem originally developed at Sun Microsystems. Available under the CDDL, it uniquely combines volume manager and filesystem into a powerful storage management solution for Unix systems. Regardless of big or small storage requirements. ZFS offers features, for free, that are usually found only in costly enterprise storage solutions. This talk will introduce ZFS and give an overview of its features like snapshots and rollback, compression, deduplication as well as replication. We will demonstrate how these features can make a difference in the datacenter, giving administrators the power and flexibility to adapt to changing storage requirements.
Real world examples of ZFS being used in production for video streaming, virtualization, archival, and research are shown to illustrate the concepts. The talk is intended for people considering ZFS for their data storage needs and those who are interested in the features ZFS provides.
Databases are a key part of any application. The storage subsystem contributes most to performance of the database. In recent days, new storage technologies like Solid State Storage (SSD) and high performance drives are becoming cheaper and more accessible, but it takes a lot of planning to use these technologies in a cost effective way for best price-performance.
Database performance tuning for SSD based storageAngelo Rajadurai
Databases are a key part of any application. The storage subsystem contributes most to performance of the database. In recent days, new storage technologies like Solid State Storage (SSD) and high performance drives are becoming cheaper and more accessible, but it takes a lot of planning to use these technologies in a cost effective way for best price-performance.
Spinning Brown Donuts: Why Storage Still CountsSparkhound Inc.
Storage, next to server hardware, is pretty commoditized and probably the least exciting thing in your datacenter. However, not properly assessing your storage needs and requirements can be the difference between a great app or resume generating event. This session will cover topics such as: Why you may not need all flash, SAN is not just NAS spelled backwards, leveraging cloud storage, why RAID is not a sound backup solution, and cutting through the marketing to make sense of it all.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!SOFTTECHHUB
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
GridMate - End to end testing is a critical piece to ensure quality and avoid...ThomasParaiso2
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
Monitoring Java Application Security with JDK Tools and JFR Events
ZFS in 30 minutes
1. In ~ 30 Minutes
bill.hathaway@gmail.com
Thursday, January 14, 2010 1
2. What is ZFS?
A modern take on storage
Safe - transaction based, checksums
Simple - 2 main commands
Efficient - share resources well
Dynamic - change on the fly
Thursday, January 14, 2010 2
Zettabyte is 2^70 bytes
ZFS was developed in response to dealing with 20+ years of complexity and limitations
around storage. Todays disk drives are a roughly a million times bigger than the first
hard disks. File systems have all sorts of whacky limitations like the number of inodes
or items in a directory. Sysadmins have to make choices about how much space to
assign to volumes and file systems and then later shrink or add space if they were
wrong. Troubleshooting physical hardware, logical volume managers, and file systems
increases complexity. Fsck times mean outages can that take hours or days.
With filesystems, safety is a hard constraint. ZFS provides safety through two mechanisms
a Copy-on-Write architecture that uses transactions to keep the on-disk format consistent
and checksums that can tell if any data is corrupted. If you have redundancy, then the
problem can be fixed. No FSCK is needed. Ever.
ZFS is easy to administer because it simplifies the administration model. There are only 2
main commands. Each of these have sub-commands, but the usage is consistent and tries to
be intuitive.
ZFS is efficient in that it uses a concept called pooled storage. This means that disk space
and disk bandwidth can be easily shared among multiple file systems.
Dynamic - storage can be added, new file systems created, or properties changed all while
running
3. Where can I use ZFS?
zfs-fuse.net
Nexenta code.google.com/p/maczfs
Thursday, January 14, 2010 3
The operating systems on the left all support ZFS natively.
Apple originally was porting ZFS to OS X, but apparently there was an unsolved licensing
dispute and that is no longer the case. A group of enthusiasts are continuing to work with
the code to make it available to Mac users.
KQ Infotech announced they would be porting ZFS to Linux and maintaining the port.
Blog at kqinfotech.wordpress.com
4. ZFS Basic Terms
Volume - block device
File system - standard POSIX FS layer
Snapshot - read-only copy of a FS
Clone - read-write copy of a FS
Dataset - any of the 4 terms above
Pool - logical set of vdevs
VDev - block storage (redundancy done here)
Thursday, January 14, 2010 4
ZFS volumes are typically used to support iSCSI luns or for swap devices
5. Copy-on-Write
1. Initial tree blocks 2. CoW some blocks
4. Rewrite uberblock 3. CoW indirect blocks
Thursday, January 14, 2010 5
Using the copy-on-write mechanism allows ZFS to operate very safely.
In this example, the original data blocks are blue, and new data is green.
6. Pooled Storage
Manage disks
more like
RAM
Thursday, January 14, 2010 6
Pooled storage means that as you add storage to ZFS you don’t need to worry about micro-
managing it. The storage is made available to all the file systems using the pool, similar to
how when you add DIMMs to a system you don’t need to reconfigure anything.
No dimmconfig
No /etc/dimmtab
No fdimm
You can create multiple pools of storage per server.
7. Pooled Storage
Underlying storage is manipulated via zpool
# zpool create
# zpool list
# zpool status
# zpool add
Thursday, January 14, 2010 7
There are also additional sub-commands for tasks such as replacing drives or scrubbing
data.
Redundancy is handled at the pool level. When you create a pool you can add drives in a
striped fashion, as mirrors, or in a parity configuration similar to RAID5 or 6.
You can also have multiple pools per a machine. An example is where you have a pool
mirrored storage to support a high-performance database using 15k RPM disks and another
pool that uses slower SATA based storage in a RAID5 like configuration holding archived data.
8. Vdev Types
Data Parity
Non-redundant Redundant
Single disk Mirror
Striped RAIDZ
Thursday, January 14, 2010 8
Here we are showing the types of vdevs that can be used as building blocks for ZFS pools.
If we start with a single disk, we can expand it later either to a mirrored vdev by attaching a
disk or we could change to a striped configuration by adding a disk.
One common complaint with ZFS is that you can’t remove disks from striped or RAIDZ
configurations, even if you have plenty of available space. This will be remedied when a
feature called “block pointer rewrite” gets integrated.
There are double and triple parity versions of RAIDZ called RAIDZ2 and RAIDZ3.
9. Pooled Storage
# zpool create data mirror c1t0d0 c2t0d0
# zpool list data
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
data 496G 164K 496G 0% 1.00x ONLINE -
# df -h /data
Filesystem size used avail capacity Mounted on
data 488G 24K 488G 1% /data
Thursday, January 14, 2010 9
Here we will create a pool consisting of mirrored storage and then run a zpool list command
to see how much space is available.
A file system mounted on the name of the pool will be available by default
10. Pooled Storage
# zpool status
pool: data
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
errors: No known data errors
Thursday, January 14, 2010 10
The status command tells us which drives are part of the pool and if they have had any errors
11. Pooled Storage
# zpool create data mirror c1t0d0 c2t0d0
# zpool list data
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
data 496G 164K 496G 0% 1.00x ONLINE -
# zpool add data mirror c3t0d0 c4t0d0
# zpool list data
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
data 992G 164K 992G 0% 1.00x ONLINE -
Thursday, January 14, 2010 11
12. Pooled Storage
# zpool status
pool: data
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c3t0d0 ONLINE 0 0 0
c4t0d0 ONLINE 0 0 0
errors: No known data errors
Thursday, January 14, 2010 12
The status command tells us which drives are part of the pool and if they have had any errors
13. Pool Evolution
zpool create data c1t0d0
zpool attach data c2t0d0
zpool add data mirror
c3t0d0 c4t0d0
zpool destroy data
Thursday, January 14, 2010 13
Here is an example where we start with a single disk.
Using the zpool attach command we can add a mirror to the original disk.
If we want more space or better performance, we can also add another pair of mirrored disks.
If we decide we are all done with the pool and want to remove all the data we can use the
zpool destroy command to free up the disks
14. File Systems
FS manipulation done via zfs command
# zfs create data/web
# zfs set compression=on data/web
# zfs snapshot data/web@before_upgrade
# zfs create data/home
# zfs create data/home/slaney
Thursday, January 14, 2010 14
zfs has several sub commands, the most common used are create, set, get, and list
15. ZFS Properties
All objects have props that change behavior
Properties are typically inherited
# zfs set property=value $dataset
# zfs get <$propname|all> $dataset
# zfs set mountpoint=/apache data/web
Thursday, January 14, 2010 15
All ZFS objects have properties that can control their behavior.
Most inherit from their parent.
16. ZFS Properties
# zfs get all data/web
NAME PROPERTY VALUE SOURCE
data/web type filesystem -
data/web used 329M -
data/web available 992G -
data/web compressratio 1.75x -
data/web quota none default
data/web mountpoint /apache local
data/web checksum on default
data/web compression on local
data/web atime off inherited from data
Thursday, January 14, 2010 16
This example shows some properties for one of our file systems.
The first few lines contain read-only properties and can’t be set.
The compress ratio shows how much space is being saved by compression if it is active
We can see that the mountpoint is set to /apache and the source is local, meaning it is due to
a setting we did explicitly to this file system
Lower we can see that the atime property is disabled and that the source shows it is inherited
from a parent object
17. File Systems
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
data 131K 488G 23K /data
data/home 21K 488G 21K /data/home
data/home/slaney 21K 488G 21K /data/home/slaney
data/web 21K 488G 21K /apache
# zfs set mountpoint=/home data/home
# zfs list | grep home
NAME USED AVAIL REFER MOUNTPOINT
data/home 21K 488G 21K /home
data/home/slaney 21K 488G 21K /home/slaney
Thursday, January 14, 2010 17
18. ZFS File Systems
zfs set quota=10g data/home
zfs set compression=on data/home
zfs create data/postgres
zfs set reservation=50g data/postgres
zfs set atime=off data/home/slaney
Thursday, January 14, 2010 18
19. Snapshots
A read-only copy of a filesystem
Can be used to roll-back to a previous state
# zfs snapshot $fs@$name
# zfs snapshot data/web@pre_upgrade
# zfs snapshot data/web@post_upgrade
# zfs snapshot data/home@`date +%F`
Thursday, January 14, 2010 19
21. Clones
A clone is a read-write copy based on a
snapshot.
# zfs snapshot data/postgres@for_test
# zfs clone data/postgres@for_test data/pgtest
# zfs set mountpoint=/postgres2 data/pg_test
Thursday, January 14, 2010 21
Clones have a dependency on the snapshot they are built from.
22. Data Replication
1. Take a snapshot
# zfs snapshot data/postgres@2009-12-31
2. Use zfs send/recieve
# zfs send data/posgres@2009-12-31 |
ssh $remote_host zfs receive $dataset
3. Later .... use optional incremental update
# zfs snapshot data/postgres@2010-01-02
# zfs send -i data/postgres@2009-12-31 data/postgres@2010-01-02 |
ssh $remote_host zfs receive $dataset
Thursday, January 14, 2010 22
The zfs send/receive stream can be piped over SSH or your transport of choice
You can obviously also use rsync
23. De-Duplication
PAIN WARNING: Still in heavy development as of
2010/01/12
To activate:
# zfs set dedup=on $dataset
To view space savings:
# zpool list $dataset
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
tpool 136G 33.7G 102G 24% 3.16x ONLINE -
Thursday, January 14, 2010 23
24. Solid State Disk
Leverages SSD + standard drives
SSD can be used as either (or both)
• Write accelerator for intent log (ZIL)
• 2nd level cache for reads (L2ARC)
Thursday, January 14, 2010 24
ZFS uses the best parts of hard disk drives (large capacity) and the best parts of SSDs (fast
access speed)
A SSD can also be partitioned so that a small portion is used for the intent log and a larger
section is used for L2ARC
25. ZFS in OpenSolaris
• Packaging system is ZFS aware
• OS upgrades works on a clone of /
• Adds extra grub entry
• Reboot into new OS rev
• If trouble, just reboot and pick previous rev
Thursday, January 14, 2010 25
26. ZFS Resources
• LMGTFY “ZFS best practice guide”
• LMGTFY “ZFS evil tuning guide”
• zfs-discuss @ opensolaris.org
• http://tinyurl.com/zfshome
Thursday, January 14, 2010 26