ZFS is a filesystem and volume manager that provides data integrity and efficiency features. It can be used with FreeNAS and PC-BSD to create storage pools by combining disks. Pools can be logically divided into datasets or zvols. Key ZFS features include RAIDZ parity protection, snapshots, clones, compression, and self-healing capabilities. ZFS is well-suited for network storage and provides advanced features for optimizing storage and protecting data.
In computing, ZFS is a combined file system and logical volume manager designed by Sun Microsystems, a subsidiary of Oracle Corporation. The features of ZFS include support for high storage capacities, integration of the concepts of file system and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z and native NFSv4 ACLs. Unlike traditional file systems, which reside on single devices and thus require a volume manager to use more than one device, ZFS file systems are built on top of virtual storage pools called zpools. A zpool is constructed of virtual devices (vdevs), which are themselves constructed of block devices: files, hard drive partitions, or entire drives, with the last being the recommended usage.[7] Thus, a vdev can be viewed as a group of hard drives. This means a zpool consists of one or more groups of drives.
In addition, pools can have hot spares to compensate for failing disks. In addition, ZFS supports both read and write caching, for which special devices can be used. Solid State Devices can be used for the L2ARC, or Level 2 ARC, speeding up read operations, while NVRAM buffered SLC memory can be boosted with supercapacitors to implement a fast, non-volatile write cache, improving synchronous writes. Finally, when mirroring, block devices can be grouped according to physical chassis, so that the filesystem can continue in the face of the failure of an entire chassis. Storage pool composition is not limited to similar devices but can consist of ad-hoc, heterogeneous collections of devices, which ZFS seamlessly pools together, subsequently doling out space to diverse file systems as needed. Arbitrary storage device types can be added to existing pools to expand their size at any time. The storage capacity of all vdevs is available to all of the file system instances in the zpool. A quota can be set to limit the amount of space a file system instance can occupy, and a reservation can be set to guarantee that space will be available to a file system instance.
These slides cover the very basics of Hadoop architecture, in particular HDFS. This was my presentation in the first Delhi Hadoop User Group (DHUG) meetup held at Gurgaon on 10th September 2011. Loved the positive feedback. I'll also upload a more elaborate version covering Hadoop mapreduce architecture as well soon. Most of the stuff covered in these slides can be found in Tom White's book as well (See the last slide)
OSDC 2016 - Interesting things you can do with ZFS by Allan Jude&Benedict Reu...NETWAYS
ZFS is the next generation filesystem originally developed at Sun Microsystems. Available under the CDDL, it uniquely combines volume manager and filesystem into a powerful storage management solution for Unix systems. Regardless of big or small storage requirements. ZFS offers features, for free, that are usually found only in costly enterprise storage solutions. This talk will introduce ZFS and give an overview of its features like snapshots and rollback, compression, deduplication as well as replication. We will demonstrate how these features can make a difference in the datacenter, giving administrators the power and flexibility to adapt to changing storage requirements.
Real world examples of ZFS being used in production for video streaming, virtualization, archival, and research are shown to illustrate the concepts. The talk is intended for people considering ZFS for their data storage needs and those who are interested in the features ZFS provides.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
1. ZFS 101
aka did you know that ZFS is cool and
that you could be using it?
Dru Lavigne
Documentation Lead, iXsystems
OLF, September 14, 2013
2. Outline
Discuss ZFS features and describe how these
features are configured in the following operating
systems:
FreeNAS: open source NAS (Network Attached
Storage)
PC-BSD: open source desktop (GUI) or server
(CLI)
Latest versions of both operating systems use the
most recent open source ZFS (v5000 or feature
flags)
3. What is ZFS?
128-bit COW (Copy on Write) filesystem and
logical volume manager with a maximum pool/file
size of 16 exabytes
Unlike traditional Unix filesystems, you are not
limited to the partition sizes and mount points
defined at filesystem creation time
Instead, disks are fed to a “pool” and the size of
that pool can increase as the need for disk
capacity increases (does require pre-planning)
4. Pool
Root (parent) volume which can be logically sub-
divided as needed--can create one or multiple
pools
The number of disks added at a time is known as
a “vdev”
To optimize performance and resilvering time,
number of disks per vdev is limited
As more capacity is needed, add identical vdevs--
these will be striped into the pool
5. RAIDZ
RAIDZ* levels designed to overcome hardware
RAID limitations such as the write-hole and
corrupt data written over time before the controller
provides an alert
Designed for commodity disks so no RAID
controller is needed
Can also be used with a RAID controller, but it
should be put in JBOD mode
6. RAIDZ1
Parity blocks are distributed across all disks
Up to one disk can fail per vdev without losing
pool
Pool can be lost if second disk in a vdev fails
before resilver completes
Optimized for vdev of 3, 5, or 9 disks
7. RAIDZ2
Double-parity solution similar to RAID6
Parity blocks are distributed across all disks
Up to two disks can fail per vdev without losing
pool, with no restrictions on which disks can fail
Optimized for vdev of 4, 6, or 10 disks
8. RAIDZ3
Triple-parity solution
Parity blocks are distributed across all disks
Up to three disks can fail per vdev without losing
pool, with no restrictions on which disks can fail
Optimized for vdev of 5, 7, or 11 disks
11. ZIL
ZFS Intent Log
Effectively a filesystem journal that manages
writes
A dedicated SSD as a secondary log device
(SLOG) can increase synchronous write
performance, will have no effect on asynchronous
writes
12. ARC
ARC refers to read cache in RAM
Expect a miss for random reads and a hit for
cached reads
Takes time for ARC to populate; if high misses
continue for cached reads, the system needs to
be tuned
Freenas adds ARC stats to top(1) and includes
arc_summary.py and arcstat.py tools for ARC
monitoring
13. L2ARC
Optional, secondary ARC which can be installed
on SSD or disk in order to increase random read
performance
Always add as much RAM as possible first
L2ARC is populated over time with “hot” reads
15. Datasets
Pool can be divided into datasets
Similar to a folder in that it supports permissions
Similar to a filesystem in that you can set
properties such as quotas and compression (built-
in LZ4 compression is near real-time speed)
A well thought out design can optimize storage for
the type of data being stored
16. Properties
Dozens of configurable properties such as: atime
(access time), canmount, compression, copies,
dedup, exec, quota, userquota, groupquota,
readonly, recordsize, reservation, setuid, etc.
Descriptions can be found at
http://www.freebsd.org/cgi/man.cgi?query=zfs
19. Zvols
Pool can also be divided into zvols
Essentially, a virtual, raw block device
Ideal for iSCSI device extents
Regardless of the filesytem the zvol is formatted
with by the iSCSI initiator, the underlying disk
blocks still benefit from all of the features provided
by ZFS
21. Snapshots
Provide read-only, point-in-time image of the
specified pool, dataset, or zvol
Snapshots can be recursive (atomic inclusion of
all child datasets)
Initial size is 0 bytes as COW, snapshot increases
in size as changes are written to disk
Can be replicated to another system
22. Clones
Provide read-write copies of read-only snapshots
Initial size of clone is 0 bytes
Clones can be mounted and used to access data
from that point in time (e.g. to restore an earlier
revision of a file)
25. Snapshot Restore
In PC-BSD 9.2, the Life Preserver utility can be
used to automate full system backups to another
system running rsync+ssh, or to a FreeNAS
system
The backup can be used to perform an operating
system restore from a PC-BSD install media,
should the system become unusable
27. Scrubs
ZFS was designed to be self-healing; it creates
and verifies checksums as data is written to disk
A scrub verifies the checksum in each disk block
and attempts to correct data as necessary
I/O intensive so should be scheduled
appropriately
Reading the scrub results can provide an early
indication of possible disk failure
29. Deduplication
Used to free blocks containing identical data
(updates reference pointers)
Can improve performance on datasets containing
duplicate data (e.g. virtual images)
Dedup tables should fit into L2ARC; systems with
limited RAM or no L2ARC might freeze hard
Recommended is at least 5 GB RAM/L2ARC per
TB of storage to be deduplicated
30. Boot Environments
A snapshot of the dataset the operating system
resides on can be taken before an upgrade or a
system configuration change
This saved “boot environment” is automatically
added to the GRUB boot manager
Should the upgrade or configuration change fail,
simply reboot and select the previous boot
environment from the boot menu