The document discusses synchronization and consistency issues in distributed file systems. It covers several consistency models including UNIX semantics, session semantics, and immutable files. It also discusses transaction semantics and file locking techniques. The document then focuses on consistency and replication, covering client-side caching, server-side replication, and replication in peer-to-peer systems. It provides examples of caching techniques used in NFS including caching with server control and open delegation. It also discusses the use of leases to maintain consistency across cached copies.
This document discusses task migration in distributed systems. It defines task migration as the preemptive transfer of a partially executed task to another node. The document outlines the group members working on the task migration project and provides an introduction to terminology used. It then discusses reasons for task migration like load distribution and improving communication. The main issues discussed are state transfer costs, location transparency, migration mechanism structure, and performance impacts.
This document discusses concurrency and processes. It begins by defining threads of execution and how operating systems provide the illusion of multiple processors through time-sharing techniques. It then discusses how modern operating systems abstract processes to provide protection between threads through separate address spaces and the virtualization of hardware resources. The document covers techniques like context switching, scheduling, and inter-process communication to enable concurrency while maintaining isolation between processes.
1. Multicore processors require applications to be parallelized to avoid performance stagnation. Data management, programming models, and on-chip communication all impact performance.
2. Symmetric multiprocessors (SMPs) have uniform memory access times, while distributed memory systems have faster local memory access but slower remote access.
3. Shared memory architectures allow any processor to access any memory location directly using load/store instructions, while message passing involves explicit data transfer between processes using send and receive calls.
This document discusses data replication in distributed database management systems. It covers the purposes of replication like availability, performance, and scalability. It also discusses consistency models for replicated data, including mutual consistency and transaction consistency. For update management strategies, it describes eager and lazy propagation as well as centralized and distributed techniques. Specific replication protocols are also outlined, such as single master, primary copy, eager centralized/distributed, and lazy centralized/distributed protocols.
Lecture 7, 8, 9 and 10 Inter Process Communication (IPC) in Operating SystemsRushdi Shams
This document discusses inter-process communication and solutions to classical synchronization problems. It describes three issues in IPC: how processes can pass information, avoiding interference with shared resources, and proper sequencing of dependent processes. Race conditions are explained using an example of a printer queue. Solutions to achieve mutual exclusion are explored, including disabling interrupts, lock variables, strict alternation, and Peterson's algorithm. The producer-consumer problem is presented and solved using semaphores. Barriers and the dining philosophers problem are also introduced.
The document discusses concurrency in operating systems. It notes that operating systems must manage multiple concurrent processes through techniques like multiprogramming and multiprocessing. This introduces challenges around sharing resources and non-deterministic execution orders. It provides examples of race conditions that can occur without proper synchronization and discusses requirements for implementing mutual exclusion like critical sections to avoid issues like deadlock and starvation.
OSI Model Layers and Internet Protocol StackShivam Mitra
1. Advantages of dividing a complex system into layers or modules
2. Taking example of airline system to explain advantage of layering
3. OSI model and Internet Protocol Stack
4. Functions of each layers in OSI model
This document discusses task migration in distributed systems. It defines task migration as the preemptive transfer of a partially executed task to another node. The document outlines the group members working on the task migration project and provides an introduction to terminology used. It then discusses reasons for task migration like load distribution and improving communication. The main issues discussed are state transfer costs, location transparency, migration mechanism structure, and performance impacts.
This document discusses concurrency and processes. It begins by defining threads of execution and how operating systems provide the illusion of multiple processors through time-sharing techniques. It then discusses how modern operating systems abstract processes to provide protection between threads through separate address spaces and the virtualization of hardware resources. The document covers techniques like context switching, scheduling, and inter-process communication to enable concurrency while maintaining isolation between processes.
1. Multicore processors require applications to be parallelized to avoid performance stagnation. Data management, programming models, and on-chip communication all impact performance.
2. Symmetric multiprocessors (SMPs) have uniform memory access times, while distributed memory systems have faster local memory access but slower remote access.
3. Shared memory architectures allow any processor to access any memory location directly using load/store instructions, while message passing involves explicit data transfer between processes using send and receive calls.
This document discusses data replication in distributed database management systems. It covers the purposes of replication like availability, performance, and scalability. It also discusses consistency models for replicated data, including mutual consistency and transaction consistency. For update management strategies, it describes eager and lazy propagation as well as centralized and distributed techniques. Specific replication protocols are also outlined, such as single master, primary copy, eager centralized/distributed, and lazy centralized/distributed protocols.
Lecture 7, 8, 9 and 10 Inter Process Communication (IPC) in Operating SystemsRushdi Shams
This document discusses inter-process communication and solutions to classical synchronization problems. It describes three issues in IPC: how processes can pass information, avoiding interference with shared resources, and proper sequencing of dependent processes. Race conditions are explained using an example of a printer queue. Solutions to achieve mutual exclusion are explored, including disabling interrupts, lock variables, strict alternation, and Peterson's algorithm. The producer-consumer problem is presented and solved using semaphores. Barriers and the dining philosophers problem are also introduced.
The document discusses concurrency in operating systems. It notes that operating systems must manage multiple concurrent processes through techniques like multiprogramming and multiprocessing. This introduces challenges around sharing resources and non-deterministic execution orders. It provides examples of race conditions that can occur without proper synchronization and discusses requirements for implementing mutual exclusion like critical sections to avoid issues like deadlock and starvation.
OSI Model Layers and Internet Protocol StackShivam Mitra
1. Advantages of dividing a complex system into layers or modules
2. Taking example of airline system to explain advantage of layering
3. OSI model and Internet Protocol Stack
4. Functions of each layers in OSI model
The document discusses processes and process management in an operating system. A process is an instance of a computer program being executed and contains the program code and current activity. Processes go through various states like ready, running, waiting, and terminated. The operating system uses a process control block (PCB) to maintain information about each process like its state, program counter, memory allocation, and other details. Key process operations include creation, termination, and context switching between processes using the PCB.
The document discusses processes and threads in operating systems. It describes how processes contain multiple threads that can run concurrently on multicore systems. Each thread has its own execution state and context stored in a thread control block. When a thread is not actively running, its context is saved so it can resume execution later. The document provides examples of how threads are implemented in Windows and Linux operating systems.
This document discusses multithreading and the differences between tasks and threads. It explains that operating systems manage each application as a separate task, and when an application initiates an I/O request it creates a thread. Multithreading allows a single process to support multiple concurrent execution paths. Benefits of threads include less overhead for creation, termination, and context switching compared to processes. The document concludes that threads enhance efficiency by sharing resources within a process.
Replication in computing involves sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.
The document provides a historical overview of Linux and its origins from Unix. It discusses how Unix was initially developed at Bell Labs in the 1960s-1970s and was then further developed into various commercial and non-commercial versions. It also outlines how the GNU project was founded to create a free Unix-like operating system and how Linux was later created by Linus Torvalds to provide the missing kernel to GNU. Finally, it briefly describes the structure of Linux systems including the kernel, user space, and popular desktop environments.
Inter-process communication (IPC) allows processes to communicate and synchronize actions. There are two main models - shared memory, where processes directly read/write shared memory, and message passing, where processes communicate by sending and receiving messages. Critical sections are parts of code that access shared resources and must be mutually exclusive to avoid race conditions. Semaphores can be used to achieve mutual exclusion, with operations P() and V() that decrement or increment the semaphore value to control access to the critical section. For example, in the producer-consumer problem semaphores can suspend producers if the buffer is full and consumers if empty, allowing only one process at a time in the critical section.
This document discusses various consistency protocols for replicated data in distributed systems. It covers primary-based protocols where each data item has a primary replica, and replication-based protocols where writes can be done at multiple replicas. For primary-based protocols, it distinguishes between remote-write protocols where operations are done remotely at the primary and local-write protocols where the primary may be copied locally. It also discusses quorum-based replication protocols that use voting to achieve consistency across replicas.
Hyper-V High Availability and Live MigrationPaulo Freitas
This document provides an overview of a Microsoft Virtual Academy training program on Hyper-V virtualization. The program is split into two halves, with the first half covering topics like Hyper-V infrastructure, networking, storage, and management. The second half focuses on high availability, disaster recovery, and integrating Hyper-V with System Center. It also discusses capabilities like live migration, replication, clustering and improving application availability and redundancy through virtualization.
Yabusame: postcopy live migration for qemu/kvmIsaku Yamahata
Yabusame is a postcopy live migration technique for QEMU/KVM. It was developed by Isaku Yamahata of VALinux Systems Japan K.K. and Takahiro Hirofuchi of AIST. The project aims to improve live migration performance by allowing the guest VM to resume execution at the destination host before memory pages have been fully copied. This is achieved through asynchronous page fault handling during the postcopy phase. Evaluation shows the technique can improve CPU utilization and reduce total migration times compared to traditional precopy approaches. Future work includes upstream integration, support for KSM/THP, multithreading optimizations, and integration with management platforms like libvirt and OpenStack.
This document discusses parallel architecture and parallel programming. It begins with an introduction to von Neumann architecture and serial computation. Then it defines parallel architecture, outlines its benefits, and describes classifications of parallel processors including multiprocessor architectures. It also discusses parallel programming models, how to design parallel programs, and examples of parallel algorithms. Specific topics covered include shared memory and distributed memory architectures, message passing and data parallel programming models, domain and functional decomposition techniques, and a case study on developing parallel web applications using Java threads and mobile agents.
This is the Complete Information about Data Replication you need, i am focused on these topics:
What is replication?
Who use it?
Types ?
Implementation Methods?
The operating system performs key functions like scheduling, synchronization, and memory management. Scheduling involves allocating time to programs and switching between threads. Synchronization ensures mutual exclusion so only one process accesses shared resources at a time using techniques like semaphores. Memory management uses virtual memory to allow processes to access more memory than physically available by swapping pages between disk and RAM.
This document discusses key components of an operating system including the kernel, file manager, device drivers, memory manager, scheduler, and dispatcher. It provides details on how the file manager maintains file records and permissions. Device drivers communicate with peripheral devices. The memory manager coordinates main memory usage. The scheduler maintains process information and chooses which to run next using algorithms like priority scheduling. The dispatcher oversees process execution through time slicing and context switching. The document also covers semaphores, deadlocks, and their prevention.
RTLinux is a real-time operating system that allows real-time applications to run on top of Linux. It modifies the Linux kernel to add a virtual machine layer with a separate task scheduler that prioritizes real-time tasks over standard Linux processes. This enables RTLinux to support hard real-time deadlines. Programming in RTLinux involves creating modules that can be loaded and unloaded from the kernel using specific commands. Real-time threads and synchronization objects like mutexes are implemented using POSIX interfaces.
This document discusses real-time operating systems (RTOS) and real-time Linux. It begins with an overview of operating systems and their tasks before discussing what real-time means in terms of deterministic and predictable response times. It then describes what qualifies as a real-time operating system and provides some examples. The document notes that while Linux can support some real-time capabilities, it was not designed as a RTOS. It explores some real-time Linux options like RTAI, Xenomai, and PREEMPT-RT that aim to enhance Linux's real-time performance and determinism. It also briefly mentions the LITMUS RT testbed for evaluating real-time scheduling algorithms on multiprocessors
Client-Centric Consistency
Provide guarantees about ordering of operations only for a single client, i.e.
Effects of an operations depend on the client performing it
Effects also depend on the history of client’s operations
Applied only when requested by the client
No guarantees concerning concurrent accesses by different clients
Assumption:
Clients can access different replicas, e.g. mobile users
The document discusses three common multithreading models: many-to-one, one-to-one, and many-to-many. It also describes common high-level program structures for multithreaded programs like the boss/workers model, pipeline model, up-calls, and using version stamps to keep shared information consistent.
This slides indicate an introduction on the definition of real time and RTOSes, then you can find information on introducing RT Linux approaches and comparing them with each other, then finally you can see a latency measurement test done by "Linutronix" in the slides
Network File System (NFS) is a distributed file system protocol that allows users to access and share files located on remote computers as if they were local. NFS runs on top of RPC and supports operations like file reads, writes, lookups and locking. It uses a stateless client-server model where clients make requests to NFS servers, which are responsible for file storage and operations. NFS provides mechanisms for file sharing, locking, caching and replication to enable reliable access and performance across a network.
This document discusses distributed file systems. It begins by defining key terms like filenames, directories, and metadata. It then describes the goals of distributed file systems, including network transparency, availability, and access transparency. The document outlines common distributed file system architectures like client-server and peer-to-peer. It also discusses specific distributed file systems like NFS, focusing on their protocols, caching, replication, and security considerations.
The document discusses processes and process management in an operating system. A process is an instance of a computer program being executed and contains the program code and current activity. Processes go through various states like ready, running, waiting, and terminated. The operating system uses a process control block (PCB) to maintain information about each process like its state, program counter, memory allocation, and other details. Key process operations include creation, termination, and context switching between processes using the PCB.
The document discusses processes and threads in operating systems. It describes how processes contain multiple threads that can run concurrently on multicore systems. Each thread has its own execution state and context stored in a thread control block. When a thread is not actively running, its context is saved so it can resume execution later. The document provides examples of how threads are implemented in Windows and Linux operating systems.
This document discusses multithreading and the differences between tasks and threads. It explains that operating systems manage each application as a separate task, and when an application initiates an I/O request it creates a thread. Multithreading allows a single process to support multiple concurrent execution paths. Benefits of threads include less overhead for creation, termination, and context switching compared to processes. The document concludes that threads enhance efficiency by sharing resources within a process.
Replication in computing involves sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.
The document provides a historical overview of Linux and its origins from Unix. It discusses how Unix was initially developed at Bell Labs in the 1960s-1970s and was then further developed into various commercial and non-commercial versions. It also outlines how the GNU project was founded to create a free Unix-like operating system and how Linux was later created by Linus Torvalds to provide the missing kernel to GNU. Finally, it briefly describes the structure of Linux systems including the kernel, user space, and popular desktop environments.
Inter-process communication (IPC) allows processes to communicate and synchronize actions. There are two main models - shared memory, where processes directly read/write shared memory, and message passing, where processes communicate by sending and receiving messages. Critical sections are parts of code that access shared resources and must be mutually exclusive to avoid race conditions. Semaphores can be used to achieve mutual exclusion, with operations P() and V() that decrement or increment the semaphore value to control access to the critical section. For example, in the producer-consumer problem semaphores can suspend producers if the buffer is full and consumers if empty, allowing only one process at a time in the critical section.
This document discusses various consistency protocols for replicated data in distributed systems. It covers primary-based protocols where each data item has a primary replica, and replication-based protocols where writes can be done at multiple replicas. For primary-based protocols, it distinguishes between remote-write protocols where operations are done remotely at the primary and local-write protocols where the primary may be copied locally. It also discusses quorum-based replication protocols that use voting to achieve consistency across replicas.
Hyper-V High Availability and Live MigrationPaulo Freitas
This document provides an overview of a Microsoft Virtual Academy training program on Hyper-V virtualization. The program is split into two halves, with the first half covering topics like Hyper-V infrastructure, networking, storage, and management. The second half focuses on high availability, disaster recovery, and integrating Hyper-V with System Center. It also discusses capabilities like live migration, replication, clustering and improving application availability and redundancy through virtualization.
Yabusame: postcopy live migration for qemu/kvmIsaku Yamahata
Yabusame is a postcopy live migration technique for QEMU/KVM. It was developed by Isaku Yamahata of VALinux Systems Japan K.K. and Takahiro Hirofuchi of AIST. The project aims to improve live migration performance by allowing the guest VM to resume execution at the destination host before memory pages have been fully copied. This is achieved through asynchronous page fault handling during the postcopy phase. Evaluation shows the technique can improve CPU utilization and reduce total migration times compared to traditional precopy approaches. Future work includes upstream integration, support for KSM/THP, multithreading optimizations, and integration with management platforms like libvirt and OpenStack.
This document discusses parallel architecture and parallel programming. It begins with an introduction to von Neumann architecture and serial computation. Then it defines parallel architecture, outlines its benefits, and describes classifications of parallel processors including multiprocessor architectures. It also discusses parallel programming models, how to design parallel programs, and examples of parallel algorithms. Specific topics covered include shared memory and distributed memory architectures, message passing and data parallel programming models, domain and functional decomposition techniques, and a case study on developing parallel web applications using Java threads and mobile agents.
This is the Complete Information about Data Replication you need, i am focused on these topics:
What is replication?
Who use it?
Types ?
Implementation Methods?
The operating system performs key functions like scheduling, synchronization, and memory management. Scheduling involves allocating time to programs and switching between threads. Synchronization ensures mutual exclusion so only one process accesses shared resources at a time using techniques like semaphores. Memory management uses virtual memory to allow processes to access more memory than physically available by swapping pages between disk and RAM.
This document discusses key components of an operating system including the kernel, file manager, device drivers, memory manager, scheduler, and dispatcher. It provides details on how the file manager maintains file records and permissions. Device drivers communicate with peripheral devices. The memory manager coordinates main memory usage. The scheduler maintains process information and chooses which to run next using algorithms like priority scheduling. The dispatcher oversees process execution through time slicing and context switching. The document also covers semaphores, deadlocks, and their prevention.
RTLinux is a real-time operating system that allows real-time applications to run on top of Linux. It modifies the Linux kernel to add a virtual machine layer with a separate task scheduler that prioritizes real-time tasks over standard Linux processes. This enables RTLinux to support hard real-time deadlines. Programming in RTLinux involves creating modules that can be loaded and unloaded from the kernel using specific commands. Real-time threads and synchronization objects like mutexes are implemented using POSIX interfaces.
This document discusses real-time operating systems (RTOS) and real-time Linux. It begins with an overview of operating systems and their tasks before discussing what real-time means in terms of deterministic and predictable response times. It then describes what qualifies as a real-time operating system and provides some examples. The document notes that while Linux can support some real-time capabilities, it was not designed as a RTOS. It explores some real-time Linux options like RTAI, Xenomai, and PREEMPT-RT that aim to enhance Linux's real-time performance and determinism. It also briefly mentions the LITMUS RT testbed for evaluating real-time scheduling algorithms on multiprocessors
Client-Centric Consistency
Provide guarantees about ordering of operations only for a single client, i.e.
Effects of an operations depend on the client performing it
Effects also depend on the history of client’s operations
Applied only when requested by the client
No guarantees concerning concurrent accesses by different clients
Assumption:
Clients can access different replicas, e.g. mobile users
The document discusses three common multithreading models: many-to-one, one-to-one, and many-to-many. It also describes common high-level program structures for multithreaded programs like the boss/workers model, pipeline model, up-calls, and using version stamps to keep shared information consistent.
This slides indicate an introduction on the definition of real time and RTOSes, then you can find information on introducing RT Linux approaches and comparing them with each other, then finally you can see a latency measurement test done by "Linutronix" in the slides
Network File System (NFS) is a distributed file system protocol that allows users to access and share files located on remote computers as if they were local. NFS runs on top of RPC and supports operations like file reads, writes, lookups and locking. It uses a stateless client-server model where clients make requests to NFS servers, which are responsible for file storage and operations. NFS provides mechanisms for file sharing, locking, caching and replication to enable reliable access and performance across a network.
This document discusses distributed file systems. It begins by defining key terms like filenames, directories, and metadata. It then describes the goals of distributed file systems, including network transparency, availability, and access transparency. The document outlines common distributed file system architectures like client-server and peer-to-peer. It also discusses specific distributed file systems like NFS, focusing on their protocols, caching, replication, and security considerations.
Course 102: Lecture 27: FileSystems in Linux (Part 2)Ahmed El-Arabawy
This lecture goes through the different types of Filesystems and some commands that are used with filesystems. It introduces the filesystems ext2/3/4 , JFFS2, cramfs, ramfs, tmpfs, and NFS.
Video for this lecture on youtube:
http://www.youtube.com/watch?v=XPtPsc6uaKY
Check the other Lectures and courses in
http://Linux4EnbeddedSystems.com
or Follow our Facebook Group at
- Facebook: @LinuxforEmbeddedSystems
Lecturer Profile:
Ahmed ElArabawy
- https://www.linkedin.com/in/ahmedelarabawy
Concurrency Control in Distributed Systems.pptxMArshad35
Concurrency control in distributed systems allows multiple transactions to execute simultaneously by coordinating access to shared data. There are several concurrency control algorithms like two-phase locking and timestamp ordering that prevent interference between transactions. Two-phase locking works by acquiring locks before reads/writes and releasing them after transaction completion. Timestamp ordering assigns timestamps to transactions and ensures serializability. Optimistic concurrency control allows transactions to run without locking by checking for conflicts at commit time and aborting transactions if conflicts exist. Concurrency control aims to maximize throughput while maintaining consistency across distributed transactions.
The Google Chubby lock service for loosely-coupled distributed systemsRomain Jacotin
The Google Chubby lock service presented in 2006 is the inspiration for Apache ZooKeeper: let's take a deep dive into Chubby to better understand ZooKeeper and distributed consensus.
Linux is a family of open-source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991, by Linus Torvalds. Linux is typically packaged in a Linux distribution.
in this presentation, we will show you a short description of the Linux/Unix.
I hope you guys enjoy it.
----------------------------------------------------------------------------------------------------
Who Saurabh Upadhyay?
Experienced Technical Support Engineer with a demonstrated history of 2 years working in the Technical field.
Have hands-on experience in System Support, Remote Support, Network Support. Strong engineering professional with a Bachelor of Technology (B.Tech.) majoring in Computer Science and Engineering from Dr.APJ Abdul Kalam Technical University.
This document provides an overview of the Linux architecture. It describes Linux as a free UNIX-like kernel that forms the GNU/Linux operating system together with the GNU project software. The kernel acts as an intermediary between hardware and software, managing processes, scheduling, and resources. Processes can be in different states like running, waiting, or sleeping. The kernel uses preemptive multitasking to switch between processes and prioritize interrupts. System calls allow processes to be created and managed.
The document provides an outline on operating systems memory storage and management. It discusses how the OS must manage memory to ensure each process has enough space to execute without interfering with other processes. It describes different types of memory like cache, RAM, and disk and how the OS uses these properly. Specific topics covered include cache memory, RAM, virtual memory using swap files, the kernel, kernel types, shells, types of shells, shell scripting, and the four freedoms of open source software.
Linux kernel development_ch9-10_20120410huangachou
Kernel synchronization methods are required to prevent race conditions when shared resources are accessed concurrently by multiple threads. The Linux kernel supports various synchronization primitives including atomic operations, spin locks, semaphores, mutexes, completion variables, and the big kernel lock. Each method has advantages and limitations depending on whether the resource can be accessed in interrupt context or while sleeping. Locking schemes must prevent deadlocks and ensure scalability to work effectively in multi-processor and preemptible kernel environments.
Kernel synchronization methods are required to prevent race conditions when shared resources are accessed concurrently by multiple threads. The Linux kernel supports various synchronization primitives including atomic operations, spin locks, semaphores, mutexes, completion variables, and the big kernel lock. Each method has advantages and limitations depending on whether the resource can be accessed in interrupt context or while sleeping. More complex schemes like reader-writer locks and sequential locks also provide alternatives to binary locks.
Lustre is an open-source, object-based file system designed for large clusters providing petabytes of storage and high throughput. It features object protocols, intent-based locking, and adaptive locking policies for concurrency along with aggressive caching.
NFSv4 was motivated by issues with prior versions like lack of guarantees on caches, failure semantics, and data coherency. It uses a stateful protocol with compound operations, lease-based locking, delegation to clients, and close-open consistency to provide distributed transparent access across heterogeneous networks.
Security was improved using GSS-API and requiring implementations support Kerberos v5 and LIPKey authentication.
chapter4-processes nd processors in DS.pptaakarshsiwani1
This document discusses processes, threads, and distributed systems. It covers:
- The differences between processes and threads, with threads sharing an address space.
- Models for organizing threads, including dispatcher-worker, team, and pipeline models.
- Issues in designing thread packages, such as thread management, scheduling, and synchronization using mutexes and condition variables.
- Implementing thread packages at the user level, where the runtime system handles scheduling, and at the kernel level, where the OS schedules threads.
Ibm spectrum scale fundamentals workshop for americas part 4 Replication, Str...xKinAnx
The document provides an overview of IBM Spectrum Scale Active File Management (AFM). AFM allows data to be accessed globally across multiple clusters as if it were local by automatically managing asynchronous replication. It describes the various AFM modes including read-only caching, single-writer, and independent writer. It also covers topics like pre-fetching data, cache eviction, cache states, expiration of stale data, and the types of data transferred between home and cache sites.
The document discusses basic operating system concepts including resource management, abstraction, and virtualization as main goals of an OS. It describes system calls as entry points for users to request OS services, and some common UNIX system calls. It also reviews key OS concepts like processes, threads, scheduling, synchronization, and memory management.
The document provides information about various components of an operating system including:
- The kernel acts as an interface between hardware and software, allocating resources and managing tasks.
- Operating systems support single/multi-user and single/multi-tasking capabilities.
- Linux is an open source, multi-user operating system based on the Unix kernel that is used widely today.
This document summarizes key aspects of distributed file systems (DFS), including their structure, naming and transparency, remote file access using caching, stateful versus stateless service models, file replication, and examples like the Sun Network File System (NFS). A DFS manages dispersed storage across a network, using caching to improve performance of remote file access and dealing with issues of consistency between cached and server copies. NFS provides a specific implementation of a DFS that integrates remote directories transparently and uses stateless remote procedure calls along with caching for efficiency.
Taking AI to the Next Level in Manufacturing.pdfssuserfac0301
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Digital Marketing Trends in 2024 | Guide for Staying AheadWask
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
Main news related to the CCS TSI 2023 (2023/1695)Jakub Marek
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Building Production Ready Search Pipelines with Spark and MilvusZilliz
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Programming Foundation Models with DSPy - Meetup SlidesZilliz
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfChart Kalyan
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Webinar: Designing a schema for a Data WarehouseFederico Razzoli
Are you new to data warehouses (DWH)? Do you need to check whether your data warehouse follows the best practices for a good design? In both cases, this webinar is for you.
A data warehouse is a central relational database that contains all measurements about a business or an organisation. This data comes from a variety of heterogeneous data sources, which includes databases of any type that back the applications used by the company, data files exported by some applications, or APIs provided by internal or external services.
But designing a data warehouse correctly is a hard task, which requires gathering information about the business processes that need to be analysed in the first place. These processes must be translated into so-called star schemas, which means, denormalised databases where each table represents a dimension or facts.
We will discuss these topics:
- How to gather information about a business;
- Understanding dictionaries and how to identify business entities;
- Dimensions and facts;
- Setting a table granularity;
- Types of facts;
- Types of dimensions;
- Snowflakes and how to avoid them;
- Expanding existing dimensions and facts.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Speck&Tech
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
GraphRAG for Life Science to increase LLM accuracyTomaz Bratanic
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providersakankshawande
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
3. Synchronization
• Is an issue only if files are shared
• Sharing in a distributed system is often
necessary, and at the same time can
affect performance in various ways.
• In the following discussion we assume file
sharing takes place in the absence of
process-implemented synchronization
operations such as mutual exclusion.
4. UNIX File Semantics
• In a single-processor (or SMP) system, any
file read operation returns the result of the
most recent write operation.
• Even if two writes occur very close
together, the next read returns the result of
the last write.
• It is as if all reads and writes are time-
stamped from the same clock. Operation
order is based on strict time ordering.
5. UNIX Semantics in DFS
• Possible to (almost) achieve IF…
– There is only one server
– There is NO caching at the client
• In this case every read and write goes directly to
the server, which processes them in sequential
order.
– Even so, the order of simultaneous writes depends on
network transmission times.
• Any file read operation returns the result of the
most recent write operation as seen by the
server.
6. Caching and UNIX Semantics
• Single-server + no client caching leads to
poor performance, so most file systems
allow users to make local copies of files
(or file blocks) that are currently in use.
• Now UNIX semantics are problematic: a
write executed on a local copy only will not
be seen by another client that reads the
file from the server, or from other clients
that have the file cached.
7. Write-Through
• A partial solution is to require all changes
to local copies to be immediately written to
the server. Now a new user will see all the
changes.
– Still inefficient – caching is no longer as useful
– Not a total solution: what happens when two
users have the same file cached?
8. Consistency Models
• Recall discussion of consistency models in
Chapter 7
• Realistically, strict consistency can’t be
achieved without synchronization
techniques such as transactions or locks
• Here we consider what the file system can
do in the absence of user-enabled
methods.
9. Session Semantics
• Instead of trying to implement UNIX
semantics where it is not possible, define
a new semantic:
– Local changes to a file are not made
permanent until the file is closed. In the
meantime, if another user opens the file, she
gets the original version.
– This approach is common in DFS’s.
10. Simultaneous Caching
• What if two users concurrently cache and
modify the same file? How do we
determine the “new” state of the file?
• Most likely option:
– The most recently closed file becomes the
new “official” version
11. File foo
X = 10
Open file Write x = x+6
foo Close foo
to foo
Process 1
Process 2
Open Write x=4*x Close foo
foo to foo
Using session semantics, what is the final value of x in file foo after both
processes close the file?
Assume the horizontal lines represent real time & close operations are
seen at the server in the order shown
12. Immutable Files
• The only operations on a file are,
effectively, create, read, and replace.
– Once a file is created it can be read but not
changed.
– A new file (incorporating changes to a current
file) can be created and placed in the
directory instead of the original version.
• If several users try to replace an existing
file at the same time, one is chosen: either
the last to close, or non-deterministically.
13. Review: File System Semantics
• UNIX semantics • Every file operation is instantly
visible to all processes
• Session semantics • No changes are visible until
the file is closed.
• Immutable files • No updates are possible; files
can only be replaced
14. Transaction Semantics
• Transactions are a way of grouping
several file operations together and
ensuring that they are either all executed
or none is executed.
– We say they are atomic.
• The transaction system is responsible for
ensuring that all of the operations are
carried out in order, without any
interference from concurrent transactions.
15. The Transaction Model
• Transaction: a set of operations which
must be executed entirely, or not at all.
• Processes in a transaction can fail at
random
– Failure causes: hardware or software
problems, network problems, lost messages,
etc.
• Transactions will either commit or abort:
– Commit => successful completion (All)
– Abort => partial results are undone (Nothing)
16. Transaction Model
• Transactions are delimited by two special
primitives:
Begin_transaction // or something similar
transaction operations
(read, write, open, close, etc.)
End_transaction
• If the transaction successfully reaches the end
statement, it “commits” and all changes become
permanent; otherwise it aborts.
17. ACID Properties of Transactions
• Atomic: either all or none of the operations in a
transaction are performed
• Consistent: the transaction doesn’t affect system
invariants; e.g., no money “lost” in a banking
system
• Isolated (serializable): one transaction can’t
affect others until it completes
• Durability: changes made by a committed
transaction are permanent, even if the process
or server fails.
18. Atomicity
• An atomic action is one that appears to be
“indivisible and instantaneous” to the rest
of the system. For example, machine
language instructions.
• Transactions support the execution of
multiple instructions as if they were a
single atomic instruction.
19. Consistent
• A state is consistent if invariants hold
• An invariant is a predicate which states
a condition that must be true.
• Invariants for the airline ticket example:
– seatsLeft = seatsTotal – seatsSold
– seatsLeft >=0
• In the bank case (simplified)
– balancefinal = balanceoriginal – withdrawals +
deposits
20. Isolated
• No other transaction will see the
intermediate results of a transaction.
• Concurrent transactions have the same
effect on the database as if they had run
serially. Notice the similarity to critical
sections, which do run serially.
• This characteristic is enforced through
special concurrency control measures.
21. AD Properties
• ACID is a commonly used term, but
somewhat redundant.
• Transactions that execute atomically will
be consistent and isolated.
• Atomicity and durability capture the
essential qualities.
22. Semantics of File Sharing in
Distributed Systems
• UNIX semantics • Every file operation is instantly
visible to all processes
• Session semantics • No changes are visible until
the file is closed.
• Immutable files • No updates are possible; files
can only be replaced
• Transactions • All changes occur and are
visible atomically – or not at all
23. File Locking
• UNIX file semantics are not possible in
DFS
• Session semantics and immutable files do
not always support the kind of sharing
processes need.
• Transactions have a heavy overhead.
• Thus some additional form of
synchronization is desirable to enable the
server to enforce mutual exclusion on
writes.
24. Locking in NFS
• Early versions of NFS (through V3) were
stateless, and so could not implement
locks.
• An add-on, NLM (Network Lock Manager)
worked in the NFS environment to enforce
advisory locking
– If one process has locked a byte sequence,
any other process requesting a lock on that
sequence will be denied.
25. File Locking in NFSv4
• NFSv4 added a similar locking discipline to the
basic protocols.
• Lock managers in NSF, as in other file systems,
are based on the centralized scheme discussed
in Chapter 6
– Client requests lock
– Lock manager grants lock (if it is free)
– Client releases lock (or it expires after a time)
• In NSF, if a client requests a lock which cannot
be granted, the client is not blocked – must try
again later.
26. Denied Requests
• If a client’s request for a lock is denied, it
receives an error message.
– Poll the server later for lock availability
• Clients can request to be put on a FIFO queue;
when a lock is released it is reserved for the first
process on the queue; if that process polls within
a certain amount of time it gets the lock.
– How is this different from the centralized mutual
exclusion algorithm in the textbook?
27. File locking in NFS
• Two types of locks:
– Reader locks, which can be held
simultaneously,
– Writer locks, which guarantee exclusive
access.
• The lock operation is applied to
consecutive byte sequences in the file,
rather than to the whole file.
28. NFSv4 Lock Related Operations
Operation Description
• Lock • Create a lock for a range of
bytes
• Lockt • Test whether a conflicting lock
has been granted
• Locku • Remove a lock from a range of
bytes
• Renew • Renew the lease on a lock
29. Leases
• Locks are granted for a specific time
interval.
– What problem does this address?
• At the end of that interval the lock is
removed unless the client has requested
an extension.
30. Share Reservations in NFS
• An open request specifies the kind of
access the application requires: READ,
WRITE, BOTH
• It also specifies the kind of access that
should be denied other clients: NONE,
READ, WRITE, BOTH
• If requirements can’t be met, open fails
• Share reservations = implicit locking
• Used in NFS for Windows-based systems
31. Share Reservations - Example
• Client tries to open a file for reading and
writing, and deny concurrent write access.
– If no other client has the file open, the request
succeeds.
– If another client has opened the file for reading
(and hasn’t blocked write access), the request
succeeds
– If another client has opened the file for writing,
the request fails.
– If another client has the file open and has
denied read or write access, the request fails.
32. 11.5 Summary
• UNIX semantics not possible in DFS
• Session semantics is a common choice
• File locking (usually advisory) can provide
additional protection if needed.
• In parallel programs, mutual exclusion
techniques can be used to protect file
operations.
• In database systems, transactions are
used.
33. 11.6: Consistency and Replication
• Client-Side Caching
• Server-Side Replication
• Replication in P2P Systems
34. Introduction
• Replication (and caching) => multiple copies
of something
• Two reasons for replication:
– Reliability (protection against failure, corruption)
– Performance (size of user base, geographical
extent of system)
• Replication can cause inconsistency: at least
one copy is different from the rest.
35. Caching in a DFS
• Caching in any DFS reduces access
delays due to disk access times or
network latency.
• Caches can be located in the main
memory of either the server or client
and/or in the disk of the client
36. Caching in a DFS
• Client-side caching (memory or disk)
offers most performance benefits, but also
leads to potential inconsistencies.
• However, because in practice file sharing
is relatively rare, client-side caching
remains a popular way to improve
performance in a DFS.
37. Cache Consistency Measures
• Server-initiated consistency: server
notifies client if its data becomes stale
– e.g., another client closes its copy of the file,
which was opened for writing.
• Client-initiated consistency: client is
responsible for consistency of data
– e.g., client side software can periodically
check with server to see if file has been
modified.
38. Caching in NFS
• NFSv3 did not define a caching protocol.
– Individual implementations made decisions
• “Stale” data could exist for periods ranging
from a few seconds to ½ minute
• NFSv4 made some improvements but
many details are still implementation
dependent.
• General structure of NFS cache model
follows
39. Client Side Caching in NFS
Figure 11-21.
Memory NFS
Client
Cache applica- server
tion
Disk
cache
Network
40. What Do Clients Cache?
• File data blocks
• File handles – for future reference
• Directories
• Two approaches to caching in NFS
– Caching with server control
– Caching with open delegation
41. Caching Data with Server Control
• The simplest approach to caching allows the
server to retain control over the file.
• Procedure
– Client opens file
– Data blocks are transferred to the client (by read ops)
– Client can read and write data in the cache.
– When the file closes, flush changes back to server
• Session semantics & NFS: the last (most recent)
process to close a file has its changes become
permanent. Changes made by processes that run
concurrently but close earlier are lost.
42. Caching with Server Control
• In caching with server control
– All clients on a single machine may read and write the
same cached data if they have access rights
– data remaining in the cache after a file closes doesn’t
need to be removed, although changes must be sent
to server.
• If a new client on the same machine opens a file
after it has been closed, the client cache
manager usually must validate local cached data
with the server
– If the data is stale, replace it.
43. Caching With Open Delegation
• Allows a client machine to handle some
local open and close operations from
other clients on the same machine.
– Normally the server decides if a client can
open a file
• Delegation can improve performance by
limiting contact with the server
• The client machine gets a copy of the
entire file, not just certain blocks.
44. Open delegation – Examples*
• Suppose a client machine has opened a
file for writing, and has been delegated
rights to control the file locally.
– If another local client tries to lock the file, the
local machine can decide whether or not to
grant the lock
– If a remote client tries to lock the file (at the
server) the server will deny file access
• If a client has opened the file for reading,
only, local clients desiring write privileges
must still contact the server.
45. Delegation and Callbacks
• Server may need to “undelegate” the file –
perhaps when another client needs to
obtain access.
• This can be done with a callback, which is
essentially an RPC from server to client.
• Callbacks require the server to maintain
state (knowledge) about clients – a reason
for NFS to be stateful.
46. Caching Attributes*
• Clients can cache attributes as well as data.
– (size of file, number of links, last date modified, etc.)
• Cached attributes are kept consistent by the
client, if at all
– No guarantee that the same file cached at two sites
will have the same attributes at both sites
• Attribute modifications should be written through
to the server (write through cache coherence
policy), although there’s no requirement to do so
47. Leases*
• Lease: cached data is automatically
invalidated after a certain period of time.
– Applies to file attributes, file handles (mapping
of name to file handle), directories, and
sometimes data.
– When lease expires, must renew data from
server
– Helps with consistency and protects against
errors.
48. An Implementation of Leases*
• Data blocks have time-stamps applied by the
server that indicate when they were last
modified.
• When a block is cached at a client, the server’s
time-stamp is also cached.
• After a period of time, the client confirms the
validity of the data
– Compare timestamp at the client to timestamp at
server
– If server timestamp is more recent, invalidate client
data
49. Coda
A Prototype Distributed File System
• Developed at CMU – M. Satarayanan
• Started in 1987 as an improvement on the
Andrew file system ( a classic research
FS)
– Andrew strongly influenced NFSv4 and some
versions of Linux
• Most recent version of Coda (6.9.4) was
released 1/05/2009 (
http://www.coda.cs.cmu.edu/news.html )
50. Objectives of Coda
• Support disconnected operation (server
goes down, laptop is disconnected from
network, etc.)
• Client side caching is extensive
– Uses client disk cache
• Replication contributes to availability, fault
tolerance, scalability
51. Caching in Coda
• Critical, because of Coda’s objectives
• Caching achieves scalability; provides
more fault tolerance for the client in case it
is disconnected from the server.
• When a client opens a file, the entire file is
downloaded. This is true for reads and
writes.
52. Concurrent Access
• In Coda, many clients may have a file
open for reading, but only one for writing.
– Multiple readers and single writer may exist
concurrently
– In NFS and most other file systems, multiple
readers and multiple writers can exist
concurrently unless locks are used to prohibit
sharing.
53. Callbacks/Server Initiated Cache
Consistency
• A Coda callback is an agreement between
the server and a client. Server agrees to
notify client when a file has been modified by
another client, closed, and written back to
server.
• At this time, the client may purge the file from
its cache, but it may also continue reading the
outdated copy.
• This is a blend of session and transaction
semantics.
54. Coda Callbacks
• Callback promise: server’s commitment to
notify client when file changes
• Callback break: notice from server that the
client’s file is stale; called a “break”
because it terminates the agreement.
There will be no further callbacks unless
the client renews it.
55. Figure 11-23, page 523
• Local copies of files can be used as long
as the client still has an outstanding
callback promise
– No other client has closed a modified file.
56. client 1 cache
server
client 2 cache
Suppose clients 1 & 2 have cached the same file.
Client 1 modifies the file
How/when does client2 know?
What role, if any, does the server have?
Are Coda and NFS different in this respect?
57. 11.6.2: Server-Side Replication
• Caching: replication at the client side.
– Initiated implicitly by client request
– Cached data is temporary
– Unit of caching = a file, or less (usually)
– Purpose: improved performance
• Server replication
– Mainly for fault tolerance & availability
– May actually degrade performance (overhead)
– Replicated data is permanent
58. Caching & Replication in Coda
• Unit of replication = volume (group of
related files)
• Each volume is stored on several servers,
its Volume Storage Group (VSG)
• Available Volume Storage Group (AVSG)
is the set of servers a client can actually
reach
• Contact one server to get permission to R/
W, contact all when closing an updated
file.
59. Server
S1 Server
S3
Server Broken
S2 network
Client Client
A B
Open(f) Open(f)
Figure 11-24. Two clients with a different AVSG for the same file
60. Writing in Disconnected Systems
• Each file has a Coda version vector (CVV),
analogous to vector timestamps, one
component per server. Starts at (1, 1, 1)
• Update local component after a file is
updated.
• As long as all servers get all updates, all
timestamps will be equal
61. Detecting Inconsistencies
• In the previous example, both A and B will
be allowed to open a file for writing.
• When A closes, it will update S1 and S2, but
not S3; B will update S3, but not S1, S2.
• The timestamp at S1 and S2 will be [2, 2, 1].
• The timestamp at S3 will be [1, 1, 2].
• It is easy to detect the inconsistency, but
knowing how to resolve them is application
dependent.
62. Replication in P2P Systems
• In P2P systems replication is more
important because
– P2P members are less reliable – may leave
the system or remove files
– Load balance is important since there are no
designated servers
• File usage in P2P is different: most files
are read only, updates consist of adding
new files, so consistency is less of an
issue.
63. Unstructured P2P Systems
(each node knows n neighbors)
• Look-up = search (in structured systems,
lookup is directed by some algorithm)
• Replication speeds up the process
• How to allocate files to nodes (it may not
be possible to force a node to store files)
– Uniformly distribute n copies across network
– Allocate more replicas for popular files
– Users who download files are responsible for
sharing them with others (as in BitTorrent)
64. Structured P2P Systems
• Replication is used primarily for load
balance
• Possible approaches:
– Store a replica at each node in the search
path (concentrates replicas near the prime
copy, but may unbalance some nodes)
– Store replicas at nodes that request a file,
store pointers to it at nodes along the way.
65. 11.7: Fault Tolerance in DFS*
• Review of Fault Tolerance
• Handling Byzantine Failures
• High Availability in P2P systems
66. Basic Concepts - Review
• Distributed systems may experience partial failure
• Build systems to automatically recover from
crashes.
• Continue to operate normally while failures are
being repaired; i.e., be fault tolerant.
• Fault tolerant systems exhibit dependabilty.
– Availability: the system is immediately ready to use
– Reliability: the system can run continuously without
failing.
• (remember availability/reliability example)
– Safety: system failure doesn’t have disastrous
consequences
– Maintainability: easy to repair
67. Failure Models
• Failure may be due to an error at any place in
the system:
– The server crashes
– The network goes down
– A disk crashes
– Security violations occur
• Crash failure, omission failure, Byzantine
failure:
– Incorrect, but undetectable;
– malicious servers produce deliberately wrong results,
– ...
68. Handling Byzantine Failures in
Distributed File Systems
• Replication handles many errors in DFS
but Byzantine errors are harder to solve.
• Text presents an algorithm by Castro and
Liskov that works as long as no more than
1/3 of the nodes is faulty at any moment.
• Clients must get the same answer from
k+1 servers (in a system with 3k +1) to be
sure the answer is correct.
69. Availability in P2P Systems
• Possible approaches
– Replication (although must be at very high
levels due to unreliability of nodes)
– Erasure coding: divides a file into m
fragments, recodes them into n > m fragments
such that any set of m fragments can be used
to reconstruct the entire file. Distribute
fragments, rather than entire file replicas
• Requires less redundancy than full replication.