The document summarizes the Andrew File System (AFS), a distributed file system developed at Carnegie Mellon University. It describes the initial prototype of AFS implemented in the 1980s, which used client-server architecture with Venus processes on client workstations caching files from Vice servers. The prototype showed performance bottlenecks due to high server CPU usage from frequent cache validation checks and pathname traversals. Subsequent versions improved performance by reducing cache checks, having clients perform name resolution, using a single server process with lightweight threads, and identifying files by unique IDs instead of pathnames.
The document describes HDFS's implementation of file truncation, which allows reducing a file's length. It evolved HDFS's write-once semantics to support data mutation. Truncate uses the lease and block recovery framework to truncate block replicas in-place, except when snapshots exist, where it uses "copy-on-truncate" to preserve the snapshot. The truncate operation returns immediately after updating metadata, while block adjustments occur in the background.
Coordinating Metadata Replication: Survival Strategy for Distributed SystemsKonstantin V. Shvachko
Hadoop Summit, April 2014
Amsterdam, Netherlands
Just as the survival of living species depends on the transfer of essential knowledge within the community and between generations, the availability and reliability of a distributed computer system relies upon consistent replication of core metadata between its components. This presentation will highlight the implementation of a replication technique for the namespace of the Hadoop Distributed File System (HDFS). In HDFS, the namespace represented by the NameNode is decoupled from the data storage layer. While the data layer is conventionally replicated via block replication, the namespace remains a performance and availability bottleneck. Our replication technique relies on quorum-based consensus algorithms and provides an active-active model of high availability for HDFS where metadata requests (reads and writes) can be load-balanced between multiple instances of the NameNode. This session will also cover how the same techniques are extended to provide replication of metadata and data between geographically distributed data centers, providing global disaster recovery and continuous availability. Finally, we will review how consistent replication can be applied to advance other systems in the Apache Hadoop stack; e.g., how in HBase coordinated updates of regions selectively replicated on multiple RegionServers improve availability and overall cluster throughput.
There are different dimensions for scalability of a distributed storage system: more data, more stored objects, more nodes, more load, additional data centers, etc. This presentation addresses the geographic scalability of HDFS. It describes unique techniques implemented at WANdisco, which allow scaling HDFS over multiple geographically distributed data centers for continuous availability. The distinguished principle of our approach is that metadata is replicated synchronously between data centers using a coordination engine, while the data is copied over the WAN asynchronously. This allows strict consistency of the namespace on the one hand and fast LAN-speed data ingestion on the other. In this approach geographically separated parts of the system operate as a single HDFS cluster, where data can be actively accessed and updated from any data center. The presentation also cover advanced features such as selective data replication.
Extended version of presentation at Strata + Hadoop World. November 20, 2014. Barcelona, Spain.
http://strataconf.com/strataeu2014/public/schedule/detail/39174
A Distributed File System(DFS) is simply a classical model of a file system distributed across multiple machines.The purpose is to promote sharing of dispersed files.
DFS allows administrators to consolidate file shares across multiple servers so users can access files from a single location. It provides benefits like centralized resources management, high accessibility regardless of physical location, fault tolerance through replication, and optimized workload distribution. HDFS is the distributed file system of Hadoop. It has a master/slave architecture with a single NameNode managing the file system namespace and DataNodes storing and retrieving blocks. The NameNode maintains metadata and regulates client access while DataNodes store and serve blocks upon instruction from the NameNode, providing streaming data access at large scale for batch processing workloads.
The slides were created for one University Program on Apache Hadoop + Apache Apex workshop.
It explains almost all the hdfs related commands in details along with the examples.
The document provides an overview of distributed file systems, including NFS, AFS, Lustre, and others. It discusses key aspects like scalability, consistency, caching, replication, and fault tolerance. Lustre is highlighted as an example of a distributed file system that aims to remove bottlenecks and achieve high scalability through an object-based design with separate metadata and storage servers.
The document describes HDFS's implementation of file truncation, which allows reducing a file's length. It evolved HDFS's write-once semantics to support data mutation. Truncate uses the lease and block recovery framework to truncate block replicas in-place, except when snapshots exist, where it uses "copy-on-truncate" to preserve the snapshot. The truncate operation returns immediately after updating metadata, while block adjustments occur in the background.
Coordinating Metadata Replication: Survival Strategy for Distributed SystemsKonstantin V. Shvachko
Hadoop Summit, April 2014
Amsterdam, Netherlands
Just as the survival of living species depends on the transfer of essential knowledge within the community and between generations, the availability and reliability of a distributed computer system relies upon consistent replication of core metadata between its components. This presentation will highlight the implementation of a replication technique for the namespace of the Hadoop Distributed File System (HDFS). In HDFS, the namespace represented by the NameNode is decoupled from the data storage layer. While the data layer is conventionally replicated via block replication, the namespace remains a performance and availability bottleneck. Our replication technique relies on quorum-based consensus algorithms and provides an active-active model of high availability for HDFS where metadata requests (reads and writes) can be load-balanced between multiple instances of the NameNode. This session will also cover how the same techniques are extended to provide replication of metadata and data between geographically distributed data centers, providing global disaster recovery and continuous availability. Finally, we will review how consistent replication can be applied to advance other systems in the Apache Hadoop stack; e.g., how in HBase coordinated updates of regions selectively replicated on multiple RegionServers improve availability and overall cluster throughput.
There are different dimensions for scalability of a distributed storage system: more data, more stored objects, more nodes, more load, additional data centers, etc. This presentation addresses the geographic scalability of HDFS. It describes unique techniques implemented at WANdisco, which allow scaling HDFS over multiple geographically distributed data centers for continuous availability. The distinguished principle of our approach is that metadata is replicated synchronously between data centers using a coordination engine, while the data is copied over the WAN asynchronously. This allows strict consistency of the namespace on the one hand and fast LAN-speed data ingestion on the other. In this approach geographically separated parts of the system operate as a single HDFS cluster, where data can be actively accessed and updated from any data center. The presentation also cover advanced features such as selective data replication.
Extended version of presentation at Strata + Hadoop World. November 20, 2014. Barcelona, Spain.
http://strataconf.com/strataeu2014/public/schedule/detail/39174
A Distributed File System(DFS) is simply a classical model of a file system distributed across multiple machines.The purpose is to promote sharing of dispersed files.
DFS allows administrators to consolidate file shares across multiple servers so users can access files from a single location. It provides benefits like centralized resources management, high accessibility regardless of physical location, fault tolerance through replication, and optimized workload distribution. HDFS is the distributed file system of Hadoop. It has a master/slave architecture with a single NameNode managing the file system namespace and DataNodes storing and retrieving blocks. The NameNode maintains metadata and regulates client access while DataNodes store and serve blocks upon instruction from the NameNode, providing streaming data access at large scale for batch processing workloads.
The slides were created for one University Program on Apache Hadoop + Apache Apex workshop.
It explains almost all the hdfs related commands in details along with the examples.
The document provides an overview of distributed file systems, including NFS, AFS, Lustre, and others. It discusses key aspects like scalability, consistency, caching, replication, and fault tolerance. Lustre is highlighted as an example of a distributed file system that aims to remove bottlenecks and achieve high scalability through an object-based design with separate metadata and storage servers.
This document discusses distributed file systems. It describes distributed file systems as implementing a common file system that can be shared across autonomous computers. A client-server model is presented where file servers store files and clients access them. Key services like name servers and caching are described. Design issues around naming, caching, writing policies, availability, scalability, and semantics are also summarized.
Coda (Constant Data Avaialabilty) is a distributed file system developed at Carnegie Mellon University . This presentation explains how it works and different aspects of it.
The document discusses key concepts related to distributed file systems including:
1. Files are accessed using location transparency where the physical location is hidden from users. File names do not reveal storage locations and names do not change when locations change.
2. Remote files can be mounted to local directories, making them appear local while maintaining location independence. Caching is used to reduce network traffic by storing recently accessed data locally.
3. Fault tolerance is improved through techniques like stateless server designs, file replication across failure independent machines, and read-only replication for consistency. Scalability is achieved by adding new nodes and using decentralized control through clustering.
The document discusses several distributed file systems including NFS, AFS, and CODA. NFS uses a stateless design with UDP and allows clients to access remote files transparently like local files. AFS uses whole file caching at clients and call backs to ensure cache coherence. CODA extends AFS to support disconnected operation and user mobility through caching and logging updates locally when disconnected.
Operating System : Ch17 distributed file systemsSyaiful Ahdan
This document discusses distributed file systems (DFS), which allow files to be shared across multiple machines. It covers key topics in DFS including naming and transparency, remote file access using caching, consistency issues, and a comparison of caching versus stateful remote file access approaches. The goal of a DFS is to manage dispersed storage across machines while providing a unified file system interface to hide the distributed nature from clients.
This document provides an overview and summary of key concepts related to distributed file systems. It begins with an introduction to file systems and their properties. It then discusses the Sun Network File System (NFS), including its architecture, operations, optimizations like caching, and how it achieves various requirements like transparency and performance. The document concludes with a brief overview of recent advances in file services beyond NFS.
The Google File System (GFS) was designed by Google to store massive amounts of data across cheap, unreliable hardware. It uses a single master to coordinate metadata and mutations across multiple chunkservers. Files are divided into fixed-size chunks which are replicated for reliability. The design focuses on supporting huge files that are written once and read through streaming, while tolerating high failure rates through replication and relaxed consistency. GFS has proven successful at meeting Google's storage needs at massive scale.
This document discusses distributed file systems (DFS), which allow files to be dispersed across networked machines. A DFS includes clients that access files, servers that store files, and services that provide file access. Key features of DFS include mapping logical file names to physical storage locations, transparency so file locations are hidden, and replication to improve availability and performance. DFS supports either stateful or stateless access, with stateful requiring unique identifiers but enabling features like read-ahead. Namespaces and replication help organize files across multiple servers.
File Replication : High availability is a desirable feature of a good distributed file system and file replication is the primary mechanism for improving file availability. Replication is a key strategy for improving reliability, fault tolerance and availability. Therefore duplicating files on multiple machines improves availability and performance.
Replicated file : A replicated file is a file that has multiple copies, with each copy located on a separate file server. Each copy of the set of copies that comprises a replicated file is referred to as replica of the replicated file.
Replication is often confused with caching, probably because they both deal with multiple copies of data. The two concepts has the following basic differences:
A replica is associated with server, whereas a cached copy is associated with a client.
The existence of cached copy is primarily dependent on the locality in file access patterns, whereas the existence of a replica normally depends on availability and performance requirements.
Satynarayanana [1992] distinguishes a replicated copy from a cached copy by calling the first-class replicas and second-class replicas respectively
This document discusses key aspects of distributed file systems including file caching schemes, file replication, and fault tolerance. It describes different cache locations, modification propagation techniques, and methods for replica creation. File caching schemes aim to reduce network traffic by retaining recently accessed files in memory. File replication provides increased reliability and availability through independent backups. Distributed file systems must also address being stateful or stateless to maintain information about file access and operations.
This document discusses distributed file systems. It begins by defining key terms like filenames, directories, and metadata. It then describes the goals of distributed file systems, including network transparency, availability, and access transparency. The document outlines common distributed file system architectures like client-server and peer-to-peer. It also discusses specific distributed file systems like NFS, focusing on their protocols, caching, replication, and security considerations.
This document outlines and compares two NameNode high availability (HA) solutions for HDFS: AvatarNode used by Facebook and BackupNode used by Yahoo. AvatarNode provides a complete hot standby with fast failover times of seconds by using an active-passive pair and ZooKeeper for coordination. BackupNode has limitations including slower restart times of 25+ minutes and supporting only two-machine failures. While it provides hot standby for the namespace, block reports are sent only to the active NameNode, making it a semi-hot standby solution. The document also briefly mentions other experimental HA solutions for HDFS.
Hadoop is a distributed processing framework for large data sets across clusters of commodity hardware. It has two main components: HDFS for reliable data storage, and MapReduce for distributed processing of large data sets. Hadoop can scale from single servers to thousands of machines, handling data measuring petabytes with very high throughput. It provides reliability even if individual machines fail, and is easy to set up and manage.
The document provides an overview of distributed file systems including NFS, Lustre, AFS, Coda, and Sprite. It discusses key concepts such as naming, caching, consistency, performance, replication, and security. Examples of different distributed file system architectures and consistency models are presented. Key goals and tradeoffs for different distributed file systems are outlined.
Distributed file systems allow files to be shared across multiple computers even without other inter-process communication. There are three main naming schemes for distributed files: 1) mounting remote directories locally, 2) combining host name and local name, and 3) using a single global namespace. File caching schemes aim to reduce network traffic by storing recently accessed files in local memory. Key decisions for caching schemes include cache location (client/server memory or disk) and how/when modifications are propagated to servers.
File service architecture and network file systemSukhman Kaur
Distributed file systems allow users to access and share files located on multiple computer systems. They provide transparency so that clients can access local and remote files in the same way. Issues include maintaining consistent concurrent updates and caching files for improved performance. Network File System (NFS) is an open standard protocol that allows remote file access like a local file system. It uses remote procedure calls and has evolved through several versions to support features like locking, caching, and security.
Distribution File System DFS TechnologiesRaphael Ejike
Distribution File System (DFS) technologies allow administrators to group shared folders located on different servers into logically structured namespaces that appear to users as single shared folders. DFS Namespaces provide this functionality. DFS Replication is used to synchronize files between servers over local and wide area networks, while minimizing bandwidth usage. Domain-based and stand-alone DFS differ in their path structure, location, availability, server type, and whether a domain controller is required.
A simple replication-based mechanism has been used to achieve high data reliability of Hadoop Distributed File System (HDFS). However, replication based mechanisms have high degree of disk storage requirement since it makes copies of full block without consideration of storage size. Studies have shown that erasure-coding mechanism can provide more storage space when used as an alternative to replication. Also, it can increase write throughput compared to replication mechanism. To improve both space efficiency and I/O performance of the HDFS while preserving the same data reliability level, we propose HDFS+, an erasure coding based Hadoop Distributed File System. The proposed scheme writes a full block on the primary DataNode and then performs erasure coding with Vandermonde-based Reed-Solomon algorithm that divides data into m data fragments and encode them into n data fragments (n>m), which are saved in N distinct DataNodes such that the original object can be reconstructed from any m fragments. The experimental results show that our scheme can save up to 33% of storage space while outperforming the original scheme in write performance by 1.4 times. Our scheme provides the same read performance as the original scheme as long as data can be read from the primary DataNode even under single-node or double-node failure. Otherwise, the read performance of the HDFS+ decreases to some extent. However, as the number of fragments increases, we show that the performance degradation becomes negligible.
Hadoop Institutes : kelly technologies is the best Hadoop Training Institutes in Hyderabad. Providing Hadoop training by real time faculty in Hyderabad.
HDFS is a distributed file system designed for large data sets and high throughput access. It uses a master/slave architecture with a Namenode managing the file system namespace and Datanodes storing file data blocks. Blocks are replicated across Datanodes for fault tolerance. The system is highly scalable, handling large clusters and files sizes ranging from gigabytes to terabytes.
This document discusses distributed file systems. It describes distributed file systems as implementing a common file system that can be shared across autonomous computers. A client-server model is presented where file servers store files and clients access them. Key services like name servers and caching are described. Design issues around naming, caching, writing policies, availability, scalability, and semantics are also summarized.
Coda (Constant Data Avaialabilty) is a distributed file system developed at Carnegie Mellon University . This presentation explains how it works and different aspects of it.
The document discusses key concepts related to distributed file systems including:
1. Files are accessed using location transparency where the physical location is hidden from users. File names do not reveal storage locations and names do not change when locations change.
2. Remote files can be mounted to local directories, making them appear local while maintaining location independence. Caching is used to reduce network traffic by storing recently accessed data locally.
3. Fault tolerance is improved through techniques like stateless server designs, file replication across failure independent machines, and read-only replication for consistency. Scalability is achieved by adding new nodes and using decentralized control through clustering.
The document discusses several distributed file systems including NFS, AFS, and CODA. NFS uses a stateless design with UDP and allows clients to access remote files transparently like local files. AFS uses whole file caching at clients and call backs to ensure cache coherence. CODA extends AFS to support disconnected operation and user mobility through caching and logging updates locally when disconnected.
Operating System : Ch17 distributed file systemsSyaiful Ahdan
This document discusses distributed file systems (DFS), which allow files to be shared across multiple machines. It covers key topics in DFS including naming and transparency, remote file access using caching, consistency issues, and a comparison of caching versus stateful remote file access approaches. The goal of a DFS is to manage dispersed storage across machines while providing a unified file system interface to hide the distributed nature from clients.
This document provides an overview and summary of key concepts related to distributed file systems. It begins with an introduction to file systems and their properties. It then discusses the Sun Network File System (NFS), including its architecture, operations, optimizations like caching, and how it achieves various requirements like transparency and performance. The document concludes with a brief overview of recent advances in file services beyond NFS.
The Google File System (GFS) was designed by Google to store massive amounts of data across cheap, unreliable hardware. It uses a single master to coordinate metadata and mutations across multiple chunkservers. Files are divided into fixed-size chunks which are replicated for reliability. The design focuses on supporting huge files that are written once and read through streaming, while tolerating high failure rates through replication and relaxed consistency. GFS has proven successful at meeting Google's storage needs at massive scale.
This document discusses distributed file systems (DFS), which allow files to be dispersed across networked machines. A DFS includes clients that access files, servers that store files, and services that provide file access. Key features of DFS include mapping logical file names to physical storage locations, transparency so file locations are hidden, and replication to improve availability and performance. DFS supports either stateful or stateless access, with stateful requiring unique identifiers but enabling features like read-ahead. Namespaces and replication help organize files across multiple servers.
File Replication : High availability is a desirable feature of a good distributed file system and file replication is the primary mechanism for improving file availability. Replication is a key strategy for improving reliability, fault tolerance and availability. Therefore duplicating files on multiple machines improves availability and performance.
Replicated file : A replicated file is a file that has multiple copies, with each copy located on a separate file server. Each copy of the set of copies that comprises a replicated file is referred to as replica of the replicated file.
Replication is often confused with caching, probably because they both deal with multiple copies of data. The two concepts has the following basic differences:
A replica is associated with server, whereas a cached copy is associated with a client.
The existence of cached copy is primarily dependent on the locality in file access patterns, whereas the existence of a replica normally depends on availability and performance requirements.
Satynarayanana [1992] distinguishes a replicated copy from a cached copy by calling the first-class replicas and second-class replicas respectively
This document discusses key aspects of distributed file systems including file caching schemes, file replication, and fault tolerance. It describes different cache locations, modification propagation techniques, and methods for replica creation. File caching schemes aim to reduce network traffic by retaining recently accessed files in memory. File replication provides increased reliability and availability through independent backups. Distributed file systems must also address being stateful or stateless to maintain information about file access and operations.
This document discusses distributed file systems. It begins by defining key terms like filenames, directories, and metadata. It then describes the goals of distributed file systems, including network transparency, availability, and access transparency. The document outlines common distributed file system architectures like client-server and peer-to-peer. It also discusses specific distributed file systems like NFS, focusing on their protocols, caching, replication, and security considerations.
This document outlines and compares two NameNode high availability (HA) solutions for HDFS: AvatarNode used by Facebook and BackupNode used by Yahoo. AvatarNode provides a complete hot standby with fast failover times of seconds by using an active-passive pair and ZooKeeper for coordination. BackupNode has limitations including slower restart times of 25+ minutes and supporting only two-machine failures. While it provides hot standby for the namespace, block reports are sent only to the active NameNode, making it a semi-hot standby solution. The document also briefly mentions other experimental HA solutions for HDFS.
Hadoop is a distributed processing framework for large data sets across clusters of commodity hardware. It has two main components: HDFS for reliable data storage, and MapReduce for distributed processing of large data sets. Hadoop can scale from single servers to thousands of machines, handling data measuring petabytes with very high throughput. It provides reliability even if individual machines fail, and is easy to set up and manage.
The document provides an overview of distributed file systems including NFS, Lustre, AFS, Coda, and Sprite. It discusses key concepts such as naming, caching, consistency, performance, replication, and security. Examples of different distributed file system architectures and consistency models are presented. Key goals and tradeoffs for different distributed file systems are outlined.
Distributed file systems allow files to be shared across multiple computers even without other inter-process communication. There are three main naming schemes for distributed files: 1) mounting remote directories locally, 2) combining host name and local name, and 3) using a single global namespace. File caching schemes aim to reduce network traffic by storing recently accessed files in local memory. Key decisions for caching schemes include cache location (client/server memory or disk) and how/when modifications are propagated to servers.
File service architecture and network file systemSukhman Kaur
Distributed file systems allow users to access and share files located on multiple computer systems. They provide transparency so that clients can access local and remote files in the same way. Issues include maintaining consistent concurrent updates and caching files for improved performance. Network File System (NFS) is an open standard protocol that allows remote file access like a local file system. It uses remote procedure calls and has evolved through several versions to support features like locking, caching, and security.
Distribution File System DFS TechnologiesRaphael Ejike
Distribution File System (DFS) technologies allow administrators to group shared folders located on different servers into logically structured namespaces that appear to users as single shared folders. DFS Namespaces provide this functionality. DFS Replication is used to synchronize files between servers over local and wide area networks, while minimizing bandwidth usage. Domain-based and stand-alone DFS differ in their path structure, location, availability, server type, and whether a domain controller is required.
A simple replication-based mechanism has been used to achieve high data reliability of Hadoop Distributed File System (HDFS). However, replication based mechanisms have high degree of disk storage requirement since it makes copies of full block without consideration of storage size. Studies have shown that erasure-coding mechanism can provide more storage space when used as an alternative to replication. Also, it can increase write throughput compared to replication mechanism. To improve both space efficiency and I/O performance of the HDFS while preserving the same data reliability level, we propose HDFS+, an erasure coding based Hadoop Distributed File System. The proposed scheme writes a full block on the primary DataNode and then performs erasure coding with Vandermonde-based Reed-Solomon algorithm that divides data into m data fragments and encode them into n data fragments (n>m), which are saved in N distinct DataNodes such that the original object can be reconstructed from any m fragments. The experimental results show that our scheme can save up to 33% of storage space while outperforming the original scheme in write performance by 1.4 times. Our scheme provides the same read performance as the original scheme as long as data can be read from the primary DataNode even under single-node or double-node failure. Otherwise, the read performance of the HDFS+ decreases to some extent. However, as the number of fragments increases, we show that the performance degradation becomes negligible.
Hadoop Institutes : kelly technologies is the best Hadoop Training Institutes in Hyderabad. Providing Hadoop training by real time faculty in Hyderabad.
HDFS is a distributed file system designed for large data sets and high throughput access. It uses a master/slave architecture with a Namenode managing the file system namespace and Datanodes storing file data blocks. Blocks are replicated across Datanodes for fault tolerance. The system is highly scalable, handling large clusters and files sizes ranging from gigabytes to terabytes.
The document describes the key features and architecture of the Hadoop Distributed File System (HDFS). HDFS is designed to reliably store very large data sets across clusters of commodity hardware. It uses a master/slave architecture with a single NameNode that manages file system metadata and regulates client access. Numerous DataNodes store file system blocks and replicate data for fault tolerance. The document outlines HDFS properties like high fault tolerance, data replication, and APIs.
This document summarizes key aspects of the Hadoop Distributed File System (HDFS). HDFS is designed for storing very large files across commodity hardware. It uses a master/slave architecture with a single NameNode that manages file system metadata and multiple DataNodes that store application data. HDFS allows for streaming access to this distributed data and can provide higher throughput than a single high-end server by parallelizing reads across nodes.
The Hadoop Distributed File System (HDFS) has a master/slave architecture with a single NameNode that manages the file system namespace and regulates client access, and multiple DataNodes that store and retrieve blocks of data files. The NameNode maintains metadata and a map of blocks to files, while DataNodes store blocks and report their locations. Blocks are replicated across DataNodes for fault tolerance following a configurable replication factor. The system uses rack awareness and preferential selection of local replicas to optimize performance and bandwidth utilization.
Redis. Seattle Data Science and Data Engineering MeetupAbhishek Goswami
Redis has become one of the critical tools in a Data Engineers toolkit. In this meetup we will take a gentle introduction to Redis, and also discuss some internals and usage patterns.
Hadoop Distributed File System (HDFS) is modeled after Google File System and optimized for large data sets and high throughput. HDFS uses a master/slave architecture with a NameNode that manages file system metadata and DataNodes that store data blocks. The NameNode tracks locations of data blocks and the DataNodes provide streaming access to blocks for batch processing applications. HDFS replicates data blocks across multiple DataNodes for fault tolerance.
Hadoop-professional-software-development-course-in-mumbaiUnmesh Baile
The document provides an overview of the Hadoop Distributed File System (HDFS). It describes HDFS's key features like fault tolerance, high throughput, and support for large datasets. The core architecture uses a master/slave model with a Namenode managing metadata and Datanodes storing file data blocks. HDFS employs replication for reliability and data is placed across racks for high availability. The document also discusses HDFS protocols, APIs, and how it ensures data integrity and handles failures.
Hadoop professional-software-development-course-in-mumbaiUnmesh Baile
Vibrant Technologies is headquarted in Mumbai,India.We are the best Hadoop training provider in Navi Mumbai who provides Live Projects to students.We provide Corporate Training also.We are Best Hadoop classes in Mumbai according to our students and corporates
The document discusses the key features and architecture of the Hadoop File System (HDFS). HDFS is designed for large data sets and high fault tolerance. It uses a master/slave architecture with one namenode that manages file metadata and multiple datanodes that store file data blocks. HDFS replicates blocks across datanodes for reliability and provides interfaces for applications to access file data.
Distributed systems allow for parallel and distributed computing across multiple autonomous computers connected through a network. There are different types of distributed systems such as high performance computing systems, distributed information systems, and distributed systems for pervasive services. Processes and threads are important concepts in distributed systems - a process is the instance of a program in execution and has state information stored in a process control block, while threads are lightweight processes that execute concurrently within a process and share the same memory address space. Remote procedure calls allow processes on different machines to communicate and invoke procedures remotely through a client-server model.
Big data refers to large and complex datasets that are difficult to process using traditional methods. Key challenges include capturing, storing, searching, sharing, and analyzing large datasets in domains like meteorology, physics simulations, biology, and the internet. Hadoop is an open-source software framework for distributed storage and processing of big data across clusters of computers. It allows for the distributed processing of large data sets in a reliable, fault-tolerant and scalable manner.
Distributed file systems allow files to be accessed across a network in a transparent manner. They achieve high availability by allowing users easy access to files irrespective of physical location. A DFS uses name servers to map file names to storage locations and cache managers to copy remote files to clients for faster access. Key components include caching, replication for availability, and consistency protocols to ensure clients see up-to-date file data. Design goals involve scalability to large numbers of clients and servers as well as maintaining strong file semantics even in distributed systems.
The document discusses several topics related to distributed operating systems including:
- Distributed shared memory, which implements shared memory across distributed systems without physical shared memory.
- Central server and migration algorithms for managing shared data in distributed shared memory systems.
- Read replication and full replication algorithms that allow multiple nodes to read or write shared data.
- Memory coherence and coherence protocols for maintaining consistency across processor caches.
- Key components of distributed file systems such as naming, caching, writing policies, availability, scalability, and cache consistency.
This chapter discusses configuring major network services on Linux servers including dynamic routing protocols, DNS servers, email servers, and web servers. It covers setting up routing protocols like RIP and OSPF, configuring the BIND DNS server software, setting up a sendmail email server, and using the Apache web server on Linux.
The document discusses distributed file systems. It defines a distributed file system as a classical model of a file system distributed across multiple machines to promote sharing of dispersed files. Key aspects discussed include:
- Files are accessed using the same operations (create, read, etc.) regardless of physical location.
- Systems aim to make file locations transparent to clients through techniques like replication and unique file identifiers.
- Caching is used to improve performance by retaining recently accessed data locally to reduce remote access.
- Consistency must be maintained when copies are updated.
The document provides an introduction to the key concepts of Big Data including Hadoop, HDFS, and MapReduce. It defines big data as large volumes of data that are difficult to process using traditional methods. Hadoop is introduced as an open-source framework for distributed storage and processing of large datasets across clusters of computers. HDFS is described as Hadoop's distributed file system that stores data across clusters and replicates files for reliability. MapReduce is a programming model where data is processed in parallel across clusters using mapping and reducing functions.
This document provides an overview of distributed file systems (DFS), including their structure, naming and transparency, remote file access, caching, and example systems. Some key points:
- A DFS manages dispersed storage devices across a network to provide shared files and storage. It provides location transparency so the physical location of files is hidden from users.
- Remote file access in DFS is enabled through caching, where frequently accessed data is cached locally to reduce network traffic. This raises cache consistency issues in keeping cached copies up-to-date.
- The Andrew distributed computing environment example presented a DFS with a shared name space spanning over 5,000 workstations using whole-file caching for remote access.
This document discusses key reinstallation attacks against the WPA2 protocol. It describes how attackers can force nonce reuse during the 4-way handshake by manipulating messages, allowing them to decrypt encrypted data frames. This enables attacks such as stealing sensitive user information. The document outlines different techniques for different client implementations, including Android and macOS. It also discusses attacks against the group key handshake and 802.11r fast BSS transition handshake. The authors are researchers who discovered these vulnerabilities in the combination of encryption protocols and handshake procedures in WPA2.
CPU scheduling algorithms aim to share CPU resources fairly and efficiently among processes. Lottery scheduling is an algorithm that assigns each process a number of "tickets" proportional to its desired share of CPU time. The scheduler then randomly selects a ticket to determine the next process to run, simulating a lottery. This provides probabilistic fairness by ensuring each process's chance of running is equal to its proportion of tickets over time. Lottery scheduling has advantages over traditional priority-based and fair share algorithms in providing both long-term fairness and preventing indefinite starvation.
IEEE 802.11i is a security standard that defines authentication and encryption for wireless networks. It introduced stronger authentication methods like 802.1X and improved encryption protocols like TKIP and CCMP to replace the weaker WEP encryption. The standard defines a multi-phase process for wireless security that includes network discovery, security capabilities negotiation, authentication, and key management. Robust authentication is important for wireless security as it establishes a station's identity before allowing access to network resources.
Fueling AI with Great Data with Airbyte WebinarZilliz
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Driving Business Innovation: Latest Generative AI Advancements & Success StorySafe Software
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfMalak Abu Hammad
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
“An Outlook of the Ongoing and Future Relationship between Blockchain Technologies and Process-aware Information Systems.” Invited talk at the joint workshop on Blockchain for Information Systems (BC4IS) and Blockchain for Trusted Data Sharing (B4TDS), co-located with with the 36th International Conference on Advanced Information Systems Engineering (CAiSE), 3 June 2024, Limassol, Cyprus.
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
OpenID AuthZEN Interop Read Out - AuthorizationDavid Brossard
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
CAKE: Sharing Slices of Confidential Data on BlockchainClaudio Di Ciccio
Presented at the CAiSE 2024 Forum, Intelligent Information Systems, June 6th, Limassol, Cyprus.
Synopsis: Cooperative information systems typically involve various entities in a collaborative process within a distributed environment. Blockchain technology offers a mechanism for automating such processes, even when only partial trust exists among participants. The data stored on the blockchain is replicated across all nodes in the network, ensuring accessibility to all participants. While this aspect facilitates traceability, integrity, and persistence, it poses challenges for adopting public blockchains in enterprise settings due to confidentiality issues. In this paper, we present a software tool named Control Access via Key Encryption (CAKE), designed to ensure data confidentiality in scenarios involving public blockchains. After outlining its core components and functionalities, we showcase the application of CAKE in the context of a real-world cyber-security project within the logistics domain.
Paper: https://doi.org/10.1007/978-3-031-61000-4_16
Infrastructure Challenges in Scaling RAG with Custom AI modelsZilliz
Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceIndexBug
Imagine a world where machines not only perform tasks but also learn, adapt, and make decisions. This is the promise of Artificial Intelligence (AI), a technology that's not just enhancing our lives but revolutionizing entire industries.
3. Distributed File System
◎File System (FS) controls how data is stored
and retrieved
◎A Distributed File System (DFS) is a set of
client and server services that allow access
multiple hosts sharing information via
network
3
4. Andrew File System
◎Andrew is a distributed computing
environment that has been under development
at Carnegie Mellon University since 1983
◎Each individual at CMU may eventually possess
an Andrew workstation
◎A fundamental component of Andrew is the
distributed file system
4
5. Andrew File System
◎It’s the first wide DFS for large number of
users
◎Andrew File System presents a homogeneous,
location-transparent file name space to all
the client workstations
5
6. Andrew File System
◎The goal of Andrew File System is scalability
◎Large scale affects a distributed system in two
ways
○ it degrades performance
○ it complicates administration and day-to-day
operation
◎Andrew copes successfully with these
concerns
6
7. 1° Prototype
◎Validate the basic file system architecture
○ obtain feedback on our design as rapidly as possible
○ build a system that was usable enough to make that
feedback meaningful
◎Prototype used by 400 users
◎100 workstations Sun2 with 65MB local disks
◎6 servers Sun2s or Vax-750s each with 2-3 disk
of 400MB
◎Client-Server nodes run 4.2BSD of UNIX
7
8. 1° Prototype
◎VICE: a set of trusted servers, it defines a
homogeneous, location-transparent file name
space to all the client
◎VENUS: client process, it caches files from
Vice and stores modified copies of files back
on the servers they came from
8
10. ● Venus would rendezvous with a process listening at a
well-known network address on a server
● This process then created a dedicated process to deal
with all future requests from the client
● All communication and manipulation of data structures
between server processes took place via files in the
underlying file system
10
11. ● Each server contained a directory hierarchy mirroring
the structure of the Vice files stored on it
● The directory hierarchy contained Stub directories that
represented portions of the Vice name space located on
other servers. If a file were not on a server, the search
for its name would end in a stub directory which
identified the server containing that file
11
12. The Benchmark
◎This benchmark consists of a command script
that operates on a collection of files
constituting an application program
◎Load Unit: load placed on a server by a single
client workstation running this benchmark
◎A load unit corresponds to about five Andrew
users
◎The input to the benchmark is a read-only
source subtree consisting of about 70 files
12
13. Phases of the benchmark
◎ MakeDir: constructs a target subtree that is identical in
structure to the source subtree
◎ Copy: copies every file from the source subtree to the
target subtree
◎ ScanDir: recursively traverses the target subtree and
examines the status of every file in it. It doesn’t actually
read the contents of any file
◎ ReadAll: scans every byte of every file in the target
subtree once
◎ Make: compiles and links all the files in the target
subtree
13
15. Performance Observations
◎Venus used two caches
○ one for files
○ the other for status information about files
◎The most frequent system calls
○ TestAuth, validated cache entries
○ GetFileStat, obtained status information about files
absent from the cache
15
16. ● 12 machines
○ 81% file-cache hit ratio
○ 82% status-cache hit ratio
○ 6% Fetch and Stores Calls
16
17. ● The total running time for the benchmark as a function
of server load
● The table also shows the average response time for the
most frequent Vice operation, TestAuth
17
18. ● A software on servers to maintain statistics about CPU
and disk utilization and about data transfers to and
from the disks
● Table IV presents these data for four servers over a 2-
week period
● As the CPU utilizations in the table show, the servers
loads were not evenly balanced
● CPU utilization of about 40 percent for two most used
servers.
18
19. Performance Observations
◎The performance bottleneck in our prototype
was the server CPU (high CPU utilization)
○ the frequency of context switches between the many
server processes
○ the time spent by the servers in traversing full
pathnames presented by workstations
19
20. Performance Observations
◎Significant performance improvement is
possible if
○ we reduce the frequency of cache validity checks
○ reduce the number of server processes
○ require workstations rather than the servers to do
pathname traversals
○ balance server usage by reassigning users
20
21. Changes for performance
◎Unchanged aspects
○ workstations cache entire files from a collection of
dedicated autonomous servers
○ venus and server code run as user-level processes
○ communication between servers and clients is based
on the RPC paradigm
○ the mechanism in the workstation kernels to
intercept and forward file requests to Venus is the
same as in the prototype
21
22. Changes for performance
◎Changed aspects
○ To enhance performance
● Cache management
● Name resolution
● Communication and server process structure
● Low-level storage representation
○ To improve the operability of the system
22
23. Cache management
◎Caching, the key to Andrew’s ability to scale
well
◎Venus now caches the contents of directories
and symbolic links in addition to files
◎There are still two separate caches:
○ for status
○ for data
◎Venus uses LRU to keep each of them
bounded in size
23
24. Cache management
◎ Modifications to a cached file are done locally and are
reflected back to Vice when the file is closed
◎ Venus intercepts only the opening and closing of files
and doesn’t participate in the reading or writing of
individual bytes on a cached copy
◎ For the integrity, modifications to a directory are made
directly on the server responsible for that directory
◎ Venus reflects the change in its cached copy to avoid
refetching the directory
24
25. Cache management
◎Callback mechanism to keep consistent cache
entries
○ Venus assumes that cache entries are valid unless
otherwise notified
○ When a workstation caches a file, the server promises
to notify it before allowing a modification by any
other workstation
◎Callback reduces the number of cache
validation requests received by servers
25
26. Cache management
◎Merits
○ reducing cache validation traffic
○ callback reduces the load on servers considerably
○ callback makes it feasible to resolve pathnames on
workstations
◎Defects
○ callback complicates the system because each server
and Venus now maintain callback state information
○ there is a potential for inconsistency if the callback
state maintained by a Venus gets out of sync with the
corresponding state maintained by the servers
26
27. Name Resolution
◎4.2BSD system
○ inode: a file has a unique, fixed-length name
○ one or more variable-length pathnames that map to
this inode
◎The routine that performs this mapping,
namei, is usually one of the most heavily used
◎CPU overhead on the servers and was an
obstacle to scaling
27
28. Name Resolution
◎The notion of two-level names
◎Each Vice file or directory is now identified by a
unique fixed-length Fid (96 bit)
◎Each entry in a directory maps a component of a
pathname to a fid
◎Venus performs the logical equivalent of a namei
operation, and maps Vice pathnames to fids
◎Servers are presented with fids and are unaware
of pathnames
28
29. Communication and Server Process Structure
◎The use of a server process per client didn’t scale
well
◎Server processes couldn’t cache critical shared
information in their address spaces
◎4.2BSD doesn’t permit processes to share virtual
memory
◎The redesign solves these problems by using a
single process to service all clients of a server
29
30. Communication and Server Process Structure
◎An user-level mechanism to support multiple
nonpreemptive Lightweight Processes (LWPs)
within one process
◎Context switching between LWPs is only on the
order of a few procedure-call times
◎Number of LWPs is typically five
◎RPC mechanism is implemented entirely outside
the kernel and is capable of supporting many
hundreds or thousands of clients per server
30
31. Low-Level Storage Representation
◎ Limited cost of the namei operations involved in
accessing data via pathnames
◎ Access files by their inodes
◎ The internal inode interface is not visible to user-level
processes
◎ Added an appropriate set of system calls
◎ The vnode information for a Vice file identifies the
inode of the file storing its data
◎ Data access consists of an index of a fid into a table to
look up vnode information
◎ An iopen call to read or write the data
31
32. Overall Design
◎Suppose a user process opens a file with
pathname P on a workstation
◎The kernel detects that it is a Vice file and
passes it to Venus on that workstation
◎Venus now uses the cache to examine each
directory component D of P in succession
32
33. Overall Design
1° Case
If D is in the cache
and has a callback
on it, it’s used
without any network
communication
2° Case
If D is in the cache
but has no callback
on it, the
appropriate server is
contacted, a new
copy of D is fetched
if it has been
updated, and a
callback is
established on it
3° Case
If D isn’t in the
cache, it is fetched
from the appropriate
server, and a
callback is
established on it
33
34. Overall Design
File F is
identified
34
Cache copy
is created
Venus
returns to
the kernel
Kernel opens
the cached
copy of F and
returns its
handle to the
user process
Directories
and F are in
the cache
with
callbacks
Venus regains
control when F
is closed.If it
has been
modified
locally,updates
it on the server
35. Overall Design
◎Future references to this file will involve no
network communication at all, unless a
callback is broken on a component of P
◎An LRU replacement algorithm is periodically
run to reclaim cache space
◎Problem: since the other LWPs in Venus may
be concurrently servicing file access requests
from other processes, accesses to cache data
structures must be synchronized
35
36. Overall Design
◎Design converged on the following consistency
semantics
○ Writes to an open file by a process on a workstation are
visible to all other processes on the workstation
immediately but are invisible elsewhere in the network
○ Once a file is closed, the changes made to it are visible to
new opens anywhere on the network
○ All other file operations are visible everywhere on the
network immediately after the operation completes
○ Application programs have to cooperate in performing the
necessary synchronization if they care about the
serialization of these operations
36
37. 2° Prototype
◎Two questions
○ It has the anticipated improvement in scalability been
realized?
○ What are the characteristics of the system in normal
operation?
◎Client workstations
○ IBM-RT
○ Storage 40 or 70MB
○ First Models with 4MB RAM
37
38. ● Benchmark
○ Andrew workstation is 19%
slower than a stand-alone
○ The prototype was 70% slower
○ Copy and Make phases are
most susceptible to server load
38
39. ● CPU and disk utilization on the server during the benchmark
○ CPU - from 8.1% to 70.9%
○ Disk - from 2.7% to 23.6%
● CPU still limits performance in our system
● Our design changes have improved scalability considerably
○ At a load of 20, the system is still not saturated (50 Andrew users)
39
41. Comparison with a remote-open file system
◎Considerations about caching of entire files
on local disks in the AFS
○ Locality: server load and network traffic are reduced
○ Whole-file transfer approach contacts servers only on
opens and closes
○ Disk caches retain their entries across reboots
○ Caching of entire files simplifies cache management
41
42. Comparison with a remote-open file system
◎Workstations require local disks for
acceptable performance
◎Files that are larger than the local disk cache
cannot be accessed at all
◎Concurrent read and write semantics across
workstations is impossible
42
43. Comparison with a remote-open file system
◎Could an alternative design have produced
equivalent or better results?
◎How critical to scaling are caching and whole-
file transfer?
◎AFS vs NFS
43
44. Network File System
◎NFS developed by Sun Microsystems in 1984
◎builds on the Open Network Computing
Remote Procedure Call (ONC RPC) system
◎Client-Server Protocol
◎Sharing file, directory or FS
◎SO Unix
44
47. AFS vs NFS
◎Scalability assessment, to compare with NFS
○ NFS is a mature product, not a prototype
○ Sun has spent a lot in NFS
○ Different architecture, same hardware
◎Benchmark
○ 18 workstations Sun3
○ A server Sun3
○ Client-Server on a 10MB Ethernet
○ Experiments on Andrew consists in a Cold Cache and
Warm Cache
47
48. 48
● NFS performs slightly
better than Andrew at
low loads, not with huge
loads
● ScanDir, ReadAll, Make
show different
performance
● Andrew wins on NFS in
ReadAll
49. 49
● Benchmark in function of load unit
● CPU utilization in function of load unit
○ Load 1: 22% NFS - 3% AFS
○ Load 18: 100% NFS - 38% AFS in Cold
Cache and 42% in Warm Cache
50. Server Side
● Load 1
○ NFS, Disk 1: 9%
○ NFS, Disk 2: 3%
○ AFS, Disk 1: 4%
● Load 18
○ NFS, Disk 1: 95%
○ NFS, Disk 2: 19%
○ AFS, Disk 1: 33%
50
51. Conclusions
◎ AFS is more scalable than NFS
◎ AFS would reduce its costs if moved in kernel side
◎ Andrew is implemented in user space
◎ NFS in kernel space
51
52. Conclusions
◎ AFS testing
○ 400 workstations, 16 server, 3500 Andrew users
◎ Goals
○ Scaling until 5000 workstation
○ Moving Venus and server code in kernel
◎ AFS is a great FS
◎ Excellent Performance
◎ It compares favorably with the most prominent
alternative distributed file system
52
Andrew File System presenta uno spazio di nomi di file omogeneo e trasparente alla posizione su tutte le workstation client
Il nostro obiettivo principale nella creazione di un prototipo era di convalidare l'architettura di base del file system
Per massimizzare il numero di client che possono essere supportati da un server, la maggior parte del lavoro possibile viene eseguita da Venus anziché da Vice
Poiché 4.2BSD non consente la condivisione di spazi di indirizzi tra processi, tutte le comunicazioni e la manipolazione di strutture di dati tra processi del server sono avvenute tramite file nel file system sottostante
Venus si incontrerà con un processo in ascolto su un noto indirizzo di rete su un server
Questo processo ha quindi creato un processo dedicato per gestire tutte le richieste future del client
Tutte le comunicazioni e la manipolazione delle strutture di dati tra i processi del server sono avvenute tramite file nel file system sottostante
Ogni server conteneva una gerarchia di directory che rispecchiava la struttura dei file Vice memorizzati su di esso.
La gerarchia della directory conteneva le directory Stub che rappresentavano porzioni dello spazio dei nomi dei vice situati su altri server. Il database delle posizioni che mappa i file sui server è stato quindi incorporato nella struttura dei file. Se un file non si trovava su un server, la ricerca del suo nome terminerebbe in una directory stub che identificava il server contenente quel file
Questo benchmark consiste in uno script di comando che opera su una raccolta di file che costituiscono un programma applicativo
Unità di carico: carica posta su un server da una singola workstation client che esegue questo benchmark
Un'unità di carico corrisponde a circa cinque utenti Andrew
L'input per il benchmark è una sottostruttura sorgente di sola lettura composta da circa 70 file
MakeDir: crea una sottostruttura di destinazione identica nella struttura alla sottostruttura di origine
Copia: copia ogni file dalla sottostruttura di origine alla sottostruttura di destinazione
ScanDir: attraversa in modo ricorsivo il sottostruttura di destinazione ed esamina lo stato di ogni file in esso contenuto. In realtà non legge il contenuto di alcun file
ReadAll: esegue la scansione di ogni byte di ogni file nella sottostruttura di destinazione una volta
Crea: compila e collega tutti i file nella sottostruttura di destinazione
Su una workstation Sun 2 con un disco locale, questo benchmark impiega circa 1000 secondi per il completamento quando tutti i file sono ottenuti localmente. I tempi corrispondenti per le altre macchine sono mostrati nella Tabella I.
La voce TestAuth ha convalidato le voci della cache, mentre GetFileStat ha ottenuto informazioni sullo stato dei file assenti dalla cache.
Un'istantanea delle cache di 12 macchine ha mostrato un tasso medio di riscontri tra cache e file dell'81%, e un rapporto di riscontri tra cache e stato dell'82%, La tabella mostra anche che solo il 6% delle chiamate a Vice (Recupera e archivia) ha effettivamente comportato il trasferimento di file e che il rapporto delle chiamate di recupero per le chiamate di archivio era approssimativamente 2:1.
Il tempo totale di esecuzione del benchmark in funzione del carico del server
La tabella mostra anche il tempo medio di risposta per l'operazione Vice più frequente, TestAuth
Un software sui server per mantenere le statistiche sull'uso della CPU e del disco e sui trasferimenti di dati da e verso i dischi
La Tabella IV presenta questi dati per quattro server nell'arco di un periodo di 2 settimane
Come mostrano gli utilizzi della CPU nella tabella, i carichi dei server non erano equamente bilanciati
Utilizzo della CPU di circa il 40 percento per i due server più utilizzati.
la frequenza degli switch di contesto tra i molti processi del server e il tempo trascorso dai server nel percorrere i percorsi completi presentati dalle workstation.
il miglioramento significativo delle prestazioni è possibile se riduciamo la frequenza dei controlli di validità della cache, riduciamo il numero di processi del server, richiediamo workstation anziché i path per attraversare il percorso e bilanciamo l'utilizzo del server riassegnando gli utenti.
Le wrokstations memorizzano nella cache interi file da una raccolta di server autonomi dedicati
venus e il codice del server vengono eseguiti come processi a livello di utente
la comunicazione tra server e client si basa sul paradigma RPC
il meccanismo nei kernel delle workstation per intercettare e inoltrare le richieste di file a Venus è lo stesso del prototipo
RPC = Remote Procedure Call
Quindi l'RPC consente a un programma di eseguire subroutine "a distanza" su computer remoti, accessibili attraverso una rete. Essenziale al concetto di RPC è l'idea di trasparenza: la chiamata di procedura remota deve essere infatti eseguita in modo il più possibile analogo a quello della chiamata di procedura locale; i dettagli della comunicazione su rete devono essere "nascosti" (resi trasparenti) all'utilizzatore del meccanismo.
Per migliorare le prestazioni
Gestione della cache
Risoluzione del nome
Struttura del processo di comunicazione e server
Rappresentazione dello storage di basso livello
Per migliorare l'operabilità del sistema
Caching, la chiave per la capacità di Andrew di scalare bene
Venus ora memorizza nella cache il contenuto delle directory e dei collegamenti simbolici oltre ai file
Ci sono ancora due cache separate:
per lo stato
per i dati
Venus usa LRU per mantenere ognuna di esse delimitata in dimensioni
Le modifiche a un file memorizzato nella cache vengono eseguite localmente e vengono riflesse su Vice quando il file viene chiuso
Venus intercetta solo l'apertura e la chiusura di file e non partecipa alla lettura o scrittura di singoli byte su una copia cache
Per l'integrità, le modifiche a una directory vengono effettuate direttamente sul server responsabile di quella directory
Venus riflette la modifica nella sua copia memorizzata nella cache per evitare il refetching della directory
Meccanismo di callback per mantenere le voci della cache coerenti
Venus presume che le voci della cache siano valide salvo diversa comunicazione
Quando una workstation memorizza nella cache un file, il server promette di notificarlo prima di consentire una modifica da parte di un'altra workstation
La richiamata riduce il numero di richieste di convalida della cache ricevute dai server
A small amount of cache validation traffic is still present, usually to replace callbacks lost because of machine or network failures
Una piccola quantità di traffico di validazione della cache è ancora presente, in genere per sostituire i callback persi a causa di errori della macchina o della rete
meriti
riducendo il traffico di validazione della cache
la funzione di callback riduce considerevolmente il carico sui server
la funzione di callback rende possibile la risoluzione dei nomi di percorso sulle workstation
difetti
la callback complica il sistema poiché ogni server e Venus ora gestiscono le informazioni sullo stato del callback
esiste una possibilità di incoerenza se lo stato di callback mantenuto da una Venus non è sincronizzato con lo stato corrispondente gestito dai server
In un sistema convenzionale 4.2BSD, un file ha un nome univoco, a lunghezza fissa, il suo inode e uno o più nomi di percorsi a lunghezza variabile che si associano a questo inode. La routine che esegue questo mapping, namei, è in genere una delle parti del kernel più pesantemente utilizzate e che richiede tempo.
Ciò ha comportato un notevole sovraccarico della CPU sui server ed è stato un ostacolo alla scalabilità
SOLUZIONE :
La nozione di nomi a due livelli
Ogni file o directory di Vice è ora identificata da un Fid di lunghezza fissa unico (96 bit)
Ogni voce in una directory associa un componente di un percorso a un fid
Venus esegue l'equivalente logico di un'operazione denominata e mappa i nomi dei percorsi di Vice in fids
I server sono presentati con i fids e non sono a conoscenza dei nomi dei percorsi
L'utilizzo di un processo server per client non ha avuto una buona scalabilità
I processi del server non potevano memorizzare nella cache informazioni critiche condivise negli spazi degli indirizzi
4.2BSD non consente ai processi di condividere la memoria virtuale
La riprogettazione risolve questi problemi utilizzando un singolo processo per servire tutti i client di un server
Un meccanismo a livello di utente per supportare più processi leggeri non penetrabili (LWP) all'interno di un unico processo
Il cambio di contesto tra LWP è solo nell'ordine di poche procedure, i tempi di chiamata
Il numero di LWP è in genere cinque (Il numero di LWP (in genere cinque) viene determinato quando un server viene inizializzato e rimane fisso in seguito)
Il meccanismo RPC è implementato interamente al di fuori del kernel ed è in grado di supportare molte centinaia o migliaia di client per server
Come menzionato nella Sezione 3.2, siamo stati cauti sul costo delle operazioni namei coinvolte nell'accesso ai dati tramite i nomi dei percorsi. Pertanto, abbiamo deciso di accedere ai file tramite i loro inode anziché tramite i nomi dei percorsi. Poiché l'interfaccia inode interna non è visibile ai processi a livello utente, abbiamo dovuto aggiungere un insieme appropriato di chiamate di sistema. Le informazioni di vnode per un file Vice identificano l'inode del file che memorizza i suoi dati.
Aggiunto un set appropriato di chiamate di sistema
Le informazioni di vnode per un file Vice identificano l'inode del file che memorizza i suoi dati
L'accesso ai dati consiste in un indice di un fid in una tabella per cercare informazioni su vnode
Una chiamata di iopen per leggere o scrivere i dati
Test di accesso ad un file remoto
Supponiamo che un processo utente apra un file con il percorso P su una workstation. Il kernel, nel risolvere P, rileva che è un file Vice e lo passa a Venus su quella workstation. Uno dei LWP che comprende Venus ora utilizza la cache per esaminare in successione ciascun componente di directory D di P:
-Se D è nella cache e ha una richiamata su di esso, viene utilizzato senza alcuna comunicazione di rete.
-Se D è nella cache ma non contiene alcuna richiamata, viene contattato il server appropriato, una nuova copia di D viene recuperata se è stata aggiornata e su di essa è stata stabilita una richiamata.
-Se D non è nella cache, viene prelevato dal server appropriato e su di esso viene stabilita una richiamata.
Quando viene identificato il file di destinazione F, viene creata una copia cache corrente nello stesso modo. Venus quindi ritorna al kernel, che apre la copia cache di F e restituisce il suo handle al processo utente. Quindi, alla fine del pathname traversal, tutte le directory intermedie e il file di destinazione sono nella cache con i callback su di essi. I futuri riferimenti a questo file non implicheranno alcuna comunicazione di rete, a meno che una callback non venga interrotta su un componente di P. Venus riprende il controllo del file quando è chiuso e, se è stato modificato localmente, lo aggiorna sul server appropriato. Un algoritmo di sostituzione LRU viene periodicamente eseguito per recuperare spazio nella cache.
Cache consistency and concurrent control
I futuri riferimenti a questo file non implicheranno alcuna comunicazione di rete, a meno che una callback non venga interrotta su un componente di P
Un algoritmo di sostituzione LRU viene periodicamente eseguito per recuperare spazio nella cache
Problema: poiché gli altri LWP di Venus possono essere contemporaneamente utilizzati per le richieste di accesso ai file da altri processi, gli accessi alle strutture di dati della cache devono essere sincronizzati
Il design convergeva sulla seguente semantica della coerenza
Le scritture su un file aperto da un processo su una workstation sono immediatamente visibili a tutti gli altri processi sulla workstation, ma sono invisibili altrove nella rete
Una volta che un file è chiuso, le modifiche apportate ad esso sono visibili alle nuove aperture in qualsiasi punto della rete
Tutte le altre operazioni sui file sono visibili ovunque sulla rete immediatamente dopo il completamento dell'operazione
I programmi applicativi devono cooperare per eseguire la sincronizzazione necessaria se si preoccupano della serializzazione di queste operazioni
Due domande
È stato realizzato il miglioramento anticipato della scalabilità?
Quali sono le caratteristiche del sistema nel normale funzionamento?
Table VI shows that the Copy and Make phases are most susceptible to server load.
La tabella VIII presenta CPU del server e utilizzi del disco in Andrew. Le cifre mostrate sono medie su un periodo di un'ora dalle 9:00 A.M. alle 5:00 P.M. nei giorni della settimana. La maggior parte dei server mostra un utilizzo della CPU tra il 15 e il 25 percento. Uno dei server, vice4, mostra un utilizzo del 35,8 percento, ma l'utilizzo del disco non è corrispondentemente alto. L'elevata deviazione standard per l'utilizzo della CPU ci porta a credere che questa anomalia sia stata causata da attività di manutenzione del sistema che sono state eseguite inaspettatamente durante il giorno piuttosto che durante la notte. Server vice9, d'altra parte, mostra un utilizzo della CPU del 37,6 percento con una piccola deviazione standard. L'utilizzo del disco è del 12,1 percento, il più alto di qualsiasi server. L'elevato utilizzo è spiegato dal fatto che questo server memorizza le bacheche elettroniche, una raccolta di directory che sono spesso accessibili e modificate da molti utenti diversi.
La memorizzazione nella cache di interi file su dischi locali in Andrew File System era motivata principalmente dalle seguenti considerazioni di scala:
-Località dei riferimenti di file da parte degli utenti tipici rende attraente la cache: il carico del server e il traffico di rete sono ridotti.
-Un approccio di trasferimento di file completi contatta i server solo su apre e chiude. Le operazioni di lettura e scrittura, che sono molto più numerose, sono trasparenti per i server e non causano traffico di rete.
- lo studio di Ousterhout et al. [4] ha mostrato che la maggior parte delle tessere in un ambiente 4.2BSD sono lette nella loro interezza. Il trasferimento di file interi sfrutta questa proprietà consentendo l'uso di efficienti protocolli di trasferimento di dati alla rinfusa.
-Le cache dei dischi mantengono le loro voci attraverso i riavvii, un evento sorprendentemente frequente negli ambienti workstation. Poiché alcuni dei file a cui si accede da un utente tipico sono suscettibili di essere modificati altrove nel sistema, la quantità di dati recuperati dopo un riavvio è in genere ridotta.
-Infine, il caching di interi file semplifica la gestione della cache. Venere deve solo tenere traccia delle tessere nella sua cache, non delle loro singole pagine.
Le workstation richiedono dischi locali per prestazioni accettabili
Non è possibile accedere a file più grandi della cache del disco locale
La semantica simultanea di lettura e scrittura tra le workstation è impossibile
Un progetto alternativo potrebbe aver prodotto risultati equivalenti o migliori?
Quanto sono importanti per il ridimensionamento del caching e il trasferimento di interi file?
AFS vs NFS
Gli esperimenti di Andrew consistevano in due sottoinsiemi: un set di Cold Cache, in cui le cache venivano cancellate prima di ogni prova e un set di Warm Cache, in cui le cache venivano lasciate inalterate.
Valutazione della scalabilità, da confrontare con NFS
NFS è un prodotto maturo, non un prototipo
Sun ha speso molto in NFS
Architettura diversa, stesso hardware
segno di riferimento
18 postazioni di lavoro Sun3
Un server Sun3
Client-Server su Ethernet da 10 MB
Gli esperimenti su Andrew consistono in Cold Cache e Warm Cache