The document discusses the Google File System (GFS), which was developed by Google to handle large files across thousands of commodity servers. It provides three main functions: (1) dividing files into chunks and replicating chunks for fault tolerance, (2) using a master server to manage metadata and coordinate clients and chunkservers, and (3) prioritizing high throughput over low latency. The system is designed to reliably store very large files and enable high-speed streaming reads and writes.

1. Guided By: Developed By:
Prof. Jinal V. Purohit Sharma Shivam(X-46)

2.  Introduction
 Motive To Build
 Design Overview
 System Architecture
 System Interaction
 Working of GFS
 Garbage Collection
 Fault tolerance
 Conclusion

3. ◌ GFS is scalable distributed file system for large distributed
data-intensive applications.
◌ Provides fault tolerance, serving large number of clients with
high aggregate performance.
◌ The field of Google is beyond the searching.
◌ Google store the data in more than 15 thousands commodity
hardware.
◌ Handles the exceptions of Google and other Google specific
challenges in their distributed file system.

4. ◌ Google had key observations upon which they decided to
build their own Distributed File System.
◌ Cost Effective:
◌The system is built using inexpensive commodity components
where components failure is the norm and not the exception.
◌So the system must detect, tolerate, and recover from failures on
a routine basis.
◌ File Size:
◌ Multi GB files are the common case, so the system must be optimized in
managing large files.
◌ Small files also are supported but no need to optimize for them.

5. ◌Read Operation:
◌Large Data Streams
◌An operation reads hundreds of KBs or maybe 1 MB
or more.
◌Successive operations from the same client reads
usually from the same file region.
◌Random Reads
◌An operation reads a few KBs staring from an arbitrary
offset.
◌Performance - conscious applications usually patch
and sort their small reads to advance steadily in the
file instead going back and forth.

6. ◌Write Operation:
◌ Are the same in size as the read operations.
◌ Once written the files are seldom modified.
◌ Write operations are in the form of sequential
append.
◌ Random writes are supported but not efficient.

7. ◌ From many inexpensive commodity components that
often fail.
◌ Stores a modest number of large files.
◌ Workloads consist of large streaming reads and
small random reads.
◌ Workloads also have many large, sequential writes
that append data to files.
◌ Efficiently implement well-defined semantics for
multiple clients.
◌ High sustained bandwidth is more important than low
latency.

8. ◌ GFS cluster consists of a single master and multiple
chunkservers.
◌ The basic analogy of GFS is master , client ,
chunkservers.

9. ◌ Files are divided into fixed-size chunks.
◌ Chunkservers store chunks on local disks as
Linux files.
◌ Master maintains all file system metadata.
◌ Includes the namespace, access control
information, the mapping from files to chunks, and
the current locations of chunks.
◌ Clients interact with the master for metadata
operations.
◌ Chunkservers need not cache file data .

10. Chunk:-
◌ Similar to the concept of block in file systems.
◌ Compared to file systems, the size of chunk is 64
MB.
◌ Less chunks and less metadata for chunks in the
master.
◌ Problem in this chunk size is developing a hotspot.
◌ Property of chunk is chunks are stored in
chunkserver as file, chunk handle, i.e., chunk file
name.

11. Metadata :-
◌ Master stores three major types of metadata: the
file and chunk namespaces, the mapping from
files to chunks, and the location of each chunk’s
replicas.
◌ First two types are kept persistent to an operation
log stored on the master’s local disk.
◌ Metadata is stored in memory, master operations
are fast.

12. ◌ Easy and efficient for the master to periodically scan .
◌ Periodic scanning is used to implement chunk
garbage collection, re-replication and chunk
migration.
Master:-
◌ Single process ,running on a separate machine that
stores all metadata.
◌ Clients contact master to get the metadata to
contact the chunkservers.

13. o GFS provides an interface to:
 Create
 Delete
 Open
 Close
 Read
 Write
 Snapshot (Copy)
 Record Append

14. 1. Application originates the read request.
2. GFS client translates the request form (filename, byte range) -
> (filename, chunk index), and sends it to master.
3. Master responds with chunk handle and replica locations (i.e.
chunkservers where the replicas are stored) .
Read Algorithm

15. 4. Client picks a location and sends the (chunk
handle, byte range) request to the location.
5. Chunkserver sends requested data to the client.
6. Client forwards the data to the application.

16. 1. Application originates the write request.
2. GFS client translates request from (filename, data) ->
(filename, chunk index), and sends it to master.
3. Master responds with chunk handle and (primary +
secondary) replica locations.
Write Algorithm

17. 4. Client pushes write data to all locations. Data is
stored in chunkservers’ internal buffers.
5. Client sends write command to primary.

18. 6. Primary determines serial order for data instances stored in its
buffer and writes the instances in that order to the chunk.
7. Primary sends the serial order to the secondary and tells them
to perform the write
8. Secondary respond to the primary.
9. Primary responds back to the client.

19. 1. Application originates record append request.
2. GFS client translates requests and sends it to master.
3. Master responds with chunk handle and (primary + secondary)
replica locations.
4. Client pushes write data to all replicas of the last chunk of the
file.
5. Primary checks if record fits in specified chunk.
6. If record doesn’t fit, then the primary:
- Pads the chunk.
- Tell secondary to do the same.
- And informs the client.
- Client then retries the append with the next chunk.
Record Append Algorithm

20. 7. If record fits, then the primary:
- Appends the record.
- Tells secondary to write data at exact offset.
- Receives responses from secondary.
- And sends final response to the client.

21. • Factors for choosing where to place the initially empty replicas:
 (1)We want to place new replicas on chunkservers with
below-average disk space utilization.
 (2) We want to limit the number of “recent” creations on
each chunkserver.
 (3)Spread replicas of a chunk across racks.
• Master re-replicates a chunk.
• Chunk that needs to be re-replicated is prioritized based on
how far it is from its replication goal.
• Finally, the master rebalances replicas periodically.
Creation , Re-replication and Balancing Chunks

23. • Garbage collection at both the file and chunk levels.
• Deleted by the application, the master logs the deletion
immediately.
• File is just renamed to a hidden name.
• The file can be read under the new, special name and
can be undeleted.
• Memory metadata is erased.

24. High Availability
 Fast Recovery
 Chunk Replication
 Master Replication
Data Integrity
 Chunkserver uses check summing.
 Broken up into 64 KB blocks.

25. Diagnostic Tools
 Extensive and detailed diagnostic logging
has helped immeasurably in problem
isolation, debugging, and performance
analysis, while incurring only a minimal
cost .
 RPC requests, replies etc.

26.  Different than previous file systems.
 Supporting large-scale data
processing.
 Provides fault tolerance.
 Tolerate chunkserver failures.
 Delivers high throughput.
 Storage platform for research and
development.