This document describes how to distribute computation to sort massive data over multiple slaves. It involves: 1) Dividing the data into chunks and distributing the chunks across 4 slave nodes to perform sorting in parallel. 2) The master node takes 1000 elements from each sorted chunk and performs a heap sort, continuing until all data is sorted. 3) The design includes fault tolerance by having the master node check the status of each slave node through a heartbeat mechanism to ensure slaves remain alive during processing.

1. Distributed Computation to
Sort massive data over
by
Parth Sawant , Suhas Pillai & Siddesh Pillai
0101001 10110100 10110010 00110010 00110010 10111001 101101000
01010011 01110101 01101000 01100001 01110011
&
01010000 01100001 01110010 011101000 1101000

2. Example Scenario
• 8 MB data containing (char + int) strings and you have 4 slaves.
MASTER
SLAVESSLAVESSLAVESSLAVES

4. Master
SlaveSlaveSlaveSlave
CHUNKS CHUNKSCHUNKSCHUNKS
sort sortsortsort

5. Store file objects
in a List
Sorted
chunks
Sorted
chunks
Sorted
chunks
Sorted
chunks
Sorted
chunks
MASTER

6. File
Objects
File
Objects
File
Objects
File
Objects
File
Objects
File
Objects
File
Objects
File
Objects
TAKE 1000 elements from
every file and do heap sort
and continue till all are read
Sorted
file

8. Fault Tolerance
SLAVE
SLAVE
SLAVE
SLAVE Heart
beat
MASTER
Checks whether the
slave is alive or dead or
may be in coma

9. • Similar design pattern as task one.
• Store the values in Hash Map (provenance and sum)
• Reduction on master.
Task - 2

10. Questions