Introduction to ha loop

1,231 views

Published on

Published in: Technology, Travel
  • Be the first to comment

  • Be the first to like this

Introduction to ha loop

  1. 1. Introduction To HaLoop<br />xiafei.qiu@PCA<br />
  2. 2. How hadoop works<br />
  3. 3. Page Rank in Hadoop<br />
  4. 4.
  5. 5. Page Rank in Hadoop<br />
  6. 6. Differences<br />
  7. 7.
  8. 8. Loop-aware Scheduling<br />place on the same physical machines those map and reduce tasks that occur in different iterations but access the same data.<br />
  9. 9. Scheduling Algorithm<br />the number of reduce tasks should be invariant across iterations, so that the hash function assigning mapper outputs to reducer nodes remains unchanged.<br />the master node maintains a mapping from each slave node to the data partitions that this node processed in the previous iteration.<br />
  10. 10. Caches<br />Reducer Input Cache<br />Same key hashed to same reducer.<br />f must be deterministic, same across iterations, take tuple t as only the input.<br />Number of reducers remains unchanged.<br />Reducer Output Cache<br />That is, if two Reduce function calls produce the same output key from two different reducer input keys, both reducer input keys must be in the same partition so that they are sent to the same reduce task.<br />Mapper Input Cache<br />
  11. 11. Inspirations<br />

×