Twister4Azure : Iterative MapReduce for Azure Cloud<br />ThilinaGunarathne, Judy Qiu, Geoffrey Fox<br />{tgunarat, xqiu,gc...
MapReduceRolesfor Azure<br />Familiar MapReduce programming model<br />Built using highly-available and scalable Azure clo...
MapReduceRolesfor Azure<br />
Twister for Azure<br /><ul><li>Merge Step
In-Memory Caching of static data
Cache aware hybrid scheduling using Queues as well as using a bulletin board (special table) </li></li></ul><li>Twister fo...
Performance – Kmeans Clustering<br />Performance with/without data caching. <br />Speedup gained using data cache<br />Inc...
Performance Comparisons<br />BLAST Sequence Search<br />Smith Watermann Sequence Alignment<br />Cap3 Sequence Assembly<br />
Upcoming SlideShare
Loading in …5
×

Twister4Azure - Iterative MapReduce for Azure Cloud

3,325 views

Published on

Twister4Azure is an iterative MapReduce framework, which support development and execution of Iterative MapReduce and traditional MapReduce application in Microsoft Azure cloud.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
3,325
On SlideShare
0
From Embeds
0
Number of Embeds
1,658
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • We use Azure Queues for scheduling, Tables to store meta-data and monitoring data, Blobs for input/output/intermediate data storage.
  • In our current work, we extend MR4Azure to support iterative mapreduce computations. We added an additional merge step to the programming model, where the computations decides whether to go for a new iteration or not. We also support in-memory caching of static data between iterations and we developed a hybrid scheduling strategy to perform cache aware scheduling.
  • We don’t have a master node, who has the global knowledge about the cached data.. Hence in each iteration, tasks will be posted to the bulleting board, where workers will first check to identify any tasks that require a data product they have in cache. If not they fall back to the queue.
  • KMeans iterative MapReduce performance.16 Azure Small instances,6 iterations,8 to 48 million 20-D data points.Left: Performance with and without data caching.Right: Speedup obtained from using the data cacheLeft: Scaling speedup with increasing number of instances (Azure Small) &amp; data for 10 iterations.Right: Increasing number of iterations using 16 million data points with caching using 16 Azure Small instances.
  • Twister4Azure - Iterative MapReduce for Azure Cloud

    1. 1. Twister4Azure : Iterative MapReduce for Azure Cloud<br />ThilinaGunarathne, Judy Qiu, Geoffrey Fox<br />{tgunarat, xqiu,gcf}@indiana.edu<br />CCA 2011<br />April 12 – 13, 2011<br />
    2. 2. MapReduceRolesfor Azure<br />Familiar MapReduce programming model<br />Built using highly-available and scalable Azure cloud services<br />Co-exist with eventual consistency & high latency of cloud services<br />Decentralized control<br />No single point of failure.<br />Supports dynamically scaling up and down of the compute resources.<br />MapReduce fault tolerance<br />
    3. 3. MapReduceRolesfor Azure<br />
    4. 4. Twister for Azure<br /><ul><li>Merge Step
    5. 5. In-Memory Caching of static data
    6. 6. Cache aware hybrid scheduling using Queues as well as using a bulletin board (special table) </li></li></ul><li>Twister for Azure<br />
    7. 7. Performance – Kmeans Clustering<br />Performance with/without data caching. <br />Speedup gained using data cache<br />Increasing number of iterations<br />Scaling speedup<br />
    8. 8. Performance Comparisons<br />BLAST Sequence Search<br />Smith Watermann Sequence Alignment<br />Cap3 Sequence Assembly<br />
    9. 9. Conclusion<br />Enables users to easily and efficiently perform large scale iterative data analysis and scientific computations on Azure cloud. <br />Utilizes a novel hybrid scheduling mechanism to provide the caching of static data across iterations. <br />Utilize cloud infrastructure services effectively to deliver robust and efficient applications.<br />http://salsahpc.indiana.edu/twister4azure<br />

    ×