Scaling Genetic Algorithms using MapReduce<br />AbhishekVerma, Xavier Llora, <br />David E. Goldberg, Roy H. Campbell<br />
Motivation<br />Genetic Algorithms (GAs)<br />applied to very large scale data-intensiveproblems<br />Current approach: MP...
Outline<br />Motivation<br />MapReduce<br />Genetic Algorithm<br />Approach<br />Experimental Results<br />Conclusion<br /...
MapReduce Overview<br />k1<br />v1<br />k1<br />v1<br />k2<br />v2<br />k1<br />v3<br />k1<br />v3<br />k1<br />v5<br />k2...
Genetic Algorithm<br />Initialize population with random individuals. <br />Evaluate fitness value of individuals.<br />Se...
Genetic Algorithm<br />Initialize population with random individuals. <br />Evaluate fitness value of individuals. <br />R...
MapReducing Genetic Algorithm<br />7<br />Random<br />partitioner<br />00010<br />10000<br />01001<br />&lt;00010, 1&gt;<b...
MapReducing Genetic Algorithm (2)<br />Modifications<br />Mappers write to DFS so that clients can evaluate convergence cr...
Experimental Results<br />9<br />Experimental setup<br />52 nodes: 16GB RAM, 2TB hard drives<br />Each node runs 5 mappers...
Scaling GAs to 100 million variables<br />10<br />Intelligent Systems Design and Applications 2009<br />
Conclusion<br />Modeled GAs in MapReduce<br />Scales on a commodity clusters to 100 million variables<br />Can also use Pt...
Questions?<br />
Thank You<br />
Upcoming SlideShare
Loading in...5
×

Isda

327

Published on

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
327
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Simple == Just two functions: Map and Reduce, Scalable == automatic parallelization across machines, fault tolerance, speculative execution,
  • Modify this slide to show partitioning function
  • Isda

    1. 1. Scaling Genetic Algorithms using MapReduce<br />AbhishekVerma, Xavier Llora, <br />David E. Goldberg, Roy H. Campbell<br />
    2. 2. Motivation<br />Genetic Algorithms (GAs)<br />applied to very large scale data-intensiveproblems<br />Current approach: MPI<br />Requires detailed knowledge of h/w architecture<br />Complicated to program, debug, checkpoint<br />Does not scale on commodity clusters<br />MapReduce: simple and scalable abstraction<br />Use MapReduce to scale GAs<br />2<br />Intelligent Systems Design and Applications 2009<br />
    3. 3. Outline<br />Motivation<br />MapReduce<br />Genetic Algorithm<br />Approach<br />Experimental Results<br />Conclusion<br />3<br />Intelligent Systems Design and Applications 2009<br />
    4. 4. MapReduce Overview<br />k1<br />v1<br />k1<br />v1<br />k2<br />v2<br />k1<br />v3<br />k1<br />v3<br />k1<br />v5<br />k2<br />v2<br />k2<br />v4<br />k2<br />v4<br />k1<br />v5<br />Input<br />records<br />h(k1)<br />Output<br />records<br />Map<br />Reduce<br />h(k1)<br />h(k2)<br />Split<br />h(k1)<br />Reduce<br />Map<br />h(k2)<br />Split<br />Shuffle<br />4<br />Intelligent Systems Design and Applications 2009<br />
    5. 5. Genetic Algorithm<br />Initialize population with random individuals. <br />Evaluate fitness value of individuals.<br />Select good solutions by using tournament selection without replacement.<br />Create new individuals by recombining the selected population using uniform crossover.<br />Evaluate the fitness value of all offspring.<br />Repeat steps 3-5 until some convergence criteria are met.<br />5<br />Intelligent Systems Design and Applications 2009<br />
    6. 6. Genetic Algorithm<br />Initialize population with random individuals. <br />Evaluate fitness value of individuals. <br />Repeat steps 4-5 to 2 until some convergence criteria are met.<br />Select good solutions by using tournament selection without replacement.<br />Create new individuals by recombining the selected population using uniform crossover.<br />6<br />Map<br />Reduce<br />Intelligent Systems Design and Applications 2009<br />
    7. 7. MapReducing Genetic Algorithm<br />7<br />Random<br />partitioner<br />00010<br />10000<br />01001<br />&lt;00010, 1&gt;<br />&lt;10000, 1&gt;<br />&lt;01001, 2&gt;<br />Map<br />10110<br />00001<br />Reduce<br />&lt;01001, 2&gt;<br />10001<br />01000<br />10001<br />01000<br />Reduce<br />10101<br />10000<br />00000<br />&lt;10101, 3&gt;<br />&lt;10000, 1&gt;<br />&lt;00000, 0&gt;<br />Map<br />&lt;10101, 3&gt;<br />Distributed File System<br />Intelligent Systems Design and Applications 2009<br />
    8. 8. MapReducing Genetic Algorithm (2)<br />Modifications<br />Mappers write to DFS so that clients can evaluate convergence criteria and control next iteration<br />Random partitioner function<br />Maintain a window of individuals in each reducer<br />Optimizations<br />Create the initial population in 0th MapReduce<br />Compactly represent bits in array of long ints<br />8<br />Intelligent Systems Design and Applications 2009<br />
    9. 9. Experimental Results<br />9<br />Experimental setup<br />52 nodes: 16GB RAM, 2TB hard drives<br />Each node runs 5 mappers + 3 reducers<br />Population set to nlog(n)<br />Intelligent Systems Design and Applications 2009<br />
    10. 10. Scaling GAs to 100 million variables<br />10<br />Intelligent Systems Design and Applications 2009<br />
    11. 11. Conclusion<br />Modeled GAs in MapReduce<br />Scales on a commodity clusters to 100 million variables<br />Can also use Pthreads(Phoenix), GPUs(Mars), …<br />Future Work<br />Demonstrate scalability for practical applications<br />MapReduce Compact GAs and Extended Compact GAs<br />Comparison with MPI implementation<br />11<br />Intelligent Systems Design and Applications 2009<br />
    12. 12. Questions?<br />
    13. 13. Thank You<br />
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×