Ironfan: Build a Hadoop Cluster in 20 minutes

Ironfan: Build a Hadoop Cluster in 20 Minutes Dhruv Bansal Nathaniel Eliot @dhruvbansal @temujin9Saturday, May 19, 2012

Saturday, May 19, 2012

OutlineSaturday, May 19, 2012

Why Ironfan? Outline • Why we love Chef • Why we don’t • Why we wrote IronfanSaturday, May 19, 2012

Why Ironfan? Outline • Why we love Chef • Why we don’t • Why we wrote Ironfan Ironfan in Action • Build a Hadoop cluster • Run a Hadoop job • Put that cluster awaySaturday, May 19, 2012

Why Ironfan? Outline • Why we love Chef • Why we don’t • Why we wrote Ironfan Ironfan in Action • Build a Hadoop cluster • Run a Hadoop job • Put that cluster away Ironfan roadmap • Better environments • Cloud agnosticism • Continuous integration • StacksSaturday, May 19, 2012

Why we love Chef Infrastructure as code: Version control Testable Shareable RecapitulableSaturday, May 19, 2012

Why we love Chef Modular, composable units: MySQL Nginx SOLR My ApplicationSaturday, May 19, 2012

Why we love ChefSaturday, May 19, 2012

Why we don’t love Chef Anything is possible Nothing is simple And there’s much repetition (not DRY)Saturday, May 19, 2012

Why we don’t love Chef Write Upload Compile Converge Development Machine Managed Server Hard Coded Dynamic Too much is hard-coded at development/upload time!Saturday, May 19, 2012

Why we don’t love Chef How do we make @server_ips dynamic?Saturday, May 19, 2012

Enter Ironfan We wrote Ironfan toSaturday, May 19, 2012

Enter Ironfan We wrote Ironfan to • Simplify, unify, and standardize our usage of the Chef toolsetSaturday, May 19, 2012

Enter Ironfan We wrote Ironfan to • Simplify, unify, and standardize our usage of the Chef toolset • Build further abstractions on top of ChefSaturday, May 19, 2012

Enter Ironfan We wrote Ironfan to • Simplify, unify, and standardize our usage of the Chef toolset • Build further abstractions on top of Chef • Give us superpowers that Chef doesn’t have yetSaturday, May 19, 2012

Enter Ironfan We wrote Ironfan to • Simplify, unify, and standardize our usage of the Chef toolset • Build further abstractions on top of Chef • Give us superpowers that Chef doesn’t have yet http://github.com/infochimps-labs/ironfanSaturday, May 19, 2012

What does Ironfan do?Saturday, May 19, 2012

What does Ironfan do? ChefSaturday, May 19, 2012

What does Ironfan do? Ironfan Simple helpers in the silverware cookbook abstract common Chef patterns and keep things DRY ChefSaturday, May 19, 2012

What does Ironfan do? Dynamic service discovery:Saturday, May 19, 2012

What does Ironfan do? A simple DSL for defining clusters of machinesSaturday, May 19, 2012

DEMO TIME!Saturday, May 19, 2012

Thanks for coming! Learn more about what Ironfan and Infochimps can sales@infochimps.com do for you. Come work at a fast- growing, cutting-edge careers@infochimps.com technology company based in Austin, TX. http://github.com/infochimps-labs/ironfan http://github.com/infochimps-labs/wukongSaturday, May 19, 2012

Ironfan: Build a Hadoop Cluster in 20 minutes

by temujin9

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 10

Statistics