Ironfan: Build a Hadoop Cluster in 20 minutes

2,592 views
2,530 views

Published on

The slides from our talk at ChefConf 2012

Published in: Business, Technology

Ironfan: Build a Hadoop Cluster in 20 minutes

  1. 1. Ironfan: Build a Hadoop Cluster in 20 Minutes Dhruv Bansal Nathaniel Eliot @dhruvbansal @temujin9Saturday, May 19, 2012
  2. 2. Saturday, May 19, 2012
  3. 3. OutlineSaturday, May 19, 2012
  4. 4. Why Ironfan? Outline • Why we love Chef • Why we don’t • Why we wrote IronfanSaturday, May 19, 2012
  5. 5. Why Ironfan? Outline • Why we love Chef • Why we don’t • Why we wrote Ironfan Ironfan in Action • Build a Hadoop cluster • Run a Hadoop job • Put that cluster awaySaturday, May 19, 2012
  6. 6. Why Ironfan? Outline • Why we love Chef • Why we don’t • Why we wrote Ironfan Ironfan in Action • Build a Hadoop cluster • Run a Hadoop job • Put that cluster away Ironfan roadmap • Better environments • Cloud agnosticism • Continuous integration • StacksSaturday, May 19, 2012
  7. 7. Why we love Chef Infrastructure as code: Version control Testable Shareable RecapitulableSaturday, May 19, 2012
  8. 8. Why we love Chef Modular, composable units: MySQL Nginx SOLR My ApplicationSaturday, May 19, 2012
  9. 9. Why we love ChefSaturday, May 19, 2012
  10. 10. Why we don’t love Chef Anything is possible Nothing is simple And there’s much repetition (not DRY)Saturday, May 19, 2012
  11. 11. Why we don’t love Chef Write Upload Compile Converge Development Machine Managed Server Hard Coded Dynamic Too much is hard-coded at development/upload time!Saturday, May 19, 2012
  12. 12. Why we don’t love Chef How do we make @server_ips dynamic?Saturday, May 19, 2012
  13. 13. Enter Ironfan We wrote Ironfan toSaturday, May 19, 2012
  14. 14. Enter Ironfan We wrote Ironfan to • Simplify, unify, and standardize our usage of the Chef toolsetSaturday, May 19, 2012
  15. 15. Enter Ironfan We wrote Ironfan to • Simplify, unify, and standardize our usage of the Chef toolset • Build further abstractions on top of ChefSaturday, May 19, 2012
  16. 16. Enter Ironfan We wrote Ironfan to • Simplify, unify, and standardize our usage of the Chef toolset • Build further abstractions on top of Chef • Give us superpowers that Chef doesn’t have yetSaturday, May 19, 2012
  17. 17. Enter Ironfan We wrote Ironfan to • Simplify, unify, and standardize our usage of the Chef toolset • Build further abstractions on top of Chef • Give us superpowers that Chef doesn’t have yet http://github.com/infochimps-labs/ironfanSaturday, May 19, 2012
  18. 18. What does Ironfan do?Saturday, May 19, 2012
  19. 19. What does Ironfan do? ChefSaturday, May 19, 2012
  20. 20. What does Ironfan do? Ironfan Simple helpers in the silverware cookbook abstract common Chef patterns and keep things DRY ChefSaturday, May 19, 2012
  21. 21. What does Ironfan do?Saturday, May 19, 2012
  22. 22. What does Ironfan do? Dynamic service discovery:Saturday, May 19, 2012
  23. 23. What does Ironfan do? Dynamic service discovery:Saturday, May 19, 2012
  24. 24. What does Ironfan do?Saturday, May 19, 2012
  25. 25. What does Ironfan do? A simple DSL for defining clusters of machinesSaturday, May 19, 2012
  26. 26. DEMO TIME!Saturday, May 19, 2012
  27. 27. Thanks for coming! Learn more about what Ironfan and Infochimps can sales@infochimps.com do for you. Come work at a fast- growing, cutting-edge careers@infochimps.com technology company based in Austin, TX. http://github.com/infochimps-labs/ironfan http://github.com/infochimps-labs/wukongSaturday, May 19, 2012

×