Ironfan: Principle-Driven Scalable Ops Tools

939 views
854 views

Published on

Ironfan is the foundation for your Big Data stack, making provisioning and configuring your Big Data infrastructure simple. Spin up clusters when you need them, kill them when you don't, so you can spend your time, money, and engineering focus on finding insights, not getting your machines ready.

Learn more at http://infochimps.com

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
939
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
13
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Ironfan: Principle-Driven Scalable Ops Tools

  1. 1. Ironfan:Principle-DrivenScalable Ops Tools http://github.com/infochimps-labs/ironfan
  2. 2. Ironfan• Open-source core (Apache) with enterprise features• http://github.com/infochimps-labs/ironfan• Based on Chef (http://www.opscode.com/chef)
  3. 3. Principles• Progressive - match what you’d whiteboard• Declaratory - show all (and only) what’s interesting• Coherent - full state is versioned as a whole• Decoupled - minimize shared knowledge• Helpful - common things are simple• Inevitable - declarations imply actions
  4. 4. Ironfan:your system diagram made real
  5. 5. Declaratory:simple stories, simply told
  6. 6. HBase Cluster• HBase Master• HBase RegionServers Simple, right?
  7. 7. HBase Cluster• one-ish: • HBase master • HBase secondary master • Hadoop Namenode • many: • Hadoop 2NN • HBase RegionServer • Hadoop Jobtracker • HBase Stargate • Zookeeper Quorum x3 • HBase Thrift • Hadoop Datanode • Hadoop Tasktracker
  8. 8. DecouplePowerful Black Boxes Beautiful Glue
  9. 9. Data Stores in Production• HBase • MySQL• ElasticSearch • Redis• Cassandra • sqlite• TokyoTyrant • whisper (graphite)• SimpleDB • file system• MongoDB • S3
  10. 10. Programs Used for This Talk• Emacs • Skitch• Keynote • finder• Preview • flickr.com• Chrome • google image search• ruby (pry) • ssh
  11. 11. A Database is just a Bucket
  12. 12. Service Discovery:game recognize game
  13. 13. gibbon-hadoop-namenode gibbon-hadoop-jobtracker ack ack ackgibbon-nnjt-0 ¿dónde está mi namenode? gibbon-nnjt-0 gibbon- worker-0
  14. 14. gibbon-hadoop-namenode gibbon-hadoop-jobtracker ack ack ackgibbon-nnjt-0 ¿dónde está mi namenode? gibbon-nnjt-0 ¿dónde está mi jobtracker? gibbon-nnjt-0 gibbon- worker-0
  15. 15. gibbon-hadoop-namenode gibbon-hadoop-jobtracker ack ack ackgibbon-nn-0 gibbon-jt-0 ¿dónde está mi namenode? gibbon-nn-0 ¿dónde está mi jobtracker? gibbon-jt-0 gibbon- worker-0
  16. 16. declaratory + decoupled = Recapitulation ex: DB Migrations the Ironfan Way
  17. 17. alicealicebobalicebobbob
  18. 18. Aspects:Cross-cutting Concerns
  19. 19. Amenities• Logs• Ports• Daemons• Dashboards• Libs / Executables
  20. 20. Amenities:It’s good to be king
  21. 21. Amenities• Logs: rotation, archiving• Ports: remote monitoring (on and off)• Daemons: process monitoring• Dashboards: Dashboard dashboard• Libs / Executables: Decoupled imports• Versions: Shims, Discovery
  22. 22. Inevitability:born this way
  23. 23. Chef:Ironfan:
  24. 24. 10 lines
  25. 25. 10 lines, 5 flawscan you spot all 5?
  26. 26. 10 lines, 5 flaws hard coded SECURITY HOLE daemon-writeable conf dir fails on bsdnot DRY ‘0755’ not 755
  27. 27. Ironfan
  28. 28. Inevitabilitymust do this: to get a log dir so when you announce, gibbon-hadoop-namenode AWESOMENESS IS INEVITABLE
  29. 29. Contract:word is bond yo
  30. 30. Resource Sharing:can’t we all just get along?
  31. 31. HBase• Region Server (dominate)• TaskTracker + Children (wussy)• DataNode (moderate) Hadoop• DataNode (moderate)• TaskTracker + Children (dominate)
  32. 32. Rails Apps > 5k LOC considered Harmfulone big app many apps tax
  33. 33. Resource Sharing:can’t we all just get along?
  34. 34. Monolithic Software means Meetings
  35. 35. Meetingsare Death
  36. 36. Decentralize. Decouple.
  37. 37. It’s not about Performance Fast Resilient Simple
  38. 38. jobs@infochimps.comWe’re Hiring github.com/ infochimps-labs
  39. 39. ...
  40. 40. • System Diagram made real• Decoupled Components connect• Components Announce their capabilities• Announcements enable Service Discovery• Components announce cross-cutting Aspects• Aspects enable zero-conf Amenities• Announcement as Contract• Contracts enable zero-conf specification testing• Specs + monitoring enable zero-conf integration testing• Systems Bind to provisioned resources• Resource Sharing

×