Ironfan:  Principle-Driven Scalable Ops Tools
 

Ironfan: Principle-Driven Scalable Ops Tools

on

  • 1,079 views

Ironfan is the foundation for your Big Data stack, making provisioning and configuring your Big Data infrastructure simple. Spin up clusters when you need them, kill them when you don't, so you can ...

Ironfan is the foundation for your Big Data stack, making provisioning and configuring your Big Data infrastructure simple. Spin up clusters when you need them, kill them when you don't, so you can spend your time, money, and engineering focus on finding insights, not getting your machines ready.

Learn more at http://infochimps.com

Statistics

Views

Total Views
1,079
Views on SlideShare
1,075
Embed Views
4

Actions

Likes
4
Downloads
12
Comments
0

1 Embed 4

http://www.linkedin.com 4

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Ironfan:  Principle-Driven Scalable Ops Tools Ironfan: Principle-Driven Scalable Ops Tools Presentation Transcript

  • Ironfan:Principle-DrivenScalable Ops Tools http://github.com/infochimps-labs/ironfan
  • Ironfan• Open-source core (Apache) with enterprise features• http://github.com/infochimps-labs/ironfan• Based on Chef (http://www.opscode.com/chef)
  • Principles• Progressive - match what you’d whiteboard• Declaratory - show all (and only) what’s interesting• Coherent - full state is versioned as a whole• Decoupled - minimize shared knowledge• Helpful - common things are simple• Inevitable - declarations imply actions
  • Ironfan:your system diagram made real
  • Declaratory:simple stories, simply told
  • HBase Cluster• HBase Master• HBase RegionServers Simple, right?
  • HBase Cluster• one-ish: • HBase master • HBase secondary master • Hadoop Namenode • many: • Hadoop 2NN • HBase RegionServer • Hadoop Jobtracker • HBase Stargate • Zookeeper Quorum x3 • HBase Thrift • Hadoop Datanode • Hadoop Tasktracker
  • DecouplePowerful Black Boxes Beautiful Glue
  • Data Stores in Production• HBase • MySQL• ElasticSearch • Redis• Cassandra • sqlite• TokyoTyrant • whisper (graphite)• SimpleDB • file system• MongoDB • S3
  • Programs Used for This Talk• Emacs • Skitch• Keynote • finder• Preview • flickr.com• Chrome • google image search• ruby (pry) • ssh
  • A Database is just a Bucket
  • Service Discovery:game recognize game
  • gibbon-hadoop-namenode gibbon-hadoop-jobtracker ack ack ackgibbon-nnjt-0 ¿dónde está mi namenode? gibbon-nnjt-0 gibbon- worker-0
  • gibbon-hadoop-namenode gibbon-hadoop-jobtracker ack ack ackgibbon-nnjt-0 ¿dónde está mi namenode? gibbon-nnjt-0 ¿dónde está mi jobtracker? gibbon-nnjt-0 gibbon- worker-0
  • gibbon-hadoop-namenode gibbon-hadoop-jobtracker ack ack ackgibbon-nn-0 gibbon-jt-0 ¿dónde está mi namenode? gibbon-nn-0 ¿dónde está mi jobtracker? gibbon-jt-0 gibbon- worker-0
  • declaratory + decoupled = Recapitulation ex: DB Migrations the Ironfan Way
  • alicealicebobalicebobbob
  • Aspects:Cross-cutting Concerns
  • Amenities• Logs• Ports• Daemons• Dashboards• Libs / Executables
  • Amenities:It’s good to be king
  • Amenities• Logs: rotation, archiving• Ports: remote monitoring (on and off)• Daemons: process monitoring• Dashboards: Dashboard dashboard• Libs / Executables: Decoupled imports• Versions: Shims, Discovery
  • Inevitability:born this way
  • Chef:Ironfan:
  • 10 lines
  • 10 lines, 5 flawscan you spot all 5?
  • 10 lines, 5 flaws hard coded SECURITY HOLE daemon-writeable conf dir fails on bsdnot DRY ‘0755’ not 755
  • Ironfan
  • Inevitabilitymust do this: to get a log dir so when you announce, gibbon-hadoop-namenode AWESOMENESS IS INEVITABLE
  • Contract:word is bond yo
  • Resource Sharing:can’t we all just get along?
  • HBase• Region Server (dominate)• TaskTracker + Children (wussy)• DataNode (moderate) Hadoop• DataNode (moderate)• TaskTracker + Children (dominate)
  • Rails Apps > 5k LOC considered Harmfulone big app many apps tax
  • Resource Sharing:can’t we all just get along?
  • Monolithic Software means Meetings
  • Meetingsare Death
  • Decentralize. Decouple.
  • It’s not about Performance Fast Resilient Simple
  • jobs@infochimps.comWe’re Hiring github.com/ infochimps-labs
  • ...
  • • System Diagram made real• Decoupled Components connect• Components Announce their capabilities• Announcements enable Service Discovery• Components announce cross-cutting Aspects• Aspects enable zero-conf Amenities• Announcement as Contract• Contracts enable zero-conf specification testing• Specs + monitoring enable zero-conf integration testing• Systems Bind to provisioned resources• Resource Sharing