The Other Way of Doing Big Data

How to Win at Scale and itsInﬂuence on People Philip (ﬂip) Kromer CTO, Infochimps.com

Big Data is InevitableIt Demands a New Approach

There’s Another Way

There’s Another WayYou’re Going to Have to follow It

There’s Another WayYou’re Going to Have to follow ItIt Might be a Better Way

The Other Way

Massive component count

Federated Truth email MySQL HBase s3spreadsheets elasticsearch elasticsearch HDFS hipchat redis mongo MongoDB log ﬁles salesforce zabbix hubspot ADP Chargify BC/BS ZenDesk google docs

Low Coupling

Reliable Resilient

• Manage 100s of machines: architecture as code• Contain system complexity: relentlessly decouple• Maintain coherency: federated truth• Manage true costs: optimize for people not machines• Manage failure & change:resiliency engineering

The Other WayDeclarative, not HomogenousDecoupled, not Standardized Federated, not Centralized Simple, not Performant Resilient, not Reliable

Declarative

Architecture as Code Lightweight Lightweight Dashboard Dashboard HBase HBase API Data Transport ESh ﬂume ElasticSearch ElasticSearch Operations ApplicationIronfan + ops ics.com Hadoop On-Demand Hadoop Chef

Lightweight Lightweight Dashboard Dashboard HBase HBase APIData Transport ESh ﬂume ElasticSearch ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop HM NN ZK RS RS RS RS RS RS

provision machinerun statesettingsstandard componentscluster-speciﬁcfacet groups

Lightweight Dashboard Lightweight Dashboard HBase HBase HM NN ZK APIData Transport ESh ﬂume ElasticSearch RS RS ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop RS RS RS RS regionserver ssh nfs datanode zbx stargate log tasktracker fw zookeeper

Winsfrom Declarative Lightweight Lightweight Dashboard Dashboard HBase HBase API Data Transport ESh ﬂume ElasticSearch ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop

Recapitulatable

Portable

Decoupled

Our Stack Lightweight Lightweight Dashboard Dashboard HBase HBase APIData Transport ESh ﬂume ElasticSearch ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop

Our Stack

Engineer : System = 1:10• >60 distinct components• 50-150 machines• 1 ops + 5 hackers + 1 analyst

Self-similar Lightweight Lightweight Dashboard Dashboard HBase HBase APIData Transport ESh ﬂume ElasticSearch ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop HM NN ZK RS RS ssh ssh hb 2d mstr hb master nfs nfs RS RS namenode zbx 2d nn zbx log jobtracker log zookeeper RS RS fw zookeeper fw alpha beta regionserver ssh regionserver ssh nfs nfs datanode datanode zbx zbx stargate log stargate log tasktracker fw tasktracker fw zookeeper gamma delta

Example: ScraperScraper disk tail’er decorator sink Jobs database

Scraper ﬂumeScraper disk tail’er decorator sink Jobs database while true: get_job fetch_url dump_to_disk

Scraper ﬂumeScraper disk tail’er decorator sink Jobs database while true: ensures get_job reliable fetch_url delivery dump_to_disk

Scraper ﬂumeScraper disk tail’er decorator sink Jobs database while true: ensures parse get_job reliable raw fetch_url delivery => dump_to_disk objects

Scraper ﬂumeScraper disk tail’er decorator sink Jobs database while true: ensures parse store get_job reliable raw object fetch_url delivery => => dump_to_disk objects database

alicealicebobalicebobbob

Simple

• Immediately Understandable• Clear Interface• Few Moving Parts

Federated

Data Stores in Production• HBase • MySQL• ElasticSearch • Redis• Cassandra • sqlite• TokyoTyrant • whisper (graphite)• SimpleDB • ﬁle system• MongoDB • S3

Programs Used for This Talk• Emacs • Skitch• Keynote • ﬁnder• Preview • ﬂickr.com• Chrome • google image search• ruby (pry) • ssh

How’s my Batch Job Going?• 1 x Job Status• 1 x Counters & App Metrics• N x Task Status• M x Machine System Stats• 1 x Cloud Status• 1 x Chef Server

Dataﬂow is All

Lightweight Lightweight Dashboard Dashboard HBase HBase APIData Transport ESh flume ElasticSearch ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop System Diagram Dataflow Workflow

Lightweight Lightweight Dashboard Dashboard HBase HBase APIData Transport ESh flume ElasticSearch ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop System Diagram Dataflow Workflow Org Chart

Robots are CheapPeople are Important

Expensive / Not Expensive1 trillion 10 kb objects: • 100 % in RAM: $ 212,000 /mo • 10% in Ram: $ 21,000 /mo • On Disk: $ 3,000 /mo • On S3: $ 1,200 /mo

Expensive / Not Expensive1 trillion 10 kb objects: • 100 % in RAM: $ 212,000 /mo • 10% in Ram: $ 21,000 /mo • On Disk: $ 3,000 /mo • On S3: $ 1,200 /mo1 Intern, part-time: $ 1,500 /mo

Scalability is People

Monolithic Software means Meetings

Meetingsare Death

Decentralize. Decouple.

n^2 law of coupling100 things 5 + 3 + 2 things + 2 (tax)

n^2 law of coupling 2500 + 900 + 400 + 400 =10,000 things 4200 thingsto go wrong to go wrong

Infochimps.com 2011 text search Planet of the API acctg APIs infochimps.com models A/B testing cloud services

Infochimps.com 2012 datasets catalog API API docs text search content dashboards Planet of the API acctg APIs auth & payment layout console models A/B testing blog press cloud services collateral

Infochimps.com 2012 (infochimps) icsexpl catalog API (saas) capuchin elasticsrch kanzi beergoggls Planet of the MongoDB APIs george george alphamale MySQL redis WPEngine totem cloud services hubspot

this drawing ﬁts in my head datasets catalog API this app ﬁts in my head, and my laptop

Infochimps.com 2012 (infochimps) icsexpl catalog API (saas) capuchin elasticsrch kanzi beergoggls Planet of the MongoDB APIs george george alphamale MySQL redis WPEngine totem cloud services hubspot

ﬁn. http://infochimps.comhttp://github.com/infochimps-labs

The Other Way of Doing Big Data

by Infochimps, Inc. on Jun 13, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

The Other Way of Doing Big Data Presentation Transcript