Large Scale Data Analysis Tools
Upcoming SlideShare
Loading in...5

Large Scale Data Analysis Tools



My talk on Hadoop, Storm, and other big data tools - DevNexus - 3/21/2012

My talk on Hadoop, Storm, and other big data tools - DevNexus - 3/21/2012



Total Views
Views on SlideShare
Embed Views



2 Embeds 14 8 6



Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • 90 slides - coffee\n\nBig Data guy - Data Scientist?\n\nScaling Data helps our customers tackle this new Big Data space - their whole stack\n
  • if you write applications that are JVM-based and you’re not using Metrics, you are doing it wrong\n\ninstrument your running production code to get real intelligence on what’s going on AS your running production code creates business value\n
  • At scaling data, people give us money for crunching data.\n
  • the reason they pay us so much money is that we crunch data that generates business value.\n
  • I thought this was going to be about big data\n
  • topline\n
  • recommendations for other complimentary products, driving overall spend higher\n\ncustomer classification and scoring - offer good customers deals for repeat business\n\ntransactional retargeting - abandoned shopping carts are mined, and personalized ads are returned to that specific user\n
  • cell tower data used to track where people go for lunch - identify a new restaurant site\n\nwhat roads are used so we can target billboards - demand higher prices\n\nmunicipal planning\n
  • cost cutting\n
  • pattern recognition in the power signature can point to imminent failure for expensive equipment\n
  • imagine a diagnosis that was cured with 17 procedures at immense cost\n\nsame diagnosis was cured with 5 procedures elsewhere\n\nanalyzing patient histories across the country / world can get us here\n\n
  • because we like more money... \n
  • \n
  • \n
  • \n
  • \n
  • We have even more types of data,\nbecoming ever more complex,\ndistributed across multiple existences,\nand we are left with the task of parsing out terabytes of noise to get to a megabyte of signal.\n
  • \n
  • Ever more data to try to find the business value\n\nCurrent tools are straining under the load, (banks) my talk last year\n\nThere is significant pain while using these big data tools - Why are they so hot now?\n\ngetting better\n
  • put it on disk in a database\n
  • SAN\n
  • even with the SAN... so you get a bigger machine\n
  • Oracle loves you for this!\n\n37signals approach - basecamp 1 server\n
  • \n
  • EMC loves you for this!\n\n\n
  • \n
  • IBM, HP, Sun loves (or loved) you for this!\n\nmore processors, more memory, more disk\n\n
  • \n
  • mounting costs are not good for...\n
  • the new approach, starting about 5 years ago\n\nNoSQL?\n\nNewSQL?\n
  • \n
  • \n
  • \n
  • so you’re sold on ‘scale out’\n
  • if you want your ops co-workers to be outside of their happy space, this is the ticket\n
  • lots of commodity hardware boxes ... racks\n
  • haproxy is a good one\n
  • things will break - fault tolerance\n\ndistribution of data - rebalancing\n\ntask coordination - leader election / masterless\n
  • reduce ops headache - Chef, Puppet\n
  • I still have the pain... I want to go forward with this\n
  • Cambrian explosion 530 million years ago\n\nappearance of most major animal phyla\n\ndiversification of organisms as earth warms, forms different climates\n
  • small records/files\n\nfixed schema, semi-structured, totally unstructured\n\ncolumn store, graph store\n
  • how will you ask for the data?\n\nkey lookup\n\ntable scan otherwise\n\nsecondary indices for oft-queried fields? mostly roll-your-own\n\n
  • per-request speed - fast = column db\n\namount of requests - availability of reads/writes under load becomes important\n
  • cassandra - read/write speed impressive\n\ndynamo-based clusters\n\nvery capable data stores\n
  • \n
  • hadoop rules the batch world for massive data sets\n
  • \n
  • probably 40-50 satellite projects that are non-core hadoop\n
  • distributions - should be matched to your use-case\n
  • \n
  • data --> business value\n
  • logging only, from Facebook\n\nkind of old and busted\n\nbut still on every Facebook server (or was at one time), so battle-tested\n
  • near-realtime: minutes\n\nreliability: getting better with recent releases\n\nmgmt: complicated\n\nsupport: apache project\n
  • a more general data ingest tool, although it started with log files\n\nnear-realtime: seconds\n\nreliability: best effort, store+retry on failure, and end-to-end mode \nthat uses acks and a write ahead log.\n\nmgmt: master or masters, then smooth from there\n\nsupport: cloudera\n
  • if you have a realtime component, use Storm\n\nit’s already distributed, reliable, easily manageable.\n
  • big files\n\nrecent performance improvements\n\nships with hadoop\n
  • unique for small files\n\nperformance over hdfs\n\nsnapshotting\n
  • low-latency column store\n\nfast key-based access\n\nalso have MR to do in batch/background\n
  • time series schema for hbase\n\nStumbleupon\n
  • a framework for processing in parallel on large clusters\n\nmap - nodes process local data\n\nreduce - reduces the ‘map output’ in some way (sum, count, etc)\n\n(shuffle & sort are in between M & R)\n
  • high-level language built on top of MR\n\noften favored for data movement, but can be used for querying / processing too\n
  • \n
  • \n
  • \n
  • high-level language built on top of MR\n\nstriving for SQL-like language\n
  • \n
  • high-level language built on top of MR\n\nmultiple MR jobs linked together\n\ncomplex query workflows\n
  • querying DSL written in Clojure\n
  • Excel-like frontend tool on top of Hadoop\n\nspreadsheet-like interface targets business users\n\njoins, data ingest too\n\n
  • released with Hadoop 0.23\n\nsplit JobTracker into:\n - ResourceManager (RM)\n - ApplicationMaster (AM), which does job scheduling/monitoring\n\nyou can run different applications now (next slide)\n
  • \n
  • one of the highest levels of ‘gaining insight’\n\nRecommendation\nClassification\nClustering / Segmentation\nPredictive Analytics\nSimilarity\n
  • loose federation of machine learning algorithms that run on hadoop\n\nHadoop not best system for some of these, although MRv2 is now here\n\nsome algos are better than others - you have been warned\n
  • output targets of Hadoop jobs\n
  • I’m not a hater!\n\nGreat tool for 40 years\n\n
  • mongo, redis\n
  • back into the cluster for use in another MR job\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • sexy, complicated algorithms are very insightful\n\nBUT more data and a shittier / more basic algorithm wins\n\ndata can overcome “known truths” and organizational inertia\n\n
  • for your organization, start small\n\ndon’t bet the farm... maybe 10-15% of your analytics budget\n\nskunkworks projects, hackers, etc.\n
  • \n
  • we need more Big Data people!\n
  • \n

Large Scale Data Analysis Tools Large Scale Data Analysis Tools Presentation Transcript

  • Large ScaleData Analysis Tools Brad Anderson @boorad
  • shameless borrowing
  • I crunch data.
  • data
  • data business value
  • What the hell isbusiness value?
  • Business value isanything which makes people more likely to give us money.
  • shopping cart analysis
  • mobile device tracking
  • Business value is anything whichsaves us money.
  • smart gridsubstations
  • healthcare
  • We want to generatemore business value.
  • ever-growing sources of big data
  • web logs
  • mobile devices
  • sensors rfid tagssmart metersocean buoys
  • parsing terabytes of noiseto get a megabyte of signal
  • How did we get here?
  • your data doesn’t fit in local memory
  • your data doesn’t fit on local disk
  • your data doesn’t fit on one machine
  • scale up
  • $
  • SAN
  • $$
  • big db iron
  • $$$
  • business value.
  • scale out
  • move the datato the processors
  • functiondata datadata datadata datadata datadata data
  • functiondata datadata datadata datadata datadata data ship code not data
  • function function data function function data data function functiondata data data datadata data function functiondata data data datadata datadata data function function data data function data ship code not data
  • add more machines
  • shit gets interesting
  • clusters
  • load balancers
  • distributed systems problems opportunities
  • configurationmanagement
  • What systems do I use?
  • data shape
  • query patterns
  • latency and throughput requirements
  • cassandra riakbigcouch
  • batch vs. realtime
  • Hadoop
  • hdfsmapreduce
  • ecosystem
  • Cloudera IBMAmazon EMR MapRHortonworks EMC
  • data ingest storagequerying / processing output
  • processes RDBMS batch Hadoop CacheRaw NoSQL AppsData processes realtime Storm NoSQL
  • data ingestscribe
  • data ingestchukwa
  • data ingestflume
  • data ingesthomegrown?
  • storagehdfs
  • storageMapR
  • storagehbase
  • storageopentsdb
  • querying / processingmapreduce
  • querying / processingpig
  • querying / processing
  • querying / processingExample Pig Script
  • Equivalent MR Java code
  • querying / processinghive
  • querying / processing Example Hive QueryFROM pv_usersINSERT OVERWRITE TABLE pv_gender_sumSELECT pv_users.gender, count(DISTINCT pv_users.userid)GROUP BY pv_users.genderINSERT OVERWRITE DIRECTORY /user/facebook/tmp/pv_age_sumSELECT pv_users.age, count(DISTINCT pv_users.userid)GROUP BY pv_users.age;
  • querying / processingcascading
  • querying / processingcascalog
  • querying / processingDatameer
  • querying / processingMRv2
  • querying / processing MRv2 allows MRv1 (of course) SparkBulk Synchronous Parallel Graphs MPI
  • querying / processingmachine learning algorithms
  • querying / processingmahout
  • outputflat files
  • outputrdbms
  • outputcache
  • outputhdfs
  • realtime
  • Storm
  • streamsTuple Tuple Tuple Tuple Tuple Tuple Tuple Unbounded sequence of tuples
  • spoutsSource of streams
  • spout examples•Read from Kestrel queue• Read from Twitter streaming API
  • boltsProcesses input streams and produces new streams
  • bolts• Functions• Filters• Aggregation• Joins• Talk to databases
  • topologiesNetwork of spouts and bolts
  • data business value
  • The UnreasonableEffectiveness of Data
  • Start small
  • But definitely start!
  • Please start!
  • Thank you.