Downloaded 142 times
More Related Content
Similar to Large Scale Data Analysis Tools
Large Scale Data Analysis Tools
- 1. Large Scale Data AnalysisTools Brad Anderson brad@scalingdata.com @boorad
- 2.
- 3.
- 4.
- 5. data business value
- 6. What the hellis business value?
- 7. Business value is anythingwhich makes people more likely to give us money.
- 8. shopping cart analysis
- 9. mobile device tracking
- 10. Business value is anything which saves us money.
- 11.
- 12.
- 13. We want togenerate more business value.
- 14. ever-growing sources of big data
- 15.
- 16.
- 17. sensors rfidtags smart meters ocean buoys
- 18. parsing terabytes ofnoise to get a megabyte of signal http://www.kaushik.net/avinash/big-data-imperative-driving-big-action/
- 20. How did weget here?
- 21. your data doesn’tfit in local memory
- 22. your data doesn’tfit on local disk
- 23. your data doesn’tfit on one machine
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32. move the data tothe processors
- 33. function data data data data data data data data data data
- 34. function data data data data data data data data data data ship code not data
- 35. function function data function function data data function function data data data data data data function function data data data data data data data data function function data data function data ship code not data
- 36.
- 37.
- 38.
- 39.
- 40. distributed systems problems opportunities
- 41.
- 42. What systems doI use?
- 44.
- 45.
- 46. latency and throughput requirements
- 47. cassandra riak bigcouch
- 48.
- 49.
- 50.
- 51.
- 52. Cloudera IBM Amazon EMR MapR Hortonworks EMC
- 53. data ingest storage querying / processing output
- 54. processes RDBMS batch Hadoop Cache Raw NoSQL Apps Data processes realtime Storm NoSQL
- 55.
- 56.
- 57.
- 58.
- 59.
- 60.
- 61.
- 62.
- 63.
- 64.
- 65.
- 66. querying / processing ExamplePig Script
- 67.
- 68.
- 69. querying / processing Example Hive Query FROM pv_users INSERT OVERWRITE TABLE pv_gender_sum SELECT pv_users.gender, count(DISTINCT pv_users.userid) GROUP BY pv_users.gender INSERT OVERWRITE DIRECTORY '/user/facebook/tmp/pv_age_sum' SELECT pv_users.age, count(DISTINCT pv_users.userid) GROUP BY pv_users.age;
- 70.
- 71.
- 72.
- 73.
- 74. querying / processing MRv2 allows MRv1 (of course) Spark Bulk Synchronous Parallel Graphs MPI
- 75. querying / processing machinelearning algorithms
- 76.
- 77.
- 78.
- 79.
- 80.
- 81.
- 82.
- 83. streams Tuple Tuple Tuple Tuple Tuple Tuple Tuple Unbounded sequence of tuples
- 84.
- 85. spout examples •Read fromKestrel queue • Read from Twitter streaming API
- 86. bolts Processes input streamsand produces new streams
- 87. bolts • Functions • Filters •Aggregation • Joins • Talk to databases
- 88. topologies Network of spoutsand bolts
- 89. data business value
- 90. The Unreasonable Effectiveness ofData http://bit.ly/x407Ln
- 91.
- 92.
- 93.
- 94.
Editor's Notes
- #2 90 slides - coffee\n\nBig Data guy - Data Scientist?\n\nScaling Data helps our customers tackle this new Big Data space - their whole stack\n
- #3 if you write applications that are JVM-based and you’re not using Metrics, you are doing it wrong\n\ninstrument your running production code to get real intelligence on what’s going on AS your running production code creates business value\n
- #4 At scaling data, people give us money for crunching data.\n
- #5 the reason they pay us so much money is that we crunch data that generates business value.\n
- #6 I thought this was going to be about big data\n
- #7 topline\n
- #8 recommendations for other complimentary products, driving overall spend higher\n\ncustomer classification and scoring - offer good customers deals for repeat business\n\ntransactional retargeting - abandoned shopping carts are mined, and personalized ads are returned to that specific user\n
- #9 cell tower data used to track where people go for lunch - identify a new restaurant site\n\nwhat roads are used so we can target billboards - demand higher prices\n\nmunicipal planning\n
- #10 cost cutting\n
- #11 pattern recognition in the power signature can point to imminent failure for expensive equipment\n
- #12 imagine a diagnosis that was cured with 17 procedures at immense cost\n\nsame diagnosis was cured with 5 procedures elsewhere\n\nanalyzing patient histories across the country / world can get us here\n\n
- #13 because we like more money... \n
- #14 \n
- #15 \n
- #16 \n
- #17 \n
- #18 We have even more types of data,\nbecoming ever more complex,\ndistributed across multiple existences,\nand we are left with the task of parsing out terabytes of noise to get to a megabyte of signal.\n
- #19 \n
- #20 Ever more data to try to find the business value\n\nCurrent tools are straining under the load, (banks) my talk last year\n\nThere is significant pain while using these big data tools - Why are they so hot now?\n\ngetting better\n
- #21 put it on disk in a database\n
- #22 SAN\n
- #23 even with the SAN... so you get a bigger machine\n
- #24 Oracle loves you for this!\n\n37signals approach - basecamp 1 server\n
- #25 \n
- #26 EMC loves you for this!\n\n\n
- #27 \n
- #28 IBM, HP, Sun loves (or loved) you for this!\n\nmore processors, more memory, more disk\n\n
- #29 \n
- #30 mounting costs are not good for...\n
- #31 the new approach, starting about 5 years ago\n\nNoSQL?\n\nNewSQL?\n
- #32 \n
- #33 \n
- #34 \n
- #35 so you’re sold on ‘scale out’\n
- #36 if you want your ops co-workers to be outside of their happy space, this is the ticket\n
- #37 lots of commodity hardware boxes ... racks\n
- #38 haproxy is a good one\n
- #39 things will break - fault tolerance\n\ndistribution of data - rebalancing\n\ntask coordination - leader election / masterless\n
- #40 reduce ops headache - Chef, Puppet\n
- #41 I still have the pain... I want to go forward with this\n
- #42 Cambrian explosion 530 million years ago\n\nappearance of most major animal phyla\n\ndiversification of organisms as earth warms, forms different climates\n
- #43 small records/files\n\nfixed schema, semi-structured, totally unstructured\n\ncolumn store, graph store\n
- #44 how will you ask for the data?\n\nkey lookup\n\ntable scan otherwise\n\nsecondary indices for oft-queried fields? mostly roll-your-own\n\n
- #45 per-request speed - fast = column db\n\namount of requests - availability of reads/writes under load becomes important\n
- #46 cassandra - read/write speed impressive\n\ndynamo-based clusters\n\nvery capable data stores\n
- #47 \n
- #48 hadoop rules the batch world for massive data sets\n
- #49 \n
- #50 probably 40-50 satellite projects that are non-core hadoop\n
- #51 distributions - should be matched to your use-case\n
- #52 \n
- #53 data --> business value\n
- #54 logging only, from Facebook\n\nkind of old and busted\n\nbut still on every Facebook server (or was at one time), so battle-tested\n
- #55 near-realtime: minutes\n\nreliability: getting better with recent releases\n\nmgmt: complicated\n\nsupport: apache project\n
- #56 a more general data ingest tool, although it started with log files\n\nnear-realtime: seconds\n\nreliability: best effort, store+retry on failure, and end-to-end mode \nthat uses acks and a write ahead log.\n\nmgmt: master or masters, then smooth from there\n\nsupport: cloudera\n
- #57 if you have a realtime component, use Storm\n\nit’s already distributed, reliable, easily manageable.\n
- #58 big files\n\nrecent performance improvements\n\nships with hadoop\n
- #59 unique for small files\n\nperformance over hdfs\n\nsnapshotting\n
- #60 low-latency column store\n\nfast key-based access\n\nalso have MR to do in batch/background\n
- #61 time series schema for hbase\n\nStumbleupon\n
- #62 a framework for processing in parallel on large clusters\n\nmap - nodes process local data\n\nreduce - reduces the ‘map output’ in some way (sum, count, etc)\n\n(shuffle & sort are in between M & R)\n
- #63 high-level language built on top of MR\n\noften favored for data movement, but can be used for querying / processing too\n
- #64 \n
- #65 \n
- #66 \n
- #67 high-level language built on top of MR\n\nstriving for SQL-like language\n
- #68 \n
- #69 high-level language built on top of MR\n\nmultiple MR jobs linked together\n\ncomplex query workflows\n
- #70 querying DSL written in Clojure\n
- #71 Excel-like frontend tool on top of Hadoop\n\nspreadsheet-like interface targets business users\n\njoins, data ingest too\n\n
- #72 released with Hadoop 0.23\n\nsplit JobTracker into:\n - ResourceManager (RM)\n - ApplicationMaster (AM), which does job scheduling/monitoring\n\nyou can run different applications now (next slide)\n
- #73 \n
- #74 one of the highest levels of ‘gaining insight’\n\nRecommendation\nClassification\nClustering / Segmentation\nPredictive Analytics\nSimilarity\n
- #75 loose federation of machine learning algorithms that run on hadoop\n\nHadoop not best system for some of these, although MRv2 is now here\n\nsome algos are better than others - you have been warned\n
- #76 output targets of Hadoop jobs\n
- #77 I’m not a hater!\n\nGreat tool for 40 years\n\n
- #78 mongo, redis\n
- #79 back into the cluster for use in another MR job\n
- #80 \n
- #81 \n
- #82 \n
- #83 \n
- #84 \n
- #85 \n
- #86 \n
- #87 \n
- #88 \n
- #89 sexy, complicated algorithms are very insightful\n\nBUT more data and a shittier / more basic algorithm wins\n\ndata can overcome “known truths” and organizational inertia\n\n
- #90 for your organization, start small\n\ndon’t bet the farm... maybe 10-15% of your analytics budget\n\nskunkworks projects, hackers, etc.\n
- #91 \n
- #92 we need more Big Data people!\n
- #93 \n