Uploaded bycwensel
KEY, PDF1,237 views

Cascading and BigData Problems

The document discusses several case studies of companies using Hadoop and MapReduce for large-scale data processing problems. Common themes across the case studies include using Cascading and AWS Elastic MapReduce to develop multi-stage ETL pipelines to extract, transform and analyze large amounts of daily log and user behavior data. The case studies demonstrate how Hadoop can be used to gain business insights through modeling, machine learning, and predictive analytics on petabytes of user data.

Embed presentation

Downloaded 99 times
Cascading and BigData Problems Chris K Wensel Concurrent, Inc. Copyright Concurrent, Inc. 2011. All rights reserved.
About Me • Concurrent, Inc., Founder • Cascading support and tools • http://concurrentinc.com/ • Cascading, Lead Developer (started Sept 2007) • An alternative API to MapReduce • http://cascading.org/ • Formerly Hadoop mentoring and training • Sun - Apple - HP - LexisNexis - startups - etc • Formerly Systems Architect & Consultant • Thomson/Reuters - TeleAtlas - startups - etc Copyright Concurrent, Inc. 2011. All rights reserved.
Overview • Case Studies • What’s in common? • Where does Hadoop fit? • Processing vs Innovation Copyright Concurrent, Inc. 2011. All rights reserved.
Case Studies • ShareThis • BestBuy • FlightCaster • Etsy • Ion Flux Copyright Concurrent, Inc. 2011. All rights reserved.
Summary • All running in production with Hadoop • All use AWS, most use Elastic MapReduce • All production processing was implemented in Cascading • Various other tools used at different stages of development Copyright Concurrent, Inc. 2011. All rights reserved.
Share This • Cascading + AWS (pre-EMR) • Daily event log processing, initially multiple TB and growing • Details in the O’Reilly Hadoop book from Tom White Copyright Concurrent, Inc. 2011. All rights reserved.
Lessons every Y hrs on crawl completion every X hrs logprocessor crawler indexer ... ... • Mark data as bad and why, never discard • useful for upstream debugging • Data is seasonal, cyclical, and bursty • Tune your app and cluster to the workload • (garbage collect Hadoop clusters) Copyright Concurrent, Inc. 2011. All rights reserved.
BestBuy - Behavioral Ad- Targeting • Cascading + AWS (Elastic MapReduce) • Daily automated User Behavior Segmentation • 6wks dev, 3T/day, $13k/mo • 500% increase in return on ad spend from a similar campaign a year before • http://aws.amazon.com/solutions/case-studies/ razorfish/ Copyright Concurrent, Inc. 2011. All rights reserved.
Cluster Amazon Web Services Elastic MapReduce Slaves Ad System Map/Reduce behavior app HDFS input output S3 E-Commerce Site • 200+ nodes, 9-12 hour runs • 30+ days of history + 3TB daily • Remote HTTP update of ad-server • of only changed data Copyright Concurrent, Inc. 2011. All rights reserved.
Road Blocks • No one really understood the data • Character formats (UTF-8 vs ...) • Zero byte chars • Unique columns not unique • Outliers in the data • Creating test data • QAing the data • result data was also big Copyright Concurrent, Inc. 2011. All rights reserved.
FlightCaster - Predicting Flight Delays • Clojure + Cascading + AWS • Scours data on every domestic flight for the past 10-years and matches it to real- time conditions • Machine learning on Cascading, Scoring on app server • 3mos dev, 10G day, <$2k/mos Copyright Concurrent, Inc. 2011. All rights reserved.
Lessons • Even with a good abstraction, you must intuit the underlying model (MapReduce) to improve throughput • i.e. Logical vs Physical plans • we still need DBAs after decades of query planner dev Copyright Concurrent, Inc. 2011. All rights reserved.
Etsy - Online Marketplace • JRuby + Cascading + AWS • 1B page-views & multi-T data/mo, of logs • 40-50 cascading.jruby jobs a night • http://codeascraft.etsy.com/2010/02/24/ analyzing-etsys-data-with-hadoop-and-cascading/ • http://www.concurrentinc.com/casestudies/etsy Copyright Concurrent, Inc. 2011. All rights reserved.
Initially • JRuby for the ‘analysts’ • Log pre-processing, • db snap shot diffs, • nightly and ad-hoc analytics Copyright Concurrent, Inc. 2011. All rights reserved.
Data Driven Products • Search index/scoring (under dev) • Taste Test • Facebook gift recommender • Suggested shops • Top query list, etc... • Many more on the way Copyright Concurrent, Inc. 2011. All rights reserved.
Ion Flux - Gene Sequencing • Cascading + AWS • Sequence Alignment • http://aws.amazon.com/solutions/case- studies/ion-flux/ Copyright Concurrent, Inc. 2011. All rights reserved.
Cluster • 10-30 nodes, using new HPC instances • 200-500 cores, • runs up to 50 hours Copyright Concurrent, Inc. 2011. All rights reserved.
Architecture Delivery Ion Flux - Annotation Server Clinical Lab Ion Torrent - Torrent Sequencer Ion Torrent - Torrent Server (EC2) Ion Flux - Pipeline Controller FastQ Ion Torrent RAW Data FTP FTP RAW Data FTP Upload Annotation Sample Prep Chip Measure DNA File File Basecalling Sequence Wait Start Pipeline Third Party Clients Client Server File Server Complete? Database (RDS) Client App Annotation ReST Server Ion Flux - Flux Capacitor Ion Flux - LIMS FastQ FastQ Compressed DNA Sample FTP Transfer Cloud LIMS ReST Chip LIMS Sequence Split File Sequence Compress Sequence Transfer Agent Client Agent Input Server (EC2) Metadata Database File Chunk File (RDS) Ion Flux - Variant Server (EC2) Variants ReST Variant Server Database Ion Flux - (RDS) Client Website External Variant Complete Partners Report Runs (EC2) AWS - S3 Storage FastQ Software & Performance PILEUP Sequence Data Data Variants Chunks Heavy Lifting Ion Flux - Sequencing Pipeline AWS - EMR Cluster SAM Corrected Bootstrap Cluster Configure SAM Sort by Sorted SAM Split to PILEUP TMAP Alignment SRMA SAM PILEUP Nodes Pipeline Alignments position Alignments Bins Variants Bins Alignments Create Cluster Shutdown Cascading Cluster Cleanup Cluster Start Node Performance Profiler Data Copyright Concurrent, Inc. 2011. All rights reserved.
Common Architecture intermediate data raw data loggers ? valuable loggers data loggers Analyst Producer Consumer Developer Value • New data continuously arriving • Actively incorporating the new with the old • Updating backend systems Copyright Concurrent, Inc. 2011. All rights reserved.
Common Constraints • Speed of light • Understanding the data • Creating tests and validating the results • Lifecycle phases have different environments • dev vs. integration vs. prod • Better algorithms, less cost, more complexity Copyright Concurrent, Inc. 2011. All rights reserved.
Apps Have Many Stages • Heavy Lifting • Scoring • Modeling & • Processing Learning Copyright Concurrent, Inc. 2011. All rights reserved.
Heavy Lifting • ETL Style processes hampered by physics • Moving/Transferring/Packaging data • Data cleansing and value normalization Copyright Concurrent, Inc. 2011. All rights reserved.
Modeling & Learning • Also known as “Data Mining” • Ask lots of questions to understand the data • Machine learning, or • Ad-hoc queries • Where the innovation happens Copyright Concurrent, Inc. 2011. All rights reserved.
Processing • Transforming and/or combining multiple data sets into new data sets or models • Analytics, • indexing (w/ • statistics, scoring), • enrichment, • feature reduction, • entity extraction, • matching Copyright Concurrent, Inc. 2011. All rights reserved.
Scoring • Apply what’s learned • Sometimes batch (as part of Processing) • indices with search result ranking • Sometimes transactional, req/resp • prediction, recommendations, etc Copyright Concurrent, Inc. 2011. All rights reserved.
In Summary collection cleansing processing delivery event data signal info knowledge normalization scoring mining The point of computing systems is to make data more valuable Copyright Concurrent, Inc. 2011. All rights reserved.
Where does Hadoop fit? Copyright Concurrent, Inc. 2011. All rights reserved.
Hadoop Cluster Rack Rack Rack Node Node Node Node ... Global Compute-space Global Namespace • Distributed replicated storage for large files • Distributed fault tolerant exec of batch processes • Scale out vs (legacy) scale up • Java API allows complex analysis, more freedom Copyright Concurrent, Inc. 2011. All rights reserved.
MapReduce • A “divide and conquer” strategy for parallelizing workloads against collections of data • Map & Reduce are two user defined functions chained via Key Value Pairs • It’s really Map->Group->Reduce where Group is built in Copyright Concurrent, Inc. 2011. All rights reserved.
Keys and Values • Map translates input to keys and values to new keys and values [K1,V1] Map [K2,V2]* • System Groups each unique [K2,V2] Group [K2,{V2,V2,....}] key with all its values [K2,{V2,V2,....}] Reduce [K3,V3]* • Reduce translates the values of each unique key to new keys and values * = zero or more Copyright Concurrent, Inc. 2011. All rights reserved.
Word Count Mapper [0, "when in the course of human events"] Map ["when",1] ["in",1] ["the",1] [...,1] ["when",1] ["when",1] ["when",1] ["when",1] Group ["when",{1,1,1,1,1}] ["when",1] Reducer ["when",{1,1,1,1,1}] Reduce ["when",5] Copyright Concurrent, Inc. 2011. All rights reserved.
Divide and Conquer Parallelism • Since the ‘records’ entering the Map and ‘groups’ entering the Reduce are independent • That is, there is no expectation of order or requirement to share state between records/ groups • Arbitrary numbers of Map and Reduce function instances can be created against arbitrary portions of input data Copyright Concurrent, Inc. 2011. All rights reserved.
Cluster Cluster Rack Rack Rack Node Node Node Node ... map map map map map reduce reduce reduce • Multiple instances of each Map and Reduce function are distributed throughout the cluster Copyright Concurrent, Inc. 2011. All rights reserved.
Another View [K1,V1] Map [K2,V2] Combine Group [K2,{V2,...}] Reduce [K3,V3] Mapper Task same code Mapper Reducer Shuffle Task Task Mapper Reducer Shuffle Task Task Mapper Reducer Shuffle Task Task Mapper Task Mappers must complete before Reducers can begin split1 split2 split3 split4 ... part-00000 part-00001 part-000N file directory Copyright Concurrent, Inc. 2011. All rights reserved.
Architectural Components NameNode DataNode DataNode DataNode DataNode data block ns read/write operations Secondary ns operations read/write ns operations read/write mapper mapper child jvm mapper child jvm jobs tasks child jvm Client JobTracker TaskTracker reducer reducer child jvm reducer child jvm child jvm • Solid boxes are unique applications • Dashed boxes are child JVM instances on same node as parent • Dotted boxes are blocks of managed files on same node as parent Copyright Concurrent, Inc. 2011. All rights reserved.
Deployment Topology Node Node Node jobs tasks Client JobTracker TaskTracker DataNode Node NameNode Not uncommon to Node be same node Secondary • Job Client may run on any node • NameNode and JobTracker may run on same node (Master) • DataNode and TaskTracker instances should run on same node (Slaves) • NameNode and SecondaryNode shouldn’t typically run on same node Copyright Concurrent, Inc. 2011. All rights reserved.
Complex job assemblies • Real applications are many MapReduce jobs chained together • Linked by intermediate (usually temporary) files • Executed in order, by hand, from the ‘client’ application Count Job Sort Job [ k, [v] ] [ k, [v] ] Map Reduce Map Reduce [ k, v ] [ k, v ] [ k, v ] [ k, v ] File File File [ k, v ] = key and value pair [ k, [v] ] = key and associated values collection Copyright Concurrent, Inc. 2011. All rights reserved.
Tokenize Count Job Map Reduce Map Reduce File File File Join Tokens/Counts Job File Map Reduce File Sort/Prefix Filter Job Map Reduce File Match two sets Self Join Job Map Reduce using prefix File filtering Unique Pairs Job Map Reduce File Join LHS Job Map Reduce File Join RHS / Match Job Map Reduce File Copyright Concurrent, Inc. 2011. All rights reserved.
Real World Apps [37/75] map+reduce [54/75] map+reduce [41/75] map+reduce [43/75] map+reduce [42/75] map+reduce [45/75] map+reduce [44/75] map+reduce [39/75] map+reduce [36/75] map+reduce [46/75] map+reduce [40/75] map+reduce [50/75] map+reduce [38/75] map+reduce [49/75] map+reduce [51/75] map+reduce [47/75] map+reduce [52/75] map+reduce [53/75] map+reduce [48/75] map+reduce [23/75] map+reduce [25/75] map+reduce [24/75] map+reduce [27/75] map+reduce [26/75] map+reduce [21/75] map+reduce [19/75] map+reduce [28/75] map+reduce [22/75] map+reduce [32/75] map+reduce [20/75] map+reduce [31/75] map+reduce [33/75] map+reduce [29/75] map+reduce [34/75] map+reduce [35/75] map+reduce [30/75] map+reduce [7/75] map+reduce [2/75] map+reduce [8/75] map+reduce [10/75] map+reduce [9/75] map+reduce [5/75] map+reduce [3/75] map+reduce [11/75] map+reduce [6/75] map+reduce [13/75] map+reduce [4/75] map+reduce [16/75] map+reduce [14/75] map+reduce [15/75] map+reduce [17/75] map+reduce [18/75] map+reduce [12/75] map+reduce [60/75] map [62/75] map [61/75] map [58/75] map [55/75] map [56/75] map+reduce [57/75] map [71/75] map [72/75] map [59/75] map [64/75] map+reduce [63/75] map+reduce [65/75] map+reduce [68/75] map+reduce [67/75] map+reduce [70/75] map+reduce [69/75] map+reduce [73/75] map+reduce [66/75] map+reduce [74/75] map+reduce [75/75] map+reduce [1/75] map+reduce 1 app, 75 jobs green = map + reduce purple = map blue = join/merge orange = map split Copyright Concurrent, Inc. 2011. All rights reserved.
Cascading Word Count/Sort Flow Map Reduce Map Reduce [ f1,f2,.. ] [ f1,f2,.. ] [ f1,f2,.. ] Parse Group Count Sort [ f1,f2,.. ] [ f1,f2,.. ] Data [ f1, f2,... ] = tuples with field names Data • Alternative model & API to MapReduce • pipe/filters of re-usable operations • For rapidly implementing Data Processing Systems Copyright Concurrent, Inc. 2011. All rights reserved.
Cascading • Allows for Unit testing independent of integration • Re-usable libraries • Integration is first class • Homogeneous framework for scheduling • Any JVM based host language Copyright Concurrent, Inc. 2011. All rights reserved.
Elastic MapReduce Amazon Web Services Elastic MapReduce User CLI Console Master Slaves Client mr mr Map/Reduce temp HDFS input output S3 jar • Clusters typically single purpose • S3 used for storage between runs Copyright Concurrent, Inc. 2011. All rights reserved.
Architecture Isn’t Innovation operationalization collection cleansing processing delivery event data signal info knowledge normalization scoring mining innovation Rate of innovation and arrival of answers are proportional Copyright Concurrent, Inc. 2011. All rights reserved.
Big vs Lots Lots of "Big" Data Data Data ! = Hadoop Mining* ! ? ? = RDBMS, R, etc Data * Data Warehousing Processing ! ! • Big - too much to fit in/on any one thing • Lots - complexity arising from keeping track of all the bits Copyright Concurrent, Inc. 2011. All rights reserved.
At Rest vs In Motion data mining raw data ETL data warehousing loggers loggers loggers ETL Analyst Data At Rest raw data data processing valuable loggers data loggers loggers Consumer process Data In Motion • Hub/Spoke vs Incremental Layers • Static Schema vs Dynamic Views • Monolithic vs Distributed Copyright Concurrent, Inc. 2011. All rights reserved.
Hadoop for Processing Value Creation Scalability Simplicity • Delivering Value from Innovation • Scalability, Not Performance • Simplifies Infrastructure Copyright Concurrent, Inc. 2011. All rights reserved.
Simplicity Cluster Rack Rack Rack Node Node Node Node ... cpus Global Compute-space disks Global Namespace • Virtualization across resources, not within (PaaS) • A single FileSystem across disks - no DBA • A single Execution System across CPUs - less IT • One app installed and managed across hardware Copyright Concurrent, Inc. 2011. All rights reserved.
Scalability Users Cluster Client Rack Rack Rack Node Node Node Node ... Client job job job Client • Scalability - continued reliability and met expectations as demand changes • Application Scalability - data grows, app/infra expand • Organizational Scalability - simpler infra and apps Copyright Concurrent, Inc. 2011. All rights reserved.
Delivering Value events reporting raw data loggers loggers data processing loggers Hadoop + Hadoop etlCascading analytics Cascading Producer Consumer product operational Value • Unconstrained processing model • Data processing requires integration • Processing must not fail or fall behind Copyright Concurrent, Inc. 2011. All rights reserved.
Data In Motion raw data data processing valuable loggers data loggers loggers Consumer process Data In Motion • Data always arriving, results being delivered • Not paying the upfront cost of indexing • No upfront schema design • “ETL” is built into the processing pipeline Copyright Concurrent, Inc. 2011. All rights reserved.
Where to Innovate? Lots of "Big" Data Data Data ! = Hadoop Mining* ! ? ? = RDBMS, R, etc Data * Data Warehousing Processing ! ! • Depends on the problem whether Hadoop makes sense as your innovation platform Copyright Concurrent, Inc. 2011. All rights reserved.
Hadoop for Innovating value innovation innovation innovation latency degrees of freedom • Need to ask similar questions repeatedly • Indexes help here • Need a reasonably high abstraction • Existing libraries and a simple syntax • Third-party Tool support Copyright Concurrent, Inc. 2011. All rights reserved.
Innovation Abstractions • Syntax • Pig • Hive - now has some indexing support • Language (easier to operationalize) • Cascalog • Cascading.jruby • 3 new Scala languages pending release Copyright Concurrent, Inc. 2011. All rights reserved.
Data At Rest data mining raw data ETL data warehousing loggers loggers loggers ETL Analyst Data At Rest • Hadoop becomes a warehouse (with Schemas) • and without indexes, high latency queries • ETL becomes an independent architecture Copyright Concurrent, Inc. 2011. All rights reserved.
Don’t throw out the baby with the bath water • Need low latency responses • Need support for existing tools • Need to not retrain analysts • RDBMS (Aster, • SAS GreenPlum, Vertica, Oracle) • MicroStrategies • R • Tableaux Copyright Concurrent, Inc. 2011. All rights reserved.
Bailing Wire & Bubble Gum • Integrating them with Hadoop adds brittleness and inefficiencies • Hadoop Streaming • RHIPE, etc.. Copyright Concurrent, Inc. 2011. All rights reserved.
Operationalizing operationalization collection cleansing processing delivery event data signal info knowledge normalization scoring mining innovation • Minimize the number of processing tech (debt) • Don’t lose sight of the physical model/plan • XML is not a programming language • String concatenation isn’t programming Copyright Concurrent, Inc. 2011. All rights reserved.
Resources • Chris K Wensel •chris@wensel.net •@cwensel • Cascading & Cascalog •http://cascading.org •@cascading • Concurrent, Inc. •http://concurrentinc.com •@concurrent •http://concurrentinc.com/careers Copyright Concurrent, Inc. 2011. All rights reserved.

More Related Content

PDF
2012 11 Openstack China
byOpenCity Community
 
PDF
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...
byAdrian Cockcroft
 
PDF
RESTing in the ALPS Mike Amundsen's Presentation from QCon London 2013
byCA API Management
 
PDF
Netflix in the cloud 2011
byAdrian Cockcroft
 
PPTX
SQL Data Service Overview
byEric Nelson
 
PDF
Netflix Architecture Tutorial at Gluecon
byAdrian Cockcroft
 
PDF
The Netflix Open Source Platform
byRuslan Meshenberg
 
PDF
Elaster Labs, the 1st PaaS in Taiwan
byFred Chiang
 
2012 11 Openstack China
byOpenCity Community
 
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...
byAdrian Cockcroft
 
RESTing in the ALPS Mike Amundsen's Presentation from QCon London 2013
byCA API Management
 
Netflix in the cloud 2011
byAdrian Cockcroft
 
SQL Data Service Overview
byEric Nelson
 
Netflix Architecture Tutorial at Gluecon
byAdrian Cockcroft
 
The Netflix Open Source Platform
byRuslan Meshenberg
 
Elaster Labs, the 1st PaaS in Taiwan
byFred Chiang
 

Viewers also liked

ODP
On cascading small decision trees
byJulià Minguillón
 
PDF
SMART HEALTH PREDICTION USING DATA MINING by Dr.Mahboob Khan Phd
byHealthcare consultant
 
PDF
ACM Bay Area Data Mining Workshop: Pattern, PMML, Hadoop
byPaco Nathan
 
PPT
The Future of the Workplace
byamanda gore
 
PDF
Rw 2014 data visulization
byJackie Wirz, PhD
 
PPT
Crime Mapping & Analysis – Georgia Tech
byJonathan D'Cruz
 
PPTX
Crime Analytics: Analysis of crimes through news paper articles
byChamath Sajeewa
 
PDF
Real Time Data Streaming using Kafka & Storm
byRan Silberman
 
On cascading small decision trees
byJulià Minguillón
 
SMART HEALTH PREDICTION USING DATA MINING by Dr.Mahboob Khan Phd
byHealthcare consultant
 
ACM Bay Area Data Mining Workshop: Pattern, PMML, Hadoop
byPaco Nathan
 
The Future of the Workplace
byamanda gore
 
Rw 2014 data visulization
byJackie Wirz, PhD
 
Crime Mapping & Analysis – Georgia Tech
byJonathan D'Cruz
 
Crime Analytics: Analysis of crimes through news paper articles
byChamath Sajeewa
 
Real Time Data Streaming using Kafka & Storm
byRan Silberman
 

Similar to Cascading and BigData Problems

PPT
Dreamforce_2012_Hadoop_Use_Cases
byNarayan Bharadwaj
 
PDF
Hadoop + Forcedotcom = Like
byNarayan Bharadwaj
 
PDF
Hadoop Summit San Diego Feb2013
byNarayan Bharadwaj
 
PDF
Building Scale Free Applications with Hadoop and Cascading
bycwensel
 
PDF
Google Compute and MapR
byMapR Technologies
 
PDF
SAM SIG: Hadoop architecture, MapReduce patterns, and best practices with Cas...
bycwensel
 
PDF
Nimbula lspe-2012-04-19
byChris Westin
 
PDF
A (very) short intro to Hadoop
byKen Krugler
 
PDF
The METISfiles - Data Centers and the Cloud
byPim Bilderbeek
 
PDF
Zi1one Presentation Rev7 Eng(Sep2011)
byGiancarlo Mancinelli
 
PPTX
Introduction to Cloud Data Center and Network Issues
byJason TC HOU (侯宗成)
 
PPTX
AWS Re:Invent - Optimizing Costs with AWS
byCoburn Watson
 
PDF
Cloud Foundry Anniversary: Technical Slides
bymarklucovsky
 
PDF
Architecting the Future of Big Data & Search - Eric Baldeschwieler
bylucenerevolution
 
PDF
USAREUR Cloud Computing Training Class Presentation Heidelberg 2
byGovCloud Network
 
PPTX
NYC-Meetup- Introduction to Hadoop Echosystem
byAL500745425
 
PDF
8 mattwoodaws-intro-pdf-110411093115-phpapp01
byCarl Chesal
 
PDF
Performance engineeringforcloudcomputing lero
bythreesixty
 
PDF
DevOps or: How I Learned to Stop Worrying and Love the Cloud
byHirokazu MORIKAWA
 
PDF
Ramakrishnan Keynote Ladis2009
byyarapavan
 
Dreamforce_2012_Hadoop_Use_Cases
byNarayan Bharadwaj
 
Hadoop + Forcedotcom = Like
byNarayan Bharadwaj
 
Hadoop Summit San Diego Feb2013
byNarayan Bharadwaj
 
Building Scale Free Applications with Hadoop and Cascading
bycwensel
 
Google Compute and MapR
byMapR Technologies
 
SAM SIG: Hadoop architecture, MapReduce patterns, and best practices with Cas...
bycwensel
 
Nimbula lspe-2012-04-19
byChris Westin
 
A (very) short intro to Hadoop
byKen Krugler
 
The METISfiles - Data Centers and the Cloud
byPim Bilderbeek
 
Zi1one Presentation Rev7 Eng(Sep2011)
byGiancarlo Mancinelli
 
Introduction to Cloud Data Center and Network Issues
byJason TC HOU (侯宗成)
 
AWS Re:Invent - Optimizing Costs with AWS
byCoburn Watson
 
Cloud Foundry Anniversary: Technical Slides
bymarklucovsky
 
Architecting the Future of Big Data & Search - Eric Baldeschwieler
bylucenerevolution
 
USAREUR Cloud Computing Training Class Presentation Heidelberg 2
byGovCloud Network
 
NYC-Meetup- Introduction to Hadoop Echosystem
byAL500745425
 
8 mattwoodaws-intro-pdf-110411093115-phpapp01
byCarl Chesal
 
Performance engineeringforcloudcomputing lero
bythreesixty
 
DevOps or: How I Learned to Stop Worrying and Love the Cloud
byHirokazu MORIKAWA
 
Ramakrishnan Keynote Ladis2009
byyarapavan
 

Recently uploaded

PPTX
Tech Trends 2026: AI Agents, Quantum Computing, Robotics & Cybersecurity
byridwansassman
 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 
PDF
Agent to agent service discovery using HashiCorp Consul and Vault
byBram Vogelaar
 
PPTX
GenerationAI Paris 2025 | AI in Tech: Beyond Expectations, Into Execution
byapidays
 
PPTX
Celonis Optimizing your future with process
bydevjeremyappleyard
 
PPTX
CTO Strategy OS 2026: The Tech, AI & Cloud Playbook Boards Want
byridwansassman
 
PDF
20260212 Security-JAWS activity results for 2025 and activity goals for 2026
byTyphon 666
 
PDF
Digital Twin in IBM for Accelerated Discovery of Climate & Sustainability, K...
byMichiaki Tatsubori
 
PDF
UiPath Automation Developer Associate Training Series 2025 - Session 4
byDianaGray10
 
PDF
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
PDF
Preserve workload integrity during cross-architecture migration
byPrincipled Technologies
 
PDF
final.pdf
byomarbishtawi04
 
PDF
Explaining specific examples of purpose-specific BOM
byakipii ogaoga
 
PDF
Escape from the Forbidden Zone: Smuggling green and inclusive tech past the g...
byBookNet Canada
 
PDF
GenerationAI_Paris_2025_Architecting_Intelligence.pdf
byapidays
 
PDF
Empower your IT team with cloud-based PC management using Dell Management Por...
byPrincipled Technologies
 
PDF
Oracle Cloud Infrastructure 2025 Architect Professional (1Z0-1127-25) Master ...
byExamCert App
 
PDF
From Hallucinations to High-Confidence AI with Data Enrichment.pdf
byPrecisely
 
PPTX
PPTX game guess the logo with a twistppt
byAdrianAnig
 
PDF
GTM-and-Sales-Plan for a cyber security product
byAshish Jangir
 
Tech Trends 2026: AI Agents, Quantum Computing, Robotics & Cybersecurity
byridwansassman
 
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 
Agent to agent service discovery using HashiCorp Consul and Vault
byBram Vogelaar
 
GenerationAI Paris 2025 | AI in Tech: Beyond Expectations, Into Execution
byapidays
 
Celonis Optimizing your future with process
bydevjeremyappleyard
 
CTO Strategy OS 2026: The Tech, AI & Cloud Playbook Boards Want
byridwansassman
 
20260212 Security-JAWS activity results for 2025 and activity goals for 2026
byTyphon 666
 
Digital Twin in IBM for Accelerated Discovery of Climate & Sustainability, K...
byMichiaki Tatsubori
 
UiPath Automation Developer Associate Training Series 2025 - Session 4
byDianaGray10
 
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
Preserve workload integrity during cross-architecture migration
byPrincipled Technologies
 
final.pdf
byomarbishtawi04
 
Explaining specific examples of purpose-specific BOM
byakipii ogaoga
 
Escape from the Forbidden Zone: Smuggling green and inclusive tech past the g...
byBookNet Canada
 
GenerationAI_Paris_2025_Architecting_Intelligence.pdf
byapidays
 
Empower your IT team with cloud-based PC management using Dell Management Por...
byPrincipled Technologies
 
Oracle Cloud Infrastructure 2025 Architect Professional (1Z0-1127-25) Master ...
byExamCert App
 
From Hallucinations to High-Confidence AI with Data Enrichment.pdf
byPrecisely
 
PPTX game guess the logo with a twistppt
byAdrianAnig
 
GTM-and-Sales-Plan for a cyber security product
byAshish Jangir
 

Cascading and BigData Problems

  • 1.
    Cascading and BigData Problems Chris K Wensel Concurrent, Inc. Copyright Concurrent, Inc. 2011. All rights reserved.
  • 2.
    About Me • Concurrent, Inc., Founder • Cascading support and tools • http://concurrentinc.com/ • Cascading, Lead Developer (started Sept 2007) • An alternative API to MapReduce • http://cascading.org/ • Formerly Hadoop mentoring and training • Sun - Apple - HP - LexisNexis - startups - etc • Formerly Systems Architect & Consultant • Thomson/Reuters - TeleAtlas - startups - etc Copyright Concurrent, Inc. 2011. All rights reserved.
  • 3.
    Overview • Case Studies •What’s in common? • Where does Hadoop fit? • Processing vs Innovation Copyright Concurrent, Inc. 2011. All rights reserved.
  • 4.
    Case Studies • ShareThis •BestBuy • FlightCaster • Etsy • Ion Flux Copyright Concurrent, Inc. 2011. All rights reserved.
  • 5.
    Summary • All runningin production with Hadoop • All use AWS, most use Elastic MapReduce • All production processing was implemented in Cascading • Various other tools used at different stages of development Copyright Concurrent, Inc. 2011. All rights reserved.
  • 6.
    Share This • Cascading+ AWS (pre-EMR) • Daily event log processing, initially multiple TB and growing • Details in the O’Reilly Hadoop book from Tom White Copyright Concurrent, Inc. 2011. All rights reserved.
  • 7.
    Lessons every Y hrs on crawl completion every X hrs logprocessor crawler indexer ... ... • Mark data as bad and why, never discard • useful for upstream debugging • Data is seasonal, cyclical, and bursty • Tune your app and cluster to the workload • (garbage collect Hadoop clusters) Copyright Concurrent, Inc. 2011. All rights reserved.
  • 8.
    BestBuy - BehavioralAd- Targeting • Cascading + AWS (Elastic MapReduce) • Daily automated User Behavior Segmentation • 6wks dev, 3T/day, $13k/mo • 500% increase in return on ad spend from a similar campaign a year before • http://aws.amazon.com/solutions/case-studies/ razorfish/ Copyright Concurrent, Inc. 2011. All rights reserved.
  • 9.
    Cluster Amazon Web Services Elastic MapReduce Slaves Ad System Map/Reduce behavior app HDFS input output S3 E-Commerce Site • 200+ nodes, 9-12 hour runs • 30+ days of history + 3TB daily • Remote HTTP update of ad-server • of only changed data Copyright Concurrent, Inc. 2011. All rights reserved.
  • 10.
    Road Blocks • Noone really understood the data • Character formats (UTF-8 vs ...) • Zero byte chars • Unique columns not unique • Outliers in the data • Creating test data • QAing the data • result data was also big Copyright Concurrent, Inc. 2011. All rights reserved.
  • 11.
    FlightCaster - Predicting Flight Delays • Clojure + Cascading + AWS • Scours data on every domestic flight for the past 10-years and matches it to real- time conditions • Machine learning on Cascading, Scoring on app server • 3mos dev, 10G day, <$2k/mos Copyright Concurrent, Inc. 2011. All rights reserved.
  • 12.
    Lessons • Even witha good abstraction, you must intuit the underlying model (MapReduce) to improve throughput • i.e. Logical vs Physical plans • we still need DBAs after decades of query planner dev Copyright Concurrent, Inc. 2011. All rights reserved.
  • 13.
    Etsy - Online Marketplace • JRuby + Cascading + AWS • 1B page-views & multi-T data/mo, of logs • 40-50 cascading.jruby jobs a night • http://codeascraft.etsy.com/2010/02/24/ analyzing-etsys-data-with-hadoop-and-cascading/ • http://www.concurrentinc.com/casestudies/etsy Copyright Concurrent, Inc. 2011. All rights reserved.
  • 14.
    Initially • JRuby forthe ‘analysts’ • Log pre-processing, • db snap shot diffs, • nightly and ad-hoc analytics Copyright Concurrent, Inc. 2011. All rights reserved.
  • 15.
    Data Driven Products •Search index/scoring (under dev) • Taste Test • Facebook gift recommender • Suggested shops • Top query list, etc... • Many more on the way Copyright Concurrent, Inc. 2011. All rights reserved.
  • 16.
    Ion Flux -Gene Sequencing • Cascading + AWS • Sequence Alignment • http://aws.amazon.com/solutions/case- studies/ion-flux/ Copyright Concurrent, Inc. 2011. All rights reserved.
  • 17.
    Cluster • 10-30 nodes,using new HPC instances • 200-500 cores, • runs up to 50 hours Copyright Concurrent, Inc. 2011. All rights reserved.
  • 18.
    Architecture Delivery Ion Flux - Annotation Server Clinical Lab Ion Torrent - Torrent Sequencer Ion Torrent - Torrent Server (EC2) Ion Flux - Pipeline Controller FastQ Ion Torrent RAW Data FTP FTP RAW Data FTP Upload Annotation Sample Prep Chip Measure DNA File File Basecalling Sequence Wait Start Pipeline Third Party Clients Client Server File Server Complete? Database (RDS) Client App Annotation ReST Server Ion Flux - Flux Capacitor Ion Flux - LIMS FastQ FastQ Compressed DNA Sample FTP Transfer Cloud LIMS ReST Chip LIMS Sequence Split File Sequence Compress Sequence Transfer Agent Client Agent Input Server (EC2) Metadata Database File Chunk File (RDS) Ion Flux - Variant Server (EC2) Variants ReST Variant Server Database Ion Flux - (RDS) Client Website External Variant Complete Partners Report Runs (EC2) AWS - S3 Storage FastQ Software & Performance PILEUP Sequence Data Data Variants Chunks Heavy Lifting Ion Flux - Sequencing Pipeline AWS - EMR Cluster SAM Corrected Bootstrap Cluster Configure SAM Sort by Sorted SAM Split to PILEUP TMAP Alignment SRMA SAM PILEUP Nodes Pipeline Alignments position Alignments Bins Variants Bins Alignments Create Cluster Shutdown Cascading Cluster Cleanup Cluster Start Node Performance Profiler Data Copyright Concurrent, Inc. 2011. All rights reserved.
  • 19.
    Common Architecture intermediate data raw data loggers ? valuable loggers data loggers Analyst Producer Consumer Developer Value • New data continuously arriving • Actively incorporating the new with the old • Updating backend systems Copyright Concurrent, Inc. 2011. All rights reserved.
  • 20.
    Common Constraints • Speedof light • Understanding the data • Creating tests and validating the results • Lifecycle phases have different environments • dev vs. integration vs. prod • Better algorithms, less cost, more complexity Copyright Concurrent, Inc. 2011. All rights reserved.
  • 21.
    Apps Have ManyStages • Heavy Lifting • Scoring • Modeling & • Processing Learning Copyright Concurrent, Inc. 2011. All rights reserved.
  • 22.
    Heavy Lifting • ETLStyle processes hampered by physics • Moving/Transferring/Packaging data • Data cleansing and value normalization Copyright Concurrent, Inc. 2011. All rights reserved.
  • 23.
    Modeling & Learning •Also known as “Data Mining” • Ask lots of questions to understand the data • Machine learning, or • Ad-hoc queries • Where the innovation happens Copyright Concurrent, Inc. 2011. All rights reserved.
  • 24.
    Processing • Transforming and/orcombining multiple data sets into new data sets or models • Analytics, • indexing (w/ • statistics, scoring), • enrichment, • feature reduction, • entity extraction, • matching Copyright Concurrent, Inc. 2011. All rights reserved.
  • 25.
    Scoring • Apply what’slearned • Sometimes batch (as part of Processing) • indices with search result ranking • Sometimes transactional, req/resp • prediction, recommendations, etc Copyright Concurrent, Inc. 2011. All rights reserved.
  • 26.
    In Summary collection cleansing processing delivery event data signal info knowledge normalization scoring mining The point of computing systems is to make data more valuable Copyright Concurrent, Inc. 2011. All rights reserved.
  • 27.
    Where does Hadoop fit? Copyright Concurrent, Inc. 2011. All rights reserved.
  • 28.
    Hadoop Cluster Rack Rack Rack Node Node Node Node ... Global Compute-space Global Namespace • Distributed replicated storage for large files • Distributed fault tolerant exec of batch processes • Scale out vs (legacy) scale up • Java API allows complex analysis, more freedom Copyright Concurrent, Inc. 2011. All rights reserved.
  • 29.
    MapReduce • A “divide and conquer” strategy for parallelizing workloads against collections of data • Map & Reduce are two user defined functions chained via Key Value Pairs • It’s really Map->Group->Reduce where Group is built in Copyright Concurrent, Inc. 2011. All rights reserved.
  • 30.
    Keys and Values • Map translates input to keys and values to new keys and values [K1,V1] Map [K2,V2]* • System Groups each unique [K2,V2] Group [K2,{V2,V2,....}] key with all its values [K2,{V2,V2,....}] Reduce [K3,V3]* • Reduce translates the values of each unique key to new keys and values * = zero or more Copyright Concurrent, Inc. 2011. All rights reserved.
  • 31.
    Word Count Mapper [0,"when in the course of human events"] Map ["when",1] ["in",1] ["the",1] [...,1] ["when",1] ["when",1] ["when",1] ["when",1] Group ["when",{1,1,1,1,1}] ["when",1] Reducer ["when",{1,1,1,1,1}] Reduce ["when",5] Copyright Concurrent, Inc. 2011. All rights reserved.
  • 32.
    Divide and Conquer Parallelism • Since the ‘records’ entering the Map and ‘groups’ entering the Reduce are independent • That is, there is no expectation of order or requirement to share state between records/ groups • Arbitrary numbers of Map and Reduce function instances can be created against arbitrary portions of input data Copyright Concurrent, Inc. 2011. All rights reserved.
  • 33.
    Cluster Cluster Rack Rack Rack Node Node Node Node ... map map map map map reduce reduce reduce • Multiple instances of each Map and Reduce function are distributed throughout the cluster Copyright Concurrent, Inc. 2011. All rights reserved.
  • 34.
    Another View [K1,V1] Map [K2,V2] Combine Group [K2,{V2,...}] Reduce [K3,V3] Mapper Task same code Mapper Reducer Shuffle Task Task Mapper Reducer Shuffle Task Task Mapper Reducer Shuffle Task Task Mapper Task Mappers must complete before Reducers can begin split1 split2 split3 split4 ... part-00000 part-00001 part-000N file directory Copyright Concurrent, Inc. 2011. All rights reserved.
  • 35.
    Architectural Components NameNode DataNode DataNode DataNode DataNode data block ns read/write operations Secondary ns operations read/write ns operations read/write mapper mapper child jvm mapper child jvm jobs tasks child jvm Client JobTracker TaskTracker reducer reducer child jvm reducer child jvm child jvm • Solid boxes are unique applications • Dashed boxes are child JVM instances on same node as parent • Dotted boxes are blocks of managed files on same node as parent Copyright Concurrent, Inc. 2011. All rights reserved.
  • 36.
    Deployment Topology Node Node Node jobs tasks Client JobTracker TaskTracker DataNode Node NameNode Not uncommon to Node be same node Secondary • Job Client may run on any node • NameNode and JobTracker may run on same node (Master) • DataNode and TaskTracker instances should run on same node (Slaves) • NameNode and SecondaryNode shouldn’t typically run on same node Copyright Concurrent, Inc. 2011. All rights reserved.
  • 37.
    Complex job assemblies • Real applications are many MapReduce jobs chained together • Linked by intermediate (usually temporary) files • Executed in order, by hand, from the ‘client’ application Count Job Sort Job [ k, [v] ] [ k, [v] ] Map Reduce Map Reduce [ k, v ] [ k, v ] [ k, v ] [ k, v ] File File File [ k, v ] = key and value pair [ k, [v] ] = key and associated values collection Copyright Concurrent, Inc. 2011. All rights reserved.
  • 38.
    Tokenize Count Job Map Reduce Map Reduce File File File Join Tokens/Counts Job File Map Reduce File Sort/Prefix Filter Job Map Reduce File Match two sets Self Join Job Map Reduce using prefix File filtering Unique Pairs Job Map Reduce File Join LHS Job Map Reduce File Join RHS / Match Job Map Reduce File Copyright Concurrent, Inc. 2011. All rights reserved.
  • 39.
    Real World Apps [37/75] map+reduce [54/75] map+reduce [41/75] map+reduce [43/75] map+reduce [42/75] map+reduce [45/75] map+reduce [44/75] map+reduce [39/75] map+reduce [36/75] map+reduce [46/75] map+reduce [40/75] map+reduce [50/75] map+reduce [38/75] map+reduce [49/75] map+reduce [51/75] map+reduce [47/75] map+reduce [52/75] map+reduce [53/75] map+reduce [48/75] map+reduce [23/75] map+reduce [25/75] map+reduce [24/75] map+reduce [27/75] map+reduce [26/75] map+reduce [21/75] map+reduce [19/75] map+reduce [28/75] map+reduce [22/75] map+reduce [32/75] map+reduce [20/75] map+reduce [31/75] map+reduce [33/75] map+reduce [29/75] map+reduce [34/75] map+reduce [35/75] map+reduce [30/75] map+reduce [7/75] map+reduce [2/75] map+reduce [8/75] map+reduce [10/75] map+reduce [9/75] map+reduce [5/75] map+reduce [3/75] map+reduce [11/75] map+reduce [6/75] map+reduce [13/75] map+reduce [4/75] map+reduce [16/75] map+reduce [14/75] map+reduce [15/75] map+reduce [17/75] map+reduce [18/75] map+reduce [12/75] map+reduce [60/75] map [62/75] map [61/75] map [58/75] map [55/75] map [56/75] map+reduce [57/75] map [71/75] map [72/75] map [59/75] map [64/75] map+reduce [63/75] map+reduce [65/75] map+reduce [68/75] map+reduce [67/75] map+reduce [70/75] map+reduce [69/75] map+reduce [73/75] map+reduce [66/75] map+reduce [74/75] map+reduce [75/75] map+reduce [1/75] map+reduce 1 app, 75 jobs green = map + reduce purple = map blue = join/merge orange = map split Copyright Concurrent, Inc. 2011. All rights reserved.
  • 40.
    Cascading Word Count/Sort Flow Map Reduce Map Reduce [ f1,f2,.. ] [ f1,f2,.. ] [ f1,f2,.. ] Parse Group Count Sort [ f1,f2,.. ] [ f1,f2,.. ] Data [ f1, f2,... ] = tuples with field names Data • Alternative model & API to MapReduce • pipe/filters of re-usable operations • For rapidly implementing Data Processing Systems Copyright Concurrent, Inc. 2011. All rights reserved.
  • 41.
    Cascading • Allows forUnit testing independent of integration • Re-usable libraries • Integration is first class • Homogeneous framework for scheduling • Any JVM based host language Copyright Concurrent, Inc. 2011. All rights reserved.
  • 42.
    Elastic MapReduce Amazon Web Services Elastic MapReduce User CLI Console Master Slaves Client mr mr Map/Reduce temp HDFS input output S3 jar • Clusters typically single purpose • S3 used for storage between runs Copyright Concurrent, Inc. 2011. All rights reserved.
  • 43.
    Architecture Isn’t Innovation operationalization collection cleansing processing delivery event data signal info knowledge normalization scoring mining innovation Rate of innovation and arrival of answers are proportional Copyright Concurrent, Inc. 2011. All rights reserved.
  • 44.
    Big vs Lots Lots of "Big" Data Data Data ! = Hadoop Mining* ! ? ? = RDBMS, R, etc Data * Data Warehousing Processing ! ! • Big - too much to fit in/on any one thing • Lots - complexity arising from keeping track of all the bits Copyright Concurrent, Inc. 2011. All rights reserved.
  • 45.
    At Rest vsIn Motion data mining raw data ETL data warehousing loggers loggers loggers ETL Analyst Data At Rest raw data data processing valuable loggers data loggers loggers Consumer process Data In Motion • Hub/Spoke vs Incremental Layers • Static Schema vs Dynamic Views • Monolithic vs Distributed Copyright Concurrent, Inc. 2011. All rights reserved.
  • 46.
    Hadoop for Processing Value Creation Scalability Simplicity • Delivering Value from Innovation • Scalability, Not Performance • Simplifies Infrastructure Copyright Concurrent, Inc. 2011. All rights reserved.
  • 47.
    Simplicity Cluster Rack Rack Rack Node Node Node Node ... cpus Global Compute-space disks Global Namespace • Virtualization across resources, not within (PaaS) • A single FileSystem across disks - no DBA • A single Execution System across CPUs - less IT • One app installed and managed across hardware Copyright Concurrent, Inc. 2011. All rights reserved.
  • 48.
    Scalability Users Cluster Client Rack Rack Rack Node Node Node Node ... Client job job job Client • Scalability - continued reliability and met expectations as demand changes • Application Scalability - data grows, app/infra expand • Organizational Scalability - simpler infra and apps Copyright Concurrent, Inc. 2011. All rights reserved.
  • 49.
    Delivering Value events reporting raw data loggers loggers data processing loggers Hadoop + Hadoop etlCascading analytics Cascading Producer Consumer product operational Value • Unconstrained processing model • Data processing requires integration • Processing must not fail or fall behind Copyright Concurrent, Inc. 2011. All rights reserved.
  • 50.
    Data In Motion raw data data processing valuable loggers data loggers loggers Consumer process Data In Motion • Data always arriving, results being delivered • Not paying the upfront cost of indexing • No upfront schema design • “ETL” is built into the processing pipeline Copyright Concurrent, Inc. 2011. All rights reserved.
  • 51.
    Where to Innovate? Lots of "Big" Data Data Data ! = Hadoop Mining* ! ? ? = RDBMS, R, etc Data * Data Warehousing Processing ! ! • Depends on the problem whether Hadoop makes sense as your innovation platform Copyright Concurrent, Inc. 2011. All rights reserved.
  • 52.
    Hadoop for Innovating value innovation innovation innovation latency degrees of freedom • Need to ask similar questions repeatedly • Indexes help here • Need a reasonably high abstraction • Existing libraries and a simple syntax • Third-party Tool support Copyright Concurrent, Inc. 2011. All rights reserved.
  • 53.
    Innovation Abstractions • Syntax • Pig • Hive - now has some indexing support • Language (easier to operationalize) • Cascalog • Cascading.jruby • 3 new Scala languages pending release Copyright Concurrent, Inc. 2011. All rights reserved.
  • 54.
    Data At Rest data mining raw data ETL data warehousing loggers loggers loggers ETL Analyst Data At Rest • Hadoop becomes a warehouse (with Schemas) • and without indexes, high latency queries • ETL becomes an independent architecture Copyright Concurrent, Inc. 2011. All rights reserved.
  • 55.
    Don’t throw outthe baby with the bath water • Need low latency responses • Need support for existing tools • Need to not retrain analysts • RDBMS (Aster, • SAS GreenPlum, Vertica, Oracle) • MicroStrategies • R • Tableaux Copyright Concurrent, Inc. 2011. All rights reserved.
  • 56.
    Bailing Wire &Bubble Gum • Integrating them with Hadoop adds brittleness and inefficiencies • Hadoop Streaming • RHIPE, etc.. Copyright Concurrent, Inc. 2011. All rights reserved.
  • 57.
    Operationalizing operationalization collection cleansing processing delivery event data signal info knowledge normalization scoring mining innovation • Minimize the number of processing tech (debt) • Don’t lose sight of the physical model/plan • XML is not a programming language • String concatenation isn’t programming Copyright Concurrent, Inc. 2011. All rights reserved.
  • 58.
    Resources • Chris K Wensel •chris@wensel.net •@cwensel • Cascading & Cascalog •http://cascading.org •@cascading • Concurrent, Inc. •http://concurrentinc.com •@concurrent •http://concurrentinc.com/careers Copyright Concurrent, Inc. 2011. All rights reserved.

Editor's Notes