Intridea ajn-rttos OA NYC Summit
Upcoming SlideShare
Loading in...5
×
 

Intridea ajn-rttos OA NYC Summit

on

  • 3,748 views

OANYC Summit

OANYC Summit

Statistics

Views

Total Views
3,748
Views on SlideShare
693
Embed Views
3,055

Actions

Likes
0
Downloads
2
Comments
0

37 Embeds 3,055

http://todobi.blogspot.com.es 1189
http://todobi.blogspot.com 937
http://www.ikanow.com 232
http://todobi.blogspot.com.ar 227
http://todobi.blogspot.mx 153
http://cloud.feedly.com 79
http://www.todobi.blogspot.com 55
http://todobi.blogspot.com.br 38
http://todobi.blogspot.de 17
http://192.254.196.224 17
http://feeds.feedburner.com 17
http://todobi.blogspot.co.nz 16
http://todobi.blogspot.co.uk 8
http://todobi.blogspot.fr 8
http://newsblur.com 7
http://todobi.blogspot.in 6
http://todobi.blogspot.ca 5
http://www.todobi.blogspot.com.es 5
http://todobi.blogspot.pt 4
http://todobi.blogspot.jp 4
http://todobi.blogspot.be 4
http://todobi.blogspot.it 3
http://feedreader.com 3
http://todobi.blogspot.ie 2
http://webcache.googleusercontent.com 2
http://todobi.blogspot.nl 2
http://todobi.blogspot.co.at 2
http://todobi.blogspot.dk 2
http://todobi.blogspot.com.au 2
http://www.directrss.co.il 2
http://todobi.blogspot.tw 1
http://translate.googleusercontent.com 1
http://131.253.14.66 1
http://todobi.blogspot.ch 1
http://dev.newsblur.com 1
http://prlog.ru 1
http://www.google.com.au 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Intridea ajn-rttos OA NYC Summit Intridea ajn-rttos OA NYC Summit Presentation Transcript

  • Anthony NyströmFellow, Managing Director of EngineeringTuesday, June 18, 13
  • Tuesday, June 18, 13
  • What is Intridea?Tuesday, June 18, 13
  • What is Intridea?We design anddevelop apps:Web, Mobile and DataTuesday, June 18, 13
  • What is Intridea?We design anddevelop apps:Web, Mobile and DataFounded inWashington, DCTuesday, June 18, 13
  • What is Intridea?We work with coolclients – really!We design anddevelop apps:Web, Mobile and DataFounded inWashington, DCTuesday, June 18, 13
  • What is Intridea?40+ Intrideans:Designers/Developers/Scientists+ Smart biz folksWe work with coolclients – really!We design anddevelop apps:Web, Mobile and DataFounded inWashington, DCTuesday, June 18, 13
  • What is Intridea?40+ Intrideans:Designers/Developers/Scientists+ Smart biz folksWe work with coolclients – really!We work from anywhere!We design anddevelop apps:Web, Mobile and DataFounded inWashington, DCTuesday, June 18, 13
  • What is Intridea?40+ Intrideans:Designers/Developers/Scientists+ Smart biz folksWe work with coolclients – really!We work from anywhere!We design anddevelop apps:Web, Mobile and DataFounded inWashington, DCWe are growingTuesday, June 18, 13
  • What is Intridea?40+ Intrideans:Designers/Developers/Scientists+ Smart biz folksWe work with coolclients – really!We work from anywhere!We hire the best andthe smartestWe design anddevelop apps:Web, Mobile and DataFounded inWashington, DCWe are growingTuesday, June 18, 13
  • Anthony NyströmFellow, Managing Director of EngineeringTuesday, June 18, 13
  • The guy on stageIntridean:Anthony NyströmFellow, Managing Director of EngineeringTuesday, June 18, 13
  • Tuesday, June 18, 13
  • Data Science in the NOW!It takes an army of TOOLSTuesday, June 18, 13
  • Tuesday, June 18, 13
  • An Army of Tools you say?Tuesday, June 18, 13
  • An Army of Tools you say?• I am going to talk about what NOW means in Data Science• Databases, Streaming Engines, Query Engines and Interfaces• We are going to look at many of them and single out a few• Each has a respected and in some cases competing set offeaturesTuesday, June 18, 13
  • Tuesday, June 18, 13
  • Why is NOW in data Special?Tuesday, June 18, 13
  • Why is NOW in data Special?Actionable Intelligence & KnowledgeTuesday, June 18, 13
  • Why is NOW in data Special?Actionable Intelligence & KnowledgeNOW has innate contextTuesday, June 18, 13
  • Why is NOW in data Special?Actionable Intelligence & KnowledgeNOW has innate contextTIME is THE natural facet for our minds &life!Tuesday, June 18, 13
  • Tuesday, June 18, 13
  • Why is NOW in data Special?Tuesday, June 18, 13
  • Why is NOW in data Special?Trends | Patterns | ExtractionTuesday, June 18, 13
  • Why is NOW in data Special?Trends | Patterns | ExtractionData Centric TrendsTuesday, June 18, 13
  • Why is NOW in data Special?Trends | Patterns | ExtractionData Centric TrendsPattern Extraction (ML/NLP)Tuesday, June 18, 13
  • Why is NOW in data Special?Trends | Patterns | ExtractionData Centric TrendsPattern Extraction (ML/NLP)Signature Extraction (Binary, Encoded)Tuesday, June 18, 13
  • Why is NOW in data Special?Trends | Patterns | ExtractionData Centric TrendsPattern Extraction (ML/NLP)Signature Extraction (Binary, Encoded)Not user input data like Google, Yahoo etc.Tuesday, June 18, 13
  • Why is NOW in data Special?Trends | Patterns | ExtractionData Centric TrendsPattern Extraction (ML/NLP)Signature Extraction (Binary, Encoded)Not user input data like Google, Yahoo etc.“I am looking for data that conforms to a learned or known pattern”Tuesday, June 18, 13
  • Why is NOW in data Special?Trends | Patterns | ExtractionData Centric TrendsPattern Extraction (ML/NLP)Signature Extraction (Binary, Encoded)Not user input data like Google, Yahoo etc.“I am looking for data that conforms to a learned or known pattern”“I am looking for data that matches a predefined signature”Tuesday, June 18, 13
  • Tuesday, June 18, 13
  • Why is NOW in data Special?Tuesday, June 18, 13
  • Why is NOW in data Special?Routing | Transformation | ComputationTuesday, June 18, 13
  • Why is NOW in data Special?Routing | Transformation | ComputationIntelligent RoutingTuesday, June 18, 13
  • Why is NOW in data Special?Routing | Transformation | ComputationTransformation & ComputationIntelligent RoutingTuesday, June 18, 13
  • Why is NOW in data Special?Routing | Transformation | ComputationTransformation & ComputationIntelligent Routing“I need to replicate/fork that of criteria x portions of this datastream”Tuesday, June 18, 13
  • Why is NOW in data Special?Routing | Transformation | ComputationTransformation & ComputationIntelligent Routing“I need to replicate/fork that of criteria x portions of this datastream”“I need to transform certain fields” or “I need to computea some value on certain fields”Tuesday, June 18, 13
  • Tuesday, June 18, 13
  • Why is NOW in data Special?Tuesday, June 18, 13
  • Why is NOW in data Special?Algorithmic SpecialityTuesday, June 18, 13
  • Why is NOW in data Special?ConceptsAlgorithmic SpecialityTuesday, June 18, 13
  • Why is NOW in data Special?RegressionConceptsAlgorithmic SpecialityTuesday, June 18, 13
  • Why is NOW in data Special?RelationshipsRegressionConceptsAlgorithmic SpecialityTuesday, June 18, 13
  • Why is NOW in data Special?RelationshipsRegressionConceptsAlgorithmic SpecialityWhat does a value represent or infer (NLP/ML/k-NN)Tuesday, June 18, 13
  • Why is NOW in data Special?RelationshipsRegressionConceptsAlgorithmic SpecialityWhat does a value represent or infer (NLP/ML/k-NN)How is a value related to another value orHow can we predict such relationsTuesday, June 18, 13
  • Why is NOW in data Special?RelationshipsRegressionConceptsAlgorithmic SpecialityWhat does a value represent or infer (NLP/ML/k-NN)How is a value related to another value orHow can we predict such relationsTopological, Ontological, Forest(Evolutionary/Random) (NLP)Tuesday, June 18, 13
  • Tuesday, June 18, 13
  • Tuesday, June 18, 13
  • Tuesday, June 18, 13
  • Point of Sale System• Terminal• Admin• TabletTuesday, June 18, 13
  • Tuesday, June 18, 13
  • Merck• RT Persona• RT Data• BrowserTuesday, June 18, 13
  • Tuesday, June 18, 13
  • Where is NOW in data?Tuesday, June 18, 13
  • Where is NOW in data?Data Creation Time | Data Consumption TimeTuesday, June 18, 13
  • Tuesday, June 18, 13
  • LatencyTuesday, June 18, 13
  • LatencyData Creation Time | Data Consumption TimeTuesday, June 18, 13
  • LatencyStandard - NOPE!Data Creation Time | Data Consumption TimeTuesday, June 18, 13
  • LatencyStandard - NOPE!Depends upon the Medium - YEP!Data Creation Time | Data Consumption TimeTuesday, June 18, 13
  • LatencyStandard - NOPE!Depends upon the Consumer - YEP!Depends upon the Medium - YEP!Data Creation Time | Data Consumption TimeTuesday, June 18, 13
  • LatencyStandard - NOPE!Depends upon the Consumer - YEP!Depends upon the Medium - YEP!Depends upon Technology - YEP!Data Creation Time | Data Consumption TimeTuesday, June 18, 13
  • Tuesday, June 18, 13
  • NOW and LatencyTuesday, June 18, 13
  • NOW and LatencyReal-TimeTuesday, June 18, 13
  • NOW and LatencyReal-TimeNear Real-TimeTuesday, June 18, 13
  • NOW and LatencyReal-TimeSome-TimeNear Real-TimeTuesday, June 18, 13
  • NOW and LatencyReal-TimeSome-TimeData that is consumed immediately after creationNear Real-TimeTuesday, June 18, 13
  • NOW and LatencyReal-TimeSome-TimeData is consumed within seconds/minutesData that is consumed immediately after creationNear Real-TimeTuesday, June 18, 13
  • NOW and LatencyReal-TimeSome-TimeData is consumed when requested & is NOT RT nor NRTData is consumed within seconds/minutesData that is consumed immediately after creationNear Real-TimeTuesday, June 18, 13
  • Tuesday, June 18, 13
  • Physiological LatencyTuesday, June 18, 13
  • Perception:Research suggests that the human retina transmits data to the brain at therate of 10 million bits per second, which is close to that of 10 base Ethernetconnection!We can perceive changes in reality at ~ 13-15 frames per second (fps, orHz), Our perception of reality fully refreshes itself ~ once every 77Physiological LatencyTuesday, June 18, 13
  • Perception:Research suggests that the human retina transmits data to the brain at therate of 10 million bits per second, which is close to that of 10 base Ethernetconnection!We can perceive changes in reality at ~ 13-15 frames per second (fps, orHz), Our perception of reality fully refreshes itself ~ once every 77Stock Exchange ~ 5-100 milliseconds (ms)Physiological LatencyTuesday, June 18, 13
  • Web Sites ~ 50-400 milliseconds (ms)Perception:Research suggests that the human retina transmits data to the brain at therate of 10 million bits per second, which is close to that of 10 base Ethernetconnection!We can perceive changes in reality at ~ 13-15 frames per second (fps, orHz), Our perception of reality fully refreshes itself ~ once every 77Stock Exchange ~ 5-100 milliseconds (ms)Physiological LatencyTuesday, June 18, 13
  • Web Sites ~ 50-400 milliseconds (ms)Perception:Research suggests that the human retina transmits data to the brain at therate of 10 million bits per second, which is close to that of 10 base Ethernetconnection!We can perceive changes in reality at ~ 13-15 frames per second (fps, orHz), Our perception of reality fully refreshes itself ~ once every 77Games (FPS) ~ 10-150 milliseconds (ms)Stock Exchange ~ 5-100 milliseconds (ms)Physiological LatencyTuesday, June 18, 13
  • Web Sites ~ 50-400 milliseconds (ms)Perception:Research suggests that the human retina transmits data to the brain at therate of 10 million bits per second, which is close to that of 10 base Ethernetconnection!We can perceive changes in reality at ~ 13-15 frames per second (fps, orHz), Our perception of reality fully refreshes itself ~ once every 77Games (FPS) ~ 10-150 milliseconds (ms)Social/Games ~ 200 ms -1 secondStock Exchange ~ 5-100 milliseconds (ms)Physiological LatencyTuesday, June 18, 13
  • Tuesday, June 18, 13
  • Real-Time (DB’s, Index’s, FS’s)Tuesday, June 18, 13
  • Real-Time (DB’s, Index’s, FS’s)No particular orderTuesday, June 18, 13
  • Real-Time (DB’s, Index’s, FS’s)• MySQLNo particular orderTuesday, June 18, 13
  • Real-Time (DB’s, Index’s, FS’s)• MySQL• SQL ServerNo particular orderTuesday, June 18, 13
  • Real-Time (DB’s, Index’s, FS’s)• MySQL• PostgreSQL• SQL ServerNo particular orderTuesday, June 18, 13
  • Real-Time (DB’s, Index’s, FS’s)• MySQL• PostgreSQL• Neo4j (Graph)• SQL ServerNo particular orderTuesday, June 18, 13
  • • MongoReal-Time (DB’s, Index’s, FS’s)• MySQL• PostgreSQL• Neo4j (Graph)• SQL ServerNo particular orderTuesday, June 18, 13
  • • MongoReal-Time (DB’s, Index’s, FS’s)• MySQL• PostgreSQL• Neo4j (Graph)• SQL Server• Elastic Search (Lucene)No particular orderTuesday, June 18, 13
  • • MongoReal-Time (DB’s, Index’s, FS’s)• MySQL• PostgreSQL• Neo4j (Graph)• SQL Server• Elastic Search (Lucene)• SolrNo particular orderTuesday, June 18, 13
  • • MongoReal-Time (DB’s, Index’s, FS’s)• MySQL• PostgreSQL• Neo4j (Graph)• SQL Server• Elastic Search (Lucene)• Solr• HDFSNo particular orderTuesday, June 18, 13
  • • MongoReal-Time (DB’s, Index’s, FS’s)• MySQL• PostgreSQL• Neo4j (Graph)• SQL Server• Elastic Search (Lucene)• Solr• HDFS• HBaseNo particular orderTuesday, June 18, 13
  • • MongoReal-Time (DB’s, Index’s, FS’s)• MySQL• PostgreSQL• Neo4j (Graph)• SQL Server• Elastic Search (Lucene)• Solr• HDFS• Oracle• HBaseNo particular orderTuesday, June 18, 13
  • • MongoReal-Time (DB’s, Index’s, FS’s)• MySQL• PostgreSQL• Neo4j (Graph)• SQL Server• Elastic Search (Lucene)• Solr• HDFS• Oracle• ERTFS• HBaseNo particular orderTuesday, June 18, 13
  • • MongoReal-Time (DB’s, Index’s, FS’s)• MySQL• PostgreSQL• Neo4j (Graph)• SQL Server• Elastic Search (Lucene)• Solr• Redis• HDFS• Oracle• ERTFS• HBaseNo particular orderTuesday, June 18, 13
  • • MongoReal-Time (DB’s, Index’s, FS’s)• MySQL• PostgreSQL• Neo4j (Graph)• SQL Server• Elastic Search (Lucene)• Solr• Redis• HDFS• Oracle• ERTFS• HBase• CassandraNo particular orderTuesday, June 18, 13
  • • MongoReal-Time (DB’s, Index’s, FS’s)• MySQL• PostgreSQL• Neo4j (Graph)• SQL Server• Elastic Search (Lucene)• Solr• Redis• HDFS• Oracle• ERTFS• HBase• Cassandra• RiakNo particular orderTuesday, June 18, 13
  • Tuesday, June 18, 13
  • HBaseTuesday, June 18, 13
  • HBaseRegions and HDFSTuesday, June 18, 13
  • HBaseRegions and HDFSScalingTuesday, June 18, 13
  • HBaseRegions and HDFSHadoopScalingTuesday, June 18, 13
  • HBaseRegions and HDFS“Regions” Data files for regions are stored in HDFS and replicated tomultiple nodes in the cluster. As well, allocation in to the cluster israther automaticHadoopScalingTuesday, June 18, 13
  • HBaseRegions and HDFS“Regions” Data files for regions are stored in HDFS and replicated tomultiple nodes in the cluster. As well, allocation in to the cluster israther automaticHadoopScalingFault ToleranceCommodity MachinesTuesday, June 18, 13
  • HBaseRegions and HDFSRuns on top of HadoopMapReduce Integration“Regions” Data files for regions are stored in HDFS and replicated tomultiple nodes in the cluster. As well, allocation in to the cluster israther automaticHadoopScalingFault ToleranceCommodity MachinesTuesday, June 18, 13
  • Tuesday, June 18, 13
  • CassandraTuesday, June 18, 13
  • CassandraAlways WritableTuesday, June 18, 13
  • CassandraAlways WritableScalingTuesday, June 18, 13
  • CassandraAlways WritableMore...ScalingTuesday, June 18, 13
  • CassandraAlways WritableEven when internally the write fails. However, the data will eventuallybecome consistent (Tunable)More...ScalingTuesday, June 18, 13
  • CassandraAlways WritableEven when internally the write fails. However, the data will eventuallybecome consistent (Tunable)More...ScalingCan span data centersPeer-to-Peer communication between nodes (Gossip)Tuesday, June 18, 13
  • CassandraAlways WritableSupports MapReduceSupports Range QueriesEven when internally the write fails. However, the data will eventuallybecome consistent (Tunable)More...ScalingCan span data centersPeer-to-Peer communication between nodes (Gossip)Tuesday, June 18, 13
  • Tuesday, June 18, 13
  • RedisTuesday, June 18, 13
  • RedisTransactionsTuesday, June 18, 13
  • RedisTransactionsAn evolutionary Key-Value StoreTuesday, June 18, 13
  • RedisTransactionsPub-SubAn evolutionary Key-Value StoreTuesday, June 18, 13
  • RedisTransactionsAtomic operations (MULTI/EXEC/Discard) Queue your operations andEXEC/Commit as transaction. Allows for Roll-back support.Pub-SubAn evolutionary Key-Value StoreTuesday, June 18, 13
  • RedisTransactionsAtomic operations (MULTI/EXEC/Discard) Queue your operations andEXEC/Commit as transaction. Allows for Roll-back support.Pub-SubAn evolutionary Key-Value StoreSupports complex types that are closely related to fundamental datastructures. No need for abstraction layer.Tuesday, June 18, 13
  • RedisTransactionsPublish - Push messages to a channelSubscribe - Listen to a channelAtomic operations (MULTI/EXEC/Discard) Queue your operations andEXEC/Commit as transaction. Allows for Roll-back support.Pub-SubAn evolutionary Key-Value StoreSupports complex types that are closely related to fundamental datastructures. No need for abstraction layer.Tuesday, June 18, 13
  • Tuesday, June 18, 13
  • Near Real-Time & Real-TimeTuesday, June 18, 13
  • Near Real-Time & Real-TimeQueries and StreamsTuesday, June 18, 13
  • Near Real-Time & Real-Time• StormQueries and StreamsTuesday, June 18, 13
  • Near Real-Time & Real-Time• Storm• KafkaQueries and StreamsTuesday, June 18, 13
  • Near Real-Time & Real-Time• Storm• Drill/Dremel• KafkaQueries and StreamsTuesday, June 18, 13
  • Near Real-Time & Real-Time• Storm• Drill/Dremel• Hadoop• KafkaQueries and StreamsTuesday, June 18, 13
  • • MapReduceNear Real-Time & Real-Time• Storm• Drill/Dremel• Hadoop• KafkaQueries and StreamsTuesday, June 18, 13
  • • MapReduceNear Real-Time & Real-Time• Storm• Drill/Dremel• Hadoop• MapReduce v2 (YARN)• KafkaQueries and StreamsTuesday, June 18, 13
  • • MapReduceNear Real-Time & Real-Time• Storm• Drill/Dremel• Hadoop• MapReduce v2 (YARN)• Pig• KafkaQueries and StreamsTuesday, June 18, 13
  • • MapReduceNear Real-Time & Real-Time• Storm• Drill/Dremel• Hadoop• MapReduce v2 (YARN)• Pig• Hive• KafkaQueries and StreamsTuesday, June 18, 13
  • • MapReduceNear Real-Time & Real-Time• Storm• Drill/Dremel• Hadoop • Cascalog• MapReduce v2 (YARN)• Pig• Hive• KafkaQueries and StreamsTuesday, June 18, 13
  • • MapReduceNear Real-Time & Real-Time• Storm• Drill/Dremel• Hadoop • Cascalog• MapReduce v2 (YARN)• Pig• Hive• Kafka• DataTurbineQueries and StreamsTuesday, June 18, 13
  • Tuesday, June 18, 13
  • MapReduce/HadoopTuesday, June 18, 13
  • MapReduce/HadoopScaleTuesday, June 18, 13
  • MapReduce/HadoopScaleDevelopmentTuesday, June 18, 13
  • MapReduce/HadoopScaleBatchDevelopmentTuesday, June 18, 13
  • MapReduce/HadoopScale100’s to 1000’s of server nodesExtreme and cheapSimple programming modelBatchDevelopmentTuesday, June 18, 13
  • MapReduce/HadoopScale100’s to 1000’s of server nodesExtreme and cheapSimple programming modelBatchDevelopmentJava, Python, Grep & Others...Tuesday, June 18, 13
  • MapReduce/HadoopScaleComplex Multi-Step Processing100’s to 1000’s of server nodesExtreme and cheapSimple programming modelBatchDevelopmentJava, Python, Grep & Others...Tuesday, June 18, 13
  • Tuesday, June 18, 13
  • StormTuesday, June 18, 13
  • StormFASTTuesday, June 18, 13
  • StormFASTIntegrationTuesday, June 18, 13
  • StormFASTAssuranceIntegrationTuesday, June 18, 13
  • StormFASTOver a million tuples processed per second per nodeAssuranceIntegrationTuesday, June 18, 13
  • StormFASTOver a million tuples processed per second per nodeAssuranceIntegrationIntegrates with any queueing system and any database systemHandles the parallelization, partitioning, and retrying onfailures when necessaryTuesday, June 18, 13
  • StormFASTScalable, Fault-Tolerant, Guarantees your data will be processed!Over a million tuples processed per second per nodeAssuranceIntegrationIntegrates with any queueing system and any database systemHandles the parallelization, partitioning, and retrying onfailures when necessaryTuesday, June 18, 13
  • Tuesday, June 18, 13
  • CQL/StreamQL/SparQL/QL-RTDB/Tuesday, June 18, 13
  • CQL/StreamQL/SparQL/QL-RTDB/LanguagesTuesday, June 18, 13
  • CQL/StreamQL/SparQL/QL-RTDB/LanguagesScalableTuesday, June 18, 13
  • CQL/StreamQL/SparQL/QL-RTDB/LanguagesSQL IdiomsScalableTuesday, June 18, 13
  • CQL/StreamQL/SparQL/QL-RTDB/LanguagesHuman ReadableSQL IdiomsScalableTuesday, June 18, 13
  • CQL/StreamQL/SparQL/QL-RTDB/LanguagesHuman ReadableSQL IdiomsScalableSimultaneous n Queries upon both stream data and staticTuesday, June 18, 13
  • CQL/StreamQL/SparQL/QL-RTDB/LanguagesAll support to a large degree what you would expect from SQLHuman ReadableSQL IdiomsScalableSimultaneous n Queries upon both stream data and staticTuesday, June 18, 13
  • Tuesday, June 18, 13
  • PIGTuesday, June 18, 13
  • PIGLanguageTuesday, June 18, 13
  • PIGLanguageParallelizationTuesday, June 18, 13
  • PIGLanguageUnderneathParallelizationTuesday, June 18, 13
  • PIGLanguageHigh Level and easy to understand (Pig Latin)UnderneathParallelizationTuesday, June 18, 13
  • PIGLanguageHigh Level and easy to understand (Pig Latin)UnderneathParallelizationIt is trivial to achieve parallel execution of simple, "embarrassinglyparallel" data analysis tasksTuesday, June 18, 13
  • PIGLanguageEssentially a MapReduce sequence compilerHigh Level and easy to understand (Pig Latin)UnderneathParallelizationIt is trivial to achieve parallel execution of simple, "embarrassinglyparallel" data analysis tasksTuesday, June 18, 13
  • Tuesday, June 18, 13
  • PIGTuesday, June 18, 13
  • PIGExample Pig ScriptTuesday, June 18, 13
  • PIGExample Pig ScriptTuesday, June 18, 13
  • Tuesday, June 18, 13
  • PIGTuesday, June 18, 13
  • PIGThat same example using MR Java codeTuesday, June 18, 13
  • Tuesday, June 18, 13
  • The perfect Army!Tuesday, June 18, 13
  • The perfect Army!In MemoryTuesday, June 18, 13
  • The perfect Army!In MemoryIdentify and PlanTuesday, June 18, 13
  • The perfect Army!In MemoryConsumerIdentify and PlanTuesday, June 18, 13
  • The perfect Army!In MemoryKeep as much as you can IN MEMORY! Think Redis...ConsumerIdentify and PlanTuesday, June 18, 13
  • The perfect Army!In MemoryKeep as much as you can IN MEMORY! Think Redis...ConsumerIdentify and PlanWhat data can be batch processed and what can’t! ThinkHadoop and Storm (for stream) and HBase (for adhoc)Tuesday, June 18, 13
  • The perfect Army!In MemoryWho is the data consumer? Person or Process? Think Pig or xQL’s forboth!Keep as much as you can IN MEMORY! Think Redis...ConsumerIdentify and PlanWhat data can be batch processed and what can’t! ThinkHadoop and Storm (for stream) and HBase (for adhoc)Tuesday, June 18, 13
  • www.intridea.comAnthony NyströmFellow, Managing Directorof Engineeringanthony@intridea.com@AnthonyNystromThank YouGraciasMerciDankeTuesday, June 18, 13