Budapest Big Data Meetup Real-time stream processing
StarschemaExperience and Innovation
• Who we are and what we are doing• Big Data era• BSP (Bulk synchronous parallel)• Apache Giraph• Storm• Our use case• ConclusionTopics todayStarschemaExperience and Innovation
Continuous growth25 FTE plus external resources,over $1.5million EBITOpen source projectsShare the knowledge with the public.Open source project in ETL and datawarehousing fields.Founded in 2006Company was founded by privateowners with decade of BI and datawarehouse backgroundR&DCooperation with Obuda University,NKE, EU co-founded technologyresearch and developmentCOMPANY DataFacts about StarschemaStarschemaExperience and Innovation
Big Data eraThe rise of HadoopStarschemaExperience and InnovationGoogle Year of WP Apache YearGFS 2003 HDFS 2007MapReduce 2004 Hadoop MR 2007BigTable 2006 HBase 2007Chubby Lock Service 2006 ZooKeeper 2007Pregel 2009 Giraph 2011Dremel 2010 Drill 2012 ?Which is next? (Curator, Falcon, MRQL, etc.)
• Leslie Valiant - article in nov. 1990• Supersteps• Data stored in local memory• Asynchronous data processing• Barrier sync• Optimal load balacing (more logical processesthan physcal processors, random allocation ofprocesses)• Solution differences (procotols, buffermanagement, routing strategies)• No deadlock or any other race conditions(since no circular dependency)• Use casesBSP (Bulk synchronous parallel)What is it? What is it good for?StarschemaExperience and Innovation
Storm Apache GiraphStarschemaExperience and Innovation
Apache GiraphStarschemaExperience and Innovation• A loose implementation of Pregel• Avery Chink: We cant use it at Yahoo, thats too bad• Developed at Yahoo• Runs on existing MapReduce infrastructure• Netty based comm. instead of Hadoop RPC• In-memory• Fault tolerant• Internal state is saved at user-defined intervals• Master/slave architectureWhat is it?
StormStarschemaExperience and Innovation• Storm is a free and open source distributed real timecomputation system• Developed at BackType, open-sourced by Twitter in 2011• Guaranteed data processing• Horizontal scalability• Fault tolerance• ZeroMQ for message passing• Processing unboundedsequence of tuples• GroupingsWhat is it?
StormStarschemaExperience and InnovationWhat is it for?• Analyze, clean, normalize• Real-time calculation• Real-time ETL• Failure detection from log files• Machine data analysis• IT early-warning systems, security and fraud detection• Traffic information, DOS attack• Stream processing - Continous computation -Distributed RPC
Our use caseStarschemaExperience and Innovation• Real-time calculation• Error detection• Horizontal scalability• Fast implementation• High-availability• Error predictionPOC: Processing machine data from sensorsRequirements
Our use case part 2StarschemaExperience and Innovation• Choosen tool: Storm• One spout for each sensor• Dynamic add and remove of spouts• Error detection based on statistical calculations• ~ 200 lines• HA capability of StormPOC: Processing machine data from sensorsSolution:
Conclusion• Extend existing infrastructure• Answer to new questions• Re-think old problems• New solutions, new features• Happy customers/users• $$$StarschemaExperience and Innovation
StarschemaExperience and InnovationWhat else to use?• Yahoo S4 (Apache Incubator project)• Apache Hama (Top level Apache project)• GoldenOrb• Signal/Collect
QUESTIONS & ANSWERSQ…AStarschemaExperience and Innovationborosg@starschema.netwww.starschema.net