Fast Data Open Source Platforms
•Download as PPTX, PDF•
3 likes•1,061 views
An overview of Fast Data, focusing on Open Source Platforms (Spark, Flink, Kafka, Storm, Samza, et al).
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Fast Data Open Source Platforms
Similar to Fast Data Open Source Platforms (20)
Recently uploaded
Recently uploaded (20)
Fast Data Open Source Platforms
- 2. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Fast Data Open Source An Overview Chuck Scyphers Big Data Lead East Coast
- 3. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 3
- 4. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 4 • 25+ years experience with highly available, highly scalable, high throughput, globally-spanning enterprise class systems. • 7 startups (2 wins, 2 break-evens, 3 losses); last two built on Hadoop and NoSQL systems (resume analytics and behavioral analysis of network traffic • Chief Data Architect, US-Visit (Department of State) and US Department of Energy SLD Project Chuck Scyphers
- 5. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Agenda 5 Fast Data Definition Popular Open Source Platforms What Do We Want To Be When We Grow Up? Refreshments
- 6. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Agenda 6 Fast Data Definition Popular Open Source Platforms What Do We Want To Be When We Grow Up? Refreshments
- 7. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | What Is Fast Data? “Fast data is the application of big data analytics to smaller data sets in near- real or real-time in order to solve a problem or create business value. The goal of fast data is to quickly gather and mine structured and unstructured data so that action can be taken.” 7
- 8. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | New Concepts for a Modern Data Platform Architecture Polyglot Fit for Purpose Data Lambda Speed Layer Batch Layer Data Sources Data Services Kappa Data Services Data PipelineData Sources
- 9. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | How Data Impacts The Organization 9 67% executives who say drawing intelligence from data is top priority Source: Oracle Research Study - From Overload to Impact: An Industry Scorecard on Big Data Business Challenges, July 2012
- 10. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | How Data Impacts The Organization 10 89% executives who would grade themselves C or lower in preparedness Source: Oracle Research Study - From Overload to Impact: An Industry Scorecard on Big Data Business Challenges, July 2012
- 11. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | How Data Impacts The Organization 11 % believe their organization is losing revenue as a result of not being able to fully leverage information Source: Oracle Research Study - From Overload to Impact: An Industry Scorecard on Big Data Business Challenges, July 2012
- 12. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Velocity Matters of executives say too much critical information is delivered too late Source: Aberdeen Group – January 2012, survey of 247 executives - Data Management for BI – Big Data, Bigger Insight, Superior Performance
- 13. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Why Do We Care? It’s about getting more from in-flight data It’s about faster action, faster insights It’s about visibility and predictability It’s about running your business in real-time
- 14. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Key Value Drivers of Timely Accurate Action 14 Delivering Tangible Results With Fast Data Higher Quality In Operations Improved Efficiency New Services Better Customer Experience
- 15. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Fast Data Is Universal Across Industries 15 Financial Services Transportation & Logistics Telecommunications Manufacturing & Retail Utilities & Oil and GasHealth carePublic Sector
- 16. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Fast Data Characteristics 16 ANALYZEMOVE & TRANSFORM FILTER & CORRELATE ACT
- 17. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Fast Data Characteristics Oracle Confidential – Internal/Restricted/Highly Restricted 17 ANALYZEMOVE & TRANSFORM FILTER & CORRELATE ACT Complete In-Flight Event Processing • Eliminate, Consolidate, Correlate, And/Or Filter Data While In Flight • Analyze Data Streams • Enrich Data For More Accurate Decisions • Process Data In The Stream To Free Up Back End Resources
- 18. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Fast Data Characteristics Oracle Confidential – Internal/Restricted/Highly Restricted 18 ANALYZEMOVE & TRANSFORM FILTER & CORRELATE ACT Work With The Stream • Apply Basic Filtering At Capture • Improve Trusted Quality Of Information • Move Data (duh)
- 19. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Fast Data Characteristics Oracle Confidential – Internal/Restricted/Highly Restricted 19 ANALYZEMOVE & TRANSFORM FILTER & CORRELATE ACT Speed Up The OODA Loop • Get Actionable Insights • Predict Outcomes
- 20. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Fast Data Characteristics Oracle Confidential – Internal/Restricted/Highly Restricted 20 ANALYZEMOVE & TRANSFORM FILTER & CORRELATE ACT Make Decisions That Matter Faster • Deliver Real-Time Decisions And Recommendations To Customers/Employees • Automatically Render Decisions Within A Process With Tailored Messaging • Integrate Human Workflow, Process Management, Activity Monitoring
- 21. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Fast Data Customers Oracle Confidential – Internal/Restricted/Highly Restricted 21 From Oracle (naturally)
- 22. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Fast Data And Financial Services Oracle Confidential – Internal/Restricted/Highly Restricted 22 Improving Customer Experiences • Improve Customer Experience: The goal is to connect all data about the customers to improve customer service experience and to lower the burden of hiring new representatives. • Reduce Staffing Demands: For customers calling to discuss a claim or their coverage, it means fewer annoying waits as an agent accesses data from any of dozens of different places. • Consolidate information in real-time: All a customer’s transactions: claims, records, status, possible cross-sell information (e.g., if someone lives in an apartment and might need renter’s insurance)
- 23. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Fast Data And Retail Oracle Confidential – Internal/Restricted/Highly Restricted 23 • Price optimization - leveraging analytics to price goods and services on the fly based on real-time metrics such as competitor pricing, supply chain and inventory data, market data and consumer behavior data. • Product placement analysis - processing video data to identify shopping trends, assesses effectiveness of displays to improve store layouts and product placements. • Staffing - The largest retailers are analyzing weather forecasts, promotional campaigns and dates to effectively meet staffing requirements on holidays all year round.
- 24. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Fast Data And Public Sector Oracle Confidential – Internal/Restricted/Highly Restricted 24 LA City Planning And Traffic Analysis • Dynamic Pricing for Toll Lanes: if a driver is paying to drive in the HOT (high-occupancy tolling) lane, he’s guaranteed a consistent speed of 45 miles per hour. If traffic starts backing up, prices for individual cars will rise to discourage them from entering, saving the lanes for high-occupancy vehicles • Express Park: It’s not enough to know how to set the price, you have to make sure that data gets to users in real time. Drivers also need to know parking spots will still be there when they arrive in 40 minutes. • Combining M2M: The answer lies in combining information from other sources, such as mass-transit systems, toll highways, traffic sensors and weather data to paint a real-time picture of what traffic actually looks like
- 25. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Fast Data And Telecomm Oracle Confidential – Internal/Restricted/Highly Restricted 25 Location Based Mobile Billboard Advertising at Turkcell • Processing over 800,000 subscriber related events per second (with 1.5 Billion Events Daily) • Provided and executed over 50 simultaneous campaigns • Ensured customer responsiveness with less than 1 second times with a scalable architecture, ready to expand on demand
- 26. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Agenda 26 Fast Data Definition Popular Open Source Platforms What Do We Want To Be When We Grow Up? Refreshments
- 27. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | HDFS Based • Spark • HBase • Impala • H20 • Apex Other Based • Druid • Flink • Storm • Kafka • Samza • ElasticSearch • Lucene/Solr 27 Composite • SMACK • PANCAKE Open Source Platforms General Classifications
- 28. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | HDFS Based • Spark • HBase • Impala • H20 • Apex Other Based • Druid • Flink • ElasticSearch • Storm • Kafka • Lucene/Solr 28 Composite • SMACK • PANCAKE Open Source Platforms General Classifications
- 29. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Spark HDFS Based HDFS Based • Spark • HBase • Impala • H20 • Apex 29 • In-Memory Distributed Processing Framework • Will Spill To Disk As Needed • Handles Streaming Data Through Micro-batching
- 30. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | HBase HDFS Based HDFS Based • Spark* • HBase • Impala • H20 • Apex 30 • A NoSQL Columnar Store Built On Top Of HDFS • Provides A Big Table–esque Processing Model • Compression • In-memory • Bloom Filters By Column • Offers Both Real Time Read/Write Access And Random Access To HDFS
- 31. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Impala HDFS Based HDFS Based • Spark* • HBase • Impala • H20 • Apex 31 • Real-time SQL queries over data stored in HDFS or HBase • No MapReduce processing • Uses a MPP query engine on the Hadoop cluster • Utilizes Hive metastore for metadata repository • Leveraged by numerous BI tools and applications • Not ANSI SQL
- 32. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | H20 HDFS Based HDFS Based • Spark* • HBase • Impala • H20 • Apex 32
- 33. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Apex HDFS Based HDFS Based • Spark* • HBase • Impala • H20 • Apex 33
- 34. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | HDFS Based • Spark* • HBase • Impala • H20 • Apex Other Based • Druid • Flink • Storm • Kafka • Samza • ElasticSearch • Lucene/Solr 34 Composite • SMACK • PANCAKE Open Source Platforms General Classifications
- 35. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Open Source Platforms • Druid • Flink • Storm • Kafka • Samza • ElasticSearch • Lucene/Solr 35 Other Based
- 36. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Other Based • Druid • Flink • Storm • Kafka • Samza • ElasticSearch • Lucene/Solr 36 Druid MySQL Zookeeper
- 37. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Other Based • Druid • Flink • Storm • Kafka • Samza • ElasticSearch • Lucene/Solr 37 Flink
- 38. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Other Based • Druid • Flink • Storm • Kafka • Samza • ElasticSearch • Lucene/Solr 38 Storm
- 39. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Other Based • Druid • Flink • Storm • Kafka • Samza • ElasticSearch • Lucene/Solr 39 Kafka
- 40. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Samza Other Based 40 • Druid • Flink • Storm • Kafka • Samza • ElasticSearch • Lucene/Solr
- 41. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Search Based • Druid • Flink • Storm • Kafka • Samza • ElasticSearch • Lucene/Solr 41 ElasticSearch
- 42. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Search Based • Druid • Flink • Storm • Kafka • Samza • ElasticSearch • Lucene/Solr 42 Lucene/Solr
- 43. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | A Quick Comparison 43 Guarantee Throughput Fault Tolerance Overhead Computation Model Windowing Memory Management DAG Based Batch Support Latency Stateful Operations Spark Exactly Once 100k+ records/sec Low Microbatches Time Based Moving towards automatic yes Yes seconds yes Flink Exactly Once Low Continuous Flow Operation Record Based / User Defined Automatic Yes milliseconds Storm At least Once/Exactly Once (+ Trident) 100k+ records/sec Continuous Flow Operation yes No (unless paired with Trident) milliseconds no (unless with Trident) Samza At least Once 10k+ records/sec Continuous Flow Operation milliseconds yes Hadoop Lower High Batch Only Nope YARN is helping No Only
- 44. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | HDFS Based • Spark* • HBase • Impala • H20 • Apex Other Based • Druid • Flink • ElasticSearch • Storm • Kafka • Samza • Lucene/Solr 44 Composite • SMACK • PANCAKE Open Source Platforms General Classifications
- 45. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Open Source Platforms Composite • SMACK • PANCAKE 45 Composite
- 46. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Open Source Platforms • SMACK • PANCAKE 46 SMACK Stack Spark Mesos Akka Cassandra Kafka
- 47. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Open Source Platforms • SMACK • PANCAKE 47 PANCAKE Pile Presto Arrow NiFi Cassandra AirFlow Kafka Elastic Search
- 48. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Open Source Platforms • SMACK • PANCAKE 48 PANCAKE STACK Presto Arrow NiFi Cassandra AirFlow Kafka ElasticSearch Spark TensorFlow Algebird CoreNLP Kibana
- 49. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Agenda 49 Fast Data Definition Popular Open Source Platforms What Do We Want To Be When We Grow Up? Refreshments
- 50. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | What Do We Want This Meetup To Be? • How Often Do We Want To Meet? • Where? (other than here) • From Whom Do We Want To Hear? – Vendors? – Never Vendors? • Demos & Code? • Sponsors? • Who’s Hiring? Who’s Looking? 50
- 51. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Agenda 51 Fast Data Definition Popular Open Source Platforms What Do We Want To Be When We Grow Up? Refreshments
- 52. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Refreshments 52 Reston Town Center 1888 Explorer St Reston VA
- 53. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 53
Editor's Notes
- First, the industry has embraced polyglot persistence – accepting and embracing that we should store and manage data in a “fit for purpose” approach that is optimized for the data at hand. Second, that we can parallelize our MPP data foundation for both speed and size, this is crucial for next-gen data services and analytics that can scale to any latency and size requirements. Third, MPP data pipelines that allow us to treat data events in a moving time windows at variable latencies; in the long run this will change how we do ETL for most use cases.
- The value of applying a fast data strategy as an end-to-end solution has become more apparent across many industry segments; it’s also becoming more mainstream now. We’re seeing many of Oracle customers adopting solutions which address these requirements around fast data; for example: Transportation: Monitor all airline's operational events to eliminate flight delays Retail: Customer service centers are using Fast Data for click-stream analysis and customer experience management. Telco: Location based offers, or Intelligent Network Management to drive new services and lower costs. Healthcare: Monitoring Medical Device Data to help save lives Manufacturing: Real-time corrective action for reducing maintenance costs or risk outages
- Oracle Data Integration (Oracle GoldenGate and Oracle Data Integrator) provide best-in class real-time capture and big data transformation: Capture changed data and events in real time Apply basic filtering and transformation at capture point Transform and load structured or unstructured data for analysis Improve the trusted quality of information
- In-memory analytics to provide actionable insight from large amounts of data - fast. Oracle TimesTen In-Memory Database for Oracle Exalytics has also been certified to work with both Oracle GoldenGate Real-Time Replication technology, as well as Oracle Data Integrator, allowing more flexibility for customers to report on events as they happen. With these certifications, Oracle Exalytics data can be updated either via replication or via incremental updates, making the refreshes quicker and more efficient. Trust your data model is current and accurate : customers benefit from the Common enterprise information model provides centralized metadata management, common query request generation and data access, and a rich spectrum of visualization, collaboration, and search features Quickly organize and explore diverse and unstructured Big data from inside and outside your organization – Endeca allows business users to freely explore and discover meaningful new insight from both structured and unstructured sources to help identify root causes and new associations.
- In-memory analytics to provide actionable insight from large amounts of data - fast. Oracle TimesTen In-Memory Database for Oracle Exalytics has also been certified to work with both Oracle GoldenGate Real-Time Replication technology, as well as Oracle Data Integrator, allowing more flexibility for customers to report on events as they happen. With these certifications, Oracle Exalytics data can be updated either via replication or via incremental updates, making the refreshes quicker and more efficient. Trust your data model is current and accurate : customers benefit from the Common enterprise information model provides centralized metadata management, common query request generation and data access, and a rich spectrum of visualization, collaboration, and search features Quickly organize and explore diverse and unstructured Big data from inside and outside your organization – Endeca allows business users to freely explore and discover meaningful new insight from both structured and unstructured sources to help identify root causes and new associations.
- In-memory analytics to provide actionable insight from large amounts of data - fast. Oracle TimesTen In-Memory Database for Oracle Exalytics has also been certified to work with both Oracle GoldenGate Real-Time Replication technology, as well as Oracle Data Integrator, allowing more flexibility for customers to report on events as they happen. With these certifications, Oracle Exalytics data can be updated either via replication or via incremental updates, making the refreshes quicker and more efficient. Trust your data model is current and accurate : customers benefit from the Common enterprise information model provides centralized metadata management, common query request generation and data access, and a rich spectrum of visualization, collaboration, and search features Quickly organize and explore diverse and unstructured Big data from inside and outside your organization – Endeca allows business users to freely explore and discover meaningful new insight from both structured and unstructured sources to help identify root causes and new associations.
- In general, I see three main categories: HDFS ecosystem based, ones that are built on other ecosystems (and/or), and the composite systems
- HDFS based does not mean that is *must* be HDFS based; Spark and H20 can all run outside of HDFS. However, most times you will find them, there is going to be HDFS running around as well.
- An effort to make R both more scalable and faster. Can run on top of HDFS, but other platforms as well
- YARN based processing Includes Malhar, a “lego box” of premade operators and widgets to speed adoption
- An in memory OLAP store, designed to ingest event/log data, chunking and compressing that data into column-based queryable segments. Data Ingestion Data is ingested by Druid directly through its real-time nodes, or batch-loaded into historical nodes from a deep storage facility. Real-time nodes accept JSON-formatted data from a streaming datasource. Batch-loaded data formats can be JSON, CSV, or TSV. Real-time nodes temporarily store and serve data in real time, but eventually push the data to the deep storage facility, from which it is loaded into historical nodes. Historical nodes hold the bulk of data in the cluster. Real-time nodes chunk data into segments, and are designed to frequently move these segments out to deep storage. To maintain cluster awareness of the location of data, these nodes must interact with MySQL to update metadata about the segments, and with Apache ZooKeeper to monitor their transfer. Query Management Client queries first hit broker nodes, which forward them to the appropriate data nodes (either historical or real-time). Since Druid segments may be partitioned, an incoming query can require data from multiple segments and partitions (or shards) stored on different nodes in the cluster. Brokers are able to learn which nodes have the required data, and also merge partial results before returning the aggregated result. Cluster Management Operations relating to data management in historical nodes are overseen by coordinator nodes, which are the prime users of the MySQL metadata tables. Apache ZooKeeper is used to register all nodes, manage certain aspects of internode communications, and provide for leader elections. Features Low latency (real-time) data ingestion Arbitrary slice and dice data exploration Sub-second analytic queries Approximate and exact computations
- Flink is intended to be a framework for unified stream and batch process (Kappa architecture). Flink also can handle backpressure on the queues more gracefully than other platforms (::cough:: storm) Flink executes arbitrary dataflow programs in a data-parallel and pipelined manner.[3] Flink's pipelined runtime system enables the execution of bulk/batch and stream processing programs.[4][5] Furthermore, Flink's runtime supports the execution of iterative algorithms natively.[6
- Stream processor only. Bolts & Spouts
- Samza’s goal is to provide a lightweight framework for continuous data processing. Samza continuously computes results as data arrives which makes sub-second response times possible. It’s unlike batch processing systems such as Hadoop which typically has high-latency responses which can sometimes take hours. Samza might help you to update databases, compute counts or other aggregations, transform messages, or any number of other operations. It’s been in production at LinkedIn for several years and currently runs on hundreds of machines across multiple data centers. Our largest Samza job is processing more than one million messages per-second during peak traffic hours. Architecture & Concepts Streams and Jobs are the building blocks of a Samza application: A stream is composed of immutable sequences of messages of a similar type or category. In order to scale the system to handle large-scale data, we break down each stream into partitions. Within each partition, the sequence of messages is totally ordered and each message’s position is uniquely identified by its offset. At LinkedIn, streams are provided by Apache Kafka. A job is the code that consumes and processes a set of input streams. In order to scale the throughput of the stream processor, jobs are broken into smaller units of execution called Tasks. Each task consumes data from one or more partitions for each of the job’s input streams. Since there is no defined ordering of messages across the partitions, it allows tasks to operate independently. Samza assigns groups of tasks to be executed inside one or more containers – UNIX processes running a JVM that execute a set of Samza tasks for a single job. Samza’s container code is single threaded (when one task is processing a message, no other task in the container is active), and is responsible for managing the startup, execution, and shutdown of one or more tasks.
- SPARK – fast, general purpose engine for distributed processing (everywhere) MESOS – cluster management and resource isolation (blue hexagon) AKKA – runtime for highly concurrent, distributed, message-driven applications (blue triangle) CASSANDRA – distributed NoSQL store optimized for reads (eye) KAFKA – high throughput, low latency pub/sub messaging system (orange pipe)