Stream Analytics
•Download as PPTX, PDF•
0 likes•348 views
Between 2017 and 2022, the market for event stream processing platforms is estimated to grow 15% annually. Streaming analytics software can filter, aggregate, enrich, and analyze high throughput data from multiple sources to identify patterns and provide context to automate actions and dynamically adapt. Popular open source platforms include Apache Storm, Apache Flink, and Spark Streaming which provide real-time, parallel, and fault-tolerant processing of streaming data.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Stream Analytics
Similar to Stream Analytics (20)
More from Software Infrastructure
More from Software Infrastructure (20)
Recently uploaded
Recently uploaded (20)
Stream Analytics
- 1. Streaming Data Analytics Gizem Akman | Software Infrastructure Nov. 21, 2017
- 2. Between 2017 and 2022, Gartner estimates that the market for event stream processing (ESP) platforms will grow 15% year over year (compound annual growth rate).
- 3. Streaming Analytics – What? Analytics Real Time «Data in motion» Batch «Data at rest»
- 5. Streaming Analytics – What? Software that can filter, aggregate, enrich, and analyze a high throughput of data from multiple, disparate live data sources and in any data format to identify simple and complex patterns to provide applications with context to detect opportune situations, automate immediate actions, and dynamically adapt.
- 6. What is Real Time?
- 9. Paradigms – Processing Types • Atomic • Micro Batching • Windowing
- 10. Paradigms – Data Handling Guarantees • At Most Once • At Least Once • Exactly Once
- 11. Requirements & Characteristics • Data must be real time (current) • High volume & High Velocity Data • «Perishable Data» • Analytic logic must be predefined • Ultra-High performance messaging • Unbounded - Execution never stops
- 12. Streaming Analytics – How? • Analytic logic must be predefined • in-memory • Parallel – scale out • faster chips or GPUs • efficient algorithms (ex. minimizing context switches) • Leveraging innovative data architectures (ex. hashing) • Compromise on flexibility ( such as limiting random data access)
- 13. Zero-Copy
- 14. Popular Platforms Open Source Apache Storm Apache Flink Spark Streaming Apache Samza Vendors IBM Streams Software AG – Apama Streaming Analytics Azure Stream Analytics SAP Event Stream Processor Oracle Stream Analytics SAS Event Stream Processing Cisco Streaming Analytics Amazon Kinesis Google Cloud Dataflow TIBCO Event Analytics Informatica Striim DataTorrent StreamAnalytix SQLStream Blaze Data Artisans Impetus Technologies EsperTech
- 16. Basics • Free & open source distributed realtime computing engine «Hadoop for real time» • Fast over a 1M tuples processed per second per node
- 17. Architecture
- 18. Data Model
- 19. Groupings
- 21. Fault Tolerance «fail-fast, auto restart» • When a worker dies: the supervisor will restart it. If it continuously fails on startup and is unable to heartbeat to Nimbus, Nimbus will reschedule the worker. • When a node dies: The tasks assigned to that machine will time-out and Nimbus will reassign those tasks to other machines. • When Nimbus dies: The Nimbus is fail-fast (process self-destructs whenever any unexpected situation is encountered) and stateless (all state is kept in Zookeeper or on disk, so restart like nothing happened. • If you lose the Nimbus node, the workers will still continue to function. Additionally, supervisors will continue to restart workers if they die. However, without Nimbus, workers won't be reassigned to other machines when necessary (like if you lose a worker machine).
- 22. Parallelism
- 23. Parallelism
- 24. Integration • Apache Kafka • Apache Hbase • Apache HDFS • Apache Hive • Apache Solr • Apache Cassandra • JDBC • JMS • Redis • Event Hubs • Elasticsearch • MQTT • Mongodb • OpenTSDB • Kinesis • Druid • Kestrel With External Systems, and Other Libraries With Containers, and Resource Management Systems • YARN • Mesos • Docker • Kubernetes
- 25. And many others>> http://storm.apache.org/Powered-By.html
Editor's Notes
- *visualize real time *detect urgent situations *automate immediate actions Normal Programming >> code execution controls data Streaming Applications >> incoming data controls the code
- «loosely real time»
- pipeline
- Atomic: (aka one-key-value-at-a-time) Processes each inbound data event as a separate element. Seems logical but this is the most computationally expensive design. For example, it’s used to guarantee fastest processing of individual events with least delay in transmitting the event to the subscriber. Seen often for customer transactional inputs so that if some element of the event block fails the entire block is not deleted but moved to a bad record file that can later be processed further. Apache Storm uses this paradigm. Micro batching: Yes these are batches of events but typically those that occur within only a few milliseconds. You can adjust the time window. This makes the process somewhat more efficient. Spark Streaming uses this paradigm. Windowing: Similar to batching, Windowing allows the design of micro batches that may be simple time-based batches, but also allows for many more sophisticated interpretations such as sliding windows (e.g. everything that occurred in the last X period of time). This can be very useful for aggregating events or determining outliers when compared to averages or standard deviation.
- At most once: data loss possible At least once: no data loss- redelivery attemps made, duplication possible Exactly once: typically require use of idempotent updates
- Compute Intensity, the number of arithmetic operations per I/O or global memory reference. In many signal processing applications today it is well over 50:1 and increasing with algorithmic complexity. Data Parallelism exists in a kernel if the same function is applied to all records of an input stream and a number of records can be processed simultaneously without waiting for results from previous records. Data Locality is a specific type of temporal locality common in signal and media processing applications where data is produced once, read once or twice later in the application, and never read again. Intermediate streams passed between kernels as well as intermediate data within kernel functions can capture this locality directly using the stream processing programming model.
- Most of these can be hidden within modern analytics products so that the user does not have to be aware of exactly how they are being used. Analytic logic must be predefined – daha önceki datalara dayanmamalı, independent belirlenmiş kurallar pipelineından geçiyor Stream processing is essentially a compromise, driven by a data-centric model that works very well for traditional DSP or GPU-type applications (such as image, video and digital signal processing) but less so for general purpose processing with more randomized data access (such as databases). By sacrificing some flexibility in the model, the implications allow easier, faster and more efficient execution. Depending on the context, processor design may be tuned for maximum efficiency or a trade-off for flexibility.
- Zero-copy Kafka neden hızlı? **Arada context switch yok** Zero Copy - basically it calls the OS kernel direct rather than at the application layer to move data fast. Batch Data in Chunks - Kafka is all about batching the data into chunks. This minimises cross machine latency with all the buffering/copying that accompanies this. Avoids Random Disk Access - as Kafka is an immutable commit log it does not need to rewind the disk and do many random I/O operations and can just access the disk in a sequential manner. This enables it to get similar speeds from a physical disk compared with memory. Can Scale Horizontally - The ability to have thousands of partitions for a single topic spread among thousands of machines means Kafka can handle huge loads
- Storm is designed for massive scalability, supports fault-tolerance with a “fail fast, auto restart” approach to processes, and offers a strong guarantee that every tuple will be processed. Storm defaults to an “at least once” guarantee for messages, but offers the ability to implement “exactly once” processing as well. Storm is written primarily in Clojure and is designed to support wiring “spouts” (think input streams) and “bolts” (processing and output modules) together as a directed acyclic graph (DAG) called a topology. Storm topologies run on clusters and the Storm scheduler distributes work to nodes around the cluster, based on the topology configuration. You can think of topologies as roughly analogous to a MapReduce job in Hadoop, except that given Storm’s focus on real-time, stream-based processing, topologies default to running forever or until manually terminated. Once a topology is started, the spouts bring data into the system and hand the data off to bolts (which may in turn hand data to subsequent bolts) where the main computational work is done. As processing progresses, one or more bolts may write data out to a database or file system, send a message to another external system, or otherwise make the results of the computation available to the users. Hadoop: Store & Query. Store the data first, and indefinitely. Analyse when you like. Storm: Analyze while data is being produced. No need to store anything at all.
- Spouttan her tuple unique bir messageid ile çıkıyor ve bu messageidnin ack veya fail edilmesi gerekiyor. Ack edilmeyenler failed sayılıp spouttan tekrar tetikleniyor. Her boltun işlediği tupleı acklemesi lazım.
- Storm cbdevapp02 Redis nsidevapp01 redis/redis-4.0.2/src/redis-cli psubscribe WordCountTopology redis/redis-4.0.2/src/redis-server /WebSphere85/AppServer/java_1.8_64/bin/java -jar storm/Storm.jar 1>/dev/null 2>&1