Streaming and Messaging

•Download as PPTX, PDF•

0 likes•872 views

Part-1: Distributed Computing and Storage (Resource Management & Job/Data Management) Part-2: Streaming and Messaging (Apache Storm & Apache Kafka) Part-3: Streaming in User Behavior Analysis (Issues & Practices, New Streaming/Messaging Technologies)

Streaming & Messaging
- Xin Wang
July 2017, Beijing, Apache RocketMQ Meetup

Xin Wang
• Apache Storm Committer & PMC member
• Five years distributed system exprience
• Love open source
• Focus on distributed technologies, especially stream processing
• https://github.com/vesense

Index
Part-1: Distributed Computing and Storage
Part-2: Streaming and Messaging
Part-3: Streaming in User Behavior Analysis

01
Distributed Computing & Storage
「分而治之」

What's new in Storm 1.0
• Nimbus HA - Failover
• Pacemaker - Replaces Zookeeper for Heartbeats
• Streaming Window API
• Distributed Cache API - Allows sharing of files among topologies
• State Management - Statefule Bolts with Automatic Checkpointing
• Automatic Backpressure
• Resource Aware Scheduler
• Dynamic Log Levels
• Dynamic Worker Profiling
• Streaming SQL(since 1.1)
• ......

What's new in Storm 2.0
• Port Clojure to Java
• Lambda Expression Support - bolt: `tuple -> System.out.println(tuple)`
• Apache Beam Runner
• Storm-SQL Improvements
• Metrics V2 (or release in 1.2)
• Worker/Threading model Redesign
• Worker Classloader Isolation
• Dynamic Topology Updates
• ......

03
Streaming in User Behavior Analysis
「用之有方」

User Behavior Analysis 2.0
3G/4G
HBase
StormKafka
HDFS/Hive
MySQL
Solr
WLAN
Cable
Topic-In-1
Topic-In-N
Topic-Out-1
Topic-Out-N
Topology-1
Topology-2
Topology-3
Topology-N
Other

User Behavior Analysis 3.0 - Neymar
• Unified streaming API
• Pluggable Tagging module
• Dynamic Rule/Schema update
• UDF support for Schema fields
• Built-in main Serializer/Deserializer
• 16 vcore+32GB RAM *12node
• 60 billion/day, about 700K tps
• No data lose

Issues&Practices
• STORM-643 KafkaUtils repeatedly fetches messages whose offset is out of range
• STORM-329 Fix cascading Storm failure by improving reconnection strategy and buffering messages
• STORM-763 Nimbus reassigned worker A to another machine, but other worker's netty client can't
connect to the new worker A
• Worker down because of some Kafka ZK related bugs KAFKA-1382,KAFKA-1387,KAFKA-1451,KAFKA-
1585
• Be careful incompatible changes when upgrading Storm from 0.9 to 1.0. Scheme changed from
`deserialize(byte[] ser)` to `deserialize(ByteBuffer ser)`
• Worker heavy GCs due to large in-memory cache
• Put the lightweight logics into the same bolt
• Do NOT set too many executors/tasks in one worker, higher parallelism sometimes causes lower
throughput
• MQ->Streaming->MQ is better than MQ->Streaming->HDFS(throughput, delay, stability)
• Never log the logs unnecessay

Future
• Storm 2.0 & Kafka 0.11
• Apache Beam & Open-Messaging
• Investigating new Streaming Engine: Apache Flink, Spark-Streaming
• Investigating new MQ: Apache RocketMQ
• ROCKETMQ-81: RocketMQ-Spark Integration
• ROCKETMQ-82: RocketMQ-Flink Integration
• STORM-2349/ROCKETMQ-79: RocketMQ-Storm Integration

Storm-RocketMQ
• RocketMQSpout - Now only RocketMQ push mode supported, pull mode is in the plan.
The default Deserializer is StringScheme, you can override the value by setting
`RocketMQConfig.SCHEME`.
• RocketMQBolt - Async sending by default, or you can change the value by invoking
`withAsync(boolean async)`
• RocketMQState - For users using Storm Trident API
• TopicSelector - Selecting a topic based on the input Storm tuple
• TupleToMessageMapper - Mapping a Storm tuple to a RocketMQ message, you can
implement the MessageBodySerializer interface to serialize the message body. The
default implementation of MessageBodySerializer is
`body.toString().getBytes(StandardCharsets.UTF_8)`
• MessageRetryManager - Retry policy for failed messages
More details please refer to: https://github.com/apache/storm/tree/master/external/storm-rocketmq

This document discusses how to manage large amounts of data with Akka Streams. It introduces Reactive Streams, which defines standard interfaces for asynchronously processing potentially unbounded streams of data with backpressure. It discusses problems with traditional Java stream management approaches and how Akka Streams provides a better solution through composable, pure functional streams that allow data to be piped and transformed between a producer and consumer in a type-safe manner. Specific examples are given around using Akka Streams to handle large HTTP requests by normalizing and asynchronously saving file chunks to storage without buffering the entire file in memory.

Experience with Kafka & Storm

Otto Mok

This document discusses AcuityAds' use of Apache Kafka and Storm for processing over 10 billion daily ad impressions. It describes their architecture with Kafka used to ingest bid request data from multiple sources into partitions. Storm topologies read from Kafka and processed the data to calculate metrics like daily impressions by site. Initial issues included unbalanced Kafka partitions and low Storm uptime due to exceptions. Future improvements involved upgrading versions and adding monitoring capabilities.

Multi-tenant Apache Storm as a service

Robert Evans

Apache Kafka at LinkedIn

Discover Pinterest

Jay Kreps is a Principal Staff Engineer at LinkedIn where he is the lead architect for online data infrastructure. He is among the original authors of several open source projects including a distributed key-value store called Project Voldemort, a messaging system called Kafka, and a stream processing system called Samza. This talk gives an introduction to Apache Kafka, a distributed messaging system. It will cover both how Kafka works, as well as how it is used at LinkedIn for log aggregation, messaging, ETL, and real-time stream processing.

Real-time streaming and data pipelines with Apache Kafka

Joe Stein

Get up and running quickly with Apache Kafka http://kafka.apache.org/ * Fast * A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. * Scalable * Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers * Durable * Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact. * Distributed by Design * Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.

Stream Processing using Apache Spark and Apache Kafka

Abhinav Singh

This document provides an agenda for a session on Apache Spark Streaming and Kafka integration. It includes an introduction to Spark Streaming, working with DStreams and RDDs, an example of word count streaming, and steps for integrating Spark Streaming with Kafka including creating topics and producers. The session will also include a hands-on demo of streaming word count from Kafka using CloudxLab.

The document summarizes updates and new features in Apache Storm 1.0 including: - Pacemaker replaces Zookeeper for distributed coordination allowing larger clusters - A distributed cache API allows sharing files between topologies - High availability Nimbus allows Nimbus hosts to join/leave dynamically - Native streaming windows support event time processing and out of order tuples - State management provides automatic checkpointing for stateful bolts - Automatic back pressure throttles spouts when buffers reach high watermarks - The resource aware scheduler allocates tasks based on component resource requirements - Usability improvements include dynamic log levels, tuple sampling for debugging, and distributed log searching - New integrations support Cassandra, Sol

Spark streaming and Kafka

Iraj Hedayati

This document discusses two approaches for receiving data from Kafka in Spark Streaming - the receiver-based approach and direct approach. The receiver-based approach uses Kafka's high-level API and enables exactly-once processing semantics but requires writing to WAL. The direct approach fetches offsets manually, provides simplified parallelism with 1:1 mapping of partitions, and more efficiency without needing WAL, but does not guarantee exactly-once processing. It also covers how to set up a Spark Streaming application with Kafka, including library dependencies, Kafka consumer properties, subscribing to topics, and location strategies.

Kafka Summit NYC 2017 - Running Hundreds of Kafka Clusters with 5 People

confluent

Tom Crayford discusses his experience running hundreds of Apache Kafka clusters on Heroku with a small team. Some key points discussed include: - Using automation to manage clusters and reduce manual work required - Common issues encountered like disk growth from log compaction bugs and addressing them by scanning clusters for anomalies - Kafka's built-in high availability and how it helped during an AWS EBS failure event - Novel failure cases encountered like a JVM memory leak from gzip usage and working to fix it - Importance of taking breaks and not wasting time when operating clusters at scale.

Exactly-once Stream Processing with Kafka Streams

Guozhang Wang

kafka

Ariel Moskovich

This document discusses the evolution of Kafka clusters at AppsFlyer over time. The initial cluster had 4 brokers and handled hundreds of millions of messages with low partitioning and replication. A new cluster was designed with more brokers, replication across availability zones, and higher partitioning to support billions of messages. However, this led to issues like uneven leader distribution and failures. Various solutions were implemented like increasing brokers, splitting topics, and hardware upgrades. Ongoing testing and monitoring helped identify more problems and improvements around replication, partitioning, and automation. Key lessons learned included balancing replication and leaders, supporting dynamic changes, and thorough testing of failure scenarios.

Integrating Apache Pulsar with Big Data Ecosystem

StreamNative

Lessons from managing a Pulsar cluster (Nutanix)

StreamNative

In this presentation, we will cover: - How to performance test and optimize a Pulsar cluster. We will present how we load tested Pulsar with locust and, following this, how we tuned our configurations for our use cases. - Event sourcing pattern with Apache Pulsar. Avro schema usage, compatibility choices and schema evolution on pulsar topics that worked for us. - Bonus: How we source Apache Flink from apache pulsar and run our workflows. By attending this webinar, you can expect to come away with: - How to performance test a Pulsar cluster for your use case. - How to leverage the highly configurable broker and Bookkeeper to suit your needs. - Event sourcing patterns on top of Apache Pulsar. - Avro schema usage, compatibility choices, and evolution. - Familiarise with pulsar connector for Flink and possible use cases.

Introducing Exactly Once Semantics To Apache Kafka

Apurva Mehta

Multi cluster, multitenant and hierarchical kafka messaging service slideshare

Allen (Xiaozhong) Wang

This document discusses Netflix's approach to scaling Apache Kafka through a hierarchical, multi-cluster architecture. It addresses the challenges of scaling a single Kafka cluster, such as increased latency and vulnerability to failures. Netflix uses multiple "fronting" Kafka clusters for data collection and buffering optimized for producers. Consumer clusters scale by adding brokers or moving partitions to new clusters. Producers and consumers can operate across multiple clusters simultaneously using routing services. This multi-cluster approach allows independent scaling of producers and consumers while balancing data distribution.

Spark streaming + kafka 0.10

Joan Viladrosa Riera

Spark Streaming has supported Kafka since it's inception, but a lot has changed since those times, both in Spark and Kafka sides, to make this integration more fault-tolerant and reliable.Apache Kafka 0.10 (actually since 0.9) introduced the new Consumer API, built on top of a new group coordination protocol provided by Kafka itself. So a new Spark Streaming integration comes to the playground, with a similar design to the 0.8 Direct DStream approach. However, there are notable differences in usage, and many exciting new features. In this talk, we will cover what are the main differences between this new integration and the previous one (for Kafka 0.8), and why Direct DStreams have replaced Receivers for good. We will also see how to achieve different semantics (at least one, at most one, exactly once) with code examples. Finally, we will briefly introduce the usage of this integration in Billy Mobile to ingest and process the continuous stream of events from our AdNetwork.

Query Pulsar Streams using Apache Flink

StreamNative

This document discusses querying Apache Pulsar streams using Apache Flink. It provides an overview of Pulsar and how it can be used for pub/sub messaging and streaming applications. It then describes how Flink connectors allow querying Pulsar streams through Flink's streaming and table APIs, including support for Pulsar schemas, exactly-once processing, and integration with the Pulsar catalog. The integration provides a unified data processing stack of Pulsar for streaming data storage and Flink for querying and analysis.

What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019

confluent

Data stream processing is built on the core concept of time. However, understanding time semantics and reasoning about time is not simple, especially if deterministic processing is expected. In this talk, we explain the difference between processing, ingestion, and event time and what their impact is on data stream processing. Furthermore, we explain how Kafka clusters and stream processing applications must be configured to achieve specific time semantics. Finally, we deep dive into the time semantics of the Kafka Streams DSL and KSQL operators, and explain in detail how the runtime handles time. Apache Kafka offers many ways to handle time on the storage layer, ie, the brokers, allowing users to build applications with different semantics. Time semantics in the processing layer, ie, Kafka Streams and KSQL, are even richer, more powerful, but also more complicated. Hence, it is paramount for developers, to understand different time semantics and to know how to configure Kafka to achieve them. Therefore, this talk enables developers to design applications with their desired time semantics, help them to reason about the runtime behavior with regard to time, and allow them to understand processing/query results.

Kafka Summit NYC 2017 - Introducing Exactly Once Semantics in Apache Kafka

confluent

The document introduces Apache Kafka's new exactly once semantics that provide exactly once, in-order delivery of records per partition and atomic writes across multiple partitions. It discusses the existing at-least once delivery semantics and issues around duplicates. The new approach uses idempotent producers, sequence numbers, and transactions to ensure exactly once delivery and coordination across partitions. It also provides up to 20% higher throughput for producers and 50% for consumers through more efficient data formatting and batching. The new features are available in Apache Kafka 0.11 released in June 2017.

Real Time Streaming Data with Kafka and TensorFlow (Yong Tang, MobileIron) Ka...

confluent

This document discusses using Kafka and TensorFlow for real-time streaming machine learning. It introduces TensorFlow 2.0 capabilities for data processing and machine learning. It then discusses challenges in integrating streaming data with machine learning frameworks and formats. It proposes using KafkaDataset, a TensorFlow dataset for reading and writing from Kafka. KafkaDataset allows streaming data in and out of TensorFlow models for real-time inference and prediction.

Kafka meetup JP #3 - Engineering Apache Kafka at LINE

kawamuray

This document summarizes a presentation about engineering Apache Kafka at LINE. Some key points: - LINE uses Apache Kafka as a central data hub to pass data between services, handling over 140 billion messages per day. - Data stored in Kafka includes application logs, data mutations, and task requests. This data is used for tasks like data replication, analytics, and asynchronous processing. - Performance optimizations have led to target latencies below 1ms for 50% of produces and below 10ms for 99% of produces. - SystemTap, a Linux tracing tool, helped identify slow disk reads causing delayed Kafka responses, improving performance. - Having a single Kafka cluster as a data hub makes inter-service

When it Absolutely, Positively, Has to be There: Reliability Guarantees in Ka...

confluent

In the financial industry, losing data is unacceptable. Financial firms are adopting Kafka for their critical applications. Kafka provides the low latency, high throughput, high availability, and scale that these applications require. But can it also provide complete reliability? As a system architect, when asked “Can you guarantee that we will always get every transaction,” you want to be able to say “Yes” with total confidence. In this session, we will go over everything that happens to a message – from producer to consumer, and pinpoint all the places where data can be lost – if you are not careful. You will learn how developers and operation teams can work together to build a bulletproof data pipeline with Kafka. And if you need proof that you built a reliable system – we’ll show you how you can build the system to prove this too.

Introducing Exactly Once Semantics in Apache Kafka with Matthias J. Sax

Databricks

HBaseCon 2015: Blackbird Collections - In-situ Stream Processing in HBase

HBaseCon

Blackbird is a large-scale object store built at Rocket Fuel, which stores 100+ TB of data and provides real time access to 10 billion+ objects in a 2-3 milliseconds at a rate of 1 million+ times per second. In this talk (an update from HBaseCon 2014), we will describe Blackbird's comprehensive collections API and various examples of how it can be used to model collections like sets, maps, and aggregates on these collections like counters, etc. We will also illustrate the flexibility and power of the API by modeling custom collection types that are unique to the Rocket Fuel context.

Have your cake and eat it too

Gwen (Chen) Shapira

Current and Future of Apache Kafka

Joe Stein

3.2 Streaming and Messaging

振东刘

Kafka Explainaton

NguyenChiHoangMinh

What's hot

The Future of Apache Storm

P. Taylor Goetz

Spark streaming and Kafka

Iraj Hedayati

Kafka Summit NYC 2017 - Running Hundreds of Kafka Clusters with 5 People

confluent

Exactly-once Stream Processing with Kafka Streams

Guozhang Wang

kafka

Ariel Moskovich

Integrating Apache Pulsar with Big Data Ecosystem

StreamNative

Lessons from managing a Pulsar cluster (Nutanix)

StreamNative

Introducing Exactly Once Semantics To Apache Kafka

Apurva Mehta

Multi cluster, multitenant and hierarchical kafka messaging service slideshare

Allen (Xiaozhong) Wang

Spark streaming + kafka 0.10

Joan Viladrosa Riera

Query Pulsar Streams using Apache Flink

StreamNative

What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019

confluent

Kafka Summit NYC 2017 - Introducing Exactly Once Semantics in Apache Kafka

confluent

Real Time Streaming Data with Kafka and TensorFlow (Yong Tang, MobileIron) Ka...

confluent

Kafka meetup JP #3 - Engineering Apache Kafka at LINE

kawamuray

When it Absolutely, Positively, Has to be There: Reliability Guarantees in Ka...

confluent

Introducing Exactly Once Semantics in Apache Kafka with Matthias J. Sax

Databricks

HBaseCon 2015: Blackbird Collections - In-situ Stream Processing in HBase

HBaseCon

Have your cake and eat it too

Gwen (Chen) Shapira

Current and Future of Apache Kafka

Joe Stein

What's hot (20)

The Future of Apache Storm

Spark streaming and Kafka

Kafka Summit NYC 2017 - Running Hundreds of Kafka Clusters with 5 People

Exactly-once Stream Processing with Kafka Streams

kafka

Integrating Apache Pulsar with Big Data Ecosystem

Lessons from managing a Pulsar cluster (Nutanix)

Introducing Exactly Once Semantics To Apache Kafka

Multi cluster, multitenant and hierarchical kafka messaging service slideshare

Spark streaming + kafka 0.10

Query Pulsar Streams using Apache Flink

What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019

Kafka Summit NYC 2017 - Introducing Exactly Once Semantics in Apache Kafka

Real Time Streaming Data with Kafka and TensorFlow (Yong Tang, MobileIron) Ka...

Kafka meetup JP #3 - Engineering Apache Kafka at LINE

When it Absolutely, Positively, Has to be There: Reliability Guarantees in Ka...

Introducing Exactly Once Semantics in Apache Kafka with Matthias J. Sax

HBaseCon 2015: Blackbird Collections - In-situ Stream Processing in HBase

Have your cake and eat it too

Current and Future of Apache Kafka

Similar to Streaming and Messaging

3.2 Streaming and Messaging

振东刘

Kafka Explainaton

NguyenChiHoangMinh

Past, Present, and Future of Apache Storm

P. Taylor Goetz

This document discusses the past, present, and future of Apache Storm. It summarizes key releases of Storm including 0.9.x which moved Storm to Apache, 0.10.x which focused on enterprise readiness, 1.0 which improved performance and maturity, and 1.1.0 which added features like streaming SQL. It outlines current features such as the Resource Aware Scheduler, Kafka integration improvements, and new integrations. It concludes by discussing Storm 2.0 and beyond, including transitions to Java from Clojure and improved performance, as well as integrating Apache Beam and allowing bounded spouts and dynamic topology updates.

HTTP/2 Comes to Java: Servlet 4.0 and what it means for the Java/Jakarta EE e...

Edward Burns

Servlet is very easily the most important standard in server-side Java. The much awaited HTTP/2 standard is now complete, was fifteen years in the making and promises to radically speed up the entire web through a series of fundamental protocol optimizations. In this session we will take a detailed look at the changes in HTTP/2 and discuss how it may change the Java ecosystem including the foundational Servlet 4 specification included in Java/Jakarta EE 8.

World of Tanks Experience of Using Kafka

Levon Avakyan

DataConf.TW2018: Develop Kafka Streams Application on Your Laptop

Yu-Jhe Li

This document discusses developing Kafka Streams applications on a laptop by: 1) Explaining what Kafka Streams is and its architecture using processors, topologies, and state stores. 2) Detailing how to set up a local Kafka and Zookeeper environment using Docker Compose. 3) Walking through a sample Kafka Streams application that joins sample user data from two topics and demonstrates updating state.

Lessons Learned: Using Spark and Microservices

Alexis Seigneurin

The document discusses lessons learned from building a real-time data processing platform using Spark and microservices. Key aspects include: - A microservices-inspired architecture was used with Spark Streaming jobs processing data in parallel and communicating via Kafka. - This modular approach allowed for independent development and deployment of new features without disrupting existing jobs. - While Spark provided batch and streaming capabilities, managing resources across jobs and achieving low latency proved challenging. - Alternative technologies like Kafka Streams and Confluent's Schema Registry were identified to improve resilience, schemas, and processing latency. - Overall the platform demonstrated strengths in modularity, A/B testing, and empowering data scientists, but faced challenges around

Apache Kafka - Scalable Message-Processing and more !

Guido Schmutz

Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target. This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.

Storm worker redesign

Roshan Naik

The document discusses proposed enhancements to improve the performance of Apache Storm 2.0. It analyzes the current messaging architecture and identifies bottlenecks. Preliminary testing shows the redesigned messaging architecture improves latency by 116x and increases throughput by 50% over Storm 1.0 and the 2.0 master branch. Further optimizations to grouping, tuple implementation, and the acking mechanism could potentially yield even higher throughput of 15 million tuples per second. A new threading and execution model is also proposed to improve performance.

A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0

Legacy Typesafe (now Lightbend)

Reactive Streams 1.0.0 is now live, and so are our implementations in Akka Streams 1.0 and Slick 3.0. Reactive Streams is an engineering collaboration between heavy hitters in the area of streaming data on the JVM. With the Reactive Streams Special Interest Group, we set out to standardize a common ground for achieving statically-typed, high-performance, low latency, asynchronous streams of data with built-in non-blocking back pressure—with the goal of creating a vibrant ecosystem of interoperating implementations, and with a vision of one day making it into a future version of Java. Akka (recent winner of “Most Innovative Open Source Tech in 2015”) is a toolkit for building message-driven applications. With Akka Streams 1.0, Akka has incorporated a graphical DSL for composing data streams, an execution model that decouples the stream’s staged computation—it’s “blueprint”—from its execution (allowing for actor-based, single-threaded and fully distributed and clustered execution), type safe stream composition, an implementation of the Reactive Streaming specification that enables back-pressure, and more than 20 predefined stream “processing stages” that provide common streaming transformations that developers can tap into (for splitting streams, transforming streams, merging streams, and more). Slick is a relational database query and access library for Scala that enables loose-coupling, minimal configuration requirements and abstraction of the complexities of connecting with relational databases. With Slick 3.0, Slick now supports the Reactive Streams API for providing asynchronous stream processing with non-blocking back-pressure. Slick 3.0 also allows elegant mapping across multiple data types, static verification and type inference for embedded SQL statements, compile-time error discovery, and JDBC support for interoperability with all existing drivers.

Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example

confluent

This document introduces Kafka Streams and provides an example of using it to process streaming data from Apache Kafka. It summarizes some key limitations of using Apache Spark for streaming use cases with Kafka before demonstrating how to build a simple text processing pipeline with Kafka Streams. The document also discusses parallelism, state stores, aggregations, joins and deployment considerations when using Kafka Streams. It provides an example of how Kafka Streams was used to aggregate metrics from multiple instances of an application into a single stream.

Stream processing using Kafka

Knoldus Inc.

Cloud lunch and learn real-time streaming in azure

Timothy Spann

Apache Kafka

emreakis

Apache Kafka is a distributed streaming platform used for building real-time data pipelines and streaming apps. It provides a unified, scalable, and durable platform for handling real-time data feeds. Kafka works by accepting streams of records from one or more producers and organizing them into topics. It allows both storing and forwarding of these streams to consumers. Producers write data to topics which are replicated across clusters for fault tolerance. Consumers can then read the data from the topics in the order it was produced. Major companies like LinkedIn, Yahoo, Twitter, and Netflix use Kafka for applications like metrics, logging, stream processing and more.

Stream processing in python with Apache Samza and Beam

Hai Lu

Apache Samza is the streaming engine being used at LinkedIn that processes around 2 trillion messages daily. A while back we announced Samza's integration with Apache Beam, a great success which leads to our Samza Beam API. Now an UPGRADE of our APIs - we're now supporting Stream Processing in Python! This work has made stream processing more accessible and enabled many interesting use cases, particularly in the area of machine learning. The Python API is based on our work of Samza runner for Apache Beam. In this talk, we will quickly review our work on Samza runner, and then how we extended it to support portability in Beam (Python specifically). In addition to technical and architectural details, we will also talk about how we bridged Python and Java ecosystems at LinkedIn with the Python API, together with different use cases.

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...

Athens Big Data

OpenStack and Windows

Alessandro Pilotti

OpenStack is an open source cloud computing platform that can manage large networks of virtual machines and physical servers. It uses a distributed architecture with components like Nova (compute), Swift (object storage), Cinder (block storage), and Quantum (networking). OpenStack has been successful due to its scalability, support for multiple hypervisors including Hyper-V, and compatibility with popular programming languages like Python. While OpenStack is best suited for large public and private clouds, its complex installation and lack of unified deployment tools can present challenges, especially for small to mid-sized clouds.

Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)

Kai Wähner

[Big Data Spain] Apache Spark Streaming + Kafka 0.10: an Integration Story

Joan Viladrosa Riera

This document provides an overview of Apache Kafka and Apache Spark Streaming and their integration. It discusses what Kafka and Spark Streaming are, how they work, their benefits, and semantics when used together. It also provides examples of code for using the new Kafka integration in Spark 2.0+, including getting metadata, storing offsets in Kafka, and achieving at-most-once, at-least-once, and exactly-once processing semantics. Finally, it shares some insights into how Billy Mobile uses Spark Streaming with Kafka to process large volumes of data.

Openstack Cactus Survey

Pjack Chen

OpenStack is an open-source cloud computing platform that was launched in 2010. It provides services for managing compute, storage, and networking resources in a data center. The core OpenStack services are Nova (compute), Swift (object storage), and Glance (image service). OpenStack uses a RESTful API and supports many hypervisors, operating systems, and storage backends. It allows for public and private cloud deployments and supports common industry standards.

Similar to Streaming and Messaging (20)

3.2 Streaming and Messaging

Kafka Explainaton

Past, Present, and Future of Apache Storm

HTTP/2 Comes to Java: Servlet 4.0 and what it means for the Java/Jakarta EE e...

World of Tanks Experience of Using Kafka

DataConf.TW2018: Develop Kafka Streams Application on Your Laptop

Lessons Learned: Using Spark and Microservices

Apache Kafka - Scalable Message-Processing and more !

Storm worker redesign

A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0

Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example

Stream processing using Kafka

Cloud lunch and learn real-time streaming in azure

Apache Kafka

Stream processing in python with Apache Samza and Beam

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...

OpenStack and Windows

Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)

[Big Data Spain] Apache Spark Streaming + Kafka 0.10: an Integration Story

Openstack Cactus Survey

Recently uploaded

GenAI Pilot Implementation in the organizations

kumardaparthi1024

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

shyamraj55

Monitoring and Managing Anomaly Detection on OpenShift.pdf

Tosin Akinosho

Monitoring and Managing Anomaly Detection on OpenShift Overview Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices. Key Topics Covered 1. Introduction to Anomaly Detection - Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems. 2. Understanding Edge (IoT) - Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source. 3. What is ArgoCD? - Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices. 4. Deployment Using ArgoCD for Edge Devices - Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD. 5. Introduction to Apache Kafka and S3 - Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions. 6. Viewing Kafka Messages in the Data Lake - Learn how to view and analyze Kafka messages stored in a data lake for better insights. 7. What is Prometheus? - Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices. 8. Monitoring Application Metrics with Prometheus - Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system. 9. What is Camel K? - Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes. 10. Configuring Camel K Integrations for Data Pipelines - Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow. 11. What is a Jupyter Notebook? - Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text. 12. Jupyter Notebooks with Code Examples - Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.

Taking AI to the Next Level in Manufacturing.pdf

ssuserfac0301

Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as: 1. How quickly AI is being implemented in manufacturing. 2. Which barriers stand in the way of AI adoption. 3. How data quality and governance form the backbone of AI. 4. Organizational processes and structures that may inhibit effective AI adoption. 6. Ideas and approaches to help build your organization's AI strategy.

How to Get CNIC Information System with Paksim Ga.pptx

danishmna97

みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...

名前です男

Video Streaming: Then, Now, and in the Future

Alpen-Adria-Universität

In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.

Climate Impact of Software Testing at Nordic Testing Days

Kari Kakkonen

My slides at Nordic Testing Days 6.6.2024 Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.

HCL Notes and Domino License Cost Reduction in the World of DLAU

panagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/ The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this! We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model. Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward. These topics will be covered - Reducing license cost by finding and fixing misconfigurations and superfluous accounts - How do CCB and CCX licenses really work? - Understanding the DLAU tool and how to best utilize it - Tips for common problem areas, like team mailboxes, functional/test users, etc - Practical examples and best practices to implement right away

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

Malak Abu Hammad

Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers: * What is Vector Search? * Importance and benefits of vector search * Practical use cases across various industries * Step-by-step implementation guide * Live demos with code snippets * Enhancing LLM capabilities with vector search * Best practices and optimization strategies Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications. #MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology

Columbus Data & Analytics Wednesdays - June 2024

Jason Packer

Full-RAG: A modern architecture for hyper-personalization

Zilliz

Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.

Choosing The Best AWS Service For Your Website + API.pptx

Brandon Minnick, MBA

Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API? Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose? Which one is cheapest? Which one is fastest? Which one will scale to meet our needs? Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!

How to use Firebase Data Connect For Flutter

Daiki Mogmet Ito

Things to Consider When Choosing a Website Developer for your Website | FODUU

FODUU

Programming Foundation Models with DSPy - Meetup Slides

Zilliz

20240605 QFM017 Machine Intelligence Reading List May 2024

Matthew Sinclair

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?

Speck&Tech

ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune. Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile. BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).

Infrastructure Challenges in Scaling RAG with Custom AI models

Zilliz

Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.

20240609 QFM020 Irresponsible AI Reading List May 2024

Matthew Sinclair

Recently uploaded (20)

GenAI Pilot Implementation in the organizations

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

Monitoring and Managing Anomaly Detection on OpenShift.pdf

Taking AI to the Next Level in Manufacturing.pdf

How to Get CNIC Information System with Paksim Ga.pptx

Video Streaming: Then, Now, and in the Future

Climate Impact of Software Testing at Nordic Testing Days

HCL Notes and Domino License Cost Reduction in the World of DLAU

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

Columbus Data & Analytics Wednesdays - June 2024

Full-RAG: A modern architecture for hyper-personalization

Choosing The Best AWS Service For Your Website + API.pptx

How to use Firebase Data Connect For Flutter

Things to Consider When Choosing a Website Developer for your Website | FODUU

Programming Foundation Models with DSPy - Meetup Slides

20240605 QFM017 Machine Intelligence Reading List May 2024

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?

Infrastructure Challenges in Scaling RAG with Custom AI models

20240609 QFM020 Irresponsible AI Reading List May 2024

Streaming and Messaging

1. Streaming & Messaging - Xin Wang July 2017, Beijing, Apache RocketMQ Meetup

2. Xin Wang • Apache Storm Committer & PMC member • Five years distributed system exprience • Love open source • Focus on distributed technologies, especially stream processing • https://github.com/vesense

3. Index Part-1: Distributed Computing and Storage Part-2: Streaming and Messaging Part-3: Streaming in User Behavior Analysis

4. 01 Distributed Computing & Storage 「分而治之」

5. Distributed Computing

6. Distributed Storage

7. 02 Streaming & Messaging 「取之有道」

8. Streaming

9. What's new in Storm 1.0 • Nimbus HA - Failover • Pacemaker - Replaces Zookeeper for Heartbeats • Streaming Window API • Distributed Cache API - Allows sharing of files among topologies • State Management - Statefule Bolts with Automatic Checkpointing • Automatic Backpressure • Resource Aware Scheduler • Dynamic Log Levels • Dynamic Worker Profiling • Streaming SQL(since 1.1) • ......

10. What's new in Storm 2.0 • Port Clojure to Java • Lambda Expression Support - bolt: `tuple -> System.out.println(tuple)` • Apache Beam Runner • Storm-SQL Improvements • Metrics V2 (or release in 1.2) • Worker/Threading model Redesign • Worker Classloader Isolation • Dynamic Topology Updates • ......

11. Messaging

12. 03 Streaming in User Behavior Analysis 「用之有方」

13. User Behavior Analysis 1.0

14. User Behavior Analysis 2.0 3G/4G HBase StormKafka HDFS/Hive MySQL Solr WLAN Cable Topic-In-1 Topic-In-N Topic-Out-1 Topic-Out-N Topology-1 Topology-2 Topology-3 Topology-N Other

15. User Behavior Analysis 3.0 - Neymar • Unified streaming API • Pluggable Tagging module • Dynamic Rule/Schema update • UDF support for Schema fields • Built-in main Serializer/Deserializer • 16 vcore+32GB RAM *12node • 60 billion/day, about 700K tps • No data lose

16. Issues&Practices • STORM-643 KafkaUtils repeatedly fetches messages whose offset is out of range • STORM-329 Fix cascading Storm failure by improving reconnection strategy and buffering messages • STORM-763 Nimbus reassigned worker A to another machine, but other worker's netty client can't connect to the new worker A • Worker down because of some Kafka ZK related bugs KAFKA-1382,KAFKA-1387,KAFKA-1451,KAFKA- 1585 • Be careful incompatible changes when upgrading Storm from 0.9 to 1.0. Scheme changed from `deserialize(byte[] ser)` to `deserialize(ByteBuffer ser)` • Worker heavy GCs due to large in-memory cache • Put the lightweight logics into the same bolt • Do NOT set too many executors/tasks in one worker, higher parallelism sometimes causes lower throughput • MQ->Streaming->MQ is better than MQ->Streaming->HDFS(throughput, delay, stability) • Never log the logs unnecessay

17. Future • Storm 2.0 & Kafka 0.11 • Apache Beam & Open-Messaging • Investigating new Streaming Engine: Apache Flink, Spark-Streaming • Investigating new MQ: Apache RocketMQ • ROCKETMQ-81: RocketMQ-Spark Integration • ROCKETMQ-82: RocketMQ-Flink Integration • STORM-2349/ROCKETMQ-79: RocketMQ-Storm Integration

18. Storm-RocketMQ • RocketMQSpout - Now only RocketMQ push mode supported, pull mode is in the plan. The default Deserializer is StringScheme, you can override the value by setting `RocketMQConfig.SCHEME`. • RocketMQBolt - Async sending by default, or you can change the value by invoking `withAsync(boolean async)` • RocketMQState - For users using Storm Trident API • TopicSelector - Selecting a topic based on the input Storm tuple • TupleToMessageMapper - Mapping a Storm tuple to a RocketMQ message, you can implement the MessageBodySerializer interface to serialize the message body. The default implementation of MessageBodySerializer is `body.toString().getBytes(StandardCharsets.UTF_8)` • MessageRetryManager - Retry policy for failed messages More details please refer to: https://github.com/apache/storm/tree/master/external/storm-rocketmq

19. Storm-RocketMQ

20. Thanks everyone!

Streaming and Messaging

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Streaming and Messaging

Similar to Streaming and Messaging (20)

Recently uploaded

Recently uploaded (20)

Streaming and Messaging